Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning
Authors: Sid Bharthulwar
Stone Tao
Hao Su
Venue: NeurIPS 2025 poster ()
Keywords: reinforcement learning
robotics
ppo
on-policy
simulation
Primary area: reinforcement_learning
TL;DR: Staggered resets fix harmful nonstationarity in massively parallel RL's short synchronous rollouts by varying environment start times, improving state coverage and boosting sample efficiency, performance, and scalability.

Abstract: Massively parallel GPU simulation environments have accelerated reinforcement learning (RL) research by enabling fast data collection for on-policy RL algorithms like Proximal Policy Optimization (PPO). To maximize throughput, it is common to use short rollouts per policy update, increasing the update-to-data (UTD) ratio. However, we find that, in this setting, standard synchronous resets introduce harmful nonstationarity, skewing the learning signal and destabilizing training. We introduce staggered resets, a simple yet effective technique where environments are initialized and reset at varied points within the task horizon. This yields training batches with more uniform state visitation distributions, reducing the nonstationarity induced by synchronized rollouts. We characterize dimensions along which RL environments can benefit significantly from staggered resets through toy environments. We then apply this technique to challenging high-dimensional robotics environments, achieving significantly higher sample efficiency, faster wall-clock convergence, and stronger final performance. Finally, this technique scales better with more parallel environments compared to naive synchronized rollouts, yielding more optimal utilization of computational resources.

Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning
Sid Bharthulwar
Stone Tao UC San Diego staoßuesd. edu
Harvard University sbharthullvar@colloge hww.://w.........a.....o.oo. harvard.
Hao Su UC San Diego
hao@sudo. ai
Abstract
Massively parallel GPU simulation environments have accelerated reinforcement learning (RL) research by enabling fast data collection for on-policy RL algorithms like Proximal Policy Optimization (PPO). To maximize throughput, it is common short rollouts per policy update, increasing the update-to-data (UTD) tio. However, we find that, in this setting, standard synchronous resets introduce harmful nonstationarity, skewing the learning signal and destabilizing training. We introduce staggered resets, a simple yet effective technique where environments are initialized and reset varied points within the task horizon. This yields training batches with greater temporal diversity, reducing the nonstationarity induced by synchronized rollouts. We characterize dimensions along which RL environments can benefit significantly from staggered resets through illustrative toy environ- ments. We then apply this technique challenging high-dimessional roboties environments, achieving significantly higher sample efficiency, faster wall-clock convergence, and stronger final performance. Finally, this technique scales better with more parallel environments compared to naive synchronized rollouts.
Introduction
Reinforcement Learning (RL) has emerged as a powerful paradigm for tackling complex sequential decision-making problems. particularly in continuous control domains like roboties [ [0]. How- ever, RL often depends on vast quantities of interaction data, requirement that can be prohibitively expensive or slow acquire in real-world settings. Massively parallel simulation environments, especially those accelerated on GPUs (14] 3) 20 16). have enabled data-generation throughput on orders of magnitude greater than traditional CPU-based setups. The increase in data throughput has enabled far faster training of robotics models with successful sim2real deployments of locomotion
21 15
state-based manipulation [ B
and vision- based manipulation 26 32 24
Despite this paradigm shift in data generation capabilities, the core algorithms, particularly on on-policy methods like Proximal Policy Optimization (PPO) [22], have often been adapted with only superficial Shanges-ttypially larger batch sizes and shorter per-environment rollouts (K) to increase the update- to-data (UTD) ratio (1 This strategy, while seemingly maximizing hardware utilization, overlooks a critical interaction between the data collection process and the learning algorithm's 's stability when the task horizon (H) significantly exceeds the rollout length (K < H).
We argue that stable and efficient learning in this massively parallel regime requires more than just algorithmic re-tuning; it necessitates modification the environment interaction protocol itself.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

We introduce staggered resets, simple yet highly effective technique that breaks this harmful synchronicity. By initializing parallel environments diverse effective time steps distributed across the task horizon H, staggered resets ensure that each training batch contains rich, temporally heterogeneous mix of experiences. This provides the learner with a more stationary and representative view of the overall task dynamics within every gradient update. To summarize our contributions:
We precisely identify, evidence, and formulate the problem of cyclical batch nonstationarity stemming from synchronous full-episode resets combined with short rollouts (K < H) in massively parallel on-policy RI., explaining its detrimental impact on learning dynamics.
We propose staggered resets, an elegant and easily implementable mechanism independent of the RL algorithm itself to ensure temporal diversity within training batches by desynchro- nizing the effective starting points of parallel environments across the task horizon.
Through illustrative toy environments, we characterize the conditions under which this nonstationarity is most severe and staggered resets offer maximal benefit.
We provide compelling empirical evidence on challenging, high-dimensional robotics tasks, demonstrating that staggered resets significantly improve sample efficiency, wall-clock convergence speed, final policy performance, and scalability with increasing parallelism compared to standard synchronous reset protocols.
2
Related Work
Massively Parallel RL The inherent sample inefficiency of many reinforcement learning algorithms has significant research into scaling and parallelization to improve training time and performance. Early approaches, such as IMPALA and others 2 18 6 utilized multiple CPU workers to achieve parallelism and scalability. These typically relied on benchmarks 31
28 built on top of CPU-based simulators like MuJoCo (27), PyBullet [i. PhysX etc. More recently, GPU-accelerated simulators and other JAX-based GPU-accelerated environments 20
9 have enabled much greater degree of parallelism, resulting in considerable speedups for training complex policies. Recent work has sought to replace the popular choice of PPO as RL algorithm by improving scalability [12 25) In contrast our work is algorithm-agnosti and addresses the challenge of handling non-stationary regimes for on-policy algorithms like PPO.
14 3
data in synchronous
Nonstationarity in RL. Nonstationarity in the data distribution is a recognized challenge within reinforcement learning. Such nonstationarity can stem from various sources, including changes in environment dynamics, the reward function, or, as pertinent to our work, the data collection process itself. Previous research has explored related issues such as catastrophic forgetting in continual learning scenarios 7 and representation collapse arising from biased data. The "primacy bias," wherein early experiences exert a disproportionate influence on RL training, has also been documented [19 223. Our work specifically highlights and aims to mitigate the cyclical nonstationarity induced by the interplay of synchronous resets and short rollouts in massively parallel RL settings.
Short Rollouts in Parallel RL The use of short rollouts (K < H) in parallel RL is frequently motivated by the desire to increase update frequency (achieving a high update-to-data ratio) and maximize wall-clock training speed [30 [2]. While this strategy can be effective, underlying issues related data distribution bias are often overlooked. Some implementations incorporate partial resets reesetine environments upon task success, failure, or termination whnih which can introduce some eventual desynchronization. However, this process can be slow and may prove insufficient to counteract the initial bias, particularly when K is very small. Furthermore, while some simulation environment implementations include versions of the reset staggering technique we propose [21], these are typically not detailed in accompanying publications. To our knowledge, our work is the first to thoroughly investigate and analyze this method, as well as to characterize the environmental conditions under which it proves most effective.

---

3 Methodology
3.1 Preliminaries: PPO in Massively Parallel Environments
We consider the standard RI. setting where an agent interacts with an environment modeled as a Markov Decision Process (MDP), defined by (S, A, r.r, po. 7, HI., where S is the state space, A is the action space, P(s" s, a) is the transition probability function, r(s, a) is the reward function, Po is the initial state distribution, 7 E [0, 1) the discount factor, and H is the maximum episode horizon. The goal is to learn policy To (als), s), parameterized by A, that maximizes the expected discounted cumulative return:
H- Σ
>'r(sE, ac) 1=)
J(TO) = Pso-po - P(-(se.0a)
PPO 22 is actor-critic method that optimizes this objective using policy gradients. It alternats between collecting trajectories using the current policy Te and updating the policy parameters 0 as well as value function.
In the massively parallel setting. independent copies of the environment are simulated syn- chronously. During the data collection phase, each of the N environments executes the cur- rent policy Tew for K steps (the rollout length). This generates a batch of x K transitions {ss,, oz; Tt, Ttt S141, N,..,,,,, K-1, where de indicates if state 5t+1 is terminal. This batch is then used to compute policy and value function gradients and perform updates for several epochs.
3.2 The Problem: Cyclical Nonstationarity with Synchronous Resets
Consider the common scenario in massively parallel RL where the PPO rollout length K is chosen to be much smaller than the task horizon (K" & H). This strategy aims increase the update-to-dde (UTD) ratio and overall data throughput. In standard synchronous implementations, where all N parallel environments are reset only after completing their full H H-step episode duration:
1. Initial Synchronized Start: All N environments episodes at effective time & = 0 from an initial state distribution po.
initially reset, commencing their
2. First Rollout Batch: For the first PPO update, each enviromment executes the current policy for K steps. The resulting batch of N K transitions exclusively contains data from the time window [0, K 1] of the episode.
3. Subsequent Rollout Batches: For the second PPO update, environments continue, and the batch is now formed from transitions within the window [K, 2K - 1]. This pattern continues: for the j-th PPO update (assuming no environment has yet completed H steps), the batch consists of transitions exclusively from the window [(j 1)K - 1]. Environments that terminate prematurely within K-step window (e.g.. due task success/failure before H steps) are typically reset to / = O and continue, slightly desynchronizing from the main H-step cycle but not fundamentally altering the batch-wise temporal homogeneity.
4. Synchronized Full Reset: After approximately m [H/K] K] rollouts, all (or most) envi- ronments will have reached or exceeded H elapsed steps since their last full reset. At this point, they all synchronously reset back to effective time f O.
5. Cycle Repetition: the (m + 1)-th update) will again process batch exclusively composed of transitions from the [0, K 1] window, mirroring the temporal origin of the very first batch.
Consequently, the PPO update following this full synchronous reset (e.g..
The crucial issue stemming from this process is a cyclical nonstationarity of the training batches. While each environment eventually explores states across its entire H/-step horizon, each individual batch used for PPO gradient update is temporally homogeneous. It contains data only from a narrow K K-step slice of the overall task. The learner is thus fed data stream where the underlying state distribution within the batch shifts dramatically and predictably from one update to the next, cycling through different segments of the episode. For instance, one batch might contain only early-episode states, the next only mid-episode states, and another only late-episode states, before abruptly reverting early-episode states after the synchronous full reset of all environments. This cyclical bias prevents the learner from accurately estimating values and advantages, likely due to catastrophic forgetting

---

phenomena (empirically verified in Appendix Ep. This leads to poor performance, instability, and failure to learn long-horizon behaviors. effectively negating the benefits of parallelization and high UTD ratios. Empiries we collect in Sections 4 and justiify this claim.
3.3 Staggered Resets
Rollout
Rollout
Rollout
Rallout
Rollout Length
(b)
Task Horizon (H)
Figure 1: Data collection in massively parallel RL. Rows environments, columns time within task horizon H. Colors (red, orange, etc.) mark distinct task stages. (a) Synchronous Resets (Naive): All environments start at (red stage). Each rollout batch (e.g.. Rollout 1: all red; Rollout 2: all orange) temporally homogeneous. Batch content cycles through stages every H rollouts, causing cyclical nonstationarity for the learner. (b) Staggered Resets: Environments start and hence at varied points in the task. Each rollout batch contains mix of task stages (red, orange, green, blue, purple). This within-batch temporal diversity is maintained across rollouts, yielding more stationary and representative data distribution.
To counteract this detrimental cyclical nonstationarity, we introduce staggered resets. The core principle is to ensure that the N parallel environments are not all synchronized to the same portion of the task the beginning of each data collection phase. Instead, they are deliberately initialized to cover diverse effective time steps within the overall task. The mechanism is straightforward:
Initial Staggering Strategy: Before the main training loop commences, each of the N parallel environments is independently advanced (e.g., with random actions or the initial untrained policy) for specific number of offset steps, geffsett This offset geffset can be sampled uniformly from [0, H 1].
However, to manage the frequency reset and advance operations (which can be costly, especially in GPU environments if not batched), discretize the staggering process by dividing the N parallel environments into NB groups. Each environment i then assigned an offset gollsst sampled from a set of discrete intervals, e.g., {0 S, S.... .... (Ng 1) S). where S is the stagger step size, chosen such that NB S = A standard choice is S like {0- K, 1- K,... (H K)/ K|] K). This ine-intiializatinn phase positions environments distinct effective starting points within the task horizon H.
(the PPO rollout length), leading to offsets
The choice of Nи (number of distinct offset groups) balances temporal diversity against the complex- ity of managing reset schedules; a heuristic of NB H/K often works well, ensuring coverage of the horizon within each PPO update cycle while grouping initial advance operations.
Synchronous Rollouts with Staggered Starts: Once initially staggered, the standard synchronous PPO data collection proceeds. All N environments execute the current policy nh... for K steps. However, because each environment effectively began its current episode segment at different offset poffeet relative to the global task timeline, the aggregated batch of N x K transitions now contains experiences from much wider and more representative range of the task horizon. For instance, environments are staggered across multiples of K, the collected batch will naturally include data segments corresponding time windows like [0, - 1). [K, 2K 1]. .... H K, - 1], rather than being overwhelmingyy biased towards single segment. This concept is illustrated in Figure [b).

---

Handling Resets During Training:
Ond-of-Horizon Resets (End of H): After an environment accumulates H effective steps, it is reset to Sy Po. These operations typically occur for entire groups of environments simultaneously, enabling efficient batching.
Partial Resets (Early Termination): If an environment j terminates early (e.g., suc- cess/failure before H lifetime steps). it is flagged. To optimize wall-clock time by max- imizing batched operations, it waits for the next scheduled "reset gate" (when a group undergoes reset by reaching H elapsed lifetime steps). At this gate, environment, j is reset to 80 Po alongside others, starting its new episode from effective time / = 0. This strategy minimizes the number of batched environment reset calls align with Nn. thereby reducing wall-time costs. In practice, this approach effectively maintains the benefits of staggered data collection and its associated temporal diversity without significant degradation.
Policy Updates: The aggregated batch B now contains the temporally diverse transitions due to staggered starts/reset management. B is then used for policy and value function updates in PPO.
By ensuring temporal diversity within each batch, staggered resets provide the learner with data distribution that better approximates the true state visitation distribution p (1:H-1) encountered over complete episodes. This stabilization is crucial for allowing the use of short rollouts K (and thus high update-to-data ratios) without succumbing to the cyelical nonstationarity bias that plagues naive synchronous reset schemes. The improved data quality promotes more stable learning, better value estimates for states across the entire task, and ultimately, enhanced performance on long-horizon tasks. Our empirical results in Section 5 and Section corroborate these benefits.
4 Illustrative Experiments on Toy Environments
To dissect the impact of staggered resets and pinpoint conditions where they offer maximal benefit, we conducted experiments configurable I-dimensional toy environments. These allow controlled variation of factors that often interact in real-world tasks. Understanding these interactions in simplified settings provides insights into the efficacy of staggered resets in more complex, high- dimensional scenarios that often exhibit similar characteristics.
4.1
Toy Environment Design
Our primary toy environment is a ID chain of B discrete levels. An episode spans H steps, with each level covering H/B steps. The agent's state is its current level index b; [t/L]. In each level b, the agent selects an action a; from discrete set A. A fixed target action aj E A is assigned to each level. Correct actions (a, = a;,) yield a reward of +0.5, incorrect actions -.5.. We manipulate three key environment dynamics to explore when staggered resets are most impactful:
Rollout Length Ratio (K) H): We vary the environment task horizon H, exploring scenarios from K ~ H (short effective skill horizon, less temporal bias) to K < H (long effective skill horizon where cyclical nonstationarity with naive resets is hypothesized to be severe).
Reset Dynamics: Upon episode termination, resets can range from deterministic (AR = 0, always to level 6o 0) to stochastic (AR > O, where by Poisson(Ag) centered around Ag). This tests if inherent start-state randomness mitigates synchronous reset issues versus needing deliberate staggering. The term "Reset Homogeneity" in our plots (Figure 2b) refers to 2 An. where higher values mean more deterministic (homogeneous) resets to by 0.
Sequential Skill Gating: Progression from level to b + 1 depends on a mastery threshold Kmastery (number of correct actions uz)) and an unconditional progression probability Progg The agent advances if mastery is met or a random check (Prrog succeeds. Varying Ppng E [0. 1] (where Pprgg 0 implies hard gating requiring kimastery > 0) creates tasks from simple linear progression to those requiring
sequential skill acquisition.
5

---

4.2 Results on Toy Environments
Experiments on these toy environments (Figure 2 characterize how different environmental factors influence the efficacy of staggered resets compared to naive synchronous rollouts, particularly highlighting the impact of data stattiontoo...y
Final Average Patur vo. Hor oen Langit
Finsl Average Pature vs. Ploue Horrogennity
Average Pultam Progrosson Prozabilty
Seepeed
-00
Hortoo Lange
Roast Homogenatyy (V-
Plog station Protad by 80
(a)
(b)
(c)
Figure 2: PPO with Non-Staggered (blue) vs. Staggered (red) resets on toy environments (mean + 1 std dev). Staggered resets show robust performance as (a) horizon H increases, (b) reset homogeneity
(2 - Ax) increases, and (c) progression probability (Prog) varies, unlike non non-staggered PPO which degrades especially with longer horizons, more deterministic resets, and easier skill gates (Pprog > 0).
With fixed short rollouts (K), each batch of N x K transitions is temporally homogeneous, covering only a K-step slice of the H-step task. As H increases. the number of distinct temporal slices in the data collection cycle (H/K) also grows. This means the periodicity of revisiting any specific task segment (e.g., the initial states) becomes longer. The learner struggles to consolidate information and may forget what it learned about earlier segments by the time the data cycle returns to them, especially with many intermediate, distinct batch types (see Figure 12 for visualization of this forgetting process). Staggered resets, by providing temporally diverse data within each batch, maintain high performance irrespective of H, effectively facilitating learning across the entire task horizon even when K < H.
Horizon Length (Figure 21:
(Figure 2): This experiment, conducted with minimal skill gating (Ppreg = 1.0), varies the determinism of reset locations. As resets become more concentrated the episode's start (higher "Reset Homogeneity." " i.e., AR 0), the performance of non-staggered PPO deteriorates. While some inherent randomness in reset locations (Aς > 0) can offer minor desyn- chronization and marginal performance improvements for the naive method, deliberate staggering consistently yields substantially better results. This indicates that relying on incidental environmental randomness is insufficient to overcome the core non-stationarity induced by synchronous rollouts.
Reset CoochastiiityyHHomggnen
Skill Gating Dynamics (Figure This experiment tests how (Ppea) (ease of progressing without mastery) affects data collection strategies. Staggered PPO performs robustly across all (Ppog) levels. its comprehensive sampling ensuring sufficient data for learning all skills. Non-staggered PPO's behavior is more complex: it performs best with stringent skill gates ((Ppog O)), where the environment imposes a curriculum forcing mastery of carly skills to reach later states. As (Ppng) increases (gates soften), non- unasstagered PPO's performance degrades. With relaxed gates, random advancement occurs, but the established data collection bias towards initial states prevents learning a coherent policy for the full task, as sporadic later-state exposure without mastery insufficient. Consequently, the advantage of staggered resets widens as (Ppog) increases. This highlights that the cyclical non-ttationarity of naive rollouts a key limiter, especially when environmental structures (like hard gates) that aid exploration are weak.
The environmental factors varied in our toy experiments- length, reset stochasticity, and skill gating gating-mirror -iiror complexities encountered in -igh-dimensional robotics tasks, such as those in ManiSkill3, For instance, long manipulation sequences often mean the task horizon H much larger than practical PPO rollout lengths K, exacerbating the long periodicity issue seen in Figure Z.. "Skill gates" in toy tasks are analogous to critical bottleneck sub-tasks in robotics, like achieving a stable grasp in StackCube-ve before attempting to lift and place. If an agent can bypass such a bottleneck (akin to high Pprog) but its experience is fed in temporally homogeneous batches, learning a robust overall strategy becomes difficult. Similarly, the degree of stochasticity in initial object poses slight variations in robot starting conditions ManiSkill3 tasks relates to the reset stochasticity (Figure@p). Highly deterministic setups in robotics can make the cyclical batch problem more pronounced.

---

Experiments on High-Dimensional Robotics Tasks
5.1 Experimental Setup
Our evaluation suite includes several challenging robotics tasks from ManiSkil13 (26], GPU- accelerated robotics framework based on SAPIEN [29]. We test on StackCube-vi, a manipulation task requiring an agent to stack one cube onto another; PushT, where a T-shaped block must be pushed to target pose; TwoRobotPushCube, where two robots work together move a cube to goal; Unitree Transport Box, humanoid task where box must be transported to a table; and Anymal Reach where an Anymal C robot must move specific goal location. We also test on H-5-Humnooiee faakk, dWalk, a humanoid walking control task. These environments involve high- nigh-dimensional continuous state and action spaces, providing suitable testbed for evaluating the effectiveness of staggered resets.
5.2 State Visitation Dynamics in High-Dimensionas Robotics
(a)
Steps 0-100
Sieps 3-100
Stepa 0-100
Stepa 0-100
Stepa 0-100
(b)
Steps 0-25
Steps 26 60
Stepe 51:75
Steps 76-100
Steps 0-25
(c)
Steps
Steps
Steps
Rollout
Rollout 2
Rollout 3
Rollout 4
Rollout 5
Figure 3:
State visitation KDEs StackCube-vé over five rollouts. (a) Long Rollout (K 100):
stable, broad coverage. (b) Naive Short Rollout (K 25): cyclical non-stationarity, narrow/erratic coverage. (c) Staggered Short Rollout (K 25): stable, diverse coverage, emulating (a) despite short trajectories.
A central premise of this work is that the common practice of using short rollouts (K < H) with naive synchronous resets in massively parallel RL leads to a problematic data generation process. This process is characterized by temporally unstable and skewed state visitation distribution, which we hypothesize significantly impedes learning. Our proposed staggered resets intervention aims to rectify this by enabling short rollouts to yield data distributions more similar to those from longer, more informative trajectories.
To empirically assess this hypothesis, we visualize and analyze state visitation patterns on the challenging StackCube-vi roboties manipulation task. preent Kernel Density Estimates (KDEs) of visited states in a given rollout buffer, projected onto their first two principal components derived from the aggregate data of all rollouts across an entire training run. Each row corresponds to a difffernnt data collection strategy, and columns depict the evolution of the state distribution over five consccutive PPO data collection rollouts.
The Long Rollout PPO baseline (K = 100), shown in Figure 3h, serves as an empirical ideal, exhibiting broad and stable state coverage across all five rollouts. This pattern signifies rich and temporally consistent data stream, achieved here at the expense of a lower UTD ratio.
Figure 3p, representing the Naive Short Rollout PPO strategy (K = 25). illustrates the nonstationarity consecutive rollout buffers induced by synchronous resets. Initial rollouts show a state distribution highly concentrated near the environment's 's reset distribution. As training progresses, the distribu-

---

tion expands. but its shape and locus shift dramatically between rollouts. After the environments synchronously reset (following Rollout 4), the state distribution at Rollout 5 closely mirrors that of Rollout 1. This cyclical nonstationarity means the learner receives a constantly changing and biased view of the transition space, hindering its ability to form accurate value estimates and a robust policy.
Figure 3 depicts the state visitation dynamics for our Staggered Short Rollout PPO strategy (also K 25). The distributions achieved by staggered resets qualitatively mirror the desirable breadth and consistency of the long rollout baseline (a). This empirical evidence strongly supports our central claim: staggered resets effectively counteract the cyclical non non-stationarity induced by synchronous resets in short-rollout regimes.
5.3 Performance on High-DDmensioon Robotics Tasks
To validate the effectiveness and generality of staggered resets, we conduct experiments on two distinct sets of eigh-dimensional robotics benchmarks, using two different ofate-of-the-at on-policy algorithms: Proximal Policy Optimization (PPO) and Split and Aggregate Policy Gradients (SAPG).
StackCube
TransoSSir
Anymalt Reach
200
2.8
0.8
5.0
0.6
0.6
0.4
0.4
0.2
0.2
0.
Ereioos Stoes
Environment
0
Enwrosrior Stops
2004
40M
G0M
TwssfobeePikcuuee
510005
0.8
0.0
0.5
0.7
0.4
0.6
0.2
0.5
Enviromment Steps
Environment
2351
ADM
OOM
EDM
33M
40M
103
20M
30M
401
3000
1000
10030
1909
5004
3.50 2.50
Staggered Resets (Ours)
Synchronous Resets
Figure 4: Staggered Resets (Ours, red) consistently outperform Synchronous Resets (blue) across different on-policy algorithms and task suites. Plots show the average evaluation metric (success rate or reward) vs. environment steps. Shaded areas show the standard deviation over 10 seeds. (Top Middle Rows) On PPO with diverse ManiSkill3 tasks (StackCube, PushT, etc.), staggered resets consistently improve learning speed, final performance, and stability. Performance comparable only on the locomotion task MS- Hos-Humnoiice iWakk, where natural desynchronization reduces the severity of the problem. (Bottom Row) To demonstrate aggoritm-agnosttccss evaluate on SAPG with challenging AllegroKuka manipulation tasks. Staggered resets again yield substantial gains in sample efficiency and final reward, confirming the generality of our approach.
we
5.3.1 Performance Improvements with PPO on ManiSkill3 Tasks
As shown in the top and middle rows of Figure 4 staggered resets achieve substantially faster convergence, higher final success rates, and greater stability on wide range of PPO-based manipula- tion tasks. For instance, in PushT and StackCube- -1. staggering leads significantly higher and
8

---

more stable final performance. Similarly, in AeymalC-Reach and TwoRooootPckCCug learning is markedly quicker and reaches better asymptotic success rate. Crucially, with an increased evaluation of 10 seeds, the variance across runs is also substantially lower with staggered resets, confirming that our method improves overall training stability.
Interestingly, on the locomotion task MS- Ho--HHmmoiidaaae both methods perform comparably. As discussed in our toy experiments, locomotion environments often feature shorter effective skill horizons and highly stochastic reset behaviors (e.g., the agent falling at unpredictable times). These factors induce a degree of natural desynchronization, making the cyclical batch nonstationarity less severe and thereby reducing the marginal benefit of explicit staggering. This aligns with prior work [21] that employed a form of staggering in locomotion contexts, where our analysis suggests the benefits are less pronounced compared to more complex, longer- horgzon manipulation tasks.
5.3.2 Validating Algorithm-AAnossicismn with SAPG
central claim of work is that the benefits of staggered resets are algorithm-agnosiic, stemming from the data distribution rather than the specifics of the learning update. To substantiate this, we conduct additional experiments with a more recent on-policy algorithm, SAPG [25] on the challenging AllegroKuka dexterous manipulation tasks.
The results, shown in the bottom row of Figure 4 demonstrate that staggered resets significantly improve the performance of SAPG. Both AllegroKukaThron and AllegroKukaBegraspii dramatic improvements in sample efficiency and final asymptotic reward. Notably, these environ- ments feature frequent early resets (e.g., on task success or object drop) and significant domain randomization, which can naturally induce some desynchronization and reduce the cyclicity problem. Despite this, staggered resets still yield a pronounced performance improvement, underscoring the robustness and general applicability of our method for improving data quality in massively parallel on-policy RL.
show
5.4 Scaling with Parallel Environments and Overcoming Performance Saturation
A key challenge in massively parallel RL is effectively utilizing increased parallelism. SAPG [25), shows that beyond threshold, additional environments may not reduce wall-clock time and may even hurt performance due to higher gradient variance or commu- nication overhead. We address this issue and show that staggered resets improve convergence speed across a wide range of N, yielding higher marginal utility of increased environment parallelism.
Figure 5) shows the wall-clock time required to reach a fixed conver- gence threshold (70% success rate) function of the number of parallel environments N. Results are shown for two representative tasks: StackCube-vi (Figure 55) and Unitree G1 Transport Box (Figure 5p).
(a)
Unirwor Taapatt
For Naive PPO (blue curves), increasing parallel environments (N) initially speeds up wall-clock convergence. However, this bene- fit saturates or even reverses at larger N (e.g., beyond N R 1024
for StackCube-vl. Figure 5p degrading sharply at N 6144 for Unitree G1 Transport Box, Figure 5p). Hence, while data throughput rises, learning efficiency diminishes due to temporally homogeneous from many synchronously reset environments, off- setting parallelism gains. Conversely, Staggered PPO (red curves) demonstrates superior scaling, with wall-clock convergence time continuing to decrease as N increases across all tested values, even beyond 6000 environments.
(b)
Figure 5: Wall-clock time to convergence versus number of parallel environments (N) for (a) StackCube-v1 and (b) Unitree G1 Transport Box
Discussion
Our investigation reveals a significant challenge in modern massively parallel on-policy RI. The cyclical nonstationarity introduced by synchronous environment resets when coupled with short

---

rollouts (K < H) inadvertently biases the learning signal by repeatedly oversampling states from the initial segments of episodes. This bias can detrimentally affect learning stability, convergence speed, and ultimate policy quality, leading to issues like value function divergence and catastrophic forgetting, as we quantitatively demonstrate in Appendix F.
Staggered resets directly address this nonstationarity without any changes to the core learning algorithm. By deliberately initializing parallel environments varied effective time steps within the task horizon, we ensure that each training batch encompasses a temporally diverse set of experiences. This creates a more stationary and representative data distribution for the learner. The state visitation KDEs (Figure B) offer compelling visual evidence: staggered resets with short rollouts yield broad and stable state coverage, closely emulating the desirable properties of much longer rollouts, whereas naive short rollouts suffer from erratic, cycling state distributions.
Crucially, our work also provides empirical justification for focusing the short-rollout (K & H) regime. As shown in our appendix experiments (Table TT. this regime is not merely a heuristic but is often optimal for wall-clock efficiency. On challenging tasks like StackCube-v1, short rollouts (K = 8 16) converge 2-3x faster than long rollouts while achieving the same or better final performance, reinforcing that addressing issues within this specific setting is of high practical importance.
The illustrative toy experiments (Section further characterize the conditions where staggered resets are most impactful. Specifically, tasks with longer horizons relative to the rollout length, more deterministic (homogeneous) reset states, and weaker intrinsic task curricula (e.g., where agents not strictly forced to master early skills to progress) show pronounced benefits from the explicit temporal diversification that staggering provides.
key practical advantage of staggered resets is their ability to enhance the scalability of on-policy RL in massively parallel settings (Section 5.4 Figure 5 While naive PPO often encounters diminishing returns or even performance degradation as the number of parallel environments grows-likely due to increasingly redundant data staggered resets facilitate continued improvements in wall-clock convergence time. This indicates a more effective utilization of parallel compute resources, as the increased data volume is also more diverse and informative.
In conclusion, staggered resets provide robust, easily implementable, and computationally inexpen- sive method to significantly enhance the performance and scalability of on-policy RL. Its benefits are gegorithm-agnostee improving performance for both PPO and more recent methods like SAPG (Section 5.3.2). By directly addressing the issue of cyclical data nonstationarity, this technique allows for more stable value estimation, faster convergence, and better final policies, paving the way for more effective learning in complex, long-horizon tasks.
7
Acknowledgements
Stone Tao is supported in part by the NSF Graduate Research Fellowship Program grant under grant No. DGE2038238.
References
[1] Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. http://pyybulett.org 2016-2021
[2] Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässaan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1406-1415. PMLR, 2018. URL attp://proceedings mlr, preess//80/espehotttt 1atmemmmmm1....mmm1111
[3] Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax-a differentiable physics engine for large scale rigid body simulation. In Thirty- fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
10

---

[4] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei. Yunchao Yao, Xiaodi Yuan. Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskill2: A unified benchmark for generalizable manipulation skills. In International Conference on Learning Representations, 2023.
[5] Ankur Handa, Arthur Allshire, Viktor Makoviychuk, Aleksei Petrenko, Ritvik Singh, Jingzhou Liu. Denys Makoviichuk, Karl Van Wyk, Alexander Zhurkevich, Balakumar Sundaralingam, Yashraj Narang, Francors Lafleche, Dieter Fox, and Gavriel State. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. arXiv, 2022.
[6] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado van Hasselt, and David Silver. Distributed prioritized experience replay. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httpps:////ppeeeee/ net/forumm -it-mmddd------OO--
[7] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A Rusu, and K Milan. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):35113352 2017.
[8] Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11):11388 1274, 2013.
[9] Robert Tjarko Lange. Gymmax: Jax-based reinforcement learning environments, 2022. URL http:///ithub. com http://ittub..co////obttr//aaaa/ gymmaxx
[10] Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Sei. Robolics, (26), 2019. doi: 10. /226//CTROBOTICCSAASSS scirobotics. aau5872 2
URL htttp:://di..org:/ 10..6...1...////// 1126/
[11] Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. In Journal of Machine Learning Research, volume 17, pp. 1 1-40, 2016.
[12] Yunan Li, Viktor Makoviychuk, Yash Narang, Fabio Ramos, Marco Hutter, Arthur Allshire, and David Hoeller. Parallel advantage actor-critic for rapid end-to-end robotics training. IEEE Robotics and Automation Letters, 8(1 ((11)7113-7122) 2023.
[13] Toru Lin, Kartik Sachdev, Linxi Fan, Jitendra Malik, and Yuke Zhu. Sim-to-real reinforcement learning for vision-based dexterous manipulation on humanoids. arXhv:2502.20394 2025.
[14] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu-bau based physics simulation for robot learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[15] Gabriel Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. In Robotics: Science and Systems, 2022.
[16) Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 86):3740-3747,, 2023, doi: 10. 10.11009/RAA.2223.27700
[17] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep rein- forcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
[18] Arun Nair. Praveen Srinivasan, Sam Blackwell, Cagdas Alcicek, Rory Fearon, Alessandro De Maria, Vedavyas Panneershelvam, Mustafa Suleyman, Charles Beattie, Stig Petersen, Shane Legg, Volodymyr Mnih, Koray Kavukcuoglu, and David Silver. Massively parallel methods for deep reinforcement learning. CoRR, abs/1507.00 00..099 2015. URL htRt::ttt://// //arxiv.. org/obga abs/ 1507. 04296

---

[19] Evgenii Nikishin, Junhyuk Oh, Georg Ostrovski, Mohammad Gheshlaghi Azar. Hado van Hasselt, and Doina Lee. The primacy bias in deep reinforcement learning. arXiv preprint wrxiv:2205.07803 2022.
[20] Alexander Nikulin, Vladislav Kurenkov, Ilya Zisman, Viacheslav Sinii, Artem Agarkov, and Sergey Kolesnikov. minnnnii Scalable meta-reinforcemmnt learning environments in JAX. In Intrinsically--Motivveed and Open-Ended Learning Workshop, NeurIPS2023, 2023. URL
[21] Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Aleksandra Faust, David Hsu, and Gerhard Neumann (eds.), Proceedings of the Sth Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pp. 91-100. PMLR, 08-11 Nov 2022. URL htttps:/pproceeddngg mlr prrss/v1644rudin222 html
[22] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL htttp///axii, org/
abs/1707.06.0 06347
[23]
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Marc Bellemare, Rishabh Agar- wal, and Pablo Samuel Castro. Bigger, better, faster: Human-level Atari with human-level effi- ciency. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 30365-30380. PMLR, 23-29 Jul 2023. URL https::/proceedd/e
mlr press/ press/v202) http:///222222///a/ee hwtps//wwwwaeeeaaaaaa html
[24] Ritvik Singh, Arthur Allshire, Ankur Handa, Nathan Ratliff, and Karl Van Wyk. Dextrah- rgb: Visuomotor policies to grasp anything with dexterous hands, 2025. URL https://arxiv org/abs/2412. 01791
[25] Jayesh Singla, Ananye Agarwal, and Deepak Pathak. Sapg:
Split and aggregate policy gradients. In Proceedings of the 41 International Conference on Machine Learning (ICML 2024), Proceedings of Machine Learning Research, Vienna, Austria, July 2024. PMLR.
[26] Stone Tao, Fanbo Xiang, Arth Shukla, Yuzhe Qin, Xander Hinrichsen, Xiaodi Yuan, Chen Bao, Xinsong Lin, Yulin Liu, Tse kai Chan, Yuan Gao, Xuanlin Li. Tongzhou Mu, Nan Xiao, Arnay Gurha, Viswesh Nagaswamy Rajesh, Yong Woo Choi, Yen-Ru Chen, Zhiao Huang. Roberto Calandra, Rui Chen, Shan Luo, and Hao Su. Maniskill3: Gpu parallelized robotics simulation and rendering for generalizable embodied ai. Robotics: Science and Systems, 2025.
[27] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp- 0026-5033. IEEE, 2012. doi: 10. 109/TROS.20112.63866 2012.638610..
[28) Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Frez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. dm_ con_ttoool: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638. doi: https:// lbi.org/10.1016/j.sso wwtp:i.smmaaa....... 2020. 100022. UtL/https://
sciencedirect.....cceeeeeeeeieie www/scieeeeeeeeeeeo.co sci /
[29] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. Chang, Leonidas J. Guibas, and Hao Su. SAPIEN: A simulated part-based interactive environment. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[30] Jie Xu, Viktor Makoviychuk, Yashraj Narang, Fabio Ramos, Wojeiech Matusik, Animesh Garg. and Miles Macklin. Accelerated policy learning with parallel differentiable simulation. In International Conference on Learning Representations, 2021.
[31] Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning ( (CoRL., 2019. https :ttps://arraa //arxiv. www. /bs/
1910. 10897
12

---

[32] Kevin Zakka, Baruch Tabanpour, Qiayuan Liao. Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A. Kahrs, Carlo Sferrazza, Yu- val Tassa, and Pieter Abbeel. Mujoco playground: An open-source framework for gpu- accelerated robot learning and sim-to-real transfer., 2025. URL https:://githu..c com/ http://-deeepaaaa//ooooooooeeeee
13

---

A Motivating Experiments
A.1 Empirical Motivation for the & H Regime
To motivate the massively parallel short-rollout regime, we sweep K (1, 4, 16, 64, 100) on StackCube-v1 (H-100). Each configuration is trained for 100M steps across three seeds with identical hyperparameters. Short rollouts (K=8-10 reach the same or better final reward as long rollouts while converging 2-3x faster in wall-clock time, confirming the practical relevance of K < H in massively parallel on-policy RL.
Table 1: Performance sweep rollout length K on StackCube-vil Best results (K=8-16 16) in bold.
(a) Final reward after 100M euvironment steps.
Metric
2
4
8
16
32
64
100
Staggered 0.00
0.47++0.01
0.70+-0.00
0.74+10.00
0.72++0.03
0.75=0002
0.71=0.03
0.70+00.2
Naive
0.27110117
0.36.10.12
0.4710.02
0.62:10.01
0.79-0.03 0.02
0.71
0.70.10.02
(b) Environment steps (millions) to reach >75% success rate.
Metric
2
4
8
16
32
64
100
Staggered
DNC
DNC
DNC
16.2:0.2
17.2=0.1
23.2 1.4
50.1=5.6
49.6:19.
Naive
DNC
DNC
DNC
35.8:1..5
18.5=2.1
24.1
50.5=6.2
49.6:19.9
(c) Wall-clock time (minutes) reach >75% success rate,
Metric
4
Staggered
DNC
DNC
DNC
16.6+:3.2
15.3=1.0
22.5+2233
40.0=8.7
42.6+8877
Naive
DNC
DNC
DNC
26.2.12.1
11.6
30.0
41.114.5
43.2.19.9
Observation. The fastest convergence is achieved with short rollouts (K=8-16) on an NVIDIA RTX-4090 GPU, matching Rudin et al. (2021) and Singla et al. (2024). Training with 1=1-4 fails converge (DNC). Final reward saturates beyond K=8, reinforcing that modern massively parallel on-policy systems naturally operate in the K < H regime.
B
Implementation Details
B.1 Staggered Reset Implementation Details
The staggered reset mechanism aims to distribute the effective starting timesteps of the N parallel environments across the task horizon H. This was achieved by dividing environments into NB = [H/K] groups, with each group starting its first "effective" episode step after initial offset of j K simulation steps (typically performed with random actions or the initial policy). This ensures that each PPO batch contains data from various segments of the task horizon.
B.2 Details on Toy Environments
The toy environments described in Section 4 were designed isolate and study the effects of data nonstationarity under different environmental conditions. See Table 2 for the hyperparameters chosen for PPO in the toy environment. We describe more concretely the ablation and environment implementation details below.
B.2.1 Environment Dynamics
The environment is 1-dimensional chain of B discrete levels "blocks". An episode lasts for maximum of H time steps. Each level E {O...., , 1) covers H/B steps. The agent's state is its current level index b, cclapsed_steps,_ steps,/ LJ. At each time step t, the agent, being in
14

---

level by, chooses an action a, from a discrete set of Ag categories (e.g.. A.e 20). Each level has pre-assigned target action aj ((....,44. A. 13. The reward function is:
+0.55
if at %
r(st,0)) -0. if az + ob.
The episode terminates if elapsed_steps, steps, > H.
B.2.2 Further Details on Ablations on Toy Environments
The following parameters were varied to create the different experimental conditions shown in Figure 2
Horizon Length (H K): (Figure 2a) The task horizon H (max_steps in code) was varied across values [50, 100, 200, 300, 400, 500]. The PPO rollout length K (num_stee in PPO loop, i.c., buffer size per environment before update) was kept fixed at K = 5. The block length L was also fixed at For this experiment, skill gating was moderate (Pprog = 0.5, Kiasttryy = 3) and reset deterministic (A =).
(AR): (Figure Upon episode termination, the reset mechanism was varied. The parameter AR (reset_stochastice) stochasticity llmbddaalmaaaaaaaaaa code) controls the mean of a Poisson distribution from which the starting block bn is sampled, i.e., bo Poisson(AR) (RR). clamped to [0, B 1]. XR = 0 corresponds to deterministic reset bp The "Reset Homogeneity" axis in the plot is 2.0 AR for visualization purposes (higher values = more deterministic starts at by = 0). 2R was varied in [0.0, o.,,.... 1.0]. For this experiment, H 50, L 5, K 5, Pprog 1.0 (easy progression), Kmastery = 3.
Reset Ettchastiiity/Hooogeeeee
Skill Gating Dynamics (Pprug): (Figure 2 Progression from the current block b to +1 (when enough steps within block b have nominally passed to enter b + 1) occurs if either:
1. The agent has achieved "mastery" in block b, defined making least kimastery correct actions ag within block 6 during the current episode. (Kimsstrr = 3 was used).
2. A random chance Pprog for unconditional progression succeeds.
The probability Prog (progression_prat prob in code) was varied in 0.10, 1.0). Porog 0 means hard gating requiring mastery. For this experiment, H 200, L 5. K
AR =0=
B.3 Implementation Details and Hyperparameters for ManiSkill Experiments
This section details the Proximal Policy Optimization (PPO) configuration for experiments on ManiSkill robotics tasks (Section utilizing ManiSkill3 (26) for GPU-accelerated simulation. The exact PPO implementation is based on the one provided by ManiSkil13 baselines, which is based on LeanRL and CLeanRL. Table 3 provides comprehensive list of hyperparameters.
15

---

Table 2: PPO and Environment Hyperparameters for Toy Environment Experiments. Shaded rows categorize parameters, Values listed are defaults; specific sweeps varied H, Pprog or AR as detailed text and figures.
Hyperparameter
Value
PPO Algorithm Core Settings
Learning Rate
3 x 10
Discount Factor (7))
0.99
GAE Lambda (A)
0.95
PPO Rollout Length (K)
5 steps per environment
Number of Parallel Environments (N)
512
Total Training Updates
150
Update Epochs
4
Number of Minibatches
4 (Minibatch size: (512 5)/1 640)
PPO Clipping Coefficient (c)
0,2
Value Function Loss Coefficient
0.5
Entropy Bonus Coefficient
0.01
Max Gradient Norm
0.5
Network Architecture (Actor & Critic MLLP)
Input
Current block index (integer state)
Embedding Layer
Input block index 64-dim embedding
Hidden Layers
4
Units per Hidden Layer
Activation Function
ReLU
Policy Output
Categorical distribution over actions
Value Output
Scalar state value
Optimization
Optimizer
Adam
Toy Environment Base Parameters (Defaults for Sweeps)
Episode Horizon (IT)
Varied (e.g., 50, 100, 200, 375 for specific experiments)
Block Length (L)
5 steps
Number of Action Categories (A.)
20
Reward for Correct Action
+0.5
Reward Incorrect Action
-0.5
Success Definition
Agent is in the final block at episode end Varied (0.0 to 1.0)
Skill Gating: Progression Prob. (Pprg)
Skill Gating: Mastery Threshold (kimastery)
3 correct actions
Reset Stochasticity (AR)
Varied (Poisson
for start block, 0.0 for deterministic)
Staggered Resets Mechanism (When Enabled)
Number of Stagger Blocks (Nv)
H/K Episode Horizon/55
Stagger Step Size (S)
K =
16

---

Table 3: PPO Hyperparameters for ManiSkill Experiments. Common settings are listed first, followed by per-environment variations where applicable. Shaded rows categorize parameters.
Hyperparameter
Value / Per-Environment Specification
PPO Algorithm Core Settings
Learning Rate
3
10
Update Epochs
4
Number of Minibatches
32
PPO Clipping Coefficient (e)
0.2
Value Function Loss Coefficient
0.5
Entropy Bonus Coefficient
0.005
Max Gradient Norm
0.5
Advantage Normalization
True (Per minibatch)
Target KI. Farly Stopping
Network Architecture (Actor & Critic MLP)
Hidden Layers
3
Units per Hidden Layer
[256. 256, 256]
Activation Function
Tanh
Weight Initialization
Orthogonal
Policy Output
Gaussian mean, learaalec state-independent log std. dev.
Optimization
Optimizer
Adam
Adam Epsilon
1 x 10-5
Learning Rate Annealing
False
Environment Interaction & Data Collection (Common)
Total Training Timesteps
2 10
Partial Resets (Training)
True
Evaluation Environments
128
Evaluation Partial Resets
False
Observation Normalization
Via environment wrappers / running mean & std
Reward Scaling
Environment-dependeen (aim approx. 1)
Staggered Resets Mechanism (When Enabled)
Staggering Mode
Uniform distribution of start times H/K] (Task Horizon / Rollout Length)
Number Stagger Blocks (NB)
Stagger Step Size (S)
(Rollout Length)
Ter-Ennironment Specific Hyperparameters
Parameter
Values for: StackCube PushT AnymalC/ HumanoidWall /TooRobotCube / UnitreeBox
Rollout Length (K)
8/8/ / 16 64 16 32
Task Horizon (H)
100 100 200/ / 1000 / 100 500
Discount Factor (7)
0.8 / 0.99 0.99 / 0.97 / 0.8 / 0.8
GAE Lambda (A)
0.9 0.95 0.9 0.9
Num. Parallel Env. (N)
4096 / 4096 / 512 4096 2048 1024
17

---

B.4 Implementation Details and Hyperparameters for SAPG Experiments
This section details the configuration for the experiments on the AllegroKuka dexterous manipulation tasks (Section 5.3), which use the Split and Aggregate Policy Gradients (SAPG) algorithm. The experiments were conducted using the IsaacGym simulator. To ensure a fair comparison and isolate the effect of staggered resets, we adopt the official hyperparameters reported in the original SAPG paper [25]. Table 4 provides comprehensive list of these hyperparameters. Our method introduces only the staggered reset mechanism on top of this baseline.
Table 4: SAPG Hyperparameters for AllegroKuka Experiments, based categorize parameters.
25
Shaded rows
Hyperparameter
Value
SAPG Algorithm Core Settings
Learning Rate
x 10-
Update Epochs
Mini-bate batch Size
Num. Parallel Env. x
Clipping Coefficient (e)
0.1
Value Function Loss Coefficient
4.0
Entropy Bonus Coefficient
(default, see
25
for tuning)
Max Gradient Norm
1.0
KL Threshold for LR Update
0.016
Bounds Loss Coefficient
0.0001
Network Architecture (Recurrent Actor & Critie)
Observation Preprocessor
MLP with hidden layers [768, 512, 256]
Policy/Value Core
1-layer LSTM with 768 hidden units
Activation Function
ELU
Policy Output
Gaussian mean, learnable state-independent std. dev.
Optimization
Optimizer
Adam
Environment Interaction Data Collection
Number of Parallel Env. (N)
24576
Rollout Length (K)
16
Discount Factor (?)
0.99
GAF Lambda (A)
0.95
LSTT Sequence Length
Staggered Resets Mechanism (When Enabled)
Staggering Mode
Uniform distribution of start times
Number Stagger Blocks (Nn)
[H/K] (Task Horizon Rollout Length)
Stagger Step Size (S)
K (Rollout Length)
18

---

C Additional Results on Toy Environments
To further illustrate the impact of synchronous versus staggered resets on the data distribution and learning progress, we visualize the training dynamics in one of the toy environments (specifically. H 200, 5, K = 5, Pprog 0.5, AR 0). Figure 6 shows the training average accuracy and the mean distribution of environments across different blocks (states) over the course of training updates. For these experiments, we define accuracy as the rolling percentage of correct one-hot action guesses from the PPO agent.
Warid Jesssage Cocurayy
MY
M
0.3
(a)
(b)
Moun Eny Biock Detrribticn Over Thaining
310
136
25
15
139
80 120 140
(c)
(d)
Figure 6: Comparison of training dynamics in toy environment with (a. e) naive synchronous resets versus (b, d) staggered resets. (a) & (b): Training average accuracy over 150 PPO updates. With naive resets (a), accuracy is unstable and struggles converge. With staggered resets (b), accuracy rises quickly and stabilizes at a high level. (c) & (d): Heatmaps showing the mean number environments occupying each block (y-axis) at each PPO update index (x-axis). (c) With naive synchronous resets, environments progress through blocks in tight, synchronized waves. After approximately 40 updates (when H/K 200/5 40 rollouts complete an episode), all environments abruptly reset block O, leading to cyclical pattern where training batches are temporally homogeneous (all early-episode, then all mid mid-episode, etc.). (d) With staggered resets, the distribution of environments across blocks is far more uniform at any given update index. This indicates that each training batch contains a mix of experiences from different stages of the episode, leading to a more stationary data distribution for the learner.
Subplots (c) and (d) in Figure 6 are heatmaps where the x-axis represents the PPO training update index, the y-axis represents the block index (state) within the toy environment's episode, and the color intensity indicates the mean number of parallel environments present in that block at that specific training update.
Naive Synchronous Resets (Figure (b)) The heatmap for naive synchronous resets clearly shows distinct diagonal bands. Each band signifies that the cohort of parallel environments synchronously progressing through the episode's blocks. A crucial observation is the abrupt termination of these bands followed by an immediate restart from block (the bottom of the y-axis). This occurs approximately every 40 updates, corresponding to the episode horizon (H 200) divided by the
19

---

rollout length (K = 5). This pattern visually confirms the cyclical nonstationarity discussed in Section 3.2 at any given update, the training batch is predominantly composed of states from narrow segment of the episode, and this segment predictably cycles. For instance, for updates 1-5, data is from blocks near 0-4; for updates 35-40, data is from blocks near 35-39; then at update 41, data abruptly shifts back to blocks 0-4.
Staggered Resets (dd) In contrast, the heatmap for staggered resets shows much more diffuse and uniform pattern. While environments still progress through blocks (indicated by the general upward-right trend), there is no global, abrupt reset of all environments. At any given PPO update index, environments are distributed across a wide range of blocks. This means that each training batch collected under staggered resets contains temporally diverse mix of experiences some from early parts of episodes, some from middle, and some from later parts. This significantly reduces the cyclical nonstationarity of the data fed to the PPO algorithm.
Impact on Learning (Figure oa and 6b) The consequences of these different state visitation dynamics are evident in the training accuracy plots. With naive synchronous resets (Figure (b). the learning curve for average accuracy is highly erratic, exhibiting periodic dips and slow overall improvement, struggling to consistently achieve high accuracy. This instability likely results from the PPO learner trying to adapt to the rapidly shifting data distributions. Conversely, with staggered resets (Figure 6)), the average accuracy rises smoothly and rapidly, quickly converging to high and stable level. This demonstrates that providing the learner with more temporally diverse and stationary batches facilitates more effective and stable learning.
C.1 State Visitation Distribution Analysis
We provide further detail on the state visitation dynamics in our toy environments by visualizing the mean environment block distribution over training updates (similar to Figure and while systematically varying key environmental parameters. For all visualizations this section, the PPO rollout length K = 5, number of parallel environments N = 512, total training updates shown are 150, environment block length I, = 5, number of action categories A, = 20, and mastery threshold kmnstery 3, unless specified otherwise. Base PPO hyperparameters are detailed in Appendix [B.2] Each figure's s top row shows results for naive synchronous resets, and the bottom row shows results for staggered resets.
C.2 Impact of Progression Probability (Pprog)
Figure illlustrtes how the probability of unconditional progression, Pprog affects state visitation. The ex experiment uses a fixed horizon H = 200 and deterministic resets (A = 0).
Non-Staggered Resets (Top Row):
Pprig 0.0 (Left): With hard skill gating, environments get stuck at early blocks if they fail to achieve mastery. The heatmap shows most environments concentrated at very low block indices, with only very few managing to progress. The cyclical reset pattern is still evident for those that do run the full course or get reset.
Pprog 0.5 (Center): Partial random progression allows more environments to reach later blocks, but the density remains higher at carlier stages due to the gating. The cyclical nature of resets is clear.
Pprog = 1.0 (Right): Environments progress freely through blocks irrespective of mastery. This results in the most pronounced eyclical bands, all environments synchronously march through the episode blocks and reset together.
Staggered Resets (Bottom Row): Across all values of Pprog. staggered resets maintain a signifi- cantly more uniform distribution of environments across different blocks at any given training update. While lower Pprog means individual environments might take longer or struggle more to traverse all blocks within their own episode lifetime, the staggering ensures that the batch fed to the learner still contains diverse experiences. The overall "texture" of the heatmap becomes denser Pprgg increases, indicating that more environments are successfully exploring the full range of blocks over time, but the crucial within-batch temporal diversity is preserved by the staggered mechanism itself.
20

---

Hew Bock Busden Over Thnnnn
Figure 7: Mean environment block distribution over training for varying progression probabilities (Pprog). Top Row :Non-Staggered): From left to right, Perog 0.0, 0.5, 1.0. Bottom Row (Stag- gered): From left to right, Ppeg 0.0, 0.5, 1.0. With non-staggered resets, low Pprog (hard skill gating) leads environments bunching at early blocks, with sparse exploration of later stages. As Ppog increases, environments progress more freely, but the strong cyclical reset pattern remains. With staggered resets, coverage is more uniform across updates regardless of Prrag, though higher Pprg allows environments to explore the full range of blocks more rapidly within their individual (staggered) episode timelines.
C.3 Impact Episode Horizon Length (H)
Figure & shows the effect of varying the episode horizon length H. For these plots, progression probability Pprog = 0.5 and reset stochasticity AR = 0 are fixed. The number of blocks B = H/T. changes with H.
Non-Staggered Resets (Top Row):
H = 10 (Left): With very short horizon, the full episode cycle H/K = 10/5 = 2 updates. The cyclical pattern appears as very rapid, almost vertical bands. While cyclical, parts of this very short episode are revisited extremely frrequentl.. The y-axis shows only 2 blocks (10/5).
H = 150 (Center): The cycle period is 150/5 5 = 30 updates. Clear diagonal bands show synchronous progression and reset. The y-axis spans 30 blocks.
The cycle period becomes 375/5 = 75 updates. The bands are elongated, indicating longer time between revisiting the same episode phase. The y-axis spans 75 blocks. This long periodicity is hypothesized to partieularly detrimental.
375 (Right):
Staggered Resets (Bottom Row): Staggered resets consistently provide diverse state visitations within each batch, regardless of the horizon length H. The heatmaps show that environments are distributed across the available blocks (which scale with H) at each update. This ensures the learner receives more stationary data stream, which is particularly beneficial for longer horizons where the non-staggered approach suffers from infrequent revisitation of early-episode states.
C.4 Impact of Reset Stochasticity (Ag)
Figure 9 visualizes influence of reset stochasticity, An. which controls the mean Poisson distribution for sampling the starting block upon reset. Here, H 200 and Pprog 0.5.
21

---

Dry Back Over Triiin
Pian The Blick Diailu Traking
Figure 8: Mean environment block distribution training for varying episode horizon lengths (H). Top Row (Non-Staggered): From left to right, H = 10, 150, 375. (Note: y-axis Block Index scales with H). Bottom Row (Staggered): From left right, H 10, 150, 375. For non-staggered resets, shorter horizons (H- = 10) lead to very rapid cycles (H/K K = 10/5 5 updates per cycle). As H increases, the period of these cycles becomes longer, potentially exacerbating learning instability. Staggered resets maintain uniform coverage irrespective of H.
Figure 9: Mean environment block distribution over training for varying reset stochasticity (Az)). Top Row : From left to right, AR = 0.0, 1.0, 2.0. Bottom Row (Staggered); From left to right, An 0.0, 1.0, 2.0. For non-staggered resets, increasing Ån slightly "fuzzes" the start of each cycle after a mass reset, but the overall cyelical progression remains. Staggered resets maintain uniform coverage; ÅR primarily influences the initial state distribution within each environment's individual staggered timeline.
Non-Staggered Resets (Top Row):
>n 0.0 (Left)): Deterministic reset block This results the sharpest eyclical bands, all environments restart precisely from the same initial state after completing episode.
22

---

AR = 1.0 (Center) and AR = 2.0 (Right); Stochastic resets mean environments restart from a distribution of initial blocks centered around block (due to Poisson sampling with mean An. then clamped). This causes the beginning of each major eycle (after most environments have run for H steps) to appear slightly "fuzzier" or more spread out near block O. However, once this initial phase passes, the environments that didn't terminate early tend to re-synchronize in their progression, and the cyclical bands through the bulk of the episode remain prominent. The inherent reset stochasticity offers only minor and temporary desynchronization.
Staggered Resets (Bottom Row): With staggered resets, the distribution of environments across blocks remains largely uniform over updates, irrespective of AR. The primary mechanism for achieving temporal diversity in batches is the explicit staggering of episode start times. The reset stochasticity parameter Åg further diversifies the exact starting block for environment when its individual (staggered) episode concludes and it resets, but it does not fundamentally change the broad, uniform coverage ensured by the staggering mechanism itself. Staggered resets effective even with deterministic environment resets (An = 0.0).
These visualizations across different parameter sweeps consistently highlight the ability of stag- gered resets to create more temporally diverse and stationary training batches compared to naive synchronous resets, providing more stable learning signal for the on-policy RL agent.
D
Wall-Time Results and Additional Analysis
D.1
Wall-Time Analysis for Staggering Granularity (Nn)
A critical aspect of staggered resets is balancing the desired temporal diversity of training batches with the computational overhead associated with environment reset operations. While an ideal scenario might involve resetting each environment a unique timestep (effectively NB Rs N, or even finer if NB R H H/sim_ dd), frequent, unbatched 'env.reset()' calls can be costly, especially in GPU-accelerated simulations. The parameter NB. representing the number of distinct stagger groups or "blocks," controls this trade-off. A smaller NB means fewer, larger groups of environments are reset/advancee synchronously, reducing reset call frequency but potentially coarsening the approximation of a truly staggered (temporally diverse) data distribution.
We empirically investigated this trade-off on the StackCube-ve v1 task (H 100, with PPO rollout K = 8, thus H/K x 12.5) by measuring the wall-clock time to reach a 70% success rate while varying No. Figure T0 illustratess the results.
The findings, shown in Figure TO indicate:
Few Blocks (Ng 1): When NB 1, all environments are effectively synchronized, resembling the naive reset scheme. This results in the slowest wall-clock convergence due to the detrimental effects of cyclical batch nonstationarity.
Moderate Blocks (NB == H/K): As NB increases, wall-clock time to convergence rapidly decreases. The optimal performance is typically observed when Ng is in the vicinity of Н/K. For StackCube-ve with K = 8, this optimal is around Nn as 10 13. This granularity provides sufficient temporal diversity in training batches to stabilize learning and accelerate convergence.
Many Blocks (Np >> H/K): Further increasing NB beyond H/K leads to marginal improvements or even a slight degradation in wall-clock convergence time. While providing finer-grained staggering, the overhead of managing and executing resets for many small, distinct groups may start to outweigh the benefits from any additional (and likely minimal) gains in data diversity.
This analysis demonstrates that judicious choice of NB. typically around H/K, allows us to effectively approximate the benefits of continuously staggered reset distribution (where each environment could theoretically start at any unique step within H) while maintaining wall-clock efficiency. This approach strikes a practical balance, achieving most of the sample efficiency gains from temporal diversity without incurring the potentially significant computational costs of excessively
23

---

Performance vs. Number of Stagger Blocks on StackCube-vi
Oprimall 12.5
That Cock Time 10% (N0
30
Number of Stagger Slocks
Figure 10: Wall-clock time to convergence (70% success on StackCube-v1) as function of the number of stagger blocks (NB). Performance, measured by faster convergence (lower wall- clock time), improves significantly as NB increases from (naive synchronous resets) towards NB R H/K (here, H 100, K 8, so H/K 12.5). Beyond this point, further increasing NB yields diminishing returns or even a slight increase in wall-clock time, likely due to overhead of managing more numerous, smaller reset groups outweighing marginal gains in temporal diversity. This demonstrates that moderate number of stagger blocks (e.g., NB x Н/K) effectively balances temporal diversity benefits with wall-time efficiency, approximating a continuously staggered reset distribution without incurring prohibitive reset costs.
frequent unbatched reset operations. The default setting our experiments for staggering (see Appendix B.3) is chosen based on this principle, typically defaulting to K.
D.2 Wall-Clock Training Time for ManiSkill Environments
Beyond improvements in sample efficiency (i.e., performance per environment step), staggered resets also demonstrate significant advantages in terms of wall-clock training time. Figure I1 presents comparison of evaluation success rates against wall-clock time for several challenging ManiSkill tasks.
AnymalCCRRuch
UnitreeG11 TrarreddtTaassttBBoo
StackCube
1.2
Eel Success Razo
Eee Succaoo Rata
0.6
0.6
0.4
0.4
0.2
0.2
Wall Time
Hell Time
Well Time
5
10
20
10
20
30
50
10
20
30
50
Staggered Resets (Durs)
Synchroneus Resets
Figure 11: Comparison of evaluation success rates versus wall-clock training time for Staggered Resets (Ours, red) and Synchronous Resets (blue) on three ManiSkill tasks: AnymalC-Res Unitreed ImitrreCiTransportaae
and StackCube. The x-axis represents wall-clock time (units may vary per plot, e.g., minutes or hours, but relative comparison key). Staggered resets consistently achieve higher success rates faster, or reach comparable success rates in significantly less wall-clock time than synchronous resets. This highlights that the benefits of improved data quality and learning stability from staggered resets translate directly into more efficient use of compute time.
As illustrated in Figure II policies trained with staggered resets (red curves) consistently achieve target success rates in substantially less wall-clock time compared to those trained with naive

---

synchronous resets (blue curves). For instance, in PaymalC-Reachh staggered resets reach over 80% success much earlier than synchronous resets begin to show significant learning. Similarly, for Unit mnitreeG1Transpottee
and StackCube, the learning curves for staggered resets are considerably steeper when plotted against wall time, indicating faster convergence to high-performing policies.
This empirical evidence supports the conclusion that the improved sample efficiency and learning stability afforded by staggered resets directly translate into reduced overall training time, making the technique not only more data-efficient but also more computationally efficient in practice for achieving desired performance levels on complex robotics tasks. The overhead of managing staggered resets is outweighed by the gains from more effective learning per unit of time.
E
Quantitative Analysis of Cyclical Nonstationarity Effects
In the main text (Sections 1, 3.2, and 4.2), we claim that the cyclical nonstationarity induced by synchronous resets can destabilize value function learning. induce policy oscillations, and cause catastrophic forgetting. This appendix provides direct quantitative evidence for these claims using the toy environment described Section 4.1. We compare two settings: one using our proposed staggered resets and one using naive synchronous resets, with moderate skill gate pnoggression.robbee to highlight the differences.
E.0.1 Metric Computation Details
To substantiate our claims, we introduce three quantitative metrics computed during the toy environ- ment training runs. The following descriptions detail their implementation.
Value Function Estimation Error. To measure the stability of the critic network, we track the value function loss at each PPO update. This is computed as the Mean Squared Error (MSE) between the critic's s value predictions for the states in the rollout buffer, (**)). and the empirical Monte-Carlo returns calculated via Generalized Advantage Estimation (GAE). Large spikes in this value indicate that the critic is struggling to predict returns, often due to a sudden shift in the underlying data distribution.
Policy Oscillations. To quantify the magnitude of policy changes between updates, we measure the approximate KL divergence between the policy before an update (Tol)) and the policy after the update (Tnew). This is calculated the mean squared difference between the dog-probabilities of the actions taken during the rollout: 0.5 [E[log nne log ooua (a (sss)) Consistently high or spiky KI. values suggest the agent is making large, unstable updates rather than smooth, incremental improvements.
Catastrophic Forgetting. To directly measure the agent's ability to retain knowledge of early- episode skills while training on late-episode data, we construct "forgetting matrix." The process is as follows:
1. At every training update i, record the agent's average accuracy each discrete block b of the task, forming an accuracy matrix A(t, b).
2. For each block, we compute the best-so-far accuracy achieved up to the current update: Atse(t, b) maxes: A(t', b)..
3. We define forgetting at update for block b as the difference between the peak performance and the current performance: Forgetting(t, b) Abea (t, b) A(t, b).
high value this matrix (visualized as bright color in the plots) signifies that the agent previously had high accuracy performance drop. A dark matrix indicates stable knowledge retention.
specific block but has since "forgotten" how to solve it, resulting in
E.0.2 Analysis of Results
Figures 12 and 13 provide compelling evidence for the detrimental effects of cyclical nonstationarity. The analysis below directly compares the corresponding plots from each figure.
25

---

Fongetiing Nabric Orighes Mare hengeting!
W
10
updare ide
Figure 12: Training is stable with Staggered Resets. The left panel shows value function error (top) and policy KL divergence (bottom). Both metries remain low and stable, with the value error (MSE) never exceeding 2.5. The right panel shows the forgetting matrix, which is almost completely dark, indicating that the agent successfully retains knowledge of all task stages throughout training.
20-
20
130 149
Figure Training unstable with Synchronous (Naive) Resets. The left panel shows catastrophic spikes in value function еиrог (top, MSE > 80) that coincide with mass reset events (updates 40, 80, 120), as well as more erratic policy updates (bottom). The forgetting matrix on the right shows bright, cyclical patterns, indicating severe forgetting of early-stage skills the training data shifts to later stages.
Value Function Instability. The value function error, visualized the top-left plot of each figure, demonstrates the most dramatic effect. With synchronous resets (Figure 13), the value function experiences catastrophic prediction error spikes. The Mean Squared Error (MSE) surges to over 80 at regular intervals. These spikes align perfectly with the mass environment reset eycle (every 40 updates, as H H/K = 200/5 40), which occurs when the data distribution abruptly shifts from late-episode states back initial states. This provides clear evidence that the cyclical data stream makes impossible for the critic to maintain a consistent estimate of state values. stark contrast, the agent trained with staggered resets (Figure 12) maintains stable and low value estimation error, never exceeding an MSE of 2.5.
Policy Oscillations. The policy oscillation metric, measured by the KL divergence between consec- utive policy updates (visualized in the bottom-left plot of each figure), also shows clear difference. While both methods exhibit some variance, the policy under synchronous resets (Figure 13) becomes significantly more erratic after the first mass reset (update 40), exhibiting larger and more frequent spikes in KI. divergence. This indicates that the learning algorithm is making large, corrective updates response to the sudden distributional shift, rather than the smooth, incremental learning facilitated by the stationary data from staggered resets.
26

---

Catastrophic Forgetting, The forgetting matrices, shown in the right panel of each figure, visualize the agent's 's inability to retain knowledge. For the agent with synchronous resets (Figure [3), the matrix reveals stark sawtooth pattern of forgetting. As the training data progresses to later blocks in an episode cycle, the agent's s accuracy on early blocks (e-g., blocks 10) is almost completely lost (bright yellow/orange). When the mass reset occurs, the agent must then relearn these initial skills. Conversely, the forgetting matrix for the staggered resets agent (Figure (2) is almost entirely black, signifying near-perfect knowledge retention across all task stages throughout training. Quantitatively, accuracy with naive resets drops by an average of 0.21 relative to its peak, while with staggered resets, the average drop is only 0.015 14-fold improvement in knowledge retention. This directly validates our central hypothesis: the cyclical data stream forces the agent into a destructive learn-forget cycle, which staggered resets completely mitigates.
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state the problem of cyclical batch nonstationarity, the proposed solution of staggered resets, and the claimed benefits (improved sample efficiency, wall-clock convergence, final performance, scalability), which are then substantiated in the experimental sections (Sections 4 and 5).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Justification: The paper acknowledges that the benefits of staggered resets less pro- nounced in locomotion tasks like HSS-Humno dWalk due to shorter effective skill horizons and natural desynchronization (Section 5.3). The discussion also frames the benefits in the context of "common high-throughpu short-rollout regimes" (Section 6).
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
28

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper proposes an empirical method and supports its claims with ex- perimental results and illustrative explanations rather than formal theoretical results or mathematical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
This paper details the intervention made to standard PPO in great detail the methodology section. Due the simplicity of the intervention, along with the great detail in which it is presented, reproduction open source benchmarks such ManiSkill) is of details full hyperparameters, best practices, and instructions for reproduction of the empirics shown.
Justification:
the main experimental results (which are done on
other researchers. The appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
29

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We are attaching a preliminary and anonymized version of the code used for the experiments we run. Additionally, we describe our method in great detail, so faithful reproduction of the experimental results is highly accessible.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc/
Bublic/guides/CCee otwp://s/ooooeeuueississsooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wttpccccpppppppp...e.eeee.eo.ctt wwciic/gguiss/eeee /ooeee//emmomdmmiiicca ssiinni ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we include all hyperparameters and implementation details in the appendix for the purpose of reproduction and comprehension of results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
30

---

Answer: [Yes]
Justification:
Justification: The paper reports performance curves with shaded regions representing standard deviation over 10 seeds for the robotics experiments (e.g.. Figure 4, caption states "Shaded area show the standard deviation over 10 seeds") and mean std dev for toy experiments (Figure 2, caption states 'mean + std dev").
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we provide sufficient information on compute resources to replicate ex- periments in the Appendix. Additionally, the majority of experiments can be interpreted as GPU-agnostic, as they focus on sample efficiency, so any GPU that is able to hold the relevant batches in memory reproduce these results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: httpt:://///eet httpp////////...../eeee/ httppuubiie/tthhccccciciaaaaaaaa
Answer: [Yes]
Justification: The research focuses on algorithmic improvements for reinforcement learning within simulated environments. It does not involve human subjects, sensitive data, or direct societal deployments that would raise immediate concerns under the NeurIPS Code of
Ethics.
31

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
The paper focuses on the technical contributions and performance improve- ments of the proposed method for reinforcement learning training, and does not include specific section or discussion on broader positíve negative societal impacts, as these are not deemed relevant within the context of the project.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. degloymeent of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper proposes an algorithmic technique for training reinforcement learning agents and does not release new datasets or pre-trained models that inherently pose a high risk for misuse requiring specific safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
32

---

We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper properly credits the creators of existing assets like ManiSkill3, SAPIEN, Isaac Gym, and Brax through citations (e.g., [1]. [6], [13], [14], [17], [28]).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper introduces new technique (staggered resets) rather than releasing
new datasets, models, or formally packaged software libraries as its primary contributions.
Guidelines:
The answer NA means that the paper does release new assets. Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The research described in the paper does not involve crowdsourcing or experi- ments with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
33

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The research described in the paper does not involve human subjects and therefore does not require IRB approval or discussion of risks to participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard component.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
34


Reviews:
[{"review_id": "OzWqNupoJs", "reviewer": "Reviewer_TQcz", "summary": "This paper proposes a new environment resetting strategy for massively parallel PPO training. The problem of typical environment resetting strategy (i.e., synrchronus reset) is that each batch of data bias towards a certain region of the state space since all data in a batch were collected in similar stage of the task. The proposed method make the initial state of each parallel environment different so that each batch of data contain states at different stages of the task. The experimental results show visible performance gain over PPO with synchronous resets.", "questions": "- Figure 3: What does each row and column mean? From the caption, it seems that (a) represents a long rollout but the bottom of the figure shows rollout 1, 2, 3 .... I'm not sure if I'm following the figure.\n- Section 5.4: As SAPG also shows better scaling, a comparison with SAPG will further strengthen the result.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength: The method is simple to implement and effective.\n\nWeakness:\n- Presentation of the method can be improved. Section 3.3 introduces the method with a large bulk of text, but it would be easier to follow if a pseudo-code is provided.\n- Lack of comparison with baselines. The proposed method is positioned as a method to improve the performance of massively parallel PPO training. If so, a comparison with other works that aim to improve PPO performance in large-scale parallel training is needed. For example, SAPG [25] cited in this paper is a good baseline to compare with. In addition, other environment reset strategies may be considered. For instance, one can uniformly sample an initial state from the environment or from the history of the agent. I believe these strategies could have a similar effect, too, and are easy to implement. Secondly, the number of parallel environments studied is relatively small compared with SAPG that use 20K environments.\n\nI'd be happy to adjust the rating if more empirical", "was_revised": true, "final_justification": "My concerns have been addressed.", "rebuttal": ""}, {"review_id": "7vdvySpxsj", "reviewer": "Reviewer_8Puz", "summary": "The authors highlight a key challenge in on-policy reinforcement learning training when scaling the amount of parallel environments. They refer to this challenge as ‘cyclical batch nonstationarity’. This occurs when many parallel environments are reset synchronously, run for fixed horizon length episodes of length $H$ for short rollouts of length $K$ at a time. Since popular on-policy algorithms interleave experience gathering with training it leads to algorithms training on chunks of data that come from limited temporal segments of the full environment episodes which leads to poor training. This is particularly apparent in environments where episodes only terminate at $H$ timesteps with no intermediate failure and reset conditions and when environments are synchronously reset at episode termination.\n\nThe authors propose a fix for this, where environments are reset in a staggered way ensuring that training data contains transitions from multiple points in the episode of the underlying environment. This is achieved in practice by stepping sub groups of all environments being simulated in parallel to various positions in time before training starts. This then ensures that each batch of training data contains a temporal mixture with data from various parts of each episode. The authors do two ablation studies on the effect of this approach, isolate cases where it is particularly useful, and show how it can improve sample efficiency and wallclock time on robotic control tasks.", "questions": "* In Figure 3 systems are trained for the same amount of environment steps for all tasks except on `UnitreeG1Transport-Box`, is this because staggered resets also suffered from policy collapse there?\n* Have the authors considered the case where environments were only reset synchronously once and then allowed to automatically terminate and reset without having to wait for any reset gates? \n* How were the hyperparameters for high dimensional robotic control benchmarks chosen?\n* The 1D toy ablation environment is defined as having B blocks (or levels) when initialised, but I cannot find a value for B in the text. Can the authors please mention what this value is?\n* The method relies on using some heuristic like no-op actions or random actions to progress environments to start at varying locations in time. Is there any impact on training based on the method used to do this?\n* What if an environment can not be suitably advanced in time with a heuristic?\n* What is the rollout length $K$ that is kept fixed as the episode horizon $H$ is increased in Figure 2a?", "limitations": "The authors do not discuss any of their work’s limitations explicitly in a Section. But mention it loosely throughout the work. I believe this work is maximally effective for training on-policy algorithms specifically on robot control tasks with fixed episode horizons where the environment does not have success or failure states leading to episodes terminating early.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths:\n* The approach is straightforward to implement and well explained. The illustrative Figure 1 is very useful for understanding the method at a glance. \n* The method is clearly shown to be effective in terms of wallclock time to convergence with increased numbers of environments – this is a well known challenge when scaling on-policy RL training batch sizes. \n* The method is clearly shown to improve sample efficiency. \n* The method is algorithm agnostic.\n* The two ablation studies are well thought out and show when the approach is maximally effective.  \n    - The first ablation study shows have staggered resets allows algorithms to overcome learning collapse when the episode horizon $H$ becomes much larger than the partial rollout length $K$ used to generate training data, when episode rests favour the initial state and are not randomised and when the environment permits passing certain skill learning bottlenecks without acquiring the desired skills. These are designed to isolate cases that happen when training on robotic control tasks. \n    - The second ablation shows how staggered resets maintain good state coverage in training batches even when rollout lengths $K$ are very short. This mirrors the state coverage achieved with very long rollout lengths while having higher update-to-data ratios. \n* The method clearly improves wall-clock time to convergence with increased numbers of parallel environments.\n\n## Weaknesses:\n* In general I find the results on the high dimensional robotic control tasks (for which this method is supposed to be maximally beneficial) to be promising, but quite limited.\n* There is limited statistical significance for the performance benchmarks in Figure 4. Only 3 random seeds makes it difficult to draw strong conclusions. Especially since the authors claim that the method allows for much faster convergence on benchmarks, so running at least 10 seeds (Agarwal et al., 2021) should be possible. \n* `The StackCube` and `PushT` tasks seem to indicate greater sample efficiency but less training stability as the performance degrades over time. To me this seems contradictory to a core point the authors are making which is that staggered resets improves training stability. Further along this point there appears to be high variance for the training performance on `TwoRobotPickCube` for staggered resets.\n\n### References\nAgarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320.", "was_revised": true, "final_justification": "I expressed concerns about the limited statistical significance and evidence for the impact of the method following from the results on the high-dimensional tasks, and I had some clarifying questions about the work.\n\nMy clarifying questions were answered during the rebuttal period, for which I thank the authors. The authors have done more experiments for increased statistical significance, but have mentioned that the metrics used to measure the impact of their method were accidentally incorrect.\n\nGiven that metrics need to fixed for some of the core results of the paper, I opted to maintain my score of 4.", "rebuttal": "We are extremely grateful for the insightful review, feedback, and opportunity to improve our work! We are glad you found the approach straightforward, well-explained, and our experiments well thought out. We address all concerns and questions below:\n\n### Weaknesses:\n>  the results on the high dimensional robotic control tasks (for which this method is supposed to be maximally beneficial) to be promising, but quite limited.\n\nWe appreciate your concern about results on high-dimensional robotic control tasks being limited. We emphasize that, to our knowledge, this is the primary domain of interest in which the massively parallel on-policy RL setting is applied. Hence, we reasoned that this domain is the ideal testbed for our method’s intended application, as well as a primary area of interest for practitioners interested in applying our work. However, we are working on follow-up experiments on game environments such as Atari and Pong, which demonstrate similar long-horizon tendencies and for which there exist vectorised environments for. We hope to include these results in a future version of the manuscript for additional empirical justification of our method.\n\n> Only 3 random seeds makes it difficult to draw strong conclusions ... The StackCube and PushT tasks seem to indicate greater sample efficiency but less training stability as the performance degrades over time. To me this seems contradictory to a core point the authors are making which is that staggered resets improves training stability. Further along this point there appears to be high variance for the training performance on TwoRobotPickCube for staggered resets.\n\nWe appreciate the concern about the small number of random seeds used in the evaluation of this method. We agree with you, and hence we are running experiments with 10 seeds rather than 3. We started with `StackCube-v1` and `TwoRobotPickCube-v1` to address your subsequent concerns about training stability and high variance between training runs. **Our findings indicate that the proposed method is actually far better at improving training stability than the original empirics would indicate.** For `StackCube-v1`, we have environment evaluation reward at **0.71 +/- 0.026 with staggered resets and 0.63 +/- 0.04 without staggered resets (n = 10)**, after training for approximately 25M environment steps. For the same experiments, we also have success rate at **0.80 +/- 0.04 with staggered resets and 0.31 +/- 0.41 without staggered resets**. For `TwoRobotPickCube-v1`, we have environment evaluation reward at **0.77 +/- 0.006 with staggered resets and 0.40 +/- 0.171 without staggered resets (n = 10)**, after training for approximately 50M environment steps. For the same experiments, we also have **success rate at 0.949 +/- 0.013 with staggered resets and 0.027 +/- 0.13 without staggered resets**. In both cases, using staggered resets provides clear advantages both in terms of increased average evaluation reward and evaluation success rate, but also in reduced variance of these quantities compared to using naive synchronous rollouts, further supporting our original claim that staggered resets reduces the variance across seeds. We firmly believe these additional results strengthen the statistical significance of our results, addressing one of your foremost concerns with the paper in its current state. We plan to have the remaining empirics added as soon as possible. We also have a table with these results, since we can't share the return/success curves directly as per NeurIPS rebuttal policy:\n\n| Task | Method | Avg. Reward (n=10) | Avg. Success Rate (n=10) |\n| :--- | :--- | :--- | :--- |\n| **StackCube-v1** | Staggered | **0.71 ± 0.026** | **0.80 ± 0.040** |\n| | Naive | 0.63 ± 0.040 | 0.31 ± 0.410 |\n| **TwoRobotPickCube-v1** | Staggered | **0.77 ± 0.006** | **0.949 ± 0.013** |\n| | Naive | 0.40 ± 0.171 | 0.027 ± 0.130 |\n\n### Questions:\n> In Figure 3 systems are trained for the same amount of environment steps for all tasks except on UnitreeG1Transport-Box, is this because staggered resets also suffered from policy collapse there?\n\nWe wanted to extend the curve for the `UnitreeG1Transport-Box-v1` task since PPO with naive synchronous resets would otherwise appear to have never learned a successful policy, when in reality it simply learns far slower than when equipped with staggered resets. This explains the difference in the number of environment steps. \n\n> Have the authors considered the case where environments were only reset synchronously once and then allowed to automatically terminate and reset without having to wait for any reset gates?\n\nAllowing for early resets comes with a myriad of considerations and tradeoffs. On one hand, it allows for better utilization of the vectorized environment, as an environment that has terminated early can begin collecting useful data quicker. However, calling `env.reset()` when not in-sync with reset gates will incur additional overhead if the environment’s reset routine is involved, as discussed in the wall-time analysis section of the appendix. Additionally, there is some risk in disrupting the uniformity of the rollout buffer afforded by staggered resets. Ultimately, we didn’t observe significant empirical differences in wall-clock convergence time when enabling versus disabling early resets in the high-dimensional robotics tasks. We are happy to include more information on this tradeoff in the final manuscript. \n\n> How were the hyperparameters for high dimensional robotic control benchmarks chosen?\n\nWe simply choose the same hyperparameters as used in the ManiSkill3 baseline implementations. The chosen values are also commonly used in other works involving high-dimensional robotics tasks in the massively parallel training regime, such as SAPG [A]. \n\n>The 1D toy ablation environment is defined as having B blocks (or levels) when initialised, but I cannot find a value for B in the text. Can the authors please mention what this value is?\n\n The value of $B$ is dependent on the horizon length $(H)$ and block length $(L)$. We fixed $L$ as 5 steps across all experiments. For instance, a 200-step horizon toy environment would yield $B = 40$ blocks/levels in the environment. We will update the paper to better clarify this. \n\n> The method relies on using some heuristic like no-op actions or random actions to progress environments to start at varying locations in time. Is there any impact on training based on the method used to do this?\n\nStaggered resets can be implemented either by (1) advancing certain environments using heuristic actions or (2) modifying the elapsed steps attribute of the vectorized environment itself. We found that, experimentally, there is no difference in performance or training based on either method, and that the choice between the two mainly depends on engineering/implementation semantics. For instance, some vectorized environments may not provide mutable access to the vector storing the number of elapsed steps for each environment. Similarly, some vectorized environments may not easily support only resetting or stepping a certain subset of the total environments. We appreciate the thoughtful question, and will add a section on these implementation considerations to the final manuscript.\n\n> What if an environment can not be suitably advanced in time with a heuristic?\n\nWe appreciate your thoughtful question. We tackle this question in our skill gating toy experiment (Figure 2c). In that experiment, when the progression probability is low, the agent cannot advance to later stages without first mastering earlier skills. Our results demonstrate that the advantage of staggered resets diminishes as these skill gates become stricter, as the environment itself imposes a curriculum that naturally forces the agent to spend more time on early-episode states, reducing the impact of cyclical nonstationarity. However, we note that, even with $p_{prog} = 0$ (the strictest case of skill gating), **the agent initialized with staggered resets achieves a return of 60, compared to 20 without staggered resets, a ~3x increase in reward**. Our explanation for staggered resets’ better performance even with strong skill gating is that, despite the *states* not being advanced when the heuristic action is performed at initialization, the *long-term reset behavior* is changed for the length of the training run, and hence when the effective skill horizon increases as training progresses, staggered resets begin to benefit the agent when cyclical nonstationarity may start to emerge. We hope this clarifies your question, and we are happy to elaborate further and incorporate elements of this explanation into the revised manuscript. \n\n> What is the rollout length K that is kept fixed as the episode horizon H is increased in Figure 2a?\n\nRollouts are 5 steps long for the experiments shown in Figure 2a. This detail $(K = 5)$ is included in the appendix, line 518, and we can revise the main manuscript to make this clearer. \n\nWe would like to once again thank you for your exceptionally thorough and constructive review. We hope our response address your concerns. In light of these clarifications, would you be able to raise your score? We are happy to address any more issues you have.\n\n[A] SAPG: Split and Aggregate Policy Gradients, Jayesh Singla et al. ICML, 2024."}, {"review_id": "gF9pDKyBwu", "reviewer": "Reviewer_jQjk", "summary": "This paper studies the problem of the rollout horizon in reinforcement learning with massively parallel environments. The paper argues that an issue in this setting is that a small rollout length can lead to a temporal non-stationary as the collected data will only ever contain data from the same timesteps in every batch. The paper demonstrates that this phenomenon can be harmful on a toy environment. Then, it proposes a solution where resets of the environments are asynchronous leading to de-correlation on the time axis. An experimental section demonstrates the efficacy of the approach in first on a toy task and then evaluates the method on a set of ManiSkill tasks.", "questions": "Q1 Can you elaborate on the structure of the toy MDP? If the progression probability is 1, why does the agent not immediate go through all levels. I seem to misunderstand something here.", "limitations": "The work does not have an explicit limitations section. It argues that the limitations are discussed by stating that the method will not work well if the effective horizons are short. That may be sufficient to establish that limitations have been mentioned.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n**Motivation**\n* The motivation of this work is that in practice, short rollouts in tasks that have long horizon dependencies are detrimental to performance and that many practitioners ignore this fact. I believe this is an interesting direction to study. \n\n**Clarity**  \n* The text is easy to follow and well written.  \n* All figures have appropriate descriptions of what can be taken away from them.  \n\n**Method**\nThe proposed method is simple and elegant.\n\n**Claims and evidence**  \n* The claim that the paper articulates the problem is well supported by both section 3 and the experiments in section 4.\n* The claim that the text characterizes the severity of non-stationarity is sufficiently supported by the toy tasks and the kernel density experiment.\n__________ \n**Neutral Points**\n\n**Novelty**  \n* The proposed method (as pointed out by the text itself) is not necessarily novel since practical implementations with staggered resets definitely exist. However, I am a defender of studies that help us better understand algorithmic pieces that often make it into implementations without proper documentation. As such, I don’t categorize the novelty of this manuscript as strength or weakness. However, such works do need to do a very thorough investigation of the presented algorithm choice.\n__________ \n**Weaknesses**\n\n**Motivation**\n* The problem itself is not well established in the text. Whether or not short rollouts are in fact needed is unclear and is only evidenced by one study in the paper (L28, [21]). Right now, the text simply states that people use short rollouts without evidence. It is not clear that using longer rollouts would yield worse results in the first place and whether the motivation for the study is one that arises artificially by choosing incorrect hyperparameters. This could be remedied either by early on providing more related work that demontrates the drawback of short rollouts or by demonstrating the drawbacks of long rollouts in a short experiment.\n\n**Clarity**\n* The toy task’s description is a bit vague. It is not clear to me what a level is, what the states in the MDP are or what the transition dynamics are. It is also unclear to me what a skill gate is exactly (see Q1).\n\n**Claims and evidence**  \n* There are various issues with claims that the text makes and does not support with evidence either from prior work of by conducting the relevant experiments.\n  * L36 “cyclical shift in the input distribution can destabilize value function learning, induce policy oscillations and hinder the agent’s ability to consolidate knowledge across the entire task horizon” At no point does the manuscript measure value functions, policy oscillations or a form of forgetting/information gain. Actually evaluating these metrics than reward would significantly strengthen the paper and increase its depth.\n  * In L68, the paper argues that the proposed approach is algorithm-agnostic. While on paper that may be true, whether staggered resets bring benefits to other algorithms has not been studied. I'm willing to believe this phenomenon might not only be relevant to PPO only but the manuscript would be improved by demonstrating this.\n  *  L220. The learner struggles to consolidate information and may forget what it learned about earlier segments by the time the data cycle returns to them, especially with many intermediate, distinct batch types. There is no measure of information consolidation or forgetting that is being reported. As mentioned before, more metrics would benefit the paper\n  * L135 This cyclical bias prevents the learner from accurately estimating values and advantages, likely due to catastrophic forgetting phenomena” - No evidence is presented for this.\n  *  L301. “Concretely, locomotion environments feature extremely large task horizons (sometimes H ≃ 1000), but the actual locomotion skill itself is much shorter horizon.” This is a conjecture that is not supported by evidence. \n*  The text claims that the problem of cyclic non-stationarity is a general problem for on-policy RL in the introduction but only results on one algorithm (i.e. PPO) are presented. The claims should probably be made more specific to PPO or more algorithms should be evaluated. This is especially relevant given that we know that PPO has rather unstable gradients [1] and issues with how batches are handeled [2].\n\n**Relation To Scientific Literature**  \n* There are various claims in the related work section that are not supported by citations. e.g.\n  * L70: \"Nonstationarity in the data distribution is a recognized challenge within reinforcement learning. Such nonstationarity can stem from various sources, including changes in environment dynamics, the reward function, or, as pertinent to our work, the data collection process itself.\"\n  * L80: \"While this strategy can be effective, underlying issues related to data distribution bias are often overlooked.\"\n  * L81: \"Some implementations incorporate partial resets—resetting environments upon task success, failure, or termination—which can introduce some eventual desynchronization. However, this process can be slow and may prove insufficient to counteract the initial bias, particularly when K is very small. \n* The related work section on non-stationarity is rather brief. Non-stationarity has played a large role in RL over the past decade and it’s representation in the text could likely be more thorough.\n\n**Experimental Design and Analyses**\n*  Given that tasks are only chosen from a single benchmark, the experimental section lacks variety. For instance, an interesting study might be to evaluate the method on PQN [3] and MinAtar [4]. This would extend the results beyond on-policy algorithms, and would increase algorithm variety and task variety.\n* The number of seeds  in the experiments is insufficient. [5, 6]\n  * It is unclear whether the presented results are statistically significant.\n  * The statement \"The variance across seeds (n = 3) is also lower with staggered resets\" can not be used to argue increased stability unless more statistically significant experiments are presented. \n\n__________ \n**Summary**\nOverall, I think this paper studies an interesting problem and a practical solution to a potentially severe problem. However, pointed out the work is not especially novel and as such, I would expect a very thorough study of the effects of this algorithmic choice. While the toy experiments are nice to gain some intuition, the results on Maniskill need to be made statistically significant, 3 seeds is simply not enough to establish superiority here. Additionally, I believe this paper could be much stronger by demonstrating generality of the problem across domains and algorithms. Finally, various claims throughout the text need to be adjusted and supported with citations; this is majorly a writing point and not so much a criticism of the method. For instance, in order to claim generality of the phenomenon, the technique needs to be evaluated on more algorithms. I think the choice of evaluation tasks is fine but several claims are being made about the results on these tasks that are not supported by quantitative evidence. If more space is required, I think the conclusion section can be significantly shortened as it mostly repeats the findings.\n\n[1] Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms? Ilyas et al.  \n[2] Batch size-invariance for policy optimization. Jacob Hilton et al.   \n[3] Simplifying Deep Temporal Difference Learning. Matteo Gallici et al.  \n[4] MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments. Kenny Young, Tian Tian.  \n[5] Deep Reinforcement Learning that Matters. Henderson et al.   \n[6] How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. Cédric Colas et al.", "was_revised": true, "final_justification": "Overall, this paper studies an interesting aspect of on-policy learning by examining the effect of time correlation in the data. After the discussion phase the authors have included a large set of new experiments that have addressed my concerns. This includes my concerns about motivation, statistical validity and generalization to other algorithms. I changed my scores for quality and significance to 3. My remaining worry is that the paper needs to go through a large chunk of revisions to adequately integrate all the new data and experiments. Given this, I will raise my score to 4 as I lean towards acceptance now. If the AC thinks that these changes are likely to be integrated accurately, I recommend the paper be accepted.", "rebuttal": ""}, {"review_id": "4lOZvKZ5XZ", "reviewer": "Reviewer_a4QY", "summary": "This paper addresses the issue of temporal correlation in on-policy reinforcement learning when using many parallel environments with long episodes. The authors point out that synchronously resetting all environments can introduce non-stationarity, as the agent's observations vary significantly depending on its position within the episode. To mitigate this, they propose a staggered reset strategy, where environments are divided into groups that are reset at different times, evenly spaced throughout the training process. Their experiments show that this approach improves performance, reduces the number of environment steps required, and scales more effectively with the number of parallel environments.", "questions": "1. Line 163: It's unclear why you specify values for $H$ and $K$ here.\n2.  I'm a bit confused by the design of your toy environment. Since the states evolve independently of the agent's actions, it seems more appropriate to describe this setup as a bandit problem rather than an MDP—is that accurate? This makes line 509 in Appendix A.3.1 puzzling, where you define \"success in an episode\" as reaching the final block $(b_t=B-1)$ at termination. But isn't the agent always in the final block at the end due to the deterministic structure of time steps? Additionally, the notion of \"success\" in this context is not clearly defined—could you clarify what it means?\nIn line 204, I also disagree with the statement that $b_0$ is \"centered around 0\" - it's centered around $\\lambda_R$. Why do you only consider $\\lambda_R \\in [0,2]$? Wouldn't it be informative to explore larger values of $\\lambda_R$? My expectation is that for larger $\\lambda_R$, the difference between the two methods in Figure 2.b) should vanish—is that correct? \n3. Line 303: You mention that \"locomotion environments feature highly stochastic dynamics.\" Could you elaborate on this? Aren't locomotion tasks essentially deterministic?\n4. It would be helpful if these figures were larger, as the axis titles are currently very difficult to read without excessive zooming. Also, is the number of blocks fixed at $H/K$ across all values of $N$ in these plots? A clarification in the caption or text would help.\n5. Building on my earlier concern (weakness 2), did you consider formulating the problem using a discounted infinite-horizon setting, where each block has an episode length $H$ drawn from a geometric distribution with parameter $1-\\gamma'$ for some $\\gamma' > \\gamma$ (and resampled for each block after termination)? This would make the effective step count a random variable, naturally introducing staggered resets and potentially offering similar performance with more stable sampling and emergent desynchronization. I’m curious whether you see specific drawbacks to this alternative compared to your current approach.\n6. The choice of combining long episode horizons (e.g., $H = 500$) with a relatively low discount factor (e.g., $\\gamma = 0.8$) in your experiments is intriguing. If I understand correctly, you assume that initial states are sampled from a relatively localized distribution. An alternative way to address the coverage issue you're targeting could be to sample initial states from a more diverse distribution (without staggering environments). Could you elaborate on whether there are any theoretical differences between this alternative and your approach? Also, what are the practical limitations: for instance, should the reset functions of environments be designed to allow more diverse initial states?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper addresses an important issue in parallel on-policy reinforcement learning. The authors do a solid job motivating the problem and highlighting its practical relevance. While I do not find the proposed solution particularly creative or original (see weakness 1 below), I am inclined to support acceptance of this paper, as it effectively illuminates the underlying issue and has the potential for high practical relevance.\n2. The proposed solution is straightforward and appears broadly applicable across different environments.\n3. The experimental section presents a range of perspectives, and I find the results to be thorough and well-documented.\n\n\nWeaknesses\n1. I am concerned about the originality of the proposed method. As the authors acknowledge: \"while some simulation environment implementations include versions of the reset staggering technique we propose [21], these are typically not detailed in accompanying publications\". While the paper claims to be the first to analyze and systematically evaluate this approach, the core idea is trivial. Given this, and the fact that the theoretical contributions are limited, the value of the paper rests heavily on empirical results—which, although solid, are not particularly surprising.\n2. While the method’s simplicity is a strength, I am concerned about the use of a deterministic sampling scheme. It's not hard to imagine environments where this determinism could be exploited or lead to failure. Some discussion or evaluation of this limitation would strengthen the paper.", "was_revised": true, "final_justification": "The authors analyze an important problem in on-policy reinforcement learning. Although the community was somewhat aware of this issue, as noted by the authors, it is valuable to have the problem clearly stated and analyzed as in this paper. The proposed method is algorithm-agnostic and requires only a straightforward modification of the underlying algorithm. Nonetheless, the authors provide a suitable and thorough analysis of the approach and demonstrate significant empirical benefits. I believe this work can have a substantial impact on the area of parallel on-policy learning and therefore recommend it for acceptance.", "rebuttal": "We are extremely grateful for the insightful review, feedback, and opportunity to improve our work! We are glad you found the issue we address important, our solution straightforward and broadly applicable, and our results thorough and well-documented. We address all concerns and questions below:\n\n#### Weaknesses:\n\n> I am concerned about the originality of the proposed method. \n\n We believe the primary contribution of this work is the systematic analysis, characterization, and empirical justification of this technique. We introduce and characterize the problem setting, empirically demonstrate its prevalence in modern RL training regimes through empirics, identify which RL environments can maximally benefit from staggered resets through the toy experiments, and provide practical implementation details such as balancing the tradeoff between environment reset overhead and rollout uniformity. By doing so, we hope to provide others in the field with a best practice implementation step that can improve performance on long-horizon tasks in the massively parallel (and popular) RL setting. We note that many priors works published at past top conferences are similarly motivated; their value is derived from investigating the \"nuts and bolts\" of RL algorithms since small implementation details can massively impact training performance [A, B, C]. \n\n> While the method’s simplicity is a strength, I am concerned about the use of a deterministic sampling scheme. It's not hard to imagine environments where this determinism could be exploited or lead to failure. \n\nOur choice of a deterministic grid-based staggering approach was a deliberate design decision to balance theoretical purity with computational efficiency and learning stability. On modern GPU environments (IsaacGym, ManiSkill, etc), calling `env.reset()` in serial incurs computational overhead, so there is a fundamental tradeoff between how deterministic / granular the staggering scheme is versus how parallelized environment resets are. We empirically validate this design choice in an experiment featured in the appendix (starting L648), where we find that, in real-world wall-clock performance, the deterministic scheme with fewer reset gates outperforms the case in which there is a reset gate at every step, which is provably the same as a randomized staggering scheme.  \n\n#### Questions:\n\n> Line 163: It's unclear why you specify values for $H$ and $K$ here.\n\nThis is an error in the manuscript, as the claim indeed holds whenever $H >> K$. We will correct the future version of the manuscript to remove this specification.\n\n> I'm a bit confused by the design of your toy environment. Since the states evolve independently of the agent's actions, it seems more appropriate to describe this setup as a bandit problem rather than an MDP—is that accurate?\n\nWhile the toy environments are somewhat inspired by bandits, the setting is different as states depend on the agent’s actions whenever $p_{prog} < 1$. For the majority of the toy experiments, $p_{prog} < 1$ (except for when we directly vary it to measure the effect of skill gating, and then for one experiment $p_{prog} = 1$), and hence we are not operating in the bandit setting for the vast majority of toy experiments. \n\n> But isn't the agent always in the final block at the end due to the deterministic structure of time steps? Additionally, the notion of \"success\" in this context is not clearly defined—could you clarify what it means? \n\nYou are correct that, when $p_{prog} = 1$, success as we define it is guaranteed. We found return to be a far more illustrative metric, and hence all of our empirical analysis of the toy environments focuses on return rather than success. We appreciate your observation and will modify the future manuscript, as the inclusion of success criteria is unnecessary and understandably confusing to the reader. \n\n> In line 204, I also disagree with the statement that $b_0$ is \"centered around 0\" - it's centered around $\\lambda_R$\n\nThis is an error in the manuscript. We will promptly update the manuscript to reflect this.\n\n> Why do you only consider $\\lambda_R \\in [0, 2]?$ \n\nWe vary $\\lambda_R$ from 0 to 2 because, as shown in Figure 2b, the difference between naive synchronous rollouts and staggered resets shrinks and the intended effect saturates. To provide some empirics, we ran the experiment again with $\\lambda_R = 10$ for 3 seeds and found that staggered resets achieves final return of $18.148 \\pm 0.149$ and naive achieves $17.998 \\pm 0.223$, confirming your expectation that the difference vanishes as $\\lambda_R$ increases. In the final manuscript, we will increase the range of values of $\\lambda_R$ in Figure 2b to better demonstrate this effect. \n\n> Line 303: You mention that \"locomotion environments feature highly stochastic dynamics.\" Could you elaborate on this? Aren't locomotion tasks essentially deterministic?\n\nOur claim referred to the overall training scheme which deliberately introduces significant domain randomization into the RL environment for sim-to-real transfer [D]. Furthermore, as we claim, a crucial source of randomness comes from the reset behavior itself, as robots attempting locomotion fall at different, unpredictable times. The combination of domain randomization and asynchronous failures introduces natural desynchronization, explaining (inline with the analysis from our toy experiments section) why the marginal benefit of our method is less pronounced for pure locomotion tasks. We will revise our claim in the future manuscript to be more precise.\n\n > Is the number of blocks fixed at $H / K$ across all values of $N$ in these plots? \n\nWe fix the number of blocks to be $H / L$, where $H$ is the maximum horizon and $L$ is the block length. This is detailed in the appendix but we will revise the main text to be more clear here.\n\n> Did you consider formulating the problem using a discounted infinite-horizon setting, where each block has an episode length $H$ drawn from a geometric distribution with parameter $1 - \\gamma'$ for some $\\gamma' > \\gamma$ (and resampled for each block after termination)?\n\nThis is an alternate way to vary the level of stochasticity in environment dynamics. Instead of varying the stochasticity of the reset distribution, your idea involves varying the stochasticity of inter-block dynamics, prior to reset. In the toy setting, we could imagine both approaches yielding similar results, except for the behavior of early resets. We believe this phenomenon of frequent partial resets relates to our findings on the high-dimensional robotics tasks. Locomotion, which experiences frequent partial resets due to early failures or successes, benefits less from staggering resets compared to other long-horizon robotics tasks, which only reset early when they succeed, which requires the policy to have almost fully converged. Hence, the natural uniformity in rollout buffer that can be achieved by frequent early resets usually does not occur in practice, because early resets generally require the training to already be near / at completion. \n\n> An alternative way to address the coverage issue you're targeting could be to sample initial states from a more diverse distribution (without staggering environments). Could you elaborate on whether there are any theoretical differences between this alternative and your approach?\n\nWe thought about whether a modified reset distribution could, in theory, afford the same benefits as staggering. consider a task with $H = 100$, 100 parallel environments, and 100 states. Scenario 1: Let the standard reset distribution $D$ just be state 1. Say we applied staggered resets with 100 reset gates. Scenario 2: Let the modified reset distribution $D’$ be the uniform distribution across all 100 states. One could imagine, under certain stationarity assumptions of the environment’s state space, these two scenarios to be provably equivalent. However, we find that, despite this interesting theoretical connection, staggered resets is more practical than solutions focusing on the *states* that an environment resets to because obtaining a suitable $D'$ is highly nontrivial. This is because, for high-dimensional tasks, resetting is highly nontrivial and current environment resetting routines in GPU simulations involve far more than sampling random vectors. Additionally, for tasks in which the majority of the set of reachable states is not known, staggered resets afford rollout uniformity whereas $D’$ would have to change with each rollout to incorporate newly visited states. Because, for high-dimensional RL tasks, the set of all reachable states is a superset of the support of the reset distribution, we would constantly need to add to an ever-growing support of states in our reset distribution. This requires far more overhead from an engineering/implementation perspective. In general, we (empirically) and the field at large find that more diverse reset distributions yield better performance [D]. However, as previously mentioned, this is highly nontrivial to achieve performantly at scale, and hence staggered resets are highly appealing as a lightweight drop-in solution that can be applied to any environment, regardless of the semantics of its reset dynamics. \n\nWe would like to once again thank you for your exceptionally thorough and constructive review. We hope our response address your concerns. In light of these clarifications, would you be able to raise your score? We are happy to address any more issues you have.\n\nReferences:\n\n[A] FastTD3: Simple, Fast, and Capable Reinforcement Learning for Humanoid Control, Younggyo Seo et al. 2025\n\n[B] Implementation Matters in Deep RL: A Case Study on PPO and TRPO, Logan Engstrom et al. ICLR 2020\n\n[C] Efficient Online Reinforcement Learning with Offline Data, Philip J. Ball et al. ICML 2023 \n\n[D] Learning to Walk in Minutes using Massively Parallel Deep Reinforcement Learning. Nikita Rudin et al. CoRL 2021"}]

Author rebuttal:

Meta-review: The work focuses on massive GPU utilization where it is found that the current approach of using parallel environments with synchronized short rollouts result in non-stationarity and loss of performance and scalability. The work proposes to stagger the resets to mitigate some of the non-stationarity effects. The approach is simple and yet provides better scalability. The reviewers overall appreciated the work but noted limited seeds and novelty. This work indeed points out a widespread issue in the current way of utilizing parallel environments, which this work will help avoid. Hence, I recommend accepting it. The authors are also highly recommended to improve their camera-ready version based on their discussion with the reviewers, e.g., increasing the number of seeds.
Decision: Accept (poster)