Title: Staggered Environment Resets Improve Massively Parallel On-Policy Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

robotics

ppo

on-policy

simulation
Abstract: Massively parallel GPU simulation environments have accelerated reinforcement learning (RL) research by enabling fast data collection for on-policy RL algorithms like Proximal Policy Optimization (PPO). To maximize throughput, it is common to use short rollouts per policy update, increasing the update-to-data (UTD) ratio. However, we find that, in this setting, standard synchronous resets introduce harmful nonstationarity, skewing the learning signal and destabilizing training. We introduce staggered resets, a simple yet effective technique where environments are initialized and reset at varied points within the task horizon. This yields training batches with more uniform state visitation distributions, reducing the nonstationarity induced by synchronized rollouts. We characterize dimensions along which RL environments can benefit significantly from staggered resets through toy environments. We then apply this technique to challenging high-dimensional robotics environments, achieving significantly higher sample efficiency, faster wall-clock convergence, and stronger final performance. Finally, this technique scales better with more parallel environments compared to naive synchronized rollouts, yielding more optimal utilization of computational resources.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the issue of temporal correlation in on-policy reinforcement learning when using many parallel environments with long episodes. The authors point out that synchronously resetting all environments can introduce non-stationarity, as the agent's observations vary significantly depending on its position within the episode. To mitigate this, they propose a staggered reset strategy, where environments are divided into groups that are reset at different times, evenly spaced throughout the training process. Their experiments show that this approach improves performance, reduces the number of environment steps required, and scales more effectively with the number of parallel environments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper addresses an important issue in parallel on-policy reinforcement learning. The authors do a solid job motivating the problem and highlighting its practical relevance. While I do not find the proposed solution particularly creative or original (see weakness 1 below), I am inclined to support acceptance of this paper, as it effectively illuminates the underlying issue and has the potential for high practical relevance.\n2. The proposed solution is straightforward and appears broadly applicable across different environments.\n3. The experimental section presents a range of perspectives, and I find the results to be thorough and well-documented.\n\n\nWeaknesses\n1. I am concerned about the originality of the proposed method. As the authors acknowledge: \"while some simulation environment implementations include versions of the reset staggering technique we propose [21], these are typically not detailed in accompanying publications\". While the paper claims to be the first to analyze and systematically evaluate this approach, the core idea is trivial. Given this, and the fact that the theoretical contributions are limited, the value of the paper rests heavily on empirical results—which, although solid, are not particularly surprising.\n2. While the method’s simplicity is a strength, I am concerned about the use of a deterministic sampling scheme. It's not hard to imagine environments where this determinism could be exploited or lead to failure. Some discussion or evaluation of this limitation would strengthen the paper.\n\nQUESTIONS:\n1. Line 163: It's unclear why you specify values for $H$ and $K$ here.\n2.  I'm a bit confused by the design of your toy environment. Since the states evolve independently of the agent's actions, it seems more appropriate to describe this setup as a bandit problem rather than an MDP—is that accurate? This makes line 509 in Appendix A.3.1 puzzling, where you define \"success in an episode\" as reaching the final block $(b_t=B-1)$ at termination. But isn't the agent always in the final block at the end due to the deterministic structure of time steps? Additionally, the notion of \"success\" in this context is not clearly defined—could you clarify what it means?\nIn line 204, I also disagree with the statement that $b_0$ is \"centered around 0\" - it's centered around $\\lambda_R$. Why do you only consider $\\lambda_R \\in [0,2]$? Wouldn't it be informative to explore larger values of $\\lambda_R$? My expectation is that for larger $\\lambda_R$, the difference between the two methods in Figure 2.b) should vanish—is that correct? \n3. Line 303: You mention that \"locomotion environments feature highly stochastic dynamics.\" Could you elaborate on this? Aren't locomotion tasks essentially deterministic?\n4. It would be helpful if these figures were larger, as the axis titles are currently very difficult to read without excessive zooming. Also, is the number of blocks fixed at $H/K$ across all values of $N$ in these plots? A clarification in the caption or text would help.\n5. Building on my earlier concern (weakness 2), did you consider formulating the problem using a discounted infinite-horizon setting, where each block has an episode length $H$ drawn from a geometric distribution with parameter $1-\\gamma'$ for some $\\gamma' > \\gamma$ (and resampled for each block after termination)? This would make the effective step count a random variable, naturally introducing staggered resets and potentially offering similar performance with more stable sampling and emergent desynchronization. I’m curious whether you see specific drawbacks to this alternative compared to your current approach.\n6. The choice of combining long episode horizons (e.g., $H = 500$) with a relatively low discount factor (e.g., $\\gamma = 0.8$) in your experiments is intriguing. If I understand correctly, you assume that initial states are sampled from a relatively localized distribution. An alternative way to address the coverage issue you're targeting could be to sample initial states from a more diverse distribution (without staggering environments). Could you elaborate on whether there are any theoretical differences between this alternative and your approach? Also, what are the practical limitations: for instance, should the reset functions of environments be designed to allow more diverse initial states?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of the rollout horizon in reinforcement learning with massively parallel environments. The paper argues that an issue in this setting is that a small rollout length can lead to a temporal non-stationary as the collected data will only ever contain data from the same timesteps in every batch. The paper demonstrates that this phenomenon can be harmful on a toy environment. Then, it proposes a solution where resets of the environments are asynchronous leading to de-correlation on the time axis. An experimental section demonstrates the efficacy of the approach in first on a toy task and then evaluates the method on a set of ManiSkill tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n**Motivation**\n* The motivation of this work is that in practice, short rollouts in tasks that have long horizon dependencies are detrimental to performance and that many practitioners ignore this fact. I believe this is an interesting direction to study. \n\n**Clarity**  \n* The text is easy to follow and well written.  \n* All figures have appropriate descriptions of what can be taken away from them.  \n\n**Method**\nThe proposed method is simple and elegant.\n\n**Claims and evidence**  \n* The claim that the paper articulates the problem is well supported by both section 3 and the experiments in section 4.\n* The claim that the text characterizes the severity of non-stationarity is sufficiently supported by the toy tasks and the kernel density experiment.\n__________ \n**Neutral Points**\n\n**Novelty**  \n* The proposed method (as pointed out by the text itself) is not necessarily novel since practical implementations with staggered resets definitely exist. However, I am a defender of studies that help us better understand algorithmic pieces that often make it into implementations without proper documentation. As such, I don’t categorize the novelty of this manuscript as strength or weakness. However, such works do need to do a very thorough investigation of the presented algorithm choice.\n__________ \n**Weaknesses**\n\n**Motivation**\n* The problem itself is not well established in the text. Whether or not short rollouts are in fact needed is unclear and is only evidenced by one study in the paper (L28, [21]). Right now, the text simply states that people use short rollouts without evidence. It is not clear that using longer rollouts would yield worse results in the first place and whether the motivation for the study is one that arises artificially by choosing incorrect hyperparameters. This could be remedied either by early on providing more related work that demontrates the drawback of short rollouts or by demonstrating the drawbacks of long rollouts in a short experiment.\n\n**Clarity**\n* The toy task’s description is a bit vague. It is not clear to me what a level is, what the states in the MDP are or what the transition dynamics are. It is also unclear to me what a skill gate is exactly (see Q1).\n\n**Claims and evidence**  \n* There are various issues with claims that the text makes and does not support with evidence either from prior work of by conducting the relevant experiments.\n  * L36 “cyclical shift in the input distribution can destabilize value function learning, induce policy oscillations and hinder the agent’s ability to consolidate knowledge across the entire task horizon” At no point does the manuscript measure value functions, policy oscillations or a form of forgetting/information gain. Actually evaluating these metrics than reward would significantly strengthen the paper and increase its depth.\n  * In L68, the paper argues that the proposed approach is algorithm-agnostic. While on paper that may be true, whether staggered resets bring benefits to other algorithms has not been studied. I'm willing to believe this phenomenon might not only be relevant to PPO only but the manuscript would be improved by demonstrating this.\n  *  L220. The learner struggles to consolidate information and may forget what it learned about earlier segments by the time the data cycle returns to them, especially with many intermediate, distinct batch types. There is no measure of information consolidation or forgetting that is being reported. As mentioned before, more metrics would benefit the paper\n  * L135 This cyclical bias prevents the learner from accurately estimating values and advantages, likely due to catastrophic forgetting phenomena” - No evidence is presented for this.\n  *  L301. “Concretely, locomotion environments feature extremely large task horizons (sometimes H ≃ 1000), but the actual locomotion skill itself is much shorter horizon.” This is a conjecture that is not supported by evidence. \n*  The text claims that the problem of cyclic non-stationarity is a general problem for on-policy RL in the introduction but only results on one algorithm (i.e. PPO) are presented. The claims should probably be made more specific to PPO or more algorithms should be evaluated. This is especially relevant given that we know that PPO has rather unstable gradients [1] and issues with how batches are handeled [2].\n\n**Relation To Scientific Literature**  \n* There are various claims in the related work section that are not supported by citations. e.g.\n  * L70: \"Nonstationarity in the data distribution is a recognized challenge within reinforcement learning. Such nonstationarity can stem from various sources, including changes in environment dynamics, the reward function, or, as pertinent to our work, the data collection process itself.\"\n  * L80: \"While this strategy can be effective, underlying issues related to data distribution bias are often overlooked.\"\n  * L81: \"Some implementations incorporate partial resets—resetting environments upon task success, failure, or termination—which can introduce some eventual desynchronization. However, this process can be slow and may prove insufficient to counteract the initial bias, particularly when K is very small. \n* The related work section on non-stationarity is rather brief. Non-stationarity has played a large role in RL over the past decade and it’s representation in the text could likely be more thorough.\n\n**Experimental Design and Analyses**\n*  Given that tasks are only chosen from a single benchmark, the experimental section lacks variety. For instance, an interesting study might be to evaluate the method on PQN [3] and MinAtar [4]. This would extend the results beyond on-policy algorithms, and would increase algorithm variety and task variety.\n* The number of seeds  in the experiments is insufficient. [5, 6]\n  * It is unclear whether the presented results are statistically significant.\n  * The statement \"The variance across seeds (n = 3) is also lower with staggered resets\" can not be used to argue increased stability unless more statistically significant experiments are presented. \n\n__________ \n**Summary**\nOverall, I think this paper studies an interesting problem and a practical solution to a potentially severe problem. However, pointed out the work is not especially novel and as such, I would expect a very thorough study of the effects of this algorithmic choice. While the toy experiments are nice to gain some intuition, the results on Maniskill need to be made statistically significant, 3 seeds is simply not enough to establish superiority here. Additionally, I believe this paper could be much stronger by demonstrating generality of the problem across domains and algorithms. Finally, various claims throughout the text need to be adjusted and supported with citations; this is majorly a writing point and not so much a criticism of the method. For instance, in order to claim generality of the phenomenon, the technique needs to be evaluated on more algorithms. I think the choice of evaluation tasks is fine but several claims are being made about the results on these tasks that are not supported by quantitative evidence. If more space is required, I think the conclusion section can be significantly shortened as it mostly repeats the findings.\n\n[1] Are Deep Policy Gradient Algorithms Truly Policy Gradient Algorithms? Ilyas et al.  \n[2] Batch size-invariance for policy optimization. Jacob Hilton et al.   \n[3] Simplifying Deep Temporal Difference Learning. Matteo Gallici et al.  \n[4] MinAtar: An Atari-Inspired Testbed for Thorough and Reproducible Reinforcement Learning Experiments. Kenny Young, Tian Tian.  \n[5] Deep Reinforcement Learning that Matters. Henderson et al.   \n[6] How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments. Cédric Colas et al.\n\nQUESTIONS:\nQ1 Can you elaborate on the structure of the toy MDP? If the progression probability is 1, why does the agent not immediate go through all levels. I seem to misunderstand something here.\n\nLIMITATIONS:\nThe work does not have an explicit limitations section. It argues that the limitations are discussed by stating that the method will not work well if the effective horizons are short. That may be sufficient to establish that limitations have been mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors highlight a key challenge in on-policy reinforcement learning training when scaling the amount of parallel environments. They refer to this challenge as ‘cyclical batch nonstationarity’. This occurs when many parallel environments are reset synchronously, run for fixed horizon length episodes of length $H$ for short rollouts of length $K$ at a time. Since popular on-policy algorithms interleave experience gathering with training it leads to algorithms training on chunks of data that come from limited temporal segments of the full environment episodes which leads to poor training. This is particularly apparent in environments where episodes only terminate at $H$ timesteps with no intermediate failure and reset conditions and when environments are synchronously reset at episode termination.\n\nThe authors propose a fix for this, where environments are reset in a staggered way ensuring that training data contains transitions from multiple points in the episode of the underlying environment. This is achieved in practice by stepping sub groups of all environments being simulated in parallel to various positions in time before training starts. This then ensures that each batch of training data contains a temporal mixture with data from various parts of each episode. The authors do two ablation studies on the effect of this approach, isolate cases where it is particularly useful, and show how it can improve sample efficiency and wallclock time on robotic control tasks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n* The approach is straightforward to implement and well explained. The illustrative Figure 1 is very useful for understanding the method at a glance. \n* The method is clearly shown to be effective in terms of wallclock time to convergence with increased numbers of environments – this is a well known challenge when scaling on-policy RL training batch sizes. \n* The method is clearly shown to improve sample efficiency. \n* The method is algorithm agnostic.\n* The two ablation studies are well thought out and show when the approach is maximally effective.  \n    - The first ablation study shows have staggered resets allows algorithms to overcome learning collapse when the episode horizon $H$ becomes much larger than the partial rollout length $K$ used to generate training data, when episode rests favour the initial state and are not randomised and when the environment permits passing certain skill learning bottlenecks without acquiring the desired skills. These are designed to isolate cases that happen when training on robotic control tasks. \n    - The second ablation shows how staggered resets maintain good state coverage in training batches even when rollout lengths $K$ are very short. This mirrors the state coverage achieved with very long rollout lengths while having higher update-to-data ratios. \n* The method clearly improves wall-clock time to convergence with increased numbers of parallel environments.\n\n## Weaknesses:\n* In general I find the results on the high dimensional robotic control tasks (for which this method is supposed to be maximally beneficial) to be promising, but quite limited.\n* There is limited statistical significance for the performance benchmarks in Figure 4. Only 3 random seeds makes it difficult to draw strong conclusions. Especially since the authors claim that the method allows for much faster convergence on benchmarks, so running at least 10 seeds (Agarwal et al., 2021) should be possible. \n* `The StackCube` and `PushT` tasks seem to indicate greater sample efficiency but less training stability as the performance degrades over time. To me this seems contradictory to a core point the authors are making which is that staggered resets improves training stability. Further along this point there appears to be high variance for the training performance on `TwoRobotPickCube` for staggered resets.\n\n### References\nAgarwal, Rishabh, et al. \"Deep reinforcement learning at the edge of the statistical precipice.\" Advances in neural information processing systems 34 (2021): 29304-29320.\n\nQUESTIONS:\n* In Figure 3 systems are trained for the same amount of environment steps for all tasks except on `UnitreeG1Transport-Box`, is this because staggered resets also suffered from policy collapse there?\n* Have the authors considered the case where environments were only reset synchronously once and then allowed to automatically terminate and reset without having to wait for any reset gates? \n* How were the hyperparameters for high dimensional robotic control benchmarks chosen?\n* The 1D toy ablation environment is defined as having B blocks (or levels) when initialised, but I cannot find a value for B in the text. Can the authors please mention what this value is?\n* The method relies on using some heuristic like no-op actions or random actions to progress environments to start at varying locations in time. Is there any impact on training based on the method used to do this?\n* What if an environment can not be suitably advanced in time with a heuristic?\n* What is the rollout length $K$ that is kept fixed as the episode horizon $H$ is increased in Figure 2a?\n\nLIMITATIONS:\nThe authors do not discuss any of their work’s limitations explicitly in a Section. But mention it loosely throughout the work. I believe this work is maximally effective for training on-policy algorithms specifically on robot control tasks with fixed episode horizons where the environment does not have success or failure states leading to episodes terminating early.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new environment resetting strategy for massively parallel PPO training. The problem of typical environment resetting strategy (i.e., synrchronus reset) is that each batch of data bias towards a certain region of the state space since all data in a batch were collected in similar stage of the task. The proposed method make the initial state of each parallel environment different so that each batch of data contain states at different stages of the task. The experimental results show visible performance gain over PPO with synchronous resets.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: The method is simple to implement and effective.\n\nWeakness:\n- Presentation of the method can be improved. Section 3.3 introduces the method with a large bulk of text, but it would be easier to follow if a pseudo-code is provided.\n- Lack of comparison with baselines. The proposed method is positioned as a method to improve the performance of massively parallel PPO training. If so, a comparison with other works that aim to improve PPO performance in large-scale parallel training is needed. For example, SAPG [25] cited in this paper is a good baseline to compare with. In addition, other environment reset strategies may be considered. For instance, one can uniformly sample an initial state from the environment or from the history of the agent. I believe these strategies could have a similar effect, too, and are easy to implement. Secondly, the number of parallel environments studied is relatively small compared with SAPG that use 20K environments.\n\nI'd be happy to adjust the rating if more empirical\n\nQUESTIONS:\n- Figure 3: What does each row and column mean? From the caption, it seems that (a) represents a long rollout but the bottom of the figure shows rollout 1, 2, 3 .... I'm not sure if I'm following the figure.\n- Section 5.4: As SAPG also shows better scaling, a comparison with SAPG will further strengthen the result.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}