The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Authors: Walter Mayor
Johan Obando-Ceron
Aaron Courville
Pablo Samuel Castro
Venue: ICML 2025 poster (2025)
Keywords: Reinforcement learning
deep RL
PPO
PQN
Primary area: reinforcement_learning->deep_rl
TL;DR: We conduct an empirical analysis of these trade-offs on PPO, one of the most popular RL algorithms that uses parallel actors, and establish connections to network plasticity and, more generally, optimization stability.

Abstract: The use of parallel actors for data collection has been an effective technique used in reinforcement learning (RL) algorithms. The manner in which data is collected in these algorithms, controlled via the number of parallel environments and the rollout length, induces a form of bias-variance trade-off; the number of training passes over the collected data, on the other hand, must strike a balance between sample efficiency and overfitting. We conduct an empirical analysis of these trade-offs on PPO, one of the most popular RL algorithms that uses parallel actors, and establish connections to network plasticity and, more generally, optimization stability. We examine its impact on network architectures, as well as the hyper-parameter sensitivity when scaling data. Our analyses indicate that larger dataset sizes can increase final performance across a variety of settings, and that scaling parallel environments is more effective than increasing rollout lengths. These findings highlight the critical role of data collection strategies in improving agent performance.

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Walter Mayor" !
Johan Obando-Ceron 23 Aaron Courville 23 Pablo Samuel Castro 2.34
Abstract
This interaction forms the foundation of RL's ability to dis- cover and refine strategies that can handle nigh-dimensional and dynamic decision spaces. However, many RL tasks, particularly those involving complex environments, suffer from a critical bottleneck: the need for extensive and diverse training data to effectively generalize across the state space (Cobbe et al., 2020). This bottleneck is further compounded in online RL, where data arrives sequentially over time in non-stationary fashion (Khetarpal et al., 2022).
The use of parallel actors for data collection has been an effective technique used in reinforcement learning (RL) algorithms. The manner in which data is collected in these algorithms, controlled via the number of parallel environments and the rollout length, induces a form of bias-variance trade-off; the number of training passes over the collected data, on the other hand, must strike a balance between sample efficiency and overfit- ting. We conduct an empirical analysis of these trade-offs on PPO, one of the most popular RL algorithms that uses parallel actors, and establish connections to network plasticity and, more gener- ally, optimization stability. We examine its impact network architectures, well as the hyper- parameter sensitivity when scaling data. Our anal- yses indicate that larger dataset sizes can increase final performance across variety of settings, and that scaling parallel environments is more effec- tive than increasing rollout lengths. These find- ings highlight the critical role of data collection strategies in improving agent performance.
Traditional RI. methods often rely on single-environmen simulations which limits the speed and diversity of data collection. This approach, while straightforward, is inher- ently slow and can restrict the scope of exploration, leading to suboptimal policy learning (Rudin al., 2022; Singla et al., 2024). To address these limitations, recent advances in parallelized simulations, powered by GPU technology and distributed computing, have enabled simultancous inter- action with multiple instances the environment (Handa et al., 2023; Gallici et al., 2024; Singla et al., 2024). This innovation allows RL agents to gather significantly more data within given timeframe, which can result in more diverse training data and richer training signals (Horgan et al., 2018; Espeholt et al., 2018; Petrenko et al., 2020).
Parallelized data collection has the potential to facilitate better state space coverage, reduce variance in gradient es- timates, and accelerate the learning process by providing a larger and more diverse data for each policy update (Schul- man et al., 2017). However, the practical realization of these benefits might be hindered by intrinsic challenges within deep RL algorithms. One such challenge is the loss of plas- ticity, which refers to a network's 's ability to adapt to new information without degrading its performance on previ- ously learned tasks (Berariu et al., 2021; I; Lyle et al., 2023; Juliani & Ash, 2024; Moalla et al., 2024). Recent work has also demonstrated that scaling data can often saturate an agent's 's performance (Singla et al., 2024).
1. Introduction
Reinforcement Learning (RL) is promising framework for addressing a wide range of decision-making tasks, from robotics and autonomous driving to game playing and re- source optimization (Schwarzer et al., 2023; Vinyals al., 2019; Bellemare al., 2020; Fawzi al., 2022). Central to the success of RL. is the iterative process of data collection and policy learning, wherein agents interact with envi- ronment to gather experience, which is subsequently used to improve decision-making policies (Sutton & Barto, 2018).
Equal contribution 'Independent Researcher Mila Québec AI Institute "Université de Montréal 'Google DeepMind. Corre- spondence Johan Obando-Ceron Ceron <jobando0730@ 0awoo00000000m0mmmmmmmmmm gmail.com>> Pablo Samuel Castro <psc@e googlc.com.
With this work we systematically investigate the interplay between parallelized data collection, plasticity, learned rep- resentations, and sample efficiency. Our findings reveal that parallelized data collection can help mitigate common optimization challenges in deep RL, ultimately resulting in a significant positive impact on final performance. We
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
show that increasing the number of parallel environments (Nenv) leads to stabler training dynamics, as evidenced by reductions in weight norm and gradient kurtosis, and that this strategy is generally more effective than increasing roll- out length (NBo) for a fixed data budget (see Sec. 4 and See. 5). We conclude by providing guidance for maximiz- ing data efficiency when increasing the data collected via parallelization.
Our work builds on these insights by systematically analyz- ing how different modes of data scaling (via Nenvs and NRO) affect performance, sample efficiency, and network plas- ticity in PPO and PQN. This study isolates these variables under fixed data budgets and links their effects to optimiza- tion stability and final returns across discrete and continuous control tasks.
3. Background
2. Related Work
RI is a machine learning paradigm that enables agents to map observations to actions. When these actions are exe- cuted in an environment, the environment provides numer- ical reward and transitions the agent to new state. This typically formalized as a Markov Decision Process (MDP) (Puterman, 1994). M := (S, A, R, P, 7), where S is finite set of states, is finite set of actions available to the agent, R S x A -> [Rmin: Rmax] is the reward function that gives the immediate reward received after taking action a in state 8, P: x A + (SS) is the transition probability, where P(s, (s") represents the probability of transition- ing to state s' when the agent takes action a in state 8, and 7 E [0, is a discount factor.
Early efforts to scale RL focused on distributing computa- tion across multiple actors and learners. A3C (Mnih et al., 2016) introduced an asynchronous multi-threaded frame- work to parallelize experience collection and learning. IM- PALA (Espeholt et al., 2018) extended this idea using a decoupled actor-learner architecture with off- off-policy correc- tion, enabling efficient scaling to thousands of environments. Other systems explored synchronous data collection using large batches (Stooke& & Abbeel, 2018; Horgan et al., 2018 8), improving hardware utilization but often increasing training instability.
Recent advances in GPU-accelerated simulators such as Isaac Gym (Makoviychuk et al., 2021), EnvPool (Weng et al., 2022), and PGX (Koyamada et al., 2024) have signifi- cantly increased the throughput of data collection, enabling thousands of environment instances to run in parallel on a single device. These tools have enabled researchers to scale simulation- based learning to previously inaccessible do- mains, such as dexterous manipulation (Rudin al., 2021), legged locomotion (Agarwal et al., 2022), and complex coordination tasks (Handa et al., 2023).
The primary objective of reinforcement learning is to learn policy æ that maximizes cumulative rewards over time. A policy, which defines the behavior of an agent, is represented as π : S + (A). Two key approaches to obtaining the best policy are value-based (McKenzie & McDonnell, 2022) and policy-based (Sutton et al., 1999) methods. In value- based methods, the policy is determined by learning state- action value function and selecting actions with the highest estimated value. In contrast, policy-based methods directly optimize over the policy (Schulman et al., 2017).
Several recent studies have revisited the role of parallel data collection in improving learning performance. Li et al. (2023) showed that increasing the number of environments during training can improve exploration and stabilize leam- ing. Liu et al. (2024) demonstrated that combining parallel environment rollout with distributed learners leads to faster convergence and stronger final performance. Despite these benefits, scaling rollout length (NRo) versus environment count (Nenvs) remains underexplored.
3.1. Proximal Policy Optimization (PPO)
PPO is policy- method that alternates between sam- pling and optimization (Schulman et al., 2017). In the sam- pling step, the algorithm collects a batch B from multiple enviromments and rollouts. The agent then trains by sam- pling mini- mini-batches from B over multiple epochs; the size of B thus determines how much data is collected, while the number of epochs determines the number of times each data point is (re-)used for learning. PPO is designed to optimize the policy while maintaining stability and preventing large, potentially destabilizing, policy updates. To achieve this, PPO introduces a clipping mechanism in the objective func- tion, ensuring that policy updates remain within a predefined trust region:
While high-throughput simulators provide vast amounts of experience, prior work has shown that simply increasing batch size or rollout length is often suboptimal. Singla al. (2024) showed diminishing returns when scaling data vol- ume without accounting for structural factors such as rollout diversity or update frequency. Similar results were observed in single-environmen settings, where larger batches hin- dered generalization and slowed adaptation (Obando Ceron et al., 2024). These findings suggest that the structure of collected data, not just its volume, is critical for effective and stable training.
CLIP (0) E. [min (r. A., cl), - c, + c)A,)),
= [st) where (0) Te (ael=e] represents the probability ratio between the new policy Ty and the old policy Te... The term A, is the advantage function, which quantifies the relative
2

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
(OM Human Normalized Score
1.50
1.50
1.50
1.25
1.25
1.25
1.00
1.00
1.00
0.75
0.75
0.75
0.50
0.50
Nows =8 & Neo 128
0.50
128
0.25
Ne -8 & Nao 128
0.25
Nao x 2
0.25
Hava x
Ner. 128 & Nao
Narus x 2
Mho
0.00
0.00
0
20
40
60
80
100
0
60 Total Timesteps (in millions)
20
40
80
100
8
16
Total Timesteps (in millions)
Number of epochs
(e)
(b)
(e)
Figure/ 1. (a) Collecting the data with a large News is more effective; both experiments contain the same amount of data. (b) Scaling parallel data collection improve sample efficiency and final performance. Sealing data by increasing the News is more effective than N8O. (c) Scaling parallel data collection diminishes performance degradation as the number of epochs increases. Performance collapse is better mitigated by increasing the News All the experiments were run on the Atari-10 benchmark (Aitchison et al., 2023) for PPO (Schulman al., 2017). See Sec. 4 for training details.
improvement of an action over the expected value of the state. The hyperparameter controls the amount of clip- ping, ensuring that policy updates are restricted to prevent excessively large changes. The complete PPO objective is expressed as: [() = CCLIP(O) (0) ci cL (0) + c2 C (0), where (8) represents the Value Function Loss, given by (v.) - y y² CENT (0) is the Entropy Bonus encourag- ing exploration, and ei and 0g are coefficients that balance the contributions of each.
4. Data collection and efficiency
Our work aims to develop a better understanding of the trade- offs mentioned above, with the aim of providing guidance for how best scale data collection for more performant agents. In this section we examine how parallelized data collection can impact agent performance and plasticity.
4.1. Experimental Setup
We use the CleanRL implementation of PPO (Huang et al., 2022) with default hyperparameter settings. Our evaluation was conducted on the Arcade Learning Environment (ALE) (Bellemare et al., 2013), focusing on the Atari-10 games (Aitchison et al., 2023), which were shown to be reflective of performance on the full suite, trained for 100 million total timesteps, which is equivalent the total number of envi- ronment steps!. Following the evaluation protocol proposed by Agarwal et al. (2021), each experiment was executed using 5 independent random seeds. We report the human- normalized interquantile mean (IQM) scores, aggregated across games, configurations, and seeds, along with 95% stratified bootstrap confidence intervals. In all our figures, default configurations will be shown in black. All experi- ments were performed on NVIDIA Tesla A A100 GPUs, with each experiment requiring approximately 2-3 days.
3.2. Parallelized Data Collection
The batch B previously mentioned is obtained by run- ning News environments in parallel, each for rollout of length NRO. Their product thus defines the batch size [B] News X Noo and, as we will demonstrate below, the choice of these two factors can have a large impact on agent performance. The choice of News can have an impact on the diversity of data gathered, as governed by the support of the starting state distribution. A larger value can result in greater state-action coverage, but excessive paralleliza- tion can also result in (possibly) conflicting learning signals, highlighting the need to balance coverage and specificity. Varying NRO. on the other hand, induces a bias-variance tradeoff: longer rollouts can result in less biased estimates of returns (approaching Monte-Carlo estimates), while exces- sively long rollouts may produce updates with high variance, resulting in training instabilities.
4.2. Data collection and efficiency
Ideally, deep RL algorithms should be maximally sample- efficient by training over B multiple times. Doing so, how- ever, has been shown to overfit to the data, leading to poor performance and plasticity (Nikishin et al., 2022; D'Oro et al., 2023; Juliani & Ash, 2024). The diversity induced by increasing News can help mitigate this risk. but it may not be sufficient when the number of epochs is high (Moalla
Fixed data budgets Given a fixed data budget B News X NRo. Fig. I(a) illustrates the impact of varying News and Ngo while maintaining B fixed. Specifically, we compare the default settings (N = 8, NoO = 128) with a variant that increases the number of environments and reduces the rollout length (Nam's = 128 NRO = 8). This
'The number of environment interactions is fixed across all experiments when varying News or N8o.
et al., 2024).
3

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
result suggests that, for fixed data budget, scaling N Weavs is preferable scaling NgO-
Mao 120
128 & No
Amioal
Scaling data We next explore settings where the data budget is increased. Fig. 1(b) illustrates the impact of in- creasing B by varying Newss and NRO. While results show that increasing either leads improved performance, in- creasing News appears to yield more gains, consistent with our previous result.
Dxg-0:132
Cug-D
Name Theeeaehemm
00
Ex0-000030
Ovg-D..00 -0006)
Phoeeta in
Data reuse The number of epochs controls the number of times we train over each data point. In Fig. I(c) we ex- plore increasing the default value of 4 by factor of 2 and 4, while simultaneously increasing News and NoO. Consistent with prior works (Nikishin et al., 2022; Sokar et al., 2023), overly training on the same data, via increasing the number of epochs, negatively impacts performance. However, it can be seen that the gains obtained from increasing data, in particular by scaling Newws can help to counteract this degradation. For instance, the performance decline when training twice as many times over the data (with 8 epochs) is completely overcome by doubling the number of environ- ments.
Ov3-0.184
Ovg-0.420 0.44
42
Riverwe
Exgm0.1344
OvgmD.160
Figure 2. State-space coverage under varying parallel rollout configurations across Atari games. (I.eft) fewer environments, more rollouts per environment (Nems 8, NRo = 128). (Right) more environments, fewer rollouts (Nenvs = 128, Noo = 8).
Increasing the number of environments improves spatial coverage and distributional spread, as quantified by the coverage metric (Cvg) shown in each plot. See Sec. B.7 for Cvg metric details. The color scale indicates the critic value associated with each projected point: higher critic values are shown yellow, and lower values in bluc.
Data diversity In on-policy RL, the diversity of collected trajectories plays a key role in determining agent perfor- mance by influencing the quality of the training signal. Fig. 2 shows how varying the number of parallel environ- ments (Nenvs and rollouts per environment (NRo) affects the coverage of the learned state distribution. Increasing Nemss enhances spatial diversity by exposing the policy to wider range of initial conditions and environment stochas- ticity, while increasing NRo improves temporal depth but may induce stronger trajectory correlations. Configurations with higher News (right column) consistently yield greater state-space coverage and higher performance across Atari games, measured by iisitation-based metric. This high- lights that increasing the News, while keeping total samples constant, can lead to diverse policy updates. Fig. 9 further supports this observation by showing consistent improve- ments in state-space coverage across different Nems values.
5. Analyses
Having demonstrated the benefits of scaling environments, in this section we explore its connection to various learning dynamics and algorithmic components, with the aim to better understand the reasons for the observed gains.
5.1. Learning dynamics
To assess how learning dynamics are affected by changing data collection strategies, in Fig. we examine the following metries throughout training four representative games: feature rank, dormant neurons, weight norm and gradient kurtosis; these metrics can serve as proxy indicators of feature collapse and loss of plasticity. The results on the remaining games are provided in Fig. 10, and results on extra games in Fig. 15.
While longer rollouts might improve temporal credit assign- ment, our results suggest the gains from better state-action coverage induced by more environments are more effective at improving performance and mitigating overfitting. It is likely that this tradeoff is not fixed throughout training, and future work could explore adjusting Nenvs and NRo dynami- cally throughout training. Taken together, our results in this section can be summarized as follows.
Feature Rank Feature rank reflects the effective dimen- sionality of the representations learned by the agent's neural network (see Sec. B.1 for further details). Higher feature rank suggests richer and more diverse representations, which can contribute to improved policy learning. We consistently observe an increase in feature rank when increasing Ne suggesting that the resulting increased data diversity can improve learned representations. This is consistent with prior techniques, such as data augmentation, which have been shown to aid in learned representations.
When dealing with a fixed or scaling data budget, or when increasing the number of epochs, it is more effective scale News over NRO-
4

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Nenus & =Rc= 128
N & =Rc=122 128
Games Metrics Proximal Policy Optimization (PPO) agent.
>io/Average Episodic Retum
x10
Feature Rank
5.0
% Dormant Neurons
X20
Weigit Norm
5.0
Gradient Kurtosis
2.0
Amidar
1.5
2.0
3.0
1.5
1.0
2.0
1.0
0.5
1.0
2.0
0.0
1.0
X10
300
x30
X107
1.4
NameThisGame
1.0
6.0-
1.0
1.0
0.5
4.0
4.0
0.5
0.2
2.0
0.0
0.4
4.0
5.0
2.0
4.0-
2.0
Phoenix
3.0
1.
1.0
1.0
1.0
0.5
0.0
0.0
X160
410
K201
X107
2.5
3.0
3.0
2.0
Riverraid
2.0
2.5
1.0-
1.0
2.0
1.5
0.5
0.5
1.5
75
100
50
75
25
100
100
Total Timesteps (in millions)
Figure 3. Empirical analyses for four representative games with different amount of parallel data for PPO (Schulman al., 2017). From left to right: training returns, feature rank, dormant neurons percentage (Sokar et al., 2023), weight norm and gradient kurtosis. All results averaged over seeds, shaded areas represent 95% confidence intervals. See Sec. 4 for training details.
Dormant Neurons We measure the fraction of neurons that have gone dormant, inactive, during training (see Sec. B.2 for details), which can indicate inefficient usage of network capacity (Sokar et al., 2023; Ceron et al., 2024c;b). We observe a negative correlation between Nenvx and the level of neuron dormancy. This suggests that the increased data diversity induced by more environments can help miti- gate dormancy and thereby increase network utilization and plasticity.
butions (Garg al., 2021). High kurtosis can indicate sharp minima and unstable training which implies an increase in heavy-tailedness of the gradient distribution, whereas lower kurtosis may correspond to smoother gradients. We observe strong negative correlation between gradient kurtosis and Newsi it is possible that the increased state-action diversity induced by higher number of parallel environments has regularizing effect on the gradients, which in turn leads stable learning.
Weight Norm Weight norm measures the magnitude of the network weights (see Sec. B.3 for details) and has been shown to be correlated with training instability (Krogh & Hertz, 1991; Bartlett, 1996; Hinton al., 2012; Neyshabur al., 2015). Our results show an inverse correlation be- tween News and weight norm, suggesting that increased data collection via more parallelization can help stabilize network training.
Policy Variance and ESS Policy variance measures variability the x--) and is correlated with the network's "churn" (Schaul al., 2022). Effective Sample Size (ESS) measures the proportion of independent and high-quality samples contributing to policy updates, with higher ESS values suggesting more effective learning (Martino et al., 2017). Fig. shows that increasing News leads to lower policy variance and higher ESS, which further helps explain the observed increased performance.
Gradient Kurtusis Gradient kurtosis (see Sec. B.4 for de- tails) quantifies the sharpness or variability of gradient distri-
5

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
N & 128
16 16 & 128
dently optimize its feature representations, potentially en- hancing learning efficiency and performance, particularly in complex environments. This separation can be especially beneficial when scaling parallel data, given that in Sec. 5.1 we demonstrated that scaling Nenv. helps in learning more expressive and robust representations.
Games vs Motrics for Proximal Policy Optimization (PPO) agent.
pp aurhenagee Splencic Return
Palcc Sartnce
ESS percertage
2.-
100
Fig. 5 shows that when employing separate networks for the actor and critic, agents benefit more from scaling paral- lel data collection, leading improved final performance. However, our findings suggest that separate encoders alone do not always provide substantial performance gains over shared architectures unless paired with large-scale data col- lection. This indicates that data diversity plays a crucial role in maximizing the benefits of independent actor-critic representations.
3.0
1.00
Phoerto
0.75
0.50-
Ryarra
5.3. Hyper-parameter sensitivity
The effect of varying one algorithmic parameter is known to be tied the settings of other syper-parameters. Given our focus on increasing batch size, often via increasing rollouts, in this section we investigate the sensitivity of our findings to the choice of two related components: the learning rate and the discount factor 7.
Timesteps millions)
Figure 4. Increasing Nems for parallel data collection leads to lower policy variance and higher ESS, which results in higher average episodic return. All results averaged over 5 seeds, shaded areas represent 95% confidence intervals. See Sec. 4 for training details.
Learning rate The learning rate plays a pivotal role in the stability of RI. optimization (Andrychowicz et al.; Ceron et al., 2024a), and is generally advised to be re-tuned for larger batch sizes (Shallue et al., 2019). It bears question- ing whether the strong performance gains we have thus far observed by increasing News is sensitive to the particular choice of learning rate. Indeed, Hilton et al. (2022) sug- gest scaling the learning rate by a factor proportional to the change in B; specifically, either or vk, where l is the multplicative factor used to increase News- Fig. 11 illus- trates that the current default learning rate generally yields the best performance. While we observe some variability with different choices for the learning rate, these differences are not significant.
Key observations on increased Nems:
Increases representational diversity and pre- vents feature collapse.
Leads to broader state space coverage, ensuring better exploration.
Results in networks that are both more expres- sive and exhibit greater plasticity.
Reduces weight norm, which tend to result in stabler learning.
Exhibits low heavy-tailedness in gradients, which offers optimization stability.
Length of Rollouts and Y The choice of Nno can signif- icantly impact training efficiency and stability (Ceron et al., 2024a). A larger NoO provides more temporally extended trajectories, allowing reduced estimation bias by incorpo- rating more future rewards. However, high values of Noo can result in over-correlated data with higher variance, neg- atively affecting updates. As Fig. demonstrates, we can scale data collection via increasing News and reducing NRo-
Reduces policy variance and increases ESS, leading to stabler and more effective learning.
5.2. Decoupled architectures
The architectural choice between shared and separate en- coders for the actor and critic networks can significantly in- fluence an agent's performance (Cobbe et al., 2021). While shared encoders streamline the model and reduce computa- tional overhead, they may inadvertently constrain learning capacity by requiring a unified representation to simultane- ously optimize both policy and value functions. In contrast, using dedicated encoders allows each network to indepen-
The choice of the discount factor y is intimately tied with the rollout length, as it affects how much transitions are weighted along a trajectory, as well as in general advantage estimation (Schulman et al., 2015), which is a core compo- nent of PPO. Thus, it worth investigating how sensitive
6

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Average Episodic Return
x10a
Amidar
x10
NameThisGame
X10a
Phoenix
x10a
Riverraid
2.0
20
400
1.5
15
Shared
Decoupled
200
Dafaurt
0.5
5
0.0
0
0
25
50
75
100
25
75
100
25
50
75
100
25
50
75
100
Total Timesteps (In millions)
Figure When using separate networks to represent policy and value function (Cobbe et al., 2021), scaling parallel data collection improves final performance. Using decoupled architectures with default settings collapses performance; this collapse is mitigated by scaling News. See Sec. for training details.
Nenus & = 128
= 16 & -A0=128
Games vs Metrics Parallelised Q-network (PQN) agent. Weightt Norm
AvlAver Episodic Return
Feature Rank
3.b
% Dormant Nourons
Gradient Kurtosis
15
Amidar
2.0
1.0
0.5
1.0
NameThisGame
X20
3.0
.10
4.6
8.0
x10
4.5
2.0
6.0
4.0
1.5
1.0
1.0
1.0
0.5
2.5
0.0
5.
x10
4.0
2.0
Phoenix
2.5
3.0
1.0-
2.0
0.5
1.0
0.0
K30
1.6
x107
6.0
2.0
Riverraid
3.0
4.0-
1.0
0.4
2.0
1.0-
1.0
2.0
0.0
30
15
100
23
50
75
100
0.0
25
100
100
Total Timesteps (in millions)
Figure 6. Scaling parallel data collection on Parallelised Q-network (PQN) (Gallici et al., 2024) marginally mitigates representation
deterioration and slightly improves final performance. From to right: training returns, feature rank, dormant neurons percentage (Sokaret et al., 2023), weight norm and gradient kurtosis. All results averaged over seeds, shaded areas represent 95% confidence intervals. Sec. for training details.
the performance gains observed in Fig. 1(b) are to different values of 7. In Fig. 12 we swept over number of values for ry and found that the observed performance gains remain unaffected.
6.1. Value-based methods
PPO (Schulman et al., 2017) and Parallel Q Network (PQN) (Gallici et al., 2024>² both collect on policy parallel data, but they differ fundamentally the nature of their loss functions. PQN is a value-based learning method that simplifies and improves deep temporal difference learning approaches such as DQN (Mnih et al., 2015). It eliminates the need for both replay buffer and a target network. PQN collects batch B of experiences from multiple environments and rollouts and samples mini- bani-batches for training, similar to PPO.
6. Analysis in other settings
We extend our analysis value-based methods, which offer alternative learning paradigms, and assess the impact of parallel data collection across additional environments to evaluate the generalization of our findings.
"W use the PQN implementation from CleanRI.

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
X10
AllegroHand
x10*
Humanoid
785
7050
Havia 4096 & Map
Norralizad Final Raturn
1.0
Nerve
6053
Havs No 4096
512
0.8
2048
4800
4096
0.6
8192
0.4
612
2
0.2
262-
100
200
300
400
500
600
100 200
300
400
500
600
0.0
AllegroHand
Humanoid
AnymalTerrain
Total Timesteps (in millions)
Totall Timestops (in millions)
Isaac Gym Environment
Figure 7. Proximal Policy Optimization (PPO) (Schulman et al., 2017; Huang et al., 2022) on Isaac Gym environments (Makoviychuk a... (Left) Scaling data by increasing the News is more effective than NRo: both experiments contains the same amount data. (Right) Increasing News for parallel data collection improves final performance. We report returns over 5 runs for each experiment.
The learning objective for PQN is to minimize the mean squared error (MSE) between the predicted Q-values Qa(s;, a:), where ø are the network parameters, and the Bellman target for each sample i from B:
x10
BigRish
x10
StarPilot
Returns
1 B
((0)=
Σ B
(target: - ai(s),
Norus 64
256
Naves X
N1o x
10 15
20
25
15 20
25
target; = Ti + + T. max,' Q0(s4, d'); where 8i, as Ti, df and s/ is sampled from the current batch. We conduct similar analysis as with PPO to examine if the same benefits can be observed from scaling Newss and NRO. In Fig. 6 and Fig. 14 we can see that, although larger values of Newws do not appear to yield significant performance improvements, we do observe some mild improvements on some of the learning dynamics metries. We observe a similar trend across the remaining Atari-10 suite; see Fig. 13. Although there are number of differences between PPO and PQN, hypothesize that these qualitative differences are largely due to the choice of loss function and, more generally, the distinction between value-based and policy-based methods.
Total Timesteps (in millions)
Total Timesteps (in millions)
Figure 8. Scaling parallel data collection improves final perfor- mance on the BigFish and StarPilot games from the proegen Benchmark (Cobbe et al., 2020). We observe that both increasing News and NRo leads to greater performance, with Nens the more effective of the two, as discussed in Sec. 3.2.
stochasticity and initial states, reducing redundancy and promoting wider coverage of the state space. As shown in Fig. 9, higher Newvs configurations lead to broader empirical support in the embedded state distribution, suggesting im- proved exploration dynamics even under fixed sample bud- gets. This structural diversity can complement algorithmic exploration strategies, leading to more robust and sample efficient learning. This effect is particularly beneficial in challenging exploration tasks such as MONTEZUMA' s RE- VENGE and PHOENIX, where progress depends on rare event chains. Fig. 15 and Fig. 16 show mid-training improvements in these environments, motivating further investigation.
6.2. Evaluating on a separate benchmark
To further assess the generalization and scalability of paral- lel data collection, we extend our analysis to two environ- ments from the Proegen suite (Cobbe et al., 2020), which uses procedural generation for each level; and three Isaac Gym environments (Makoviychuk et al.). In Fig. 7 and Fig. 8 we observe the same general tendency: increasing B improves performance, but it is more effective to do by scaling Newss-
7. Discussion
The findings of this study provide insights into the funda- mental trade-offs RL training with regards to data collec- tion using parallel actors (Singla al., 2024). Our results indicate that larger dataset sizes, driven by increased News or Ngo. can enhance final agent performance. This aligns with previous research emphasizing the importance of data availability in deep lcarning and RL for improving gener- alization and robustness (Taiga et al., 2023; Kumar et al., 2022). Our analysis suggests that scaling Newva is a more effective strategy than increasing NRO. which can be an im-
6.3. Parallel environments and exploration
In ppware hard-exploration settings, increasing the number of parallel environments enhances the chance of encountering rare but informative events by amplifying the agent's 's exposure to diverse trajectories. Unlike increasing NRO. which can yield temporally correlated data, scaling the N News provides independent instantiations of environment
8

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Nemss & NrO
128
Narvs 4 & Nao 128
Nenva 8 & NRo 128
Nanus 16 & Nao 128
1.0
Amidar
Cvg=0.030
055
Cvg=0. 128
Cvg=0.10 147
-0.8
Name NameThisGame
0.6
Cvg=0. 144
0,g=0.165
Cvg=0.181
Cvg=0.238
Phoenix
0.4
Cvg=0.062
Cvg= 0.18
Cvg=0.211
Cvg=0.421 0.421
0.2
Riverrald
Cvg=0.056
Cvg=0.128
Cvg=0.141
Cvg=0.162
0.0
Figure 9. State-space coverage under varying levels of parallel data collection. Increasing the News leads to improved spatial coverage across Atari games. This suggests scaling New promotes broader exploration and diverse training data under fixed sample budget. The color scale indicates the critic value associated with each projected point: higher critic values are shown in yellow, and lower values blue.
portant consideration in settings where one is constrained by computational resources.
given the similarity to PPO in terms of data collection. We hypothesize that this is due to their difference in loss func- tions; if so, investigating this further could shed light on the fundamental and practical differences between policy- and value-based methods.
Recent works (Moalla et al., 2024; Juliani & Ash, 2024) have observed representation deterioration and loss of plas- ticity in the on-policy deep RL setting, particularly on PPO. Juliani & Ash (2024) show that plasticity loss is widespread under domain shift in this regime and that several existing methods designed to address in other contexts (Sokar al., 2023; Nikishin et al., 2022) often fail, sometimes even performing worse than taking no intervention at all. We provide evidence of the impact of parallel data collection in addressing optimization challenges, such as feature collapse and loss of plasticity.
Future work: In our analyses, like most prior works, we have maintained the values of Newv and Ngo fixed through- out training. This is most likely sub-optimal as it is known that the learning dynamics in RL vary throughout training (Nikishin et al., 2022; Lyle et al., 2024) Thus, even when maintaining the data budget fixed, the optimal trade-off be- tween News and NRO will likely vary throughout training. Future work will explore setting these values, and the size of B itself, dynamically. The recent growth in popularity and impact of machine learning, in particular large language models (LLMss,, has been largely driven by the ability of supervised learning methods to leverage massive amounts of pre-existing data. Online RL algorithms typically do not have access to this type of dataset, and must instead collect data throughout training. While parallelization can help speed this collection, agent performance is still hampered by many of the standard challenges in RL. such as those outlined in the deadly triad (Sutton & Barto, 2018). 10. If RL is to achieve scale and impact comparable to that of LLMs (beyond its use for fine-tuning LLMs), we need to develop better understanding of the challenges, and successful strategies, in parallel data collection. Our work is step in that direction, and provides guidance for future research to continue developing these insights.
In this work, we use GPU-vectorized PPO and PQN algo- rithms explore how large-batch data collection affects network plasticity and learning dynamics. We find that scal- ing the batch of data can mitigate representation collapse and reduce dormant neurons. Additionally, we observe negative correlation between the number of environments Nenvs and both weight norm and gradient kurtosis, suggest- ing that increasing data collection through parallelization stabilize network training.
Scaling the batch of data via increased parallel environments can be enhanced by exploring alternate network architec- tures. In Fig. 5 we demonstrated that, by doubling News, PPO is able to avoid collapse and obtain strong performance. The relatively low performance gains observed when scaling the batch of data PQN warrants more detailed analysis,

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Acknowledgements
form for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013. doi: 10.1613/jair.391..
The authors would like to thank Ghada Sokar, Roger Creus Castanyer, Olya Mastikhina, Dhruv Sreenivas, Ali Saheb Pasand, Ayoub Echchahed and Gandharv Patil for valuable discussions during the preparation of this work. Ghada Sokar deserves special mention for providing us valuable feed-back on an early draft of the paper. We thank the anonymous reviewers for their valuable help in improving our manuscript.
Bellemare, M. G., Candido, S., Castro, P. S., Gong, J.. Machado, M. C., Moitra, S., Ponda, S. S., and Wang, Z. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588:77 82, 2020.
Berariu, T., Czarnecki, W., De, S., Bornschein, J., Smith, S.,
Pascanu, R., and Clopath, C. A study on the plasticity of neural networks. arXiv preprint
100.20000000022 2021.
We want to acknowledge funding support from Google, CIFAR Al and compute support from Digital Research Al- liance of Canada and Mila IDT. We would also like to thank the Python community (Van Rossum & Drake Jr, 1995; Oliphant, 2007) for developing tools that enabled this work, including NumPy (Harris et al., 2020), Matplotlib (Hunter, 2007), Jupyter (Kluyver et al., 2016), and Pandas (McKin- ney, 2013).
Burda, Y., Edwards, H., Storkey, A., and Klimov, O. Exploration by random network distillation. In In- ternational Conference on Learning Representations, 2019, URL battps://openrevtiee net forum? id=R1133nR5Ym.
Ceron, J. S. O., Araújo, J. G. M., Courville, A., and Castro, S. On the consistency of hyper-parameter selection in value-based deep reinforcement learning. In Reinforce- ment Learning Conference, 2024a.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Ceron, J. S. O., Courville, A., and Castro, P. S. In value- based deep reinforcement learning, a pruned network is a good network. In International Conference on Machine Learning, pp. 38495-38519 PMLR, 2024b.
References
Ceron, J. S. O., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J. N., Dziugaite, G. K., Precup, D., and Cas- tro, S. Mixtures of experts unlock parameter scaling for deep rl. In International Conference on Machine Learning, pp. 58520-3854p PMLR, 2024c.
Agarwal, A., Kumar, A., Malik, J., and Pathak, D. Legged locomotion in challenging terrains using egocentric vi- sion. In 6th Annual Conference on Robot Learning, 2022. URL https:/ http://pweeeeeeee...eeee (d=Re3NjjwfOORZ
https:///peereevew...t
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Lever- aging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Confer- ence on Machine Learning, ICML'20. JMLR.org, 2020.
Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A. C., and Bellemare, M. Deep reinforcement learning at the edge of the statistical precipice. Advances in neural in- formation processing systems, 34:29304-29320, 2021.
Cobbe, K. W., Hilton, J., Klimov, O., and Schulman, J. Pha- sic policy gradient. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2020-2027. PMLR, 18-24 Jul 2021. URL https: http::///occedeeeeeeee ml press/ v v139 www.e/la.b...oo html.
Aitchison, M., Sweetser, P., and Hutter, M. Atari-5: Distill- ing the arcade learning environment down to five games. In International Conference on Machine Learning, pp- 421-438 PMLR, 2023.
Andrychowicz, M., Raichuk, A., Stańezyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., et al. What matters for on-policy deep actor-critic methods? a large-scale study. In International conference on learning representations.
D'Oro, P., Schwarzer, M., Nikishin, F.,, Bacon, P.-L.,, Belle- mare, M. G., and Courville, A. Sample-efficient rein- forcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations, 2023. URL https: openrevien..e 9aBBVJe.
Bartlett, P. For valid generalization the size of the weights is more important than the size of the network. Advances in neural information processing systems, 9, 1996.
net met//forum?ice
Bellemare, M. G., Naddaf, Y., Veness, J., and Bowling, M. The arcade learning environment: An evaluation plat-
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning,
10

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
I., et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In Interna- tional conference on machine learning, pp. 1407-1416. PMLR. 2018.
Juliani, A. and Ash, J. T. A study of plasticity loss in on- policy deep reinforcement learning. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024. URL htt.s: /pppreeieeeeeeeeeeeeeeeeeee / forum? i MMMM
.nt// net/
Fawzi, A., Balog, M., Huang, A., Hubert, T., Romera- Paredes, B., Barekatain, M., Novikov, A., Ruiz, J.,., Schritwieser, J., Swirszez, G., et al. Discovering faster matrix multiplication algorithms with reinforce- ment learning. Nature, 610(7930):47-53, 2022.
Khetarpal, K., Riemer, M., Rish, I., and Precup. D. To- wards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401-1476, 2022.
Gallici, M., Fellows, M., Ellis, B., Pou, B., Masmitja, I., Foerster, J. N., and Martin, M. Simplifying deep temporal difference learning. arXiv preprint 1xXiv:2407.04811 2024.
Kluyver, T., Ragan- Kelley, B., Pérez, F., Granger, B., Bus- sonnier, M., Frederic, J., Kelley, K., Hamrick, J., Grout, J., Corlay, S., Ivanov, P., Avila, D., Abdalla, S., Will- ing, C., and Jupyter Development Team. Jupyter Note- books a publishing format for reproducible computa- tional workflows. In IOS Press, pp. 87-90. 2016. doi:
Garg, S., Zhanson, J., Parisotto, E., Prasad, A., Kolter, Z., Lipton, Z., Balakrishnan, S., Salakhutdinov, R., and Ravikumar, P. On proximal policy optimization's heavy- tailed gradients. In International Conference on Machine Learning, pp. 3610-3619, PMLR, 2021.
Koyamada, S., Okano, S., Nishimori, S., Murata, Y., Habara, K., Kita, H., and Ishii, S. Pgx: Hardware-acceleratte parallel game simulators for reinforcement learning. Ad- vances in Neural Information Processing Systems, 36, 2024.
Handa, A., Allshire, A., Makoviychuk, V., Petrenko, A., Singh, R., Liu, J., Makoviichuk, D., Van Wyk, K., Zhurke- vich, A., Sundaralingam, B., et al. Dextreme: Transfer of agile in-hand manipulation from simulation to reality. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5977-5984. IEEE, 2023.
Krogh, A. and Hertz, J. A simple weight decay can im- prove generalization. Advances in neural information processing systems, 4, 1991.
Harris, C. R., Millman, K. J., Van Der Walt, S. J., Gommers, R., Virtanen, P. Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. Array programming with numpy. Nature, 585(7825):357-36) 2020.
Kumar, A., Agarwal, R., Geng, X., Tucker, G., and Levine, S. Offline q-learning on diverse multi-task data both scales and generalizes. In The Eleventh International Conference on Learning Representations, 2022.
Hilton, J., Cobbe, K., and Schulman, J. Batch size- invariance for policy optimization. Advances in Neural Information Processing Systems,35% 15:16686-77098, 2022.
Li, Z., Chen, T., Hong, Z.-.., Ajay, A., and Agrawal, P. Parallel learning: Scaling off- off-policy reinforcement learning under massively parallel simulation. In Inter- national Conference Machine Learning, pp. 19440- 19459. PMLR, 2023.
Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. R. Improving neural networks by preventing co-adaptation of feature detectors. arxiv 2012. arXiv preprint arXiv: 1207.0580, 2012.
Liu, Z., Xu, X., Qiao, P., and Li, D. Acceleration for deep reinforcement learning using parallel and distributed com- puting: A survey. ACM Computing Surveys, 57(4):1-35, 2024.
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hes- sel, M., van Hasselt, H., and Silver, D. Distributed prioritized experience replay. In International Confer- ence on Learning Representations, 2018. URL https: //openreview net/ forum? wawiammddd--D-------
Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural net- works. In International Conference Machine Learning, pp. 23190-23211, PMLR, 2023.
Huang. S., Dossa, F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and AraÂsjo, J. G. Cleanrl: High-quality single-fil file implementations of deep reinforcement leam- ing algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
Lyle, C., Zheng, Z., Khetarpal, K., Martens, J., van Has- selt, H., Pascanu, R., and Dabney, W. Normalization and effective learning rates in reinforcement learning. In The Thirty-eighth Annual Conference on Neural In- formation Processing Systems, 2024. URL https: / /poenreevi ew net/ fwr.mmrummmmmmm paterummidzzbrrr d=zb JJGGNNKK
Hunter, J. D. Matplotlib: A 2d graphics environment. Com- puting in science engineering, 9(03):00-95 2007.

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Makoviychuk, V., Wawrzyniak, L.,, Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., et al. Isaac gym: High performance gpu based physics simulation for robot learning. In Thirty-fiihh Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Systems, 2024. URL https: httttps//openrvview... forum? dd
Neyshabur, B., Tomioka, R., and Srebro, N. Norm-based capacity control in neural networks. In Conference on learning theory, pp. 1376-1401. PMLR, 2015.
Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., and Storey, K. Miles 430 macklin, david hoeller, nikita rudin, arthur allshire, ankur handa, and gavriel state. isaac 431 gym: High performance gpu based physics simulation for robot learning. Proceedings of the 432 Neural In- formation Processing Systems Track on Datasets and Benchmarks, 433, 2021.
Nikishin, E., Schwarzer, M., D'Oro, P., Bacon, P.-L.. and Courville, A. The primacy bias in deep reinforce- ment learning. In Chaudhuri, K., Jegelka, S., Song, L.. Szepesvari, C., Niu, G., and Sabato, S. (eds.), Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16828- PMLR, 17-23 Jul 2022. URL httLttttpp//pcceeeeeeee v1 62 1162/nikkshin222 html.
mlr press/
Martino, I.,, Elvira, V., and Louzada, F. Effective sam- ple size for importance sampling based on discrepancy measures. Signal Processing, 131:386 401, 2017.
Obando Ceron, J., Bellemare, M., and Castro, P. S. Small batch deep reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03442 2018.
Oliphant, T. E. Python for scientific computing. Computing in Science & Engineeringg 903):10-20, 2007. doi: 10. 1 1109/MCEE.207..5
McKenzie, M. C. and McDonnell, M. D. Modern value based reinforcement learning: A chronological review. IEEE Access, 10: 3770444-43347 2022. doi: 10.1 ACCESS.2002.33226441
Petrenko, A., Huang, Z., Kumar, T., Sukhatme, G., and Koltun, V. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. In International Conference on Machine Learn- ing. pp. 7652-7662 PMLR, 2020.
McKinney, W. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. O'Reilly Media, edition, February 2013. ISBN 978935110000 100065.
URL hwtp:ttp:/// http:://wwwwaaaaa.. wttp:/aawt.....ccc/c///accccccee obiios/
Puterman, M. L.. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., USA, 1tt edition, 1994, ISBN 0471619779.
ASIN/144931977
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Ve-
ness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wier- stra, D., Legg, S., and Hassabis, D. Human-level control through deep reinforcement learning. Nature, 518(7540): 529-533, February 2015.
Rudin, N., Hoeller, D., Reist, P., and Hutter, M. Learning to walk in minutes using massively parallel deep rein- forcement learning. In 5th Anmual Conference on Robot Learning, 2021. URL https: /pwpr/vipppee..eeeeeeeeeee net/
Rudin, N., Hoeller, D., Reist, P., and Hutter, M. Learning to walk minutes using massively parallel deep rein- forcement learning. In Conference on Robot Learning. pp. 90-100. PMLR, 2022.
Mnih, V., Badia, P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asyn- chronous methods for deep reinforcement learning. In Balcan, M. F. and Weinberger, K. Q. (eds.), Proceedings of The 33rd International Conference on Machine Learn- ing, volume 48 of Proceedings of Machine Learning Re- search, pp. 1928-1999 1937, New York, New York, USA, 20- 22 Jun 2016. PMLR. URL https: : mlr. plrss / 4 8 mmiha166 html.
Schaul, T., Barreto, A., Quan, J., and Ostrovski, G. The phenomenon of policy churn. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Advances in Neural Information Processing Systems, vol- ume 35, pp. 2537-2549. Curran Associates, Inc., 2022.
httpcppp:p//poeeeeeee /pwp/pooeediii/ igg. ngs.
Moalla, S., Miele, A., Pyatko, D., Pascanu, R., and Gulcehre, C. No representation, no trust: Connecting representation, collapse, and trust issues in PPO. In The ehirth-eighth Annual Conference on Neural Information Processing
Schulman, J., Moritz, P., Levine, S., Jordan, M. and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.0244 506.0248, 2015.
12

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv: 1iXiv:177.006445 2017.
Van Rossum, G. and Drake Jr, F. L. Python reference man- ual. Centrum voor Wiskunde Informatica Amsterdam, 1995.
Schwarzer, M., Obando Ceron, J. S., Courville, A., Belle- mare, M. G., Agarwal, R., and Castro, P. S. Bigger, better, faster: Human-level Atari with human-level ef- ficiency. In Krause, A., Brunskill, E., Cho, K., En- gelhardt, B., Sabato, S., and Scarlett, J. (eds.), Pro- ceedings of the 40th International Conference chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 90365-30380, PMLR, 23-29 Jul 2023. URL https: /ttpppp:://prrceee http///oocccccccccooooooo ngs. mlr, ...eess///// w202/schwazeer23a html.
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M.,
Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575 (7782):350-2) -04, 2019.
Ma-
Weng. J., Lin, M., Huang, S., Liu, B., Makoviichuk, D., Makoviychuk, V., Liu, Z.. Song, Y., Luo, T., Jiang, Y., et al. Envpool: A highly parallel reinforcement learn- ing environment execution engine. Advances in Neural Information Processing Systems, 35:22409-22421, 2022.
Shallue, C. J., Lee, J., Antognini, J., Sohl-Dickstein, J., Frostig, R., and Dahl, G. E. Measuring the effects of data parallelism on neural network training. Journal of Machine Learning Research, 20(1 20(112):1-49) 2019.
Yang, Y., Zhang, G., Xu, Z., and Katabi, D. Harnessing planning and reinforcement
structures for learning, arXiv preprint arXiv: (909999999222 /2255, 2019.
Singla, J., Agarwal, A., and Pathak, D. Sapg: split and aggregate policy gradients. In Proceedings of the 41st In- ternational Conference on Machine Learning, 1CML'24. JorR. 2024.
Sokar, G., Agarwal, R., Castro, P. S., and Evci, U. The dor- mant neuron phenomenon in deep reinforcement learning. In International Conference Machine Learning, pp.
32145-32168, PMLR, 2023.
Stooke, A. and Abbeel, Accelerated methods for deep reinforcement learning. arXiv preprint arXiv:1803.0255 2018.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. A Bradford Book, Cambridge, MA, USA, 2018. ISBN 0262039249.
Sutton, R. S., McAllester, D., Singh, S., and Mansour, Y. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the 13th In- lernational Conference on Neural Information Process- ing Systems, NIPS 99, pp. 1057-1063, Cambridge, MA, USA, 1999. MIT Press.
Taiga, A. A., Fedus, W., Machado, M. C., Courville, A., and Bellemare, M. G. On bonus based exploration methods in the arcade learning environment. In International Confer- ence on Learning Representationss, 2020. URL https: //openreview net //tppp/neeeeeeeeneoe/ooooonorcca /ttp://rrmmmm--deeeeseeom
Taiga, A. A., Agarwal, R., Farebrother, J., Courville, A., and
Bellemare, M. G. Investigating multi-task pretraining and generalization in reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023. URL httpa: / /pppnrevieww.....////e n Forum? 1d= RRRSSRO.
13

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
A. Training Details
In this section, we offer a detailed overview the hyperparameters and configurations for each experimental section. To ensure robustness and reliability of our results, each experimental setup is conducted with five random seeds.
Tasks We evaluate PPO and PQN 10 games from the Arcade Learning Environment (Bellemare al., 2013). We the same set of games evaluated by Aitchison et al. (2023). This includes: Amidar, Bowling, Frostbite, KungFuMaster, Riverraid, BattleZZne,, DoubleDunk, NameThisGamg Phoenix, Qbert.
We use the default hyper-parameter values for PPO (Schulman et al., 2017) and PQN (Gallici et al., 2024) agents. We share the details of these values Tab. and Tab. 2.
Hyper-parameters
Table J. Default hyper-parameters setting for PPO and PQN agents.
Atari
Hyper-parameter
PPO
PQN
Adam's 's (e)
le-5
1e-8
Adam's learning rate
2.5e-4
2.5e-4
Conv. Activation Function
ReLU
ReLU
Convolutional Width
32,64.64
32,64,64
Dense Activation Function
ReLU
ReLU
Dense Width Normalization
512
512
None
LayerNorm
Discount Factor
0.99
0.99
Exploration e Exploration decay
0.01
0.011
250000
250000
Number of Convolutional Layers
3
Number of Dense Layers
2
2
Reward Clipping
True
True
Weight Decay
0
0
Table 2. Default hyper parameters setting for PPO agent.
Procgen
Hyper-parameter
PPO
Adam's 's (6)
le-5
Adam's learning rate
5c-4
Conv. Activation Function
ReLU
Convolutional Width
16,32,32
Dense Activation Function
ReLU
Dense Width Normalization
256
None
Discount Factor
0.999
Exploration e Exploration decay
0.01
250000
Number of Convolutional Layers
Number of Dense Layers
Reward Clipping Weight Decay
True
14

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
B. Metrics
B.1. Feature Rank
This metric evaluates representation quality in deep RL by finding the smallest subspace retaining 99% variance, improving interpretability, efficiency, and stability. high feature rank indicates diverse representations and is computed using the approximate rank from Yang al., 2019; Moalla al., 2024.
NT
Where denotes the singular values of the feature matrix, represents the total number of singular values, and T is threshold (e.g., 99%) used to determine the rank. The value k, referred to as the "feature rank," corresponds to the smallest number principal components (or singular values) needed to preserve at least the total variance in the data.
B.2. Dormant Neuron
This metric quantifies the number of inactive neurons with near-zero activations, limiting network expressivity. It helps detect learning inefficiencies and improve model performance. A high number of dormant neurons means many are inactive or rarely contribute to the model's output. Its computation follows Sokar et al., 2023.
Σ 1==1
1 ((a < c) x 100,
N
where N is the total number of neurons, is the activation of neuron i, e is small threshold (e.g., 10-5), and is the indicator function.
B.3. Weight Norm
This metric quantifies the magnitude of neural network weights, helping evaluate model complexity, stability, generalization, and overfitting risks. A high weight norm indicates that the model's parameters have large magnitudes, reducing its ability to fit new targets over time. calculated as in Moalla et al., 2024; Lyle et al., 2023.
|10112
Σ 02
V
where 02 are the weights of the layer.
B.4. Kurtosis
This metric evaluates the sharpness and tail heaviness of weight gradients, highlighting distribution shape and outliers. High kurtosis indicates extreme gradient values, heavy-tailed distribution, and large gradient magnitude disparities. Following Garg et al., 2021, we additionally apply logarithmic transformation to minimize the impact of extreme values and emphasize differences between small and large gradients.
E (LLLPL))
of.
where L, log( + e) are the log- transformed absolute gradients, μ4,, is the mean of the log- -og-trannformed gradients, o the variance of the log- tran-rranformed gradients, G2 represents each individual gradient and is small positive constant prevent undefined values when G; 0.
B.5. Effective Sample Size
This metric estimates the number of independent samples in importance sampling by assessing weight dispersion, reflecting sampling efficiency and estimator accuracy. A high ESS indicates low weight variance, efficient sampling, and accurate estimator performance. It is computed using the ESS approximation from Martino et al. 2017.
ESS
ΣN (0)
15

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
where (0) represents the normalized ratio, which is computed as:
(0(
(0) Σ (3)
where r=( = mo(ae]se)
[ represents the probability ratio between the new policy we and the old policy zow The variable
N denotes the number of samples drawn from Te.. and weighted according to (0). The ESS takes values in the range 1 < ESS < N. Based this, we propose using the percentage of the ESS as:
ESS N
ESS%
x 100
B.6. Policy variance
The metric quantifies action probability dispersion across states, reflecting policy diversity and consistency within batch. It computed similarly to the policy variance measure Moalla et al. 2024.
A
policy
a
where A represents the total number of possible actions in the action space of the policy and o is the variance of the probability of action across different states, computed as:
B Pia
Pa)²
B
where B is the number of states (batch size), Pi.a is the probability of selecting action in state s, and Pa is the mean probability of selecting action a across all states.
B.7. UMAP and Coverage Metric (Cvg)
This visual and numerical metric allows us to quantify the cumulative dispersion of batch data across iterations. We begin by projecting the high-dimensional batch data into a 2D space using UMAP (McInnes et al., 2018), popular technique for visualizing high-dimensional data. UMAP is particularly suitable for this task as it preserves both the local neighborhood structure and the global layout of the data manifold, enabling a faithful representation of how the internal representations are distributed over time.
Let the resulting 2D projection be denoted as:
= (( mi) ,,, where F4 Us 10
We partition the 2D space into a uniform grid of size G G. Each point (z, y:) is assigned to grid cell as follows:
idx; min (Lar, GJ, 1), idx. = min (Lys GJ, G C--1) 1)
We define the set occupied grid cells as:
O (idx, idx!) V 1,...,N .J
Finally, the spatial coverage is computed as the fraction of grid cells visited by at least point:
O
Coverage
G
where O denotes the number of unique occupied cells, and G2 the total number of cells in the grid.
16

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Mava -8 Nig-128
N6 & Nig-129
Games vs Metrics for Proximal Policy Optimization (PPO) agent.
sppffuerage Eriendle Setum
Feature Sone
36 Dormant Nearons
Weight Norm
Gradient Brrlock
4.0
2.0
0.2-
2.5
4.0
1.0
1.0
2.0
3.0
0.21
DoubleDunk
2:0-
0.2-
3.0
Frocibue
0.0
NamnTThoGuree
0.2
2.2
5.5
1.0
3.2-
2.0
2.0
3.0
2.04
1.2
25
0.5
50
75
100
25
75
200
22
30
100
25
32
75
120
Total Timesteps millions)
Figure 10. Empirical analyses for ALE games with different amount of parallel data for PPO (Schulman et al., 2017). From left right: training returns, feature rank, dormant neurons percentage (Sokar al., 2023), weight norm and gradient kurtosis, All results averaged over 5 seeds, shaded areas represent 95% confidence intervals. See Section 4.1 for training details.
17

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
= x 10
(Default)
Ir x k
Irx vk
Ir+
Ir +
vik
x10²
Average Episodic Return
x10*
Average Episodic Return
4.0
NameThisGame
3.0
Amidar
2.0
1.0
0.0
x10
x10
2.5
4.0
2.0
Phoenix
Riverraid
1.5
2.0
1.0
0.5
0.0
25
50
100
Total Timesteps (in millions)
Figure 11. Varying the learning rate by factor proportional to the increase to Nums- In this figure Newe was scaled by factor of 2, as Hilton et al. (2022) suggest increasing the learning accordingly. In addition to that increase, we explore decreasing the learning by the same factor. Consistent with Hilton et al. (2022). PPO is robust to learning rate changes; the default learning is 2.5 x 10
0.95 (Default)
0.99
0.97
0.9
x10
Average Episodic Return
x10
Average Episodic Return
Name ThisGamee
3.0
Amidar
1.5
2.0
1.0
1.0
0.0
x10
x10
2.5
6.0
Phoenix
Riverraid
2.0
4.0
1.5
1.0
2.0
0.5
0.0
25
50
75
100
0
25
50
75
100
Total Timesteps (in millions)
Figure 12. Performance gains from NBo x are unaffected by varying v. where the default value isy = 0.95.
18

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
Mara Nig=128
Nanua =16 Nxp=128
Q Q-network (PQN) agent
30
200
-0
2.0
6.7
0.04
4.0-
5.0
2.2-
3.0
DoubleDunk
0.5
LungPuNasser
2.0
7.0
0.04
5.0-
2.0
3.0
Biveraid
3.0
20
4.0m
4.0
3.2
255
2.0
0.04
120
Total Timesteps (in millions)
Figure 13. Empirical analyses for ALE games with different amount of parallel data for PON (Gallici al., 2024). From left right: training returns, feature rank, dormant neurons percentage (Sokar et al., 2023), weight norm and gradient kurtosis. All results averaged over 5 seeds, shaded areas represent 95% confidence intervals. See Section for training details.
19

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
1QM Human Normalized Score
2.0
Nenus = & =Ro=128
2.0
Nenus 128 & ~
1.5
1.5
1.0
1.0
Nenus= =8 & NRo=128
0.5
0.5
NRO
Nanus 2
0.0
0.0
0
20
40
60
80
100
0
20
40
60
80
100
Total Timesteps (in millions)
Total Timesteps (in millions)
Figure Evaluating impact scaling parallel data collection PQN agent (Gallici al., 2024) on Atari- 10 benchmark (Aitchison et al., 2023). See Section 4.1 for training details.
Nerus Nao=128
Nerus #16 & Nao=128
Games vs Metrics for Proximal Policy Optimization (PPO) agent.
340
Average Episodic Return
Feature Rank
% Dormant Neurons
Weight Norm
1.0
3.3
2.0-
0.8
1.2
Freeway
3.2
1.0
0.4
0.8
0.5
0.2
0.5
3.0
x10
x1.0
x10
3.0
1.5
Gravitar
2.0
1.2
1.0
2.0
1.0
1.0
1.0
0.5
0.0
100
X307
6000
X307
MontezumaRevenge
3.0
4.0
0.6
1.0
0.8
0.2
0.5
0.0
0.0
25
50
75
100
25
50
75
100
25
50
75
100
0
25
50
100
Total Timesteps (in millions)
Figure 15. Evaluating the impact of increasing Nans on separate set ALE games. These are the so-called "hard exploration games" from Taiga al. (2020). We observe that increased batch size results in equivalent or improved performance, and overall improvement on the learning dynamics measures. All results averaged over seeds, shaded areas represent 95% confidence intervals. See Section for training details.
20

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
MontezumaRevenge
Venture
1750
Nenvs 128
1500
Nenvs
1500
1250
Nenvs 64
1250
Returns
1000
750
750
500
500
250
40
80
100
20
Total Timesteps (in millions)
Total Timesteps (in millions)
Figure 16. Evaluating the impact of increasing Newss separate set of ALE games. These the so-called "hard exploration games" from Taiga et al. (2020). Scaling data by increasing the News is more effective than NRo. All results averaged over 5 seeds, shaded areas represent 95% confidence intervals. For this experiment, PPO+RND (Burda et al., 2019) algorithm from Cleanrl (Huang al., 2022) library.
Nerus =1
N4rw5
Novs-8
Nows-16
Metrics Proximal Pallcy Optimization (PPO) agent.
Average Episodic
Feature Rank
Dormant Neurons Percentage
Weight
Gradienn Kurtosis
2.0
2.5
6.0
Anidar
1.5
2.0
0.5
1.0
0.5-
0.0
0.0
-10
+30
K20
-32
NomeTTssaane
1.5
0.5
1.0
0.6
3.0
1.0-
0.0-
0.2
0.0-
5.0
+10
2.5
-30
-30
300
8.0
2.5-
1.a
2.0
4.0-
0.5
1.0
1.0
Biverall
2.0
0.0.
1.0
10-
15a
4.0
2.0
1.5
0.6-
0.4-
25
30
75
150
25
30
75
1.00
25
50
150
25
30
100
25
50
75
150
Total Timesteps (in millions)
Figure 17. Empirical analyses for four representative games with different amount of parallel data for PPO (Schulman al., 2017). From left to right: training returns, feature rank, dormant neurons percentage (Sokar et al., 2023), weight norm and gradient kurtosis. Inereasing Nems for parallel data collection improves final performance. All results averaged over 5 sceds, shaded areas represent 95% confidence intervals.
21

---

The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networks
x10
Alle
Nenvs
8.0
8192
4096
6.0
Returns
2048
512
4.0
2.0
0.0
100
200
300
400
500
600
Total Timesteps (in millions)
x10
Humanoid
AnymalTerrain
8.0
8
6.0
6
5.0
4.0
4
3.0
2
2.0
1.0
0
0.0
100
200
300
400
500
600
20
40
60
80
100
Total Timesteps (in millions)
Total Timesteps (in millions)
Figure 18. Proximal Policy Optimization (PPO) (Schulman et al., 2017; Huang et al., 2022) Isaac Gym environments (Makoviychuk al.) when increasing News. Increasing New. for parallel data collection improves final performance. We report returns over 5 runs for each experiment.
22


Reviews:
[{"review_id": "L6jWLmHuad", "reviewer": "Reviewer_ZCcE", "summary": "The paper claims that larger batch sizes in deep reinforcement learning obtained by parallelized data collectors help mitigate several optimization challenges, listed below, and give recommendations on the scaling of the dimensions of the batch size (num envs vs rollout size).\n1) Performance gains: Increasing the batch size improves sample efficiency of PPO and PQN.\n2) Batch dimensions recommendation: It is better to increase the number of parallel environments than the rollout size.\n3) Better learning dynamics: \n  a) a higher batch size improves plasticity and learned representations (higher feature rank, less dormant neurons, lower weight norm, lower kurtosis, less policy variance, higher effective sample size)\n  b) A higher batch can be more important when using separate actor and critic networks.\n4) Connection to other hyperprameters: it seems unnecessary to vary the learning rate and discount factor when varying the dimensions of the batch size.\n\n## update after rebuttal\nI have read the other reviews and comments from the authors, and I maintain the score of my review. The authors agreed to make their contributions clearer and explained the validity of their experimental protocol, which I encourage them to make much clearer in the final version of the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback! We are happy that the reviewer found that “the claims made in the paper are interesting and give important insights to RL researchers and practitioners” and that “the claims are backed by sufficient empirical evidence”.\n\nWe respond to their main concerns below.\n\n## Only use individual env curves, which could be prone to cherry-picking.\n\nDue to our limited compute budget, we reported individual game curves only for the subset of games included in the main analysis of the paper. However, we will run the experiments for the remaining games in the suite and share the corresponding aggregated plots in the final version of the paper. Note that Phoenix and NameThisGame were the two games used for studying representation collapse on PPO [1].\n\n\n[1]. Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. No representa- tion, no trust: Connecting representation, collapse, and trust issues in PPO. In Advances in Neural Information Processing Systems, 2024.\n\n\n## It's not easy to identify the precise main claims of the paper from the abstract and the introduction. \n\n\nThank you for the suggestion regarding presentation and clarity. We will include precise statements of our claims in both the abstract and the introduction in the final version.\n\n\n## … the benefit of increasing the batch size comes with induced extra compute, and the paper does not compare to other methods that could use this extra compute more effectively \n\n\nWe agree that increasing batch size comes with additional computational cost, and we acknowledge that our current comparisons focus primarily on data collection strategies within a fixed algorithmic framework (mainly PPO). Our intention with this study is to isolate and understand the trade-offs inherent in data collection design, specifically, how varying the number of parallel environments and rollout lengths affects learning dynamics and final performance. In the final version, we will clarify this scope more explicitly and emphasize that our results are complementary to other lines of work focusing on algorithmic improvements or more compute-efficient approaches.", "claims_and_evidence": "All the claims are backed by sufficient empirical evidence.\n\n1) Figure 1\n2) Figure 1\n3a) Figure 2, Figure 3\n3b) Figure 4\n4) Appendix Figure and Figure 9.\n\nClaims 1, 2 are well illustrated by using aggregated performance metrics across the 10 envs, however, claims 3b and 4, which also argue about performance, only use individual env curves, which could be prone to cherry-picking. While it may be the case that the paper wanted to present more details by including individual envs, I believe that these claims would be better illustrated (have a complete point) by using the aggregate performance over the 10 envs like done in Figure 1.", "methods_and_evaluation_criteria": "The Atari benchmark with the subset of Atari-10 is a good choice for this work, as it's been widely used in similar work on plasticity.\nThe work does not need to compare to other methods to make its claim; a comparison with the base PPO is enough.", "theoretical_claims": "The paper does not make any theoretical claim.", "experimental_designs_or_analyses": "The paper makes a good use of aggregate metrics (Figure 1) when claiming overall performance improvement and using individual env training curves (Figures 2, 3, 4, 5) to show improvements on learning dynamics, which cannot easily be aggregated.\nNevertheless, I have made comments in the section about claims on how I believe the paper could make better use of these two reporting strategies.\n\nThe choice of the number of environments and seeds makes a good balance for statistical significance.\n\nThe paper uses a safe and sound experimental protocol by using an implementation by a popular codebase (CleanRL) with its default hyperprameters.", "relation_to_broader_scientific_literature": "The paper related its finding to the plasticity loss literature, but it is likely that other optimization features are impacted by varying the batch size, the paper does not discuss such broader relations.", "essential_references_not_discussed": "The essential references are discussed.", "other_strengths_and_weaknesses": "Presentation & clarity\n\nIt's not easy to identify the precise main claims of the paper from the abstract and the introduction. Both of these sections present a global claim that a higher batch size through parallelized data collectors helps mitigate several optimization challenges, but nothing more precise.\nThe reader has to wait for the conclusion of each section to get the precise claim. I believe the paper would be better if the claims were presented in a precise way earlier in the paper.\n\nImpact\n\nThe claims made in the paper are interesting and give important insights to RL researchers and practitioners, however, they are quite narrow as the benefit of increasing the batch size comes with induced extra compute, and the paper does not compare to other methods that could use this extra compute more effectively.", "other_comments_or_suggestions": "No extra comments.", "questions_for_authors": "No additional questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IxSPOLrS4t", "reviewer": "Reviewer_qnDC", "summary": "This paper focuses on the problem of reinforcement learning with multiple environments, which has gained increasing interest over the past years due to GPU utilization. Through empirical analysis of the effect of the number of environments and the length of rollouts, the authors provide recommendations (e.g., increasing the number of parallel environments) on how to improve the performance of deep RL agents under this parallel environment setting.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback! We are glad that the reviewer finds that “the study is well-timed”, and “nice to have a paper dedicated to studying this setting and finding best practices”.\n\nWe address their main concerns below.\n\n## two variables using two data points, which is unsubstantiated.  To claim such a trend, the authors need to show the performance on the y-axis and different values of # environments on the x-axis. That is, the performance increases when using x2/x3/x4/etc the number of environments.\n\nWe agree with the reviewer that more data points are needed to support this claim. We conducted additional experiments by varying the number of environments ([see Figs 1,2,3](https://shorturl.at/Ycvgx)); following your suggestion, we also plot the number of environments (#Envs) on the x-axis against returns ([see Fig 5](https://shorturl.at/Ycvgx)) and agree that this helps clarify our message. The conclusions remain the same: increasing the number of environments mitigates optimization-related issues in deep RL and improves performance, unlike simply increasing the number of rollouts.\n\n\n## Another example is where the authors claim a positive correlation between  and performance/feature-rank.\n\n\nTo support our claims, we report training returns, feature rank, percentage of dormant neurons, weight norm, and gradient kurtosis for different values of #Envs. ([see Figure 1](https://shorturl.at/Ycvgx)) shows a strong correlation between #Envs and the metrics used to study the dynamics of deep RL agents [1, 2]. Larger #Envs mitigate feature collapse and reduce dormant neuron percentage.\n\n## contradictory results with PQN, discrepancy with original paper, and inconclusive results\n\nWe would like to highlight that the focus of our paper is on PPO. PQN was chosen solely to conduct a comparative analysis with PPO, as both algorithms use on-policy parallelized data collection but differ in their loss functions. We will soften our claims to avoid overstating the findings.\n\nThe discrepancy with the published results likely arises from differences in the experimental framework and hyperparameter tuning. Given that we are studying PQN strictly in reference to PPO, we chose to use the same set of hyperparameters across both algorithms to isolate the effects of the loss function and data collection strategy. Notably, while PPO uses 4 epochs by default, the authors of PQN used 2, which results in improved performance. We will clarify these differences in the final version.\n\nWe realize that perhaps including PQN in the background section gives the impression that PQN is part of the main focus of the paper, and propose to move section 2.2 to the appendix to avoid this confusion.\n\n\n## number of independent runs (3 runs) is very low\n\n\nThe 3 runs specified in Figures 7 and 10 are incorrect, the actual number of runs were 5; we apologize for the confusion and will correct the typo. We followed the experimental setup from [1,2], where they run experiments with 5 seeds.  Further, we are running additional experiments with 5 more seeds to increase statistical significance and will include them in the final version of the paper.\n\n##  is the number of total environment interactions the same when comparing algorithms?\n\nThe number of environment interactions is in fact not the same when comparing with different batch sizes (whether by changing the number of environments or the rollout length). PPO (and PQN) leverage parallelism via simulated environments for their training setup, and it is this setting we explore in our work. As such, we compare the various methods across gradient steps as opposed to environment interactions, as this is a better indication of how well parallelized data collection can be leveraged for learning. Thank you for raising this, as it is a subtle point that we will clarify in our final version.\n\n\n## suggestions for figure clarity and citation update\n\n\nThank you for raising these points, we will correct them.\n\n\n## References\n\n\n[1]. Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. No representa- tion, no trust: Connecting representation, collapse, and trust issues in PPO. NeurIPS’24.\n\n[2]. Obando-Ceron, J., Sokar, G., Willi, T., Lyle, C., Farebrother, J., Foerster, J. N., Dziugaite, G., Precup, D., and Castro, P. S.. Mixtures of experts unlock parameter scal- ing for deep rl. ICML’24", "claims_and_evidence": "The authors tend throughout the paper to claim a relationship between two variables using two data points, which is unsubstantiated. \n- For example, one of the main claims in the paper is that increasing the number of environments yields more gains with or without fixing the update budget (Figure 1a, 1b). To claim such a trend, the authors need to show the performance on the y-axis and different values of # environments on the x-axis. That is, the performance increases when using x2/x3/x4/etc the number of environments. The claim can be made if we can observe a positive correlation between the two variables. Additionally, it would be even better if the authors could show this for multiple update budgets.\n- Another example is where the authors claim a positive correlation between $N_{\\text{envs}}$ and performance/feature-rank. Also, they claim a negative correlation between $N_{\\text{envs}}$ and the level of neuron-dormancy/weight norm. Those conclusions are followed by experiments on PQN, which mostly contradicts what the authors discovered with PPO. The results with PQN should allow the authors to reconcile their claims and never overstate the conclusions.\n- The results with PQN seem to be contradictory to the results from the PQN paper. The authors here showed that increasing the number of environments doesn’t help, whereas the PQN paper shows it does. The authors need to explain this discrepancy.\n- In Figure 5, The authors claimed that increasing $N_{\\text{envs}}$ in PQN mitigates representation deterioration and slightly improves final performance, but the results are inconclusive.", "methods_and_evaluation_criteria": "The authors evaluate the Atari arcade environments, which is a well-known benchmarking suite and suitable for the scope of the paper and its parallel environment setting.", "theoretical_claims": "No theoretical claims are presented in the paper.", "experimental_designs_or_analyses": "The main issue in the evaluation is the number of independent runs (3 runs) is very low, so I suspect the statistical significance of the results. I highly encourage the authors to increase the number of runs to at least 10.\n\n**A minor issue:** the authors need to show the dependent variable on the y-axis and the name of the environment as the title, not the other way around.", "relation_to_broader_scientific_literature": "The related works section covers prior work.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "The study is well-timed since there is an increasing interest in using RL under the parallel environment setting, which is parallelizable with modern GPUs. Usually, parallelizable methods such as PQN/PPO are presented with a specific set of hyperparameters, so it is nice to have a paper dedicated to studying this setting and finding best practices that apply to this category of methods.\n\nI have concerns about the experiments and the validity of the conclusions. Please refer to my comments under the claims and evidence section.", "other_comments_or_suggestions": "In line 92, value-based (McKenzie & McDonnell, 2022) -> this reference needs to be updated with an older reference.", "questions_for_authors": "To have a fair comparison, I want to make sure that the number of total environment interactions ($N_{\\text{envs}}\\times T$, where T is the number of interactions in each environment) is the same when comparing algorithms that use different $N_{\\text{envs}}$. Can the authors confirm that this is the case?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ela8T2D0oM", "reviewer": "Reviewer_NUj3", "summary": "The paper investigates how scaling parallel data collection (i.e., the product of the number of parallel environments\n$N_\\text{envs}$ and rollout length ($N_\\text{RO}$) affects the performance and representation quality of deep RL agents, focusing primarily on PPO (and briefly on a value-based variant, PQN).\n\nThe authors shop empirically, that for the same batch size $B$ using a larger $N_\\text{envs}$ yields better learning performance on Atari-10. They attempt to correlate this finding with known metrics that are linked to loss of plasticity showing that larger $N_\\text{envs}$ appear better in this regard.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback, useful comments, and address their concerns below. \n\n## The claim is too general and suggests that there should be a trend (potentially a scaling law)\n\nWe ran extra experiments varying the number of environments, rollout lengths, and across different domains to support the claims made in the paper. Please [see Figures 1,2,3,4,5](https://shorturl.at/Ycvgx). Figs 1, 2, and 5 show that increasing #Env improves final performance. Additionally, Fig 1 demonstrates that representation collapse and the percentage of dormant neurons are mitigated when using a larger #Env. Figs 3 & 4 confirm that increasing #Env is more effective than increasing #Rollouts. Notably, Fig 4 also shows performance boosts in sparse reward games.\n\n##  higher dormant neurons does not mean plasticity has been lost\n\nWe agree that increased dormancy or weight norm changes do not directly imply a loss of plasticity. We were motivated to investigate these metrics given their use as a proxy for plasticity loss in recent papers [1,2,3,4,5].  Our intent was to show a correlation between these features and training dynamics, not to assert causality.\n\n## why PPO and Atari?\n\nWhile PPO was originally proposed for continuous control, it has been widely and successfully used in discrete action settings, especially in the Atari domain [6]. We chose PPO due to its popularity, and relevance in benchmarking studies and studying deep RL learning dynamics [1]. Additionally, we chose ALE, as it  is a well known benchmark for studying issues such as loss of plasticity [7,8], so using it provides a useful reference point for our findings.\n\nNonetheless, we have added extra experiments on IsaacGym [9] which confirm our findings (see [Figures 2,3,5](https://shorturl.at/Ycvgx)).\n\n\n## unsure about the significance of PQN “learning dynamics metrics” improvements\n\nWhile we agree that the performance improvements with PQN are not as pronounced, our findings do suggest that using a larger number of environments leads to more parameter efficient learning. Nevertheless, we will soften our claims on PQN to avoid misrepresenting our results.\n\n## can same behaviour be shown on continuous control tasks?\n\nWe ran experiments on Isaac Gym [9], and the results support the main thesis of the paper. Larger #Envs mitigate optimization issues and improve performance, [see Figs 2,3,5](https://shorturl.at/Ycvgx).\n\n## seperate conclusion section\n\nWe will restructure the paper to include a dedicated conclusion section. \n\n## separate analysis for long-horizon tasks (or tasks with sparse reward signals)...\n\nWhile our current set of tasks includes varying levels of difficulty and reward sparsity, we agree that explicitly studying sparse-reward environments would provide deeper insight. To this end, we examine the impact of on-policy parallelized data collection on two  hard exploration [10] games using PPO+RND [11] from CleanRL [12]; [see Figure 4](https://shorturl.at/Ycvgx). We observe that the claims made in the paper generally hold in these scenarios as well. \n\n## test beyond 256 or 512 parallel environments to see if the improvements continue\n\nWhile we have begun running these experiments, they will not be able to finish before the rebuttal period as they are more computationally expensive.  Once completed, we plan to include larger-scale experiments in the final version.\n\nHowever, the new IsaacGym results suggest that our results hold when scaling to higher values ([see Figures 2, 3,5](https://shorturl.at/Ycvgx)).\n\n## References\n\n[1] Moalla et al. No representation, no trust: Connecting representation, collapse, and trust issues in PPO. NeurIPS’24.\n\n[2] Juliani & Ash. A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning. NeurIPS’24.\n\n[3]. Ahn et al. \"Prevalence of Negative Transfer in Continual Reinforcement Learning: Analyses and a Simple Baseline.\" ICLR’25.\n\n[4]. Nauman et al. \"Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning.\" ICML’24.\n\n[5]. Dohare et al. \"Loss of plasticity in deep continual learning.\" Nature (2024)\n\n[6]. Ellis et al.  Adam on Local Time: Addressing Nonstationarity in RL with Relative Adam Timesteps. NeurIPS’24.\n\n[7]. Nikishin et al. \"Deep reinforcement learning with plasticity injection.\" NeurIPS’23\n\n[8] Lyle et al. \"Understanding plasticity in neural networks.\" ICML’23\n\n[9]. Makoviychuk et al. Isaac gym: High performance gpu based physics simulation for robot learning. 2021\n\n[10]. Taiga et al. \"On Bonus Based Exploration Methods In The Arcade Learning Environment.\" ICLR’20\n\n[11]. Burda et al. Exploration by random network distillation. ICLR’19\n\n[12]. Huang et al. \"Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.\" JMLR’22", "claims_and_evidence": "> Claim 1: keeping the total batch size fixed $B$ fixed, more parallel environments yield superior performance compared to longer rollouts in fewer environments.\n\nWhile the experiments that the authors provide can suggest this, they are only performed on two different configurations ($N_\\text{envs}=8$, $N_\\text{RO} = 128$ and $N_\\text{envs}=128$, $N_\\text{RO} = 8$).\nThe claim is too general and suggests that there should be a trend (potentially a scaling law), but in order to warrant such a claim there would need to be experiments for different sets of parameter configurations for $N_\\text{envs}$ and $N_\\text{RO}$.\nCurrently, all that can be said is that the one hyperparameter configuration beats the other one. we do not know if e.g. $N_\\text{envs}=32$, $N_\\text{RO} = 32$ would be the best one for example.\n\n> Claim 2: Scaling the total collected data can mitigate issues such as loss of plasticity.\n\nWhile the authors provide some metrics that have been linked to loss of plasticity, I am unsure whether their claim is justified here. First, prior work has used to explain potential reasons for the loss of plasticity, however, simply because the fraction of dormant neurons or the weight norms are higher, does not mean that plasticity has been lost. Especially, since the agents' performance still increases, i.e. there might be a correlation here, however, it's not clear if it is causal and if loss of plasticity is happening.", "methods_and_evaluation_criteria": "I am not sure, why the authors chose the combination of PPO and Atari, i.e. an algorithm for contrinuous action environments on discrete action environments for their main analysis. Especially as PQN (which as a DQN/value-based method is the more common choice for Atari) does not show any difference in performance. The authors claim PQN still shows improved performance in the \"learning dynamics metrics\" they track, however, I am unsure about the significance here.\n\nAs mentioned above, in order to convincingly show scaling behaviour, the authors need to provide more probing points for different configurations and show some trend.\n\nFurther, the paper would also benefit if in addition the same behaviour could be shown on continuous control tasks. Especially, since there have been works on massively scaling PPO there (Rudin et al. Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning, 2021).", "theoretical_claims": "The authors make no theoretical claims in the paper.", "experimental_designs_or_analyses": "see above.", "relation_to_broader_scientific_literature": "-", "essential_references_not_discussed": "-", "other_strengths_and_weaknesses": "Strength:\n- \n\nWeaknesses:\n- Scope is limited to few discrete action environments\n- Too few parameter configurations in order to make a general claim about the scalability\n- Some experimental details are missing which should be added (e.g. batch size)\n- No theoretical analysis or insights", "other_comments_or_suggestions": "- The paper does not have a seperate conclusion section. This should be added as currently, the discussion also contrains some conclusions. The author should follow common practice to seperate this into an independent section.\n- A separate analysis for long-horizon tasks (or tasks with sparse reward signals) could highlight whether shorter rollouts are indeed always better or if certain environments require longer rollouts to reduce bias.", "questions_for_authors": "Have the authors tested or do they plan to test beyond, say, 256 or 512 parallel environments to see if the improvements continue (or saturate)? Especially given that there have been works in continuous state-action RL (Rudin et al. 2021).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates the batch size problem in parallel on-policy RL and demonstrates that scaling the number of environments offers several benefits, such as performance enhancement and improved representation.

Although the paper's scope appears narrow, it presents extensive experiments across various environments (both discrete and continuous) and hyperparameter settings. All reviewers concur that the claims made in the paper are intriguing and provide valuable insights for RL researchers and practitioners.
Decision: Accept (poster)