Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning
Authors: Ghada Sokar
Pablo Samuel Castro
Venue: NeurIPS 2025 poster ()
Keywords: Reinforcement Learning
Deep RL
Scaling RL networks
Primary area: reinforcement_learning
TL;DR: We identify the connection between the output of the encoder and the ensuing dense layers as the main underlying factor limiting scaling capabilities in deep RL

Abstract: Scaling deep reinforcement learning in pixel-based environments presents a significant challenge, often resulting in diminished performance. While recent works have proposed algorithmic and architectural approaches to address this, the underlying cause of the performance drop remains unclear. In this paper, we identify the connection between the output of the encoder (a stack of convolutional layers) and the ensuing dense layers as the main underlying factor limiting scaling capabilities; we denote this connection as the **bottleneck**, and we demonstrate that previous approaches implicitly target this bottleneck. As a result of our analyses, we present global average pooling as a simple yet effective way of targeting the bottleneck, thereby avoiding the complexity of earlier approaches.

Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning
Ghada Sokar Google DeepMind gsokar@google. com
Pablo Samuel Castro Google DeepMind pscêgoogle. com
Abstract
Scaling deep reinforcement learning in pixel-based environments presents signif- icant challenge, often resulting in diminished performance. While recent works have proposed algorithmic and architectural approaches address this, the under- lying cause of the performance drop remains unclear. In this paper, we identify the connection between the output of the encoder (a stack of convolutional layers) and the ensuing dense layers as the main underlying factor limiting scaling capabilities; we denote this connection as the boftleneck, and we demonstrate that previous approaches implicitly target this bottleneck. As result of our analyses, we present Global Average Pooling (GAP) as a simple yet effective way of targeting the bottleneck, therehy avoiding the complexity of earlier approaches.
Introduction
Reinforcement Learning (RL) is widely considered one of the most effective approaches for complex
sequential decision- problems [Mnih et al.) 2015 Vinyals et al.) 2019 Bellemare et al.) 2020
Degrave et al.) 2022 Wurman et al. 2022]. in particular when combined with deep neural networks (typically referred as deep RL). In contrast the so-called "scaling laws" observed supervised learning (where larger networks typically result in improved performance) (Kaplan et al. 2020). it difficult to scale RL networks without sacrificing performance. There has been recent line of work aimed at developing techniques for effectively scaling value-based networks, such as via the use of mixtures-of-expprrs [Obando Ceron et al., 2024)), network pruning [Obando Ceron et al. 2024), tokenization [Sokar et al. [025] and regularization [Nauman et al. 2024). Most of these techniques tend to focus on structural modifications to standard deep RL networks by leveraging sparse-network training techniques from the supervised learning literature.
Obando Ceron et al. [2024] first demonstrated that naively scaling the penultimate (dense) layer in an RI. network results in decreased performance, and proposed the use of soft mixtures-of-experrs [SoftMoEs: Puigcerver et al., 2024] to enable improved performance from this form of scaling.
Sokar et al. 2025] argued that the gains from SoftMoES were mostly due to the use of tokenization. Relatedly, Obando Ceron et al. (2024) demonstrated that naïvely scaling the convolutional layers hurts performance, and showed that incremental parameter pruning yields gains that grow with the size of the original, unpruned, network. While effective, these methods are non-trivial to implement and can result in increased computational costs.
One unifying aspect of the aforemeenioned works that they tend to be most effective on networks that process pixel inputs, such as when training on the Arcade Learning Environment (ALE) [Bellemare al. 2013). These networks are typically divided into an encoder Φ consisting of a series of convolutional layers, followed by series of dense layers a/: thus, for an input v. the network output given by (((())) Obando Ceron" " al. [2024] and [Sokar al. [2025) scaled the first layer of d Obando Ceron et [2024] scaled all layers in ø. is worth highlighting that 1é o is, in practice, a set of weights connecting the output of ((x) with the input layer of vb.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Bottleneck
Baseline PPuning)
( +)
HPW tocare (shared paramataa))))
SoftMoE-1
Q ))
C-dmmnnions
GAP
Q -)
Figure I: Illustration of the bottleneck pixel-based networks. Standard dense networks (Baseline) connect all 6 outputs with v,, resulting in // W x C' dim(v) parameters (scaled down when using pruning, shown in red). SoftMoE-1 converts ø's outputs into H x W tokens of dimension C: the sharing of learned parameters across tokens results in bottleneck with C x dim(+) parameters. GAP performs average pooling across H x W spatial dimensions, resulting in C' feature maps and C' x dim(yp parameters in the bottleneck.
In this work, argue that the underlying cause behind the effectiveness of the aforementioned methods that they result in a bottleneck between 6(x) and & (see Figure 1p. As we will argue, performance gains are mostly due a well-structured bottleneck, rather than the recently proposed architectural modifications to tb. This insight suggests that simpler architectural interventions may be just as effective, which we demonstrate by using Global Average Pooling (GAP) as effective mechanism for enabling improved performance from scaling.
Our contributions can be summarized follows: (i) We investigate the challenges leading to the performance degradation of scaled RL networks; (ii) we study the underlying reasons behind the success of existing architectural approaches in scaling: and (iii) present pooling faster, simpler method yielding superior performance. We begin in Section 2 providing the necessary background on reinforeement learning, detailing the architecture employed in pixel-based deep RL, and outlining architectural modifications proposed by recent methods for addressing scaling challenges. Section 3.1 then describes our experimental setup. Our analyses investigating the difficulties of scaling RI. networks and how existing methods attempt to address them are presented Sections 33. and 3.3
respectively. Section is dedicated our results and analysis on RL networks with GAP. Finally, Section covers the related work, followed by a conclusion and discussion in Section 6
2 Preliminaries
2.1 Reinforcement learning
Reinforcement learning involves an agent moving through a series states x E X' by select- ing action E A at discrete timesteps. After selecting action o, from state Ir. the agent will receive a reward ry (x,, a,), and its goal to maximize the discounted sum cumulative rewards Ee y're, where y [0. by finding optimal policy quantifies the agent's behavior at each state. Value-based methods [Sutton and Barto [1998] maintain estimates of the value of selecting action from state and following afterwards: Q(I, a) == E. 2Σ (r're(x,, at) (z) = x, Og = a, ag ~ x (2:)]], where π is induced from Q, for ((2.a)
X A(A) which
instance with the use of softmax: (a) = Edes (
Mnih et al. 2015 demonstrated deep neural networks can be very effective at approximating Q- values, even for complex domains such as Atari games [Bellemare et al. 2013]; their network has served as the backbone for most deep RL networks. Later, Espeholt et al. 20188 proposed

---

using ResNet based architecture which demonstrates significant performance improvements over the original convolutional neural network (CNN) architecture. For pixel- environments, this family of networks consists of a set of convolutional layers [Fukushima] 1980]. which we will collectively refer to as & (and often referred to as the encoder or representation), followed by a set of dense layers, which we will collectively refer to as tb. Thus given an input the network approximates the Q-values as (xx, -) ()(())).
The output of á is 3-dimensioni tensor H x W C. where H is height, W is width, and C' is the number of feature maps (channels); this output is typically flattened before being fed to th. Thus, the number of parameters for the functional composition & ó is equal to H x W C x dim(v). where dim( (()) is the dimensionality of the first dense layer in tb. We refer to connection between and v as the boftleneck, and it will be the focus of most of our work. See Fegure I| for an illustration.
2.2 Network scaling
As we will demonstrate below, this bottleneck has direct impact on learning efficiency. The standard approach is to flatten the output of 6 into a single vector before feeding it to v, which we refer to as an 'unstructured" representation. This unstructured representation risks diluting the spatial structure captured by o, making learning more difficult. This exacerbated when scaling the width of v, demonstrated in prior works [Obando Ceron" et al.] 2024 (Sokar et al.) 2025). We hypothesize that the effectiveness of prior architectural modifications for scaling RL networks lies in their ability
to induce some form of structure at this bottleneck. The rest of this section details these specific modifications.
2.2.1 SoftMoE
To address scaling limitations, Obando Ceron et al. [2024] proposed notable architectural change: replacing the standard dense layer in t/ with a mixture-of-experts (MoEs). This required first restructuring the 3-dimensional output of the encoder (ø) into a set of tokens to be fed into the mixture. Their investigation into tokenization strategies concluded that forming H x W tokens, each with C dimensions (the channel depth). yielded the best performance. This restructuring effectively changes the connection, reducing the bottleneck's 's parameter count to C' x dim(v) (See /SeguFi I). This architectural change has proven highly effective for various agents and at different scales.
Building on this, Sokar et al. 2025 isolated the source of these performance gains. They demon- strated that the tokenization step itself-not the MoE architecture- was the most critical component. Their evidence showed that single expert model (SoftMoF-1), which retains the tokenization but omits the expert routing, achieves performance nearly identical to that of the full multi-expert model across different scales.
2.2.2 Sparse networks
Sparse methods offer an alternative approach to managing network complexity, Sokar et al. [022] demonstrated that using sparse neural networks in place of dense ones can increase learning speed and lead to improved performance. Following this, Graesser et al. [2022] studied various ways to induce this sparsity, such as by pruming network weights during training or by using networks that are sparse from scratch. By imposing sparsity, these techniques directly structure the bottleneck, reducing the density of its connections. With a given sparsity level s, the effective number of parameters the bottleneck is reduced to x H x W x C x dim(()). These sparse approaches have shown promise for the scaling of larger architectures; indeed, Obando Ceron al.] (2024) recently demonstrated that Indeed,(bbnndo
pruning is an effective approach that facilitates the scaling of networks.
Gradual pruning This strategy begins with a standard, fully-dense network. As training progresses, connections (parameters) with low magnitudes-whicl are considered less salient to the network's function-
are are progressively removed. This pruning process often follows a polynomial schedule [Zhu and Gupta 2017]. Once the pruning schedule concludes, the network achieves its target sparsity level, and this fixed sparse architecture is maintained for the remainder of training.
Sparse from scratch In contrast to gradual pruning, this approach defines a sparse network at initialization, and this specific sparsity level is maintained throughout training. The sparse
3

---

topology can be static, meaning the set of active connections is fixed for the entire training duration. Alternativly,, the topology can be dynamic. For example, the RigL method (Evei et al.) 2020] dynamically optimizes the connections by periodically pruning a portion of the weights and growing new ones elsewhere, effectively "rewiring" the network while maintaining the overall sparsity.
Analyses
Main hypothesis
A low-density and well-structured bottleneck enables scaling deep RL networks.
We conduct a series of analyses, both quantitative and qualitative, to provide evidence for our main hypothesis. We investigate impacts on performance, plasticity, and properties of the learned features. Given the effectiveness of SoftMoFs (Obando Ceron et al., [024], Pruning [Obando Ceron et al.) 2024), and Tokenization [Sokar et al. 2025 for scaling value-based based networks, our analyses in this section will focus on these.
3.1 Experimental setup
Architectures We employ the Impala architecture [Espeholt et al. 2018 for its superior perfor- mance, and while our analysis centers on we also confirm our findings' broad applicability using the standard CNN architecture [Mnih et al. 2015]. Our main experiments and analyses present results across various scaling factors for the width of t (specifically, x1, x2, x4, and 8). When a scaling factor is not explicitly mentioned, our analysis defaults to the x4 scale. This is because, as shown in prior work [Obando Ceron * et al.) 22024, Sokarr et al.) [025 Obando Ceron et al., 2024), the x4 scale provides a substantial performance improvement, while further scaling tends to yield only marginal gains. Although layer width is our primary focus, we also include some preliminary investigations into scaling network depth in the appendix.
Agents and environments Our primary experimental setup involves the Rainbow agent Hessell ct al. 2018] evaluated on the Arcade Learning Environment (ALE) suite [Bellemare al. 2013] For direct comparison with recent work, we use the same 20-game subset from Obando Ceron R al. [024] and Sokar et al. 2025). However, we present our main results on the full ALE suite of 60 Atari games. We ran each experiment for a total of 200 million environment steps, with results averaged over 5 independent seeds, except for the experiments with increased replay ratio of 2, where we report the results at 50M steps. To further test the generality of our approach on discrete tasks, also evaluate Rainbow on the Procgen benchmark [Cobbe al. 2019]. Additionally, assess performance on the data-efficient 100k benchmark [Lukasz Kaiser et al. 2020 for which DER, version of Rainbow specifically tuned for that data-constrained setting. Finally, to demonstrate our findings extend beyond discrete action spaces, we also evaluate the Soft Actor-Critic (SAC) agent (Haarnoja et al. 2018 on continuous control tasks from the DeepMind Control (DMC) suite Tassa et al.) 2018).
Code and compute resources For all our experiments, we use the Dopamine library with Jax implementations [Castro et al. 2018). For SoftMoE (Obando Ceron * et al. 2024], we use the official implementation integrated Dopamine. For the sparse methods [Obando Ceron et [2024] Graesser et al. 2022). we use the same JaxPruner library") [Lee et al. 2024 used by [Obando Ceron et al. 2024.. All libraries have Apache-2.0 license. All experiments were run on NVIDIA Tesla P100 GPUs. The duration of each experiment ranged from 4 to 13 days, depending on the specific scale and algorithm. We present the exact run time for each case in Section
Implementation details For all algorithms, we use the default hyperparameters in the Dopamine library. For sparse-training algorithms, we follow [Gaesser al. [022] and use 90% sparsity. Additional analysis covering other sparsity levels is included in the appendix. For gradual pruning, we start pruning 8M environment steps and stop at 160M (80% into training), following the schedules
I Dopamine: hanp://gthnbccom/gggg//aaapiiiin
JaxPruner: http://ghhiiccomooogggerraaaaaaa

---

Seaquest
Spacelnwaders
Dorman neurons x
70
6D
50
20 Tom
8o
20
30
20
10
Narbe oa
Figure 2: (Left) Distribution of dormant neurons across á and & in scaled baseline across different games the end of training. The fully connected layer exhibits the highest percentage of dormancy. (Right) The performance degradation associated with scaling the entire network architecture is comparable that observed when only the bottleneck is scaled. The performance is aggregated over 20 games.
2200
Figure 3: GAP helps improve attention to relevant areas of input. Visualizing influential regions for network decisions using Grad-CAM (Selvaraju al. 2017]. (Left) The scaled baseline fails to attend to the important regions, focusing on irrelevant background details. (Right) GAP attends to the important regions in the input.
recommended by Graesser et al. 2022 and Obando Ceron et al. 2024]. For RigL [vvi et al. 2020). we use a drop fraction of 20% and a connection update interval of 5000. For SoftMoE, we use the single-expert variant (SoftMoE-1). This choice was twofold: first, it has been shown yield performance comparable to the standard multi-expert SoftMoE [Sokar et al. 2025] and second, ensures direct architectural comparison with the other methods considered in this study. We report interquartile mean (IQM) with 95% stratified bootstrap confidence intervals as recommended in [Agarwal et al. 202T) Full experimental details provided in the appendix.
3.2
Why scaling deep RL networks hurts performance
As has been previously demonstrated, naïvely scaling networks deteriorates performance (Obando Ceron et al. 2024 Obando Ceron al. 2024 Nauman al. [024]. In this section we conduct series of experiments to diagnose the underlying causes for this difficulty.
We start by analyzing the training dynamics of network where the width of all layers in both & and s are uniformly scaled by a factor of 4. We examine neuron activity by measuring the fraction of dormant neurons, metric that serves [2023). a neuron considered "dormant" if its average activation falls below a certain threshold. As shown in the left plot of Figure 2, we find that & exhibits large fraction of dormant neurons, while has low dormancy rates. This suggests that scaling mostly affects the plasticity of the bottleneck.
key indicator of network plasticity. Following Sokar et al.)
In the right plot of Figure 2 we compare the performance of the baseline network against the same network with scaled Φ and vb As consistently observed in previous studies, the scaled network exhibits degradation in performance. To investigate the contribution of v in this performance decrease, we only scale the bottleneck (via the first layer of v) to match the parameter count of the fully scaled network. As can be observed, the two scaled models has comparable performance, suggesting that the bottleneck drives most of the performance degradation when scaling.
To interpret the quality of the learned features of the scaled network, we generate saliency maps for areas that have the greatest impact on the scaled network's output using Grad-CAM [Selvaraju et al. 2017 Figure 3 (left) reveals that naively scaled networks fail to focus important regions and focus on irrelevant background areas. This suggests that scaling the bottleneck impairs network's ability to process and learn effective combinations of the representation ó.

---

Asterix
Seactved
Demonketack
Spareinsaders
200000
20000
150000
220000
50000
50303
10000
100
720 255
350
200
230
200
Figure 4: Scaling v hinders learning effective combinations encoder's features, leading to signifi- cant performance drop. However, performance dramatically improves when the scaled tb is fed with higher-level, more abstract features obtain by increasing the depth of Φ.
To further investigate potential challenges in feature learning within vú, we study the network hehavior when providing more abstract, higher-level features to Specifically, 6 by adding four additional ResNet blocks, while using the scaled bottleneck. This modification makes receive more structured, high-level features from the encoder. We present the performance throughout training in Figure 4 The fact that we observe a dramatic increase in performance when compared to the network that only scaled v suggests that structured representations helps feature learning in scaled networks.
increased the depth of
3.3 Existing techniques are mainly targetting the bottleneck
As previously mentioned, there have been num- ber of recent proposals to enable scaling deep RL networks by using sophisticated architectural modifications. We hypothesize that core reason for their effectiveness is that they are implicitly targetting the bottleneck. The strong performance of SoftMoE-1 I and tokenized baselines presented by (Sokar et al. [2025] support this claim, as they are primarily re-structuring the encoder output.
5-
2
Baseline
To validate our hypothesis on sparse methods, we evaluated the impact of sparse training techniques when limited to the bottleneck, while keeping all other layers dense. We performed this anal- ysis on gradual pruning [Obando Ceron et al., 2024). dynamic sparsity (RigL) and static spar- sity Graesser et al. 2022 Sokar et al.] [2022]. As shown in the top plot of Figure 5] restricting sparsification to the scaled bottleneck results in improved performance across all sparse training techniques. This suggests that applying sparsity only to the bottleneck sufficient to enable scaling RL networks.
Both and sparsified
Only sparsified
Pruning
RigL. Algorithm
Static
Sparm
Sparao
Sesethe
Besstine
Da
2
Intwort acela
In addition structuring the encoder output, ex- isting techniques effectively reduce the density of the bottleneck by either structuring the output as tokens (as in SoftMoE) which results in lower dimensional input to v/, or explicitly masking the majority of input weights (as in sparse methods). The bottom plot Figure 3Joonfirms this reduc- tion in parameters, and illustrates positive cor- relation between scale and performance, which is notably absent in the baseline. This suggests that low-density and structured bottlenecks fa- cilitate scaling deep RL networks.
10
Number Parameters
Figure 5: (Top) Across different sparse algo- rithms, sparsification of only & yields better per- formance than sparsifying & and vh (Bottom) The relation between performance and the effec- tive number of parameters & for different ap- proaches. Architectural methods have lower ef- fective density than the baseline which correlates with the observed performance improvements.

---

Base 116 GAP
Basefine GAP
Tayer
Replay
Figure 6: The impact of GAP across wide range of setting. From left to right: performance across different network scales, sample-efie efficienf training with various high replay ratio values (default 0.25), performance of the CNN architecture used by [Mnih et al., 2015]. and performance on the full 60 games of Atari 2600. In all cases, GAP significantly improves performance.
4 Mind the GAP!
Having identified low-density and well-structured bottlenecks as the main factor enabling scaling networks, demonstrate a simple alternative the sophisticated techniques recently explored: Global Average Pooling (GAP) [Lin et al. 2013). We demonstrate its efficacy across wide range of settings and aspects, including: (1) effectiveness in network scaling under different scales, architec- tures, and in sample-efficient regime with high replay ratios (Figure 6); (2) computational efficiency (Figure 7); (3) improved training stability and feature learning (Fiuure 8); (4) unlocking width and depth scaling (Figure 9); and (5) generalized gains across various domains (Figure IT).
Simple The output feature maps of o, denoted as E R XxXCC are processed by average pooling. For each feature map (F"), GAP computes the average its spatial dimensions, resulting in the output (g E RC), which is then fed to the fully connected layers (see /igure for an illustration):
H
W
gº HXW
Fi
Effective We evaluate the impact of this architectural change various settings. Scale: We assess the performance of GAP across different network scales. The left plot of Figure demon- strates that this simple architectural change unlocks scaling in RL networks, leading to signifi- cant performance improvements across different scales. cample-efficient regime: increasing the number of gradient updates per environment interaction (replay ratio) is favorable for sample- efficiency. Yet, higher replay ratios often hurt performance [Nikishin et al., 2022]. We study the performance of the scaled network across varying the replay ratio values (default 0.25). We find that even in this challenging setting, GAP has very strong performance compared to the baseline of the same network size, yielding more sample-efficite agents, as shown in the center left plot of Figure 6) Varying architecture: we evaluate the effect of GAP on different archi- tecture for o: the original CNN architecture used in [Mnih et al. 2015). The right center plot in Figure shows that GAP can also provide perfor- mance gain to this architecture. Full suite: we as- sess the generalization of our findings beyond the 20 games used for most results and evaluate on the full set of 60 games of Atari. The rightmost plot of Figure connfims that GAP consistently improves performance the full suite.
GAP
6.0
GA
Sparse
Sparsc
Sparse
ICM Batter
5.0
Basolinc
4.5
Metwor orale
1x
2x
Baseiie
8x
100
200
250
Time (hours) Faster
Figure 7: The computational cost versus perfor- mance across varying network scales for different algorithms. GAP offers the highest speed while obtaining substantial performance improvements.
Efficient In Figure we report the total num- ber of GPU hours per game for every scale, to compare the computational cost of the different methods. While scaling the baseline increases computational costs and degrades the performance, applying GAP to the encoder's output reduces
7

---

Asterixc
DemonAttack
-0-
20
Daseline
20
GAP
Deep
Figure 9: Impact of scaling network depth and width. Increasing depth hurts perfor- mance for the baseline networks, and scal- ing both width and depth makes even worse. GAP, however, unlocks scaling even for increased network depth, leading to sig- nificant performance gains.
Namber
Numbar
Figure 8: Scaled networks with GAP exhibit less dormant neurons than the baseline and have
lower feature norm.
the effective density of da (and hence computational cost), while yielding performance gains. Despite having a similar effective density SoftMoE, GAP's inherent simplicity makes it more efficient by avoiding the extra computations associated with SoftMoE's token construction and post-MoE projection. The high runtime observed sparse methods, despite their low effective density, stems from the fact that sparsity is only simulated with parameter masking [Hoefler et al. 2021 Mocanu ct al.
2018 Evei et al. 2020
Stable training dynamies Figure 3 (right) displays the saliency maps when training with GAP, where the network seems to be attending to the most important and relevant areas of the input. We further examine the dormant neurons (Sokar et al. 2023 and the norm of the features in Figure 8 We observe that the network exhibits fewer dormant neurons than the baseline and lower feature norms, suggesting improved plasticity and training stability.
Unlocking width-depth scaling We demonstrate the effectiveness of GAP in scaling network width which is the focus of previous works [Obando Ceron et al., 2024 Obando Ceron et al. 2024 Sokar et al. 2025). Beyond this, we also explore how increasing 's depth, or both width and depth, impacts the performance of RL. networks. Specifically, we add extra fully connected layer y and evaluate the performance with both unscaled and scaled width of th. More analysis with varying depth are included in the appendix. We present the results in Figure 9 We find that the bascline's s performance drops with increased depth, worsening significantly when scaling both dimensions. In contrast, interestingly GAP maintains strong performance across all scaling dimensions, confirming the impact of the representation learned by the bottleneck in the overall performance of the network.
Median
IQM
Mean
GAP Layer Norm WD SoftMoE-1 1
Static RigL- Pruning
Baseline
3.2
4.0
4.8
5.6
3.2
4.0
4.8
4
10
Human Normalized Score
Figure 10: Comparison against architectural and algorithmic techniques for scaling RL networks. We report Median, IQM, and Mean scores [Agarwal et al. [021] at 100M environment steps. GAP presents notably simpler alternative to the baseline approaches, while achieving the best performance.

---

starpilot
chaser
big
bossfight
Beserre
25-
23
301
25-
milkoorst
CrazyClimber
Alien
Kangaroo
MsPacman
3530
3300
1280
453
3300
4X00
CAP
26
26
53
26
30
46
28
24
20
cartpole-swinqup
cheetah-run
We walk
walker-stand
001
3800
830
6.2-
432-
GAP
430
100
230
27x Number of Trames On millons)
Uid col
Number of Trames (mimllen))
Number of Tramm In millionss)
Number Pramess millonal
Figure 11: Performance for Rainbow Procgen [Cobbe al. 2019] (top). DER on Atari1005 Fukasz Kaiser et al.] 2020) (middle), and SAC on DMC [Tassa et al. 2018 (bottom). GAP leads to performance gains for scaled networks in diverse domains.
Comparison against other methods We compare the performance of GAP against various archi- tectural techniques including pruning [Obando Ceron et al. 2024]. static and dynamic sparsity (RigL) [Graesser et al. 2022], and SoftMoE-1 [Sokar et 2025]] Morcover, we include comparison against algorithmic methods including weight decay [Sokar et al. 2023 Obando Ceron et al. 2024]. and layer normalization [Nauman et al. 2024). Figure shows that GAP outperforms all baseline methods and achieves performance comparable to SoftMoE, while being more efficient and simpler. Although GAP is more aggressive compression technique than granular approaches like per-conv tokens in SoftMoEs [Obando Ceron et al. 2024), which could lead to loss of spatial structure, it still preserves spatial information within each feature map more effectively than a flattening operation.
Generalization other domains To further assess the broad applicability of our findings, we extended our evaluation to diverse set of domains and agents. We assessed the Rainbow agent on Procgen [Cobbe et al., 2019]. the DER agent [Van Hasselt et al., 2019] the data-efficient Atari 100K setting [Lukasz Kaiser et al. 2020]. and the SAC agent [Haarnoja et al. 2018 on the continuous DeepMind Control (DMC) suite Tassa et al.] 2018]. . For SAC, we follow the CNN architecture presented in (aarats et al. 2021b and scale the embedding layer of the actor and critic by 8. Figure 11 illlstrate that these experiments align with our main results, confirming that GAP provides consistent and significant performance benefit for scaled networks across these distinct benchmarks.
5 Related Work
Several works have shown that scaling RL network causes substantial performance degradation due to training instabilities exhibited by the network (Hessel et al. 20188] Bjorck et al., 2022 Obando Ceron* et al. 2024]. Although the precise causes of these issues remain unclear, several approaches aim to mitigate them. We categorize these methods to architectural methods that alter the standard network architecture and algorithmic methods.
Scaling through architectural changes Recent works have investigated architectural modifications to improve the scaling of RI networks. Obando Ceron* et al. [2024] integrated Mixture-of- Experts (MoE) after the encoder in single-task RL networks. Their results across multiple domains

---

highlight the effectiveness of this approach for scaling RL. with SoftMoE Puigcerver et al. 2024] outperforming traditional MoE Shazeer et al. (2017). This MoE approach was later extended by Willi et al. 2024), who demonstrated its applicability in the multi-task setting. Relatedly, [Sokar jtt al. 2025 provided insights into why such methods might succeed, showing that replacing the standard flattened representation from the encoder with tokenized representation that preserves spatial structure significantly improves the performance of scaled networks. Another line of research studies replacing dense parameters by sparse ones in both online [Graesser et al. [2022] Tan et al. 2023] SSoaa et al.) 2022] and offline RL [Amob et al., [22T]] This approach has demonstrated its effectiveness in increasing the learning speed and performance of RL agents. Network sparsity is achieved either by starting with a dense network and progressively pruning weights, or by initializing with sparse network and maintaining consistent sparsity level throughout training. In the latter case, the sparse structure can be kept static or optimized dynamically during training using methods like
SET [Mocanu et al. 2018] and Rigl. (Evci et al. 2020). Recently, Obando Ceron et al.. (022) showed that gradual magnitude pruning helps in scaling RL networks, leading to improved performance. Concurrent works have independently proposed similar approaches to GAP (Trumpp al. 2025
Kooi et al. 2025), which provides extra evidence for the efficacy of this method.
Scaling through algorithmic changes A primary goal in this line of research is maintaining training stability as networks grow. Bjorck et al. [021]] show that using spectral normalization [Miyato et al. 2018) helps improve training stability and enable using large neural networks for actor-critic methods. Farebrother et al. [2023a] explore the usage of auxiliary tasks to learn scaled representations. Farebrother et al. [2024] demonstrated that training value networks using classification with categorical cross-entropy, as opposed to regression, leads to better performance in scaled networks. Other regularization techniques have been shown to be critical. [Schwarzer* et al. 2023 propose several tricks to enable scaling including weight decay, network reset [Nikishin al. 2022 increased discount factor, among others. Similarily. Nauman et al. (2024) employ a combination of layer normalization [Ba et al. 2016), weight decay, and weight reset.
6 Conclusion
Recent proposals for enabling scaling in deep reinforcement learning have relied on sophisticated architectural interventions, such as the use of mixtures-of offurees-of-xxerrts and sparse training techniques. We demonstrated that these methods are indirectly targeting the bottleneck connecting the encoder ø and dense layers vb in standard deep RL network. A consequence of our analyses is that directly targeting this bottleneck, for instance with global average pooling, can achieve the same (or higher) performance gains.
Our work highlights the importance of better understanding the training dynamics of neural networks in the context of reinforcement learning. The fact that a simple technique like global average pooling can outperform existing literature suggests that architecture design is ripe for exploration. There is also a likely connection of our findings with works exploring representation learning for RL [Castro et al. 2021, Kemertas and Aumentado- Armstrong 2021 Zhang et al.) 2021 Farebrother et 20233] given that these generally target the output of ø; indeed, most of these methods aim to structure the outputs of Φ so as to improve the generalizability and efficiency of the networks. It would be valuable to investigate whether approaches like GAP are complementary with (and ideally help enhance) more sophisticated representation learning approaches.
Limitations Although our broad set of results suggest findings are quite general, our inves- tigation has focused on pixel-based environments where there is a clear bottleneck, or separation between & (the convolutional layers) and v (the fully connected layers). is not clear whether our findings extend to non-pixel based environments or architectures where there isn 't a clear bottleneck, but would be an interesting line of future work.
Acknowledgements
The authors would like to thank Gheorghe Comanici, Joao Madeira Araujo, Karolina Dziugaite, Doina Precup, and the rest of the Google Deep team, as well as Roger Creus Castanyer and Johan Obando-Ceron for valuable feedback on this work.
10

---

References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural information processing systems, 01229344-23222 2021.
Samin Yeasar Arnob, Riyasat Ohib, Sergey Plis, and Doina Precup. Single-shot pruning for offline reinforcement learning. arXiv preprint arXhv:2112.15579, 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 17X1::6077006500 2016.
Mare G Bellemare, Yavar Naddaf. Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Marc G. Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C. Machado, Subhodeep Moitra, Sameera Ponda, and Ziyu Wang. Autonomous navigation of stratospheric balloons using reinforcement learning. Nature, 588(7836):77-88 82, 2020.
Nils Bjorck, Carla Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning with spectral normalization. Advances in neural information processing systems, 2021.
Pablo Samuel Castro, Subhodeep Moitra, Carles Gelada, Saurabh Kumar, and Marc Belle- mare. Dopamine: A research framework for deep reinforcement learning. arXiv preprint arXiv:1812.061100 2018.
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. MICo: Improved representations via sampling- based state similarity for markov decision processes. In A. Beygelz- imer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL httpstttttt///oereiiiiee
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. arXiv preprint arXiv: 1912.01.88, 2019.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese,
Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de las Casas, Craig Donner, Leslie Fritz, Cristian Galperti, Andrea Huber, James Keeling, Maria Tsimpoukelli, Jackie Kay, Antoine Merle, Jean-Mare Moret, Seb Noury, Federico Pesamosca, David Pfau, Olivier Sauter, Cristian Sommariva, Stefano Coda, Basil Duval, Ambrogio Fasoli, Pushmeet Kohli, Koray Kavukcuoglu, Demis Hassabis, and Martin Riedmiller. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, (12(7897):4144499 2022.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymir Mnih Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-leamer architectures. In Proceedings of the International Conference Machine Learning (ICML), 2018.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. International conference on machine learning, pages 2943-29229 PMLR, 2020.
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Mare G Bellemare. Proto-value networks: Scaling representation learning with auxiliary tasks. In The Eleventh International Conference on Learning Representations, 2023a.
URL http:///pppaapooooooooooooooaaaa wet..///oorummiiooo0os0t.01111
Jesse Farebrother, Joshua Greaves, Rishabh Agarwal, Charline Le Lan, Ross Goroshin, Pablo Samuel Castro, and Marc G. Bellemare. Proto-value networks: Scaling representation learning with auxiliary tasks, 2023b. URL httphtss////: www.oorg////aaaaeeeeeeeeeee abs/2304. 12567

---

Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep rl. In International Conference Machine Learning, pages 13049-13071. PMIR, 2024.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193-202 1980.
Laura Graesser, Utku Evei, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning. In International Conference on Machine Learning, pages 7766-7792. PMLR, 2022.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. Pmlr, 2018.
Matteo Hessel, Joseph Modayil, H. V. Hasselt, T. Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, M. G. Azar, and D. Silver. Rainbow: Combining improvements deep reinforcement learning. In AAAI, 2018.
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. Journal of Machine Learning Research, 22((41)) 1-1124, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.088 000301,08...001 2020.
Mete Kemertas and Tristan Aumentado-Armstroon Towards robust bisimulation metric learning. CoRR, abs/2110. 2021. URL htttps////axxi orwooogg//ab...o http///bb2110.....// 22100 14096
Jacob E. Kooi, Zhao Yang, and Vincent François-Lavet. Hadamax encoding: Elevating performance in model-free atari, 2025.
Joo Hyung Lee. Wonpyo Park. Nicole Elyse Mitchell, Jonathan Pilault, Johan Samir Obando Ceron, Han-Byul Kim, Namhoon Lee, Elias Frantar, Yun Long, Amir Yazdanbakhsh, et al. Jaxpruner: A concise library for sparsity research. In Conference on Parsimany and Learning, pages 515-528. PMLR, 2024.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. CoRR, abs/1312.4440, 2013. URL btttp://arxiv org/ hatg//bb//// 1312. 4400
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, loannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540))529-533 February 2015.
Decebal Constantin Mocanu, Elena Mocanu, Peter Stone, Phuong H Nguyen, Madeleine Gibescu, and Antonio Liotta. Scalable training of artificial neural networks with adaptive sparse connectivity inspired by network science. Nature communications, (1):23383, 2018
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoš, and Marek Cygan. Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control. In The Thihty-eighth Annual Conference Neural Information Processing Systems, 2024. URI.https https
//openreview. net//forumiia-uuDxhhhaal
Exgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias deep reinforcement learning. In International conference on machine learning, pages 16828-16848 16847. PMLR, 2022.
12

---

Johan Samir Obando Ceron, Aaron Courville, and Pablo Samuel Castro. In value- based deep rein- forcement learning, a pruned network a good network. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 38495-38575 38519. PMLR, 21-27 Jul 2024.
Johan Samir Obando Ceron*, Ghada Sokar*, Timon Willi*, Clare Lyle. Jesse Farebrother, Jakob Nico- laus Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep RL. In Forty-first International Conference on Machine Learning, 2024. URL httpt:://opeeererrr//
net/ www.:nee/foooui wwr.foormmioooo Xot//o/mm
Joan Puigcerver, Carlos Riquelme Ruiz, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts. In The Twelfth International Conference on Learning Representations, 2024.
Max Schwarzer*, Johan Samir Obando Ceron*, Aaron Courville, Mare G Bellemare, Rishabh Agarwal, and Pablo Samuel Castro. Bigger, better, faster: Human-level Atari with human-level efficiency. In Andreas Krause. Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 30365- 30380. PMLR, 23-29 Jul 2023.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-basee localiza- tion. In Proceedings of the IEEE international conference on computer vision, pages 618- 88626, 2017.
Noam Shazeer, * *Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gated mixture-of layer. In International Conference on Learning Representations, 2017. URL hntps: hetps:://ppenrevier net/forumiid-Bie Bit.idddD1BBc1gggggg
Ghada Sokar, Elena Mocanu, Decebal Constantin Mocanu, Mykola Pechenizkiy, and Peter Stone. Dynamic sparse training for deep reinforcement learning. In International Joint Conference on Artificial Intelligence, 2022.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning, pages 32145-32168, PMLR, 2023.
Ghada Sokar, Johan Samir Obando Ceron, Aaron Courville, Hugo Larochelle, and Pablo Samuel Castro. Don't flatten, tokenize! unlocking the key to softmoe's efficacy in deep RL. In The Thirteenth International Conference on Learning Representations, 2025. URL https
Richard S. Sutton and Andrew G. Barto. Introduction to Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
Yiqin Tan, Pihe Hu, Ling Pan, Jiatai Huang, and Longbo Huang. RIx2: Training a sparse deep
reinforcement learning model from scratch. In The Eleventh International Conference Learning Representations, 2023.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, et al. Deepmind control suite. arXiv preprint 18010000000 2018
Raphael Trumpp, Ansgar Schäfftlein, Mirco Theile, and Marco Caccamo. Impoola: The power of average pooling for image-based deep reinforcement learning. In Reinforcement Learning Conference, 2025.
Hado P Van Hasselt, Matteo Hessel, and John Aslanides. When to use parametric models in reinforcement learning? Advances in Neural Information Processing Systems, 32, 2019.
13

---

Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi. Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan. Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L.. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap. Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575(7782): 350-354, 2019.
Timon Willi*, Johan Samir Obando Ceron Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, and Pablo Samuel Castro. Mixture of experts mixture of RL settings. In Reinforcement Learning Conference, 2024. URL httpp::/oppneeeeiiieeeeee http://wwweeeeeeeeeeoooooorooooo
Peter Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J. Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, Leilani Gilpin, Piyush Khandelwal, Varun Kompella, HaoChih Lin, Patrick MacAlpine, Declan Oller, Takuma Seno, Craig Sherstan, Michael D. Thomure, Houmehr Aghabozorgi, Leon Barrett, Rory Douglas, Dion Whitehead, Peter Dürr, Peter Stone, Michael Spranger, and Hiroaki Kitano. Outrac- ing champion gran turismo drivers with deep reinforcement learning. Nature, 602(78966:223-228, 2022.
Denis Yarats, Ilya Kostrikov, and Rob Fergus, Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference learning representations, 2021a.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Improving sample efficiency in model-free reinforcement learning from images. Proceedings of the aaai conference on artificial intelligence, volume 35, pages 10674-10681, 2021b.
Amy Zhang, Rowan Thomas McAllister, Roberto Calandra, Yarin Gal, and Sergey Levine. Learning invariant representations for reinforcement learning without reconstruction. International Conference on Learning Representations, 2021. URL http::: net/ mettfooummid F2FCDDKRREu
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy pruning for model compression. arXiv preprint arXiv:1710.0187., 2017.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blažej Osiński, Roy H Camphell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, Afroz Mohiuddin, Ryan Sepassi, George Tucker, and Henryk Michalewski. Model based reinforcement learning for atari. In International Conference on Learning Representations, 2020. URL https://
openreview
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The paper aims at understanding the challenges in network scaling in RL and underlying reasons behind the success of existing algorithms. The abstract and introduction include this.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discussed the limitations in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide all experimental details in Section 3.1 and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The paper only uses already available open-source code, and provides sufficient instructions to faithfully reproduce.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
public/guides/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ https//ccppbbccc/uuuddeeeoo Coaee//dddeSebmmmcia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide all experimental details in Section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We run each experiment using 5 seeds. Error bars represent 95% stratified bootstrap confidence intervals.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the compute resources in Section 3.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The work conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We include the license of used libraries.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
20

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The paper does not include LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
21

---

A
Broader impacts
This paper studies the challenges in scaling networks in pixel-based deep reinforcement learning. We present simple, effective alternative current sophisticated approaches by identifying the effective network representation. The approach has a positive impact by requiring no hyperparameters. simplifying implementation and showing robust performance across various scenarios. While this research aims to advance RL agent capabilities without direct negative impact, we urge careful consideration of potential implications when building upon this work.
B Experimental Details
Table 1: Hyper-parameters for Rainbow and DER agents.
Atari
Hyper-panne prrrmmerrreete
Rainbow
DER
Adam's 's (c)
1.5e-4
0.00015
Adam's 's learning rate
6.25e-5
0.0001
Training
Batch Size
32
32
Weight Decay
0
0
Activation Function
ReLU
ReLU
Architecture
Fully connected layer Width
512
512
Replay Capacity
1000000
1000000
Minimum Replay History 20000
1600
Number of Atoms
51
51
Reward Clipping
True
True
Algorithm
Update Horizon
3
10
Update Period
4
Discount Factor
0.99
0.99
Exploration
0.01
0.01
Sticky Actions
True
False
Hyperparameter details We use the default hyperparameters for all the studied algorithms. We present the values of these parameters in Table For the dormant neuron analysis, use dormancy threshold of 0.001. For the feature learning analysis (Figure 4). we increase the depth of the encoder by adding ResNet blocks.
Atari Games [Bellemare et al., 2013) We use the set of 20 games used in Sokar et al. (0025] Obando Ceron" et al. [2024] for direct comparison. The set has the following games: Asterix, Spacelnvaders, Breakout, Pong. Qbert, DemonAttack, Seaquest, WizardOfWor. RoadRunner, Beam- Rider, Frostbite, CrazyClimber, Assault, Krull, Boxing, Jamesbond, Kangaroo, UpNDown, Gopher, and Hero. This set used most of our analysis, nevertheless we provide our main results on the full suite of 60 games.
Atari100K Games Lukasz Kaiser et al. 2020 We test the 26 games of this benchmark. It includes the following games: Alien, Amidar, Assault, Asterix, BankHeist, BattleZone, Boxing, Breakout, ChopperCommand, CrazyClimber, DemonAttack, Freeway, Frostbite, Gopher, Hero, Jamesbond, Kangaroo, Krull, KungFuMaster,, MsPacman, Pong, PrivateFye, Qbert, RoadRunner, Seaquest, UpNDown.
C Extra Experiments
Sparse methods address the bottleneck Extending our analysis in section 3 we validate our hypothesis on various sparsity levels. Consistent with our main results, sparsification only v yields better performance than sparsifying & and 26 across all sparsity levels shown in Figure 12
22

---

6-
5-
Baseline
Baseline
Both & and y sparsified
Both & and w ysparsified
Only y sparsified
Only sparsified
Pruning
Static
Pruning
Static
Algorithm
Algorithm
Figure 12: Sparsification of d yields better performance than sparsifying o and ab for 80% sparsity (left) and 95% sparsity (right).
CrazyClimber
Alien
Gopher
MsPacman
1200
33ne-
J00
3000
280
630
200
26 Number tames millind
25
Number Pamms millioe)
Numberot Frmmee millens
Mumber frames millons
CrazyClimber
Alien
Gopher
MsPacman
34904
259
3200
240
109
Numbut Pamme in miliioon
Number Framis In miller))
Numbe of hamis Cn (iiiiisns
Numser of hames milorst
Figure 13: Performance for DrQ (top) and DrQ(e) (bottom) 202T on Atari100K.
Yarats al. 2021a Agarwal et al.
More agents We evaluate two more algorithms in the sample-efficient regime: DrQ and DrQ(e) Yarats et al. 2021a Agarwal et al. 2021 on Atari 100K. Consistent with our main results, GAP improves performance scaled networks shown in Figure 13
Comparison against max pooling We compare the performance of global average pooling with global max pooling for Rainbow on the set of 20 games used in our main results. We find the GAP outperforms max pooling as demonstrated in Figure 14] We hypothesize this is due to GAP's ability retain more comprehensive information through its averaging operation.
Encoder width scaling We perform additional experiment analyzing the effect of scaling the width of layers by a factor of 4, while maintaining 1 unscaled. FFiuure 15 shows that GAP significantly improves performance.
Deeper networks While our main focus in this work is scaling the width of the network, we also performed some preliminary analysis on scaling the depth of the fully connected layers (1²), exploring architectures with 1, 2, and additional layers. shown in Figure 16) increasing the depth degrades the performance of the agent. Increasingly, GAP improves performance over the corresponding baseline network the same size across varying depth.
Performance throughout training Figure 17 and Figure 18 presennt the performance per game throughout training for Atari and Atari 100K benchmarks, respectively.
23

---

Baseline
200 S0t Marrod Marredage Scoe
Glital pa!
10W
50 75 25
306 235
Number of extra layers
of jemilan))
Figure 15: Effect of increasing the width of Φ. GAP improves performance over the scaled baseline.
Figure 16: Effect increasing the depth of v. GAP achieves better performance than the corresponding baseline with the same network size.
Figure 14: Comparison be- tween GAP and global max pooling for Rainbow.
Assa ult
Asteria
BearmRider
Boxiing
Dupemnet
3:30-
Breaknut
CrazyElimber
DemonAAtack
Frovthite
105
100
200
20
Gooheer
Herb
Jamesbond
Kangaroo
3030
Krull
Pong
Obert
RoadRunner
2000-
Scandi
Sea Seaqquest
Spaceincacers
UpDNDown
20000
Scanee
Stop
Figure 17: Performance during training on the 20 games of the Atrai benchmark.
24

---

Alen
Amidar
Assout
Astefix
NAUT
Seatre
ChopprCoomman
CrazyClmber
DemorAttack
Frocway
1000
4.1%
Frostbike
Gopher
Hore
jamesbond
Kangaroo
Kung
PrivateEyc
Qbert
ReadRunner
112
300
Seaqueest
UpNDown
1.0
Rvtum
$100
Figure 18: Performance during training the 26 games of the Atrai 100k benchmark.
25


Reviews:
[{"review_id": "fc071TGmrE", "reviewer": "Reviewer_YprL", "summary": "This paper considers the problem of scaling in pixel-based deep RL, that is, under what circumstances is it possible to improve the performance of deep RL algorithms by increasing the size of the network. They find that using global average pooling after the convolutional layers allows scaling. They show good empirical performance with this simple idea on Atari benchmarks.", "questions": "* Fig. 2 (Right) which game is this?", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n* The paper achieves strong performance with increasing model size and a simple architectural modification: global average pooling after convolutional layers in the value network.\n\n## Weaknesses\n\n* The main hypothesis of the paper says low-dimensional and well structured bottleneck allows scaling. Does that mean that a 2x2xd bottleneck doesn't allow scaling? Also, what does well-structured really mean? Does it have a clear definition? From my understanding of the paper it's only shown that global average pooling allows scaling but then why turn this into an unclear and more general statement?\n* The paper is technically void. There's not a single new or interesting equation (let alone theorem) in the paper.\n* The point of the paper could have been conveyed in one page.\n\n**Minor:**\nAn expectation is missing in the definition of Q-value.\n\n**Summary:**\nOn the one hand, the finding is interesting and can have impact, on the other hand, the paper is bloated and has the weaknesses I've mentioned. Therefore, I am borderline on this paper.", "was_revised": true, "final_justification": "I find it rather strange that the very standard (in CV) global average pooling operation has not be tried in deep RL and that this is now a discovery. But maybe it really is because I haven't seen any papers on this. They provide no sound rationale for why GAP allows scaling. I find the general explanations in the rebuttal and paper to be imprecise and not compelling. The experiments also could have compared to a wider range of previous works (e.g., MOE) instead of only comparing to a single \"baseline\" selected by the authors. The plots and figures are somewhat empty and give me the impression of an unfinished research project.\n\nThere is not really any discussion of RL in the paper. What's observed in the paper is related to representation learning and I suspect one should be able to see something similar in image classification (but perhaps less pronounced). This would be an important experiment to add and if the paper is accepted I kindly ask the AC to suggest this experiment to the authors (my bad for not doing so in my review).\n\nI maintain my score but I'm OK with democratically accepting the paper since other reviewers have a more positive opinion of the paper.", "rebuttal": "We thank the reviewer for their feedback, and we appreciate their recognition of the method's strong performance, simplicity, and impact. Below we provide responses to all comments.\n\n> **W1:** The main hypothesis of the paper says low-dimensional and well structured bottleneck allows scaling. Does that mean that a 2x2xd bottleneck doesn't allow scaling? Also, what does well-structured really mean?\n\n**A:** What we mean is organizing the encoder's output in a way that facilitates feature learning in the dense layers; maintaining the features' spatial relationships and learning more generalizable representation. Methods such as sparsity, tokenization, or GAP create a more effective structure than simply flattening the output and densely connecting all neurons. Note that it is not just a matter of parameter count. As we demonstrate in Figure 5 (right), an equally-parameterized baseline still underperforms.\n\n> **W2:** The paper is technically void. There's not a single new or interesting equation (let alone theorem) in the paper.\n\n**A:** While our work doesn't introduce new equations given its empirical nature, we respectfully disagree that this implies a lack of technical depth. The paper provides rigorous analysis, a novel and effective architectural design, and extensive validation of its efficiency across diverse settings and metrics. Further, we demonstrate that one can reduce the complexity of prior approaches (such as MoEs and tokenization) without sacrificing accuracy. We strongly believe this type of evidence-backed insight is a valid contribution to the field, and particularly in light of its simplicity. \n\n> **W3:** The point of the paper could have been conveyed in one page.\n\n**A:** The paper's value extends beyond our final finding of the architectural modification that boosts the agent’s performance. Indeed, a core part of our work is the extensive set of analyses which provide the evidence for our final claim. Without these, the paper would reduce to an anecdotal observation, rather than useful insights backed by data and analyses. \n\n> **Q1:** Fig. 2 (Right) which game is this?\n\n**A:** It is aggregated results over the same set of 20 games used in all our experiments. We will clarify this in the caption."}, {"review_id": "sC5crIc3Ai", "reviewer": "Reviewer_ZiVM", "summary": "Scaling deep reinforcement learning (RL) in vision-based environments has long posed challenges that contrast with the consistent performance gains observed in supervised learning. This paper identifies a key bottleneck in typical pixel-based RL architectures—specifically, the interface between the convolutional encoder (denoted $\\phi$) and the subsequent fully connected layers ($\\psi$). The authors offer a unifying explanation for the success of recent architectural and algorithmic innovations, arguing that these methods implicitly address this $\\phi$ → $\\psi$ bottleneck. In response, the paper proposes a principled yet minimal intervention: inserting a Global Average Pooling (GAP) layer to reduce the spatial output of $\\phi$ into a compact C-dimensional representation before passing it to $\\psi$. This simple modification yields substantial empirical benefits across multiple benchmarks (Atari 200M, Atari 100k, Procgen), enabling both width and depth scaling, improving representational quality (via reduced neuron dormancy and feature norm magnitudes), and offering superior runtime efficiency relative to more complex approaches such as mixtures-of-experts and pruning-based scheme", "questions": "1. Have the authors evaluated the effectiveness of GAP in data-augmented actor-critic settings such as DrQ or DrQ-v2? These methods have demonstrated strong performance in pixel-based continuous control tasks and incorporate architectural bottlenecks by design. Notably, the DrQ architecture includes a hard-coded low-dimensional bottleneck, and it is well known within the research community that naively removing this component leads to substantial performance degradation. It would be valuable to position the proposed bottleneck intervention in the context of such existing architectural choices.\n\n2. Figure 4 suggests that scaling both $\\phi$ and $\\psi$ improves performance, but scaling only $\\psi$ hurts performance. Could the authors include results for a $\\phi$-only scaling variant to further substantiate the attribution of degradation of only scaling $\\psi$? This ablation would help isolate the relative contributions of encoder and head scaling and offer stronger support for the claim that the performance degradation resides primarily in $\\psi$\n\n3. The paper focuses heavily on scaling the fully connected head $\\psi$, while prior work in computer vision typically emphasizes scaling the encoder $\\phi$. Could the authors elaborate on the motivation behind this choice? While it is understood that, in typical vision-based RL architectures, $\\psi$ contains the majority of the parameters, it remains unclear why encoder scaling—which is the standard approach in computer vision tasks—is deprioritized. From a representational learning perspective, enhancing the encoder capacity would seem more intuitively aligned with performance improvements.\n\n4. Given that GAP aggregates over spatial dimensions, how do the authors view its applicability in environments that require fine-grained spatial reasoning or partial observability? For instance, continuous control domains such as Adroit or the DeepMind Control Suite may suffer from the loss of spatial information. Have the authors explored or considered this trade-off?\n\n5. Minor typo – “Procegn → Procgen” in Fig 11", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n- The paper proposes a simple yet highly effective architectural modification—Global Average Pooling (GAP)—to address a longstanding challenge in scaling vision-based deep RL.\n- The authors present a unifying perspective suggesting that the performance improvements observed in recent methods such as SoftMoE and pruning primarily stem from their implicit targeting of the architectural bottleneck—namely, the connection between the encoder $\\phi$ and the head $\\psi$. This insight provides a valuable conceptual lens for understanding scaling dynamics in vision-based deep RL. Moreover, the paper offers empirical evidence that effective depth scaling is possible—an underexplored axis of architectural scaling, as prior work has predominantly focused on width alone.\n- The study features extensive empirical validation, including large-scale experiments on the full Arcade Learning Environment (ALE) with 200 million frames, the 100k ALE setting, and Procgen, thereby demonstrating the robustness and generality of the proposed method.\n- The proposed approach is not only performant but also computationally efficient and implementation-friendly, requiring minimal architectural changes and providing faster training time relative to alternatives such as SoftMoEs and pruning based methods.\n\n\nWeaknesses\n\n- While the paper presents compelling empirical evidence for the role of the $\\phi$ → $\\psi$ bottleneck in scaling performance, its explanation remains largely empirical. The authors do not provide a theoretical account or a detailed representational analysis to clarify why the absence of a well-structured bottleneck—and the use of flattened encoder outputs—leads to degraded plasticity or impaired learning dynamics. Given that this phenomenon lies at the core of the paper’s contribution, a more principled explanation would substantially strengthen the work’s impact and conceptual clarity.\n- Despite broad benchmarking, the experimental domains remain confined to pixel-based benchmarks with discrete action spaces (e.g., ALE and Procgen), with no evaluation on continuous control domains (e.g., DeepMind Control Suite, Adroit), limiting the scope of generalizability. In addition, I think that \n- The absence of parameter count reporting presents a potential concern. Since the introduction of the GAP layer substantially reduces the number of parameters by collapsing spatial dimensions, it remains unclear whether the observed performance gains result from genuine improvements in scaling capacity or merely from more efficient parameter utilization. In typical scaling discussions, increased model size is often implied; however, with GAP, the networks may actually become smaller. Clarifying this distinction—and including performance comparisons at matched parameter counts—would offer a more rigorous understanding of whether GAP enables true architectural scaling or simply more effective use of fewer parameters.", "was_revised": true, "final_justification": "Many of my concerns have been addressed through the additional experimental results provided by the authors, which I appreciate. However, I remain somewhat unconvinced about the underlying motivation and the reasons behind the effectiveness of the proposed approach. Therefore, I am adjusting my score to Borderline Accept, rather than Accept.", "rebuttal": "We thank the reviewer for their feedback! We are glad that the reviewer found our empirical validation to be extensive and the approach highly effective, computationally efficient, implementation-friendly, and requiring minimal architectural changes. Below, we provide detailed responses to each comment, incorporating the results from all requested experiments.\n\n> **W1:** why the absence of a well-structured bottleneck—and the use of flattened encoder outputs—leads to degraded plasticity or impaired learning dynamics…. a more principled explanation would substantially strengthen the work’s impact and conceptual clarity.\n\n**A:** The flattening operation transforms the encoder's output into a large 1D vector, potentially losing some of the spatial structure captured by the encoder. This flattened vector, combined with the scaled width of the next layer, creates a high-density connection, which makes it difficult to make effective use of the  features' spatial relationships, and may result in overfitting which often causes a loss of plasticity [1]. These effects can result in the model attending to unimportant features and regions in the input, as demonstrated in Figure 3. To understand this further, we demonstrated in Figure 4 that preserving the spatial structure by learning higher level features before forwarding them to the dense layers mitigates this issue.\n\n> **W2, Q4:** “Despite broad benchmarking, the experimental domains remain confined to pixel-based benchmarks with discrete action spaces (e.g., ALE and Procgen), with no evaluation on continuous control domains (e.g., DeepMind Control Suite, Adroit), limiting the scope of generalizability.” “how do the authors view its applicability in environments that require fine-grained spatial reasoning or partial observability?”\n\n**A:** Following the reviewer’s suggestion, we expand our evaluation to the continuous control domain. We evaluate SAC on DeepMind Control tasks. We ran each experiment with 5 seeds. We follow the CNN architecture presented in [1], which consists of four convolutional layers, followed by critic and actor embeddings of a size of 50 neurons.  We scale the critic and actor embedding by 8x and 16x. GAP consistently yields significant performance gains over its corresponding baseline.\n\n| Method       | cartpole-swingup    | cheetah-run         | walker-walk         | walker-stand        |\n|:-------------|:-------------------:|:-------------------:|:-------------------:|:-------------------:|\n| Baseline x8  | 512.41 ± 127.26     | 116.71 ± 94.86      | 55.85 ± 41.89       | 190.43 ± 37.63      |\n| GAP x8       | **839.03 ± 15.42**      | **389.68 ± 61.69**      | **690.53 ± 83.53**      | **938.62 ± 29.06**      |\n\n| Method       | cartpole-swingup    | cheetah-run         | walker-walk         | walker-stand        |\n|:-------------|:-------------------:|:-------------------:|:-------------------:|:-------------------:|\n| Baseline x16 | 558.57 ± 238.98     | 171.13 ± 93.32      | 119.39 ± 77.88      | 174.69 ± 27.76      |\n| GAP x16      | **830.64 ± 15.70**      | **349.97 ± 74.77**      | **545.89 ± 42.81**      | **952.04 ± 15.28**      |\n\nWe thank the reviewer for pressing us on this point, as the addition of these extra results help reinforce the generality of our findings.\n\n\n> **W3:** The absence of parameter count ….it remains unclear whether the observed performance gains result from genuine improvements in scaling capacity or merely from more efficient parameter utilization. Clarifying this distinction—and including performance comparisons at matched parameter counts—would offer a more rigorous understanding of whether GAP enables true architectural scaling or simply more effective use of fewer parameters.\n\n**A:** We reported the parameter count in Figure 5, right. Note that the parameter count of GAP matches the parameter count of SoftMoE-1 (Please see Figure 1 for illustration and Section 2 for more details). We also tested a baseline with this same parameter count. As demonstrated in Figure 5, parameter-matched baseline performs worse than other methods, confirming the performance improvements are attributable to our architectural design, not just parameter count.\n\n> **Q1:** Have the authors evaluated the effectiveness of GAP in data-augmented actor-critic settings such as DrQ or DrQ-v2?\n\n**A:** Following the reviewer's suggestion, we ran additional experiments using the DrQ (top) and DrQ-v2 (bottom) algorithms on the Atari100K benchmark with a 4x scaled network using 5 seeds. Consistent with previous findings, the results show that GAP provides performance gains in both cases.\n\n| DrQ   | CrazyClimber            | Alien              | Gopher             | MsPacman           |\n|:---------|:-----------------------:|:------------------:|:------------------:|:------------------:|\n| Baseline | 9,429.90 ± 1,138.75   | 488.95 ± 24.72     | 296.92 ± 30.44     | 663.18 ± 39.88     |\n| GAP      | **11,950.33 ± 533.28**    | **568.34 ± 8.00**      | **350.05 ± 17.22**     | **764.94 ± 18.83**     |\n\n| DrQ-v2   | CrazyClimber            | Alien              | Gopher             | MsPacman           |\n|:---------|:-----------------------:|:------------------:|:------------------:|:------------------:|\n| Baseline | 7,971.73 ± 2,800.05   | 498.15 ± 22.12     | 198.44 ± 24.24     | 725.83 ± 35.85     |\n| GAP      | **12,947.54 ± 812.50**    | **570.60 ± 18.08**     | **320.22 ± 10.47**     | 753.51 ± 41.21     |\n\nWe once again thank the reviewer for suggesting this extra set of experiments.\n\n\n> **Q2:** Figure 4 suggests that scaling both $\\phi$ and $\\psi$ improves performance, but scaling only $\\psi$ hurts performance. Could the authors include results for a $\\phi$-only scaling variant to further substantiate the attribution of degradation of only scaling $\\psi$? This ablation would help isolate the relative contributions of encoder and head scaling and offer stronger support for the claim that the performance degradation resides primarily in $\\psi$\n\n**A:** As suggested by the reviewer, we conducted an experiment with a deeper $\\phi$ while maintaining the original, unscaled size for the $\\psi$. We report the average return at 100M steps. As shown below, providing high-level features to $\\psi$ enhances performance in both its unscaled and scaled versions, with the most significant gains observed in the scaled version.     \n\n| Model                  | Asterix                    | Seaquest                   | DemonAttack                | SpaceInvaders              |\n|:----------------------------|:--------------------------:|:--------------------------:|:--------------------------:|:--------------------------:|\n| Baseline                    | 40,107.83 $\\pm$ 9,727.30     | 117,374.13 $\\pm$ 92,323.04   | 67,238.46 $\\pm$ 11,705.80    | 13,490.66 $\\pm$ 6,035.76     |\n| Deeper $\\phi$                 | 75,044.80 $\\pm$ 16,696.87    | 249,930.37 $\\pm$ 48,719.17   | 60,223.30 $\\pm$ 20,860.26    | 12,531.67 $\\pm$ 2,569.25   |\n| Scaled $\\psi$                 | 14,562.03 $\\pm$ 2,652.72     | 33,940.98 $\\pm$ 25,256.58    | 8,053.09 $\\pm$ 673.43        | 9,946.47 $\\pm$ 2,244.52      |\n| Deeper $\\phi$ + Scaled $\\psi$ | 127,310.99 $\\pm$ 12,471.02   | 230,295.82 $\\pm$ 92,301.67   | 70,754.62 $\\pm$ 21,999.48    | 15,556.92 $\\pm$ 2,924.93     |\n\n\n> **Q3:** The paper focuses heavily on scaling the fully connected head , while prior work in computer vision typically emphasizes scaling the encoder . Could the authors elaborate on the motivation behind this choice? While it is understood that, in typical vision-based RL architectures,  contains the majority of the parameters, it remains unclear why encoder scaling—which is the standard approach in computer vision tasks—is deprioritized. From a representational learning perspective, enhancing the encoder capacity would seem more intuitively aligned with performance improvements. \n\n**A:** Our findings in Figure 2, right, demonstrate that scaling the dense layers alone leads to a performance decline comparable to that of scaling both the encoder and dense layers simultaneously. This observation, along with the success of recent methods like SoftMoE—which achieved optimal performance by applying mixtures to the dense layers—motivates our focus in this paper on scaling $\\psi$. Nevertheless, we conducted an additional experiment scaling only the width of the encoder layers by 4x. Due to the high computational costs and the limited rebuttal period, we report the IQM with 95% confidence interval over 20 games at 40M steps. Consistent with our previous findings, scaling the encoder decreases the performance of the model, whereas incorporating GAP into the network leads to a significant improvement.\n\n| Model             | IQM (CIs) |\n|:-----------------------|:-------|\n| Baseline               | 3.0327 (2.847, 3.2184) |\n| Scaled $\\phi$          | 2.6721 (2.55, 2.7942) |\n| Scaled $\\phi$ w/ GAP | 5.0173 (4.817, 5.2176) |\n \n\n[1] Nikishin, Evgenii, et al. \"The primacy bias in deep reinforcement learning.\" International conference on machine learning. PMLR, 2022.\n\n[2] Yarats, Denis, et al. \"Improving sample efficiency in model-free reinforcement learning from images.\" Proceedings of the aaai conference on artificial intelligence. Vol. 35. No. 12. 2021."}, {"review_id": "YXlfFMMmvS", "reviewer": "Reviewer_BBqr", "summary": "Scaling up the performance of visual deep reinforcement learning has been a challenging task for long time, where usually scaling up the network lead to worse performance. While many approaches have been proposed in the past, including sparsity and MoE, they are usually complex. This paper identifies the problem that prevents scaling to be the size of the bottleneck layer (the output of encoder). When the size of bottleneck is large (which is usually the case in visual tasks), scaling becomes challenging. The authors introduces global average pooling (GAP) as a simple yet effective approach architectural change to the encoder that reduce the bottleneck size and allow for performance gains.", "questions": "In Figure 6, why do we see a performance drop when we scale the width by a multiple of 2 and increase with a multiple of 4?", "limitations": "The authors adequately addressed the limitations of their approach in the limitations section.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**: \n- The paper is easy-to-follow.\n- The paper provide a comprehensive set of experiments studying various aspects of the problem and solutions. I enjoyed reading the analysis and discussion.\n- The approach is simple yet effective and can be easily incorporated into various approaches which enhance its adoption.\n- The method is hyper-parameter free which means it can be included easily into visual RL methods.\n\n**Weaknesses:**\n- The width-depth scaling claims is unsupported. The authors claim that GAP unlocks width and depth scaling, but this is never supported with an experiment. I except to see two figures both having performance on the y-axis vs width and depth on the x-axis. The current experiment only shows that the performance using GAP is better than the baseline in wide, deep, and wide&deep variations. No claims about scaling can be made using this experiment. If the authors want to keep the claim, then they need to support it with the proper evidence.\n- The authors use the word scaling in a different way from its common use which makes things a bit complicated. The argument of the scaling is almost always missing each sentence. For example, one can argue a method X allows (performance) scaling with the width of the encoder layers then show a figure with the width on the x-axis and performance on the x-axis. This supports the claim clearly. On the other hand, the paper use the word scaling loosely which makes it lose its meaning altogether. The only place where such experiment is presented in Figure 6 (first two) where the performance barely scales with width and doesn't scale. with replay ratio. \n- The paper lacks providing any reason for this bottleneck phenomenon. It remains unclear why the bottleneck can severely affect performance. I think the paper should at least give an explanation or even a hypothesis for why this is the case.\n\nI think the paper provides a good contribution, but I think the authors should work on their claims to avoid overstating the results. I think the paper addresses an important problem and there is an audience for it. I highly encourage the authors to improve their writing and moderate their claims to reflect what is given in the experiments.\n\n**Minor issues:**\n- \"We conduct a series of analyses, both quantitative and qualitative, to provide evidence for our main hypothesis.\" -> typically the purpose of the experiment is to validate the hypothesis rather than giving evidence for its correctness. I suggest saying \"validate our main hypothesis\"", "was_revised": true, "final_justification": "The authors committed to making the necessary changes in the revised version that will address my concerns.", "rebuttal": "We thank the reviewer for their feedback! We are glad that the reviewer found our experiments comprehensive and that the method is both effective and readily adaptable. \n\n> **W1:** The width-depth scaling claims is unsupported. The authors claim that GAP unlocks width and depth scaling, but this is never supported with an experiment. I except to see two figures both having performance on the y-axis vs width and depth on the x-axis. The current experiment only shows that the performance using GAP is better than the baseline in wide, deep, and wide&deep variations. No claims about scaling can be made using this experiment. If the authors want to keep the claim, then they need to support it with the proper evidence.\n\n**A:** We have conducted extra experiments with varying depth on Rainbow using the ResNet architecture. Due to the high computational costs and the limited rebuttal period, we report the IQM with 95% confidence score over 20 games at 40M steps. As demonstrated below GAP consistently significantly improves performance of deeper networks. Results across varied width are presented in Figure 6.\n \n| Method   | Deeper-1 | Deeper-2 | Deeper-3 |\n| :------- | :------: | :------: | :------: |\n| Baseline |   2.33 (2.164, 2.496)  |   1.95  (1.89, 2.01)  |   1.63 (1.577,1.683)   |\n| GAP      |   3.47(3.249,3.691)   |   4.10 (3.885, 4.315)  |   3.30 (3.074,3.52 )   |\n\n  \n> **W2:** The authors use the word scaling in a different way from its common use which makes things a bit complicated. The argument of the scaling is almost always missing each sentence. For example, one can argue a method X allows (performance) scaling with the width of the encoder layers then show a figure with the width on the x-axis and performance on the x-axis. This supports the claim clearly. On the other hand, the paper use the word scaling loosely which makes it lose its meaning altogether. The only place where such experiment is presented in Figure 6 (first two) where the performance barely scales with width and doesn't scale. with replay ratio.\n\n**A:** In most of our experiments we focus on scaling the width of the dense layers, following previous work [1,2]. In the leftmost plot of Figure 6, GAP consistently outperforms the corresponding scaled baseline. Moreover, the scaled network with GAP outperforms the unscaled baseline. In the second from left plot, a constant scale of 4 is used across varied replay ratios in the x-axis; we show that GAP helps enable obtaining sample efficient agents, as shown by the improved performance over the baseline with the same replay ratio value. We will add a precise definition of scaling and further clarify these results in the revised version of the paper. \n\n> **W3:** The paper lacks providing any reason for this bottleneck phenomenon. It remains unclear why the bottleneck can severely affect performance. I think the paper should at least give an explanation or even a hypothesis for why this is the case.\n\n**A:** We believe that the flattening operation contributes to forming this bottleneck. It transforms the encoder's output into a large 1D vector which, combined with the scaled width of the next layer, creates a high-density connection. This high density makes it difficult to make effective use of the  features' spatial relationships, and may result in overfitting which often causes a loss of plasticity [3]. These effects can result in the model attending to unimportant features and regions in the input, as demonstrated in Figure 3. To understand this further, we show in Figure 4 that preserving the spatial structure by learning higher level features before forwarding them to the dense layers mitigates this issue. We will extend our discussion on this point in the revised version.\n \n> **Q1:** n Figure 6, why do we see a performance drop when we scale the width by a multiple of 2 and increase with a multiple of 4?\n\n**A:** GAP x2 outperforms or matches GAP x1 in 19 out of 20 games. The slight reduction in the aggregated score is due to a single outlier game, breakout, where we observed high variance across our five runs. We will include detailed per-game results in the revised manuscript to make this clear.\n\n> I think the paper provides a good contribution, but I think the authors should work on their claims to avoid overstating the results. I think the paper addresses an important problem and there is an audience for it. I highly encourage the authors to improve their writing and moderate their claims to reflect what is given in the experiments.\n\n**A:** We thank the reviewer for their constructive suggestion and their acknowledgment of the problem's importance and the potential interest our work holds for the community. We agree that our use of the term scaling could be clearer. In the revised manuscript, we will explicitly define it, and incorporate the new experiments presented in this response that further confirm our findings.  \n\n> Minor rephrasing\n\nThank you for providing us with suggestions for rewording, we will incorporate that in the revised manuscript.\n\n[1] Ceron, Johan Samir Obando, et al. \"Mixtures of Experts Unlock Parameter Scaling for Deep RL.\" International Conference on Machine Learning. PMLR, 2024.\n\n[2] Sokar, Ghada, et al. \"Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL.\" The Thirteenth International Conference on Learning Representations.\n\n[3] Nikishin, Evgenii, et al. \"The primacy bias in deep reinforcement learning.\" International conference on machine learning. PMLR, 2022."}, {"review_id": "A3v9RL7Jli", "reviewer": "Reviewer_iWUt", "summary": "This paper investigates the reasons behind the performance drop observed when scaling deep reinforcement learning (RL) in pixel-based environments. The authors pinpoint the connection between the convolutional encoder's output and the subsequent dense layers, which they term the \"bottleneck\", as the primary limiting factor. They argue that previous successful approaches implicitly address this bottleneck. Based on their analysis, the paper proposes Global Average Pooling (GAP) as a straightforward and effective method to target this bottleneck, offering a simpler alternative to more complex solutions found in the literature.", "questions": "It is currently unclear how 'dormant' neurons are determined. There are numerous possible approaches (e.g., activity thresholds, gradient magnitudes, L0 norm of weights). Please explicitly define the methodology used to identify 'dormant' neurons. If multiple approaches were considered, briefly comment why the chosen method was selected over others.\n\nYou mention following the work of others and aiming for 90% sparsity. The effect of increased sparsity on model changes and performance is a crucial aspect that needs further elaboration. Can you provide more details on how sparsity was implemented (e.g., pruning method, frequency). Elaborate on the observed effects of varying sparsity levels (even if only 90% was the target) on both the model's architecture (e.g., changes in connectivity, effective parameters) and its performance metrics.\n\nThe scaling by 4 factor seems arbitrary without further context, though there is likely a good reason for it. Is it related to a specific architectural choice, computational constraint, or a baseline from previous work?\n\nFigure 10 & Performance Claims. You conclude that \"the performance of the 2 scaled models is nearly identical.\" However, observation of the IQRs (presumed to be the shaded regions) shows non-overlapping distributions across many frame numbers. If the scaling of this figure were different, these deviations might appear more significant. While the overall conclusion regarding the bottleneck might be correct, it's important to acknowledge and comment on these visible deviations in the IQRs. If the differences are not considered significant, please explain why.\n\nOverlapping distributions for GAP and SoftMoE are shown [boxplots]. It's questionable whether a notched boxplot or a Wilcoxon rank-sum test would show statistically significant differences in their medians. Given this visual overlap, can you provide a more robust argument for why GAP is considered \"higher performing than all other methods.\" If statistical tests were performed, please give the results. If not, consider performing them or refining the claim.\n\nMinor Edits\nP3 Deep RL network > networks: Change \"network\" to \"networks\" for plural consistency.\nP3 tokenzing > tokenizing: Correct the spelling from \"tokenzing\" to \"tokenizing.\"\nFig 11 ProcGen [spelling]: Correct the spelling of \"ProcGen\" in the figure caption", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\nThe paper offers a simple, effective, and practical solution to a common bottleneck in supervised learning, particularly relevant to RL pixel literature. The authors have conducted a thorough empirical investigation with commendable breadth and presentation of experiments. A key strength is the demonstration that \"bigger doesn't mean better\" in many RL problems, providing an easily implementable fix. The inclusion of wall-clock times for training versus performance is a valuable addition, highlighting the practical benefits of their approach. The application of sparsity to the bottleneck is a compelling finding, with the results presented being impressive and supporting the method's benefits. The core idea of investigating 'dormant' neurons and their impact on model performance is also a useful insight.\n\nWeaknesses\nDespite its strengths, the paper has several areas that need improvement.\nFigure 9 is unclear and needs enhancement to better convey its message.\nWhile empirical evidence is strong, the paper would benefit from theoretical backing for its claims.\nTraining curves for all baseline methods, not just the full (non-GAP) network, are needed for a comprehensive comparison.\nThe methodology for determining 'dormant' neurons is unclear and needs explicit definition, along with a justification for the chosen approach.\nMore details are required on how sparsity was implemented (e.g., pruning method, frequency) and the observed effects of varying sparsity levels on model architecture and performance.\nThe claim that \"the performance of the 2 scaled models is nearly identical\" is questionable given the non-overlapping IQRs in Figure 10. A more nuanced discussion or explanation for why these deviations are not significant is needed.\nThe claim that GAP is \"higher performing than all other methods\" lacks robust statistical justification, especially given the visual overlap in distributions. Statistical test results (e.g., Wilcoxon rank-sum test) are needed to support this claim, or it should be refined.", "was_revised": true, "final_justification": "the author responses to the questions raised were thorough - I am keeping my score of 5 'accept'", "rebuttal": "We thank the reviewer for their feedback! We are glad that the reviewer found the method simple and effective and that the empirical investigation is thorough and strong.\n\n> **Q1:** It is currently unclear how 'dormant' neurons are determined.\n\n**A:** We adopt the definition from the original paper that identifies dormant neurons in deep reinforcement learning [1]. The dormancy is measured based on a neuron's activation threshold. We will add the precise formulation to the revised manuscript.\n\n> **Q2:** You mention following the work of others and aiming for 90% sparsity. The effect of increased sparsity on model changes and performance is a crucial aspect that needs further elaboration. Can you provide more details on how sparsity was implemented (e.g., pruning method, frequency). Elaborate on the observed effects of varying sparsity levels (even if only 90% was the target) on both the model's architecture (e.g., changes in connectivity, effective parameters) and its performance metrics.\n\n**A:** In the case of static sparsity, a random network topology with the specified sparsity level is initialized at the beginning of training and kept fixed throughout. In the case of pruning, we followed the same strategy of prior work that studied pruning in RL [2,3]. The network starts with a dense network and low magnitude parameters are gradually pruned throughout training according to a polynomial schedule. We will provide more experimental details of these analyses in the revised version. \n\nFollowing the reviewer’s suggestion, we have conducted extra experiments varying the sparsity levels. We report the IQM with 95% confidence interval at 40M steps aggregated over the 20 studied games. Consistent with our previous findings, sparsifying only $\\psi$ leads to higher or similar performance to sparsifying the whole network.\n\n| Static Sparsity          | Sparsity 80 (%) | Sparsity 95 (%) |\n| :-------------------------- | :-------------: | :-------------: |\n| Both $\\psi$ and $\\phi$ sparsified |     3.246 (3.08, 3.412)     |  2.444 (2.309,2.578)      |\n| Only $\\psi$ sparsified      |      3.245 (3.805, 3.405)      |      3.745   (3.452,4.0372)    |\n \n| Pruning          | Sparsity 80 (%) | Sparsity 95 (%) |\n| :-------------------------- | :-------------: | :-------------: |\n| Both $\\psi$ and $\\phi$ sparsified |   2.620  (2.499, 2.742)    | 2.558 (2.383, 2.732) |  \n| Only $\\psi$ sparsified      |    2.399  (2.215, 2.583 )     | 2.573  (2.451, 2.695) |  \n\n> **Q3:** The scaling by 4 factor seems arbitrary without further context, though there is likely a good reason for it. Is it related to a specific architectural choice, computational constraint, or a baseline from previous work?\n\n**A:** Our choice of a scaling factor of 4 for most of the analysis is based on the observation found in prior works that 4x provides substantial performance improvements, whereas further scaling offers only marginal benefits [2,4]. Nevertheless we provide our main results for four different scales (Figure 6). We will extend our discussion in Section 3.2 on the justification for this choice.\n\n> **Q4:** Figure 10 & Performance Claims. You conclude that \"the performance of the 2 scaled models is nearly identical.\" However, observation of the IQRs (presumed to be the shaded regions) shows non-overlapping distributions across many frame numbers. If the scaling of this figure were different, these deviations might appear more significant. While the overall conclusion regarding the bottleneck might be correct, it's important to acknowledge and comment on these visible deviations in the IQRs. If the differences are not considered significant, please explain why. \n\n**A:** We agree with the reviewer that the use of “nearly identical” is not precise. We will revise the text to state that the methods achieve 'comparable performance' for greater clarity.\n\n> **Q5:** Overlapping distributions for GAP and SoftMoE are shown [boxplots]. It's questionable whether a notched boxplot or a Wilcoxon rank-sum test would show statistically significant differences in their medians. Given this visual overlap, can you provide a more robust argument for why GAP is considered \"higher performing than all other methods.\" If statistical tests were performed, please give the results. If not, consider performing them or refining the claim.\n\n**A:** We will clarify that with respect to SoftMoE, GAP achieves similar performance but is both more efficient and much simpler.\n\n\n[1] Sokar, Ghada, et al. \"The dormant neuron phenomenon in deep reinforcement learning.\" International Conference on Machine Learning. PMLR, 2023.\n\n[2] Ceron, Johan Samir Obando, Aaron Courville, and Pablo Samuel Castro. \"In value-based deep reinforcement learning, a pruned network is a good network.\" International Conference on Machine Learning. PMLR, 2024.\n\n[3] Graesser, Laura, et al. \"The state of sparse training in deep reinforcement learning.\" International Conference on Machine Learning. PMLR, 2022.\n\n[4] Ceron, Johan Samir Obando, et al. \"Mixtures of Experts Unlock Parameter Scaling for Deep RL.\" International Conference on Machine Learning. PMLR, 2024."}]

Author rebuttal:

Meta-review: This paper investigates the challenges that lead to performance degradation in scaled RL networks and analyzes the reasons behind the success of existing architectural approaches to scaling. Building on these insights, it presents pooling as a faster, simpler method that yields superior performance.

All reviewers acknowledged the importance of the problem and appreciated the simplicity and effectiveness of the proposed pooling (GAP) strategy in mitigating scalability bottlenecks. While concerns were raised regarding the lack of a strong theoretical explanation, the AC notes that in reinforcement learning it is often difficult or even artificial to provide such justifications.

The AC concurs with the reviewers and recommends acceptance of this paper.
Decision: Accept (poster)