Title: Mind the GAP! The Challenges of Scale in Pixel-based Deep Reinforcement Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Deep RL

Scaling RL networks
Abstract: Scaling deep reinforcement learning in pixel-based environments presents a significant challenge, often resulting in diminished performance. While recent works have proposed algorithmic and architectural approaches to address this, the underlying cause of the performance drop remains unclear. In this paper, we identify the connection between the output of the encoder (a stack of convolutional layers) and the ensuing dense layers as the main underlying factor limiting scaling capabilities; we denote this connection as the **bottleneck**, and we demonstrate that previous approaches implicitly target this bottleneck. As a result of our analyses, we present global average pooling as a simple yet effective way of targeting the bottleneck, thereby avoiding the complexity of earlier approaches.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the reasons behind the performance drop observed when scaling deep reinforcement learning (RL) in pixel-based environments. The authors pinpoint the connection between the convolutional encoder's output and the subsequent dense layers, which they term the \"bottleneck\", as the primary limiting factor. They argue that previous successful approaches implicitly address this bottleneck. Based on their analysis, the paper proposes Global Average Pooling (GAP) as a straightforward and effective method to target this bottleneck, offering a simpler alternative to more complex solutions found in the literature.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe paper offers a simple, effective, and practical solution to a common bottleneck in supervised learning, particularly relevant to RL pixel literature. The authors have conducted a thorough empirical investigation with commendable breadth and presentation of experiments. A key strength is the demonstration that \"bigger doesn't mean better\" in many RL problems, providing an easily implementable fix. The inclusion of wall-clock times for training versus performance is a valuable addition, highlighting the practical benefits of their approach. The application of sparsity to the bottleneck is a compelling finding, with the results presented being impressive and supporting the method's benefits. The core idea of investigating 'dormant' neurons and their impact on model performance is also a useful insight.\n\nWeaknesses\nDespite its strengths, the paper has several areas that need improvement.\nFigure 9 is unclear and needs enhancement to better convey its message.\nWhile empirical evidence is strong, the paper would benefit from theoretical backing for its claims.\nTraining curves for all baseline methods, not just the full (non-GAP) network, are needed for a comprehensive comparison.\nThe methodology for determining 'dormant' neurons is unclear and needs explicit definition, along with a justification for the chosen approach.\nMore details are required on how sparsity was implemented (e.g., pruning method, frequency) and the observed effects of varying sparsity levels on model architecture and performance.\nThe claim that \"the performance of the 2 scaled models is nearly identical\" is questionable given the non-overlapping IQRs in Figure 10. A more nuanced discussion or explanation for why these deviations are not significant is needed.\nThe claim that GAP is \"higher performing than all other methods\" lacks robust statistical justification, especially given the visual overlap in distributions. Statistical test results (e.g., Wilcoxon rank-sum test) are needed to support this claim, or it should be refined.\n\nQUESTIONS:\nIt is currently unclear how 'dormant' neurons are determined. There are numerous possible approaches (e.g., activity thresholds, gradient magnitudes, L0 norm of weights). Please explicitly define the methodology used to identify 'dormant' neurons. If multiple approaches were considered, briefly comment why the chosen method was selected over others.\n\nYou mention following the work of others and aiming for 90% sparsity. The effect of increased sparsity on model changes and performance is a crucial aspect that needs further elaboration. Can you provide more details on how sparsity was implemented (e.g., pruning method, frequency). Elaborate on the observed effects of varying sparsity levels (even if only 90% was the target) on both the model's architecture (e.g., changes in connectivity, effective parameters) and its performance metrics.\n\nThe scaling by 4 factor seems arbitrary without further context, though there is likely a good reason for it. Is it related to a specific architectural choice, computational constraint, or a baseline from previous work?\n\nFigure 10 & Performance Claims. You conclude that \"the performance of the 2 scaled models is nearly identical.\" However, observation of the IQRs (presumed to be the shaded regions) shows non-overlapping distributions across many frame numbers. If the scaling of this figure were different, these deviations might appear more significant. While the overall conclusion regarding the bottleneck might be correct, it's important to acknowledge and comment on these visible deviations in the IQRs. If the differences are not considered significant, please explain why.\n\nOverlapping distributions for GAP and SoftMoE are shown [boxplots]. It's questionable whether a notched boxplot or a Wilcoxon rank-sum test would show statistically significant differences in their medians. Given this visual overlap, can you provide a more robust argument for why GAP is considered \"higher performing than all other methods.\" If statistical tests were performed, please give the results. If not, consider performing them or refining the claim.\n\nMinor Edits\nP3 Deep RL network > networks: Change \"network\" to \"networks\" for plural consistency.\nP3 tokenzing > tokenizing: Correct the spelling from \"tokenzing\" to \"tokenizing.\"\nFig 11 ProcGen [spelling]: Correct the spelling of \"ProcGen\" in the figure caption\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nScaling up the performance of visual deep reinforcement learning has been a challenging task for long time, where usually scaling up the network lead to worse performance. While many approaches have been proposed in the past, including sparsity and MoE, they are usually complex. This paper identifies the problem that prevents scaling to be the size of the bottleneck layer (the output of encoder). When the size of bottleneck is large (which is usually the case in visual tasks), scaling becomes challenging. The authors introduces global average pooling (GAP) as a simple yet effective approach architectural change to the encoder that reduce the bottleneck size and allow for performance gains.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: \n- The paper is easy-to-follow.\n- The paper provide a comprehensive set of experiments studying various aspects of the problem and solutions. I enjoyed reading the analysis and discussion.\n- The approach is simple yet effective and can be easily incorporated into various approaches which enhance its adoption.\n- The method is hyper-parameter free which means it can be included easily into visual RL methods.\n\n**Weaknesses:**\n- The width-depth scaling claims is unsupported. The authors claim that GAP unlocks width and depth scaling, but this is never supported with an experiment. I except to see two figures both having performance on the y-axis vs width and depth on the x-axis. The current experiment only shows that the performance using GAP is better than the baseline in wide, deep, and wide&deep variations. No claims about scaling can be made using this experiment. If the authors want to keep the claim, then they need to support it with the proper evidence.\n- The authors use the word scaling in a different way from its common use which makes things a bit complicated. The argument of the scaling is almost always missing each sentence. For example, one can argue a method X allows (performance) scaling with the width of the encoder layers then show a figure with the width on the x-axis and performance on the x-axis. This supports the claim clearly. On the other hand, the paper use the word scaling loosely which makes it lose its meaning altogether. The only place where such experiment is presented in Figure 6 (first two) where the performance barely scales with width and doesn't scale. with replay ratio. \n- The paper lacks providing any reason for this bottleneck phenomenon. It remains unclear why the bottleneck can severely affect performance. I think the paper should at least give an explanation or even a hypothesis for why this is the case.\n\nI think the paper provides a good contribution, but I think the authors should work on their claims to avoid overstating the results. I think the paper addresses an important problem and there is an audience for it. I highly encourage the authors to improve their writing and moderate their claims to reflect what is given in the experiments.\n\n**Minor issues:**\n- \"We conduct a series of analyses, both quantitative and qualitative, to provide evidence for our main hypothesis.\" -> typically the purpose of the experiment is to validate the hypothesis rather than giving evidence for its correctness. I suggest saying \"validate our main hypothesis\"\n\nQUESTIONS:\nIn Figure 6, why do we see a performance drop when we scale the width by a multiple of 2 and increase with a multiple of 4?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations of their approach in the limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nScaling deep reinforcement learning (RL) in vision-based environments has long posed challenges that contrast with the consistent performance gains observed in supervised learning. This paper identifies a key bottleneck in typical pixel-based RL architectures—specifically, the interface between the convolutional encoder (denoted $\\phi$) and the subsequent fully connected layers ($\\psi$). The authors offer a unifying explanation for the success of recent architectural and algorithmic innovations, arguing that these methods implicitly address this $\\phi$ → $\\psi$ bottleneck. In response, the paper proposes a principled yet minimal intervention: inserting a Global Average Pooling (GAP) layer to reduce the spatial output of $\\phi$ into a compact C-dimensional representation before passing it to $\\psi$. This simple modification yields substantial empirical benefits across multiple benchmarks (Atari 200M, Atari 100k, Procgen), enabling both width and depth scaling, improving representational quality (via reduced neuron dormancy and feature norm magnitudes), and offering superior runtime efficiency relative to more complex approaches such as mixtures-of-experts and pruning-based scheme\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper proposes a simple yet highly effective architectural modification—Global Average Pooling (GAP)—to address a longstanding challenge in scaling vision-based deep RL.\n- The authors present a unifying perspective suggesting that the performance improvements observed in recent methods such as SoftMoE and pruning primarily stem from their implicit targeting of the architectural bottleneck—namely, the connection between the encoder $\\phi$ and the head $\\psi$. This insight provides a valuable conceptual lens for understanding scaling dynamics in vision-based deep RL. Moreover, the paper offers empirical evidence that effective depth scaling is possible—an underexplored axis of architectural scaling, as prior work has predominantly focused on width alone.\n- The study features extensive empirical validation, including large-scale experiments on the full Arcade Learning Environment (ALE) with 200 million frames, the 100k ALE setting, and Procgen, thereby demonstrating the robustness and generality of the proposed method.\n- The proposed approach is not only performant but also computationally efficient and implementation-friendly, requiring minimal architectural changes and providing faster training time relative to alternatives such as SoftMoEs and pruning based methods.\n\n\nWeaknesses\n\n- While the paper presents compelling empirical evidence for the role of the $\\phi$ → $\\psi$ bottleneck in scaling performance, its explanation remains largely empirical. The authors do not provide a theoretical account or a detailed representational analysis to clarify why the absence of a well-structured bottleneck—and the use of flattened encoder outputs—leads to degraded plasticity or impaired learning dynamics. Given that this phenomenon lies at the core of the paper’s contribution, a more principled explanation would substantially strengthen the work’s impact and conceptual clarity.\n- Despite broad benchmarking, the experimental domains remain confined to pixel-based benchmarks with discrete action spaces (e.g., ALE and Procgen), with no evaluation on continuous control domains (e.g., DeepMind Control Suite, Adroit), limiting the scope of generalizability. In addition, I think that \n- The absence of parameter count reporting presents a potential concern. Since the introduction of the GAP layer substantially reduces the number of parameters by collapsing spatial dimensions, it remains unclear whether the observed performance gains result from genuine improvements in scaling capacity or merely from more efficient parameter utilization. In typical scaling discussions, increased model size is often implied; however, with GAP, the networks may actually become smaller. Clarifying this distinction—and including performance comparisons at matched parameter counts—would offer a more rigorous understanding of whether GAP enables true architectural scaling or simply more effective use of fewer parameters.\n\nQUESTIONS:\n1. Have the authors evaluated the effectiveness of GAP in data-augmented actor-critic settings such as DrQ or DrQ-v2? These methods have demonstrated strong performance in pixel-based continuous control tasks and incorporate architectural bottlenecks by design. Notably, the DrQ architecture includes a hard-coded low-dimensional bottleneck, and it is well known within the research community that naively removing this component leads to substantial performance degradation. It would be valuable to position the proposed bottleneck intervention in the context of such existing architectural choices.\n\n2. Figure 4 suggests that scaling both $\\phi$ and $\\psi$ improves performance, but scaling only $\\psi$ hurts performance. Could the authors include results for a $\\phi$-only scaling variant to further substantiate the attribution of degradation of only scaling $\\psi$? This ablation would help isolate the relative contributions of encoder and head scaling and offer stronger support for the claim that the performance degradation resides primarily in $\\psi$\n\n3. The paper focuses heavily on scaling the fully connected head $\\psi$, while prior work in computer vision typically emphasizes scaling the encoder $\\phi$. Could the authors elaborate on the motivation behind this choice? While it is understood that, in typical vision-based RL architectures, $\\psi$ contains the majority of the parameters, it remains unclear why encoder scaling—which is the standard approach in computer vision tasks—is deprioritized. From a representational learning perspective, enhancing the encoder capacity would seem more intuitively aligned with performance improvements.\n\n4. Given that GAP aggregates over spatial dimensions, how do the authors view its applicability in environments that require fine-grained spatial reasoning or partial observability? For instance, continuous control domains such as Adroit or the DeepMind Control Suite may suffer from the loss of spatial information. Have the authors explored or considered this trade-off?\n\n5. Minor typo – “Procegn → Procgen” in Fig 11\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper considers the problem of scaling in pixel-based deep RL, that is, under what circumstances is it possible to improve the performance of deep RL algorithms by increasing the size of the network. They find that using global average pooling after the convolutional layers allows scaling. They show good empirical performance with this simple idea on Atari benchmarks.

STRENGTHS AND WEAKNESSES:
## Strengths

* The paper achieves strong performance with increasing model size and a simple architectural modification: global average pooling after convolutional layers in the value network.

## Weaknesses

* The main hypothesis of the paper says low-dimensional and well structured bottleneck allows scaling. Does that mean that a 2x2xd bottleneck doesn't allow scaling? Also, what does well-structured really mean? Does it have a clear definition? From my understanding of the paper it's only shown that global average pooling allows scaling but then why turn this into an unclear and more general statement?
* The paper is technically void. There's not a single new or interesting equation (let alone theorem) in the paper.
* The point of the paper could have been conveyed in one page.

**Minor:**
An expectation is missing in the definition of Q-value.

**Summary:**
On the one hand, the finding is interesting and can have impact, on the other hand, the paper is bloated and has the weaknesses I've mentioned. Therefore, I am borderline on this paper.

QUESTIONS:
* Fig. 2 (Right) which game is this?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-06