Title: 1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
Venue: NeurIPS 2025
Accepted: yes
Keywords: Reinforcement Learning

Self-Supervised Learning

Contrastive RL

Goal-conditioned RL

Scaling
Abstract: Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 -- 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance.
Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals.
Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by $2\times$ -- $50\times$, outperforming other goal-conditioned baselines.
Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the potential of deep network architectures in self-supervised reinforcement learning (RL) to enhance scalability. Unlike most recent RL studies that use shallow networks (2-5 layers), the authors show that increasing network depth to up to 1024 layers can significantly improve performance. The experiments are conducted in an unsupervised, goal-conditioned setting without demonstrations or rewards, requiring the agent to explore and learn to reach commanded goals. Tested on simulated locomotion and manipulation tasks, the approach achieves 2x-50x performance improvements over other goal-conditioned baselines, demonstrating that greater depth not only boosts success rates but also qualitatively alters the learned behaviors.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The idea is interesting, and the writing is clear and easy to understand.\n2. The authors conducted numerous experiments to demonstrate that increasing the depth of networks is a key factor in improving the performance of CRL.\n3. The authors carried out extensive analysis experiments to explore various issues, including some that the reviewer might be concerned about. For example, whether increasing the depth of networks can enhance the performance of other baseline algorithms. Although the answer is no, combining scaling depth or network size with other algorithms may be one of the important future directions for RL.\n\n**Weaknesses**\n\n1. There is an error in the citation of the appendix at line 173, which shows as \"？？\".\n2. The paper does not mention the training and inference costs of the proposed algorithm. The training and inference costs associated with a network depth of 1000 layers may be relatively high.\n3. The innovativeness of the article has certain deficiencies. The core technologies involved (such as residual connections, normalization, activation functions, etc.) are derived from existing research. The main novelty of the article lies in the impact of the increase in model size on performance, rather than innovation in methodology.\n4. The contributions of the article are mainly experimental, with a lack of theoretical discussion on the critical parts of the experiments. For example, why does the performance undergo a sudden change when the depth reaches a certain critical point? This, to some extent, limits the article's insight.\n\nQUESTIONS:\n1. How do the training costs of the proposed algorithm compare with those of the baselines? Can anthors show it in a Table？\n2. Can the authors elaborate empirically or conceptually on why performance “jumps” occur at specific depths or why agents exhibit different behaviors at certain depths (as shown in Figure 3) in certain environments?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates scaling the depth of the networks used in self-supervised reinforcement learning for goal-reaching tasks. The work shows that scaling depth can significantly boost the performance of Contrastive RL in the JaxGCRL suite of online RL tasks. The authors also extensively analyze their design choices and provide qualitative analyses of the behavior and the value functions learned.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* **Strong empirical analysis**: the empirical analysis of the approach is extensive and I found it particularly useful. It clearly shows how scaling depth can significantly improve performance for Contrastive RL. However, it also provides analyses about: scaling width vs depth, the usefulness of residual layers, scaling actor vs critic network, etc.\n* **New SOTA on JaxGCRL**: the results obtained with scaling depth and data are impressive. Also, improving the state-of-the-art in a benchmark is useful to further push the boundaries of research on that benchmark and in the field.\n## Weaknesses \n* **Limited generality of the claims** : my main concern with this work is that the idea of scaling depth seem to apply to a very specific algorithm, i.e. contrastive RL, and in very specific settings, i.e. online RL. I appreciate the authors' attempts in testing their method in offline RL settings and using CMD-1 (in Appendix). However, the generality of their claims remain so far limited.\n\nQUESTIONS:\n- How is training time affected as you scale depth? \n\nAlso, I leave here some questions/idea for the authors to test outside of their contrastive RL+ online RL settings:\n* Given that simply scaling depth in the actor seems beneficial, would scaling depth be useful for BC and GCBC in Offline RL settings? \n* I understand the authors provide a study on CMD-1 in the Appendix, which uses the MRN network for quasimetric learning. It would be useful to also study QRL [1], which instead uses IQE for quasimetric learning\n\nMinor note: \n* the reference at Line 173 is undefined (probably due to the Figure being in the Appendix) \n\n[1] Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning\n\nLIMITATIONS:\nThe authors have adequately addressed limitations in their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors show that depth scaling properties arise in contrastive RL, whereas width scaling is prominent in recent vanilla RL literature [1,2]. They propose to stack neural network blocks with residual connection, normalization (LayerNorm), and Swish architecture. By increasing the number of blocks, authors found that goal reaching capabilities emerge after a certain threshold, shown both by increased performance measure (figure 1) and novel behavior (figure 3). In their analysis, they find that depth scaling improves contrastive representations, enhances exploration, and allows trajectory stitching.\n\n[1] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning., Lee et al.\n\n[2] Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners., Nauman et al.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Strong performance and clear depth-scaling tendency to support their claims.\n- In-depth analysis on the architecture and policy as depth scales.\n- Verification of contrastive RL algorithm compared to baseline methods.\n\nWeaknesses\n- The experiments are limited to one type of architecture. Whether the same trend would occur in other well-designed architectures in RL [1,2,3] is unsure.\n- On a similar note, no sufficient analysis was made on the design choices of the architecture, other than the residual connection.\n- The connection between depth and exploration is not well explained. The results presented in figure 13 could also be due to improved learning capability of the collector.\n- Due to the model's sequential nature, a depth-scaled model is likely to take more time to generate output than a width-scaled model with same number of parameters. The claim that depth scaling is more compute efficient than width scaling (line 225:229) should thus we accompanied by a comparison on wall-clock time.\n- The fact that their architecture does not scale in offline GCRL weakens their point.\n\nOverall, I think the paper provides interesting and useful knowledge to the community, although there are several issues that needs to be refined.\n\n[1] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning., Lee et al.\n\n[2] Hyperspherical Normalization for Scalable Deep Reinforcement Learning., Lee et al.\n\n[3] Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control., Nauman et al.\n\nQUESTIONS:\nPlease see weaknesses above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates scaling network depth in self-supervised reinforcement learning, specifically using the Contrastive RL (CRL) algorithm for goal-conditioned tasks. The authors demonstrate that increasing network depth up to 1024 layers can yield substantial performance improvements in unsupervised goal-reaching tasks. Their approach combines three key components: (1) self-supervised contrastive RL that learns without demonstrations or rewards, (2) GPU-accelerated training frameworks for increased data throughput, and (3) very deep residual networks with architectural stabilization techniques. The experiments are conducted on locomotion, navigation, and manipulation tasks from the JaxGCRL benchmark, showing 2-50× performance improvements over shallow networks typically used in RL (2-5 layers). The paper provides extensive analysis of why depth scaling works, including improved exploration, better representational capacity, and emergent qualitatively different behaviors at critical depth thresholds.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper provides a clearly-written comprehensive study showing that depth scaling (up to 1024 layers) can dramatically improve RL performance, contrary to conventional wisdom.\n2. The paper shows strong empirical evidence, by demonstrating consistent improvements across diverse tasks with proper statistical reporting (error bars across multiple seeds), across 10 diverse environments spanning locomotion, navigation, and manipulation.\n3. I appreciate the thorough analysis through visualization of Q-functions, learned representations, and emergent behaviors (wall vaulting, creative navigation). \n4. The approach provides effective adaptation of residual connections, layer normalization, and Swish activation for very deep RL networks.\n5. Good set of ablations are provided: width vs. depth, actor vs. critic scaling, batch size effects, and generalization capabilities.\n\nWeaknesses:\n1. It seems that the scaling benefits appear specific to CRL. The traditional TD methods (SAC, TD3) show no improvement or even show degradation with depth.\n2. The preliminary experiments show no benefits in offline settings, limiting applicability.\n3. The experiments are specific to unsupervised RL setting. It is unclear if the benefits may transfer to reward-rich (dense) or imitation learning settings.\n4. The paper would benefit from detailed analysis of computational overhead compared to baselines.\n5. As acknowledged by the authors, the approach requires substantial computational resources that may limit reproducibility.\n\nQUESTIONS:\n1. Can you provide any intutive explanation to why the scaling benefits appear specific to CRL?\n2. Could you provide detailed computational cost analysis comparing depth vs. width scaling? What are the FLOPS, memory, and wall-clock time trade-offs? How does this inform practical deployment decisions?\n3. In Figure 1, the performance of these approaches increases with depth, and then seems to decrease when depth is increased further. What is the reason for such behavior? Is this pattern consistent with further depth experients? If yes, do you have any thoughts on finding the critical depth for particular tasks?\n4. In line 173, the figure that compares proposed approach with SAC, SAC+HER, TD3+HER, and CRL seems to be missing?\n\nLIMITATIONS:\nPlease refer to the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}