1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
Authors: Kevin Wang
Ishaan Javali
Michał Bortkiewicz
Tomasz Trzcinski
Benjamin Eysenbach
Venue: NeurIPS 2025 oral ()
Keywords: Reinforcement Learning
Self-Supervised Learning
Contrastive RL
Goal-conditioned RL
Scaling
Primary area: general_machine_learning
TL;DR: While most RL methods use shallow MLPs (~2–5 layers), we show that scaling up to 1000-layers for contrastive RL (CRL) can significantly boost performance, ranging from doubling performance to 50x on a diverse suite of robotic tasks.

Abstract: Scaling up self-supervised learning has driven breakthroughs in language and vision, yet comparable progress has remained elusive in reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architectures (around 2 -- 5 layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance.
Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, so an agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals.
Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by $2\times$ -- $50\times$, outperforming other goal-conditioned baselines.
Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned.

1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities
Kevin Wang Princeton University
Ishaan Javali Princeton University jaavali@rinneton. edu edu
kw6487@princetoon edu
Michal Bortkiewicz
Tomasz Trzciński
Warsaw University of Technology mi chalh worttiiewicz.dokttggee edu.p! pl
Warsaw University of Technology.
Tooploox, IDEAS Research Institute
Benjamin Eysenbach Princeton University eysenbach@princeton. edu
Abstract
Scaling up self-supervised learning has driven breakthroughs in language and vision. yet comparable progress has remained elusive reinforcement learning (RL). In this paper, we study building blocks for self-supervised RL that unlock substantial improvements in scalability, with network depth serving as a critical factor. Whereas most RL papers in recent years have relied on shallow architec- tures (around 2 - layers), we demonstrate that increasing the depth up to 1024 layers can significantly boost performance. Our experiments are conducted in an unsupervised goal-conditioned setting, where no demonstrations or rewards are provided, agent must explore (from scratch) and learn how to maximize the likelihood of reaching commanded goals. Evaluated on simulated locomotion and manipulation tasks, our approach increases performance on the self-supervised contrastive RL algorithm by 2x - 50x, outperforming other goal-conditioned baselines. Increasing the model depth not only increases success rates but also qualitatively changes the behaviors learned. The project webpage and code can be found here: https://wang- wew.aangg-eiiii...... htt.:///....ttb..o///iiigggggee
Introduction
While scaling model size has been an effective recipe in many areas of machine learning, its role and impact in reinforcement learning (RL) remain unclear. The typical model size for state-based RL tasks between to 5 layers (Raffin al., 2021; Huang al., 2022). In contrast, it not uncommon to use very deep networks in other domain areas; Llama 3 (Dubey et al., 2024) and Stable Diffusion 3 (Esser et al., 2024) have hundreds of layers. fields such as vision (Radford er al., 2021; Zhai et al., 2021; Dehghani et al., 2023) and language (Srivastava et al., 2023), models often only acquire the ability to solve certain tasks once they are larger than a critical scale. the RL setting, many researchers have searched for similar emergent phenomena (Srivastava et al., 2023), but these papers typically report only small marginal benefits and typically only on tasks where small models already achieve some degree of success (Nauman al., 2024b; Lee et al., 2024; Farebrother et al., 2024). A key open question in RL today is whether it is possible to achieve similar jumps in performance by scaling RL networks.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Humanoid
Ant Big Maze
Ant U4-Maze
Ant US-Maze
Ant Hardest Maze
732
500
330
Time at Goal
602
240
150
632
75
300
302
120
200
132
50
25
100
1.00
50
1.0
120
300
600
57
330
150
201
Arm Push Easy
Arm Push Hard
Amm Binp ick Hard
Humanoid U-Maze
Humanoid Big Maze
500
230
220
60-
002
400
230
160
Time at Goal
45-
692
300
150
120
200
80
203-
100
30
1.90
120
1000
stop (M)
Erv stop (M)
Env stop
Env step LM)
step (M)
Depth
Depth
Degth 16
Depth 32
Depth 64
Figure 1: Scaling network depth yields performance gains across a suite of locomotion, navigation, and manipulation tasks, ranging from doubling performance to S0x improvements on Humanoid- based tasks. Notably, rather than scaling smoothly, performance often jumps specific critical depths (e.g.. layers Ant Big Maze, 64 Humanoid U-Maze), which correspond to the emergence of qualitatively distinct policies (see Section 4).
At first glance, it makes sense why training very large RL. networks should be difficult: the RL problem provides very few bits of feedback (e.g., only sparse reward after a long sequence of observations), so the ratio of feedback to parameters is very small. The conventional wisdom (LeCun, 2016), reflected in many recent models (Radford, 20 18; Chen et al., 2020; Goyal et al., 2019), has been that large AI systems must be trained primarily in a self-supervised fashion and that RL should only be used finetune these models. Indeed, many of the recent breakthroughs in other fields have been primarily achieved with self-supervised methods, whether in computer vision (Caron et al., 2021; Radford et al., 2021; Liu al., 2024), NLP (Srivastava et al., 2023), or multimodal learning (Zong et al.. 2024). Thus, if we hope to scale reinforcement learning methods, self-supervision will likely be a key ingredient.
In this paper, we will study building blocks for scaling reinforcement learning. Our first step is to rethink the conventional wisdom above: "reinforcement learning" and "self-supervised learning" are not diametric learning rules, but rather can be married together into self-supervised RL systems that explore and learn policies without reference a reward function or demonstrations (Eysenbach et al., 2021, 2022; Lee et al., 2022). In this work, we use one of the simplest self-supervised RL algorithms, contrastive RL (CRL) (Eysenbach et al., 2022). The second step is recognize the importance of increasing data availability. We will do this by building on recent GPU-accelerated al., 2021: Rutherford et al., 2023; Rudin et al., 2022; Bortkiewicz
RL frameworks (Makoviychuk et al., 2024). The third step is to increase network depth, using networks that are up to 100x deeper than those typically found in prior work. Stabilizing the training of such networks will require incorporating architectural techniques from prior work, including residual connections (He et al., 2015), layer normalization (Ba et al., 2016), and Swish activation (Ramachandran al., 2018). Our experiments will also study the relative importance of batch size and network width.
The primary contribution of this work is to show that a method that integrates these building blocks into a single RL approach exhibits strong scalability:
Empirical Scalability: We observe a significant performance increase, more than 20x in half of the environments and outerroormin other standard goal-conditioned baselines. These performance gains correspond to qualitatively distinct policies that emerge as the scale increases.
Scaling Depth in Network Architecture: While many prior RI. works have primarily focused on increasing network width, they often report limited or even negative returns

---

when expanding depth (Lee et al., 2024; Nauman et al., 2024b). In contrast, our approach unlocks the ability scale along the axis of depth, yielding performance improvements that surpass those from scaling width alone (see Sec. 4).
Empirical Analysis: We conduct extensive analysis of the key components in our scaling approach, uncovering critical factors and offering new insights.
We anticipate that future research may build on this foundation by uncovering additional building blocks.
2
Related Work
Natural Language Processing (NLP) and Computer Vision (CV) have recently converged in adopting similar architectures (i.e. transformers) and shared learning paradigms (i.e self-supervised learning), which together have enabled transformative capabilities of large-scale models (Vaswani et al., 2017; Srivastava et al., 2023; Zhai et al., 2021; Dehghami et al., 2023; Wei et al., 2022). In contrast, achieving similar advancements in reinforcement learning (RL) remains challenging. Several studies have explored the obstacles to scaling large RL models, including parameter underutilization (Obando- Ceron et al., 2024), plasticity and capacity loss (Lyle et al., 2024, 2022), data sparsity (Andrychowicz al., 2017; LeCun, 2016), and training instabilities (Ota et al., 2021; Henderson et al., 2018: 8;; Van Hasselt et al., 2018; Nauman et al., 2024a). As a result, current efforts to scale RI. models are largely restricted to specific problem domains, such as imitation learning (Tuyls et al., 2024), multi-agent games (Neumann and Gros, 2022), language-guides RL (Driess al., 2023; Ahn et al., 2022), and discrete action spaces (Obando-Ceron et al., 2024; Schwarzer et al., 2023).
Recent approaches suggest several promising directions, including new architectural paradigms (Obando-Ceron et al., 2024), distributed training approaches (Ota et al., 2021; Espeholt et al., 2018), distributional RL (Kumar et al., 2023), and distillation (Team et al., 2023). Compared to these approaches, our method makes a simple extension to an existing self-supervised RL algorithm. The most recent works in this vein include Lee et al. (2024) and Nauman et al. (2024b), which leverage residual connections to facilitate the training of wider networks. These efforts primarily focus on network width, noting limited gains from additional depth, thus both works use architectures with only four MLP layers. In our method, we find that scaling width indeed improves performance (Section 4.4); however, our approach also enables scaling along depth, proving to be more powerful than width alone.
One notable effort to train deeper networks is described by Farebrother et al. (2024), who cast value- based RL into a classification problem by discretizing the TD objective into a categorical cross-entropy loss. This approach draws on the conjecture that classification- stable and thus may exhibit better scaling properties than their regressive counterparts (Torgo and Gama, 1996; Farebrother et al., 2024). The CRL algorithm that we use effectively uses a cross-entropy loss as well (Eysenbach et al., 2022). Its InfoNCE objective a generalization of the cross-entropy loss, thereby performing RL tasks by effectively classifying whether current states and actions belong to the same or different trajectory that leads toward a goal state. In this vein, our work serves as a second piece of evidence that classification, much like cross-entropy's 's role in the scaling success in NLP, could be a potential building block in RL.
methods can be more robust and
3 Preliminaries
This section introduces notation and definitions for goal-conditioned RL and contrastive RL. Our focus is on online RL, where a replay buffer stores the most recent trajectories, and the critic is trained in self-supervised manner.
Cond-Conditioned Reinforcement Learning We define a goal-conditioned MDP as tuple Mg = (S,A, Po. P. P. Pg: where the agent interacts with the environment to reach arbitrary goals (Kael- bling, 1993; Andrychowicz et al., 2017; Blier et al., 2021). At every time step t, the agent observes state 8; E S and performs a corresponding action o, E A. The agent starts interaction states sam- pled from po(sa), and the interaction dynamics are defined by the transition probability distribution ((8411 St, at). Goals g E 9 are defined goal space G. which is related to via a mapping f: S + G. For example, G may correspond to a subset of state dimensions. The prior distribution
3

---

over goals is defined by Pg( The reward function is defined as the probability density of reaching the goal in the next time step rg (5t, ax) _ (1 (1))p(ss++111 = = St, az). with discount factor yy In this setting, the goal-conditioned policy xa s, receives both the current observation of the environment as well as a goal. We define the discounted state visitation distribution as py (-(-119)) (s) P (1-7) Σx= Y y'p;" "pp,' (((l-a)))) (s), where P (s) is the probability that policy visits after exactly steps. when conditioned with This last expression is precisely the Q-function of the policy TT- (,g) for the reward rg: (7 a) A
p. px(1--a(( (g s, a). The objective is to maximize the expected reward:
max E (a) rro (st oz)
Contrastive Reinforcement Learning. Our experiments will use the contrastive RL algorithm (Ey- senbach et al., 2022) to solve condiiond problems. Contrastive RL is actor-critic method; we will use fa. (s, a. g) to denote the critic and Ta(a s,g) to denote the policy. The critic parametrized with two neural networks that return state, action pair embedding d(s, a) and goal embedding (bg)) The critic's output is defined as the [2-norm between these embeddings:
(s, ()) = ( a) - (b(g)l2-- The critic is trained with the InfoNCE objective (Sohn, 2016) as in previous works (Eysenbach et al., 2022, 2021; Zheng et al., 2023, 2024; Myers et al., 2024; Bortkiewicz al., 2024). Training is conducted on batches B, where 8;, ass Pi represent the state, action, and goal (future state) sampled from the same trajectory. while zj represents a goal sampled from different, random trajectory. The objective function is defined as:
18
( |== clore(srang))
elo.o(sianin))
min Es d,10
log
K
The policy ae(a s.g) trained maximize the critic:
E (o())) (o) (e (s,v(s,
me
Po Po
Residual Connections We incorporate residual connections (He al., 2015) into architec- ture, following their successful use in RL (Fare- brother al., 2024; Lee et al., 2024; Nauman al., 2024b). A residual block transforms given repre- sentation h, by adding learned residual function (h.)) to the original representation. Mathemat- ically, this expressed as:
x XN
LayerNorm
LayerNorm
Dense
Swish
Dense
Swish
Dense
h h, + F, (h:)
X 4
where h is the output representation, h, is the input representation, and F (h,) is transforma- tion learned through the network (e.g., using one or more layers). The addition ensures that the net- work learns modifications to the input rather than entirely new transformations, helping to preserve useful features from earlier layers. Residual con- nections improve gradient propagation by introducing shortcut paths (He et al., 2016; Veit et al., 2016), enabling more effective training of deep models.
Figure 2: Architecture. Our approach integrates residual connections into both the actor and critic net- works of the Contrastive RL algorithm. The depth of this residual architecture is defined the total num- ber Dense layers across the residual blacks, which, with our residual block size of 4, equates to 4N.
4 Experiments
4.1 Experimental Setup
Environments. All RL experiments use the JaxGCRL codebase (Bortkiewicz et al., 2024), which facilitates fast online GCRL experiments based Brax (Freeman et al., 2021) and MJX (Todorov

---

et al., 2012) environments. The specific environments used are range of locomotion, navigation, and robotic manipulation tasks, for details see Appendix B. We use sparse reward setting, with r = 1 only when the agent is in the goal proximity. For evaluation, we measure the number of time steps (out of 1000) that the agent is near the goal. When reporting an algorithm's performance as a single number, we compute the average score over the last five epochs of training.
Architectural Components We employ residual connections from the ResNet architecture (He al., 2015), with each residual block consisting of four repeated units of Dense layer, a Layer Normalization (Ba et al., 2016) layer, and Swish activation (Ramachandran et al., 20. We apply the residual connections immediately following the final activation of the residual block, shown in Figure 2. In this paper, we define the depth of the network as the total number of Dense layers across all residual blocks in the architecture. In all experiments, the depth refers to the configuration of the actor network and both critic encoder networks, which are scaled jointly, except for the ablation experiment in Section 4.4.
4.2 Scaling Depth in Contrastive RL
We start by studying how increasing network depth can increase performance. Both the JaxGCRL benchmark and relevant prior work (Lee al., 2024; Nauman et al., 2024b; Zheng et al., 2024) use MLPs with depth of 4, and as such we adopt it as our baseline. In contrast, we will study networks of depth 8, 16, 32, and 64. The results Figure/ demonstrate that deeper networks achieve significant performance improvements across diverse range of locomotion, navigation, and manipulation tasks. Compared to the 4-layer models typical in prior work, deeper networks achieve 2 - 5x gains in robotic manipulation tasks, over 2 gains in long-horizon maze tasks such Ant U4-Maze and Ant U5-Maze, and over 50x gains humanoid- tasks. The full table of performance increases up depth 64 is provided in Table 1.
In Figure 12, we present results the same 10 environments, but compared against SAC, SAC+HER, TD3+HER, GCBC, and GCSL. Scaling CRL leads to substantial performance improvements, outper- forming all other baselines in 8 out of 10 tasks. The only exception is SAC on the Humanoid Maze environments, where it exhibits greater sample efficiency early on; however, scaled CRL eventually reaches comparable performance. These results highlight that scaling the depth the CRL algorithm enables state-of-the-art performance in coal-conditioned reinforcement learning.
4.3 Emergent Policies Through Depth
A closer examination of the results from the per- formance curves in Figure reveals a notable pattern: instead of a gradual improvement in performance as depth increases, there are pro- nounced jumps that occur once critical depth threshold is reached (also shown in Figure 5). The critical depths vary by environment, ranging from 8 layers (e.g. Ant Big Maze) to 64 layers in the Humanoid U-Maze task, with further jumps occurring even at depths of 1024 layers (see the Testing Limits section, Section 4.4).
Deyth 4
Depthi0t
Prompted by this observation, we visualized the learned policies at various depths and found qualitatively distinct skills and behaviors ex- hibited. This is particularly pronounced in the humanoid-based tasks, illustrated in Figure Networks with a depth of exhibit rudimentary policies where the agent either falls or throws itself toward the target. Only at critical depth of 16 does the agent develop the ability to walk upright into the goal. the Humanoid U-Maze environment, networks of depth 64 struggle to navigate around the intermediary wall, collapsing on the ground, Remarkably at depth of 256, the agent learns unique behaviors on Humanoid U-Maze. These behaviors include folding forward into a leveraged position to propel itself over walls and
Figure 3: Increasing depth results in new capabili-
ties: Row 1: A depth-4 agent collapses and throws itself toward the goal. Row 2: depth- 16 agent walks upright. Row 3: A depth-64 agent struggles and falls. Row depth-256 agent vaults the wall acrobatically.
5

---

Humanoid
Ant Big Maze
Arm
Push Easy
Reaching Critical Depth Critical Depth with Connections
615
9:7
800
480
441
800
535
702
Ficakall Connociers
510
Critic Depth
GC5
583
1.9
420
437
800
840
922
454
402
316
424
504
427
409
830
740
187
35
323
263
101
18
500
350
4.5
370
563
100
615
93
345
0.02
16
64
8
16
32
8
22
64
32
Actor Depth
Actor Depth
Actor Depth
Network Dapth
Network Depth
Figure 5: Critical depth and residual connections. Incrementally increasing depth results in marginal per- formance gains (left). However, once a critical thresh- old is reached, performance improves dramatically (right) for networks with residual connections.
Figure 6: Actor Critic. In Arm Push Easy, scaling the critic is more effective; Ant Big Maze, the actor matters more. For Humanoid, scaling both is necessary. These results suggest that actor and critic scaling can complement each other for CRL.
shifting into seated posture over the intermediary obstacle to worm its way toward the goal (one of these policies is illustrated in the fourth row of Figure 3). To the best of our knowledge, this is the first goal-conditioned approach to document such behaviors on the humanoid environment.
4.4 What Matters for CRL Scaling
Width Depth Past literature has shown that scaling network width can be effective (Lee al., 2024; Nauman et al., 2024b). In Figure 4, we find that scaling width is also helpful in our experiments: wider networks consistently out- perform narrower networks (depth held constant 4). However, depth seems to be more effec- tive axis for scaling: simply doubling the depth to 8 (width held constant at 256) outperforms the widest networks in all three environments. The advantage of depth scaling is most pronounced in the Humanoid environment (observation di- mension 268), followed by Ant Big Maze (di- mension 29) and Arm Push Easy (dimension 17), suggesting that the comparative benefit may increase with higher observation dimensionality.
Humanoid
Ant Big Maze
Arm Push Easy
Time Goal
4cc-
200
Model Params
Model Params
Model Params
with 11
Wic 2011
Figure 4: Scaling network width depth. Here, we reflect findings from previous works (Lee et al., 2024; Nauman ct al., 2024b) which suggest that in- creasing network width can enhance performance. How- ever, in contrast to prior work, method is able scale depth, yielding more impactful performance gains. For instance, in the Humanoid environment, raising the width to 2048 (depth=4) fails to match the performance achieved by simply doubling the depth to (width=256). The comparative advantage of scaling depth is more pro- nounced as the observational dimensionality increases.
Note additionally that the parameter count scales linearly with width but quadratically with depth. For comparison, network with 4 MLP layers and 2048 hidden units has roughly 35M parameters, while one with a depth of 32 and 256 hidden units has only around 2M. Therefore, when operating under fixed FLOP compute budget or specific memory constraints, depth scaling may be more computationally efficient approach to improving network performance.
Scaling the Actor Critic Networks To investigate the role of scaling in the actor and critic networks, Figure 6 presents the final performance for various combinations of actor and critic depths across three environments. Prior work (Nauman et al., 2024b; Lee et al., 2024) focuses scaling the critic network, finding that scaling the actor degrades performance. In contrast, while we do find that scaling the critic is more impactful in of the three environments (Humanoid, Arm Push Easy), our method benefits from scaling the actor network jointly, with one environment (Ant Big Maze) demonstrating actor scaling be more impactful. Thus, our method suggests that scaling both the actor and critic networks can play complementary role in enhancing performance.
Deep Networks Unlock Batch Size Scaling Scaling batch size has been well-established in other areas of machine learning (Chen et al., 2022; Zhang et al., 2024). However, this approach has not translated as effectively to reinforcement learning (RL), and prior work has even reported negative impacts value-based RL CObando-Ceron al., 2023). Indeed, in our experiments,

---

Depth
Depth
Depth 16
Depth 64
30
800
800
Time at Goal
400
600
20
300
400
200
10-
100
50
100
30
100
50
100
50
Env.stop
stop (M)
stop
Env stop (M)
Batch Size:
128
256
512
1024
2048
Figure 7: Deeper networks unlock batch size scaling. We find that as depth increases from 4 to 64 in Humanoid, larger networks can effectively leverage batch size scaling to achieve further improvements.
simply increasing the batch size for the original CRL networks yields only marginal differences in performance (Figure 7, top left).
At first glance, this might seem counterintuitive: since reinforcement learning typically involves fewer informational bits per piece of training data (LeCun, 2016), one might expect higher variance in batch loss gradients, suggesting the need for larger batch sizes compensate. At the same time, this possibility hinges on whether the model in question can actually make use of a bigger batch size-in domains of ML where scaling has been successful, larger batch sizes usually bring the most benefit when coupled with sufficiently large models (Zhang et al., 2024; Chen et al., 2022). One hypothesis is that the small models traditionally used in RL may obscure the underlying benefits of larger batch size.
To test this hypothesis, we study the effect of increasing the batch size for networks of varying depths. As shown in Figure 7, scaling the batch size becomes effective as network depth grows. This finding offers evidence that by scaling network capacity, may simultaneously unlock the benefits of larger batch size, potentially making it an important component in the broader pursuit of scaling self-supervised RL.
Training Contrastive RL with 1000+ Layers We next study whether further increasing depth beyond 64 layers further improves performance. We use the Humanoid maze tasks as these are both the most challenging environments in the benchmark and also seem benefit from the deepest scaling. The results, shown in Figure 12, indicate that performance continues to substantially improve as network depth reaches 256 and 1024 layers in the Humanoid U-Maze environment. While were unable to scale beyond 1024 layers due to computational constraints, we expect to see continued improvements with even greater depths, especially the most challenging tasks.
Humanoid
Humanoid
Ant Big Maze (Deep Collector) (Shallow Collector)
Ant Big Maze
Arm Push Easy Arm Push Easy (Deep Collector) (Shallow Collector)
(Deep Collector) (Shallow Collector)
700-
500
600-
at Goal
500-
600
400
300
400
200
200
200
100
50 Env Steps (M)
100
En Steps (M)
Env Steps (M)
Env Steps (M)
Env Steps (M)
Env Steps (M)
Collector (Depth 32/ / Depth 4)
Deep (Depth 32)
Shallow (Depth 4)
Figure 8: We disentangle the effects of exploration and expressivity on depth scaling by training three networks in parallel: "collector," plus one deep and one shallow learnrr that train only from the collector's shared replay buffer. In all three environments, when using deep collector (i.e. good data coverage). the deep learner outperforms the shallow learner, indicating that expressivity is crucial when controlling for good exploration. With a shallow collector (poor exploration), even the deep learner cannot overcome the limitations of insufficient data coverage. As such, the benefits of depth scaling arise from combination of improved exploration and increased expressivity working jointly.

---

4.5
Why Scaling Happens
Depth Enhances Contrastive Representations The long-horizon setting has been a long-standing challenge in RL particularly in unsupervised goal- conditioned settings where there no auxiliary re- ward feedback (Gupta et al., 2019). The family of U-Maze environments requires a global understand- ing of the maze layout for effective navigation. We consider a variant of the Ant U-Maze environment, the U4-maze, in which the agent must initially move in the direction opposite the goal to loop around and ultimately reach it. As shown in Figure 9, ob- serve a qualitative difference in the behavior of the shallow network (depth 4) compared the deep net- work (depth 64). The visualized Q-values computed from the critic encoder representations reveal that the depth 4 network seemingly relies on Euclidean distance to the goal as proxy for the Q value, even when wall obstructs the direct path. In contrast, the depth 64 critic network learns richer representations, enabling it to effectively capture the topology of the maze as visualized by the trail high Q values along the inner edge. These findings suggest that increasing network depth leads to richer learned representations, enabling deeper networks to better capture env enviro inviromment topology and achieve more comprehensive state-space coverage in a self-supervised manner.
Depth 4
Depth 64
Figure 9: Deeper Q-functions are qualitatively different. In the U4-Maze, the start and goal posi- tions are indicated by the @ and G symbols respec- tively. and the visualized Q values via the distance the learned representation space, i.c., ( a. g) = ll0(s, a) W(g)ll2. The shallow depth 4 network (left) naively relies on Euclidean proximity, showing high values near the start despite maze wall. In contrast, the depth network (right) clusters high Q values goal, gradually tapering along the interior.
computed
Depth Enhances Exploration and Expressivity in a Synergized Way Our earlier results suggested that deeper networks achieve greater state-action coverage. To better understand why scaling works, we sought to determine to whether improved data alone explains the benefits of scaling, or whether acts in conjunction with other factors. Thus, we designed an experiment in Figure 8 in which we train three networks in parallel: one network, the "collector," interacts with the environment and writes all experience to shared replay buffer. Alongside it, two additional "learners" one deep and one shallow, train concurrently. Crucially, these two learners never collect their own data; they train only from the collector's buffer. This design holds the data distribution constant while varying the model's capacity, so any performance gap between the deep and shallow learners must come from expressivity rather than exploration. When the collector deep (e-g., depth 32), across all three environments the deep learner substantially outperforms the shallow one across all three environments, indicating that the expressivity of the deep networks is critical. On the other hand, we repeat the experiment with shallow collectors (e.g., depth 4), which explores less effectively and therefore populates the buffer with low-coverage experience. Here, both deep and shallow learners struggle and achieve similarly poor performance, which indicates that the deep network's additional capacity does not overcome the limitations insufficient data coverage. As such, scaling depth enhances exploration and expressivity in synergized way: stronger learning capacity drives more extensive exploration, and strong data coverage is essential to fully realize the power of stronger learning capacity. Both aspects jointly contribute to improved performance.
Deep Networks Learn to Allocate Greater Representational Capacity to States Near the Goal In Figure 10 we take successful trajectory in the Humanoid environment and visualize the embed- dings of state-action encoder along this trajectory for both deep vs. shallow networks. While the shallow network (Depth 4) tends to cluster near-gou goal states tightly together, the deep network produces more "spread out" representations. This distinction important: our representations to separate states that matter- paatterrrrricclarrlyy future or goal-relevant states-from random ones. As such, we want to allocate more representational capacity such critical regions. This suggests that deep networks may learn to allocate representational capacity more effectively to state regions that matter most for the downstream task.
self-supervised setting, we want

---

Successful Trajectory Path in Humanoid Env
Trajectory in Embedding Space (Depth 4)
Trajectory in Embedding Space (Depth 64)
Start
Start
Start
Time Stop (9))
RCA
PCA Dim
PCh Dim
PCA Dim
PCA Dim
Figure 10: We visualize state-action embeddings from shallow (depth 4) and deep (depth 64) networks along successful trajectory in the Humanoid task. Near the goal, embeddings from the deep network expand across curved surface, while those from the shallow network form tight cluster. This suggests that deeper networks may devote greater representational capacity regions of the state space that are more frequently visited and play a more critical role in successful task completion.
Deeper Networks Enable Partial Experience Stitching Another key challenge in reinforce- ment learning learning policies that can gen- cralize to tasks unseen during training. To eval- uate this setting,
Train
Eval
designed a modified ver- sion of the Ant U-Maze eavironment. As shown in Figure 11 (top right), the original JaxGCRL benchmark assesses the agent's performance on the three farthest goal positions located the opposite side of the wall. However, instead of training on all possible subgoals (a superset of the evaluation state-goal pairs), we modified the
400
300
setup to train on start-goal pairs that are at most 3 units apart, ensuring that none of the evalua- tion pairs ever appear the training ure 1I demonstrates that depth 4 networks show limited generalization, solving only the easiest goal (4 units away from the start). Depth 16 networks achieve moderate success, while depth 64 networks excel, sometimes solving the most challenging goal position. These results suggest that the increasing network depth results in some degree of stitching, combining <:--nit pairs to navigate the 6-unit span of the U-Maze.
Time Goal
200
Fig-
Depth 4
Depth 16
Depth 64
Figure 11: Deeper networks exhibit improved gen- eralization. (Top left) We modify training setup of Ant U-Maze environment such that start-goal pairs are separated by S 3 units. This design guarantees that evaluation pairs (Top right) were encountered during training. testing the ability for combinatorial general- ization stitching. (Bottom) Generalization ability improves as network depth grows from 4 to 16 to 64 layers.
The (CRL) Algorithm is Key In Appendix A, we show that scaled CRL outperforms other baseline goal-conditioned algorithms and ad- vance the SOTA for goal-conditioned RL. We observe that for temporal difference methods
(SAC, SAC+HER, TD3+HER), the performance saturates for networks of depth 4, and there is either zero negative performance gains from deeper networks. This is in line with previous research showing that these methods benefit mainly from width (Lee et al., 2024; Nauman et al., 2024b). These results suggest that the self-supervised CRL algorithm is critical.
We also experiment with scaling more self- sepervueer algorithms, namely Goal-Conditioned Behav- ioral Cloning (GCBC) and Coal-Conditooned Supervised Learning (GCSL). While these methods yield zero success certain environments, they show some utility arm manipulation tasks. Inter- estingly, even very simple self-supervised algorithm like GCBC benefits from increased depth. This

---

points to a promising direction for future work of further investigating other self-supervised methods to uncover potentially different or complementary recipes for scaling self-supervised RL.
Finally, recent work has augmented goal-conditioned RL with quasimetric architectures, leveraging the fact that temporal distances satisfy triangle inequality-ban -aased invariance. In Appendix A, also investigate whether the depth scaling effect persists when applied to these quasimetric networks.
4.6 Does Depth Scaling Improve Offline Contrastive RI.?
In preliminary experiments, evaluated depth scaling in the offline goal-conditioned setting using OGBench (Park et al., 2024). We found little evidence that increasing the network depth of CRL improves performance in this offline set- ting. To further investigate this, we conducted ablations: (I) scaling critic depth while holding the actor at 4 or 8 layers, and (2) applying cold initialization to the final layers of the critic en-
Humanoid U Maze
Humanoid Big Maze
150
6.0
Time Goal
120
4.5
60-
3.0
1.5
coders (Zheng et al., 2024). In all cases, baseline depth 4 networks often had the highest success. A key direction for future work is to see if our method can be adapted to enable scaling in the offline setting.
0
40
80
120
25
50
75
160
Em step (M)
Eny step (M)
256
296
Depths:
1074
Figure 12:
Testing the limits of scale. We extend the results from Figure by scaling networks even fur- ther on the challenging Humanoid maze environments. We observe continued performance improvements with network depths of 256 and 1024 layers on Humanoid U- U-Maze. Note that for the 1024-layer networks, we ob- served the actor loss exploding at the onset of training. we maintained the actor depth at 512 while using 1024-layer networks only for the two critic encoders.
5 Conclusion
Arguably, much of the success of vision and language models today is due to the emergent capabilities they exhibit from scale (Srivastava al., 2023), leading to many systems reducing the RL problem to a vision or language problem. critical question for large Al models is: where does the data come from? Unlike supervised learning paradigms, RL methods inherently address this by jointly optimizing both the model and the data collection process through exploration. Ultimately, determining effective ways of building RL systems that demonstrate emergent capabilities may be important for transforming the field into one that trains its own large models. We believe that our work step towards these systems. By integrating key components for scaling up RL into single approach, we show that model performance consistently improves as scale increases in complex tasks. In addition, deep models exhibit qualitatively better behaviors which might be interpreted as implicitly acquired skills necessary to reach the goal.
Limitations. The primary limitations of our results are that scaling network depth comes at the cost of compute. An important direction for future work is to study how distributed training might be used leverage even more compute, and how techniques such as pruning and distillation might be used to decrease the computational costs.
Impact Statement This paper presents work whose goal is advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Acknowledgments. We gratefully acknowledge Nathaniel Chen, Galen Collier, and the full staff of Princeton Research Computing for their invaluable assistance. We also thank Colin Lu for his discussions and contributions to this work. This research was also partially supported by the National Science Centre, Poland (grant no. OD23/5//D/ST6/11// (60);; the Princeton Laboratory for Artificial Intelligence under Award 2025-97; and the Warsaw University of Technology through the Excellence Initiative: Research University (IDUB) program. Finally, we would also like to thank Jens Tuyls and Harshit Sikchi for providing helpful commends and feedback on the manuscript.
10

---

References
Ahn. M.. Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K.. Hausman.
K., Herzog, A., Ho, D., Hsu, J., Ibarz, J., Ichter, B., Irpan, A.. Jang, E., Ruano. R. J., Jeffrey, K., Jesmonth. S., Joshi, N., Julian, R. C., Kalashnikov, D., Kuang, Y., Lec, K.-H., Levine, S., Lu, Y., Luu, L.,, Parada, C., Pastor, P., Quiambao, J., Rao, K., Rettinghouse, J., Reyes, D., Sermanet, P., Sievers, N., Tan, C., Toshev, A.. Vanhoucke, V., Xia, F., Xiao, T., Xu, P., Xu, S., and Yan, M. (2022). Do as can, not as say: Grounding language in robotic affordances. Conference Robot Learning.
Andrychowicz. M.. Wolski, F., Ray, A., Schneider, J.. Fong, R.. Welinder, P., McGrew, B., Tobin, J., Pieter Abbeel. O., and Zaremba, (2017). Hindsight Experience Replay. In Neural Information Pro- cessing Systems, volume 30.
Ba, J. L., Kiros, J. R., and Hinton, G. (2016). Layer normalization. arXiv preprint arXiv: 1607.06450.
Blier, L.. Tallec. C., Ollivier, Y. (2021). Learning Successor States and Deal-Deeendent Values:
Mathematical Viewpoint.
Bortkiewicz, M., Palucki, W., Myers, V., Dziarmaga, T., Arezewski, T., Kuciñíski, ł., and Eysenbach, B. (2024).
Accelerating goal conlicondi rl algorithms and research. arXiv preprint arXiv:2408. 11052.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. (2021). Emerging properties in self-supervised vision transformers. arXiv preprint arXiv: 2104.14294
Chang. B., Meng, L., Haber, E., Tung, F., and Begert, D. (2018), Multi-level residual networks from dynamical systems view. In 6th International Conference Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net.
Chen, C., Zhang, J., Xu, Y. Chen, L.. Duan, J., Chen, Y., Tran, S. D., Zeng, B., and Chilimbi. T. (2022). Why do we need large batchsizes in contrastive learning? gradient- bias perspective. In Oh, A. H., Agarwal, A.. Belgrave, D., and Cho, K., editors, Advances in Neural Information Processing Systems.
Chen, T., Korbblith, S., Swersky, K., Norouzi. M., and Hinton, G. E. (2020). Big self-supervised models are strong semi-supervised learners. Advances in neural information processing systems, 33:22243-22255.
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A., Caron, M., Geirhos, R., Alabdulmohsin, M., Jenatton, R., Beyer, L.., Tschannen, M., Arnab, A.. Wang. X., Riquelme, C.. Minderer. M., Puigeerver, J.. Evei, U., Kumar, M., van Steenkiste, S., Elsayed, G. F,, Mahendran, A., Yu, F., Oliver, A., Huot, F. Bastings, Collier, M.. Gritsenko, A., Birodkar, V., Vasconcelos, C., Tay, Y., Mensink, T., Kolesnikov, A., Paveti'c, F., Tran, D., Kipf, T., Luvei'c, M., Zhai, X., Keysers, D., Harmsen, J., and Houlsby, N. (2023). Scaling vision transformers to 22 billion parameters. International Conference on Machine Learning.
Driess. D., Xia, F. Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong. Q. Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D.,. Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence, (2023). Palm-e: An embodied multimodal language model. International Conference on Machine Learning.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. (2024). The llma 3 herd of models. arXiv preprint arXiv:2407.2172
Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. (2018). Impala: Scalable distributed deep-rl rl with importance weighted actor-learner architectures. In International conference machine learning, pages 1407-1416. PMLR.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. (2024). Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning.
Eysenbach, B., Salakhutdinov, R., and Levine, (2021). C-Leaming: Learning to Achieve Goals via Recursive Classification. In International Conference on Learning Representations. arXiv.
Eysenbach, B., Zhang, T., Levine, S., and Salakhutdinov, R. R. (2022). Contrastive learning as goal-conditioned reinforcement learning. Advances in Neural Information Processing Systems, 35:35603-35620,
Farebrother, J., Orbay, J.. Vuong, Q. Taiga, A. A., Chebotar, Y., Xiao, T., Irpan, A., Levine, S., Castro, P. S., Faust, A., Kumar, A., and Agarwal, R. (2024). Stop Regressing: Training Value Functions via Classification for Scalable Deep RL.
11

---

Freeman, C. D., Frey, E., Raichuk, A., Girgin. S., Mordatch, I., and Bachem, O. (2021). Brax - a Differentiable Physics Engine for Large Scale Rigid Body Simulation. In NeurIPS Datasets and Benchmarks. arXiv.
Goyal, P., Mahajan, D., Cupta, A., and Misra, L. (2019), Scaling and benchmarking self- supervised visual representation learning. In Proceedings of the ieee/cvf International Conference computer vision, pages 6391-6400.
Gupta, A., Kumar, V. Lyneh, C., Levine, S., and Hausman, K. (2019). Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. Conference on Robot Learning.
He, K., Zhang, X., Ren, S., and Sun, J. (2015). Deep residual learning for image recognition. Computer Vision and Pattern Recognition.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Identity Mappings in Deep Residual Networks, pages 630-645. Springer International Publishing.
Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. (2018). Deep reinforcement learning that matters. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Huang. S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mchta, K., and Araújo, J. G. (2022). Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18.
Kaelbling, L. P. (1993). Learning to achieve goals. In ICCAI, volume 2, pages 1094-8. Citeseer.
Kumar, A., Agarwal, R., Geng, X., Tucker, G., Levine, S. (2023). Offline q-learning on diverse multi-task data both scales and generalizes. In The Eleventh International Conference on Learning Representations, ICIR 2023, Kigali, Rwanda, May 1-5, 20233 OpenReview.net.
LeCun, Y. (2016). Predictive learning. Invited talk at the 30th Conference on Neural Information Processing Systems (NIPS). Barcelona, Spain.
Lee, H., Hwang, D., Kim, D., Kim, H., Tai, J. Subramanian, K., Wurman, R., Choo, J., Stone, P., and Seno, T. (2024). SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning.
Lee, K.-H., Nachum, O., Yang, M., Lee, L.., Freeman, D., Xu, W., Guadarrama, S., Fischer, I., Jang, E., Michalewski, H., and Mordatch, (2022). Multi-Game Decision Transformers.
Liu, B., Feng. Y., Liu, Q., and Stone, P. (2023). Metric Residual Networks for Sample Efficient Goal Conl-Conditioned Reinforcement Learning.
Liu, H., Li, C., Wu, Q., and Lec, Y. J. (2024). Visual instruction tuning. Advances in neural information processing systems, 36.
Lyle, C., Rowland, M., and Dabney, W. (2022). Understanding and preventing capacity loss in reinforcement learning. arXiv preprint arXiv:224..09: 205.00204409000.
Lyle, C., Zheng, Z., Khetarpal, K., van Hasselt, H., Pascanu, R., Martens, J., and Dabney, W. (2024). Disentan- gling the causes of plasticity loss in neural networks. arXiv preprint 240.2402.18722
Makoviychuk, V., Wawrzyniak, L., Guo, Y., Lu, M., Storey, K., Macklin, M., Hoeller, D., Rudin, N., Allshire, A., Handa, A., et al. (2021). Isaac gym: High performance gpu-based physies simulation for robot learning. arXiv preprint arXiv:2108. 10470.
Myers, V., Zheng, C., Dragan, A., Levine, S., and Eysenbach, B. (2024). Learning temporal distances: Contrastive successor features can provide a metric structure for dacision-making. International Conference on Machine Learning.
Nauman, M., Bortkiewicz, M., Milos, P. Trzcinski, T., Ostaszewski, M., and Cygan, M. (2024a). Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In Forty-first International Conference on Machine Learning. ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net.
Nauman, M., Ostaszewski, M., Jankowski, K., Mitoš, P., and Cygan, M. (2024b). Bigger, Regularized, Optimistic: Scaling for compute and sample-efficient continuous control.
Neumann, O. and Gros, C. (2022). Scaling laws for a multi-agent reinforcement learning model, arXiv preprint arXiv:2210.00849
Obando- Cerndo-Ceron, J., Bellemare, M. G., and Castro, P. S. (2023). Small batch deep reinforcement learning. Neural Information Processing Systems. Published at NeurlPS 2023.
12

---

Obando-Ceron, J., Sokar, G., Willi, T., Lyle, C., Farebrother. J., Foerster, J. Dziugaite, G., Precup, D., and Castro, P. S. (2024). Mixtures of experts unlock parameter scaling for deep rl. International Conference on Machine Learning.
Ota, K., Jha, D. K., and Kanezaki, A. (2021). Training larger networks for deep reinforcement learning. arXiv preprint 1Xxv:2102.07920.
Park, S., Frans, K., Eysenbach, B., and Levine, S. (2024). Oghench: Benchmarking offline goal-conditioned rl. arXiv preprint arXiv: 2400.20092.
Radford, A. (2018). Improving language understanding by generative pre-training.
Radford, A., Kim, J. W., Hallacy, C.,, Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, (2021). Learning transferable visual models from natural language supervision. International Conference on Machine Learning.
Raffin, A., Hill, A., Glcave, A., Kanervisto, A., Ernestus, M., and Dormann, N. (2021). bsable-baselines:: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1-8.
Ramachandran, P.. Zoph, B., and Le. Q. V. (2018), Searching for activation functions. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April May 3. 2018. Workshop Track Proceedings. OpenReview.net.
Rudin, N., Hoeller, D., Reist, P., and Hutter, M. (2022). Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on Robot Learning, pages 91-100. PMLR.
Rutherford, A., Ellis, B., Gallici, M., Cook, J., Lupu, A., Ingvarsson, G., Willi, T., Khan, A., de Witt, C. S., Souly, A., al. (2023). Jaxmarl: Multi-agent rl environments and algorithms in jax. arXiv preprint arXiv:2311.10090.
Schwarzer, M., Obando-Cema J. S., Courville, A. C., Bellemare, M. G., Agarwal, R., and Castro, P. (2023). Bigger, better, faster: Human-level atari with human-level efficiency. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J., editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honoluda, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 30365 303800 PMLR.
Sohn, K. (2016). Improved Deep Metric Learning With Multi-Class N-Pair Loss Objective. Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Srivastava, A., Rastogi, A., Rao, A., et al. (2023). Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Trans. Mach. Learn. Res.
Team, A. A., Bauer, J.. Baumli, K., Baveja, S., Behbahani, F., Bhoopchand, A., Bradley-Schmile N., Chang, M.. Clay, N., Collister, A., et al. (2023). preprint urXiv:2301.07600
adaptation in an open-ended task space. arXiv
Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A Physics Engine for Model-Based Control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, IEBE.
Torgo, L. and Gama, J. (1996). Regression by classification. In Advances in Artificial Intelligence: 13th Brazilian Symposium on Artificial Intelligence, SBIA '96 Curitiba, Brozil, October 23-25, 1996 Proceedings 13, pages 51-60. Springer.
Tuyls, J., Madeka, D., Torkkola, K., Foster, D., Narasimhan, K., and Kakade, S. (2024). Scaling Laws for Imitation Learning in Single-Agent Games.
Van Hasselt, H., Doron, Y., Strub, F., Hessel, M., Sonnerat, N., and Modayil, J. (2018). Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.0264.
Vaswani. A., Shazeer, N. M.. Parmar, N., Uszkoreit, J., Jones, L.. Gomez, A. N., Kaiser, L., and Polosukhin, I. (2017). Attention all you need. nips.
Veit, A., Wilber, M., and Belongic, S. (2016). Residual networks behave like ensembles of relatively shallow networks. arXiv preprint arXiv: 1605.06431.
Wang, T., Torralba, A., Isola, P., and Zhang, A. (2023a). Optimal goal coal-reaching reinforcement learning via quasimetric learning.
13

---

Wang. T., Torralba, A., Isola, P., and Zhang, A. (2023b). Optimal goal-reaching reinforcement learning via quasimetric learning. In Krause, A., Brunskill, E.,, Cho, K.. Engelhardt, B., Sabato, S., and Scarleu, J., editors, International Conference un Machine Learning, JCML 2023, 23-29 July 2023, Honolddu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 36411 36430. PMLR.
Wei, J.. Tay, Y., Bommasani, R., Raffel, Zoph, B., Borgeaud. Yogatama. D., Bosma, M., Zhou, D..
Metzler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. (2022). Emergent abilitics of large language models. Trans. Mach. Learn. Res.
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. (2021). Sealing vision transformers. Computer Vision and Pattern Recognition.
Zhang, H., Morwani, D.. Vyas, N., Wu, J., Zou, D., Ghai, U., Foster, D., and Kakade, (2024). How does critical batch size scale in pre-training? arXiv preprint arXiv: 2410.21676.
Zheng. C.. Eysenbach, B., Walke, H., Yin, P., Fang. K., Salakhutdinov, R., and Levine, S. (2024). Stabi- lizing Contrastive RL: Techniques for Offline Goal Reaching. In International Conference on Learning Representations. Xiv.
Zheng, C., Salakhutdinov. R., and Eysenbach, B. (2023). Contrastive Difference Predictive Coding. In Twelfth International Conference on Learning Representations. arXiv.
Zong Y., Aodha, O. M., and Hospedales, T. (2024). Self-supervised multimodal learning: A survey.
14

---

A Additional Experiments
A.1
Scaled CRL Outperforms All Other Baselines 8 out of 10 Environments
Humanoid
Ant Big Maze
Ant U4-Maze
Ant US-Maze
Ant Hardest Maze
302
Time Goal
620
400
-20
403
430
200
330
150
60
1.0
100
Arm Push Easy
Arm Push Hard
Arm Binpick Hard
Himanood U-Maze
Humanoid Big Maze
252
253
202
Time at GGal
0.0
240
153
320
200
102
102
100
52
30
35
260
(N)
CRL
GCSL GCBC
TD3 HER
SAC + HER
Scaled CRL
Figure 12: Scaled CRL (Ours) outperforms baselines CRL (original), SAC, SAC+HER, TD3+HER. GCSL, and GCBC in 8 out 10 environments.
In Figure 1, we demonstrated that increasing the depth of the CRL algorithm leads to significant performance improvements over the original CRL (see also Table 1). Here, we show that these gains translate state-of-the-art results in online goal-conditioned RL, with Scaled CRL outperforming both standard TD-based methods such as SAC, SAC+HER, and TD3+HER, as well as self-supervised imitation-based approaches like GCBC and GCSL.
A.2
The CRL Algorithm is Key: Depth Scaling is Not Effective on Other Baselines
Next, we investigate whether increasing network depth in the baseline algorithms yields similar performance improvements as observed in CRL. We find that SAC, SAC+HER, and TD3+HER do benefit from depths beyond four layers, which consistent with prior findings (Lee al., 2024; Nauman et al., 2024b). Additionally, GCSL and GCBC fail to achieve any meaningful performance on the Humanoid and Ant Big Maze tasks. Interestingly, we do observe one exception, as GCBC exhibits improved performance with increased depth in the Arm Push Easy environment.
Table 1:
Increasing network depth (depth D = - 64) increases performance on CRL (Figure 1). Scaling depth exhibits the greatest benefits tasks with the largest observation dimension (Dim).
Task
Dim
D 4
D 64
Impry.
Arm Binpick Hard
14
219 115
5.7x
Arm Push Easy
17
308 133
762 1:30
2.5x
Push Hard
171 :11
113
2.4x
Ant U4-Maze
11.4 #4.1
+36
25x
Ant US-Maze
0.97 10.7
61 118
63x
29
Ant Big Maze
=20
+25
7.3x
Ant Hardest Maze
215 #:
387 +21
1.8x
Humanoid
12.6
6449
52x
Humanoid U-Maze
268
3.2 #1.2
159 +:3
50x
Humanoid Big Maze
0.06
59
1051x
15

---

Scaled CRL (Ours)
Humanoid
Ant Big Maze
Arm Push Easy
625
780
762
600
592
600
200
400
123
424
400
200
16
12
8
16
32
64
16
32
64
800
CCC
400
200
200
200
12.5
65.1
59
16
32
64
8
16
32
64
4
166
32
64
800
800
600
CC92
600
200
0.341
0.19
0.136
0,515
0.312
200
0
200-
71.0
223
26
64
HER
800
800
800
600
306
376
200
100
16
800
000
600
000-
330
2.5
309
401
401
SAC HER
600
600
400
314
400
400
472
350
6.0
401
400
32
16
32
64
16
64
Depth
Figure 13: Depth scaling yields limited gains for SAC, SAC+HER, TD3+HER, GCSL, and GCBC.
A.3 Additional Scaling Experiments: Offline GCBC, BC, and QRL
We further investigate several additional scaling experiments. As shown in Figure 14, our approach successfully scales with depth in the offline GCBC setting on the unlmaze-medium---tinn OGBench. We find that the combination of layer normalization, residual connections, and Swish activations is critical, suggesting that our architectural choices may be applied to unlock depth scaling in other algorithms and settings. We also attempt scale depth for behavioral cloning and the QRL (Wang et al., 2023a) algorithm-aa in both of these cases, however, we observe negative results.
task from
GCBC
BC
QRL (Online)
QRL (Offline)
0.8
0.666
Success
0.508
0,5Rt
0.474
0.450
0.4
0.260
0.2
0.088
0.123
0.180
0.200
0.180
2
4
16
32
2
16
32
2
8
16
32
2
8
16
32
Depth
Our approach successfully scales depth in offline GCBC antmaze-meddum--stice (OGBench). In contrast, scaling depth for BC caaimaaeegiaan-naiiiei expert SAC data) and for both online (FetchPush) and offline QRL (pointmaze- caaattmaze-giant--a stitch, OGBench) yield negative results.
Figure 14:
16

---

A.4 Can Depth Scaling also be Effective for Quasimetric Architectures?
Prior work (Wang et al., 2023b; Liu et al., 2023) has found that temporal distances satisfy an important invariance property, suggesting the use of quasimetrie architectures when learning temporal distances. Our next experiment tests whether changing the architecture affects the scaling properties of self- supervised RI.. Specifically, backward NCE loss with MRN representations. The results indicate that scaling benefits are not limited to single neural network parametrization. However, MRN's poor performance the Ant US-Maze task suggests further innovation is needed for consistent scaling with quasimetric models.
the CMDD1 algorithm (Myers al., 2024), which employs
Humanaid
Ant Big Maze
Ant U4-Maze
Ant US-Maze
Ant Hardest Maze
150
50
322
Time at Goal
75
165
10
200
Arm Push Easy
Arm Push Hard
Arm Binpick Hard
Humanoid U-Maze
Humanoid Big Maze
250
36
16
320
Time at GGal
630
230
Enn
Depth
Depth
Depth
Depth
Dopth
Figure 15: Performance of depth scaling on CRL augmented with quasimetric architectures (CMD-1).
A.5 Additional Architectural Ablations: Layer Norm and Swish Activation
We conduct ablation experiments validate the architectural choices of layer norm and swish activa- tion. Figure 16 shows that removing layer normalization performs significantly worse. Additionally, scaling with ReLU significantly hampers scalability. These results, along with Figure 5 show that all of our architectural components residual connections, layer norm, and swish activations are jointly essential to unlocking the full performance of depth scaling.
Layer Norm Ablation
Swish vs. ReLU Ablation
700
With Norm
700
Swish
Without Layer Norm
ReLU
600
600
Time at Goal
500
500
4
16
32
64
16
32
64
Depth
Depth
Figure 16: (Left) Layer Norm is essential for scaling depth. (Right) Scaling with ReLU activations leads to worse performance compared to Swish activations.
17

---

A.6 Can We Integrate Novel Architectural Innovations from the Emerging RL Scaling Literature?
Recently, Simba-v2 proposed architecture for scalable RL. Its key innovation is the replacement of layer normalization with hyperspherical normalization, which projects network weights onto the unit-norm hypersphere after each gradient update. As shown, the same depth-scaling trends hold when adding hyperspherical normalization to our architecture, and it further improves the sample efficiency of depth scaling. This demonstrates that our method can naturally incorporate new architectural innovations emerging in the RL scaling literature.
Table 2: Integrating hyperspherical normalization in architecture enhances the sample efficiency of depth scaling.
Steps to reach 2200 success
Steps reach 2 400 success
Steps to reach 2600 success
Depth 4
16
32
Depth
4 16
32
Depth
4
16
32
With
50
42
With
62
48
With
77
67
Without
64
54
Without
75
64
Without
77
A.7 Residuals Norms in Deep Networks
Prior work has noted decreasing residual activation norms in deeper layers (Chang et al., 2018). We investigate whether this pattern also holds in our setting. For the critic, the trend is generally evident, especially in very deep architectures (e.g., depth 256). The effect is not as pronounced in the actor.
Average Residual Magnitudes (L2 Norm)
blocks)
Enc. blocks)
Depth 32
0.3
G Enc. (8 blacks)
0.2
2
0.4
Actor (16 blocks)
SA Enc. (16 blocks)
Depth 64
0.2
0.1
2
10
12
16
0.4
Actor (32 blocks! Enc.
Depth 128
---
Bnc. (32 blocks)
0.2
0.1
Depth 256
SA Enc. (64 blacksi (64 blocks)
10
20
30
10
50
60
Residual Block Index
Figure 17: 1.2 norms of residual activations in networks with depths of 32, 64, 128, and 256.
18

---

A.8 Scaling Depth for Offline Goal-conditioned RL
antmaazeggiantn plainazzegaann-naiiatte--- antmaaz-meddiu--ttichhe
Success
33-
Ora Steps
Eny Sties
Depth
Depth
Depth 16
Depth 32
Depth
Figure 18: To evaluate the scalability of method in the offline setting, we scaled model depth OGBench (Park et al., 2024). In two out of three environments, performance drastically declined depth scaled from 4 to 64, while slight improvement Successfully adapting our method to scale offline GCRL is an important direction for future work.
untmaze-medduu--sticch--e
B Experimental Details
B.1 Environment Setup and Hyperparameters
Figure 19: The scaling results of this paper are demonstrated on the JaxGCRL benchmark, showing that they replicate across diverse range of locomotion, navigation, and manipulation tasks. These tasks are set the online goal-conditioned setting where there are no auxiliary rewards or demonstra- tions. Figure taken from (Bortkiewicz al., 2024).
Our experiments use the JaxGCRL suite of GPU-accelerated environments, visualized in Figure 19, and a contrastive RL algorithm with hyperparameters reported in Table 7. In particular, we 10 environments, namely: ant_big_maze maze, ant ant_hardest__me maaze, arm_binphl hrm__bnnick_haat arm_ aar__pus___ass_ easy, arm_push hard, humanoid, humanoid_bie bigiiidd_igaaaeeeee_eeea maze, humanoid_u u__ u_maze, ant_u4 @tt_u4___e maze, ant_u5_mazz
B.2 Python Environment Differences
In all plots presented in the paper, we used MJX 3.2.6 and Brax 0.10.1 to ensure fair and consistent comparison. During development, we noticed discrepancies in physics behavior between the environ- ment versions we employed (the CleanRL version of JaxGCRL) and the version recommended in more recent commit of JaxGCRL (Bortkiewicz et al., 2024). Upon examination, the performance differences (shown in Figure 20) stem from a difference in versions the MJX and Brax packages. Nonetheless, in both sets of MJX and Brax versions, performance scales monotonically with depth.
19

---

MJX=3.2.3. 0 0
MIX=3.2.6, Brax=0.10.1 (ours)
600
800
500
500
200
300
200
100
Steps (M4)
Steps (H)
Depth 4
Depth 16
Depth 32
Deprh 54
Figure 20: Scaling behavior for humanoid in two different python environments: MJX=3.2.3, Brax=0.10.5 and MJX=3.2.6, Brax=0.10.1 (ours) version JaxGCRL. Scaling depth improves the performance significantly for both versions. In the environment we used, training requires fewer environment steps to reach marginally better performance than in other Python environment.
B.3 Wall-clock Time of Our Approach
We report the wall-clock time of our approach in Table 3. The table shows results for depths of 4, 8, 16, 32, and 64 across all ten environments, and for the Humanoid U-Maze environment, scaling up to 1024 layers. Overall, wall-clock time increases approximately linearly with depth beyond a certain point.
Table 3: Wall-clock time (in hours) for Depth 4, 8, 16, 32, and 64 across all 10 environments.
Environment
Depth 4
Depth 8
Depth 16
Depth 32
Depth 64
Humanoid
1.48 + 0.00
2.13 + 0.01
3.40 + 0.01
5.92 + 0.01
10.99 0.01
Ant Big Maze
2.12 0.00
2.77 + 0.00
4.04 0.01
6.57 0.02
11.66 + 0.03
Ant U4-Maze
1.98 0.27
2.54
3.01
6.01
11.43 0.03
Ant US-Maze
9.46 1.75 1.75
10.99
16.09
1.34
46.40 1.10
Ant Hardest Maze
5.11 1 0.00
6.39 0.00
8.94
13.97
23,96 1:96:00 0,06
Arm Push Fasy
9.97 + 1.03
11.02 1.29
+
1.96 14.94
19.52 1.97
Arm Push Hard
9.74 + 1.05
10.55 + 1.20
11.98
14.40 1.64
18.53 0.06
Arm Binpick Hard
18.41 1.11
17.48 1.88
19.47 0.05
21.91
29.64 6.10
Humanoid U-Maze
8.72
11.29 0.01
16.36 0.03
26.48
46.74 0.05
Humanoid Big Maze
12.45 1.45:100
15.02 0.01
20.34 0.01
30.61 1 0.05
50.33 0.05
Table 4: Total wall-clock time (in hours) for training from Depth 4 up to Depth 1024 in the Humanoid U-Maze environment.
Depth
Time (h)
4
3,23 + 0.001
8
4.19 + 0.003
16
6.07 = 0.003
32
9.83 1 0.006
64
17.33 = 0.003
128
32.67 = 0.124
256
73.83 + 2.364
512
120.88 2.177
1024
134.15 + 0.081
20

---

B.4 Wall-clock Time: Comparison to Baselines
Since the baselines use standard sized networks, naturally our scaled approach incurs higher raw wall-clock time per environment step (Table 5). However, more practical metric is the time required reach given performance level. As shown in Table 6, our approach outperforms the strongest baseline, SAC, in of 10 environments while requiring less wall-clock time.
Table 5: Wall-clock training time comparison of our method vs. baselines across all 10 environments.
Environment
Scaled CRL
SAC
SAC+HER
TD3
GCSL,
GCBC
Humanoid
11.0 1 0.0
0,5 1 0.0
0,6 0.0
0.8 0.0
0,4 1: 0.0
0.6 1 0.0
Ant Big Maze
11.7 + 0.0
1.6
1.6 0.0
1.7
1.3
1.4 0.1
Ant U4-Maze
11.4 0.0
1.2
1.3 0.0
=.3
0.7
1.1
Ant US-Maze
4.1
5.7
6.1=00
6.2 0.0
2.1
0.5
Ant Hardest Maze
0.0
4.3
4.5 0.0
0.0
2.1
4.5
Arm Push Easy
0.6
8.3
8,5 0.0
8.4
6.1
0.3
Arm Push Hard
18.5
8.5
8.6-00 0.0
8.3-0.1
5.2
7.4
Arm Binpick Hard
29.6
2..7+
2.0
0.34
8.0 0.9
16.2
Humanoid U-Maze
46.7 0.0
3.0
3.5
5.4 0.0
3.1
0.2
Humanoid Big Maze
50,3
0.6
9.3 0.0
7.5 1.1
3.1000
0.0
11.4
Table Wall-clock time (in hours) for our approach surpass SAC's final performance. As shown, our approach surpasses SAC performance in less wall-clock time in 7 out of 10 environments. The N/A* $ entries are because in those environments, scaled CRL doesn't outperform SAC.
Environment
SAC
Scaled CRL (Depth 64)
Humanoid
0.46
6.37
Ant Big Maze
1.55
0.00
Ant U4-Maze
1,16
0.00
Ant US-Maze
5.73
0.00
Ant Hardest Maze
4.33
0.45
Arm Push Easy
8.32
1.91
Arm Push Hard
8.50
6.65
Arm Binpick Hard
20.70
4.43
Humanoid U-Maze
3.04
N/A
Humanoid Big Maze
8.55
N/A*
Table 7: Hyperparameters
Hyperparameter
Value
num_timee tim_timesteps
100M-400M (varying across tasks)
update-to-data (UTD) ratio
1:40
max_replay_size
10,000
min_replay_size
1,000
episode_lengee
1,000
discounting
0.99
num_envs
512
batch_size
512
policy_lr
3e-4
critic_lr
3e-4
contrastive__oss_fuuree
onf
energy_funche
L2
logsumexp_penal..
Network depth
depends the experiment depends on the experiment
Network width
representation dimension
64
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract contains 3 main claims: (1) Depth scaled to 1024 layers; (2) Performance increases 2-50x on CRL and outperforms other goal-conditioned baselines. (3)
These performance gains leads to qualitatively new learned behaviors. Each of these claims are clearly substantiated in the main text in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We included a Limitations section that describes the main limitation of our paper, which latency of deep networks. We also multiple times the paper demarcated where our research can be extended by future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
22

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This is an empirical paper. As such, no theoretical results that require assump- tions proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, documentation for reproducing the experiments is included alongside the anonymous code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
See link to anonymous code in Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (attps://nips.com cc/pu
blic/ Blic/guide///ee Seridde/Cooeeuumiii (wwodeSummsssiooooooooooooooo ssi aasionnooicyy for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //n /nips. mnpps.c/ppii wwwplccgguideeeeeeeppppeppbpeeee /tt.ggguuddeeeoee/biiiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Experiments section and Appendix Experimental Details
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Error bars in figures depict one standard error across random seeds. We used seeds in Figure 1. For other figures the main text, we could only run 3 seeds because of computational constraints.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute resources are detailed in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that NeurlPS Code of Ethics https: : /ttp///euipsp//eeeeee http//ubic///hcccccceeeeeee.eeee
didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the
Answer: [Yes]
Justification:
No known violations of the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The Conclusion notes that there are no immediately socictal impacts of the work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
No immediate impact to high-risk applications.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Benchmarks used are appropriately cited in the main text.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com camddataeess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Datasets and benchmark used are all from prior work and appropriately cited. Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subject experiments
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not used in writing the paper, and were only used for occasional code debugging.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy pestps://neuripr: wttp:///eeeeeeene/////////////// for what should or should not be described.
28


Reviews:
[{"review_id": "O2mftc4re2", "reviewer": "Reviewer_gFHa", "summary": "This paper investigates scaling network depth in self-supervised reinforcement learning, specifically using the Contrastive RL (CRL) algorithm for goal-conditioned tasks. The authors demonstrate that increasing network depth up to 1024 layers can yield substantial performance improvements in unsupervised goal-reaching tasks. Their approach combines three key components: (1) self-supervised contrastive RL that learns without demonstrations or rewards, (2) GPU-accelerated training frameworks for increased data throughput, and (3) very deep residual networks with architectural stabilization techniques. The experiments are conducted on locomotion, navigation, and manipulation tasks from the JaxGCRL benchmark, showing 2-50× performance improvements over shallow networks typically used in RL (2-5 layers). The paper provides extensive analysis of why depth scaling works, including improved exploration, better representational capacity, and emergent qualitatively different behaviors at critical depth thresholds.", "questions": "1. Can you provide any intutive explanation to why the scaling benefits appear specific to CRL?\n2. Could you provide detailed computational cost analysis comparing depth vs. width scaling? What are the FLOPS, memory, and wall-clock time trade-offs? How does this inform practical deployment decisions?\n3. In Figure 1, the performance of these approaches increases with depth, and then seems to decrease when depth is increased further. What is the reason for such behavior? Is this pattern consistent with further depth experients? If yes, do you have any thoughts on finding the critical depth for particular tasks?\n4. In line 173, the figure that compares proposed approach with SAC, SAC+HER, TD3+HER, and CRL seems to be missing?", "limitations": "Please refer to the weaknesses section.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper provides a clearly-written comprehensive study showing that depth scaling (up to 1024 layers) can dramatically improve RL performance, contrary to conventional wisdom.\n2. The paper shows strong empirical evidence, by demonstrating consistent improvements across diverse tasks with proper statistical reporting (error bars across multiple seeds), across 10 diverse environments spanning locomotion, navigation, and manipulation.\n3. I appreciate the thorough analysis through visualization of Q-functions, learned representations, and emergent behaviors (wall vaulting, creative navigation). \n4. The approach provides effective adaptation of residual connections, layer normalization, and Swish activation for very deep RL networks.\n5. Good set of ablations are provided: width vs. depth, actor vs. critic scaling, batch size effects, and generalization capabilities.\n\nWeaknesses:\n1. It seems that the scaling benefits appear specific to CRL. The traditional TD methods (SAC, TD3) show no improvement or even show degradation with depth.\n2. The preliminary experiments show no benefits in offline settings, limiting applicability.\n3. The experiments are specific to unsupervised RL setting. It is unclear if the benefits may transfer to reward-rich (dense) or imitation learning settings.\n4. The paper would benefit from detailed analysis of computational overhead compared to baselines.\n5. As acknowledged by the authors, the approach requires substantial computational resources that may limit reproducibility.", "was_revised": true, "final_justification": "The authors' response has addressed all my concerns. The authors properly answered all the raised questions, and also provided the required resource information in the form of four tables, which satisfied my initial concerns. Therefore, I will my high judgement score for the paper.", "rebuttal": "We thank Reviewer gFHA for the helpful feedback, questions, and suggestions.\n\n> **Question 1: Can you provide any intuitive explanation to why the scaling benefits appear specific to CRL?**\n\nSure! The key intuitive reason for scaling benefits specific to CRL lies in how it redefines the core learning challenge of RL. Traditional RL faces several well-known scalability challenges: regression to noisy and recursively bootstrapped Q-value targets, gradient instability, plasticity loss, amplifying overestimation biases, etc. In contrast, CRL fundamentally shifts the “heavy lifting” of the RL learning problem from recursively estimating fluctuating Q-values to addressing a simple classification problem: is the sampled future state likely to be along the same trajectory or along a different trajectory as my current state? This transforms the unstable RL problem into a supervised learning classification task based on cross-entropy loss—a setup known to scale exceptionally well in deep learning (and recently also in RL, see Farebrother et al., 2024 [4]). Once CRL scalably learns good contrastive representations through this supervised objective, the remaining learning burden on the actor becomes straightforward: at any given state, simply greedily choose the action that yields the representation most similar to the goal state.\n\n> **Question 2: Could you provide detailed computational cost analysis comparing depth vs. width scaling? What are the FLOPS, memory, and wall-clock time trade-offs? How does this inform practical deployment decisions?**\n\nIn **Table 1a**, we show that width and depth empirically scale similarly in terms of wall-clock time. For practical deployment, the relevant metric is wall-clock time to reach a given performance. **Table 1b** shows that scaling depth surpasses the performance of scaling width in less wall-clock time in all three environments. In terms of FLOPs and memory, width scales quadratically while depth scales linearly, making depth more efficient in these metrics as well. Thus, for practical deployment we recommend scaling depth in almost all cases.\n\n> **Question 3: In Figure 1, the performance of these approaches increases with depth, and then seems to decrease when depth is increased further. What is the reason for such behavior? Do you have any thoughts on finding the critical depth for particular tasks?**\n\nEven with techniques like residual connections and layer norm, training very deep networks is known to be tricky—exploding gradients, representation staleness, plasticity loss, etc. This is likely why many RL scaling efforts show results on width scaling but not depth. In our experiments, for tasks where degradation does occur, performance has already scaled to near-optimal. Thus, we hypothesize that additional depth yields diminishing returns while sometimes increasing the risk of instability. In these cases, to answer the question on identifying critical depth:  \n- **Start with 32 layers**: Across all 10 environments, we consistently observe stable and improving performance up to depth 32.  \n- **Scale deeper if needed**: For more complex tasks, especially those involving locomotion or high-dimensional control (e.g., Humanoid environments), we observe stable scaling up to 1000 layers.  \n- **Be conservative with manipulation tasks**: These tasks tend to be more sensitive to over-scaling and performance degradation at higher depths.\n\n> **Question 4: In line 173, the figure seems to be missing?**\n\nThis Figure is in the Appendix (Figure 10). Thanks for catching the undefined reference.\n\n> **Weakness 1: It seems that the scaling benefits appear specific to CRL. The traditional TD methods (SAC, TD3) show no improvement or even show degradation with depth.**\n\nThis is a known result from prior work that traditional TD methods do not scale well with depth [5, 2, 3]. As such, rather than viewing this as a limitation, we see it as a point of strength/novelty of our work: our results demonstrate that the benefits of scaling depth can indeed be unlocked in RL. Previous efforts to scale RL have largely focused on network width [5,2], with depth scaling either limited (e.g., up to 8 residual blocks in [3]) or harmful [5, 2]. Our approach unlocks the ability to scale depth, yielding performance improvements greater than width alone.\n\n> **Weakness 2: The preliminary experiments show no benefits in offline settings, limiting applicability.**\n\nAlthough we found negative results for scaling CRL in the offline setting, we now report new experiments demonstrating that our approach scales effectively to another widely used offline algorithm, GCBC. In **Table 2a**, we find that GCBC successfully scales with depth on the antmaze-medium-stitch task in OGBench [1] when using our architecture of layer normalization, residual connections, and swish activations. In **Table 2b**, we show that using our architecture is critical to enabling scaling in GCBC, as scaling the standard architecture degrades performance. These results suggest that our approach may be adapted to scaling the offline RL setting. Future work could further investigate what algorithms and architectural components best enable depth scaling in offline RL.\n\n> **Weakness 3: The experiments are specific to unsupervised RL setting. It is unclear if the benefits may transfer to reward-rich (dense) or imitation learning settings.**\n\nCRL is an unsupervised RL algorithm; as such, we focus on this setting as the primary scope of our paper. One potentially interesting direction for future research: although CRL itself is unsupervised, its self-supervised contrastive representations can be learned on any set of collected trajectories, including those of a reward-based RL algorithm. Prior work has explored integrating such representations via, for example, auxiliary losses or planning [6]. Our analysis of visualized Q-functions and learned representations give confidence that increasingly sophisticated representations can be learned through depth scaling. We encourage future work to investigate whether these scaling benefits can thus be integrated into reward-based RL.\n\n> **Weakness 4: The paper would benefit from detailed analysis of computational overhead compared to baselines.**\n\nBecause the baselines use standard-size networks, our scaled approach naturally incurs higher wall-clock time per environment step at a raw computation level (**Table 3a**). However, as with the width-vs-depth analysis, the more meaningful metric is the total wall-clock time required to reach a given performance level. To evaluate this, we compare our method with the strongest baseline, SAC. As shown in **Table 3b**, our approach surpasses SAC performance in less wall-clock time in 7 out of 10 environments.\n\n> **Weakness 5: The approach requires substantial computational resources that may limit reproducibility.**\n\nIn fact, all of our experiments, including those with 1000 layer networks, can be run on a single 80GB A100! \n\n### [TABLE 1a] Wall-clock Time Comparison of Depth vs. Width Scaling\n|Width 256|Width 512|Width 1024|Width 2048|Width 4096|\n|---|---|---|---|---|\n|4.52 ± 0.34|5.32 ± 0.01|5.96 ± 0.01|8.39 ± 0.02|15.90 ± 0.64|\n\n|Depth 4|Depth 8|Depth 16|Depth 32|Depth 64|\n|-|-|-|-|-|\n|4.52 ± 0.34|5.31 ± 0.43|6.55 ± 0.48|9.14 ± 0.65|14.06 ± 0.66|\n\n### [TABLE 1b] Scaling Depth Surpasses Performance of Scaling Width in Less Wall-clock time\n|Environment|Width = 4096|Depth = 64|\n|-|-|-|\n|Humanoid|22.30 ± 0.08|**5.66 ± 0.24**|\n|Ant Big Maze|11.70 ± 0.03|**4.04 ± 0.41**|\n|Arm Push Easy|13.69 ± 1.92|**2.38 ± 0.18**|\n\n### [TABLE 2a] Successful Depth Scaling on GCBC (Antmaze-medium-stitch, OGBench)\n|Depth|2|4|8|16|32|\n|-|-|-|-|-|-|\n|Performance|0.474 ± 0.11|0.572 ± 0.04|0.574 ± 0.02|0.588 ± 0.00|0.634 ± 0.05|\n\n### [TABLE 2b] Our Architecture is Critical: Depth Scaling on GCBC Fails with Standard Architecture\n|Depth|2|4|8|16|32|\n|-|-|-|-|-|-|\n|Performance|0.532 ± 0.09|0.746 ± 0.01|0.692 ± 0.02|0.698 ± 0.02|0.210 ± 0.21|\n\n### [TABLE 3a] Wall-clock Time Comparison of Our Method vs. Baselines\n|Method|Humanoid|Ant Big Maze|Ant U4-Maze|Ant U5-Maze|Ant Hardest Maze|Arm Push Easy|Arm Push Hard|Arm Binpick Hard|Humanoid U-Maze|Humanoid Big Maze|\n|-|-|-|-|-|-|-|-|-|-|-|\n|Scaled CRL (Depth 64)|10.99|11.66|11.43|46.40|23.96|19.52|18.53|29.64|46.74|50.33|\n|SAC|0.46|1.55|1.16|5.73|4.33|8.32|8.49|20.70|3.04|8.55|\n|SAC+HER|0.55|1.63|1.26|6.10|4.51|8.54|8.60|20.74|3.48|9.27|\n|TD3|0.79|1.68|1.30|6.18|5.04|8.41|8.31|18.40|5.35|7.50|\n|GCSL|0.42|1.48|0.70|2.81|2.13|6.37|5.22|7.95|3.07|5.08|\n|GCBC|0.61|1.40|1.11|5.56|4.38|8.28|7.36|16.24|7.18|11.37|\n\n### [TABLE 3b] CRL Surpasses SAC Performance in Less Wall-clock Time in 7 of 10 Environments\n|Algorithm|Humanoid|Ant Big Maze|Ant U4-Maze|Ant U5-Maze|Ant Hardest Maze|Arm Push Easy|Arm Push Hard|Arm Binpick Hard|Humanoid U-Maze|Humanoid Big Maze|\n|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|\n|**SAC**|**27.8**|92.7|69.8|343.8|259.8|499.2|509.7|1242.1|**182.6**|**512.9**|\n|**Scaled CRL (Depth 64)**|382.2|**0.0**|**0.0**|**0.0**|**27.2**|**114.7**|**398.8**|**266.0**|N/A*|N/A*|\n\n[1] Park, S., et al., ICLR 2025, OGBench: Benchmarking Offline Goal-Conditioned RL, https://arxiv.org/abs/2410.20092  \n[2] Lee, H., et al., ICLR 2025,  SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning. http://arxiv.org/abs/2410.09754  \n[3] Lee, H., et al., ICML 2025. Hyperspherical Normalization for Scalable Deep Reinforcement Learning. http://arxiv.org/abs/2502.15280  \n[4] Farebrother, J., et al., ICML 2024, Stop Regressing: Training Value Functions via Classification for Scalable Deep RL. http://arxiv.org/abs/2403.03950  \n[5] Nauman, M., et al.,  NeurIPS 2024. Bigger, Regularized, Optimistic: Scaling for compute and sample-efficient continuous control. http://arxiv.org/abs/2405.16158  \n[6] Laskin, M., et al. ICML 2020, CURL: Contrastive Unsupervised Representations for Reinforcement Learning. https://proceedings.mlr.press/v119/laskin20a.html"}, {"review_id": "MFOhE2f6Af", "reviewer": "Reviewer_5GqS", "summary": "The authors show that depth scaling properties arise in contrastive RL, whereas width scaling is prominent in recent vanilla RL literature [1,2]. They propose to stack neural network blocks with residual connection, normalization (LayerNorm), and Swish architecture. By increasing the number of blocks, authors found that goal reaching capabilities emerge after a certain threshold, shown both by increased performance measure (figure 1) and novel behavior (figure 3). In their analysis, they find that depth scaling improves contrastive representations, enhances exploration, and allows trajectory stitching.\n\n[1] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning., Lee et al.\n\n[2] Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners., Nauman et al.", "questions": "Please see weaknesses above.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Strong performance and clear depth-scaling tendency to support their claims.\n- In-depth analysis on the architecture and policy as depth scales.\n- Verification of contrastive RL algorithm compared to baseline methods.\n\nWeaknesses\n- The experiments are limited to one type of architecture. Whether the same trend would occur in other well-designed architectures in RL [1,2,3] is unsure.\n- On a similar note, no sufficient analysis was made on the design choices of the architecture, other than the residual connection.\n- The connection between depth and exploration is not well explained. The results presented in figure 13 could also be due to improved learning capability of the collector.\n- Due to the model's sequential nature, a depth-scaled model is likely to take more time to generate output than a width-scaled model with same number of parameters. The claim that depth scaling is more compute efficient than width scaling (line 225:229) should thus we accompanied by a comparison on wall-clock time.\n- The fact that their architecture does not scale in offline GCRL weakens their point.\n\nOverall, I think the paper provides interesting and useful knowledge to the community, although there are several issues that needs to be refined.\n\n[1] SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning., Lee et al.\n\n[2] Hyperspherical Normalization for Scalable Deep Reinforcement Learning., Lee et al.\n\n[3] Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control., Nauman et al.", "was_revised": true, "final_justification": "The concerns I have raised have been addressed with additional experiment; I have raised my score from 4 to 5 accordingly. I believe this paper nicely reveals the scaling potential of GCRL compared to standard RL settings.", "rebuttal": "We thank Reviewer 5GqS for the helpful feedback, questions, and suggestions.\n\n> **1. The experiments are limited to one type of architecture. Whether the same trend would occur in other well-designed architectures in RL [1,2,3] is unsure.**\n\nTo address this concern, we have run a new experiment: we find that the same trend occurs when we apply our approach to another well-designed architecture for scaling RL, Simba-v2 [1]. The key innovation in Simba-v2 is replacing layer norm with hyperspherical normalization, which projects network weights onto the unit-norm hypersphere after each gradient update. **Table 1** shows that indeed the same depth scaling trends holds when we add the hyperspherical normalization to our architecture, and, in fact, actually improves the sample efficiency of depth scaling. This suggests that our method is able to integrate new architectural innovations in the emerging RL scaling literature.\n\n\n> **2. On a similar note, no sufficient analysis was made on the design choices of the architecture, other than the residual connection.**\n\nAs suggested by the reviewer, we report new ablation experiments for the other design choices of architecture: namely, layer norm and swish activation. We show that both of these design choices are critical for effective scaling:\n- In **Table 2a**, we show that removing layer normalization significantly hampers scalability.  \n- Additionally, **Table 2b** shows that scaling with ReLU activations performs significantly worse than Swish activations.  \n\nThese results, along with Figure 5 of the manuscript, show that all of our architectural components—residual connections, layer norm, and swish activations—are jointly essential to unlocking the full performance of depth scaling.\n\n> **3. The connection between depth and exploration is not well explained. The results presented in figure 13 could also be due to improved learning capability of the collector.**\n\nTo clarify, by exploration, we specifically mean the breadth or coverage of states visited. Shallow networks, due to their limited learning capacity, tend to explore only states near their initializations. While deep networks, due to their greater learning capacity, learn effective long-horizon behaviors, resulting empirically in much broader state coverage. Figure 13’s shallow collector experiments illustrate that improved learning capacity alone is insufficient when data diversity or coverage is poor. Conversely, Figure 13’s deep collector experiments show that good data coverage alone, without adequate learning capacity, also falls short. Thus, rather than viewing learning capacity and exploration as diametrically opposed, they mutually reinforce each other: stronger learning capacity drives more extensive exploration, and rich data coverage is essential to fully realize the potential of stronger learning capacity. Both aspects jointly contribute to improved performance.\n\n> **4. Due to the model's sequential nature, a depth-scaled model is likely to take more time to generate output than a width-scaled model with same number of parameters. The claim that depth scaling is more compute efficient than width scaling (line 225:229) should thus we accompanied by a comparison on wall-clock time.**\n\nIn **Table 3a**, we report the average wall-clock time for scaled width and depth, and find that width and depth empirically scale quite similarly in terms of wall-clock time. For practical deployment, the relevant metric is often wall-clock time to reach a given level of performance. **Table 3b** shows that scaling depth surpasses the performance of scaling width in less wall-clock time in all three environments. \n\n> **5. The fact that their architecture does not scale in offline GCRL weakens their point.**\n\nAlthough we found negative results for scaling CRL in the offline setting, we now report new experiments demonstrating that our approach scales effectively to another widely used offline algorithm, GCBC. In **Table 4a**, we find that GCBC successfully scales with depth on the antmaze-medium-stitch task in OGBench [2] when using our architecture of layer normalization, residual connections, and swish activations. In **Table 4b**, we show that using our architecture is critical to enabling this scaling in GCBC, as scaling the standard architecture degrades performance. These results suggest that our approach may be adapted to scaling the offline RL setting. Future work could further investigate what algorithms and architectural components best enable depth scaling in offline RL.\n\n### [TABLE 1] Hyperspherical Normalization (Simba-v2) Improves Sample Efficiency of Depth Scaling\n\n**Steps to reach ≥ 200 success**\n| Depth | With | Without |\n| ----: | --------------------------------: | -----------------------------------: |\n|     4 |                                 – |                                    – |\n|    16 |                            **50** |                                   64 |\n|    32 |                            **42** |                                   54 |\n\n**Steps to reach ≥ 400 success**\n| Depth | With | Without |\n| ----: | --------------------------------: | -----------------------------------: |\n|     4 |                                 – |                                    – |\n|    16 |                            **62** |                                   75 |\n|    32 |                            **48** |                                   64 |\n\n**Steps to reach ≥ 600 success**\n| Depth | With | Without |\n| ----: | --------------------------------: | -----------------------------------: |\n|     4 |                                 – |                                    – |\n|    16 |                            **77** |                                    – |\n|    32 |                            **67** |                                   77 |\n\n\n### [TABLE 2a] Layer Norm Ablation\n| Depth | With Layer Norm | Without Layer Norm |\n|-------------|----------------------|----------------------|\n|       4       |     **13.43 ± 4.40**       |    6.58 ± 0.88    | \n|      16       |     **350.17 ± 44.14**     |    13.01 ± 5.23    | \n|      32       |     **628.15 ± 41.00**     |    12.34 ± 3.55    |\n|      64       |     **672.56 ± 40.01**     |    12.94 ± 2.66   |\n\n### [TABLE 2b] Swish vs. ReLU Ablation\n| Depth | Swish | ReLU |\n|-------------|----------------------|----------------------|\n|       4       |     **13.43 ± 4.40**       |     15.09 ± 1.15     |\n|      16       |     **350.17 ± 44.14**     |     38.76 ± 6.99     |\n|      32       |     **628.15 ± 41.00**     |     46.62 ± 6.81     |\n|      64       |     **672.56 ± 40.01**     |   185.25 ± 107.40    |\n\n### [TABLE 3a] Wall-clock Time Comparison of Depth vs. Width Scaling\n|Width 256|Width 512|Width 1024|Width 2048|Width 4096|\n|---|---|---|---|---|\n|4.52 ± 0.34|5.32 ± 0.01|5.96 ± 0.01|8.39 ± 0.02|15.90 ± 0.64|\n\n|Depth 4|Depth 8|Depth 16|Depth 32|Depth 64|\n|---|---|---|---|---|\n|4.52 ± 0.34|5.31 ± 0.43|6.55 ± 0.48|9.14 ± 0.65|14.06 ± 0.66|\n\n### [TABLE 3b] Scaling Depth Surpasses Performance of Scaling Width in Less Wall-clock time\n|Environment|Width = 4096|Depth = 64|\n|-|-|-|\n|Humanoid|22.30 ± 0.08|**5.66 ± 0.24**|\n|Ant Big Maze|11.70 ± 0.03|**4.04 ± 0.41**|\n|Arm Push Easy|13.69 ± 1.92|**2.38 ± 0.18**|\n\n\n### [TABLE 4a] Successful Depth Scaling on GCBC (Antmaze-medium-stitch, OGBench)\n| Depth        | 2            | 4            | 8            | 16           | 32           |\n| ------------ | ------------ | ------------ | ------------ | ------------ | ------------ |\n| Performance | 0.474 ± 0.11 | 0.572 ± 0.04 | 0.574 ± 0.02 | 0.588 ± 0.00 | 0.634 ± 0.05 |\n\n### [TABLE 4b] Our Architecture is Critical: Depth Scaling on GCBC Fails with Standard Architecture\n| Depth        | 2             | 4             | 8             | 16            | 32            |\n| ------------ | ------------- | ------------- | ------------- | ------------- | ------------- |\n| Performance | 0.532 ± 0.088 | 0.746 ± 0.014 | 0.692 ± 0.024 | 0.698 ± 0.022 | 0.210 ± 0.210 |\n\n[1] Lee, H., et al., ICML 2025. Hyperspherical Normalization for Scalable Deep Reinforcement Learning. http://arxiv.org/abs/2502.15280.  \n[2] Park, S., et al., ICLR 2025, OGBench: Benchmarking Offline Goal-Conditioned RL, https://arxiv.org/abs/2410.20092"}, {"review_id": "UjPtr0XKrj", "reviewer": "Reviewer_kTrp", "summary": "The paper investigates scaling the depth of the networks used in self-supervised reinforcement learning for goal-reaching tasks. The work shows that scaling depth can significantly boost the performance of Contrastive RL in the JaxGCRL suite of online RL tasks. The authors also extensively analyze their design choices and provide qualitative analyses of the behavior and the value functions learned.", "questions": "- How is training time affected as you scale depth? \n\nAlso, I leave here some questions/idea for the authors to test outside of their contrastive RL+ online RL settings:\n* Given that simply scaling depth in the actor seems beneficial, would scaling depth be useful for BC and GCBC in Offline RL settings? \n* I understand the authors provide a study on CMD-1 in the Appendix, which uses the MRN network for quasimetric learning. It would be useful to also study QRL [1], which instead uses IQE for quasimetric learning\n\nMinor note: \n* the reference at Line 173 is undefined (probably due to the Figure being in the Appendix) \n\n[1] Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning", "limitations": "The authors have adequately addressed limitations in their work.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n* **Strong empirical analysis**: the empirical analysis of the approach is extensive and I found it particularly useful. It clearly shows how scaling depth can significantly improve performance for Contrastive RL. However, it also provides analyses about: scaling width vs depth, the usefulness of residual layers, scaling actor vs critic network, etc.\n* **New SOTA on JaxGCRL**: the results obtained with scaling depth and data are impressive. Also, improving the state-of-the-art in a benchmark is useful to further push the boundaries of research on that benchmark and in the field.\n## Weaknesses \n* **Limited generality of the claims** : my main concern with this work is that the idea of scaling depth seem to apply to a very specific algorithm, i.e. contrastive RL, and in very specific settings, i.e. online RL. I appreciate the authors' attempts in testing their method in offline RL settings and using CMD-1 (in Appendix). However, the generality of their claims remain so far limited.", "was_revised": true, "final_justification": "The work presents depth scaling as a successful technique to scale (self-supervised) RL. The findings seem to be mostly limited to the CRL approach (and to GCBC, as shown in some experiments posted for the rebuttal). However, the work remains an interesting case study and the authors provide an insightful analysis, which I think will be useful to the community.", "rebuttal": "We thank Reviewer kTrp for the helpful feedback, questions, and suggestions.\n\n> **1. How is training time affected as you scale depth?**\n\nWe report the wall-clock metrics of our approach as we scale depth in **Table 1**. We also conduct a comparison to baselines: since the baselines use standard sized networks, naturally our scaled approach incurs higher raw wall-clock time per environment step (**Table 2a**). For practical deployment, however, the more meaningful metric is often wall-clock time to attain a given level of performance. In **Table 2b**, we compare against the strongest baseline, SAC, and show that in 7 out of 10 environments our approach surpasses SAC performance in less wall-clock time. \n\n> **2. Limited generality of the claims : my main concern with this work is that the idea of scaling depth seem to apply to a very specific algorithm, i.e. contrastive RL, and in very specific settings, i.e. online RL.**\n\nTo address this concern, we report a new experimental result: we find that our approach successfully scales depth on a different algorithm (GCBC) in a different setting (offline RL). In **Table 3a**, we find that GCBC successfully scales on the antmaze-medium-stitch task in OGBench [1] when using our architecture of layer normalization, residual connections, and swish activations. In **Table 3b**, we show that using our architecture is critical to enabling scaling in GCBC, as scaling the standard architecture degrades performance. These results suggest that our approach to depth scaling may be adapted to other algorithms, as well as the offline setting. Future work could further investigate what algorithms and architectural components best enable depth scaling in offline RL.\n\n> **3. Given that simply scaling depth in the actor seems beneficial, would scaling depth be useful for BC and GCBC in Offline RL settings?**\n\nThanks for this suggestion! For GCBC, we have discussed the results in the response above. We also attempt to scale depth for standard BC in OGBench [1]. However, in this case we find negative results (**Table 4**).\n\n> **4. I understand the authors provide a study on CMD-1 in the Appendix, which uses the MRN network for quasimetric learning. It would be useful to also study QRL [1], which instead uses IQE for quasimetric learning**\n\nWe attempt to scale depth for QRL in both the online and offline settings, using the original QRL codebase [2] and OGBench [1] respectively. However, in both settings, we find that while QRL is able to train stably with deep networks, deep networks don’t improve performance beyond that of the shallow networks. **Table 5a** and **Table 5b** show these results.\n\n### [Table 1a] Wallclock time of Depth 4, 8, 16, 32, and 64 across all 10 environments\n|Depth|Humanoid|Ant Big Maze|Ant U4-Maze|Ant U5-Maze|Ant Hardest Maze|Arm Push Easy|Arm Push Hard|Arm Binpick Hard|Humanoid U-Maze|Humanoid Big Maze|\n|---|---|---|---|---|---|---|---|---|---|---|\n|Depth 4|1.48 ± 0.00|2.12 ± 0.00|1.98 ± 0.27|9.46 ± 1.75|5.11 ± 0.00|9.97 ± 1.03|9.74 ± 1.05|18.41 ± 2.16|8.72 ± 0.01|12.45 ± 0.02|\n|Depth 8|2.13 ± 0.01|2.77 ± 0.00|2.54 ± 0.01|10.99 ± 0.02|6.39 ± 0.00|11.02 ± 1.29|10.55 ± 1.20|17.48 ± 1.88|11.29 ± 0.01|15.02 ± 0.01|\n|Depth 16|3.40 ± 0.01|4.04 ± 0.01|3.81 ± 0.01|16.09 ± 0.01|8.94 ± 0.01|12.20 ± 1.43|11.98 ± 1.49|19.47 ± 0.05|16.36 ± 0.03|20.34 ± 0.01|\n|Depth 32|5.92 ± 0.01|6.57 ± 0.02|6.35 ± 0.01|31.49 ± 0.34|13.97 ± 0.01|14.94 ± 1.96|14.40 ± 1.64|21.91 ± 1.93|26.48 ± 0.05|30.61 ± 0.05|\n|Depth 64|10.99 ± 0.01|11.66 ± 0.03|11.43 ± 0.03|46.40 ± 0.12|23.96 ± 0.06|19.52 ± 1.97|18.53 ± 0.06|29.64 ± 6.10|46.74 ± 0.04|50.33 ± 0.05|\n\n### [Table 1b] Wallclock time up to Depth 1024 in the Humanoid U-Maze environment\n| Depth        | 4            | 8            | 16           | 32           | 64            | 128           | 256           | 512            | 1024           |\n| ------------ | ------------ | ------------ | ------------ | ------------ | ------------- | ------------- | ------------- | -------------- | -------------- |\n| Time  (h) | 3.23 ± 0.001 | 4.19 ± 0.003 | 6.07 ± 0.003 | 9.83 ± 0.006 | 17.33 ± 0.003 | 32.67 ± 0.124 | 73.83 ± 2.364 | 120.88 ± 2.177 | 134.15 ± 0.081 |\n\n### [TABLE 2a] Wall-clock Time Comparison of Our Method vs. Baselines\n|Method|Humanoid|Ant Big Maze|Ant U4-Maze|Ant U5-Maze|Ant Hardest Maze|Arm Push Easy|Arm Push Hard|Arm Binpick Hard|Humanoid U-Maze|Humanoid Big Maze|\n|---|---|---|---|---|---|---|---|---|---|---|\n|Scaled CRL (Depth 64)|10.99 ± 0.00|11.66 ± 0.01|11.43 ± 0.01|46.40 ± 0.05|23.96 ± 0.03|19.52 ± 0.57|18.53 ± 0.02|29.64 ± 1.30|46.74 ± 0.02|50.33 ± 0.02|\n|SAC|0.46 ± 0.00|1.55 ± 0.00|1.16 ± 0.00|5.73 ± 0.01|4.33 ± 0.00|8.32 ± 0.00|8.49 ± 0.04|20.70 ± 0.07|3.04 ± 0.00|8.55 ± 0.01|\n|SAC+HER|0.55 ± 0.00|1.63 ± 0.00|1.26 ± 0.00|6.10 ± 0.00|4.51 ± 0.00|8.54 ± 0.03|8.60 ± 0.01|20.74 ± 0.02|3.48 ± 0.00|9.27 ± 0.01|\n|TD3|0.79 ± 0.01|1.68 ± 0.01|1.30 ± 0.00|6.18 ± 0.01|5.04 ± 0.00|8.41 ± 0.01|8.31 ± 0.07|18.40 ± 0.28|5.35 ± 0.00|7.50 ± 1.14|\n|GCSL|0.42 ± 0.01|1.48 ± 0.26|0.70 ± 0.00|2.81 ± 0.12|2.13 ± 0.57|6.37 ± 0.12|5.22 ± 0.34|7.95 ± 0.90|3.07 ± 0.08|5.08 ± 0.04|\n|GCBC|0.61 ± 0.03|1.40 ± 0.13|1.11 ± 0.08|5.56 ± 0.54|4.38 ± 0.52|8.28 ± 0.34|7.36 ± 0.50|16.24 ± 0.36|7.18 ± 0.78|11.37 ± 1.93|\n\n\n### [TABLE 2b] CRL Surpasses SAC Performance in Less Wall-clock Time in 7 of 10 Environments\n\n| Algorithm                 |     Humanoid |  Ant Big Maze |   Ant U4-Maze |   Ant U5-Maze | Ant Hardest Maze |   Arm Push Easy |    Arm Push Hard | Arm Binpick Hard | Humanoid U-Maze | Humanoid Big Maze |\n| ------------------------- | -----------: | ------------: | ------------: | ------------: | ---------------: | --------------: | ---------------: | ---------------: | --------------: | ----------------: |\n| **SAC**                   |         **27.8** |          92.7 |          69.8 |         343.8 |            259.8 |           499.2 |            509.7 |           1242.1 |           **182.6** |             **512.9** |\n| **Scaled CRL (Depth 64)** | 382.2 ± 11.3 | **0.0 ± 0.0** | **0.0 ± 0.0** | **0.0 ± 0.0** |   **27.2 ± 3.8** | **114.7 ± 9.9** | **398.8 ± 42.1** | **266.0 ± 14.7** |           N/A\\* |             N/A\\* |\n\n*N/A denotes SAC outperforms our approach, so CRL never reaches SAC performance. These account for 2 of the 3 environments where CRL fails to surpass SAC in less wall-clock time.\n\n### [TABLE 3a] Depth Scaling on GCBC (Antmaze-medium-stitch, OGBench)\n| Depth        | 2            | 4            | 8            | 16           | 32           |\n| ------------ | ------------ | ------------ | ------------ | ------------ | ------------ |\n| Performance | 0.474 ± 0.11 | 0.572 ± 0.04 | 0.574 ± 0.02 | 0.588 ± 0.00 | 0.634 ± 0.05 |\n\n### [TABLE 3b] Our Architecture is Critical: Depth Scaling on GCBC Fails with Standard Architecture\n| Depth        | 2             | 4             | 8             | 16            | 32            |\n| ------------ | ------------- | ------------- | ------------- | ------------- | ------------- |\n| Performance | 0.532 ± 0.088 | 0.746 ± 0.014 | 0.692 ± 0.024 | 0.698 ± 0.022 | 0.210 ± 0.210 |\n\n\n### [Table 4] Depth Scaling on BC (Antmaze-giant-navigate on expert SAC policy, OGBench)\n|Depth|2|4|8|16|32|\n|---|---|---|---|---|---|\n|Performance|0.696 ± 0.00|0.782 ± 0.03|0.348 ± 0.00|0.088 ± 0.02|0.123 ± 0.04|\n\n\n### [Table 5a] Depth Scaling on Online QRL (FetchPush task on original QRL codebase)\n|Depth|2|4|8|16|32|\n|---|---|---|---|---|---|\n|Performance|0.160 ± 0.04|0.200 ± 0.00|0.180 ± 0.00|0.260 ± 0.10|0.250 ± 0.01|\n\n\n### [Table 5b] Depth Scaling on Offline QRL (Pointmaze-giant-stitch, OGBench)\n|Depth|2|4|8|16|32|\n|---|---|---|---|---|---|\n|Performance|0.570 ± 0.02|0.495 ± 0.12|0.450 ± 0.00|0.600 ± 0.00|0.588 ± 0.00|\n\n[1] Park, S., et al., ICLR 2025, OGBench: Benchmarking Offline Goal-Conditioned RL, https://arxiv.org/abs/2410.20092.  \n[2] Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learning"}, {"review_id": "vFwm6BaPn4", "reviewer": "Reviewer_4F8D", "summary": "This paper explores the potential of deep network architectures in self-supervised reinforcement learning (RL) to enhance scalability. Unlike most recent RL studies that use shallow networks (2-5 layers), the authors show that increasing network depth to up to 1024 layers can significantly improve performance. The experiments are conducted in an unsupervised, goal-conditioned setting without demonstrations or rewards, requiring the agent to explore and learn to reach commanded goals. Tested on simulated locomotion and manipulation tasks, the approach achieves 2x-50x performance improvements over other goal-conditioned baselines, demonstrating that greater depth not only boosts success rates but also qualitatively alters the learned behaviors.", "questions": "1. How do the training costs of the proposed algorithm compare with those of the baselines? Can anthors show it in a Table？\n2. Can the authors elaborate empirically or conceptually on why performance “jumps” occur at specific depths or why agents exhibit different behaviors at certain depths (as shown in Figure 3) in certain environments?", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The idea is interesting, and the writing is clear and easy to understand.\n2. The authors conducted numerous experiments to demonstrate that increasing the depth of networks is a key factor in improving the performance of CRL.\n3. The authors carried out extensive analysis experiments to explore various issues, including some that the reviewer might be concerned about. For example, whether increasing the depth of networks can enhance the performance of other baseline algorithms. Although the answer is no, combining scaling depth or network size with other algorithms may be one of the important future directions for RL.\n\n**Weaknesses**\n\n1. There is an error in the citation of the appendix at line 173, which shows as \"？？\".\n2. The paper does not mention the training and inference costs of the proposed algorithm. The training and inference costs associated with a network depth of 1000 layers may be relatively high.\n3. The innovativeness of the article has certain deficiencies. The core technologies involved (such as residual connections, normalization, activation functions, etc.) are derived from existing research. The main novelty of the article lies in the impact of the increase in model size on performance, rather than innovation in methodology.\n4. The contributions of the article are mainly experimental, with a lack of theoretical discussion on the critical parts of the experiments. For example, why does the performance undergo a sudden change when the depth reaches a certain critical point? This, to some extent, limits the article's insight.", "was_revised": true, "final_justification": "The authors' response has addressed my concerns, and I will maintain my score.", "rebuttal": "We thank Reviewer 4F8D for the helpful feedback, questions, and suggestions.\n\n> **1. How do the training costs of the proposed algorithm compare with those of the baselines? Can authors show it in a table?**\n\nSure! Because the baseline algorithms use standard-sized networks (less than 4 layers), our scaled approach naturally incurs higher raw wall-clock time per environment step (**Table 1a**). However, for practical deployment the more meaningful metric is the total wall-clock time required to attain a given level of performance. To evaluate this, we compare our method with the strongest baseline, SAC. As shown in **Table 1b**, our approach surpasses SAC performance in less wall-clock time in 7 out of 10 environments.\n\n> **2. Can the authors elaborate empirically or conceptually on why performance “jumps” occur at specific depths or why agents exhibit different behaviors at certain depths (as shown in Figure 3) in certain environments?**\n\nTo elaborate conceptually why performance “jumps” occur at specific depths, we draw parallels to the broader deep learning literature of the well-documented phenomenon of *emergent abilities*, where a sudden increase in performance arises at a critical threshold of model scale. Wei et al. 2022 [1] first proposed that large language models exhibit such emergent behavior, with performance transitioning sharply from near-random at a certain threshold points in model size. Although cross-entropy loss improves smoothly with scale, the model only begins to complete tasks once a critical threshold of capability is reached. In our experiments, we observe similar patterns in the Humanoid agent: as the depth increases from 4 to 8 to 16 layers, its ability to locomote improves gradually. However, only upon reaching a critical level of coordination—when it can walk stably without falling—does the agent achieve the goal and remain upright. This manifests as a sharp, discontinuous jump in performance. Wei et al. also remark that emergent abilities often involve qualitatively novel behaviors, such as chain-of-thought reasoning in LLMs, that are absent in smaller models. We observe a parallel in the Humanoid U-Maze environment: only at a very large model scale (i.e., depth 256) does the agent achieve the complex and coordinated movement necessary to successfully vault over the wall. At smaller depths, the agent simply appears to “flounder”, its movements not yet sophisticated enough to complete the maneuver. This behavior manifests as a qualitatively distinct, “emergent” ability and is reflected in the observed performance jump.\n\n\n> **3. There is an error in the citation of the appendix at line 173, which shows as \"？？\".**\n\nThank you for pointing that out. This figure comparing our approach with baselines is included as Figure 10 of the Appendix. The issue was due to a typo that resulted in an undefined reference — we have now fixed this in our local version.\n\n> **4. The paper does not mention the training and inference costs of the proposed algorithm. The training and inference costs associated with a network depth of 1000 layers may be relatively high.**\n\nThanks for this suggestion. In terms of considerations on training and inference costs, we report the wall-clock time of our approach as we scale depth (**Table 2a**, **Table 2b**). A comparison of the cost of wall-clock time vs. baselines is discussed in the first paragraph of this response above, and a comparison of wall-clock time for depth vs. width scaling can be found in **Table 3a** and **Table 3b**. We likewise also report the metrics of FLOPs and memory (**Table 4**, **Table 5**). \n\n> **5. For example, why does the performance undergo a sudden change when the depth reaches a certain critical point? This, to some extent, limits the article's insight.**\n\nWe address this in the second paragraph of this response above.\n\n### [TABLE 1a] Wall-clock Time Comparison of Our Method vs. Baselines\n|Method|Humanoid|Ant Big Maze|Ant U4-Maze|Ant U5-Maze|Ant Hardest Maze|Arm Push Easy|Arm Push Hard|Arm Binpick Hard|Humanoid U-Maze|Humanoid Big Maze|\n|---|---|---|---|---|---|---|---|---|---|---|\n|CRL Depth 4|1.48 ± 0.00|2.12 ± 0.00|1.98 ± 0.05|9.46 ± 0.28|5.11 ± 0.00|9.97 ± 0.39|9.74 ± 0.40|18.41 ± 0.82|8.72 ± 0.00|12.45 ± 0.00|\n|CRL Depth 32|5.92 ± 0.00|6.57 ± 0.01|6.35 ± 0.00|31.49 ± 0.13|13.97 ± 0.00|14.94 ± 0.88|14.40 ± 0.62|21.91 ± 0.38|26.48 ± 0.02|30.61 ± 0.02|\n|CRL Depth 64|10.99 ± 0.00|11.66 ± 0.01|11.43 ± 0.01|46.40 ± 0.05|23.96 ± 0.03|19.52 ± 0.57|18.53 ± 0.02|29.64 ± 1.30|46.74 ± 0.02|50.33 ± 0.02|\n|SAC|0.46 ± 0.00|1.55 ± 0.00|1.16 ± 0.00|5.73 ± 0.01|4.33 ± 0.00|8.32 ± 0.00|8.49 ± 0.04|20.70 ± 0.07|3.04 ± 0.00|8.55 ± 0.01|\n|SAC+HER|0.55 ± 0.00|1.63 ± 0.00|1.26 ± 0.00|6.10 ± 0.00|4.51 ± 0.00|8.54 ± 0.03|8.60 ± 0.01|20.74 ± 0.02|3.48 ± 0.00|9.27 ± 0.01|\n|TD3|0.79 ± 0.01|1.68 ± 0.01|1.30 ± 0.00|6.18 ± 0.01|5.04 ± 0.00|8.41 ± 0.01|8.31 ± 0.07|18.40 ± 0.28|5.35 ± 0.00|7.50 ± 1.14|\n|GCSL|0.42 ± 0.01|1.48 ± 0.26|0.70 ± 0.00|2.81 ± 0.12|2.13 ± 0.57|6.37 ± 0.12|5.22 ± 0.34|7.95 ± 0.90|3.07 ± 0.08|5.08 ± 0.04|\n|GCBC|0.61 ± 0.03|1.40 ± 0.13|1.11 ± 0.08|5.56 ± 0.54|4.38 ± 0.52|8.28 ± 0.34|7.36 ± 0.50|16.24 ± 0.36|7.18 ± 0.78|11.37 ± 1.93|\n\n### [TABLE 1b] CRL Surpasses SAC Performance in Less Wall-clock Time in 7 of 10 Environments\n| Algorithm                 |     Humanoid |  Ant Big Maze |   Ant U4-Maze |   Ant U5-Maze | Ant Hardest Maze |   Arm Push Easy |    Arm Push Hard | Arm Binpick Hard | Humanoid U-Maze | Humanoid Big Maze |\n| ------------------------- | -----------: | ------------: | ------------: | ------------: | ---------------: | --------------: | ---------------: | ---------------: | --------------: | ----------------: |\n| **SAC**                   |         **27.8** |          92.7 |          69.8 |         343.8 |            259.8 |           499.2 |            509.7 |           1242.1 |           **182.6** |             **512.9** |\n| **Scaled CRL (Depth 64)** | 382.2 ± 11.3 | **0.0 ± 0.0** | **0.0 ± 0.0** | **0.0 ± 0.0** |   **27.2 ± 3.8** | **114.7 ± 9.9** | **398.8 ± 42.1** | **266.0 ± 14.7** |           N/A\\* |             N/A\\* |\n\n*N/A denotes SAC outperforms our approach, so CRL never reaches SAC performance. These account for 2 of the 3 environments where CRL fails to surpass SAC in less wall-clock time.\n\n\n### [Table 2a] Wallclock time of Depth 4, 8, 16, 32, and 64 across all 10 environments\n|Depth|Humanoid|Ant Big Maze|Ant U4-Maze|Ant U5-Maze|Ant Hardest Maze|Arm Push Easy|Arm Push Hard|Arm Binpick Hard|Humanoid U-Maze|Humanoid Big Maze|\n|---|---|---|---|---|---|---|---|---|---|---|\n|Depth 4|1.48 ± 0.00|2.12 ± 0.00|1.98 ± 0.27|9.46 ± 1.75|5.11 ± 0.00|9.97 ± 1.03|9.74 ± 1.05|18.41 ± 2.16|8.72 ± 0.01|12.45 ± 0.02|\n|Depth 8|2.13 ± 0.01|2.77 ± 0.00|2.54 ± 0.01|10.99 ± 0.02|6.39 ± 0.00|11.02 ± 1.29|10.55 ± 1.20|17.48 ± 1.88|11.29 ± 0.01|15.02 ± 0.01|\n|Depth 16|3.40 ± 0.01|4.04 ± 0.01|3.81 ± 0.01|16.09 ± 0.01|8.94 ± 0.01|12.20 ± 1.43|11.98 ± 1.49|19.47 ± 0.05|16.36 ± 0.03|20.34 ± 0.01|\n|Depth 32|5.92 ± 0.01|6.57 ± 0.02|6.35 ± 0.01|31.49 ± 0.34|13.97 ± 0.01|14.94 ± 1.96|14.40 ± 1.64|21.91 ± 1.93|26.48 ± 0.05|30.61 ± 0.05|\n|Depth 64|10.99 ± 0.01|11.66 ± 0.03|11.43 ± 0.03|46.40 ± 0.12|23.96 ± 0.06|19.52 ± 1.97|18.53 ± 0.06|29.64 ± 6.10|46.74 ± 0.04|50.33 ± 0.05|\n\n### [Table 2b] Wallclock time up to Depth 1024 in the Humanoid U-Maze environment\n| Depth        | 4            | 8            | 16           | 32           | 64            | 128           | 256           | 512            | 1024           |\n| ------------ | ------------ | ------------ | ------------ | ------------ | ------------- | ------------- | ------------- | -------------- | -------------- |\n| Time (hours) | 3.23 ± 0.001 | 4.19 ± 0.003 | 6.07 ± 0.003 | 9.83 ± 0.006 | 17.33 ± 0.003 | 32.67 ± 0.124 | 73.83 ± 2.364 | 120.88 ± 2.177 | 134.15 ± 0.081 |\n\n### [TABLE 3a] Wall-clock Time Comparison of Depth vs. Width Scaling\n|Width 256|Width 512|Width 1024|Width 2048|Width 4096|\n|---|---|---|---|---|\n|4.52 ± 0.34|5.32 ± 0.01|5.96 ± 0.01|8.39 ± 0.02|15.90 ± 0.64|\n\n|Depth 4|Depth 8|Depth 16|Depth 32|Depth 64|\n|---|---|---|---|---|\n|4.52 ± 0.34|5.31 ± 0.43|6.55 ± 0.48|9.14 ± 0.65|14.06 ± 0.66|\n\n\n### [TABLE 3b] Scaling Depth Surpasses Performance of Scaling Width in Less Wall-clock time\n|Environment|Width = 4096|Depth = 64|\n|-|-|-|\n|Humanoid|22.30 ± 0.08|**5.66 ± 0.24**|\n|Ant Big Maze|11.70 ± 0.03|**4.04 ± 0.41**|\n|Arm Push Easy|13.69 ± 1.92|**2.38 ± 0.18**|\n\n\n### [TABLE 4] Comparison of Width vs. Depth Scaling: FLOPS and Memory\nFLOPs comparison of width vs. depth scaling\n|Width|256|512|1024|2048|4096|\n|---|---|---|---|---|---|\n|FLOPs (B)|4.8|17.4|66.9|262.4|1039.9|\n\n|Depth|4|8|16|32|64|\n|---|---|---|---|---|---|\n|FLOPs (B)|4.8|8.9|17.1|33.5|66.2|\n\n### [TABLE 4] Comparison of Width vs. Depth Scaling: FLOPs\n|Width|256|512|1024|2048|4096|\n|---|---|---|---|---|---|\n|FLOPs (B)|4.8|17.4|66.9|262.4|1039.9|\n\n|Depth|4|8|16|32|64|\n|---|---|---|---|---|---|\n|FLOPs (B)|4.8|8.9|17.1|33.5|66.2|\n\n### [TABLE 5] Comparison of Width vs. Depth Scaling: Memory\n|Width|256|512|1024|2048|4096|\n|---|---|---|---|---|---|\n|Memory (MB)|24.8|63.4|194.7|673.3|2494.6|\n\n|Depth|4|8|16|32|64|\n|---|---|---|---|---|---|\n|Memory (MB)|24.8|43.8|81.9|158.0|310.3|\n\n\n[1] Wei, J., et al., NeurIPS 2022, Chain-of-Thought Prompting Elicits Reasoning in Large Language Models https://arxiv.org/abs/2201.11903"}]

Author rebuttal:

Meta-review: This paper received positive and enthusiastic reviews from all reviewers. The AC agrees with and shares this enthusiasm.

This paper presents a reinforcement learning paradigm to effectively train (very) deep neural networks with self-supervised reinforcement learning. Using this framework, it then presents an analysis demonstrating that, like in other areas of self-supervised learning, self-supervised RL too can scale effectively with network depth and more and more sophisticated capabilities emerge as depth increases.

The paradigm presented in this paper is relatively straightforward and makes use of three things. First, it uses a simple self-supervised RL algorithm, contrastive RL (CRL), second it makes use of GPU-accelerated RL simulators to collect a large amount of data, and third it makes use of modern network designs. This simplicity and the included code will make it easier for the community to build upon this work.

The results and analysis are both equally impressive. On the JaxGCRL benchmarks the performance of contrastive RL improves by 2x to 50x and their scaled CRL outperforms all other methods (by up to an order of magnitude) in nearly all environments. Their analysis is extensive and well done. While there is more than can be summarized here, the AC would like to draw attention to:
1. their demonstration of new behaviors being discovered at greater depth, 
2. the analysis showing the importance of batch size scaling for deeper networks in CRL,
3. and the analysis showing that deeper networks enable the policy to stitch together its training experience of short tasks into handling long tasks as evaluation time.

Overall, this paper presents a paradigm that others can easily build on, impressive results, and analysis that will improve the communities understanding of scaling self-supervised RL. Given this, an Oral presentation is appropriate.
Decision: Accept (oral)