XQC: Well-conditioned Optimization Accelerates Deep Reinforcement Learning
Authors: Daniel Palenicek
Florian Vogt
Joe Watson
Ingmar Posner
Jan Peters
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Primary area: reinforcement learning

Abstract: Sample efficiency is a central property of effective deep reinforcement learning algorithms. Recent work has improved this through added complexity, such as larger models, exotic network architectures, and more complex algorithms, which are typically motivated purely by empirical performance. We take a more principled approach by focusing on the optimization landscape of the critic network. Using the eigenspectrum and condition number of the critic’s Hessian, we systematically investigate the impact of common architectural design decisions on training dynamics. Our analysis reveals that a novel combination of batch normalization (BN), weight normalization (WN), and a distributional cross-entropy (CE) loss produces condition numbers orders of magnitude smaller than baselines. This combination also naturally bounds gradient norms, a property critical for maintaining a stable effective learning rate under non-stationary targets and bootstrapping. Based on these insights, we introduce XQC: a well-motivated, sample-efficient deep actor-critic algorithm built upon soft actor-critic that embodies these optimization-aware principles. We achieve state-of-the-art sample efficiency across 55 proprioception and 15 vision-based continuous control tasks, all while using significantly fewer parameters than competing methods. Our code is available at danielpalenicek.github.io/projects/xqc.

Published as a conference paper at ICLR 2026
XQC: WELL-CONDITIONED OPTIMIZATION ACCELERATES DEEP REINFORCEMENT LEARNING
Daniel Palenicek" 1.2 Florian Vogt" Joe Watson" Ingmar Posner* Jan Peters'
Peters!.2.5. 12,5,6
'Technical University of Darmstadt Phessian.Al UUniversity of Freiburg *University of Oxford "German Research Center for Al (DFKI) Robotics Institute Germany (RIG)
ABSTRACT
Sample efficiency is central property of effective deep reinforcement learn- ing algorithms. Recent work has improved this through added complexity, such larger models, exotic network architectures, and more complex algorithms,, which are typically motivated purely by empirical performance. We take a more principled approach by focusing on the optimization landscape of the critic net- work. Using the eigenspeetrum and condition number of the critic's Hessian, we systematically investigate the impact of common architectural design deci- sions on training dynamics. Our analysis reveals that a novel combination of batch normalization (BN), weight normalization (WN), and distributional cross- entropy (CE) loss produces condition numbers orders of magnitude smaller than baselines. This combination also naturally bounds gradient norms, property crit- ical for maintaining a stable effective learning rate under non-stationary targets and bootstrapping. Based on these insights, we introduce XQC: a well-motivated, sample-efficient deep actor-critic algorithm built upon soft actor-critic that em- principles. We achieve state-of-the-an sample
bodies these optimization-avee efficiency across 55 proprioception and 15 vision-basee continuous control tasks, all while using significantly fewer parameters than competing methods. Our code is available
Better Conditioning
Number of Parameters
State-based (55 tasks)
Pixel-based RL (humanoid walk)
Improves RL Performacce
TQM Condition
10a
SAC
35M
TQM AUC Retum
800
Numbeb is
107
CroaQQ
0.6
600
10
0.4
0.22
10
SimbuV22
IQM
200
10*
XQC
2M
0
500
1000
IQM Return @ 1M
Env Steps (10M)
XQC (ouns)
CrossQ
SAC
BRO
SimbaV2
BRC
CrassQ + WN
MR.Q DIQ-V2
Figure 1: Well-conditioned network architectures yield state-of-the-art RL performance. Our algorithm, XQC with BN and wN-based architecture and CE loss, achieves competitive perfor- mance against state-of-the-ar baselines across 55 proprioceptive continuous control tasks from four different benchmarks with single set of hyperparameters. Notably, with 4..5x fewer parameters and 5x less compute in terms of FLOP// than SIMBA-VA the closest competitor. XQC's efficiency carries over from pixels on 15 vision-based DMC tasks, significantly improving DRQ-V2.
INTRODUCTION
Sample efficiency remains a major challenge in deep reinforcement learning (RL). Methods that can learn effectively from limited interactions are crucial for applying RL in domains such as robotics (Kober & Peters, 2008; Bohlinger et al., 2025), where generating data on real hardware costly and time-consuming. Recent advances in model-free RL have primarily been driven by paradigm of scaling larger networks, higher update-to to-data ratio (UTD) ratios, and ever- increasing computational budgets (Nikishin et al., 2022; Nauman et al., 2024; Lee al., 2025a; Palenicek et al., 2025). These works view architectural improvements primarily as a means to scale up stably and in turn improve sample efficiency. While proven effective, this 'bigger is better" paradigm comes

---

Published as a conference paper at ICLR 2026
the cost of computational efficiency and overlooks more fundamental question: Can improve performance not by adding complexity, but by creating a
optimization problem?
To develop this principled understanding,
conduct systematic investigation into three com- monly used architectural components whose roles in RL are often guided by heuristics. First, we examine normalization layers. The RL community frequently uses layer normalisation (LN) (Ba et al., 2016). e.g., Ball et al. (2023), in contrast to batch normalisation (BN) (loffe & Szegedy, 2015), which until recently was thought be problematic in the RL setting (Bhatt et al., 2024). Second, we consider weight normalisation (WN) (Lyle et al., 2024; Loshchilov et al., 2025; Palenicek et al., 2025) by periodically projecting the network's 's weights the unit sphere, permitted through the nor- malization layers' scale invariance property. A technique known to improve the effective learning rate (FIR) (Van Laarhoven, 2017). Lastly, we study the critic's loss function. Distributional critics using a categorical cross entropy (CF) loss have grown in popularity (Bellemare et al., 2017). The conventional argument for their adoption that modeling the full distribution of returns provides a better learning signal compared to mean squared error (MSE) regression (Farebrother et al., 2024); there is evidence that this loss is easier to optimize (Imani & White, 2018).
Through a systematic eigenvalue analysis of the critic's 's Hessian, we provide principled explanation for why different architectures outperform others. Our analysis first shows that BN consistently produces coniitooondi local loss landscapes than LN during learning, with condition numbers that are orders of magnitude smaller. Second, our investigation of the critic loss reveals that, beyond its representational advantages, the CE loss induces remarkably well-conditioned optimization landscape compared to the MSE loss. We find that the combination of BN, WN, and categorical CE loss works in synergy to dramatically improve the conditioning of the optimization problem and stabilize the ELR, a key metric for maintaining plasticity in deep RL. In summary, we claim the following contributions:
1. XQC, a simple and efficient extension to soft actor critic, uses the powerful synergy between BN, WN, and a distributional critic with a CE Bellman error loss for sample-efficient learning.
2. A Hessian eigenvalue analysis of modern deep RL critics, revealing the superior conditioning properties of distributional critic losses over the mean squared error.
3. Extensive empirical validation on 55 proprioception and 15 vision-based continuous control tasks, demonstrating state-of-the-arde performance against more complex, larger-scale methods.
2 PRELIMINARIES
This section briefly introduces the necessary background and notation for this paper.
Deep reinforcement learning. In this work, we assume the standard RL setting (Sutton & Barto, 2018), where an agent attempts to learn a policy that maximizes its expected discounted re- turn. Our experiments are based on the popular off-policy actor-critic algorithm soft actor-critic (SAC) (Haarnoja et al., 2018), where policy and critic are represented by neural networks. A key quantity in reinforcement learning with function approximation is the Bellman error Ao.
(1) where Q and V are the 'soft' parametric critic and value functions, respectively. Minimizing this Bellman eπror effectively is key to the success of actor-critic methods (Sutton & Barto, 2018). This error is typically minimized with the mean squared regression loss and gradient-based optimization. The distributional C51 algorithm (Bellemare et al., 2017) reformulates the task as a classification problem. Instead of scalar estimate, the function approximator outputs the logits of categorical distribution over the full support of Q. The Bellman error can then be minimized using a categorical CE loss, which has been shown improve performance and stability (Farebrother et al., 2024).
(s a) = ((s, a) Qo(s,a), Q(s a) = r(s, a) + Eg'mm((leaa) [V('))
Analyzing gradient-based optimization. To analyze the optimization of our gradient-based up- dates, we consider their first- and second-order aspects. For gradient- -hase optimization with pa- rameter normalization, we must consider the effective learning rate (Definition l).
Definition 1. (Effective learning rute, ELR, Van Laarhoven (2017)). For scale-invariant function f(0) = f()0), À > O, the "effective" learning rate î for an update f(O + +1 (f)) with gradients g() is the learning rate when taking this scale invariance into account,
S( (g())) ( (0)), : n/ ||0||2
0 = 0/|
2

---

Published as a conference paper at ICLR 2026
100
1000
10
100
317
1000
10"-1
P2
21
20000
40000
steps
Figure 2: When performing gradient-based optimization, the condition number (x) of the objec- tive's Hessian significantly impacts convergence. We illustrate this phenomenon with a simple two-dimensional quadratic example. As к increases by an order of magnitude, gradient descent converges at a lower rate. We believe this phenomenon plays a similar role when learning the critic in deep reinforcement learning, where high condition numbers lead to poor sample efficiency.
Recent work has studied ELR in the context of loss of 'plasticity" in neural networks and scaling gracefully larger UTD ratios in (Lyle et al., 2024; Palenicek et al., 2025).
To analyze the second-order properties of the loss landscape, a local quadratic approximation
1 _00 VOC(0)80
C( + 50) R [()) + VeL(0)60
illustrates the role of the Hessian VgC(0) in characterizing the curvature of the local loss land- scape, which is measured using its eigenvalues. The 3-smoothness of the objective upper-bounds the largest eigenvalue of the Hessian (Definition 2), while the ratio of largest to smallest absolute val- ues describes the condition number (Definition 3). The larger the condition number, the less effective gradient descent with fixed learning rate will be due the large range in curvature per dimension, as illustrated in Figure 2 (Nocedal & Wright, 2006). While we use an adaptive learning rate opti- mizer (Adam, Kingma Ba (2015)), whose adaptivity helps overcome issues with ill-conditioning, the loss landscape curvature remains relevant when assessing optimization difficulty.
Definition 2. (3-smoothness, Aravkin et al. (2017)). loss function L(0) said be 3-soooth if gradient is Lipschitz continuous with constant 3, i.e., ([)) - V (02)l S 8 [110, 02 which equivalent to the largest eigenvalue of its Hessian being bounded by 3, i.e., Amax (Q)(()) S B.
As such, the 8-smoothness quantifies the maximum curvature of the landscape.
In our experiments, we will look at the largest eigenvalue as a proxy for the empirical measure of 33 Definition 3. (Condition number, Nocedal & Wright (2006)). For normal matrix H E Rdxdd with eigenvalues .1...., ......,A,, Ad, its condition number k is k(H) = max, / ming [Ag), As a measure of sensitivity, low condition number describes 'well-conditioned matrix, while high condition number describes an 'ill-conditiondd' matrix.
For Hessian, the condition number is used to analyze and characterize the effectiveness of gradient- based optimization algorithms (Nocedal & Wright, 2006). Using the insights of the effective learn- ing rate, (-smoothness,, and condition numbers, now compare the optimization landscapes of different actor-critic architectures in deep RL.
THE OPTIMIZATION LANDSCAPES OF THE BELLMAN ERROR
We seek to improve sample efficiency by enhancing the critic's 's optimization landscape. This section applies the optimization insights from Section 2 to the Bellman error minimization. Hessian eigen- values have previously been used to understand the benefits of batch normalization in supervised learning (Ghorbani et al., 2019). To our knowledge, this is the first such analysis for deep RI.
3.1 AN EMPIRICAL INVESTIGATION OF CRITIC OPTIMIZATION.
To quantify the impact of common architectural components on the optimization of the Bellman error, we looked the cigenvalues of the critic's 's Hessian while learning the challenging DeepMind control suite (DMC) dog-trot environment, a high-dimensional continuous control task, ensuring the findings not artifacts of trivial toy-task.
3

---

Published as a conference paper at ICLR 2026
BN+WN
LN+WN
Dense+WN
BN
LN
Dense
CE loss
10g10 (A)
10a
MSE loss
log1o
LN
Dense
CE
MSE
lox; 0 (v))
IQM
60
1L5
110
010
015
1.0 105
115
Environment Steps (IM)
Figure 3: Eigenvalues and condition numbers dog-trot over 5 seeds for different critic architectures during training. The top and middle rows show the eigenspectra of the CE loss and MSE loss, respectively. The columns correspond to different combinations of normalization layers and WN. The bottom row shows the IQM and 90% SBCI of the condition number λ aggregated over five seeds for CE and MSE losses, respectively. Architectures using BN show more compact and stable eigenspectra over the course of training with no outliers. L.N suffers from large outlier modes and includes overall larger eigenvalues. Similarly, the CE loss significantly improves loss landscape conditioning over an MSE.
We systematically compare critic networks with combinations of common architectural components:
Normalization strategies: BN, LN, None (Dense).
Weight projection to the unit sphere: WN (L), WN
Loss functions: MSE (a), CB (o).
This results in total of 12 distinct architectural combinations, allowing for thorough dissection of each component's 's contribution. Per architecture, we run 5 random seeds for 1M environment steps and compute the Hessian eigenspectrum at 20 checkpoints throughout training using an efficient JAX (Bradbury et al., 2018) implementation of the stochastic Lanczos quadrature algorithm (Golub & Welsch, 1969; Lin al., 2016), adapted from Ghorbani et al. (2019).
Eigenvalue analysis. First, we qualitatively analyze how the cigenvalues evolve during training for the different architectures. Figure (top & middle) reveals striking differences in the curvature the loss landscape for the different components. Architectures employing BN consistently produce more compact and stable eigenspectra throughout training, with eigenvalues remaining bounded within moderate range and free of significant outliers. In stark contrast, LN architectures suffer from large, growing outlier eigenvalues, signifying sharp curvature that can destabilize training. Similarly, the CE loss significantly improves loss landscape conditioning over MSE. This is also reflected in the condition numbers Figure 3 (bottom), where BN-based architectures are consistently an order of magnitude smaller and more stable than their non-BN counterparts.
Condition numbers and /3smoothness. To make the relationship between the spectral properties and performance explicit, Figure 4 presents the data in aggregated form. Each point shows ag- gregated results over 5 seeds, correlating an architecture ' IQM condition number, IQM max(A) and IQM Kurtosis(A), A), respectively, over the entire course of training, with its sample-efficiency (IQM turn at IM timesteps). These plots show clear and strong trend: architectures with lower condition numbers and smaller maximum eigenvalues achieve higher returns. This trend provides compelling empirical evidence that, perhaps unsurprisinlly. smoother, better-conditioned optimization land- scape a key driver of performance in deep RI.. The Kurtosis provides a proxy measure for outliers in the eigenspectrum, where BN-based architectures consistently show lower Kurtosis than their LN-based counterparts. Furthermore, the results show that BN, WN, and a categorical CE loss each independently improve the landscape's 's conditioning, and when combined, their synergistic effect yields the best-conditioned landscape and the highest performance.
4

---

Published as a conference paper at ICLR 2026
LN BN
Adding WN
MSE -> CE
IQM max(A)
IQM Kurtosis(A)
1QM Cond. number
10a
107
10a
10"
10a
10a
16"
10
10
250
500
750
250 b0o 760
250 800 750
250 buo 750
250 500 750
1QM
IQM 1M
IQM
IQM
IQM 1M
The condition numbers and maximum eigenvalues against the return at 1M steps DMC dog-trot. Normalization strategies are color-coded BN, LN, Dense. Use of WN empty shape whereas no WN is represented by a filled shape m. MSE loss and CE= Q. Architectures with lower condition numbers and lower maximum eigenvalues tend to have better final returns. Also, BN, WN, and the categorical
Figure 4:
loss each improve the loss conditioning independently (columns 1-3). Combined, they result in the best conditioning and best performance o, For reference, we include SIMBA-V2 o a strong baseline with a similarly low condition number.
3.2 WIYY DOES CROSS-ENTROPY OUTPERFORM THE SQUARED ERROR?
Distributional RL and C51 were proposed to perform distributional regression of the returns, beyond predicting only the average value (Bellemare al., 2023). In this section, we motivate distributional losses from the optimization perspective (Imani & White, 2018) to explain the dramatic difference condition numbers between CE and MSE Bellman errors in Section 3.1. We show that the CE loss has desirable properties for optimization over the MSE. Firstly, Propositions I and 2 show that the gradient norm for the loss with respect to the predictions can only be bounded for the CE loss.
Proposition 1. The loss, lz. 3) 2/l ly D112 has unbounded gradients w.r.t. y. VV r)ll2 = ly U112 I oo, ÿ = fo(æ).
(3)
Proposition 2. The loss, l(t, 9) = Σi= to log t., i Softmax(() (g) has bounded gradients w.r.t y, (t. 9)112 = ||t Softmax (0)rmaaxx))1222) S v2,
fo()))
Combining Proposition with weight normalization and a Lipschitz assumption, we can upper bound the effective gradient update (Definition 1) in Theorem 1.
Theorem 1. For the cross entropy-loss and learning rate 1 2 for scale-invariant function approximator fe which is Lg Lipschitz continuous in the L2 norm with respect to 0 with fixed parameter norm ||0|| = C. the effective gradient update can be upper bounded as
n||@|12 -
Vol(t, fo())) S 13-C-V2L,
(5)
To analyze second-order properties, we must assume that can bound the eigenvalues of the func- tion approximator's Hessian so that weight decay can ensure the Hessian of the objective is positive definite and the smallest eigenvalue is greater than zero.
Assumption 1. We assume eigenvalue bounds for the function approximator Hessian (per output),
0 < | < (V.( S 12 < Vie m), [ x E X, θ.
Proposition Given Assumption 1, the eigenvalues of the Hessian of the mean squared error loss with weight decay μ²², μ 0 unbounded and the condition number cannot be upper bounded. Proposition 4. Given Assumption 1, the eigenvalues of the Hessian of the cross-entropy loss with weight decay 2 have nounrdb condition number
(V2C) < (4X%, + T + t)c,
c c 0,
when 2 2N + which provides finite upper bound when
0.
For proofs see Appendix J. In practice, we do not require weight decay to attain good performance, and result, the Hessian was observed to not always be positive definite. Nonetheless, these results provide a formal intuition for our empirical result that CE losses consistently report smaller condition numbers. Our analysis directly motivates the design of XQC algorithm, presented next.
5

---

Published as a conference paper at ICLR 2026
4 XQC: A SIMPLE & WELL-CONDITIONED ACTOR-CRITEL ARCHITECTURE
This section presents our novel XQC algorithm, a direct conclusion of our optimization analysis in Section 3. It is a simple, yet powerful archi- tecture with the purpose of improving the loss landscapes optimization behaviours, that extends the popular SAC algorithm. xQC's critic archi- tecture is motivated by a central principle: combining components that synergistically improve optimization dynamics. We provide a complete list of hyperparameters in Appendix A.
Input
BatchNorm
Linear
Batch normalization. XQC uses BN layers directly on the network in- put and after each linear layer (Figure 5). Following Bhatt et al. (2024), we implement& a joined forward pass to automatically calculate the BN running statistics on the joined (s, a) and (s', a') distribution (Bhatt et al., 2024), to successfully integrate BN in the RL loop. In contrast to Bhatt et al. (2024), we find that switching the order of normalization and Re ReLU-activation leads to better performance. It has the added bene- fit that in this order, BN's s scale invariance is preserved for any activation function, as opposed homogeneous ones only. XQC uses four hidden layers with 512 neurons each.
BatchNorm
ReLU
xN
Linear
Output
Cross-entropy Bellman loss. We a C51-style categorical critic with 101 atoms and CE loss (Bellemare et al., 2017). We use standard re- ward normalization based on running statisties of the standard deviation of the return R to effectively bound the Q values to the support of our categorical critic A = ra/a R)) (Engstrom et al., 2020). Next to im- proving the loss landscape conditioning, another desirable property of the categorical CE loss is to keep gradient norms bounded and thereby help keep the ELR constant.
Figure 5: The xQC net- work architecture con- sists of only three stan- dard components: Lin- ear, BN, and ReLU for total of 4 hidden layers.
Weight normalization. Enabled by BN's scale-invariance property, we project the weights of each dense layer to the unit sphere after each gradient step. This normalization keeps the denominator of the ELR constant, so it becomes practically constant when using the CE loss (Figure 8), so xQc maintains good plasticity. With constant ELR, we can now leverage learning rate schedule for Adam (Kingma & Ba, 2015) as previously suggested (Lyle et al., 2024; Lee et al., 2025b).
Vision encoder. For experiments the DMC vision-based environments, we use the standard DRQ- V2 (Yarats et al., 2022) image encoder. For a fair and direct comparison to DRQ-V2, we use its standard, unmodified vision encoder, which consists of convolutional layers alternated with ReLU activations, followed by a linear layer, LN, and a tanh activation. Our architectural modifications are confined the subsequent MLP layers of the actor and critic.
EXPERIMENTS
This section empirically validates our central hypothesis: that the synergistic combination of bN, WN, and a categorical CE critic loss, designed to create a well-conditioned optimization land- scape, directly translates into state-of-the-art sample efficiency and training stability. We structure our experiments to first demonstrate XQC's superior performance against strong baselines (Sec- tion 5.1), then dissect the underlying mechanics through analysis of common plasticity metrics and the ELR (Section 5.2), Finally, in Section 5.3 we analyze computational efficiency, scaling proper- ties, and present thorough ablation study (Appendix G) confirms the necessity of each of XQC's architectural components.
Evaluation metrics. For the main experiments we run 10 random seeds per environment for mil- lion environment steps and for ablations seeds, unless otherwise noted. For statistically rigorous evaluations, we report the IQM and 90% SBCI for all aggregate scores, following the recommended best practices of Agarwal et al. (2021). To aggregate IQM return curves over multiple environments and benchmarks, each score needs to be normalized. We follow standard practice, details in Ap- pendix B. In aggregated bar charts, we present area under the curve (AUC) of the IQM normalized return curve. The AUC captures both training speed as well as absolute performance simultaneously.

---

Published as a conference paper at ICLR 2026
Humancid Bench
DVC Hord
DMC Mod+Easy
Suite Suite
MuloCo
Aggregated (55 tasks)
0.6
6.8
1.0
0.8
0.4
0,0
AUC
I+
13:00
0.4
0.2
000
022
0.0
00 0211
0:2
G.5
0/R
0.2
0.0
0.0
SAC
BRC
BRO
Cross WN
MR.Q
SimbaV2
XQC (ours)
Figure 6: XQC achieves state-of-the-ar sample efficiency across 55 proprioceptive continuous control tasks. We report the IQM AUC of normalized returns. Error bars denote 90% SBCIS. The right column shows total aggregated performance across the benchmarks (55 tasks). XQC matches or outperforms strong baselines, especially on the hardest DMC and HB tasks, while using a simpler and smaller architecture (see Section 5.3).
DMC vision
DMC vision Medium
humanoid-mee
humanoid
humanoid stand
TQM ATCC Retur
0.6
IQM Return
0.4
200
500
500
0.2
8S5
0:62
0.0
XQC
DiQ-V2
0
2
0
2
0
1
2
Env Steps (10M)
Env Steps (10M)
Emy Steps (10M)
Figure 7: XQC improves sample efficiency on 15 vision-based DMC tasks. The left two columns show aggregated IQM AUC, demonstrating significant performance advantage over the strong DRQ- V2 baseline, particularly on the difficult humano tasks. The right three columns show full train- ing curves for the three humanoid d environments (IQM over 10 seeds), highlighting XQC's signifi- cantly better sample-efficienyy. For these experiments, XQC uses the standard DRQ-V2 encoder and hyperparameters, isolating performance gains to proposed well-conditioned critic architecture.
As such, it discriminates between two algorithms, which converge to the same performance but at different speeds, opposed to the IQM of the final performance.
Benchmarks. To validate xQC's effectiveness, we conduct comprehensive experiments across 70 continuous control tasks spanning five popular benchmark suites. Our evaluation covers 15 vision- based tasks from DMC, plus and additional 55 proprioceptive tasks from HumanoidBench (H3) (Sfer- razza et al., 2024) (14 tasks), DMC (Tassa al., 2018) (25 tasks), MyoSuite (Myo) (Caggiano al., 2022) (10 tasks), and MuJoCo (Todorov et al., 2012) (6 tasks). Our extensive evaluation shows XQC's generality, using a single set of hyperparameters across all tasks.
Baselines. We compare to several strong, recent model-free baselines: SIMBA-V2 (Lee et al., 2025b), BRO (Nauman et al., 2024), MRQ (Fujimoto al., 2025), BRC (Nauman et al., 2025), CROSSQ+WN (Palenicek al., 2025), SAC (Haarnoja et al., 2018), When available, we use the spective authors' evaluation results; otherwise, we run experiments using their official open-source implementations. Full details baseline results are provided in Appendix C.
5.1 SAMPLE EFFICIENCY RESULTS
We start our experiments by investigating the training performance in terms of sample efficiency and comparing it to state-of-the-un baselines. All of these results use 10 seeds per environment. First, we present the proprioception- based results and then the rision-based tasks.
Reinforcement learning from proprioception. Figure shows that XQC matches our outperforms strong baselines SIMBA-V2, MRQ, BRO and CROSSQ+WX on 4 benchmarks. The rightmost col- umn shows that on average XQC performs as well SIMBA-V2 while using significantly less network parameters and a substantially simpler architecture Figure 5. Notably XQC shows exceptional per- formance the most complex tasks HB and DMC-hard. These enm induce notoriously difficult and ill-conditioned optimization landscapes. xQC's superior performance and learning speed sug- gest that its well-conditioned critic-chiie characterized by stable ELR and bounded gradients as shown in Section 5.2 is fundamentally better equipped to handle the nta-stationary targets and bootstrap- ping emrors inherent in these challenging domains.
7

---

Published as a conference paper at ICLR 2026
Parameter Norm
Gradient Norm
BLR
Stable Kank
Feuture Norm
200
0.10
40
0,002
1.0
350
180
160
XQC (ours)
100
0.05
0.001
300
140
XQC
20
0.5
250
120
XQC MSE
XQC w/o WN
0.0
0.5
1.0
0.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Eav Steps (1M)
Env Steps (1M)
Env Steps
Eav Steps (1M)
Env Steps (IM)
Figure 8: XQC's architecture creates exceptionally stable learning dynamics. For XQC, BN +WN stabilizes the parameter norm, while BN +CE keep the gradient norm and ELR near constant.
Reinforcement learning from pixels. On the vision-based DMC environments, we compare XQC to DRQ-V2. For these results, we re-implemented XQC in the official DRQ-V2 codebase. We used the DRQ-V2 encoder and the same hyperparameters the original DRQ-V2 V2 to make the comparison as fair as possible. Figure shows that on most tasks, XQC outperforms or least matches DRQ-V performance. This is most pronounced on the much more challenging humanoid tasks. Learning the vision-encoder from scratch requires a large number of samples itself. We hypothesize that this is why the performance increase of XQC is smaller on the easier tasks and much more pronounced on the humanoids, which have a 10x overall runtime.
5.2 PLASTICITY ANALYSIS
Analysing the improvement common plasticity metric confirms the effectiveness of XQCs design principles. Figure 8 presents plasticity metrics aggregated over all 55 proprioceptive tasks. XQC w/o WN's growing parameter norms decrease the ELR towards zero over time, reconfirming the findings of Lyle et al. (2024) and Palenicek et al. (2025). We notice that the ELR appears directly coupled to the gradient norm, for all architectures employing wN. While XQC MSE controls the parameter norm, its gradients are unbounded and heavily influenced by outliers; consequently, its gradient norm and ELR grow over the course of training by about one order of magnitude (requiring second y-axi compare). xQC's CE critic loss removes this disturbance, directly reflected in remarkably stable gradient norm and ELR throughout the course of training, which are many orders of magnitude smaller. We show per benchmark plasticity metrics in Appendix F.
5.3 PARAMETER AND COMPUTE EFFCIENCY
XQC achieves competitive sample efficiency while requiring 4.5x fewer parameters than SIMBA- V2 (Figure 9). This parameter efficiency directly results high computational efficiency with ~ 5x fewer FL.O//S than SIMBA-V2 and BRO and 100x fewer FL.PP/S than BRC. We conjecture that xQC's superior computational efficiency is rooted its well-conditioned architecture. With respect practical performance, XQC shows 60% higher frames-per-second (FPS) during training, measured on the MoCoCo humanoid d on a NVIDIA RTX 4090 workstation. resulting in significantly faster wall-clock times. XQC's evaluation performance measured 1355FPS. slightly lower the SIMBA- v2 baseline due to its slightly larger policy networks, but remains more than fast enough typical control frequencies seen in deep reinforcement learning (10-50 Hz).
Purams
Train (Humanoid)
Eval (Humanoid)
35M
400
1600
100R
300
1000
10B
200
100
500
2M
XQC
BRO
MrQ
SimbaY2
BRC
Figure 9: XQC is significantly more parameter and compute efficient than competitive base- lines. 4.5x fewer parameters and 5x fewer FLOP/S than SIMBA-V2 and BRO and > 100x fewer FLOP/S than BRC. The computational efficiency translates into a significantly higher training FPS measured on the MuJoCo humanoid environment an NVIDIA RTX 4090 workstation.
8

---

Published as a conference paper at ICLR 2026
1QM AUIC AUC Return
UTD Sealing (55 tusks)
Width Sealing (55 tasks)
Depth Sealing (35 tasks)
XQC Ablations (55 tacks)
0.8
0.6
0.2
0.4
004
1000
0.71
6.1
0.5
0.0
8
10
256
512
1024
6
LN (1.5c4)
LN (3-4)
MSE
wib WN
BS 128
BS 512
AR
200
BRO
SimbaV2
XQC
XQC Skip
Figure 10: XQC scales stably with UTD, network depths, and widths. Columns 1-3: xQc's scaling in terms of UTD, layer width, and layer depth. Increasing compute and model capacity generally improves or maintains performance, demonstrating that well-conditioned design is robust to scaling. Results are aggregated across all 55 proprioception tasks and 5 seeds each. Right column: We perform multiple architectural and parameter ablations with respect the full xQc algorithm. XQC is robust to changing the action repeat (AR=1), introducing weight decay (WD), and different batch sizes (BSE {128, 512)). We further ablate all three of xQC's main components, first without WN (w/o WN), second replacing the CE loss with an MSE loss (MSE), and third replacing BN with LN. Each component's s removal results in a significant performance drop, showing their synergistic contribution.
5.4 XQC SCALING BEHAVIOUR AND ARCHITECTURE ABLATIONS
Figure 10 colums 1-3 demonstrate xQC's scaling ability. XQC improves performance with increas- ing UTD at a similar slope to SIMBA-V2. with a fraction of the parameters (Section 5.3), Similarly, XQC improves or maintains performance with larger and deeper networks. These results demon- strate robustness towards different hyperparameters and XQC's ability to scale stably, enabled by its well-conditioned architectural design. This property is desirable since increasing compute always represents a trade-off between sample efficiency and wall-clock or energy. It allows practitioners to use their available budget most efficiently. For depth scaling, we also ablate the influence of skip connections but find no significant effect for the investigated depths. Additionally, we investigate the influence of different batch sizes E {128, 512] compared to the default of 256, the use of weight decay, and the impact of action repetition. Neither of these ablations shows significant influ- ence on xQc. Appendix presents additional conditioning experiments for the tested components. To demonstrate the synergy between the proposed architectural components, three main components of the proposed XQC architecture: BN, WN, and the CE distributional critic. Results for all 55 and 5 seeds are shown in Figure 10 with per benchmark ablations in Figure 22. Our analysis confirms that each of the components is vital, especially in the most difficult DMC-hard and suits. The largest drop in performance occurs when switching BN for LN. We further investigate Ln with half the learning rate, accounting for the increased ELR of LN (See Figure 8); however, this leads to even worse performance. This in line with our finding that LNN-ased architectures show lower performance due to worse conditioning. While using a MSE critic loss has the second most significant influence, the removal of WN still substantial, but shows the lowest overall impact. In summary, combining each of the three components is vital for XQC's performance.
ablate each of the
RELATED WORK
Our work is positioned the intersection of several key research areas in deep RL that all attempt to improve sample efficiency. A prevailing trend for improving sample efficiency has been scal- ing along different axes. Compute scaling, particularly increasing the UTD ratio, has recently been a major focus. However, naively increasing UTD can lead to instability and overfitting on early experience (Nikishin al., 2022). Researchers have suggested many different regularizers sta- bly increase the UTD; From full parameter-resett (Nikishin et al., 2022; D'Oro et al., 2022). to critic ensembles (Chen et al., 2021) and drop-out (Hiraoka et al., 2021), to normalization layers like LN (Hiraoka et al., 2021; Lyle et al., 2024; Nauman et al., 2024; Lee al., 2025a), BN (Bhatt al., 2024), hyper-spherical normalization (Hussing et al., 2024; Lee et al., 2025b) and spectral normal- ization (Bjorck al., 2021). Recently, works have found that the combination of normalization layers together with WN can stabilize the ELR, helping against loss of plasticity (Lyle et al., 2024) and also enable scaling RT. high UTD ratios (Palenicek al., 2025). Recent works have worked

---

Published as a conference paper at ICLR 2026
on developing scaling laws for RL (Rybkin et al., 2025; Fu et al., 2025). Network scaling. is another path authors are exploring to increase sample efficiency. Bhatt et al. (2024) showed that BN allowed them to significantly scale the layer width. Since then, authors have looked into specific LN-based architectures (Nauman et al., 2024; Lee et al., 2025a) and network sparsity (Ma et al., 2025). Lee al. (2025a) propose 'simplicity bias' score, computed using FFT and scoring "simplicity" higher for functions with lower frequency content across random initializations. This score has no theoretical justification relating it to sample efficiency generalization. Another line of research attempts to scale network sizes dynamically during training (Liu et al., 2025; Kang et al., 2025). Concurrent work Castanyer al. (2025) combine second-order optimization and multi-skip residual connections to improve scaling and monitor the trace of the Hessian for deep palue-based RL. Or- thogonal to our work, recent methods revisit the target network design itself. Vincent al. (2026) share features between online and target networks to reduce memory while preserving stability, and Hendawy et al. (2026) take the minimum both estimates for bootstrapping to accelerate learning. Model-based methods scale computation by learning separate dynamics model, which is leveraged in the RL loop (Janner et al., 2019; Hafner et al., 2020; Palenicek et al., 2023; Hansen et al., 2024).
7 CONCLUSION & FUTURE WORK
In this work, shifted the focus from the prevailing pure scaling goal in deep RL and instead fo- cus on improving the critic's optimization landscape. Through an eigenvalue analysis of the critic's Hessian, demonstrate that specific architectural choices, namely batch normalization, weight normalization, and distributional cross-entrope loss, create a better optimization landscape with condition number orders of magnitude smaller during learning. This superior conditioning translates directly into learning performance gains. We propose XQC, an algorithm embodying these prin- ciples, which achieves state-of-ther of-tte-art sample efficiency across 70 continuous control tasks from proprioception and vision domains. XQC accomplishes this performance with significantly fewer parameters than competing methods, underscoring that a principled focus on optimization funda- mentals can yield greater performance and efficiency than brute-force scaling alone. For future work, insights from XQC can be used to accelerate actor-critic model-based and imitation learn- ing algorithms, c.g., Janner et al. (2019), Watson et al. (2023). XQC performance could be further accelerated by incorporating prior datasets, e.g., Ball et al. (2023).
ACKNOWLEDGEMENTS
This research was funded by the research cluster "Third Wave of AI", funded by the excellence pro- gram of the Hessian Ministry of Higher Education, Science, Research and the Arts, hessian.Al and by the Deutsche Forschungsgemeiisccaatt (DFG, German Research Foundation) under Germany's Excellence Strategy (EXC-3057/1 "Reasonable Artificial Intelligence". Project No. 533677015). was further supported by UKRI/EPSRC Programme Grant [EP/V000748/1] and partially sup- ported by the German Federal Ministry of Research, Technology and Space (BMFTR) under the Roboties Institute Germany (RIG).
REPRODUCIBILITY STATEMENT
We took special care ensure this work is reproducible and will make the code open source upon acceptance. To ease reproducibility, algorithm details are explained Section 4, all hyperparameters listed in Appendix A, and training curves are shown Appendices D and E.
LARGE LANGUAGE MODEL USAGE
large language model was helpful in polishing writing, improving reading flow, and identifying remaining typos.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in Neural Informa- fion Processing Systems, 2021.
Aleksandr Y. Aravkin, James V. Burke, and Dmitriy Drusvyatskiy. Convex analysis and nonsmooth optimization. 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv:1607.064500 2016.
arXiv preprint
Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learn- ing with offline data. In International Conference on Machine Learning (ICML), 2023.
Mare G. Bellemare, Will Dabney, and Rémi Munos. A distributional perspective on reinforcement learning. International Conference on Machine Learning, 2017.
Mare G Bellemare, Will Dabney, and Mark Rowland. Distributional reinforcement learning. MIT Press, 2023.
Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. CrossQ: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In International Conference on Learning Representations, 2024.
Nils Bjorek, Carla Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning with spectral normalization. In Advances in Neural Information Processing Systems, 2021.
Bernhard G. Bodmann. Matrix theory, math6304: Variational characterization of eigenvalues.
Lecture Notes, 2012. URL https: htt////www.. www.math. math. ub. htthedd//bbbb/ce wtwpb//////uuusseeeeee MMate6304//
MatrixTheory- wttpl22212111111111
Nico Bohlinger, Jonathan Kinzel, Daniel Palenicek, Lukasz Antezak, and Jan Peters. Gait in eight: Efficient on-robot learning for omnidirectional quadruped locomotion. In International Confer- ence on Intelligent Robots and Systems, 2025.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: composable transformations Python+NumPy programs, 2018. URL http:
/ //github. com/ google/jaa
Vittorio Caggiano, Huawei Wang. Guillaume Durandau, Massimo Sartori, and Vikash Kumar. Myosuite-a a contact-rich simulation suite for musculoskeletal motor control. In Learning for Dynamics and Control Conference, 2022.
Roger Creus Castanyer, Johan Obando-Ceron Lu Li, Pierre-Luc Bacon, Glen Berseth, Aaron Courville, and Pablo Samuel Castro. Stable gradients for stable learning at scale in deep rein- forcement learning. In Advances in Neural Information Processing Systems (NeurIPS), 2025.
Xinyue Chen, Che Wang. Zijian Zhou, and Keith Ross. Randomized ensembled double Q-learning: Learning fast without a model. In International Conference on Learning Representations, 2021.
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In International Conference on Learning Representations, 2022.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. In International Conference on Learning Representations, 2020.
Jesse Farebrother, Jordi Orbay. Quan Vuong, Adrien Ali Taiga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, et al. Stop regressing: Training value functions via classification for scalable deep RL. In International Conference on Machine Learning, 2024.
11

---

Published as a conference paper at ICLR 2026
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Preston Fu, Oleh Rybkin, Zhiyuan Zhou, Michal Nauman, Pieter Abbeel, Sergey Levine, and Aviral Kumar. Compute-optimal scaling for malue-based deep RL. arXiv preprint arXiv:2508. 14881, 2025.
Scott Fujimoto, Pierluca D'Oro, Amy Zhang, Yuandong Tian, and Michael Rabbat. Towards general-purpose model-free reinforcement learning. In International Conference Learning Representations, 2025.
Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao. An investigation into neural optimization via Hessian eigenvalue density. In International Conference on Machine Learning, 2019.
Gene H Golub and John H Welsch. Calculation of Gauss quadrature rules. Mathematics of compu- tation, 221106)):112300) 1969.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Schoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. arXiv preprint arXiv: / 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
Nicklas Hansen, Hao Su, and Xiaolong Wang. Td-mpe2: Scalable, robust world models for contin- uous control. In International Conference on Learning Representations, 2024.
Ahmed Hendawy, Henrik Metternich, Théo Vincent, Mahdi Kallel, Jan Peters, and Carlo D' D'Eramo. Use the online network you can: Towards fast and stable reinforcement learning. International Conference Learning Representations, 2026.
Takuya Hiraoka, Takahisa Imagawa, Taisei Hashimoto, Takashi Onishi, and Yoshimasa Tsuruoka.
Dropout fufucciins for doubly efficient reinforcement learning. In International Conference Learning Representations, 2021.
Marcel Hussing, Claas A Voelcker, Igor Gilitschenski, Amir-massoud Farahmand, and Eric Eaton. Dissecting deep RL with high update ratios: Combatting value divergence. In Reinforcement Learning Conference, 2024.
Ehsan Imani and Martha White. Improving regression performance with distributional losses. In International Conference Machine Learning, 2018.
Sergey loffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning. 2015.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model- based policy optimization. In Advances in Neural Information Processing Systems, 2019.
Zilin Kang, Chenyuan Hu, Yu Luo, Zhecheng Yuan, Ruijie Zheng, and Huazhe Xu. A forget-and- grow strategy for deep reinforcement learning scaling continuous control. In International Conference on Machine Learning, 2025.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In International
Conference on Learning Representutions, 2015.
Jens Kober and Jan Peters. Policy search for motor primitives robotics. Advances in Neural Information Processing systems, 2008.
Hojoon Lee, Dongyoon Hwang, Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subramanian, Peter R Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters in deep reinforcement learning. International Conference on Learning Represen- fations, 2025a.
12

---

Published as a conference paper at ICLR 2026
Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyper- spherical normalization for scalable deep reinforcement learning. International Conference on Machine Learning. 2025b.
Lin Lin, Yousef Saad, and Chao Yang. Approximating spectral densities of large matrices. SIAM review, 58(1):34-65, 2016.
Jiashun Liu, Johan Samir Obando Ceron, Aaron Courville, and Ling Pan. Neuroplastic expansion in deep reinforcement learning. International Conference on Learning Representations, 2025.
Ilya Loshchilov, Cheng-Ping Hsieh, Simeng Sun, and Boris Ginsburg. ngpt: Normalized trans- former with representation learning the hypersphere. In International Conference Learning Representations, 2025.
Clare Lyle, Zeyu Zheng, Khimya Khetarpal, James Martens, Hado van Hasselt, Razvan Pascanu, and Will Dabney. Normalization and effective learning rates in reinforcement learning. In Advances in Neural Information Processing Systems, 2024.
Guozheng Ma, Lu Li, Zilin Wang, Li Shen, Pierre-Luc Bacon, and Dacheng Tao. Network sparsity unlocks the scaling potential of deep reinforcement learning. International Conference on Machine Learning, 2025.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miloš, and Marek Cygan. Big- ger, regularized, optimistic: scaling for compute and sample-efficient continuous control. In Advances in Neural Information Processing Systems, 2024.
Michal Nauman, Marek Cygan, Carmelo Sferrazza, Aviral Kumar, and Pieter Abbeel. Bigger, regu- larized, categorical: High-capacity value functions are efficient multi-task learners. arXiv preprint arXiv:2505.23110, 2025.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning, In International Conference on Machine Learning, 2022.
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer, 2006.
Daniel Palenicek, Michael Lutter, João Carvalho, and Jan Peters. Diminishing return of value expan- sion methods in model-basee reinforcement learning. In International Conference on Learning Representations, 2023.
Daniel Palenicek, Florian Vogt, Joe Watson, and Jan Peters. Scaling off-policy reinforcement learn- ing with batch and weight normalization. Advances in Neural Information Processing Systems, 2025.
Oleh Rybkin, Michal Nauman, Preston Fu, Charlie Victor Snell, Pieter Abbeel, Sergey Levine, and Aviral Kumar. Value-based deep RL scales predictably. International Conference on Muchine Learning, 2025.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Hu- manoidbench: Simulated humanoid benchmark for whole-body locomotion and manipulation. In Robotics: Science and Systems, 2024.
Richard Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2018.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Ried- miller. Deepmind control suite. arXiv preprint arXiv: 1801.00690000 2018.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems, 2012.
Twan Van Laarhoven. L2 regularization versus batch and weight normalization. arXiv preprint arXiv: 00061000655555 2017.
13

---

Published as a conference paper at ICLR 2026
Théo Vincent, Yogesh Tripathi, Tim Faust, Yaniv Oren, Jan Peters, and Carlo D' Eramo. Bridging the performance gap between target-free and target-based reinforcement learning with iterated Q-learning In International Conference on Learning Representattios, 2026.
Joe Watson, Sandy H. Huang, and Nicolas Heess. Coherent soft imitation learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning. In International Conference on Learning Representations, 2022.
A HYPERPARAMETERS
Table summarizes all proprioception based experiments' hyperparameters.
T 1 [0.95, 0.995)
7 = clip
T
where I' denotes the effective episode length, calculated by dividing the episode length by the num- ber of repeated actions. We use the default hyperparameters used in their respective GitHub repos- itories for all other baselines. One exception BRC, where we reduced the number of parameters from the default 256M to 64M. As noted by the authors, using more than 64M parameters does not provide additional benefit the single-task setting, which is the focus of our work.
Table 2 contains all hyperparameters for the vision-based experiments based on the official DRQ-V2 codebase.
14

---

Published as a conference paper at ICLR 2026
Dense(dim)
IN Activation
MuloCo:
0.0003
0.0003
Adam
0.0001
L
DMC:
HB: 2
Myo: 2
MRQ
512
512
1.0
250
0.99
256
Skip Connection
Dense(dim)
MuJoCo: I
Table 1: Hyperparam expperprameters for XQC and baselines on all proprioception tasks.
BRC
LN
ReLU
Dense
LN
0.0003
2048
0.0003
256
0.1
0.0003
|AI /2 0.005
0.99
Adam W
0.0001
[-10, 10] CE
1024
DMC: HB:
Myo: 2
2
Skip Connection
2 BRO Small
Dense(dim)
MuJoCo:
BRQ
LN
ReL
Dense
LN
0.0003
512
0.0003
256
1.0
0.0003
|A| /2
0.005
0.99
Adam
0.0001
MSE
BRO
128
DMC:
HB:
Myo: 1
2
2 on MuloCo HB
on DMC / Myo
Dense(4 dim)
(ensc(dim)
SIMBA-V2 V2
Sealer
ReLU
1.2 Norm
LERP
L2 Norm
0.0001
0.0001
0,0001
A AA/2 /2
0.005
Heuristic Adam
[-5.5]
MuJoCo:
DMC: 2 HB:
Myo: 2
512
128
0.01
CE
256
2
Dense( dim) ReLU
Heuristic Adam
MuloCo:
0.0003
0.0003
0.0003
IAI |A//2 //2 0.005
DMC:
Myo: 2
SAC
256
256
MSE
256
HB:
CROSSQ+WN
Dense(dim) ReLU BN
Heuristic Adam 0.01
MuJoCo:
0.0003
0.0D03
0.0003
|AI /2
0.005
MSE 2
DMC: 2
Myo: 2
512
256
0.01
256
HB:
2
Dense (ense(dim) dim)
MuloCo: 1
Heuristic Adam
BN ReLU
0.0003
0.0003
0.0003
IAI/2 /2 0.005
[-5, 5] CE
DMC:
HB: 2
Myo: 2
XQC
512
256
0.01
256
Target network update frequeney
Temperatur learning rate
Target network momentum
Critic hidden dim Critic number of blocks
Actor number blocks
Actor learning rate Actor number blocks
Policy update delay
Categorical Support
Hyperparameter
Initial temperature
Number of crities
Block design
Critic learning
Target entropy
Weight Decay
Action repeat
Discount
Optimizer
Critic loss
Batch size
UTD
15

---

Published as a conference paper at ICLR 2026
Table 2: Hyperparameters for vision-based RI. tasks.
Hyperparameter
XQC
DRQ-V2
Dense(dim)
Dense(dim)
Block design
BN
ReLU
ReL.U
Critic learning rate
0.0001
0.0001
Critic hidden dim
1024
1024
Critic number blocks
Actor learning rate
0.0001
0.0001
Actor hidden dim
1024
1024
Actor number of blocks
4
3
Target network momentum
0.01
0.01
Target network update frequency
Number of critics
2
2
Discount
0.99
0,99
Optimizer
Adam
Adam
Categorical Support
[-5. 5]
Critic Loss
MSE
UTD
0.5
0.5
Batch size
256
256
Replay buffer capacity
IM
1M
N-step returns
Feature dim
50
50
Exploration stddev. clip
0.3
0.3
Action Repeat
2
2
easy: linear(1.0 0.1, 100K)
easy: linear([.0, 0.1, 100K)
Exploration stddev. schedule (Defined by task difficulty)
medium: linear(1 1.0, 0.1, 500K)
medium: linear( 1.0, 0.1, 500K)
hard: linear( I.O, 0.1, 2M)
hard: linear( 10, 0.1, 2M)
B ENVIRONMENT AGGREGATION DETAILS
As the magnitude of returns varies across environments, normalize them for comparability be- fore aggregating (Agarwal et al., 2021). We normalize scores to be between 0 and 1. Where the normalization protocols are benchmark-specific and follow standard practice.
For MuJoCo and Humanoidbench, we compute the normalized score as
Random Score
a Target Score Random Score'
where the random scores are obtained using uniformly random policy (Fu et al., 2020). Target scores are taken from trained TD3 policy in MuJoCo, and are provided by the authors for HB, where they represent the threshold required mark task solved.
For DMC tasks, we normalize by dividing the final score by 1000, the maximum achievable return.
MyoSuite tasks require no normalization, performance is already expressed in percentage-based success rates.
16

---

Published as a conference paper at ICLR 2026
C BASELINES
In this section, we briefly describe how the results were collected for every baseline we present in this work. Additionally, all hyperparameters are listed in Appendix A.
SIMBA-V2 (Lee et al., 2025b). We used the results made publicly available on the official GitHub repository. The results are based on seeds. For SIMBA-V2 (small) we ran the code from the official codebase ourselves, for 10 seeds.
CROSSQ+WN (Palenicek et al., 2025). We ran all experiments ourselves using our codebase for 10 seeds.
BRO (Nauman et al., 2024). We used the publicly available results on the official SIMBA-V2 GitHub repository. We only considered the 'small' version of BRO, which uses a UTD ratio of 2. The results are based on seeds.
MRQ (Fujimoto et al., 2025). For MuJoCo and DMC, we used the results provided by SIMBA-V2 on their official GitHub repository, which are based on 10 seeds. We conducted experiments for Myo and HB ourselves, by running the official MRQ codebase using random seeds due to computational reasons. We matched the action repeat used in our experiments ensure fair comparison.
SAC (Haarnoja et al., 2018). We ran all experiments ourselves using our codebase. We used the default SAC hyperparameters and ran seeds for every environment.
BRC (Nauman et al., 2025). Using the official BRC GitHub codebase, we experiment ourselves for seeds, due to computational constraints. While we used their default settings as reported the paper, reducing the parameters to 64M, since Nauman et al. (2025) noted that using more than 64M parameters provides no benefit for single-task settings. Additionally, we used the same action repeat environment wrapper used in all our other experiments, ensuring fair comparison.
DRQ-V2 (Hiraoka et al., 2021). We used the results reported in the offical GitHub repository based on 10 seeds. Our vision-based XQC experiments are based the DRQ-V2 codebased for fair comparison.
17

---

Published as a conference paper at ICLR 2026
D ALL TRAINING CURVES: REINFORCEMENT LEARNING FROM VISION-BASED DMC ENVIRONMENTS
Results from the vision-based DMC benchmarks. 2022).
We compare DRQ-V2 (Yarats al.,
humanoid-run
acrobot.swingup
finger.tum_easy
quadruped.rum
reacher_easy
1000
750
1000
400
500
500
500
200
250
humanoid-wall cartpolc. sparse finger.tum. hard
quadruped walk
reachcr-hard
IQM Norm Return
1000
1000
750
500
300
500
500
500
250
0
0
humanoid-stie stand
cheetah_run
hopper.hop
reach_duplo
walker_run
400
200
600
300
200
100
2
2
x10
X109
X10
X10
x10
Env Steps
Env Steps
Env Steps
Env Steps
Eny Steps
DrQ-V2
XQC (ours)
Figure 11: XQC and DRQ-V2 training curves for each of the 15 vision-based DMC tasks. the IQM and 90% SBCI aggregated over 10 seeds per environment.
We show
humanoid-mu
acrobot
finger Inger.tum.casy easy
quadruped run
reacher.easy
1000
750
1000
400
200
200
500
500
500
250
0
humanoid-wall
aartpole.swnngupsppee finger.tum hard
quadruped.walk
reacher
1000
1000
750
Mean Return
500
500
500
500
500
250
humanoid-stand
chectah run
hopper hop
reach reach
walker run
400
200
500
500
200
100
0
2
2
x100
x10
x10
x11
X10
Env Steps
Env Steps
Env Steps
Env Steps
Eny Steps
DrQ.V2
XQC (ours)
Figure 12: XQC and DRQ-V2 v2 training curves for each the 15 vision- basod DMC tasks. We show the mean and 90% SBCI aggregated over 10 seeds per environment.
18

---

Published as a conference paper at ICLR 2026
E ALL TRAINING CURVES: REINFORCEMENT LEARNING FROM PROPRIOCEPTION
All results for the proprioception continuous control benchmarking tasks with IQM and mean.
E.1
HUMANOIDBENCH
h1-pole-v0
h1-slide-vO)
h1-stair-vO
h1-crawl-v0
800
600
1000
400
600
400
400
200
0
0
0
h1 h1-maze-vo
h1-run-vO
h1-reach-v0
h1-walk-vO
500
800
400
6000
600
200
300
4000
400
200
100
100
2000
200
IQM IQM Norm Retur
01
h1 --stand-vO
h1 hurdle-vO
h1 ---bbalacchharr
h1 balance.simple-v
800
250
600
100
150
400
100
50
200
50
h1 -arddvO
h1 i1l-si- simplee0
800
600
400
400
200
200
0
0.0
0.5
0.5
Env Steps 10
Env Steps 1
SimbuV2
BRC
BRO
XQC (ours)
MR.Q
Figure 13: XQC and baseline training curves for each of the 14 HB tasks. We show the 1QM and 90% SBCI aggregated per environment.
19

---

Published as a conference paper at ICLR 2026
h1 hl-pole-vO
h1l-side-vO
stair--
h1 hl-crawl-vO
800
000
1000
800
600
400
100
100
200
200
200
200
h1 --mae-vv)
h1 hl-run-vO
hl bl-reach-vt)
h1 b1-walk-vO
500
300
400
300
4000
400
200
100
100
2000
200
0
0
0
Mean Return
h1 h1-stand-v0
h1 h1-hurdle-vO
h1 0
h1 sil-balance.simen salance.simple-v0
800
250
150
600
200
600
150
400
400
100
50
200
200
h1 -h-sii-hard-v0
h1 simple-s
800
800
600
600
100
100
200
200
0
0.0
0.5
1.0
0.0
0.5
1.0
Env Stepsc 109
Env Stepsk 10
SimbaV2
BRC
BRO
XQC (ouns)
MR.Q
Figure 14: XQC and baseline training curves for each of the 14 HB tasks. We show the mean and 90% SBCI aggregated per environment.
20

---

Published as a conference paper at ICLR 2026
E.2
DEEPMIND CONTROL SUITE
dog-stand
dog-walk
dog-trot
dog-run
humanoid-stee -tand
1000
600
500
500
400
500
200
0
humanoid walk
humanoid run
acrobot
acrobot-swingup swingup
cartpole-ball balance
cartpole cartpole-swingup
750
600
1000
750
200
500
400
200
250
0
cheetab run
Lnnge-sppin
finger-turn easy
finger tum hard
fish-swim
1000
1000
1000
IQM Norm Return
500
500
250
0
hopper-hop
hopper-stand
pendulum-swingup
quadruped-run
quadruped-walk
1000
1000
1000
300
750
200
500
500
100
250
0
0
reacher-easy
reacher-hard
walker-run
walker-stand
walker-walk
1000
1000
1000
750
500
500
500
250
0
0
Env Suepo
Eny Steps
Env Steps"
Env Stepe
Env Stepo "
BRC
XQC (ours)
MRQ
Figure 15: XQC and baseline training curves for each of the 25 DMC tasks. We show the IQM and 90% SBCI aggregated environment.
21

---

Published as a conference paper at ICLR 2026
dog-stand
dog-walk
dog-trol
dog-run
humamoid-stand
1000
600
500
500
400
500
200
0
humanoid-wall
humanoid-me run
acrobot-swingup
cartpolebbalanee
cartpole-swingup
750
200
750
500
100
500
250
200
0
cheetab-run
finger-spin
finger-turn.com casy
finger-tum hard
fish-swim
1000
1000
1000
Mean Return
500
500
500-
250
0
0
hopper-hop
hopper-stand
pendulum-swingup
quadruped-run
quadruped-walk
1000
1000
1000
300
750
200
500
500
500
100
250
0
0
reacher-easy
reacher-hard
walker-run
walker-stand
wa walker-walk
1000
1000
750
500
500
500
500
250
0
0
Env Steps*
Env Steps
Env Steps
Env Steps
Env Stepo" "
SimhuV2
BRC
BRO
(ours)
MR.Q
Figure 16: XQC and baseline training curves for each of the DMC tasks. show the mean and 90% SBC1 aggregated per environment.
22

---

Published as a conference paper at ICLR 2026
E.3 MYOSUITE
myo-pen-twirl-hard
myo-key-lurn-han
myo-obj-hold-harr
myo-pose-hard
1.0
1.0
0.03
0.5
0.00
0.0
0.0
0.0
00.05
myo-reach-hee
myo-pen-twirl
myo-key-tum
myo-obj-hold
IQM Norm Return
0.5
0.5
0.5
0.5
0.0
0.0
myo-pose
myo-reach
1.0
0.5
0.5
0.0
0.5
0.0
0.5
1.0
Env Stepsk 1
10
Env Stepsk 10
SimhaV2
BRC
BRO
XQC (ours)
MR.Q
Figure 17: XQC and baseline training curves for each of the 10 Myo tasks. We show the IQM and 90% SBCI aggregated per environment.
anyo-pen-twing twirl-hard
myo-key- turn- hard-
jyyo-obj-hold-bee
myo-poss--Ble pose-hard
1.0
1.0
0.5
0.5
0.00
0.0
0.0
0.0
0.05
myo-reach-hardd
myo-pen-twirl
myo-key-tun
myo-obj-hold
1.0
1.0
Mean Return
0.5
0.5
0.5
0.5
myo-pose
myo-reach
1.0
1.0
0.5
0.5
0.0
0.0
Env Steps 10
Env Stepsk 10"
SimbaV2 MR.Q
BRC
XQC (ouns)
Figure 18: XQC and baseline training curves for each of the 10 Myo tasks. We show the mean and 90% SBCI aggregated per environment.
23

---

Published as a conference paper at ICLR 2026
E.4 MuJoCo BENCHMARK
HalfCheetah--ee
Hopper-v4
Ant-v4
Walker2d-v4
8000
6000
M
6000
10000
3000
4000
5000
2000
1000
2000
IQM Norm Return
Humanoid-v4
HumanoidStandup-v4
400000
10000
300000
200000
5000
100000
0.0
0.3
1.0
0.5
1.0
Env Steps 10
Env Steps 109
SimhaV2 MR.Q
BRC
BRO
XQC (ours)
Figure 19: XQC and baseline training curves for each of the 6 MuJoCo tasks. We show the IQM and 90% SBCI aggregated per environment.
HalfCheetah-v4
Hopper-v4
Ant-v4
Walker2d-v4
8000
M
6000
10000
3000
1000
1000
5000
1000
2000
2000
Mean Return
Humanoid-va
HumanoidStandup-ve
10000
300000
200000
100000
Env Steps 10
Env Steps 10
SimbaV2 MR.Q
BRC
XQC (ouns)
Figure 20: XQC and baseline training curves for each of the MuJoCo tasks. We show the mean and 90% SBCI aggregated per environment.
24

---

Published as a conference paper at ICLR 2026
F
PLASTICITY METRICS
Parameter Norm
Gradient Norm
ELR
Stable Rank
Feature Norm
0.0010
400
200
dmme _.hard
0.04
0.05
350
150
0.0005
800
250
0.00
0.12
200
0.06
0.0010
200
dmc .easy
300
0.0005
0.10
150
200
0,0000
0.15
0.003
150
hb
0.10
0.002
100
2.3
300
100
0.001
0.05
0.000
50
200
0.6
200
0.010
250
myyo
0.1
375
0.005
200
350
0.2
100
150
0.0015
mujoco
0.06
10000
0.0010
200
300
1000
0.04
5000
0.0005
100
500
0.02
200
0.0000
0.002
180
Aggragated
0.10
350
40
160
0.001
300
100
120
0.000
Environment Steps (IM)
XQC LN
XQC MSE
XQC WN
XQC
Figure 21: Per benchmark plasticity metrics for XQC and architectural ablations.
25

---

Published as a conference paper at ICLR 2026
G ARCHITECTURE ABLATIONS
dmc hard
dmc_easy
hb
myo
mujoco
Aggregated
IQM AUC AUC Return
0.6
0.8
0.6
0.15
0.4
0.6
0122
01,2
0.4
0.5
0.2
0.0
0.0
XQC
XQC
XQC (ours)
Figure 22: Ablation study confirms the necessity of all three of XQC's components. We compare the full XQC algorithm against three variants: one replacing BN with LN (xQC LN), one replacing the CE loss with an MSE loss (XQC MSE), and one without WN (xQC w/o wN). Each component removal results in significant performance drop, demonstrating their synergistic contribution.
26

---

Published as a conference paper at ICLR 2026
H XQC UTD SCALING TRAINING CURVES
AUCTQQ Return
dmc_hard
dmc_easy
hb
myy
mujoco
Aggregated
0.8
0.8
0.6
0.6
0.75
1.00
1.5
0.8
0.6
0.4
0.50
1.0
0.4
0.2
0.25
0.3
0.2
0.0
0.0
0.0
0.00
0.0
0.0
12 4
8 10
12 6 10
12 6 10
12 6 8
12 4
8 10
6
UTD
UTD
UTD
UTD
UTD
UTD
BRO
SimbaV2
XQC
Figure 23: XQC stably improves with increased UTD ratios. We compare IQM AUC for xQc trained with UTD ratios E (1, 8, 16). Performance consistently improves with more updates, showcasing the stability of the learning process.
dmc_
dmc_easy
hb
1.00
TQM Norm Return
0.8
0.75
0.75
0.0
0.4
0.2
0.25
0.25
0.0
0.00
0.00
myo
mujoco
Aggregated
1.00
1.5
IQM Norm Return
0.75
0.75
1.0
0.50
0.5
0.25
0.25
0.0
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Fny Steps
100
Fny Steps
109
Eny Steps
100
UTD
UTD2
UTD
Figure 24: UTD scaling. We present the area under the curve of the IQM Norm Return. This measure captures fast and stable learning simultancously.
27

---

Published as a conference paper at ICLR 2026
I ADDITIONAL CONDITIONING RESULTS
Here we present additional conditioning results for alternative architectural components requested by the reviewers during the rebuttal. We ablate combining XQC with skip connections (with net- work depth in E {4, 6J. different batch sizes E {128, 256, 512] and weight decay le - 4. Further, we investigate how depth scaling influences conditioning for dense MLPs with number of layers {2, 4, 6}. The chosen architectural ablations on top of XQC do not meaningfully influence condi- tioning or performance. The overall finding in line with our main finding, that good conditioning correlates with good RI. performance.
IQM
IQM max(A)
IQM Kurtosis(A)
10a
10?
10a
MSE + Demse
10'
XQC skip
XQC butch size
10a
10
XQC weight
109
104
XQC (ours)
250
500
750
250
500
750
250
500
750
IQM Return @ IM
IQM Return @ IM
IQM Return @ IM
Figure 25:
The condition numbers and maximum eigenvalues against the return at 1M steps on DMC dog-trot. Architectures with lower condition numbers and lower maximum eigenvalues tend to have better final returns.
28

---

Published as a conference paper at ICLR 2026
J THEORETICAL ANALYSIS
The section details the proof' for bounding the gradient norms and Hessian condition numbers. Lemma 1. For the loss L(O, D) = I(Y, fx())), if f is L Lipschitz in the L.2 norm with respect O, the L2 norm of the gradient has the following upper bound,
VooC( y, a1)122 s b |1V fo(2)))l22
(6)
Proof. Using the chain rule and the
inequality,
[VoC((,, y, x)ll2 s V fo(x))) [[Vofo(=))]l s (ll. Lg-
Proof of Proposition 1. Standard calculus.
Proof of Proposition 2. Standard calculus, and then using the difference of two categorical proba- bility vectors (on a simplex) bound the largest squared error as 2.
Proof of Theorem I. Combine Lemma 1, Proposition 2, and the constrained parameter norm for the definition of the gradient update in Definition to obtain upper bound.
Lemma 2. For a symmetric mutrix E Rmxm with ranked eigenvalues X S.... < XA then the
eigenvalues of the sum of two such matrices C -A+ then X + + S X2 and A + AB 2 X This result holds for all finite sums.
Proof. Weyl's theorem applied to the sum of two Hermitian matrices (Bodmann, 2012).
The mean squared error loss, Uy Ú) + yllyyz 1.2. IRd has constant Hessian and therefore constant Hessian eigenvalues X,
Proposition 5.
V2lly, = las Aud
(8)
Proof. Standard calculus.
Proposition 6.
The cross entropy loss, I(y, 8) = -a
-im1 ya log Da has the following Hessian and
eigenvalue bounds given the model ÿ Softmax( fo (x)) where yi 2
Vy fo()) = diag( ÿ) Úū O5AS1,
as Σ'_ = 1,0 Vi S 1. The Hessian is singular due to the loss of degree-of of-freeom in categorical probabilities.
Proof. Standard calculus.
For Proposition 3 and 4, we use that the objective's Hessian can be decomposed using the chain rule,
VjC(0, Vofo(z)
Vofo()) V Vofol)) = (x) ( x. y)go (x) + g: v) He(æ).
(10)
Proof for Proposition 3. The first term of Equation 10 has rank of I it's an outer product, and g( x) g(0, x) = |lg| < Lz. so its eigenvalues X; E [0, D3]. Using Assumption 1, the eigenvalues of the second term are bounded by [-m[g] mmlggmmmx// Vy, mlgg Manx As the gradient ele- ments cannot be upper bounded (i.e., Proposition 1), the Hessian of the loss has eigenvalue range Vy, [p² m 2 2m [g] max V which leads the largest eigenvalue + oc and the the case that the smallest eigenvalue is 0 when adding eigenval- from both terms due to Weyl's theorem (Lemma 2).
unbounded condition number due to both
29

---

(see previous Published as a conference paper at ICLR 2026
Proof Proposition 4. The first term in Equation 10 has eigenvalue bounds [0, Li] proof). It's positive semi-definite so we know O is a lower bound on the eigenvaluse. Since the max eigenvalue non-zero, we know the Frobenius norm of g greater or equal than the trace of the outer product, and the trace is also the sum of eigenvalues, zero eigenvalue by Lzz The second term in Equation 10 has range A E [-2X 2 as they are bounded by [- Σg Di X E, 19:1 X H and 0 2 19sl 2 1, > 9i = 0. With Weyl's theorem (Lemma L If 2t2 2X + c, then we have À E [c, 42 + + 13),
can bound the (largest) non-
ml 2) we have À E 112 2A m- J + 2X so
4N + 13 +E
which concludes the proof. Unsurprisingly, the upper bound is only finite if the regularization sures positive definiteness of the objective's 's Hessian.
30


Reviews:
[{"review_id": "p0rTOO8rwW", "reviewer": "Reviewer_UPAo", "summary": "This paper proposes XQC, a simple modification of SAC that targets the optimization landscape of the critic. The key design is the synergistic use of batch normalization at every MLP layer, weight normalization via post-update unit-sphere projection, and a categorical distributional critic trained with cross-entropy. The authors argue this trio yields (i) bounded gradients, (ii) a nearly constant effective learning rate, and (iii) substantially better Hessian conditioning than typical alternatives. Empirically, XQC reports state-of-the-art sample efficiency across 55 proprioceptive and 15 vision DMC tasks, using fewer parameters and lower stated FLOP/s than strong baselines.", "questions": "- XQC’s MLPs do not include residual/skip connections, while some baselines do. Given residuals’ impact on conditioning, it would be useful to justify their omission and/or report a small ablation.\n- The bounded condition number result for CE relies on weight decay to enforce positive definiteness, yet the best empirical runs often avoid weight decay. The paper acknowledges this; still, a short empirical probe on “with vs without WD” vs conditioning would tighten the loop.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Originality: Reframes sample efficiency through loss-landscape conditioning with BN, WN, CE and a principled rationale.\n- Quality: Extensive experiments (10 seeds in most main results), scaling studies (UTD/width/depth), and insightful plasticity metrics.\n- Clarity: The paper is easy to follow, the role of each component is well articulated, figures are generally informative.\n- Significance: If the compute/latency picture is corroborated, XQC can influence default choices for actor–critic architectures.", "weaknesses": "- Reporting parameters and FLOP/s is helpful, yet the current figures seem potentially misleading. For a fair comparison, training/inference speed should be addressed to complete the real-world efficiency story.\n- For pixels, the authors state that XQC keeps the vision encoder unchanged for fairness. That isolates MLP effects, but it also under-tests the hypothesis: BN/WN/CE improvements could plausibly extend into the conv encoder. This limits the reported vision gains.\n- Figure 7 highlights sample efficiency but 10M buffer size is quite unrealistic in various setups. A 1M buffer setup would make the point under practical budgets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1pnGUyhTUs", "reviewer": "Reviewer_dowu", "summary": "This paper proproes XQC, studying how to make critic optimization in deep RL better conditioned by systematically revisiting three ingredients—normalization layers, weight normalization/projection, and the critic loss—and ties them together through a Hessian-based analysis. It shows that, contrary to common practice favoring LN, BN yields local loss landscapes with dramatically smaller condition numbers during learning; WN leverages scale invariance to stabilize the effective learning rate (ELR); and a categorical cross-entropy (CE) distributional loss induces a better-conditioned objective than MSE, with a bounded Hessian condition number under mild assumptions. Building on these findings, the authors propose XQC, a simple SAC extension that combines BN, WN, and a CE Bellman error, arguing the trio acts synergistically to improve conditioning and stabilize ELR, which in turn boosts sample efficiency. Extensive experiments on 55 proprioceptive and 15 vision control tasks report state-of-the-art results against larger, more complex baselines, supporting the claim that improving optimization fundamentals can outperform mere scaling.", "questions": "See above.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Principled theory that links design choices.** The paper gives a clear Hessian-based account of why BN (vs. LN), CE (vs. MSE), and WN jointly improve critic optimization—showing BN yields much smaller condition numbers, CE produces a better-conditioned objective, and WN stabilizes the effective learning rate; it even derives bounded eigenvalue ranges/condition numbers under mild assumptions. \n\n**Thorough and careful empirics at scale.** Results span 70 tasks (55 proprioceptive + 15 vision), use solid aggregate metrics (IQM AUC with 90% SBCIs, 10 seeds), and consistently show SOTA sample efficiency—including fair pixel-based comparisons by re-using DRQ-V2 encoders/hyperparameters—and diagnostic analyses that tie gains to stabilized ELR and gradients.", "weaknesses": "**Limited originality**. Similar ideas have been proposed in previous works, like BatchNorm in CrossQ (ICLR 2024); Replacing MSE with CE loss in EfficientZero (Neurips 2021), Stop Regression (ICML2024), BRC (Neurips 2025). It seems that this paper just analyzed the same problem from another perspective but reached a similar conclusion.\n\n**Limited scope**. As mentioned before, similar conclusions have been proposed in previous work, and this paper seems to be an ensemble of them. However, it would be interesting if the authors could expand the scope of this paper to include more settings, such as model-based RL, to investigate the general efficacy of the proposed architecture rather than constraining the scope to only the actor-critic and model-free settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xAK2d92yBq", "reviewer": "Reviewer_ATvv", "summary": "This paper introduces XQC, a simple actor-critic architecture that improves sample efficiency in deep reinforcement learning through well-conditioned optimization. Rather than relying on model scaling or complex architectures, XQC achieves stability and efficiency by combining batch normalization, weight normalization, and a categorical cross-entropy loss, which together yield lower Hessian condition numbers and bounded gradient norms. \n\nTheoretical analysis and large-scale empirical results - spanning 70 continuous control tasks from proprioceptive and vision-based benchmarks - demonstrate that this synergy leads to superior training stability, plasticity, and state-of-the-art performance while using significantly fewer parameters and compute than prior methods such as BRO or SIMBA-V2.", "questions": "1. Why do the authors use an action repeat wrapper in the humanoid bench? The humanoid bench uses joint position control, which does not make much sense with action repeat. Because of how joint position commands work, repeating that same position command multiple times has no new effect (ie. the robot just keeps tracking the same target position, no?)\n2.  [4] found that BRO performance significantly changes depending on the action repeat setting. Did the authors test their method without action repeat on benchmarks other than Mujoco?\n3.  Is it expected that BRC underperforms BRO? To the best of my understanding, they differ only by using categorical loss instead of MSE. Considering the presented results on the importance of the categorical loss, shouldn't BRC perform similarly to BRO?\n4. Do authors have a hypothesis as to why the performance of their approach decreases when using more layers?\n5. BRO and SimBa use residual architectures. Could the authors comment on why they opted for standard MLP here? Do authors think that it might matter for more complex environments?\n6. It would be interesting to see how DrQ+BRO or DrQ+SimBa performs as compared to DrQ+XQC, considering that BRO and SimBa papers did not include vision-based DMC experiments. Do authors think that these methods would also perform worse than XQC in vision experiments?\n7. Could the authors summarize the shortcomings of their theoretical analysis? What are the differences between the assumptions and real-world problems?\n8. It would be interesting to see how scaling improves conditioning on its own as compared to using bn + wn + categorical loss.\n9. Batch normalization is interesting because the batch statistics depend on the sampled batch (and the variance of these statistics depends on the batch size). To this end, it would be interesting to see how robust XQC is to different batch size settings and whether the increased variance improves the optimization landscape.\n\nNitpicks:\n- line 48 update-to-date -> update-to-data\n- line 099 bellmann -> bellman\n- line 331 propioceptive -> proprioceptive\n\n\n[1] Palenicek et al. Scaling CrossQ with Weight Normalization\n[2] Lee et al. Hyperspherical Normalization for Scalable Deep Reinforcement Learning\n[3] Nauman et al. Bigger, Regularized, Categorical\n[4] Voelcker et al. MAD-TD: Model-Augmented Data stabilizes High Update Ratio RL", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Studying architectures that allow for scaling actor-critic is timely and important\n- The proposed method is simple and seems to offer a more computationally efficient alternative to scaling via BRO or SimBa\n- I appreciate the analysis comparing different normalization strategies", "weaknesses": "- Limited novelty. The work combines puzzle pieces from previous works (batch norm + weight norm was used in [1], weight norm + categorical loss was used in [2], layer norm + categorical loss was used in [3]). I still appreciate showcasing that this particular combination seems to be most effective for scaling actor-critic.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j0sHiJiVV6", "reviewer": "Reviewer_79k1", "summary": "The paper starts with a systematic evaluation of three design choices in RL (normalization layer, MSE-CE loss, and weight normalization), based on the smoothness of the critic loss curvature. Their results suggest that the combination of batch normalization, CE loss, and weight normalization yields the smoothest loss curvature and thus stable optimization process. Theoretical insights are additionally provided on why CE loss generates much more stable gradients than standard MSE loss.\n\nBased on the results, authors propose a simple architecture design that encorporates the three components. With significantly smaller number of parameters compared to prior works, their architecture shows competitive or even superior performance in diverse benchmarks comprising up to 55 tasks. Their results also extend to vision-based RL, where the architecture leads to significant improvement of DrQ-v2, especially in challenging humanoid environments.\n\nFinally, their ablation results reveal that switching on of the components lead to degradation of performance, validating their design choices.", "questions": "- I believe comparing the actual train/inference speed (FPS) with prior SOTAs would also be meaningful, as it directly reflect the speed expected in practice, and may differ from FLOPs depending on aspects such as number of layers.\n- To my understanding, IQM was used to remove outlier tasks in Atari benchmark [1], and it feels unnatural to use IQM for learning curves of individual tasks. I don't necessarily go fully against using IQM, but I suggest the authors provide average results for both individual learning curves and aggregate results along with IQM.\n- In Figure 8, XQC LN seems to have twice as much ELR as XQC. Would it be possible that this difference caused the performance drop shown in Figure 9 (right)? Have you tried decreasing the base learning rate to $1.5e-4$, half of default value of XQC?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- **Well-motivated design choices**. Authors have verified their design choices based on thorought and systematic evaluation of the smoothness of the critic loss curvature.\n- **Strong performance across diverse benchmarks**. The proposed architecture shows strong results compared to prior SOTAs in total 55 tasks.", "weaknesses": "- I only see minor questions and weaknesses; please see the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)