Simplicial Embeddings Improve Sample Efficiency in Actor–Critic Agents
Authors: Johan Obando-Ceron
Walter Mayor
Samuel Lavoie
Scott Fujimoto
Aaron Courville
Pablo Samuel Castro
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
deep reinforcement learning
actor critic
representation learning
state embeddings
Primary area: reinforcement learning
TL;DR: We propose the use of simplicial embeddings in actor-critic methods to improve sample efficiency and final performance, without sacrificing runtime.

Abstract: Recent works have proposed accelerating the wall-clock training time of actor-critic methods via the use of large-scale environment parallelization; unfortunately, these can sometimes still require large number of environment interactions to achieve a desired level of performance. Noting that well-structured representations can improve the generalization and sample efficiency of deep reinforcement learning (RL) agents, we propose the use of simplicial embeddings: lightweight representation layers that constrain embeddings to simplicial structures. This geometric inductive bias results in sparse and discrete features that stabilize critic bootstrapping and strengthen policy gradients.
When applied to FastTD3, FastSAC, and PPO, simplicial embeddings consistently improve sample efficiency and final performance across a variety of continuous- and discrete-control environments, without any loss in runtime speed.

Published as a conference paper at ICLR 2026
SIMPLICIAL EMBEDDINGS IMPROVE SAMPLE EFFICIENCY IN ACTOR-CRITIC AGENTS
Johan Obando-Ceron 12 Walter Mayor Samuel Lavoie
Scott Fujimoto" 1.4
Aaron Courville Pablo Samuel Castro
"Equal contribution 'Mila- Québec Al Institute Université de Montréal
IIndependent Researcher *McGill University
SCIFAR Al Chair
[jobandoC730, wa waltermayor, Lamuel.lavois 1emm com (courvila) la, pablo- www.sa.maaea..aa..e...ee....... waw.el.castaoooooooooooooooco @mi quebee
scott. fujimoto@ma wwt.eeemmam..mm.m1..m...m mmg.ll..ca..
ABSTRACT
Recent works have proposed accelerating the wall-clock training time of actor- critic methods via the use of large-scale environment parallelization: unfortu- nately, these can sometimes still require large number of environment interactions achieve desired level of performance. Noting that well structrrec representa- tions can improve the generalization and sample efficiency of deep reinforcement learning (RL.) agents, we propose the use of simplicial embeddings: lightweight representation layers that constrain embeddings to simplicial structures. This ge- ometric inductive bias results in sparse and discrete features that stabilize critic bootstrapping and strengthen policy gradients. When applied to FastTD3, Fast- SAC, and PPO, simplicial embeddings consistently improve sample efficiency and final performance across variety of continuous- and discrete-control envi- ronments, without any loss runtime speed. Source code here.
"Order is not imposed from the outside, but emerges from within ".. Ilya Prigogine
INTRODUCTION
Deep reinforcement learning (deep RL) has delivered impressive progress continuous control, enabling agile locomotion (Smith et al., 2022; Zhuang et al., 2023; Margolis et al., 2024) and dex- terous manipulation (Popov et al., 2017: Akkaya et al., 2019; Luo et al., 2025). Yet a persistent tension remains between fraining speed (wall-clock efficiency) and sample efficiency (the number of environment interactions). Some modern agents such as TD-MPC2 (Hansen et al., 2024) and SR-SPR/BBF (D' Oro et al., 2022; Schwarzer et al., 2023) achieve strong returns with relatively few interactions, but demand substantial compute and engineering complexity. In contrast, recent fast actor- critie variants have scaled throughput with massive parallelization (Li al., 2023; Singla et al., 2024; Gallici et al., 2025; Seo et al., 2025). While methods such as FastTD3 (Seo et al., 2025) rapidly solve humanoid benchmarks, they require far more interactions reach compara- ble performance. Similar limitations have been observed in Parallel Q-Learning (Li et al., 2023) and scage actor-critic frameworks such as IMPALA and SEED RL (Espeholt al., 2018: 2020). This trade-off limits applicability in domains where interactions are expensive and time is constrained, such as robotics.
A natural objection is that, in modern simulators, environment steps cheap and can be generated in massive parallel batches, so sample efficiency is less important. However, this view overlooks
"This perspective resonates with deep RL: stability cannot be forced solely through more compute, heavier regularizers, or larger critics. Instead, inductive biases that shape the genmetry of representations can allow order emerge from within, leading stable critics and more efficient policies under non-stationarity.

---

Published as a conference paper at ICLR 2026
several practical and scientific concerns. First, algorithms that are data-hungry in simulation rarely transfer well to real-world scenarios (Tobin et al., 2017; Akkaya et al., 2019). Second, large-scale parallelization requires substantial compute and energy resources, raising both efficiency and sus- tainability concerns (Schwartz et al., 2020; Henderson et al., 2020). Third, sample efficiency is closely tied to generalization: agents that exploit structure from fewer trajectories tend to be more bust under distributional shifts (Zhang et al., 2018; Yao et al., 2025). Moreover, in high-dimensional simulators such as Isaac Gym, each step can be significantly more expensive, compounding ineffi- ciency as tasks grow harder (Makoviychuk et al., 2021; Rudin et al., 2021). These issues highlight why sample efficiency remains central even in the era of massively parallel deep RL.
Shaping representations with auxiliary losses (Anand et al., 2019; Laskin et al., 2020; Schwarzer et al., 2021; Castro et al., 2021; Fujimoto et al., 2023) has been shown to improve sample efficiency in deep RL. However, such methods increase algorithmic complexity and add computational over- head through extra forward and backward passes (Fujimoto et al., 2023). Alternatively, architectural components, such as convolutions (Fukushima, 1980; LeCun et al., 1989) and attention (Bahdanau et al., 2016), can be used to induce structure leading to desirable downstream properties.
Discrete and sparse representations have several desirable properties in comparison to their dense and continuous counterparts. Notably, sparse and discrete representations increase robustness to noise (Donoho et al., 2006), training stability by reducing catastrophic interference (Liu et al., 2019), sample efficiency (Fumero et al., 2023), interpretability (Murphy et al., 2012: Lavoie et al., 2023; Wabartha & Pineau, 2024) and improved generative modeling (Lavoie et al., 2025). In this work, we posit that several of those properties are beneficial in the context of reinforcement learning. Within RL, several successful agents like TD-MPC2, Dreamer V2/V3, IQRL, and MAD-TD (Hansen al., 2024; Alonso et al., 2024; Hafner et al., 2025; Scannell et al., 2024; Voelcker et al., 2025) use discrete latents; yet they typically rely on auxiliary model- based losses shape the representation.
While several methods exist for learning discrete representations explicitly (Jang et al., 2017: Mad- dison et al., 2017; van den Oord et al., 2017). these methods use straight-througe estimation (Bengio et al., 2013) which is a biased gradient estimator. Fortunately, discretization may be implicitly induced via Simplicial Embeddings (SEM) (Lavoie et al., 2023), an architectural component that partitions latent representation into sequence of L simplices. SEM is fully differentiable, thus avoiding the negative effect of explicit discretization while enacting some the desirable properties of discrete and sparse representations. Concretely, show that SEM improves both data efficiency and asymptotic performance across diverse environments such as Isaac Gym (Makoviychuk et al., 2021), HumanoidBench (Sferrazza al., 2024), and the Arcade Learning Environment (Bellemare et al., 2013), while preserving (and often improving) wall-clock speed.
PRELIMINARIES
2.1 ACTOR-CRITIC REINFORCEMENT LEARNING
We consider a standard Markov decision process (MDP) defined by the tuple M (S, P.r,. with state space S. action space A, transition distribution P(s'|s,a) a), reward function r S x A R. and discount factor 7 E [0. 1). The objective is to maximize the expected discounted return (()) = F, Ee (r((s) ac) where the agent follows a policy <a]ss). Actor-critic methods maintain both a parameterized policy Te (a(s) (the actor) and an action-value function Qp(s, a) (the critic). The critic is trained to minimize the Bellman error
Lo(d)= ((s,,.,.'''>> (
=)
y - B (5', a')],
(1)
where o= denotes target network parameters and D is replay buffer. The actor is updated via the policy gradient defined as (TJ( = Eg~D,a>sg V log (a)s)) Qo(s,)) While this can be effective, bootstrapped training is notoriously fragile. Errors in Qø propagate propa recursively through the target y. and when the representation used to compute Qs is poorly conditioned, these errors amplify and cause divergence or collapse (Fujimoto et al., 2018).
A recent line of work has sought to reduce the wall-clock cost of actor critic training. FastTD3 (Sco et al., 2025) builds on TD3 (Fujimoto et al., 2018) by leveraging (i) parallel simulation across many
2

---

Published as a conference paper at ICLR 2026
environment instances, (ii) large-batch critic updates, and (iii) algorithm design choices like distri- butional critics (C51) (Bellemare et al., 2017), noise scaling and clipped double Q-learning (CDQ) (Fujimoto et al., 2018). Together, these design choices enable high-throughput training while re- taining stable convergence, although FastTD3 (Seo et al., 2025) still remains sample-inefficient (see App. C for more details).
Policies and critics often rely on latent representations extracted from raw states (Lesort et al., 2018) 8)-
(s) which are then Formally, an encoder fa : + I& maps observations into embeddings Some methods share a consumed by either the critic, the actor, or both depending on the architecture. common encoder across actor and critic (e.g., SAC (Haarnoja et al., 2018), DrQ (Yarats al., 2021), DrQ-v2 (Yarats et al., 2022)), while others (c.g., DDPG (Lillicrap et al., 2015), FastTD3 (Seo et al., 2025)) maintain separate encoders. Regardless of parameter sharing. these representations play a central role in learning (Garcin et al., 2025). The critic estimates values Qo(s,a) a) = (f (s), a), and the actor conditions its policy xg (a(as)) =\t T fo (s)) on the chosen embedding. Ideally, should preserve the Markov property and expose predictive features of the reward and dynamics P.
Yet the choice and stability of such embeddings is far from guaranteed. When unconstrained, learned representations can introduce severe pathologies that destabilize value learning. For example, if (s) + 30, critics may extrapolate to arbitrarily large Q-values outside the support the re- play buffer, inflating the Bellman error. Formally, if Qa(2, = w b with linear heads, then 1Q as ||=|| + oc, leading to exploding targets y and divergent gradients. Similarly, if z exhibits strong correlations or degenerate directions, the critic's 's regression problem becomes ill- conditioned: the covariance matrix Σ E[22" 1 may approach singularity, amplifying variance in cemporal-differncee updates. These phenomena are empirically linked to representation collapse, where value estimates drift irrecoverably and policy updates follow unstable gradients (Moalla al.,
2024: Castanyer et al., 2025).
2.2 SIMPLICIAL EMBEDDINGS
Simplicial embeddings (SEM; Lavoie et al., 2023) provide a lightweight inductive bias on repre- sentation geometry by constraining latent codes to lie on product of simplices. Concretely, given encoder outputs (s) E RRLVV the latent vector partitioned into groups of size V, and a
softmax is applied within each group:
exp(Zt,e/T)
28,0
VL E (L, u V).
(2)
=V= exp(2e.//7)
where - >> 0 is a temperature parameter controlling the degree of sparsity. The resulting embedding 2 lies in the product space A x--x.... - .i..,, L categorical distributions of dimension V. This transformation ensures boundedness through group-wise normalization, induces sparsity as softmax competition (sharpened at low 7) drives near one-hot encodings, and promotes group structure by partitioning features into modular subspaces akin to mixtures-of-expee (Shazeer et al., 2017; Ceron et al., 2024c; Willi et al., 2024). In self-supervised learning and downstream classification, SEM has been shown to stabilize training and improve generalization, particularly in low-label and transfer settings (Lavoie et al., 2023). SEM does not rely on auxiliary losses or reconstruction terms; akin to activation function, it only modifies the embedding geometry with the group-wise softmax, limiting computational overhead.
NON-STATIONARITY AMPLIFIES REPRESENTATION COLLAPSE
Several works have shown that non-stationarity can lead severe degradation of learned represen- tations across different domains (Lyle et al., 2022; Kumar et al., 2021a; Lyle et al., 2025; Castanyer et al., 2025). In supervised learning, label noise and distribution shifts can induce representation collapse, where features lose diversity and neurons become inactive (Li et al., 2022; Sokar et al., 2023; Dohare et al., 2024). Similar observations have been made in deep RL: the constantly chang- ing data distribution, induced by evolving policy, exacerbates this phenomenon, often resulting in unstable critics and poor generalization (Nauman et al., 2024a; Kumar et al., 2021a). These studies suggest that collapse is not an isolated pathology of specific architectures, but general failure mode that emerges when training signals are non non-stationary. In App. B we provide a formal analysis that demonstrates the relationship between non-stationarity and neuron dormancy.
3

---

Published as a conference paper at ICLR 2026
Stationary
Nan stalioaryy
Non statinary: SEM
Performnce (loos
2.0-
Effective Rank (T)
1.5
30-
1.0
20
60 100
80
20
60 100
Epochs
Epochs
Epochs
Training dynamies CIFAR-10 with stationary non-stationary targets. In the stationary regime (fixed targets), losses decrease smoothly, neuron dormancy and effective rank remains controlled, suggesting stable representation learning. In the non-stationary regime (targets shuffled every 20 epochs), the model exhibits higher variance in losses, increased dormant neuron rates, and reduced effective rank. The addition of SEM mitigates this instability. Experiments averaged over 3 independent seeds, with shaded areas reporting 95% confidence intervals.
Fig. 1:
We illustrate this phenomenon with a toy experiment on CIFAR- 10 (Krizhevsky, 2009). We compare two training regimes: (i) stationary setting with fixed labels, and (ii) non-stationary setting where labels are periodically shuffled to mimic the bootstrap dy- namics of deep RI. (Castanyer al., 2025). Let (ar, y) be training samples with y E (1....: : K. In the stationary regime, targets are fixed, so the conditional distribution p(y|a) ) is constant and the em- pirical risk minimizer 07 remains stable up to stochastic fluctuations. In the non-stationary regime, labels are periodically shuffled that y (v)), where πe is a permutation applied every T steps. This induces inflection points in the minimizer, shifting whenever ne changes. Fig. shows that in the stationary regime, training is stable: losses decrease smoothly, dormant neuron rates remain low, and effective rank increases, indicating robust representation learning (Dohare et al., 2024; Sokar et al., 2023; Liu al., 2025c). In contrast, in the non-stationary regime, observe insta- bility: oscillating losses, rising neuron dormancy, and collapsing feature rank. Even in this simple supervised setting, instability in the target distribution alone is sufficient to undermine representa- tional integrity. Similar optimization instabilities have been observed in prior work when evaluating CIFAR- 10 under non- con-stationary conditions (Igl et al., 2021; Lee al., 2023; Galashov al., 2024).
A demonstration on CIFAR-10.
Stabilizing Representations under Non-Stationarity with SEM Simplicial Embeddings (SEM) can mitigate this effect by projecting features onto a structured space that prevents collapse. The transformation enforces energy preservation; since each block has unit mass, representations can- not vanish and tr(E,) remains bounded away from zero. It also promotes diversity, as intra-block competition spreads information across coordinates, while multiple blocks (L) increase effective rank, counteracting covariance deflation. As shown in Fig. I, critics trained with SEM retain higher effective rank, larger gradient energy, and lower neuron dormancy even when targets drift.
Takeaways:
Non-stationarity exacerbates representation collapse, as evidenced by increased neuron dormancy and reduced effective rank.
Simplicial Embeddings (SEM) introduce a simplex-batt based geometric prior that sustains feature diversity and prevent feature collapse.
4 UNDERSTANDING THE IMPACT OF SEM ON DEEP RL NETWORKS
In actor-critic methods such as FastTD3, the critic is trained against bootstrapped targets yt (8,a) a) r(s, a) + Qx (s', (T))))) Both the target distribution D. (samples (s, a, (,,r,,,, 8') from the replay buffer) and the target value ye evolve as the policy अe is updated. This continual drift produces
persistent bias term in b VL++ (0)) (a,,a)>D (Qa(s, a) 1 a)) VaQu a) which is nonzero whenever To or D, changes. Thus, the critic never optimizing a fixed objective but is instead forced chase moving target.
Representation collapse under such non-stationarity poses a fundamental barrier to stable and effi- cient deep RI (see App. A for additional context). Standard actor-critic methods particularly
4

---

Published as a conference paper at ICLR 2026
Actor
Critic
SEM Module
Baseline
ReLu
Linear
Tanh
ReLu
Linear
x<x)
(x,a)
A(Q 4))
Layer
SEM
2:
+ x(x)
(2,4) ++D
H
AQQ a))
Norm
SEM module
SEM module
Softmax
Fig. Actor-critic network architecture with SEM. The actor (left) and critic (middle) architec- tures are modified with SEM module, which partitions features into groups and applies group-wing wwie softmax (right panel), constraining them a product of simplices.
vulnerable. The critic's representations are trained against drifting targets, and the actor in turn depends on those representations to update its policy. This tight coupling amplifies instability, lead- ing to poor sample efficiency in continuous control tasks. To address this challenge, we evaluate Simplicial Embeddings (SEM) as a representation-leve regularizer. SEM aims to encourage the hid- den features of both actor and critic networks to maintain a well-structured geometric organization, preventing collapse and preserving diversity.
Sctup. Because this section involves large number of ablations and is computationally expen- sive, we restrict experiments to five benchmarks from the Humanoid suite (Sferrazza et al., 2024), evaluated on (Seo al., 2025). These tasks share the same robot-state dimension. We report aggre- gate performance across the five tasks and six seeds, with shaded areas reporting 95% confidence intervals, and provide full details in App.
Integrating SEM on Actor-Critic Algorithms. We choose FastTD3 (Seo al., 2025), our primary testbed. FastTD3 is specifically designed to be simple and compute-efficient baseline for contimuous-control and humanoid benchmarks. Its streamlined architecture yields strong perfor- mance while significantly reducing wall-clock training time. At the same time, FastTD3 inherits the critic-driven weaknesses of TD3; its bootstrapped value targets are generated online by the actor, making the critic susceptible to non-stationarity. This coupling amplifies representation collapse, as instabilities in the critic propagate to both value estimates and policy updates. We conduct most of ablations FastTD3, while later sections demonstrate that the benefits of SEM also extend to other actor-critic algorithms, such as SAC (Haarnoja et al., 2018) and PPO (Schulman et al., 2017).
SEM can either be applied to the actor, the critic, both. We build on prior work showing that the penultimate layer plays a critical role in representation quality (Ceron et al., 2024e; Sokar & Castro, 2025), and that regularizing this layer can yield substantial performance gains. Fig. 2 illustrates how SEM is integrated into the actor-critic networks of FastTD3. For the critic, SEM replaces the baseline linear head with a structured projection, regularizing value estimates in the distributional C51 setting. For the actor, SEM is applied at the penultimate layer before the final linear+tanh, ensuring that the policy conditioned on bounded and sparse features. Across the paper, dashed blue (blue, curves indicate the baseline, while solid green, (green, -) curves represent the interventions added to the baseline. Fig. 3 shows clear gains when applying SEM to the actor or to both actor and critic, and more moderate gains when applied only to the critic. Although different SEM dimensions (V) improve sample efficcency and asymptotic performance, V 64 appears most effective, We further explore the relationship between and (see sec 4), as this tradeoff was a central focus of the original SEM study (Lavoie et al., 2023). These results echo the non-stationary CIFAR-10 experiment, where SEM prevented feature collapse and stabilized learning (see Fig. 1).
The Effect of SEM on Learning Dynamics in Deep RL. We empirically evaluate the impact of SEM the stability and efficiency of actor-critic algorithms. Our analysis combines both learning performance (returns, losses, TD error, critic disagreement) and representation quality (effective rank, feature norms), allowing us connect sample-efficiende gains to underlying representational dynamics. This dual perspective highlights not only whether SEM improves performance, but also why it stabilizes training. A detailed explanation of each metric is provided in App. G.
5

---

Published as a conference paper at ICLR 2026
Average Normalized Return
Baseline (+ Actor)
Baseline SEM Critic)
Baseline Actor/Critcc
1.0
Baselin
0.3
(V-16)
SEM (V-64)
0.0
Environment Steps
Environment Steps
"10"
Environment Stops
Fig. 3: Average normalized return on 5 HumanoidBench tasks over seeds. Baseline agent
(blue.--) vs. SEM variants applied to actor, critic, both. Each curve corresponds to an embedding dimension; dim = 64 (green, -) is highlighted. SEM accelerates early learning and improves asymptotic performance, with dim = 64 giving the most stable gains. In the three figures we use L == 2 for the SEM module.
+10
1.5
3.0
1.00
2.5-
1.5
Taxt IDS SEM (Actor)
2.0
0.0
1.54
2.5
1.5
Envirrmentt Steps
Enrinsmor Stess
Snieee men Stepo
Ereiosmant Stess
Enveonment Stepo
Fig. 4: Learning and representation diagnostics on HumanoidBench tasks over 6 seeds. SEM reaches high return earlier, raises actor/critic effective rank, and keeps actor features compact.
Cplaade Fearr
Cati
122-02 meand)
-50
Pattic
224 Decar
-2.2
Ervorreet Endorgrort Stepe
Exvinneer Soes
Fig. 5: Learning dynamics on 2 HumanoidBench tasks. SEM reaches high return faster, with lower losses, smaller TD error, reduced critic disagreement, and better-calibrated value estimates.
To understand why SEM improves performance, we turn to representationn-evel diagnostics. Fig. 4 shows that SEM increases the effective rank of actor features, and bounds actor feature norms. Late in training. SEM also lifts the critic effective rank, signs of more expressive and robust value learn- ing. High effective rank is a proxy for avoiding representational collapse (Moalla et al., 2024; Mayor et al., 2025). In the deep RL literature, representation collapse under drift has been empirically sociated with capacity loss (Lyle et al., 2021), deterioration of feature rank (Kumar et al., 2021b), and implicit pader--arameteiizzation (Kumar al., 2021a). In supervised and self-supervised set- tings, techniques like orthogonality regularization and rank-preserving weight regularizers are used prevent feature collapse (He et al., 2024). These representational patterns align with our for- mal analysis, showing that SEM prevents covariance deflation and sustains gradient energy, thereby preventing feature collapse and boosting performance.
As shown in Fig. 5, SEM improves optimization stability over the baseline. Agents with SEM achieve higher returns earlier and maintain smaller, more stable TD errors, reduced critic disagree- ment, and lower critic-distrbution discrepancy. Such effects are crucial, as instability in boot- strapped critics is a primary failure mode of actor-critic methods (Fujimoto et al., 2019; Kumar et al., 2021a). By constraining representation geometry, SEM produces better-conditioned features that yield more calibrated value estimates, echoing similar findings in representation regularization for deep RL (Anand al., 2019; Laskin et al., 2020; Schwarzer al., 2021). These results indicate that SEM not only accelerates learning but also yields more calibrated value estimates, mitigating instability in bootstrapped critics.

---

Published as a conference paper at ICLR 2026
Edicede Retuan
ree dini
Postrep Gin
SemRim Entrapy Mi0
ACto Adton
03-
+131
1.0
1.0
0.5
0.0
-00"
Stops 413
Fig. 6: Sparsity, entropy, and action std on 2 HumanoidBench tasks. SEM agents achieve higher returns with sparser features, lower entropy, and more stable action scales,
V-
066
08
Suiieereet
EnviermantSieaa K1/7
Envhrammnn Stapo #10
Endrrrmmnn Stgg wse
Fig. 7: Aggregated average return on 5 HumanoidBench tasks. We constrain the encoder's
output of the actor. (left) SEM outperforms alternative methods to impart structure on the encoder's output. In SEM, representation capacity scales with L x V since the embedding consists of L simplex groups of size V. When varying L with fixed V (middle panels), performance improves as L increases from low-capacity regimes, and then saturates once L x V is sufficiently large. When varying V with fixed L. (right panels), we observe the same pattern: for small L. (e.g., I,=1), increasing V noticeably improves performance, whereas for large I. (e.g., L-64), all values of V perform similarly since the model already operates in a high-capacity regime.
In Fig. 6, we focus our lens on the SEM module itself and examine how it shapes representations and action behavior. As training proceeds, the SEM layer's 's activations become markedly sparser (higher Gini (Hurley & Rickard, 2009; Zonoobi al., 2011 I)) and more sharply peaked (lower simplex entropy), while the overall action variance from the policy also declines. This trend is consistent with SEM's design, where the block-wise softmax promotes competition and selective activation. As a result, the module imposes structured, energy-preserving constraints on its layer, encouraging more decisive feature usage and reducing noise in the downstream policy mapping. Interestingly, this pattern also resonates with prior work in deep RL and representation learning. Hernandez- Garcia & Sutton (2019) show that enforeing sparsity in representations can improve robustness and mitigate interference in Q-learning settings. Moreover, recent studies on sparse architectures in deep RL. such find that appropriately structured sparsity can enhance training stability and efficiency (Graesser et al., 2022; Ceron et al., 2024c;b; Ma et al., 2025).
Comparing SEM to other Regularization Methods To contextualize the benefits of simplicial embeddings, we compare SEM to alternative methods to induce structure the encoder's output. We compare SEM to commonly used methods for learning discrete explicit representations: Gumbel straight-through (Jang al., 2017; Maddison et al., 2017) and Vector Quantization (van den Oord et al., 2017). We also compare SEM to C-RELU (Abbas et al., 2023) which have been shown to improve the representation's 's stability. We present the results in Fig. 7 (left) and find SEM to be more efficient and to lead to higher return than alternatiee methods. We conjecture that such improvement over Gumbel ST and Vector quantization can be attributed the fact that SEM does not necessitate the use of the ttriigh-thhouuh estimator.
Analyzing SEM Parameters in Deep RL Lavoie et al. (2023) highlighted the effect of the sim- plex dimensionality V and number of simplices L, which jointly control sparsity and capacity of the representation. Investigating these parameters in deep RL is essential to understand how SEM balances representation capacity and stability under non-stationary training, and whether the same tradeoffs observed in self-supervised representation learning extend to RL. We study the effect of varying V and L. in Fig. 7 (middle and right, respectively). Our results show that performance is
7

---

Published as a conference paper at ICLR 2026
Era
Zastine
Se 2624
Average Normalized
1.0
1.0
1.0
0.5
0.5
0.5
0.3
Steps
Environment Steps
Environment Steps
Averaga Nom ainaleedd Return
No 020
.Dacire
Oiarfauzionll (C20) Divvibus enal (C33)
.Zasdine
Nole scals Nobi acall 2
-sak jetton
-20H juteot
+.2% (uddrt
2.0
1.0
0.5
0.5
0.5
3.0
0.0
0.0
Environment Steps
Environment Steps
Environment Steps
Fig. 8: Effect of core design choices on F3stTD3 with and without SEM on HumanoidBench tasks. SEM solid green, (green, -) consistently improves sample efficiency and asymptotic return across all settings, showing robustness to both hyperparameter and architectural design choices.
driven primarily by the total representational capacity L x V. In low-capacity regimes (c.g. L = 1), increasing V provides clear gains, consistent with the need for additional expressive power. How- ever, once capacity is sufficiently large (e.g.. L = 64), the effect of V largely saturates, different choices of V yield nearly identical performance; and in some cases smaller V (e.g.. V 4) per- forms slightly better. Similarly, increasing L improves performance when L x V is small, but the gains taper off once the model enters a high-capacity regime.
FDstTD3 Design Choices and Simplicial Embeddings. FastTD3 extends TD3 with several de- sign choices that improve throughput and stability, including parallel simulation, large-bator train- ing, and distributional crities (Seo al., 2025). These modifications enable actor-critic learning to scale efficiently in wall-clock time, but they do not address the geometry of the learned representa- tions. In this section, analyze how SEM complements FastTD3 by regularizing representation space and evaluate its effectiveness across the algorithmic design choices. In Fig. 8, we observe that SEM outperforms the bascline even when the agent is trained with reduced data availability (fewer environments, smaller replay buffers, or smaller batch sizes). Comparable gains also appear when algorithmic design choices such as CDQ and C51 are removed. These results demonstrate the robustness of SEM across both data- limited and simplified agent settings.
5 EMPIRICAL EVALUATION
We further evaluate the effectiveness and generality of SEM across diverse set of deep RL al- gorithms and environments. Our study spans both off-policy and on-policy methods, including FasTTD3, FastDD3-SimBav2, FastSAC (Seo et al., 2025), and PPO (Schulman et al., 2017). Ex- periments are conducted on challenging humanoid benchmarks (28-hlhand tasks), (Sferrazza et al., 2024), Isaac Lab (Mittal et al., 2023), Isaac Gym suite (Makoviychuk et al., 2021). MTBench (Joshi et al., 2025), and an extended Atari- 10 setup comprising 28 ALE games (sce D.0.3 for more details) (Bellemare et al., 2013; Aitchison al., 2023; Fedus al., 2020), covering both continuous- control and pixel-based settings. Following prior work (Seo et al., 2025; Castanyer et al., 2025), we evaluate maninuouss-cntrol tasks with six seeds and Atari results with three seeds, and aggregate performance across environments is reported, with shaded areas representing 95% confidence inter- vals. Full environment details and hyperparameter configurations are provided in App.
Fast Actor-Critic Algorithms. We first evaluate SEM on the HumanoidBench benchmark us- and FastSAC (Seo et al.,
ing three recent fast actor-critic baselines: FasTTD3, FastTD3-SimBal 2025). These algorithms represent compute-efficient variants of TD3 and SAC, designed to scale with parallel simulation while maintaining strong performance on high- digee-iinni humanoid con- incorporates hyperspherical normalization and reward scaling to acceler- ate critic training and stabilize optimization (Lee et al., 2025b); and Fa adapts the entropy- regularized SAC framework with similar thrieneput-oriented design choices, achieving high paral-
trol. B
8

---

Published as a conference paper at ICLR 2026
Roucile MM (Ndre))
Saad no
SEM (Acter)
Bacoline SEM (Actrr)
1.00
Nomalizod
0.35
0.50
025
0.00
Envirnmentt Steps
Environment Steps
Environenn Steps
Fig. 9: SEM on fast crior- algorithms. Average normalized return on HumanoidBench with FasTTD3 (left), FastTD3-SimBa (middle), and FastSAC (right). SEM, solid green, (green, consistently improves sample efficiency and yields higher final performance across all algorithms.
-201
Booster
0.3
Episode Return
0.4
Yamulza
0.3
Fast
02-
SEM (Actor)
PPO
SEM (Actor+Critic)
(Actor)
00-
25 30 3.5 4.0
40
65
320
Total Timestaps tin millions)
.39
Environment Staps Lin millions)
Environment Steps
Fig. 10: Performance of PPO with and without SEM across tasks. Atari experiments on an extended Atari-10 setup (28 ALE games; pixel-based) (left). PPO in Isaac Gym (middle). Booster TI (humanoid robot with fixed arms) comparing FastTD3. Applied SEM accelerates learning (right).
lel efficiency while preserving training stability. Across all three baselines, integrating SEM into the actor consistently accelerates early learning and improves asymptotic return. As shown in Fig. 9, SEM agents not only converge faster than their respective baselines, but also maintain lower vari- ance across seeds. These results demonstrate that SEM provides complementary benefits to fast actor-critic methods, enhancing both stability and sample efficiency without modifying their under- lying optimization procedures (see App. J for per-task learning curves). Fig. 10 (right) further shows that SEM improves policy learning on the Booster T1 humanoid robot (Sco et al., 2025), a real- robot benchmark used to validate transfer from large-scale MuJoCo training (Zakka et al., 2025) (see D.0.5 for more details). We also evaluate FastTD3 1 12-h1, 12-g1 tasks and 9-Isaac Gym tasks, where a similar pattern is observed, as shown in App. K.
Proximal Policy Optimization Algorithm. To evaluate the generality of SEM beyond off-policy methods, we integrate it into PPO (Schulman et al., 2017), a popular on-policy method, using the CleanRL implementation (Huang et al., 2022). We evaluate SEM on two distinct benchmarks, Isaac Gym for continuous control and the ALE (Bellemare et al., 2013) for pixel- based discrete control in Atari games. In both domains, SEM improves PPO by accelerating convergence and increasing final returns. The per-environment learning curves shown in Fig. 30. Aggregate results summarized in Fig. 10, with the left panel showing performance gains on the ALE suite?, and the middle panel showing improvements on the Isaac Gym tasks. These results demonstrate that SEM's benefits are not limited to TD3-style critics but extend to policy-gradient methods and vision-based RL, underscoring its broad applicability.
Multitask and Offline-to-onliee Deep RI. Recent work by Joshi al. (2025) introduced large- scale benchmark for multi-task reinforcement learning (MTRL) in robotics. Implemented in Isaac Gym, this benchmark comprises over seventy robotic control problems spanning both manipulation and locomotion, with subsets such as MT50 focused on manipulation. We compare FastTD3 (Seo al., 2025) to its SEM-augmented variants (+SEM). As shown in Fig. 10 (right), +SEM improves sample efficiency, achieving faster learning and higher returns within the same training budget. We also evaluate Flow Q-Learning (FQL; Park et al., 2025b), an actor caaeeerrtiissst style method that couples a value function with a flow-based policy generator trained to transport actions toward high-value regions. Despite differing from standard
updates, FQI. preserves the critic-guided policy improvement structure central to actor-critic algorithms. Applying SEM to the flow-based actor improves sample efficiency and stability during the offline-to-online transition (see Fig. 11),
For the SEM module, use I. = 128 and V = 4 when evaluating ALE games (Bellemare al., 2013) with PPO, whose penultimate fully connected layer has dimensionality 512.
9

---

Published as a conference paper at ICLR 2026
puzzle-4x4
cube-double
X36-
MTBench- Mt50
Eolsode Return
Peroomance
0.6
FaRTTD2
0.4
12
SEM (Actor)
(Criti)
0.2
FQL
SEM (Actor+ Cfttic)
SEM (Actor)
0.0
0
1M
214
1M Steps
2M
1M
2M
Steps
Steps
Environment Steps
Fig. 11: Left: Offiinettoonnliee RL results on 3 OGBench tasks (5 seeds) (Park al., 2025a). Online
fine-tuning starts at IM steps. Right: MTBench MT50 (robotics tasks) comparing FastTD3.
where representations must adapt from static replay data to evolving on-policy distributions. We view these observations opening opportunities to investigate whether simplex-constrained embed- dings can help mitigate representation drift and facilitate adaptation; establishing this more broadly will require further study and is left to future work.
Malue-Based Deep RL. Although our evaluation centers on actor-critic methods, this choice re- flects experimental scope rather than conceptual limitation. SEM operates on learned represen- tations and is compatible with value-based algorithms. Extending SEM to these settings, such as DQN-style architectures (Mnih et al., 2013), is an important avenue for improving sample efficiency. To explore this direction, we evaluated SEM within PQN (Gallici et al., 2025), recently proposed value-based algorithm that simplifies deep temporal-difference learning by streamlining target com- putation and reducing unnecessary complexity. Our preliminary PQN experiments indicate that, while SEM yields improvements in a few games (see Fig. 37), does not provide consistent gains in this regime overall (see Fig. 38). This suggests that a more systematic investigation is needed to determine when geometric constraints benefit value-basee methods. See App. M for more details.
6 DISCUSSION
Our results demonstrate that geometric priors on representation space can substantially improve the efficiency of deep RL agents. By constraining features to product of simplices, SEM yields bounded and sparse embeddings that avoid feature collapse and neuron dormancy under non- stationarity. This lightweight inductive bias requires no auxiliary losses, adds effectively zero com- putational cost (see Table 2), and consistently improves sample efficiency and asymptotic return across actor-critic methods and diverse benchmarks. Unlike existing model-based RL approaches using discrete state embeddings (Hansen et al., 2024; Hafner et al., 2020; 2025; Scannell et al., 2025), SEM does not require auxiliary objectives or additional networks. Surprisingly, its benefits are most pronounced when applied to the actor's 's penultimate layer, where feature geometry most directly shapes policy gradients. Our analyses indicate that SEM alleviates several optimization dif- ficulties in deep RL (Moalla et al., 2024; Juliani & Ash, 2024). By preserving effective rank, bound- ing feature norms, and reducing critic disagreement, SEM provides more reliable gradients and stabilizes the bootstrapped targets that often undermine critic training. These effects highlight rep- resentation geometry as a simple but powerful lever for stabilizing learning under non non-stationarity.
Limitations and Future Work. SEM is not universal remedy. In tasks with extreme distribution shift or very sparse rewards, feature collapse and critic drift may still occur, and SEM introduces hyperparameterrs (L, V. 7) that require light tuning balance sparsity and capacity. For consistency and computational efficiency, we adopted the baseline models' default hyperparameters across all experiments. Nonetheless, RL agents are notably sensitive these choices (Ceron al., 2024a; Patterson et al., 2024), and ideally each experimental setting would undergo a dedicated hyperpa- rameter search, though this is often computationally prohibitive. Moreover, our experiments focus on continuous control and Atari; its impact on large-scale vision or language-conditionel RL re- mains untested. Future work should investigate adaptive schedules for (L, V,T), and integration in more general-purpose algorithms such as MR.Q (Fujimoto et al., 2025), which combine multiple objectives and scale across domains. Another direction is to examine whether SEM benefits value- based algorithms, and to explore both its potential for scaling network architectures (Ceron al., 2024b; Sokar et al., 2025) and its interaction with architectural priors (e.g., MoEs, Residual Nets) (Ceron al., 2024e; Castanyer al., 2025; Kooi al., 2025).
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The authors would like to thank Ali Saheb Pasand and Lu Li for valuable discussions during the preparation of this work. We thank Younggyo Seo for his kindness in answering questions about the FasTTD3 repository. Gopeshh Subbaraj deserves special mention for providing valuable feedback on an early draft of the paper.
The research was enabled in part by computational resources provided by the Digital Research Al- liance of Canada (https: http///aaaaaannaaaaaaaeeeaaaaaaa ca) and Mila (https: ////miii quebec), Pablo Samuel Castro acknowledges funding from NSERC Discovery Grant. We want to acknowledge funding support from Google and CIFAR AI. We would also like to thank the Python community (Van Rossum & Drake Jr, 1995; Oliphant, 2007) for developing tools that enabled this work, in- cluding NumPy (Harris et al., 2020), Matplotlib (Hunter, 2007), Jupyter (Kluyver et al., 2016), and Pandas (McKinney, 2013).
ETHICS STATEMENT
This paper presents work whose goal is to advance the field of Machine Learning, and reinforcement learning in particular. There many potential societal consequences of our work, none which feel must be specifically highlighted here.
REPRODUCIBILITT STATEMENT
We provide all the details reproduce our results in the Appendix.
LLM USE
LLMs were used to assist paper editing and to write the code for plotting experiments.
REFERENCES
Zaheer Abbas, Rosie Zhao, Joseph Modayil, Adam White, and Marlos C Machado. Loss of plasticity in continual deep reinforcement learning. In Conference on lifelong learning agents, pp. 620-636. PMLR, 2023.
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning the edge the statistical precipice. Advances in Neural Informa- fion Processing Systems, 34, 2021.
Matthew Aitchison, Penny Sweetser, and Marcus Hutter. Atari-5: Distilling the arcade learning environment down to five games. In International Conference Machine Learning, pp. 421- 438. PMLR, 2023.
Igge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv:1910.07113 2019.
Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos J Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757-58791. 2024.
Ankesh Anand, Evan Racah, Sherjil Ozair, Yoshua Bengio, Marc-Alexandre Côté, and R Devon Hjelm. Unsupervised state representation learning in atari. Advances neural information pro- cessing systems, 32, 2019.
Jordan Ash and Ryan P Adams. On warm-starting neural network training. Advances in neural information processing systems, 33:3884-383 3894,, 2020.

---

Published as a conference paper at ICLR 2026
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate, 2016. URL https: htapps///axxiiv org/ abs 1409. 0473.
Mare G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning envi- ronment: evaluation platform for general agents. J. Artif. Int. Res., 47(1):253-279) May 2013. ISSN 1076-9757.
Mare Bellemare, Will Dabney, and Rémi Munos. distributional perspective on reinforcement learning. In International conference on machine learning, pp. 449 458. PMLR, 2017.
Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. In International Conference on Machine Learning, pp. 767-777. PMLR, 2020.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation, 2013. URL https: httpp:s///xxxx/o/////// abs/133 /3os/338..o 3432.
Roger Creus Castanyer, Johan Obando-Ceron Lu Li, Pierre-Luc Bacon, Glen Berseth, Aaron Courville, and Pablo Samuel Castro. Stable gradients for stable learning at scale in deep rein- forcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https::///// /wwp//eweeeeeeeeee//e/e/ee net forum? id=Vqj 65VeDOu.
Pablo Samuel Castro, Tyler Kastner, Prakash Panangaden, and Mark Rowland. Mico: Improved representations via sampling-basee state similarity for markov decision processes. Advances in Neural Information Processing Systems, 34:301 13-3012,, 2021.
Johan Samir Obando Ceron, João Guilherme Madeira Araújo, Aaron Courville, and Pablo Samuel Castro. On the consistency of payper--rrameter selection in value-based deep reinforcement learn- ing. In Reinforcement Learning Conference, 2024a. URL https: /ttp::://ppennvvee wetpttiwwwweeeeeee..........ee Forum? Li-uumuuooU
Johan Samir Obando Ceron, Aaron Courville, and Pablo Samuel Castro. In value-based deep rein- forcement learning, pruned network is a good network. In International Conference on Machine Learning, pp.
PMLR, 2024b.
Johan Samir Obando Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Nico- laus Foerster, Gintare Karolina Dziugaite, Doina Precup, and Pablo Samuel Castro. Mixtures of experts unlock parameter scaling for deep rl. In International Conference on Machine Learning. pp. 38520-38540. PMLR, 2024c.
Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. International conference on muchine learning, pp. 1096- 1105. PMLR. 2018a.
Will Dabney, Mark Rowland, Mare Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. Proceedings of the AAAI conference on artificial intelligence, volume 32, 20.8b.
Shibhansh Dohare, Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A Rupam Mah- mood, and Richard Sutton. Loss of plasticity in deep continual learning. Nature, 632(8026): 768-774, 2024.
D.L. Donoho, M. Elad, and V.N. Temlyakov. Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Transactions on Information Theory, 52(1):6-18, 2006. doi: 041109/TTT220558800440
Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Lue Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforeement learning by breaking the replay ratio barrier. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022.
Mohamed Elsayed, Qingfeng Lan, Clare Lyle, and A. Rupam Mahmood. Weight clipping for deep continual and reinforcement learning. In Reinforcement Learning Conference, 2024. URL https: //tp:////ppevvvvv/ee net forum? dd= 9Va VaggpnnPa.
12

---

Published as a conference paper at ICLR 2026
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, et al. Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference machine learning, pp. 1407-1416. PMLR, 2018.
Lasse Espeholt, Raphaël Marinier, Piotr Stanczyk, Ke Wang, and Marcin Michalski. Seed rl: Scalable and efficient deep-rl with accelerated central inference. In International Confer- ence on Learning Representations, 2020. URL https: pttt:::://peeeeeviee www.eewwweeeeooeee net/forumiid-
rkgvx1 rKwH.
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio. Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In Proceedings of the 37th International Conference on Machine Learning, ICML 20. JMLR.org. 2020.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2062, 2019.
Scott Fujimoto, Wei-Di Chang. Edward Smith, Shixiang Shane Gu, Doina Precup, and David Meger. For sale: State-action representation learning for deep reinforcement learning. Advances in neural information processing systems, (6:61573-61624, 2023.
Scott Fujimoto, Pierluca D' p'Oro. Amy Zhang. Yuandong Tian, and Michael Rabbat. Towards general-purpose model-free reinforcement learning. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. URL https: http///pppeeeeeeeeeeeeeeeeeeeeee wwt/ffoouu id=R1hIXdST22.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36:193-202, 1980.
Marco Fumero, Florian Wenzel, Luca Zancato, Alessandro Achille, Emanuele Rodolà, Stefano Soatto, Bernhard Schölkopf, and Francesco Locatello. Leveraging sparse and shared feature activations for disentangled representation learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: /ttp///weeeeeeeeee///e/ee net wor.mooommmaaa ? IHR83ufYPY.
Alexandre Galashov, Michalis Titsias, András György, Clare Lyle, Razvan Pascanu, Yee Whye Teh, and Maneesh Sahani. Non-stationary learning of neural networks with automatic soft parameter reset. Advances in Neural Information Processing Systems, 37:83197-83234 2024.
Matteo Gallici, Mattie Fellows, Benjamin Ellis, Bartomeu Pou, Ivan Masmitja, Jakob Nicolaus Foerster, and Mario Martin. Simplifying deep temporal difference learning. In The Thirteenth International Conference on Learning Representations, 2025. URL https:://openeeviet
Samuel Garein, Trevor Mclnroe, Pablo Samuel Castro, Christopher G. Lucas, David Abel, Prakash Panangaden, and Stefano V Albrecht. Studying the interplay between the actor and critic rep- resentations in reinforcement learning. In The Thirteenth International Conference on Learning Representattons, 2025. URL https: /ttpa////pppeeeeeeeeem net http:eetnnnooteeeee/eeeeeaee forummrmmmim Brr.irtttE.............
Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pascanu. Spectral normalisation for deep reinforcement learning: an optimisation perspective. In International Conference on Machine Learning, pp. 3734-3744, PMLR, 2021.
Laura Graesser, Utku Evci, Erich Elsen, and Pablo Samuel Castro. The state of sparse training in deep reinforcement learning. In International Conference on Machine Learning, pp. 7766-7792 PMLR, 2022.
Tuomas Haarooja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. Pmlr, 2018.
13

---

Published as a conference paper at ICLR 2026
Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pp. 1-7, 2025.
Nicklas Hansen, Hao Su, and Xiaolong Wang. TD-MPC2: Scalable, robust world models for contin- uous control. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttp:/:///peee/eeeeeeeee
forum? dot.mmmm...xx0xh0tt..... net wew//forum/
Charles R Harris, K Jarrod Millman, Stéfan J Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor. Sebastian Berg, Nathaniel Smith, et al. Array program- ming with numpy. Nature, 585(7825):3577-336 2020.
Junlin He, Jinxiao Du, and Wei Ma. Preventing dimensional collapse in self-supervised learning via orthogonality regularization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: : opw.reviee....eee.. http:/wwwwwweeeeeeeeeeeeeeeeeeee net. formm? 1d=Y3F JKSs Emy.
Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. To- wards the systematic reporting the energy and carbon footprints of machine learning. Journal of Machine Learning Research, 21(248): 12248))1-4333 1-0,, 2020.
D Hendrycks. Gaussian error linear units (gelus). arXiv preprint 160.,0006 2016.
J Fernando Hernandez-Garde and Richard S Sutton. Learning sparse representations incrementally in deep reinforcement learning. arXiv preprint arXiv:1912.04000 2019.
Geoffrey E Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan R Salakhutdi- nov. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint a270::/007.0580, 2012.
Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Ki- nal Mehta, and João G. M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1-18, 2022.
John D Hunter. Matplotlib: A 2d graphics environment. Computing in science & engineering, (03):90-95, 2007.
Niall Hurley and Scott Rickard. Comparing measures of sparsity. IEEE Transactions on Information Theory,
2009.
Maximilian Igl, Gregory Farquhar, Jelena Luketina, Wendelin Boehmer, and Shimon Whiteson. Transient -an-stationarity and generalisation in deep reinforcement learning. In International Conference on Learning Representations, 2021. URL https:///////. http://pppnneeeenneeeeee forumm? id= Ev4Sssy
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax, 2017. URL https: //t/r//axxa aaaaiiiiiiv.///...//aaaaaaaaaavv /rg/ abs/ /161.. 01144.
Vira Joshi, Zifan Xu, Bo Liu, Peter Stone, and Amy Zhang. Benchmarking massively parallelized
multi-task reinforcement learning for robotics tasks, arXiv preprint arXiv:2507.23172. 2025.
Arthur Juliani and Jordan T. Ash. A study of plasticity loss in on-policy deep reinforcement learning. In The eightt Annual Conference on Neural Information Processing Systems, 2024. URL https: cpperevieweee net feet/forume forum? df 8KkKTF.
Thomas Kluyver, Benjain Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Saña Abdalla, Carol Willing, and Jupyter Development Team. Jupyter Note- books-a publishing format for reproducible computational workflows. In IOS Press, pp. 87-90.
2016. doi: 10.3233/978-1
11113/977-1161199999 1-t---1--99999999 499-669-1187.
Jacob Eeuwe Kooi, Zhao Yang, and Vincent Francois-Laved Hadamax encoding: Elevating per- formance in model-free frre atari. The Thirty-ninth Anmual Conference on Neural Information Processing Systems, 2025. URL https: /ttes:///pppeeeeee www.eewwwwww.eeeeeeeeeeeeeeeeeee d=i RgM8 9.
14

---

Published as a conference paper at ICLR 2026
Alex Krizhevsky. Learning multiple layers of features from tiny images. https:///ww. toronto. Setukkri//eanninggeaattttv Reatures-2009. TR. pdf. 2009.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit ander-parameterizziin inhibits data-efficient deep reinforcement learning. In International Conference on Learning Rep-
resentations, 2021a. URL https: http:////ppeeeeeeeeeeeeee httet///oommmmiddonnniiii
Aviral Kumar, Rishabh Agarwal, Tengyu Ma, Aaron Courville, George Tucker, and Sergey Levine. DR3: Value-based deep reinforcement learning requires explicit regularization. In Deep RL Workshop NeurlPS 2021, 2021b. URL https:/ opwnrevieeeeveeeeeeeeeeeee net feet/forree forum?id-
LYWOCEpaO-A.
Aviral Kumar, Rishabh Agarwal, Xinyang Geng, George Tucker, and Sergey Levine. Offline q- learning on diverse multi-task data both scales and generalizes. In The Eleventh International Conference on Learning Representations, 2023, www.tttss:/openreeee
net/ forum?
Saurabh Kumar, Henrik Marklund, and Benjamin Van Roy. Maintaining plasticity in continual learning via regenerative regularization. In Vincenzo Lomonaco, Stefano Melacci, Tinne Tuyte- laars, Sarath Chandar, and Razvan Pascanu (eds.), Proceedings of The 3rd Conference long Learning Agents, volume 274 of Proceedings of Machine Learning Research, pp. 410- 430. PMLR, 29 Jul-01 Aug 2025. URL https:/ https:://ppocteeeeeeeeeo /proceededd ngs ml kumar25a.
Life-
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In International conference machine learning, pp. 5639- 5650. PMLR, 2020.
Samuel Lavoie, Christos Tsirigotis, Max Schwarzer, Ankit Vani, Michael Noukhovitch, Kenji Kawaguchi, and Aaron Courville. Simplicial embeddings in self- self-suuprvised learning and down- stream classification. The Eleventh International Conference on Learning Representations, 2023.
Samuel Lavoie. Michael Noukhovitch, and Aaron Courville, Compositional discrete latent code for high fidelity, productive diffusion models. In The Thirty-ninth Annual Conference on Neu- ral Information Processing Systems, 2025. URL https:/ pttt:p://opeereve/e wpwp//ppeeeeeeeeeeeeeee net forum?irmmmm ?id= lisnpztjja jbD.
Y. LeCun, B. Boser, S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L.. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541- 551, 1989. doi: 10.1 162/ncoo..... (tt.////............. 1989.1.4.541.
Hojoon Lee, Hanseul Cho, Hyunseung Kim, Daehoon Gwak, Joonkee Kim, Jaegul Choo, Se- Young Yun, and Chulhee Yun. Plastic: Improving input and label plasticity for sample effi- cient reinforcement learning. In Neural Information Processing Systems, 2023. URL https: //api. semant
aacholar.ore wwwaar.rrroooooeeeeeeeeeeee TD:
Hojoon Lee, Dongyoon Hwang. Donghu Kim, Hyunseung Kim, Jun Jet Tai, Kaushik Subrama- nian, Peter R. Wurman, Jaegul Choo, Peter Stone, and Takuma Seno. Simba: Simplicity bias for scaling up parameters deep reinforcement learning. The Thirteenth International Confer- on Learning Representations, 2025a. URL https:/ /wwpnrovweww..oc///// net wtw.::/wwwwe/eotttmeemmmmm forum?id= jXLiDXsuDo.
Hojoon Lee, Youngdo Lee, Takuma Seno, Donghu Kim, Peter Stone, and Jaegul Choo. Hyper- spherical normalization for scalable deep reinforcement learning. In Forty-second International Conference on Machine Learning, 2025b. URL https::///// https/////p/eeeee.eeoooom. net forum?ide kFYxyvCYQ4.
Timothée Lesort, Natalia Rodz-Rod,,,,ez, Jean-Franois Goudou, and David Filliat. State represen- tation learning for control: An overview. Neural Networks, 108:379-392, 2018.
Shikun Li, Xiaobo Xia, Shiming Ge, and Tongliang Liu. Selective-supervised contrastive learning with noisy labels. In Proceedings of the IÉEE/CVF conference on computer vision and pattern recognition, pp. 316-325, 2022.
15

---

Published as a conference paper at ICLR 2026
Zechu Li, Tao Chen, Zhang-Wa Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel g-learning: Scaling off-po reinforcement learning under massively parallel simulation. In International Conference on Machine Learning, pp. 19440-19459, PMLR, 2023.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv 2015.
preprint
Jiashun Liu, Johan Samir Obando Ceron, Aaron Courville, and Ling Pan. Neuroplastic expansion in deep reinforcement learning. In The Thirteenth International Conference on Learning Repre- sentations, 2025a. URL https:/ openrevieww net www.mooddommmmma.i.i..
Jiashun Liu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Ling Pan. The courage to stop: Overcoming sunk cost fallacy in deep reinforcement learning. In Forty-second International Conference Machine Learning, 2025b. URL ptaettss:///peeeevieee net/
Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Ling Pan. Measure gradients, not activations! enhancing neuronal activity in deep reinforcement learning. arXiv preprint arXiv:2505. 240,, 2025c.
Vincent Liu, Raksha Kumaraswamy, Lei Le, and Martha White. The utility of sparse represen- tations for control in reinforcement learning. In Proceedings of the Thirty-Third AAAI Con- ference on Artificial Intelligence and Thirty-Firys Innovative Applications of Artificial Intelli- gence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI' 19/IAA 19/EAAI' 19. AAAI Press, 2019. ISBN 978-1-57735-809 -.9-, doi: 10. 10.1609/aaai. 333101.33013844 URL https: //t::///oo/// wwggo//g//ooo http:/...111.//011 wtt.://///aaa..3311.0.... 3301438443..444
Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning. Science Robotics, 10(105))eads5033, 2025. doi: /..11///cooobootcs..aaee.
URL https:://// //www/wwww///////// http://wccceeeeee www.nce.ieec.c....c...o wttp:///abs///ooooe/////e///oaat /1266
scirobotics ads5033.
Clare Lyle, Mark Rowland, and Will Dabney. Understanding and preventing capacity loss in reinforcement learning. In Deep RL Workshop NeurIPS 2021, 2021. URL https://
openreview.neee http:/wwwweeeeooeeeeeeeeeeeeee feee:/forumiddd577555 tJTt.
Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynam- ics and generalization in deep reinforcement learning. In International conference on machine learning, pp. 14560-14581, PMLR, 2022.
Clare Lyle, Zeyu Zheng, Evgenii Nikishin, Bernardo Avila Pires, Razvan Pascanu, and Will Dabney. Understanding plasticity in neural networks. In International Conference on Machine Learning, pp. 13190-23211 PMLR, 2023.
Clare Lyle, Zeyu Zheng. Khimya Khetarpal, Hado van Hasselt, Razvan Pascanu, James Martens, and Will Dabney. Disentangling the causes of plasticity loss in neural networks. In Conference on Lifelong Learning Agents, pp. 750-783. PMLR, 2025.
Guozheng Ma, Lu Li, Zilin Wang, Li Shen, Pierre-Luc Bacon, and Dacheng Tao. Network spar- sity unlocks the scaling potential of deep reinforcement learning. In Forty-second International Conference on Machine Learning, 2025. URL https: pestps://openne http:///eeeeeeeeeeeeeeeee net forum?ide m. mimmqoskaa.
Chris J. Maddison, Andriy Mnih, and Yee Whye Tch. The concrete distribution: A continuous relax- ation of discrete random variables, 2017. URL https: /ttp/viiivvv..////a///.ea/eaaaaa org/ aww.abssll.... 1661.1116 00712.
Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu based physics simulation for robot learning. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Gabriel B Margolis, Ge Yang, Kartik Paigwar, Tao Chen, and Pulkit Agrawal. Rapid locomotion via reinforcement learning. The International Journal of Robotics Research, 43(4):572-587, 2024.
16

---

Published as a conference paper at ICLR 2026
Walter Mayor, Johan Obando-Cerome Aaron Courville, and Pablo Samuel Castro. The impact of on-policy parallelized data collection on deep reinforcement learning networks. In Forty-second International Conference on Machine Learning, 2025. URL forum? d=cnqy zu7hSo.
n
Wes McKinney. Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython. O'Reilly Media, 1 edition, February 2013. ISBN 9789351100065. URL http: /www://wwwwwwwww///../wt./...... amazon com/ exec/ owwoosiidoss// hos.oo///eeireetttttccceeeee.eee path=ASTM/144931199
Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, al. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 866):3740-3747, 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5660 2013.
Skander Moalla, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. No represen- tation, trust: Connecting representation, collapse, and trust issues in PPO. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: //oppnzeview wwtp:oruntttteeeeeo//eo/o fud S
Brian Murphy, Partha Talukdar, and Tom Mitchell. Learning effective and interpretable semantic models using non non-negative sparse embedding. In Martin Kay and Christian Boitet (eds.), Pro- ceedings of COLING 2012. pp. 1933-1950 Mumbai, India, December 2012. The COLING 2012 1111//1/1111//....
Organizing Committee. URL https: : / awwaanttnooooooooooooo
Michal Nauman, Michal Bortkiewicz, Piotr Milos, Tomasz Trzciński, Mateusz Ostaszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of rein- forcement learning. In Proceedings of the 41st International Conference on Machine Learning, pp. 37332-373664 2024a.
Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Mitoš, and Marek Cygan. Big- ger, regularized, optimistic: scaling for compute and sample efficient continuous control. Ad- vances in neural information processing systems, 101111338-1133 2024b.
Evgenii Nikishin, Max Schwarzer, Pierluca D'Oro, Pierre-Luc Bacon, and Aaron Courville. The primacy bias deep reinforcement learning. In International conference on machine learning, pp. 16828-16847. PMLR, 2022.
Johan Obando Ceron, Mare Bellemare, and Pablo Samuel Castro. Small batch deep reinforcement learning. Advances in Neural Information Processing Systems, 36.26003-26024, 2023.
Travis E. Oliphant. Python for scientific computing. Computing in Science & Engineering, 9(3): 10-20, 2007. doi: 20.1109/CSS.2207..55
Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine. OGBench: Benchmarking offline goal-conditioned RL. In The Thirteenth International Conference Learning Represen- tations, 2025a. URL https: http////peereeieeee//eee net wwt/ffouum forum? i 1d-M992mjgK2...
Seohong Park, Qiyang Li, and Sergey Levine. Flow q-learning. arXiv preprint arXiv:2502.02538 2025b.
Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in re- inforcement learning. Journal of Machine Learning Research, 25(318): 2024. URL
/ttt:://jmr... http://mmmm.rrr.pppeeoeee wwtg/ppers.pppaooooop /aarrrs/222233 83. html.
Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Ve- cerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep inforcement learning for dexterous manipulation. 2017. URL attps::/arxie //arxivv...////////////a/////atv orgg /bs/ 1 11704.00 03073.
17

---

Published as a conference paper at ICLR 2026
Yi Ren, Samuel Lavoie, Mikhail Galkin, Danica J. Sutherland, and Aaron Courville. Improving compositional generalization using iterated learning and simplicial embeddings, 2023. URL
https::/aaaa org/ htrg//bbe httpa/b.....1.11../..
Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In 5th Anmual Conference on Robot Learning, 2021. URL https::tp//.mmm....m owwnoeeeeeeeeee. net/ fow.mfiruuuu//a/
Aidan Scannell, Kalle Kujanpää, Yi Zhao, Mohammadreza Nakhaei, Arn Solin, and Joni Pajari- nen. iqrl-implicitly quantized representations for sample-efficient reinforcement learning. arXiv preprint ar 211v:2406.00666. 2024.
Aidan Scannell, Mohammadreza Nakhaeinezhadfard, Kalle Kujanpää, Yi Zhao, Kevin Sebastian Luck, Arno Solin, and Joni Pajarinen. Discrete codebook world models for continuous control. In The Thirteenth International Conference on Learning Representationss 2025.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Roy Schwartz, Jesse Dodge, Noah A Smith, and Oren Etzioni. Green ai. Communications of the ACM, 63(1)):54-63, 2020.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations. In The Nineth International Conference on Learning Representations (ICLR), 2021.
Max Schwarzer, Johan Samir Obando Ceron, Aaron Courville, Mare G Bellemare, Rishabh Agar- wal, and Pablo Samuel Castro. Bigger, better, faster: Human-level atari with human-level effi- ciency. In International Conference on Machine Learning, pp. 30365-30380. PMLR, 2023.
Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control. arXiv preprint wrXiiv2505.226-2 2025.
Carmelo Sferrazza, Dun-Ming Huang, Xingyu Lin, Youngwoon Lee, and Pieter Abbeel. Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint 2rXiv:2403. 2024.
Noam Shazeer, * Azalia Mirhoseini, *Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hin- ton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of- experts layer. In International Conference on Learning Representations, 2017. URL https: / //openeeview....
ckMDDqgg..
Jayesh Singla, Ananye Agarwal, and Deepak Pathak. Sapg: Split and aggregate policy gradients. In International Conference Machine Learning, pp. 45772. PMLR, 2024.
Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning, 2022. URL https:://exxi..ot abs/ 2208. 07860.
Ghada Sokar and Pablo Samuel Castro. Mind the gap! the challenges of scale pixel-based deep reinforcement learning. arXiv preprint 2rXhv:2505.17749 2025.
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon deep reinforcement learning. In International Conference on Machine Learning, pp- 92145-32166 PMLR, 2023.
Ghada Sokar, Johan Samir Obando Ceron, Aaron Courville, Hugo Larochelle, and Pablo Samuel Castro. Don't flatten, tokenize! unlocking the key to softmoe's efficacy in deep RL. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
//openreview. net/ffounm d=8ocr10aYess
18

---

Published as a conference paper at ICLR 2026
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremba, and Pieter Abbeel. Do- main randomization for transferring deep neural networks from simulation to the real world. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Aaron van den Oord, Oriol Vinyals, and koray kavukcuoglu. Neural discrete representation learn- ing. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.). Advances in Neural Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. URL httthttps///ett ptattps::/pprcceeiiggg
neurips. scc/paper_files wbc- www.rrbrppappp.........a......
paper/2017/ fi e/ 7 a f1
Guido Van Rossum and Fred L Drake Jr. Python reference manual. Centrum voor Wiskunde en Informatica Amsterdam, 1995.
Claas A Voelcker, Marcel Hussing, Eric Eaton, Amir massoud Farahmand, and Igor Gilitschenski. MAD-TD: Model-augmented data stabilizes high update ratio RL. In The Thirteenth International Conference on Learning Representations, 2025, URL https: oppprevveewweeeeeeeeeeeeeereee.e net / forum?
1d=6RtRsg52V1.
Maxime Wabartha and Joelle Pineau. Piecewise linear parametrization of policies:
Towards in- terpretable deep reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024. URL https:t//// http://///ewwee.eeeeeeeeeeeeeeee hot.eee//ooormmmohoooo5555
Timon Willi, Johan Samir Obando Ceron, Jakob Nicolaus Foerster, Gintare Karolina Dziugaite, and Pablo Samuel Castro. Mixture of experts mixture of RL settings, In Reinforcement Learning Conference, 2024. URL hwtpttttp//// http:///pppeeeeeeeeeeeee/e. net
Yuzhe Yang, Guo Zhang, Zhi Xu, and Dina Katabi. Harnessing structures for and reinforcement learning. arXiv preprint arXiv: 12255, 2019.
planning
Qingmao Yao, Zhichao Lei, Tianyuan Chen, Ziyue Yuan, Xuefan Chen, Jianxiang Liu, Faguo Wu, and Xiao Zhang. Offline RL with smooth OOD generalization in convex hull and its neighbor- hood. In The Thirteenth International Conference on Learning Representations, 2025. URL pettps:/cpperreeee
net/ forum?1d=aYYJNJE55
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learming Representa- tions, 2021. URL https:://// http:://ppeeevieeeeeee
net net/forum? forum? id= 6sTvGa.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con- trol: Improved data-augmented reinforcement learning, In International Conference Learning Representations, 2022. URL https:ttt.... http////pppeeeee/ee/e//eee net -iteenidd_______e__e_ee _yess...
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34:25476-25555 2021.
Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A Kahrs, et al. Mujoco playground. CoRR, 2025.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. study on overfitting in deep reinforcement learning. arXiv preprint arXiv: /84.v29804.06693, 2018.
Ziwen Zhuang, Zipeng Fu, Jianren Wang, Christopher G Atkeson, Sören Schwertfeger, Chelsea Finn, and Hang Zhao. Robot parkour learning. In 7th Anmual Conference on Robot Learning, 2023. URL httph:t////o / /ttp:///peeeeeeeeeeeeeee net
Dornoosh Zonoobi, Ashraf A Kassim, and Yedatore Venkatesh. Gini index sparsity measure for signal reconstruction from compressive samples. IEEE Journal of Selected Topics in Signal Processing, 5(5):927-99) 932, 2011.
19

---

Published as a conference paper at ICLR 2026
APPENDIX CONTENTS
A Related Work
22
Formal Analysis
23
Fast Actor-Critic
24
Benchmarks
24
E Deep RL Network Architectures
26
Ablation Setup
26
Metries
26
G.1 Feature Rank
27
G.2 Dormant Neurons
27
G.3 Weight Norm
27
G.4 Gini Sparsity
28
G.5 Cramer distance
28
G.6 Entropy
28
H Additional Baselines
28
H.1 Regen
29
H.2 Shrink and Perturb
29
H.3 Weight Clip
30
H.4 Reset Layer
30
H.5 Spectral Norm
31
H.6 Feature norm
32
H.7 Gelu
32
H.8 Dropout
33
H.9 MoEs
33
H.10 ReDO
33
H.11 Magnitude Pruning
34
20

---

Published as a conference paper at ICLR 2026
I Hyperparameters
35
J Learning Curves for Each Game
36
Additional Experiments on HumanoidBench
40
L Evaluating Layer-wise Interventions
43
M SEM on Value-Based Deep RL
44
21

---

Published as a conference paper at ICLR 2026
A RELATED WORK
Stability in Deep RL A longstanding challenge in reinforcement learning is the stability of value- based updates in actor-critic methods. One major source of instability is overestimation bias, which accumulates when bootstrapped critics reinforce overly optimistic targets. Twin Delayed DDPG (TD3) (Fujimoto et al., 2018) mitigates this issue with clipped double Q-laarning and delayed pol- icy updates, producing more reliable crities and improving control performance. Another direction seeks to stabilize targets by modeling full return distributions rather than point estimates. Distri- butional RL methods such as C51 (Bellemare al., 2017). QR-DQN (Dabney al., 2018b), and IQN (Dabney et al., 201 8a) show that capturing the shape of the return distribution reduces variance and provides richer learning signals.
More recently, architectural choices have been used to enhance critic stability. S2 (Lee et al., 2025b) biases networks toward simpler, well-conditioned representations via imput normalization, linear residual paths, and feature normalization, helping large models avoid divergence. Meanwhile, BRO shows that scaling critic capacity paired with strong regularization and optimistic updates yields dramatically better sample efficiency in continuous control tasks (Nauman et al., 2024b). In parallel, Ceron et al. (2024b;c); Ma et al. (2025) demonstrate that applying static network spar- sity unlocks scaling of deep RL models by mitigating gradient interference (Bengio et al., 2020; Obando Ceron et al., 2023) and plasticity loss (Lyle et al., 2023). Training regimens also play a role; SR-SPR OrrOro et al., 2022) demonstrates that periodic network resets counteract bootstrapping drift, allowing agents to sustain extremely high replay ratios without collapse. FasTTD3 (Seo et al., 2025) integrates several of these lessons, combining parallel simulation, large-batch updates, and distributional critics to achieve strong stability at high throughput. Our approach is complementary to these efforts. Rather than modifying update schedules or ensemble targets, we constrain the ge- ometry of latent representations, aiming to reduce critic variance and stabilize bootstrapped updates through structured embeddings.
Sample-Efficient RL Beyond stability, a parallel line of work targets sample efficiency, with progress spanning both representation-driien methods and algorithmic model-based improve- ments. Representation learning has emerged as powerful way to extract more information per interaction. CURL (Laskin al., 2020) applies contrastive learning to enforce invariances in en- coders trained jointly with the control objective, significantly narrowing the gap between pixel- and state-based agents. SPR (Schwarzer al., 2021) extends this idea with self- prediceede latent dynam- ics, ensuring temporal consistency and yielding thth--ff data efficiency on Atari. Building on SPR, SR-SPR (D'Oro et al., 2022) adds scheduled resets that prevent drift and enable aggressive replay-ratio scaling. Other works inject architectural or learning biases for example; SimBa (Lee et al., 2025a) introduces simplicity constraints that regularize feature representations, allowing larger networks to remain well-conditioned under nonstationary training; its successor, V2 (Lee et al., 2025b), further stabilizes scaling through hyperspherical normalization, constraining weights and activations to unit- manifolds and ensuring consistent gradient magnitudes across capac- ities. Neuroplastic Expansion (Liu et al., 2025a) complements these structural interventions by dynamically growing and pruning neurons preserve long-term adaptability, while The Courage to Stop (Liu et al., 2025b) improves behavioral efficiency by terminating unproductive trajectories early, reducing reflay-buffer contamination.
For SALE (Fujimoto et al., 2023) further enriches the representation space with state-action embed- dings, producing TD7, which substantially outperforms TD3 in continuous control. Outside of RL, simplicial embeddings (Lavoie et al., 2023) show that constraining features to products of probability simplices induces sparse, group-structured representations that generalize effectively in supervised and self-supervised settings and leads to a compositional representation (Ren et al., 2023; Lavoie et al., 2025). We draw inspiration from this idea and adapt it to reinforcement learning, inserting simplicial modules into fast actor-critic pipelines.
Algorithmic and model-based approaches provide another path to efficiency. Soft Actor-Critic (SAC) (Haarnoja et al., 2018) introduces maximum-entropy RL, balancing reward and exploration to achieve robust and data-efficient learning in continuous control. Model-based algorithms further im- prove efficiency by planning with learned dynamics. TD-MPC2 (Hansen et al., 2024) demonstrates that latent-space model predictive control scales effectively across diverse domains, achieving state- of-the-art performance with single set of hyperparameters. EfficientZero (Ye et al., 2021) 1) combines
22

---

Published as a conference paper at ICLR 2026
MuZero-style search with learned latent dynamics, reaching human-level Atari performance with ders of magnitude fewer environment steps. Our method differs from these approaches by focusing representation geometry: rather than auxiliary losses, ensembles, or world models, show that a single simplicial bottleneck can consistently improve the sample efficiency of fast actor-critic algorithms while preserving their hallmark wall-clock advantages.
Structured representation in RL Constraining the encoder's output is common in RL. C-ReLU has been shown to improve training and plasticity (Abbas et al., 2023). Feature normalization with L2 regularization of the features also improves training scalability and enables larger scale training of RL models. Closer to our work, DreamerV2 (Hafner et al., 2020) and DreamerV3 (Hafner al., 2025) encode the observation into one-hot discrete representation work. Scannell et al. (2025) also learn discrete latent space via a learned codebook and gumbel softmax with traaighttthouue esti- mator. Wabartha & Pineau (2024) also propose learn discrete encoding of the state for policy learning and show interpretable representations. However, methods with explicit discretization ne- cessitate the use of biased gradient estimator to propagate the learning signal inside the encoder. Similar to our work, Hansen et al. (2024) constrain the encoder's 's output into SEM. In this work, find that SEM crucial component for improving sample efficiency and performance in RL and study that component in details and connect the improved performance to the improved training stability coming from the sparse and structured representation endowed by SEM.
B FORMAL ANALYSIS
Theorem 1. Non-stationarity increases neuron dormancy.
Proof. Let Dr be the data distribution at iteration and consider a critic fo(x) W h5(x), trained by minimizing the (mean) squared error to targets (e(x))
C,( ELwD, (fo(z)) (x(()) {x))
(3)
Define the minimizer 07 arg ming (0) and tracking error e = 0: 07. A first-order expansion of SGD around 0 gives
C++1 = (-H C: - ab, H, - =Z by = (07).
(4)
where b = Of D441 = Dr, but b, + under drift. This shows that the optimizer must continually track moving minimizer, which destabilizes learned features. Let hz(#) Rd with covariance
Σ CoV (2) E[22 [[=
srank
(5)
Σ
In the stationary case, Σ, Σ with large stable rank, preserving feature diversity. Under stationarity, the drift term in equation 4 induces oscillations in E, and systematic covariance defla- tion (drop in srank), a hallmark of collapse. When representations collapse (covariance deflation; equation 5), feature energy shrinks. For a linear head,
V w 1
< 4E,7 tr(E.),
8, = fo(x) Me (x)),
(6)
so smaller tr(E,) directly yields smaller gradients and slower learning. With ReLU features 2 (a)) the backprop signal through unit is gated:
OL:
1lag (V((VLL,ee) da;
E
ac 80,
S Pj. EE C
where Py.ı Pr(a, 0) and ey is the j-th basis vector. Non-stationary drift (equation 4) reduces Pj. and Var(2,); together with lower tr(E.), this shrinks per-unit updates and increases neuron dormancy (Sokar al., 2023).
23

---

Published as a conference paper at ICLR 2026
C FAST ACTOR-CRITIC
FastTD3 (Seo et al., 2025) extends the standard TD3 framework by combining (i) parallel simulation many environment instances, (ii) large-batch critic updates, and (iit) algorithm design choices like distributional critics (C51) (Bellemare et al., 2017), noise scaling and clipped double Q-learning (CDQ) (Fujjmoto et al., 2018).
Instead of approximating only the expected return, FasrTD3 estimates the entire return distribution (ss, a) following C51 (Bellemare al., 2017). The action-value distribution is approximated by categorical distribution supported on N fixed atoms M.). uniformly spaced in Unin Umax]:
Umax
Umin
2: min +iAe,
A
0......,
N
The critic outputs logits {L,(s, Jio N-1 which define probabilities via (s, a) poftmax(é(s, a)), Given transition (s, a, s') and a next action a' Ttarg (s'), the cellman-updated support is
clip(r + 72j. Umin, Vmax).
1..... 1,
with target probabilities py(s', a') from the target critic at (s',a'). C51 projects this target distribu- tion back onto the fixed support via the projection operator
[
|z|-zu|
m, (DT2)(z)
a')
[x]+
max 0).
(8)
Training minimizes the cross-entrope between the projected target m and the predicted distribution:
C(s.u) = Σ mi logp;(s,
1=0
This distributional perspective reduces variance in target estimation and captures the multi modti-myd of returns in continuous control. Two other important stabilizers in actor-critic methods are noise scaling and Clipped Double Q-learning (CDQ). Noise scaling injects Gaussian perturbations into the action to balance exploration and stability:
a T (s) +G, (0,
(9)
where the scale must be tuned avoid either poor exploration (a too small) instability (a too large). On the other hand, CDQ mitigates overestimation bias by maintaining two critics Q Qo: and defining the bootstrapped target conservatively
y = r y min
Q,, no ( +
N(O, Purger I);
(10)
im 1.2
where o. delayed target networks and Oarget controls target-smoothing noise. Each critic then minimizes its own squared Bellman error:
(o( ((s.a.r,)) (Q,, v)²}. -1)². i= 1,2.
(11)
Together, these design choices underpin the high-throughput yet stable behavior of FastTD3 (Seo et al., 2025).
D BENCHMARKS
D.0.1 ISAAC GYM
For our experiments, we used the original Isaac Gym benchmark, which provides pre-built stand- alone environments and runs entirely on the GPU via PhysX backend. This setup enables both physics simulation and neural network policy training ation. Although Isaac Gym is deprecated, we used it to ensure reproducibility, specifically running the PPO algorithm from CleanRL on tasks spanning locomotion, robotic hands, and cube stacking (See Figure 12). To reproduce this task, we follow the PPO hyperparameters from CleanRL for Isaac, as presented in Table 3.
the GPU, offering high-throughput evalu-
24

---

Published as a conference paper at ICLR 2026
Fig. 12:
Environment Visualizations. We evaluate SEM across three benchmark suites such as Isaac Gym, HumanoidBench., and Atari. The first two cover state-based locomotion/manipulatio:: Atari introduces pixel-based games of varying complexity.
D.0.2 HUMANOIDBENCH
In experiments, we used the Humanoid Benchmark, suite of tasks for evaluating humanoid robot control across locomotion and manipulation, implemented on the MuloCo physics engine. We focused on three robot configurations: the Unitree HI without hands (26 DoF), the Unitree HI with hands (76 DoF), and the Unitree GI with three-finger hands (44 DoF). The benchmark defines 27 core tasks, and additionally, sit, balance, and bookshelf are each implemented in both simple and hard variants, while insert is implemented in small and normal configurations. This brings the total to 31 tasks. Our evaluations covered locomotion challenges, including walking, running, crawling. stair climbing, and balancing, and whole-body manipulation tasks such opening doors, lifting packages, operating kitchen objects, and performing insertions. Together, these tasks provided diverse and rigorous testing ground for our study of humanoid control (See Figure 12). To reproduce this task, we follow the fastTD3 hyperparameters (Sco al., 2025), as presented in Table 4.
D.0.3 ATARI
We conducted pixel-based reinforcement learning experiments using the Arcade Learning Environ- ment (ALE) (Bellemare al., 2013). We start from the 10 suite (Aitchison et al., 2023) and extend with additional environments from Atari-20 suite (Fedus et al., 2020), yielding broader subset of the ALE benchmark. In total, evaluate 28 games spanning varying difficulty levels, trained with PPO from CleanRL (Huang et al., 2022) as the baseline. Two games overlap across the subsets (Bowling and berert). We report IQM returns (see Figure 10, Left) following the evaluation protocol (Agarwal et al., 2021; ; Ceron et al., 2024c;b; Sokar & Castro, 2025).
D.0.4 МТBENCH
In our experiments, we used the Multi-Taan Benchmark for Robotics, an open-source suite built on the GPU-accelerated Isaac Gym simulator. Specifically, we worked with the 50 manipulation tasks adapted from Mela-World, where a single-armed robot interacts with one or two objects through actions such as pushing, picking, and placing. Each task provides parametric variations in object initialization and target positions, adding diversity and complexity. For evaluation, we adopted the MT50 setting, which encompasses the full set of 50 tasks.
D.0.5 BOOSTER GYM HUMANOID ROBOT
As reported in the FasITD3 paper (Seo et al., 2025), the authors configured the Booster Gym hu- manoid robot to operate within the MuJoCo Playground environment, which supports 12 degrees of freedom (DOF). They trained the policy entirely in simulation and successfully transferred it to real robot, demonstrating an effective sim-to-real deployment of an off-policy RL method on a full-sized humanoid. In experiments, within MuJoCo Playground. Our configuration led to faster convergence and improved performance in simulation compared to the reported baseline.
used the same robot model and hyperparameters FastTD3, also
25

---

Published as a conference paper at ICLR 2026
Table Default hyper-parameters setting for actor-critic MLP
Hyper-parameter
Value
Critic Hidden Dim
1024
Actor Hidden Dim
512
Critic Learning Rate
3e-4
Actor Learning Rate
3e-4
E DEEP RL NETWORK ARCHITECTURES
E.0.1
MLP
We modified the FastTD3 architecture, specifically in the actor-critte critic design, where both net- works are implemented as multilayer perceptrons (MLPs). The critic receives concatenated ob- servation-action inputs, while the actor processes only the observations. In both cases, the inputs first pass through two linear layers with ReLU activations. At this point, we intro- duced the SEM mechanism, which can be enabled disabled, and applied selectively to the actor, the critic, or both. For the critic, if SEM not used, the representation is processed by sequence of Linear-ReU--Linaa layers, with the final linear layer outputting di- mension num atoms. If SEM is enabled, the sequence becomes SEM+Linear, again pro- ducing output of size num atoms. For the actor, the representation without SEM follows a Linea r-RRLU-L wwetiieareee near-Tane 4Tah sequence, while with SEM it follows SEM+LI www.eiineaaaaaniii r+Tanh, where the final Tanh ensures bounded continuous actions. Table I, we present the fixed hyper- parameters used across all environments. Other hyperparameters, such as num atoms or num env, varied depending on the environment, in which case we adopted the values proposed by (Seo al., 2025).
E.0.2 CNN
For our pixel- experiments, we modified the PPO implementation from CleanRL, which fol- lows actor-critic design. The shared backbone consists of three convolutional layers, each followed by a ReLU activation, producing a flattened representation that then processed by two-layer MLP with ReLU activations. This representation is used by both the actor and the critic. In our intervention, we introduced the SEM block into the actor: when enabled, the rep- resentation passes through the SEM block before final linear layer; when disabled, it follows Linear-Re r- Li-LLLenninn sequence. The critic remains unchanged, while the actor architecture is varied depending the use of SEM. We adopted the PPO hyperparameters for Atari from CleanRL (Huang et al., 2022), as summarized in Table 5.
F ABLATION SETUP
Given our constrained computational budget, we performed experiments on a subset Hu- manoidBench, consisting of five robotics tasks. These tasks are part of the benchmark evaluated stand, run, stair,
with FastTDD3 (Seo et al., 2025) and correspond to h h1hand-(walk, slide) -vo. All ablation experiments conducted this subset using six random seeds (see Fig. 13 and Fig. 14).
G METRICS
To better understand the dynamics training and the quality of learned representations, we report diverse set of metrics beyond standard returns. These measures capture complementary aspects of learning, including representation diversity, network expressivity, parameter stability, gradient behavior and sampling efficiency. Results of these analyses provided in section 4.
26

---

Published as a conference paper at ICLR 2026
10-
64
Fig. 13: Effect of core hyperparameters.. SEM Actor compared to the baseline across (left) number of parallel environments, (middle) replay buffer size, and (right) batch size. SEM consistently scales better and achieves higher returns.
Oherr dal Sieardyy
2011
02
02
62
Drvirmmet Saga
Greesnmen Steps
Fig. 14: Robustness to design choices. SEM Actor vs. baseline across (left) clipped double Q- learning, (middle) distributional critic (C51), and (right) exploration noise scale. SEM remains robust, while the baseline is more sensitive.
G.1 FEATURE RANK
This metric assesses the quality of learned representations in deep RL by identifying the smallest subspace that retains 99% of the variance, thereby enhancing interpretability, efficiency, and sta- bility. A higher feature rank indicates more diverse representations. The computation follows the approximate rank from (Yang al., 2019; Moalla al., 2024):
2T.
where o; are the singular values of the feature matrix, n the total number of singular values, and 7 is the variance threshold (e.g., 99%). The feature rank k is the smallest number of principal components required preserve least 5 of the total variance.
G.2 DORMANT NEURONS
This metric quantifies the proportion of neurons with near-zero activations, which limits the net- work's expressivity. It serves to detect inefficiencies learning, as a high proportion of dormant neurons implies that many units are inactive rarely contribute to the output. The computation follows (Sokar et al., 2023):
EN 1(a. <)) x 100,
N
where N is the total number of neurons, is the activation of neuron i, e is small threshold (c.g., 10-9), and is the indicator function.
G.3 WEIGHT NORM
This metric measures the magnitude of neural network weights. providing insight into model com- plexity, stability, and overfitting risk. Large weight norms indicate parameters with high magnitudes, which may hinder generalization. The metric is computed as in (Moalla al., 2024; Lyle et al., 2021):
110||2
Σ 0z. V
where 0, are the weights of a given layer.
27

---

Published as a conference paper at ICLR 2026
G.4 GINI SPARSITY
The Gini metric is used to quantify the sparsity of neural representations. A high Gini value indicates sparse representation, where only few neurons are strongly active while most remain near zero; this often improves interpretability, makes more efficient use of network capacity, and can help reduce overfitting.
contrast, low Gini value corresponds to dense representations, where many neurons are active simultaneously, allowing the network to capture richer information but often the cost of reduced interpretability and potentially noisier features. In practice, we observed direct relationship between the Gini metric and the return when using SEM, with better performance associated with higher Gini values. The Gini value computed using the following equation.
G 1+
Σ 1=1
nn + 1 - (G)
-i=1
where where
(|11), [222....., 1
is the vector of all activations, taken in absolute value and stacked into one vector. The Gini metric has been explored in the papers (Hurley & Rickard, 2009; Zonoobi et al., 2011).
G.5 CRAMER DISTANCE
The Cramér distance defined as the squared L2 distance between the cumulative distribution functions (CDFs) of two probability distributions. When the distributions similar, their CDFs overlap closely and the Cramér distance approaches zero. Conversely, when the distributions differ, the CDFs diverge and the distance increases. In practice, lower Cramér distance indicates that the learned distribution is closer to the target distribution, which is desirable. Empirical results also suggest a correlation between lower Cramér distance and improved returns. This measure computed using the following equation:
Deeamer (P1.P2) = Σ (2))))) j=1
-((p
(z,)) - Fpz Foa(t))) A
where P1 and P2 probability distributions, and Fp, Fpz denote their corresponding cumulative distribution functions (CDFs).
G.6 ENTROPY
This metric measures the average entropy of the representations. High entropy indicates that the representation is more dispersed, less concentrated, and carries more uncertainty. Low entropy corresponds to a more concrete representation, with higher sparsity. In practice we observe a re- lationship where lower entropy is associated with better returns and higher Gini measure. This metric defined by the following equation;
Piá
Pij = Ex Pi.k +
1 B
Σ
Σ Pig log(ps. +e)
entropy
1=1
where B is the batch size, and is the non-negative representation normalized to form probability distribution.
H ADDITIONAL BASELINES
We expand our comparison to include a broader set of interventions commonly used to address plas- ticity loss, representation collapse, and optimization pathologies in deep RL.. Following the setup
28

---

Published as a conference paper at ICLR 2026
described in section 4 (Comparing SEM other Regularization Methods), evaluate eight addi- tional methods: Reset (Nikishin et al., 2022), L2 regularization (Kumar et al., 2023), ReGen (Ku- mar et al., 2025), Dropout (Hendrycks, 2016), Shrink-and-Perttee
(Ash & Adams, 2020), GELU activation (Hinton et al., 2012), Weight Clipping (Elsayed et al., 2024), and Spectral Normaliza- tion (Gogianu et al., 2021). Each method is evaluated across 6 random seeds and 5 HumanoidBench tasks.
These experiments in addition those reported Fig. 7, which already include other relevant baselines such as Gumbel with straight-through estimation (Jang et al., 2017; Maddison et al., 2017) 7), Vector Quantization (van den Oord al., 2017), and C-ReLU (Abbas al., 2023). We demonstrate that SEM offers a lightweight, stable, and effective mechanism for mitigating representation collapse deep RI. without requiring complex architectural modifications or extensive hyperparameter tun- ing.
H.1 REGEN
We applied ReGen by zeroing tiny weights using three thresholds, T = 10 10 10 and = 10 allowing compare how different magnitudes influence sparsity and model stability (Kumar et al., 2025).
Baseline
Regen (t 10-*)
SEM Actor
Regen (r= 10 9)
Regen (T= 10-²)
Average Normalized Return
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0
2
Environment Steps
x10*
Fig. 15: Average normalized return on 5 HumanoidBench tasks over 6 seeds. We compare the Baseline and ReGen variants with SEM Actor. SEM variation consistently outperforms all ReGen thresholds, achieving faster learning and higher normalized returns across training steps.
H.2 SHRINK AND PERTURB
We ran Shrink and Perturb with shrink factors 0.99, 0.9, 0.8, and 0.5 compare its performance and training behavior directly against our proposed method (Ash & Adams, 2020).
29

---

Published as a conference paper at ICLR 2026
Baseline
(0.05) S&P (0.2)
S&P (0.1)
SEM Actor
Average Normalized Return
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0
1
2
Environment Steps
x10*
Fig. 16: Average normalized return on 5 HumanoidBench tasks over 6 seeds. We compare the Baseline and arkk--nd-Peturbbb variants with SEM Actor. SEM variation maintains clear advantage across all S&P levels, achieving faster learning and higher normalized returns.
H.3 WEIGHT CLIP
We ran Weight Clipping with ranges (-0.01, 0.01), (-0.001, 0.001), and (-0.1, 0.1) to assess how different clipping strengths perform and to compare them directly against proposed method (Elsayed et al., 2024).
Baseline
Weight Clip (0.001,0.001) Weight Clip (-0.1,0.1)
SEM Actor
Weight Clip (-0.01,0.01)
Average Normalized Return
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0
1
2
3
4
Environment Steps
x10*
Fig. 17: Average normalized return 5 HumanoidBench tasks seeds. We compare the Baseline and Weight Clipping variants with SEM Actor. SEM variation achieves faster improvement and higher normalized returns across environment steps.
H.4 RESET LAYER
We applied the reset layer baseline by reinitializing weights with Xavier initialization, allowing us evaluate how fully resetting a layer compares to the performance and stability of proposed method (Nikishin et al., 2022).
30

---

Published as a conference paper at ICLR 2026
Baseline
SEM Actor
Reset
1.2
Average Normalized Return
1.0
0.8
0.6
0.4
0.2
0.0
1
4
6
Environment Steps
x x104
Fig. 18: Average normalized return on 5 HumanoidBench tasks over 6 seeds. We compare the Baseline and Reset Layer with SEM Actor. SEM variation accelerates learning and improves normalized return across environment steps, outperforming both the baseline and the reset-layer strategy.
H.5 SPECTRAL NORM
We applied Spectral Normalization to constrain layer norms and stabilize training, allowing us to directly compare its behavior and performance against the results achieved by our proposed method (Gogianu et al., 2021).
Baseline
SEM Actor
Spectral Norm
Average Normalized Return
1.2
1.0
0.8
0.6
0.4
0.2
0.0
1
4
6
8
Environment Steps
10
Fig. 19: Average normalized return on HumanoidBench tasks over 6 seeds. We compare the Baseline and Spectral Norm with SEM Actor. SEM variation outperforms both the baseline and Spectral Norm across training, achieving faster learning and higher normalized returns.
31

---

Published as a conference paper at ICLR 2026
H.6 FEATURE NORM
We applied Feature Norm by normalizing activations to unit L2 norm, allowing us to evaluate its effect representation stability and directly compare its behavior with the performance of proposed method (Kumar et al., 2023).
Baseline
SEM Actor
Feature Norm
Average Normalized Return
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0
1
2
3
Environment Steps
x10*
Fig. 20: Average normalized return on HumanoidBench tasks over seeds. We compare the Baseline and Feature Norm with SEM Actor SEM variation outperforms both methods, showing faster learning and higher normalized returns throughout training.
H.7 GELU
We evaluated the GELU activation to assess its impact learning dynamics and representation qual- ity, enabling a direct comparison between this widely used baseline and the performance achieved by proposed method (Hinton al., 2012).
Baseline
SEM Actor
GELU
1.2
Average Normalized Return
1.0
0.8
0.6
0.4
0.2
0.0
0
1
2
3
4
Environment Steps
x10
Fig. 21: Average normalized return on 5 HumanoidBench tasks over 6 seeds. We compare the Baseline and GELU activation with SEM Actor. SEM variation consistently surpasses both approaches, achieving faster learning and higher normalized returns across training.
32

---

Published as a conference paper at ICLR 2026
H.8 DROPOUT
We evaluated Dropout using probabilities 0.1, 0.05, and 0.2 to assess its regularization behavior and directly compare performance and stability against the results achieved by our proposed method (Hendrycks, 2016).
Baseline ne
Drepout (0.05)
SEM Actor
Dropout (0.1)
Dropout (0.2)
Average Normalized Return
1.0
0.8
0.4
0.0
Environment Steps
x10*
Fig. 22: Average normalized return on 5 HumanoidBench tasks over seeds. We compare the Baseline and Dropout variants with SEM Actor SEM variation consistently shows faster learning progress and achieves higher normalized returns across training.
H.9 MoEs
We evaluate cetture-of-Experri (MoE) architectures with and experts to study how expert multi- plicity influences model capacity, learning dynamics, and overall stability (Ceron et al., 2024c; Willi et al., 2024). These comparisons allow us to assess whether increasing routing flexibility or expert specialization match the gains provided by SEM. As shown in Fig. 23, both SoftMoF: and Top1- MoE track the baseline closely during training, with modest improvements 4-expert configuration. However, performance achieved by SEM Actor SEM consistently learns faster and attains higher normalized returns.
later stages for the of the MoE variants approach the sample efficiency final
Baseline
SoftMoE
Top1 MoE
SEM Actor
Bascline
SoftMoE
Topl MoE
SEM Actor
1.2
Nerrage Normaltzed Return
1.0
1.0
0.8
0.8
0.4
0.4
0.0
0.0
Stops
Stops
Fig. 23: Average normalized return on 5 HumanoidBench tasks over seeds. We compare the Baseline, SoftMoE, Top1-MoE, and SEM Actor. The left plot shows MoE models with I expert, and the right plot shows models with experts. MoE variants offer limited gains over the baseline and show slower learning dynamics, while SEM consistently achieves faster progress and higher final returns throughout training.
H.10 REDO
We evaluated ReDo (Sokar et al., 2023) by varying its dormancy thresholds (0.1, 0.01, 0.001) to study the sensitivity of neuron-reset frequency and to compare its stability and performance to SEM. These thresholds control how aggressively inactive neurons reset, allowing us examine whether ReDo's representational refreshing mechanism can match the benefits provided by SEM. As shown in Fig. 24, all ReDo variants track the baseline closely throughout training, with limited
33

---

Published as a conference paper at ICLR 2026
improvements in early or final performance. In contrast, SEM Actor consistently learns faster and achieves higher normalized returns.
Baseline
ReDo
SEM Actor
ReDo (0.1)
ReDo (0.001)
Average Normalizzed Return
1.0
0.6
0.4
0.2
0.0
Environment Steps
x10
Fig. 24: Average normalized return on 5 HumanoidBench tasks over seeds. We compare the Baseline, three ReDo variants (thresholds 0.1, 0.01, 0,001), and SEM Actor. All ReDo configu- rations remain close to the baseline, showing limited gains across training. SEM achieves faster learning and higher final returns, highlighting the effectiveness of structured embedding modifica- tions over neuron-reset-based approaches.
H.11 MAGNITUDE PRUNING
We evaluate Magnitude Pruning under two target sparsity levels (50% and 95%) analyze how pruning severity affects performance and to provide direct comparison with our proposed method (Graesser et al., 2022; Ceron et al., 2024b). Fig. reports results on HumanoidBench tasks. Overall, both pruning variants lag behind SEM Actor across the entire training horizon. Moderate pruning (50%) retains reasonable learning progress but consistently underperforms SEM in both sample efficiency and final returns. Severe pruning (95%) leads to early collapse in training, indi- cating that aggressive weight removal significantly harms representational capacity in this setting. In contrast, SEM maintains stable optimization and achieves substantially higher returns, highlighting the robustness of structured embedding modifications compared to unstructured pruning.
Baseline
SEM Actor
Pruning (50%)
Pruning (95%)
Average Normalized Return
1.2
1.0
0.8
0.6
0.4
0.2
0.0
Environment Steps
x10
Fig. 25: Average normalized return on 5 HumanoidBench tasks over 6 seeds. We compare the Baseline, SEM Actor, and Magnitude Pruning at 50% and 95% target sparsity. SEM consistently leams faster and reaches higher final returns than both pruning variants. Moderate pruning (50%) provides limited gains but remains inferior to SEM, while high-sparsity pruning (95%) collapses training early, demonstrating the instability of aggressive unstructured pruning.
34

---

Published as a conference paper at ICLR 2026
I HYPERPARAMETERS
In this section, we list the hyperparameters used across our experimental settings.
Table 2: Wall-clock training time (hh:mm) for the actor on the HI Illd humanoid benchmark under default settings. We compare Fas(TD3 and FasTTD3+SEM; lower better.
Actor
Game
FastID3
FastID3+SEM
h 1Thhnn-wand walk
2:31
2:42 h
h hhanddssand
2:29
2:20 h
h hannd-rrun
2:46
2:34
hand-sttair
4:09 h
4:13 h
h1 hand--slide
5:35
5:24
Table Default hyperparameter settings for the PPO agent on Isaac Gym.
Value Adam's learning rate
Hyper-parameter
Adam's (c)
1e-5
2.6e-3
Dense Activation Function
Tanh
Dense Width
256
Discount Factor
0.99
Number of Dense Layers
3
Number of environments
4096
Table 4: Default hyperparameter settings for the fastTD3 agent the humanoid bench.
Hapee-parr
Value
Critic Hidden Dim
1024
Actor Hidden Dim
512
Critic Learning Rate
3e-4
Actor Learning Rate
3e-4
Discount Factor
0.99
Dense Activation Function
ReLU
Number of Dense Layers
Number of environments
128
Number of atoms
101
35

---

Published as a conference paper at ICLR 2026
Table 5: Default hyperparameter settings for the PPO agent on Atari.
Hyper-parameter
Value
Adam's (c)
le-5
Adam's learning rate
2.5e-4
Conv. Activation Function
ReLU
Convolutional Width
32,64,64
Dense Activation Function
ReLU
Dense Width
512
Normalization
None
Discount Factor
0.99
Number of Convolutional Layers
3
Number of Dense Layers
2
Reward Clipping
True
Weight Decay
0
J LEARNING CURVES FOR EACH GAME
To complement the aggregate results reported in the main text (see section 4 and section 5), we provide full learning curves for each environment in the benchmark. These plots illustrate training dynamics across seeds and highlight differences in sample efficiency and stability between +SEM and its corresponding baseline BBatTTDD/FatTTD--SSmbbb Si--Simmaa///FsAAAC The of robotics tasks
follows those used in the FastTD3 benchmark (Seo et al., 2025).
hilhandwwalee
h1haandrrnn-00
x12'
h
Epiedde Natten batam
0
x16
simple-v0
x16
X10
Ephonde fetur
FastTD3 (SimbaV2)
(Actor)
0.2
0.4
0.6
1.0
12
1.4
x100
xLP
K300
<307
Rahand-pole-vo vo
x207
h1 hand--oom--v
atanddd
125
fplodd batian
6-
0.50
0.25
0.8 1.0
Environment
x12
Broisrmant
X12*
Endrromment Stops
K20*
Fig. 26: Learning curves on h1hand tasks. FastTD3+Simba (blue, vs. + SEM (Actor) (green, -). Curves show the mean episode return across seeds. Axes are independently scaled per subplot for readability. SEM (Actor) consistently accelerates learning and achieves higher or comparable final returns on most tasks.
36

---

Published as a conference paper at ICLR 2026
vo
huhand-run-v0
h1hand hurdle
Edhende Danum
1.5
1.0
0.5
<107
hahhandbblaanee simple.vo
x107
h1hand-stalrrv0
x10
h1hand-sliden
[pledd Baturn
Fast
(Actor)
-20
h1haand-ole--00
x10
h1hand-room-vo
x10
hllhend-stand-v0
Epivdd Setumn
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Envisomment Stape
x107
Enwironmentt Steps
x10F
Environment Staps
X30
Fig. 27: Learning curves 9 h1hand tasks. FastSAC (blue, SEM (Actor) (green, -) Curves show the mean episode return across 6 seeds. Axes are independently scaled per subplot for readability. SEM (Actor) generally accelerates learning and achieves higher final returns most
tasks.
I5aac- Lift-Cubeteeetttee Franka-v0
Cube
1saac-
x10"
Velociyy Flat-G1-v0
122
3.75
3.5
3.50
Ephodd teeum
0.75
0.50
2.75
0.25
2.50
0.00
Repose Aule-A
Isaac SaaccVelbcity-Rouugt--
Iseac BalacVVeoclly--oog-GG...o
x20
Eptoode Retum
3.5
FastTD3 (Simbav2)
2.0
SEM (Actor)
Cube Shadoww D0
Humanol AMP-ad------------- Direct
Isacc-tuuanoiid-o0
3.0
Eptoade Retumm
0.6
1.0
2.5
2.0
1.5
0.4
02
0.5
0.5
0.5
1.0
1.5
1.2
1.5
Enviroement
X12'
X30*
X20*
Fig. 28: Learning curves on 9 IsaacGym tasks. FastSAC (blue, -) vs.
SEM (Actor) (green,
Curves show the mean episode return across seeds. Axes are independently scaled per subplot for readability. SEM (Actor) generally accelerates learning.
37

---

Published as a conference paper at ICLR 2026
2007
hzhand wa
hahandstarrvv
hchardspoor-00
hlaand-turdt--00
kirandeur-2
-1rand-dide-w
+1rand puchoc
F25 Ene 015 3 125
175
Nitandmave v0
1.5
1.2
033
24
0.6
14
1lhand-oubeed
Nitandel emplesd
-55
1lbordisabbrce Fardivd
X30
109
hahandst_hardvo
bahardbackchee
2.
bitessbbrrrtt hardso
Alhord-met momat
hihandiiseer lrv)
+lbond-droest
15
1.0
-12
Fig. 29: Learning curves on 28 hlhand tasks (Sferrazza et al., 2024). FasITD3 (blue,
vs. + SEM (Actor) (green, -). Curves show the mean episode return across 6 seeds. Axes are independently scaled per subplot for readability. SEM (Actor) typically achieves faster learning and equal higher final return on most tasks.
38

---

Published as a conference paper at ICLR 2026
2014
RutieZenevé
227
Deuthc2urk-6
2014
Phoonics5
3.-
0.50
110
SDM (Actor)
32-
3.4
-20
Qberti5
Riverad-t
Amiidaus
Sowilogs5
325
15
1.30
3%
2.0
-104
Prodtalle/S5
-100
Murgfiutaddeww
-10
Acterteu
Demoskmaccku
<12"
Scamfidere5
407
221
ChosserSnmand-r
12
10-
05
6.0
26
Spacelmadee---5
30
38
0.0
0.5
16-
Xa2
412
Freewayus
Seaquestiv5
-10
Panguv
Aithaded
Jamistondad
WizandoWWerrd
Astords
Fig. 30: Learning curves on Atari game (Aitchison et al., 2023). PPO (blue, -) vs. SEM (Actor) (green, -). Curves show the mean episode return across 3 seeds. SEM (Actor) typically achieves faster learning and equal or higher final return on most tasks.
39

---

Published as a conference paper at ICLR 2026
K ADDITIONAL EXPERIMENTS ON HUMANOIDBENCH
We evaluate +SEM beyond the tasks proposed in FastTD3 by considering additional environments from the Humanoid benchmark (Sferrazza et al., 2024). These experiments assess the scalability of SEM across different robot morphologies and task sets. We include environments featuring the HI robot without hands and the Unitree GI with three-finger hands.
p1-aur-0
g3-mtaddol
2-
-100
0lccmeive
01 01-pporaat
Othurdiest
-10
31 31-sta/re0
FactTD3
SEM (Axtor)
439
c1-colerce smpp-vd
g1-3elance Nertre
c1-p
5l-dzoeuD
04
X29"
-10
gl-oubexd
93 Lona
25
1.0
Cnviorman:Sap
Dnvercment Stape
Fig. 31: Learning curves 16 h1 tasks (Sferrazza et al., 2024). FastTD3 (blue, vs. SEM (Actor) (green, -), Curves show the mean episode return across 6 seeds. Axes independently scaled per subplot for readability. SEM (Actor) typically achieves faster learning and equal or higher final return on most tasks.
40

---

Published as a conference paper at ICLR 2026
2012
11
233
21
2.-
92:cpeive
S1-pporein
Gthurdeeo
$1-sorive
2017
02:wirdenww00
1387
2037
gl mazsuv
2257
91-4it ampreev
c1-talence sirpled
gl-catance hartvn
51-po ex0
51-dcoeu6
1.0
11 loibevi
[1 g1-ocmao
-19"
-15"
31-4ssert sormallv
16
0.5
Eniismart Sacs
Dra/orman:2es:
Envinomentt Stps
Fig. 32: Learning curves on gl tasks (Sferrazza al., 2024). FastTD3 (blue, - vs. SEM (Actor) (green,- -), Curves show the mean episode return across 6 seeds. Axes are independently scaled per subplot for readability. SEM (Actor) typically achieves faster learning and equal or higher final return on most tasks.
x102
Episode Return
x101
Critic
Net e-rank
h1hand- walk-v0
3
2
5
FastTD3 + +SM (Actor)
1
0
0.5
1.0
0.5
1.0
Environment Steps
x10
Environment Steps
x10
Fig. 33: Learning and representation diagnostic h1hand-walk-vo task. SEM reaches high return earlier and critic effective rank.
41

---

Published as a conference paper at ICLR 2026
Baseline
SEM Actor
Single Simplex
Average Normalized Return
1.2
1.0
0.8
0.6
0.4
0.2
0.0
0
2
4
5
7
Environment Steps
x10
Fig. 34: Average normalized return 5 HumanoidBench tasks over seeds. We compare the baseline agent (blue, --) with SEM variants applied to the actor. SEM variants accelerate early learning, though a single-simplex configuration tends to plateau.
42

---

Published as a conference paper at ICLR 2026
L EVALUATING LAYER-WISH WISE INTERVENTIONS
Selecting which layer to modify is non-trivial, as the search space grows with model depth and archi- tectural complexity. Throughout the paper, apply SEM to the penultimate layer, following prior work (Gogianu et al., 2021; Kumar et al., 2023; Ceron et al., 2024c; Sokar & Castro, 2025). These studies highlight that the penultimate layer plays a key role in shaping and constraining learned representations in deep RL.
Here, we extend this analysis by evaluating the effect of applying SEM to individual actor layers as well as to all layers simultaneously. Following the experimental setup in section 4, we report results in Fig. and Fig. 36. Across both settings, we observe that applying SEM to deeper layers to early
the entire network leads to faster learning and higher final returns compared to intervening layers. Notably, applying SEM solely to Layer-3 achieves performance comparable to the full-layer intervention, indicating that modifying single well-chosen layer is sufficient to capture most of SEM's benefits.
Baseline
SEM Actor (Layer-2)
SEM Actor (All Layers)
SEM Actor (Layer-1)
SEM Actor (Layer-3)
1.2
Average Normalized Return
1.0
0.8
0.6
0.4
0.0
Environment Steps
x10*
Fig. 35: Average normalized return on 5 HumanoidBench tasks over 6 seeds. We evaluate the effect of applying SEM to individual actor layers (Layer-1, Layer- Layer-3) and to all layers with dim 64.
Baseline
SEM Actor (Layer-2)
SEM Actor (All Layers)
SEM Actor (Layer-1)
SEM Actor (Layer-3)
1.2
Average Normalized Retum
0.8
0.4
0.0
Environment Steps
x10*
Fig. 36: Average normalized return on 5 HumanoidBench tasks over 6 seeds. We evaluate the effect of applying SEM to individual actor layers (Layer-1, Layer-2, Layer-3) and to all layers with dim = 128.
43

---

Published as a conference paper at ICLR 2026
M SEM ON VALUE-BASED DEEP RL
We additionally evaluate SEM in value-based deep RL settings. Specifically, we run PQN (Gallici al., 2025) on 28 Atari games following the experimental setup from section 5 using seeds and training for 40M environment steps. As shown in Fig. 38, SEM does not yield consistent improvements over the baseline when averaging performance across all games. However, examining per gere learning curves (see Fig. 37) reveals that SEM provides meaningful gains in both sample efficiency and final performance on several titles (e.g., Asterix, BeamRider). These results suggest that the effectiveness of SEM in value-based methods may depend on game-specific dynamics and representation structure, raising interesting research questions that we leave for future investigation.
222
Phoonavdd
M
PCN SEM (Actor. V-4)
03
Qbanus
Riversid-uS
Bowlingu
1.9
X12"
k32"
2007
Frastaiie-s
+000
KungtoMaster-i5
-237
Demoskaack
18.
52-
0.0
31-
6
Bearrfiddrrd
Endurceb
Cravtiarid
ChoparCCommnde55
62-
XIx"
2007
7hoost-w5
Specetrateeees
'235
WerKeuergewf
-22"
MoParmanoti
0.0
Treeagor
SeaquentivS
Poreus
Desalzubu5
3.0 4104
AvRaidiv
23 430
pareebooiis
4394
25
Astenktous
2.0
-12"
430
Fig. 37: Learning curves on 28 Atari games (Aitchison et al., 2023). PQN (blue, -)) (Lavoie et al., 2023) vs. + SEM (Actor) (green,- -). Curves show the mean episode return across 3 seeds.
44

---

Published as a conference paper at ICLR 2026
IQM Human Normalized Score
x10
4
PQN
+ SEM (Actor)
1
2
3
4
x107
Environment Steps
Fig. 38: IQM scores computed over 40M environment steps 18 games, with 3 independent runs each, and error bars showing 95% stratified bootstrap confidence intervals. PQN (blue, -) (Lavoie et al., 2023) vs. + SEM (Actor) (green,- -).
45


Reviews:
[{"review_id": "wfFvLTmQjW", "reviewer": "Reviewer_WyjG", "summary": "The paper investigates the use of a particular rank encouraging representation for state embedding in actor-critic reinforcement learning methods. The particular representation used is the simplicial embedding, which has been introduced in prior work which constrains the embedding space to be the concatenation of equal dimension simplexes. This representation can be used as an embedding in the actor, critic, or both. The novel contribution of this work is to show that this previously proposed embedding can improve the performance of actor-critic RL algorithms in terms of a combination of sample efficiency, returns and variance reduction in the learning curves. The paper mostly validates its claims empirically, while providing some possible intuition behind the performance differences observed with the use of simplicity embeddings. The paper does conduct meaningful empirical analysis to do the same, and I feel most of my questions regarding the claims were sufficiently answered by the experiments. I list some remaining concerns in the weaknesses section below.", "questions": "Besides the 3 points raised in weaknesses section, please address the following:\n1. In the experiments where you are averaging over different tasks (like Fig. 3), were the state dimensions the same?\n2. What is the value of $L$ in Fig. 3?\n\nRequested Changes:\n1. Given that the dependence of perfomance on representation capacity, please explicitly state all $L$, $V$ values in the figures/captions where not already done. If the state dimensions vary, please mention those as well.\n2. Minor typo: line 204: contex -> context.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Most importantly, the proposed use of SEMs does seem to show promising performance improvements consistently across a variety of benchmarks and tasks at least for continuous control problems.\n2. The SEM formulation is quite simple and should not add significant extra computational cost in actor-critic training, which I appreciate.", "weaknesses": "1. I am unsure about the baseline choices of other representation methods used in Fig. 7 (left). Given that the authors' hypothesis about SEM working well is that it encourages the encoding to have high effective rank, are the methods compared against in Fig. 7 (left) also methods that specifically focus on creating high effective rank? If so, please mention this, if not, then I think other baselines should be used.\n2. One baseline I think should have been used for all experiments throughout the paper but has not been used is the case when $V=1$, i.e., instead of a group of $L$ $\\Delta^{V-1}$ simplexes, use a single $\\Delta^{LV-1}$. While technically this single simplex case is also a type of SEM, I this comparison is needed for the claim of SEMs promoting diversity (lines 182-184) through the group structure.\n1. Lines 365-366 (referring to Fig. 7 right)) : `We find that increasing V generally improves performance, but only up to a certain point.` Respectfully, I disagree with this claim. In the two right most plots of Fig 7., we see $V=4$ having significantly worse performance than others *only* when $L=1$. I think this trend is more aptly explained by the low overall capacity here (as $LV=4$). When $L=64$, we see all choices of $V$ performing comparably, with $V=4$ in fact slightly better.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HkhjsIcGCs", "reviewer": "Reviewer_PVgk", "summary": "The paper proposes the use of simplicial embeddings (SEM) to improve representation stability in reinforcement learning (RL), particularly under distributional shifts. By incorporating SEM into the penultimate layers of actor and critic networks, the authors report faster learning across several RL environments.", "questions": "- How are the groups chosen for normalization within the softmax operation? Is it just connected parts of the latent vectors?\n- In Figure 4, the \"Critic - Net e-rank\" curve continues to rise at 100k steps. What happens beyond this range?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "- The authors demonstrate that using simplicial embeddings generally leads to faster or comparable learning speed across multiple environments.\n- The paper investigates which components of actor-critic architectures benefit most from SEM, analyzes the impact of hyperparameters, and compares SEM with other representation-learning baselines.", "weaknesses": "- The use of shuffled CIFAR-10 labels as a proxy for distributional shift is questionable. The claim that this setup \"mimics the bootstrap dynamics of RL\" is conceptually weak: label shuffling introduces discrete, externally imposed changes rather than the gradual shifts seen in RL. It is also unclear what bootstrapping dynamics are in this context.\n- In the accompanying Figure, the authors show the loss of classifiers for stationary targets and for label shuffling with and without SEM. Before the first shuffling occurs, the distribution shift without SEM and the stationary variants should behave similarly, but they do not. This discrepancy suggests confounding factors may explain the poorer performance of the shifting variant without SEM.\n- The claim that SEM improves critic representations is made before any RL experiments are introduced, which makes this conclusion premature and poorly supported by the evidence presented.\n- The Appendix incorrectly describes the \"Atari-10\" benchmark as including 26 games. In reality, Atari-10 consists of 10 games, while 26-game subsets are part of benchmarks such as Atari-100k.\n- Overall, the paper's presentation is the weakest aspect. Clarity and experimental justification need substantial improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3IaJK1mm4", "reviewer": "Reviewer_A2Cz", "summary": "The paper studies representation collapse as a root cause of poor sample efficiency and training instability in deep RL, arguing that non-stationary, bootstrapped targets make critics ill-conditioned and prone to drifting value estimates. It proposes Simplicial Embeddings (SEM)—a lightweight, group-wise softmax projection that constrains latent features to a product of simplices—to bound features, induce sparsity, and maintain diversity, thereby stabilizing learning. The proposed method is proven to be effective on different base RL algorithms across mainstream benchmarks.", "questions": "1. What's the best setting of L and V? Did you change the values of L and V depending on the benchmark and task?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Clear problem framing.** Ties critic instability to non-stationarity and representation collapse with an accessible toy study.\n\n**Simple yet effective, plug-and-play mechanism:** SEM is like an activation with better gradient properties; no extra losses or estimators. \n\n**Broad robustness:** Benefits persist across data-limited settings and simplified agent variants; compares against reasonable discrete/structured alternatives.", "weaknesses": "**Inappropriate toy examples.** The CIFAR-10 supervised learning experiment with shuffled labels is not a straightforward toy example for demonstrating the non-stationarities in value learning.\n\n**L, V selection.** It is weird that the limited SEM capacity leads to better performance. Especially the setting that $L=1, V=64$ almost achieves the best performance in Fig. 7 if the y-axes are shared. Doesn't that mean the softmax operator is the best regularizer?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sjaeJayCe6", "reviewer": "Reviewer_d8VX", "summary": "This paper proposes the use of simplicial embeddings (which lead to sparse and discrete features) as a way to improve the sample efficiency, stability and performance of Reinforcement Learning agents. Furthermore, the paper adds to the body of work showing that non-stationary targets can cause a loss of plasticity, and demonstrates that SEM can mitigate this effect.", "questions": "How does this method compare against prior plasticity loss mitigation strategies? Does this method provide better performance? Is it better at preventing representation collapse?\n\nIt is unclear to me why this paper focuses on actor-critic algorithms specifically. Line 323 provides literature suggesting that non-actor-critic approaches can also benefit from sparsity in representations. Given this, it seems odd to me that the paper is so heavily tied to actor-critic methods (including the title). Why is this the case?\n\nWhat do the shaded areas on the different figures represent (Standard error, 95% Confidence intervals, etc)? Additionally, how many seeds does Figure 1 use?\n\nI think this paper is valuable to the RL community; however, given the lack of comparison against prior methods, I cannot yet advocate for acceptance. If this issue is remedied, and the method demonstrates superior properties to prior methods, I am willing to raise my score.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper presents a simple, practical idea as a quick and effective way to improve sample efficiency in Reinforcement Learning, which can be easily used in almost any algorithm. The analysis on why SEM improves performance is deep and uses a variety of metrics previously proposed in the literature. Furthermore, the evaluation is quite thorough, including different algorithms and environments.", "weaknesses": "My main problem with this paper is the lack of comparison to prior methods that solve very similar problems. This paper correctly cites a wide range of prior work that propose solutions to the problem of degradation of learned representations under non-stationarity ([1, 2, 3] and many more), causing plasticity loss and reduced performance. Despite this large variety of previous similar work, the paper also exclusively compares against baselines with no previous similar solutions. While many figures show improvement, none show whether this is actually an improvement over prior existing work. For this work to be accepted at a venue of this standard, I would expect that Figures 1, 9 and 10 would include a comparison against prior strategies.\n\nFigures 9 and 10 have inconsistent axes. The left and middle plots present very similar data (million of frames on the x axis, and performance on the y axis), yet are presented under different scales. This inconsistency makes the results a little harder to read.\n\nNo source code was included in the supplementary material, and there is no mention of source code being released after review period. This hurts the reproducibility of this work.\n\nFigure 10 uses the Atari-10 benchmark [4], but presents results as IQM performance. The purpose of the Atari-10 paper was to predict the **median** performance, using a specific regression procedure provided by the authors (which weights the importance of each game). Given that the paper reports IQM performance, it appears the procedure was not followed.\n\nThere is a typo on line 215.\n\n[1] Lyle, Clare, et al. \"Disentangling the Causes of Plasticity Loss in Neural Networks.\" Conference on Lifelong Learning Agents. PMLR, 2025.\n\n[2] Ceron, Johan Samir Obando, Aaron Courville, and Pablo Samuel Castro. \"In value-based deep reinforcement learning, a pruned network is a good network.\" International Conference on Machine Learning. PMLR, 2024.\n\n[3] Willi, Timon, et al. \"Mixture of Experts in a Mixture of RL settings.\" Reinforcement Learning Conference.\n\n[4] Aitchison, Matthew, Penny Sweetser, and Marcus Hutter. \"Atari-5: Distilling the arcade learning environment down to five games.\" International Conference on Machine Learning. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)