Partially Equivariant Reinforcement Learning in Symmetry-Breaking Environments
Authors: Junwoo Chang
Minwoo Park
Joohwan Seo
Roberto Horowitz
Jongmin Lee
Jongeun Choi
Venue: ICLR 2026 Poster (2026)
Keywords: Group Equivariance
Reinforcement Learning
Robotics
Primary area: reinforcement learning
TL;DR: Partial equivariance in reinforcement learning to deal with symmetry-breaking

Abstract: Group symmetries provide a powerful inductive bias for reinforcement learning (RL), enabling efficient generalization across symmetric states and actions via group-invariant Markov Decision Processes (MDPs). However, real-world environments almost never realize fully group-invariant MDPs; dynamics, actuation limits, and reward design usually break symmetries, often only locally. Under group-invariant Bellman backups for such cases, local symmetry-breaking introduces errors that propagate across the entire state--action space, resulting in global value estimation errors.
To address this, we introduce Partially Group-Invariant MDP (PI-MDP), which selectively applies group-invariant or standard Bellman backups depending on where symmetry holds. This framework mitigates error propagation from locally broken symmetries while maintaining the benefits of equivariance, thereby enhancing sample efficiency and generalizability.
Building on this framework, we present practical RL algorithms -- Partially Equivariant (PE)-DQN for discrete control and PE-SAC for continuous control -- that combine the benefits of equivariance with robustness to symmetry-breaking.
Experiments across Grid-World, locomotion, and manipulation benchmarks demonstrate that PE-DQN and PE-SAC significantly outperform baselines, highlighting the importance of selective symmetry exploitation for robust and sample-efficient RL. Project page: https://pranaboy72.github.io/perl_page/

Published as a conference paper at ICLR 2026
PARTIALLY EQUIVARIANT REINFORCEMENT LEARN- ING IN SYMMETRY-BREANKE ENVIRONMENTS
Junwoo Chang', Minwoo Minwoo Jongeun Choi
Joohwan Sco2, Roberto Horowitz', Jongminn Lee";
School of Mechanical Engineering, Yonsei University, Seoul, South Korea
2 *Department of Artificial Intelligence, Yonsei University, Seoul, South Korea
3 DDepartment of Mechanical Engineering, University of California, Berkeley, United States
i junwoochan,, minwoopark, jongminlee, jongeunchoi]eyonsee jochwan.seo, [orowitz]@berkelee edd
ABSTRACT
Group symmetries provide a powerful inductive bias for reinforcement learning (RL), enabling efficient generalization across symmetric states and actions via group-invariant Markov Decision Processes (MDPs). However, real-world envi- ronments almost never realize fully group-invariant MDPs; dynamics, actuation limits, and reward design usually break symmetries, often only locally. Under group-invariant Bellman backups for such cases, local cymmetry-breakeee duces errors that propagate across the entire state- action space, resulting in global value estimation errors. To address this, we introduce Partially Group-Invariant MDP (PI-MDP), which selectively applies group-invariant or standard Bellman backups depending on where symmetry holds. This framework mitigates error propagation from locally broken symmetries while maintaining the benefits of equivariance, thereby enhancing sample efficiency and generalizability. Build- ing this framework, we present practical RL algorithms Partially Equivariant (PE)-DQN for discrete control and PE-SAC for continuous control that combine the benefits of equivariance with robustness to symmetry-breaking Experiments across Grid-World, locomotion, and manipulation benchmarks demonstrate that PE-DQN and PE-SAC significantly outperform baselines, highlighting the im- portance selective symmetry exploitation for robust and sample-efficte RL. Project page:
intro-
girhub io/ ptrp_paarrrreaappe
INTRODUCTION
Group symmetries provide powerful inductive bias in machine learning, enabling models gener- alize efficiently. In robotics and continuous control, leveraging equivariance has been shown to im- prove data efficiency in both behavior cloning (Zeng et al.) [2021] Ryu et al. | [2233] [0244] Waang al.,
2024 Tie et al. 2025 Huang et al. 2024 Zhao et al. 2025 Seo et al. [2024] 2023 2025a a b). where
the data collection costly, and reinforcement learning (RL) (an der Pol et al. 22200] Kohler al,
2024 Wang et al.) 2022a b. Tangri et al.) 2024 Nguyen et al. 2023 Finzi et al. 2022 12] Park et al.,
2025 where exploration can inefficient. Most existing equivariant RL methods are grounded in
the notion of group-invariant Markov Decision Process (MDP) (Wang et al. (2022h 2021), where invariance of the reward and transition functions implies symmetry in the optimal policy.
In practice, however, these symmetry assumptions rarely hold exactly. Real-world environments introduce byramery- factors such as dynamics, actuation limits, or reward shaping. Under the Bellman backups based the group-invariant MDP, even local violations of symmetry can introduce errors that propagate across the state-action space, leading to degraded value estimates, suboptimal policies, even training failure. Prior works on approximate equivariance (Finzi al.) 2021a Park et al. 2025) attempt to mitigate this challenge by relaxing equivariance globally, e.g., by modifying architectures tolerate violations. While effective to extent, these methods often
"Equal advising

---

Published as a conference paper at ICLR 2026
41 (8,)
Goal
g0z (g5g))))
Rotate the goal (+t) and the starting poses A by (90*)
Start
WN (g52)
5
my (822
gag my (g5
Start
Fixed Obstacle (yymmetry-breaking)
Figure 1: Overview of Partially Equivariant RL. Assume a simple goal-reaching task where the state space includes only agent and goal poses. When states are rotated by 90", the purple agent's s equivariant action gaı remains valid. Conversely, the green agent's rotated action gō2 collides with the obstacle, illustrating local bymmetry-beeaking in the true dynamics and restricting the use of strict rotational symmetry. To address this, our framework learns a gating function that detects such violations and activates an unconstrained policy nn for d2. retaining the sample efficiency of cquivariance while falling back to T only where group symmetry is invalid.
lose the sample efficcincy benefits of strict equivariance and can become unstable when symmetry- breaking is extensive, since equivariance still applied indiscriminately across the entire space.
To overcome this limitation, we introduce the framework of the Partially Group-Invariant MDP (PI-MDP), which selectively applies the group-invariant structure only in regions where symmetry preserved (Fig. [ Our approach builds on the derivation that local symmetry-breaking leads to one-step backup errors that propagate globally. By routing updates to standard Bellman updates under the true MDP, we limit the propagation of one-step backup errors across the space. In particu- lar, we detect symmetry- brrekkig-bbaiggg regions via predictor disagreement outliers between an equivariant and an unconstrained one-step predictor, and apply standard rather than equivariant updates on those outliers while retaining equivariance elsewhere. Building on this framework, we develop practical reinforcement learning algorithms for both discrete and continuous control that retain the benefits of equivariance in symmetric regions while remaining robust to substantial symmetry-breaking.
The contributions our work are summarized as follows: 1) We analyze how local symmetry violations induce global value error via one-step backups, clarifying when selective symmetry is beneficial. 2) We introduce the Partially Group-Invariant MDP (PI-MDP) and a practical RL for- mulation that uses equivariance where symmetry holds and falls back to standard updates where breaks. 3) Across state-based discrete and continuous control experiments, we show that our method retains the sample efficiency gains of equivariance in symmetric regions and remains robust as cenumeetr--brakinng increases, outperforming strict and parrooiiaateequivariin baselines.
2 RELATED WORK
Group equivariance in continuous control. Recent works have applied group equivariance to imitation learning and classical control (Zeng al. 2021 Ryu al. 2023 2024 Wang et al. 2024] Tie et al. 2025: Huang et al. 2024 Zhao et al. 2025, Seo et al. 2024 2023 2025a), demonstrat-
ing high data efficiency and generalization over baseline models. Parallel efforts have investigated
group equivariance in reinforcement learning (RL) (Van der Pol et al. 2020 Kohler et al. 2024
Wang et al. 2022a b. Tangri et al. 2024 Nguyen et al. 2023). showing improved sample effeiency compared to the conventional RL approaches. However, the effectiveness of equivariant RL remains limited in more general settings, such as robotic control tasks, where inherent symmetry-breate
often arises from factors including occlusions, environmental asymmetries, kinematic singularities, and complex dynamics.
Approximate equivariance. Recent studies have proposed relaxing strict group equivariance to handle symmetry- -ymmetry-breakingg in data (Finzi al. 2021a; Park et al. 2025 Wang et al. 2202c Romero & Lohit 2022 van der Ouderaa et al. [022] Hofgard et al. 2024). Such approaches introduce ap- proximate equivariance, enabling models to remain robust when exact symmetries do not hold. In
2

---

Published as a conference paper at ICLR 2026
reinforcement learning, approximate equivariant architectures have also shown improved robustness and efficiency against moumetry-breakinng (Finzi et al., 2021a Park et al.] 2025). However, these methods primarily focus on global relaxations of equivariance at the representation level. In trast, our approach addresses symmetry-breakee by minimizing local equivariance errors during the Bellman backup, thereby preventing their global propagation through value updates.
PRELIMINARIES
Reinforcement learning. We consider Markov decision process (MDP) defined as M (S, A, P, ?) where S is the state space, is the action space, R S A - IR the reward function, P(. 8. a.) is the transition kernel, and E (0, is the discount factor. The agent learns a policy to maximize the expected return, J E,, - I'1
s0 =
a. The Bellman
operator under policy i is (7 (s,Q) = -R(ss + B, P() the optimal (hard) Bellman operator is (TQ)(s, R(s,a) + Ba max Q(s', a')].
E (aa (Q a²)] while
Group equivariance. A symmetry is a transformation that preserves certain properties of system, and the set of all symmetries forms group (Bronstein al. 2021). A group representation is a homomorphism
G + GL(n) that maps each group element g E G to an invertible n x n matrix. A function X + Y is equivariant if py (x) f(ex (g)x), and group-invariant fz) = f(xx (g) Vg E G. X.
Group-invariant MDP. A group-invariant MDP (Wang al. 2022b
2021 is abstract denoted as MMa (8, A, P, 7). The optimal policy and optimal Q-function of the original MDP recoverable from the abstract MDP provided the reward and transition kernel are group-invariant:
MDP based on MDP homomorphisms (Ravindran & Barto 2001 2004).
R(s, a) R(gs,ga),
P(s ss,a) = P(gs' gs, Vg G.
SYMMETRY- BREAKKNG IN GROUP-INVARIAN MDPs
Most equivariant RL approaches assume the existence of group- invariant MDP (Sec. 3 (Wang et al. 2021 2022b, Mondal et al. 2022 Van der Pol et al. 2020 Tangri et al. 2024). However, many continuous control tasks (e.g., robotics) violate these assumptions in certain regions of the state-action space. We begin by analyzing how such and subsequently propagates into the learned value function.
-erentrr-bbeaaingg perturbs Bellman backups
Let (N Ry. PN,,), 7) denote the standard environment, and let MME(S, A, Rgg PE, 7) be group-invariant MDP defined on the same spaces. To construct such group invariant MDP from MN. we average the original rewards and dynamics over the symmetry group G:
RE(s, a) = -fc RN RN(gs, ga) du(g) PE(d'(s,a)= Pr(gs' ggs du(g),
where du(g) is the normalized Haar measure on (uniform measure for finite groups). This av- eraging ensures that RS and Pr satisfy the group-invariance condition, thereby making Mr the canonical group-invariant approximation of My. For (s,a) E x A, define pointwise discrepan- cies between My and Mg via
ca(s. = (R)(s,a) Re(s,a)),
cp(s, a) = / PN s, - PE(s') ssa))
(I)
where ER is the absolute reward difference and ep is the total-variation distance between next-state kernels. Using these pointwise discrepancies, we formalize ceesstteecciion symmetry-breaking as follows.
summetry-breaking Consider the true MDP My. We say that symmetry is preserved at (s, a) if both (n(s, a) = 0 and cp(s. 0: in this case, the group- invariant approximation My locally coincides with My. Conversely, if a) > 0 or ()(
Definition 1 (Per-state-actee
say that symmetry is broken at (8,
3

---

Published as a conference paper at ICLR 2026
In the terminology of Wang et al. 2023 itateacttion pairs with cn(s,a) > 0 or cp a) > fall into yymmetr-breakkin regimes (e.g., incorrect equivariance), where the assumed group action does not match the true MDP dynamics. Our formalism captures this mismatch at the MDP level via the reward and transition discrepancies (en, <p) and the resulting Bellman error in Lemma
We quantify how these local mismatches perturb single Bellman backup and how this error propagates to the optimal value functions. Let T, denote the Bellman optimality operator in MDP i E {N, E}. We assume rewards are uniformly bounded as [R((s,)) Rmax and that the discount factor satisfies E (0, 1); define Viaa :- Rmax/ (1 -7)-
Lemma (One-step Bellman error). For any bounded Q and any (s, 6Sx
(Tx)))ss (TEQ) a)) & SER(S, a) + 2 |VVoloo a).
If Q is an action-value function, then |101 S V max- hence ||Volloo S Vmax and we define the pointwise bound
Here Vi((') = max,, Q('', a') and [Voll < 11Q
8(s,a) (s 2 max ps,,x)(sss))
We next show that this local emror lifts to global bound on the optimal action-valu functions via contraction.
Proposition 1 (Value- function gap). Let Q7 be the optimal action-value function
Then MDP funciinn in
1
1 QElloo I
1101
The proofs of Lemma] I and Proposition are provided Appendix A.1
Intuition. Local symmetry-breaking introduces a one-step Bellman backup error 0(s, a) which prop- agates through repeated backups and is amplified by the factor (1 ( due to contraction. This results in global deviation bounded by -_=116|| which can cause suboptimal policies or unstable training. We visualize this propagation in a Frrid-World example, and show that strictly equivari- ant policy can fail learn (Appendix E}. Prior works mitigate such errors with global relaxations FFizzi et al. 202 la Park et al., 2025), whereas approach employs local corrections that are less conservative and effective when symmetry holds only piecewise.
PARTIAL GROUP-INVARIANCE IN MARKOV DECISION PROCESSES
In what follows, we present efficient method for handling local commetry--beakkinn Specifically, we propose a Partially Group-Invariant MDP (PI-MDP) that interpolates, for each state-action pair, between group-invariant MDP and the true environment.
5.1 PARTIALLY GROUP-INVARIAN MDP
Definition 2 (PI-MDP). Let the true MDP be MN = (S,A, Rx, PN,"N.N) and the group-invariant MDP be Mg = (S, A, Rg, PE,") 7)). sharing the same (S,A, Define a Partially Invariant MDP (PI-MDP) Ми = (S,ARRA RH,PH,,A, 7,)) with measurable gating function À S x A [0,1]
RH a) = (1 X(s, Rg(s, a) + x(s. Rx(s, Pa s,a) = (1 A(s. a)) Pe( s, + A(s, a) PN s.
Since 0 < A(s, a) < for all (x,a) and both (RE, Pæ) and (RN, Px) are valid, (RH, Pi)) defines valid MDP. We then characterize the partially group-invariant optimality operator induced by the gating function.
Theorem 1 (Partially group- invup-nnvariant optimality operator). Let Te denote the (hard) Bellman opti- mality operutor MDP E (E, N, H), (T:Q) = R2(s, a) + (x [maxa Q(8', a d')].
For any bounded Q A RR and all (s, a),
(Tn a) (1 Xss, a)) (TE a) Ms,a) (T,))
(3)
REl. |Rv| & Rmax and 7 (0, 1), then Tn is -ccottraction and has unique fixed point Qi
4

---

Published as a conference paper at ICLR 2026
We next bound the deviation of the fixed point from the true optimum. Corollary (Proximity bound). Let Q'v be the optimal action-value of the true MDP Mx, and let 5(s,a) a) be the one-step pointwise Bellman error bound. Then
1 11 - lx - 1-7
- (1- 2) 1)0
(4)
Moreover, the right-hand side of Eq. (4) is zero whenever, at every (s, either X(s, a) (the gating function routes to the true MDP) or the group-invariant MDP coincides with the true MDP at (s, a), that is, Rg(s, a. (s, a) and PE(. 8, (- | s, a). Consequently, symmetric pairs contribute zero via MDP coincidence, and byrmetry- pairs contribute zero when À correctly gates
The proofs of Theorem [and Corollary cann be found in Appendix A..
Intuition. By gating the reward and transition kernels, the PI-MDP is itself a valid MDP. Its opti- mality operator satisfies the affinity identity in Eq. (3). Since convex combination of c-contraction is again a 7-contraction, Tи admits a unique fixed point Q2z Corollary | bounds the deviation from the true optimum: the gap 1|Q Q7 is controlled by the gated mismatch term on the right-hand side of Eq. (4). scaled by (1 - ----)--- When À correctly localizes symmetry- tnrmeitr--beakkngg Qi closely tracks Q7 while reverting the group-invariant MDP where symmetry holds. We provide the extension of the PI-MDP to the entropy-regularizee (soft) setting in Appendix A.33
Remark (Hard gating). All results above hold for any measurable gating function X: S A 10, 1]. When a(s,a) E {0, 1}, the PI-MDP routes pointwise (Rg, PE) symmetric pairs and (RN, Px) otherwise. In our algorithms, we adopt this hard- gating regime for simplicity and empir- ically more stable training (Fig.
6 PARTIALLY EQUIVARIANT REINFORCEMENT LEARNING
This section introduces partially equivariant reinforcement learning (Algorithm I for the PI-MDP setting (Sec. (.1).. We (i) learn a gating function A(s, a) that localizes conmetry--reakinng and (ii)
couple À to equivariant and unconstrained value/policy heads.
6.1 LEARNING A(s, a) VIA DISAGREEMENT SUPERVISION
By Corollary [ the value gap vanishes when X(s, a) = 1 on symmetry-breakinn pairs and A(s, a) = 0 where the proxy and true MDPs coincide (assuming an oracle binary gate under local symmetry). To approximate this behavior, we train a gating function (s, a) E {0, 1} using the disagreement between two one-step predictors: an equivariant predictor PE : S x A + IR" constrained to respect the group symmetries of Mg and an unconstrained predictor PN S A + R", where is the dimension of the predictor output.
Concretely, Pg and PN are trained on transitions (s, a, r, s') to approximate the one-step MDP components, i.e., the transition kernel and, when used, the reward function. We define a scalar disagreement score
d(s,a) = D(PE( s, a)) Px | s,a)),
where D is a discrepancy measure (e.g., squared error between next-state predictions, total-variation distance between transition distributions, or an é distance between predicted rewards). Detailed implementations of PE. PN. and D are given in Appendix B.I
At symmetric pairs (i.e., where ( a) = < a) = 0), the equivariant predictor Pg consistent with Mg. which coincides with the true MDP My, and the unconstrained predictor Py can match the same behavior. In this case, the disagreement d(s,a) remains small. At symmetry-breakee pairs, the assumption of group-invariance is violated: by construction, PE can only represent the group-averaged surrogate PE. whereas PN can approximate the true kernel PN, so their predictions diverge. Consequently, d(s, a. tends to be larger precisely in those regions where (RE, Rx) or (Pg, Pw) disagree, providing an indirect detector of symmetry- breaky-. We model symmetry- breaking transitions as belonging to the upper tail of the empirical distribution of disagreement scores d(s,, a) and use these high- gigh-disagremment samples to define pseudo-labee labels y(s, a) E {0, 1) for training d, with binary cross-entropy loss:
5

---

Published as a conference paper at ICLR 2026
Algorithm Partially Equivariant Reinforcement Learning (PERL)
Require: Replay buffer D, critics QE: QN. policies я EE, TN
Require: Dynamies predictors Pg. PN. gating functions Aw, Aç, targets Q, Au
1: Initialize all networks
2: Initialize running statistics (ji, .) for disagreement
3: for t= 1 to 7 do
4: Sample az ~ TTl- 8t)
> gated policy from Eq.
5: Store (st, (s, Pts Sr+1) in D
6: Train predictors Pr. and PN to minimize the predictive loss Spred
see App.
B.
7:
Compute disagreement d(s,a) = D(PE(: s, s, Px(- (- (1.ss,a)) s,)))
P see Sec. 6.1
App.
B.1
8:
Update the running statisties (n, o)) over the disagreement d(s,aa
9:
Update λ and ) using BCE-loss (Eq. (5)) and expectile regression (Eq. (8)
Update the crities with the objective (Eq.. (9)
11: Update the actor with the objective (Eq. (10)
SAC only; DQN uses greedy arg max
12: Soft update Q and An
13: end for
Cx(w)= (ssaa-/ y(s, , a) log Aw (s, - -(1 (1- y(s, a)) log(1 (s,
(5)
where D is the replay buffer. The gating network is optimized only through Eq. (5) during RL updates of the value function and policy, we treat A as fixed and do not backpropagate RL gradients into its parameters (see Appendix for implementation details).
6.2 PARTIALLY EQUIVARIANT REINFORCEMENT LEARNING
We couple the learned gating function to the critic and the actor, thereby implementing the PI-MDP framework under function approximation while training entirely in the true environment MN.
Gated value mixtures under the true MDP. We parameterize the critic as gated mixture:
Qo(s, a) = (1 - (s, a)) QE, + \w a) (s)
(6)
where QE is equivariant critic constrained by group symmetries and QN is unconstrained critic with no symmetry bias. The gating function A x A + {0, 1} routes between the two networks. Conditioned
the binary gating d (cached per minibatch and used with stop-gra -radddrd,,n),, our TD-based critic (e.g., DQN, SAC) learns under MN the best approximation within this mixed hypothesis class. With binary gating, the mixture reduces a hard switch, activating either QE or Q N depending on whether the state-action lies in a symmetric or bymmetry-breakkng region.
Idealized compatibility (binary oracle gating). If X(s, a) E {0.1) perfectly separates symmet- ric from broken regions and, on symmetric regions, the averaged dynamics coincide (Pg, Rg) (PN, R then the partially group-invariant operator Th identical the true operator Ty- this idealized case, our TD targets exactly match (T=Q) solution in Theorem This motivates the use of as a "local oracle" for symmetry-breakee In practice, approximate this oracle by the learned gating function X producing binary decisions as described above.
and the mixture recovers the interpolating
Gated policy and actor gating function. For the policy, employ state-only gating function Ac : S + {0, 1} and define a product-of-experts (PoE) blend
( T. s) 1~A((s) TN ()) Ac(s).
This form naturally arises from SAC policy improvement: given the critic mixture Qo (1 >. >=)QE= + N.. the information projection in SAC yields a PoE between the energy models exp(Qu/a) and exp(QN// /a) (see Appendix A.4 for details). While a fully state-action gate in would be theoretically appealing, it is intractable in practice because the normalization constant of Eq. (7) would depend on a. Instead, we use a state-only gate A²(s), which is aligned with the critic gating function via conservative aggregation loss. This conservativeness is crucial: since

---

Published as a conference paper at ICLR 2026
Hemmetry--reakkiie may occur only for subset of actions, As(s) should activate whenever any tion at state s is flagged by (s, a). This conservative choice does not compromise optimality, as taking the maximum ensures that any critical symmetry breakingg is accounted for while leaving the optimal policy unchanged.
Cx(C) = B(sa)) L -
(8)
where Ly is the expectile loss (Kostrikov et al. 2022). Taking + approximates the max, op- erator, ensuring that {s(s) conservatively reflects the maximum symmetry-breakinn signal across actions. Per sample, Eq. 7 thus collapses to hard switch between zg and nN, retaining inter- pretability and computational tractability (details in Appendix(B.2)
Besides the learned state-dependent gate Aç(s), we also consider a sampled-max -ax variant that ap- proximates max, Am (8, a) by taking the maximum over (8, for K sampled actions. In our ablations (Fig. 6). K E {4,8) performs similarly to )e(s), making this reasonable choice for lighter architecture, though the learned state gate is slightly more robust when symmetry-breaking is sparse and casy to miss with few sampled actions.
Training. We train Qo and Ta using standard objectives from deep RL: DQN (Mnih et al. 2013) for value-based methods and SAC (Haarnoja et al., 2018) for actor-critic methods, substituting in our gated parameterizations. standard off-the-shelf algorithms, while the gates A and )ę provide adaptive control over when equivariance exploited and when is suppressed.
this way, the partially equivariant framework is realized within
JQ(0)= = E(s.a,.'' 1
B ())(s - t max max(Q((s'aa))
(9)
where d denotes target parameters and, Qo(s, = (1 (s, a)) (s, a) QN.ols,a..
(Ø) = E
aa log To(a log 8) 11n;-1,2 Q0, Q0, (s.
a= tanh (go(s, (10)
anD (0.
where is the entropy temperature used SAC (Haarnoja al. 2018 , and log la (s) log T s) (s{s) log TN s). Please refer to A ggrrthm I for the pseudocode, and Appendix B for more details.
For the network architecture, we use separate trunks for the critics, the policy, and the one-step predictors, which provides stable training and a clean separation between equivariant and non- equivariant components. We also explored several trunk-sharing variants for parameter efficiency, but they did not consistently improve performance and sometimes harmed stability (see Fig. 8)
EXPERIMENTS
Our experiments aim to answer two main questions: (1) How does our method compare terms of sample efficiency against the conventional RL and strictly equivariant methods? (2) How robust is our method to symmetry-breakingg relative to the other approximately equivariant approaches?
7.1 EXPERIMENTAL SETUP
We evaluate across discrete and continuous control settings: (1) Grid-World for intuitive analysis,
and (2) Locomotion and Manipulation benchmarks using MuJoCo (Brockman et al. 2016)) Fetch Reach (Plappert et al. 2018), and the UR5e manipulator (Tassa et al. 2018:] Chuang] (023). To systematically test robustness, we introduce fixed obstacles in the Grid- World that cause the transi- tion kernel to deviate from ideal rotational symmetry. In the continuous tasks, symmetry-breakinn
naturally arises from realistic constraints like contacts and joint limits. We compare PE-DQN and
PE-SAC against vanilla RL,, strictly equivariant methods, and approximately equivariant baselines
Finzi et al. 202 Park et al. 2025). Details on the environment are provided in Appendix D
Grid-World. We use a discrete C--symmetric gridworld as a lightweight testbed for analyzing robustness to symmetry- The MDP state consists only of the agent and goal coordinates;
7

---

Published as a conference paper at ICLR 2026
Grid World
Hopper
Ant
Swimmer
FetchReach
UR5eReach
(a) Discrete space
(b) Continuous space
(c) Continuous space (manipulation)
(locomotion)
Figure 2: Benchmark environments. We evaluate our method across both discrete and continuous control tasks under gymmetry-breakiing conditions. Specifically, use the Grid-World environment for the discrete case, and locomotion and manipulation tasks for the continuous case.
obstacles are part of the environment layout but metry is broken by placing fixed obstacles that cause the induced transition kernel (N a) over these coordinates deviate from the ideal rotational symmetry. By varying the number of obstacles, we control the degree of remmetry-breakeee and can clearly examine how PE-DQN adapts as the extent of symmetry violation increases.
encoded in this state representation. Sym-
To study reward-level symmetry bbeackin, we also consider a variant in which subset of obstacles becomes passable but returns a negative reward when traversed, leaving the transition structure un- changed while altering the reward function. Finally, we construct a stochastic variant with numerous obstacles, in which actions result in randomized neighboring transitions, creating complex dynamics that test the robustness of our disagreement-baaee gating under complex dynamics.
Locomotion. We evaluate continuous-coni MuJoCo benchmarks using the same symmetry specifications as RPP (Finzi et al. 2021a), which include both exact and approximate symmetries. This setting allows to test whether PE-SAC can extend the sample-efficinnyy benefits of equivari- ance from discrete Grid-World to challenging continuous-control tasks, while remaining robust to bymmetry- factors such as external forces or reward perturbations. All baselines are trained with SAC.
Manipulation. We evaluate in manipulation settings, considering two reach tasks with 50(3) symmetry. Fetch Reach serves a simpler case, where the end-effector constrained perpendicular to the floor and the goal is specified only by (x,, y. 2) position. In contrast, UR5e Reach allows free end-effector orientation in addition position, with goal specified as SE(3) pose that includes both position and orientation. The inclusion of orientation control makes the task more represen- tative of real-world manipulators. This progression from Fetch to URSe enables us test whether PE-SAC scales from constrained to more realistic manipulation scenarios. Symmetry-breakke urally arises from collisions, floor contacts, and kinematic singularities. All methods use the same SAC backbone for comparability.
nat-
7.2 ANALYSIS
In Fig. we show returns in Grid-World as the number of obstacles increases. When no symmetry- breaking factors are present, PE-DQN quickly converges to X = O and behaves like purely equiv- ariant agent, matching the performance of strictly equivariant DQN. As obstacles are added, strictly equivariant DQN degrades much more rapidly than the other baselines, while approximately equiv- ariant methods offer only minor gains over vanilla DQN both sample efficiency and final return. In contrast, PE-DQN maintains strong performance across all obstacle counts, indicating robustness to localized commetty--braainng and aligning with our theory that value errors remain controlled by en and Ep when the gate routes away from equivariance in mismatched regions.
Fig. 4 evaaluates Grid-World variants that isolate reward-level symmetry-breakinn and complex dy- namees. For reward-level wymmetry-breakingg (s, (N, F.) and define disagreement the sum of transition- and reward-level terms
augment each predictor with reward head
8

---

Published as a conference paper at ICLR 2026
Obstacles
Obstacles
Obstacles
Avarage Retum
0.5
0.5
0.5
0.0
Avarage
0.0
Avarage
0.0
0.5 Steps
Stepe
x16
Steps
x107
x10
Obstacles
40 Obstacles
Bvemag Raturn
Average Riturn
PE-DQN (Ours)
RPP-DON
APPROXEQUI-DON
-10
-.0
Stepe
x167
Steps
<10
Figure 3: Performance comparison the discrete space (Grid-World) environment. We eval- uate the average performance over 100K steps with five random seeds. Shaded regions denote standard error. We vary the number of obstacles, which act as symmetry-breaking factors. PE-DQN consistently outperforms the baselines, and the performance gap widens as symmetry- breaking creases, demonstrating both robustness and sample efficiency.
(a) Transition reward symmetry-breakinn
(b) Stochastic transitions
Obstacles
Obstacles
40 Obstacles
Return
Average Return
0.6
PE (Ours)
EQUIDOO
Average
0.0
RPP.DON DQP
-6
-0.
DQN
APPROXEQUI-DON
0.5 Steps
0.6
x10
Sleps
Steps
Figure 4: Performance comparison Grid- World under reward-level symmetry-breakieee and complex dynamics. Results averaged over 100K environment steps with five random seeds; shaded regions denote standard error. (a) Reward-level yymmetry-breaking is introduced by making half of the obstacles passable while assigning negative reward upon traversal, in layouts with 10 and 30 obstacles. (b) Complex dynamics setting with stochastic transitions in 40-obstacle layout. PE-DQN consistently outperforms the baselines both settings, indicating robustness to reward- level symmetry-breakee and challenging dynamics.
(Appendix [.I]] This setting converts a subset of obstacles into passable but penalized cells, so sym- metry is broken purely through the reward while the transition structure remains unchanged. Across the 10- and 30-obstacle layouts, PE-DON consistently achieves the highest returns, with RPP-DQN as the strongest baseline but still trailing in both sample efficiency and final performance.
In the complex-dynamics dynamies variant with 40 obstacles and stochastic transitions, random slips par- tially mask transition-level symmetry- breaakng (e.g., the agent can occasionally bypass obstacles by chance), which allows strictly equivariant and Approximately Equivariant DQN to recover reason- able performance. Nonetheless, PE-DQN continues to attain the best returns and learning speed, indicating that the disagreement- based gate remains effective even when the underlying dynamics noisy and harder to model.
In Fig. 5, we report results on continuous-comiee control locomotion and manipulation tasks. In Hopper, PE-SAC learns faster than all baselines and reaches strong plateau, although vanilla SAC attains slightly higher asymptotic return. In Ant, PE-SAC clearly dominates both sample efficiency and final performance. In Swimmer, where symmetry is nearly exact, the strictly equivariant and Ap-

---

Published as a conference paper at ICLR 2026
Hopper
Ant
Swimmer
2000
Average Raturn
Average
2000
4002
Average
s0-
1000
Steps
x10
Steps
K10*
Steps
X10
FetchReach
UR5e
-100-
PE: (Ours)
RPP-SAC
Average
-200-
EQUI-SAC
APPROXEQU-SAC
-002
Steps
x107
Steps
K107
Figure 5: Performance comparison in the continuous space environments. Results are averaged over 1M training steps in MuJoCo tasks, and 30K, 500K steps in the Fetch, UR5e Reach environ- ment, using eight random seeds from locomotion tasks and five random seeds from manipulation tasks. Shaded regions denote standard error. For RPP (Finzi et al. 2021a), we re-ran the official code. Discrepancies with the reported numbers arise because RPP reports "max over steps" rather than average performance. PE-SAC consistently outperforms all baselines across these tasks.
proximately Equivariant SAC baselines achieve the highest final returns, while PE-SAC converges quickly to slightly lower but competitive level, reflecting the advantage of enforcing exact symme- try in this regime. Fetch Reach, PE-SAC, exact equivariant SAC, and Approximately Equivariant SAC perform similarly. In UR5e Reach, where symmetry-breakien dynamics and free or ientation, the strictly equivariant and Approximately Equivariant SAC variants become unstable or collapse, whereas PE-SAC remains stable and attains the best overall returns by shifting toward the non-equivariant head.
is substantial due realistic
Overall, these results support our central claim: by selectively mitigating local equivariance rors. PE-DQN and PE-SAC retain the sample efficiency benefits of equivariance in symmetric re- gions while remaining robust in symmetry-bree broken regimes across discrete, continuous, and realistic robotic environments.
CONCLUSION
In this work, we introduced the PI-MDP, framework that mitigates global error propagation from local symmetry- breakkng Building on this foundation, we developed Partially Equivariant RL (PE-RL) algorithms-PE-DD
for discrete control and PE-SAC for continuous control-that con- sistently improved sample efficiency and robustness over conventional RL, exact-equivariant meth- ods, and approximate baselines.
The main limitation is computation: auxiliary predictors and gates increase training time. Fur- thermore, under pervasive symmetry-breaking (c.g., gravity), the framework largely defaults to non-equivariant networks, offering marginal benefits standard RL. However, since symmetry- breaking in typical control tasks is localized, our approach successfully preserves both sample effi- ciency and robustness to symmetry- breaktrrybbbeaig....
Future work includes extending PE-RL to vision-based control, advancing the practicality of symmetry-aware reinforcement learning for real-world continuous control.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We would like thank Hyunwoo Ryu for insightful discussions that helped improve this manuscript. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the This work was also supported by the Korea Institute of Science and Technology (KIST) Institutional Program (Project No.2E33801-25 (-88001-2--15) the Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea government (MSIT) (No. RS-2024-00457882, AI Research Hub Project), and the Institute of Information Communications Technology Planning Evaluation (ITTP) grant (RS- 0020-I120136 Artificial Intelligence Graduate School Program (Yonsei University)). Joohwan Sco and Roberto Horowitz are funded by the Hong Kong Center for Construction Robotics Limited (HKCRC).
Korea government (MSIT)
REFERENCES
Greg Brockman, Vicki Cheung. Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym, 2016.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13477 2021.
Tony F Chan, Gene H Golub, and Randall J LeVeque. Algorithms for computing the sample vari-
ance: Analysis and recommendations. The American https://gittubb.co//aarroo
1983.
lan Chuang.
Manipulator-mujoco..
pelitouttor--oo Mu joco 2023.
Mare Finzi, Gregory Benton, and Andrew G Wilson. Residual pathway priors for soft equivariance constraints. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pp. 30037-30049, 2021a.
Mare Finzi, Max Welling, and Andrew Gordon Wilson. practical method for constructing equiv- ariant multilayer perceptrons for arbitrary matrix groups. In International Conference on Machine Learning (ICML). pp. 3318-3328. PMLR, 2021b.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Confer- ence on Machine Learning (ICML), pp. 1861-1870, PMLR, 2018.
Elyssa Hofgard. Rui Wang. Robin Walters, and Tess Smidt. Relaxed equivariant graph neural net- works. ICML. Workshop on Geomelry-gruundel Representation Learning and Generative Mod-
eling (GRaM), 2024. Extended Abstract.
Haojie Huang, Owen Howell, Dian Wang, Xupeng Zhu, Robin Walters, and Robert Platt. Fourier transporter: Bi-equivariant robotic manipulation in 3d. In International Conference on Learning Representations (ICLR), 2024.
Diederík P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
Colin Kohler, Anuj Shrivatsav Srikanth, Eshan Arora, and Robert Platt. Symmetric models for visual force policy learning. In IEEE International Conference on Robotics and Automation (ICRA), pp- 3101-3107. IEEE, 2024.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-
learning. International Conference on Learning Representations (ICLR), 2022.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1.1iv:1312.555 2013.
11

---

Published as a conference paper at ICLR 2026
Arnab Kumar Mondal, Vineet Jain, Kaleem Siddiqi, and Siamak Ravanbakhsh. Eqr: Equivariant representations for data-efficient reinforcement learning. In International Conference on Machine Learning (ICML), pp. 15908-159266 PMLR, 2022.
Hai Huu Nguyen, Andrea Baisero, David Klee, Dian Wang, Robert Platt, and Christopher Amato. Equivariant reinforcement learning under partial observability, In Conference on Robot Learning (CoRL), pp. 3309-3320. PMLR, 2023.
Jung Yeon Park, Sujay Bhatt, Sihan Zeng. Lawson LS Wong, Alec Koppel, Sumitra Ganesh, and Robin Walters. Approximate equivariance in reinforcement learning. In International Conference on Artificial Intelligence and Statistics (AISTA7S), 2025.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Pow- ell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, al. Multi-goal reinforce- ment learning: Challenging robotics environments and request for research. arXiv preprint IrXiv:1802..0 09464, 2018.
Balaraman Ravindran and Andrew Barto. Symmetries and model minimization in markov deci- sion processes, 2001.
Balaraman Ravindran and Andrew G Barto. Approximate homomorphisms: A framework for non- exact minimization in markov decision processes. 2004.
David W Romero and Suhas Lohit. Learning partial equivariances from data. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. -6446,, 2022.
Hyunwoo Ryu, Hong-in Lee, Jeong-Hoon Lee, and Jongeun Choi. Equivariant descriptor fields: SE(3)-equivarian energy-base -ased models for end-to-end visual robotic manipulation learning. In International Conference Learning Representations (ICLR), 2023.
Hyunwoo Ryu, Jiwoo Kim, Hyunseok An, Junwoo Chang, Joohwan Sco, Tachan Kim, Yubin Kim, Chaewon Hwang, Jongeun Choi, and Roberto Horowitz. Diffusion-edfs: Bi-equivariant denoising generative modeling SE(3) for visual robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18007-18018, 2024.
Joohwan Seo, Nikhil Potu Surya Prakash, Alexander Rose, Jongeun Choi, and Roberto Horowitz. Geometrie impedance control on SE(3) for robotic manipulators. IFAC-PapersOOlline, 56(2): 276-283, 2023.
Joohwan Seo, Nikhil PS Prakash, Xiang Zhang, Changhao Wang, Jongeun Choi, Masayoshi Tomizuka, and Roberto Horowitz. Contact-rich SE(3)-cquivarian robot manipulation task learn- ing via geometric impedance control. IEEE Robotics and Automation Letters, 1(2)1100815515, 2024.
Joohwan Sco, Arvind Kruthiventy, Soomi Lee, Megan Teng, Xiang Zhang, Scoyeon Choi, Jongeun Choi, and Roberto Horowitz. Equicontact: A hicrarchical SE(3) vision-to-force equivariant policy for spatially generalizable contact-rich tasks. arXiv preprint arXiv:2507. 100. 2025a.
Joohwan Seo, Soochul Yoo. Junwoo Chang, Hyunseok An, Hyunwoo Ryu, Soomi Lee, Arvind Kruthiventy, Jongeun Choi, and Roberto Horowitz. SE(3)ecqivariaat robot learning and control: A tutorial survey. International Journal of Control, Automation and Systems, 2375):12 2025b.
Arsh Tangri, Ondrej Biza, Dian Wang, David Klee, Owen Howell, and Robert Platt. Equivariant offline reinforcement learning. arXiv preprint arXiv:2406. 13961, 2024.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmalcki, Josh Merel, Andrew Lefrancq, al. Deepmind control suite. arXiv preprint 1801 00690, 2018.
Chenrui Tie, Yue Chen, Ruihai Wu, Boxuan Dong. Zeyi Li, Chongkai Gao, and Hao Dong. Et- seed: Efficient trajectory-level SE(3) equivariant diffusion policy. In International Conference on Learning Representations (ICLR), 2025.
12

---

Published as a conference paper at ICLR 2026
Tycho van der Ouderaa, David W. Romero, and Mark van der Wilk. Relaxing equivariance straints with non-stationary continuous filters. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pp. 03818-33830, 2022.
Elise Van der Pol, Daniel Worrall, Herke van Hoof, Frans Oliehoek, and Max Welling. MDP homo- morphic networks: Group symmetries in reinforcement learning. In Advances in Neural Informa- tion Processing Systems (NeurIPS), volume 33, pp. 1199-42101 2020.
Dian Wang, Robin Walters, Xupeng Zhu, and Robert Platt. Equivariant q learning in spatial action spaces. In Conference Robot Learning (CoRL), pp. 1713-1723, PMLR, 2021.
Dian Wang, Mingxi Jia, Xupeng Zhu, Robin Walters, and Robert Platt. On-robot learning with equivariant models. Conference on Robot Learning (CoRL), 2022a.
Dian Wang, Robin Walters, and Robert Platt. SQQ2)equivvarian reinforcement learning. In Interna- tional Conference on Learning Representations (ICLR), 2022b.
Dian Wang, Xupeng Zhu, Jung Yeon Park, Mingxi Jia, Guanang Su, Robert Platt, and Robin Wal- ters. A general theory of correct, incorrect, and extrinsic equivariance. In Advances in Neural Information Processing Systems (NeurIPS), volume 36, pp. 40006-40029, 2023.
Dian Wang, Stephen Hart, David Surovik, Tarik Kelestemur, Haojie Huang, Haibo Zhao, Mark Yeatman, Jiuguang Wang, Robin Walters, and Robert Platt. Equivariant diffusion policy. In Conference on Robot Learning (CoRL., 2024.
Rui Wang, Robin Walters, and Rose Yu. Approximately equivariant networks for imperfectly sym- metric dynamics. In International Conference on Machine Learning (ICML), pp. 23078-22822 23091.. PMLR, 2022c.
Maurice Weiler and Gabriele Cesa. General B(2) steerable CNNs. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al. Transporter networks: Rear- ranging the visual world for robotic manipulation. In Conference on Robot Learning (CoRL), pp. 726-747. PMLR, 2021.
Haibo Zhao Dian Wang, Yizhe Zhu, Xupeng Zhu, Owen Lewis Howell, Linfeng Zhao, Yaoyao Qian, Robin Walters, and Robert Platt. Hierarchical equivariant policy via frame transfer. In International Conference on Machine Learning (ICML), 2025.
13

---

Published as a conference paper at ICLR 2026
A THEORETICAL PROOFS
A.1 PROOF OF LEMMA[ANT AND PROPOSITION 1
Lemma (One-step Bellman error), For any bounded Q and any (s,a) E S A
(TxQ)(s, (TeQ)(s,a)) ER(S,a) + 27lVVeloo
Proof. By the triangle inequality,
((TN))s, (TE
Rx( RE(s, a) Px (o(8') By Pa(toe))))))))))))
- ER(s, a) + Epx [V] - Eps [oll
Using the total-variation inequality |pl.f] Eolf]| s 2 QV with the definition of ep
in Eq. I and f = VQ.
EP [Vo Erg Voll s V/llolcc
Combining the bounds leads to the lemma.
Proposition 1. (Value-function gap). Let Q be the optimal action-valee function in MDP i. Then,
1
11Q - Qirlloo I 1181100
Proof. Since Qiv = TNQi and Q3 TEQE we have
12 - V8l1 = || Q - TEQEll I 1 - T + I[TQ TEQElloo
The Bellman optimality operator a c-contraction in the sup norm, so
|T - loo S >/12% - Qilloo
By ( with Q Q' and the bounded [| |l S Viaxs we have
TN - TeQEello < ||0]|oo,
where 8(s, a) represents the pointwise one-step approximation error defined in Eq. 2 the inequalities gives
Combining
1i Qilloo <>11 - Qi + |18|
Rearranging results
1 - 110|
1 - Qooloo <
which completes the proof.
A.2 PROOF OF THEOREM 1 AND COROLLARY
Theorem (Partially group-invariant optimality operator). Let Te denote the (hard) Bellman opti- mality operator MDP: E (E, N, H). (T, Ra(s, a) + Ee (-(laa) [max Q(s', a')). For any bounded Q A IR and all (s,a),
(T. a) (1 X(s a)) (TEQ) a) + Ms, a) (TxQ)(s,a).
(11)
If Rgl, Rxd S Rmax and 7 E (0, 1), then Tn is a c-cottraction and admits a unique fixed point Q
14

---

Published as a conference paper at ICLR 2026
Proof. Identity Eq. 11 By Definition 2 for any (s,
(Tn())s,aa) ((s,a)) ()(s, + Pa(laa)) max ((xx((s
+ *(8, Rx + Ee Bewpx max Q(5'
which equals (1- Q XTvQ pointwise.
Contraction. Let Q1. Q2 be bounded. Using Eq. and that TE, TN are 7-contractions,
= -1 M(s, (T) (8, T a)) + A(s,
- (1 M(s, a)) ||T - TEQ21lx ((s, [T - Q2
>> Q211
Taking the supremum over (s. a) gives
I TAQ22l S > - Q211ax
Bounded rewards ensure Tx maps bounded Q into bounded Q. By Banach's fixed point theorem, Ti has unique fixed point Qi+-
Corollary (Proximity bound). Let QNy be the optimal action-value of the true MDP MN, and let ö(s, a) be the one-step pointwise Bellman error bound. Then
11 - 22o S 1-7
(1 2)
(12)
Proof.
11Qh 1 1Tn TNQNIls
S 11T 1wol 1|T -N 1ox
- 11 - Qivllx 1-1(-A))TE05 -x
Expanding pointwise,
(TEQ% - = Rx)s Ry (B)-PE())a)))
Vx((')] (Vr B P(a
By the definition of total variation distance,
EPp [~ Ep [v - 2cp(s.a) Vmax:
where Vina = Rmax/ (1 - ep, a) 1 fs Px 8, a) Pr (6611,,a) a)) ds' introduced in Eq.
and 6(s, a) defined in Eq. 2 Rearranging gives Eq. 12
A.3 PARTIAL GROUP-INVARIANCE SOFT MDPs
Since the PI-MDP defined in Definition is valid MDP, the soft policy iteration framework (Haarnoja al. 2018) applies unchanged. We show the evaluation identity and the standard im- provement step for completeness.
Policy evaluation. For a fixed policy m, define the soft state value (S((S) := Bana(tle) Q(s.a) a alogs(a (a [s]] with temperature > 0. The soft Bellman operator under M is
(T)))(s, a) = R& (s, a) + 7 [
Writing A= = A(s,a) a) for brevity, RH and Pи (Definition leads to the pointwise identity
(THQ) = (1 A) Rg (s, + X (s,
X) VV
(s') + +A (Bsx-
= =(1 -11-A)(T - (s)
Thus, soft evaluation under M1r is the same convex combination the component evaluation as in hard (max) case.
15

---

Published as a conference paper at ICLR 2026
Policy improvement. Treating À as fixed, the soft policy improvement step follows the SAC for- mulation:
exp (0*T (x(s)
TktI [s) arg min DKL x(. 8)
(13)
where (s) the normalizing constant. Alternating evaluation under (T and the update Eq. (33
is exactly soft policy iteration on Mu Under the standard assumptions of Haarnoja et al. (018). this admits a unique soft fixed point and corresponding policy.
A.4 POLICY PARAMETERIZATTIN AND TRACTABILITY FOR PE-SAC
PoE from SAC policy improvement. For fixed gating function À 8 A [0, and Q+ (1 - E + AQN, the SAC information projection (for each 8)
exp(Qo(8,-)/c Zo(s)
( 6) arg min DKL T(. [s)
has unique solution
x (a 8) x exp (1- (1=)))(((a))22Qxx
= [[xp(Qx(s,a)/o))
1-X(s,a) [exp (cxp(QNss,a//o)) M(.a)
is state-only, X()), then the normalizers of exp(QF/O and exp(Qx/a)) are constant and factor out, leading to the geometric mixture of normalized policies:
" (- (s x TE(. 8) 1-)(s) TN(- s) A(#)
where
TE( 18) x exp(Q )/a), TN(- (s) x exp(Qx(s,))/a)
Why an action-dependent gating function breaks reparameterization. Write the energies fE = Qs/a and fN = QN/a. Define the unnormalized density
Zo(s) / uola s) da.
ua(a exp a(s, fe(s,a) as
When x A(s,a) a), the normalizer (s(s) has no closed form and its gradient with respect to the parameters inside X, fE, fN is intractable. Therefore,
log Ta(a s) (1 A) a) AfN a) log (s)
cannot be evaluated with a tractable pathwise sampler = (8, t), so the reparameterized SAC actor objective
(() = E.,. (a log (a |s) Qs(s, a)]
tractable. This motivates state-only gating function in the actor.
Gaussian policy with squashing (state-only gating). Following SAC, we use an unbounded Gaussian for a pre-squash variable IR' and apply an elementwise tanh obtain bounded actions = tanh(u) u). Let the two pre-squash Gaussian densities be
(E(u s) = N(u (s)s), (s(s)), PN(u |) = =Nuu;
En (s)),
and let the gating function be state-only, À A(s) E [0, 1]. Define the unnormalized product
(u s) (u s) 1-X(s, (v(u s) AGe)
Since the exponents are constants for fixed s, Per proportional to a Gaussian. particular,
(u s) Auu; (s). (s)),
Mn (s) (1 E=) X (s), (s),
(14)
(s) (s) (1 >(s)) (s)PE(s) E))
16

---

Published as a conference paper at ICLR 2026
With tanh(u) and the variiables formula (cf. SAC SaCmHaamoja al. 2018 Eqs.
(2)),,
da tu
(H 8) pH("l det
log TH s) lug
Σ
log tanh" (u+)):
where u arctanh(a) and the Jacobian 0a/Ou gating is binary, A(s) {0, 1). Eq. (14) reduces the corresponding expert.
/u is diagonal with entries - -tanh (ui). When the
B IMPLEMENTATION DETAILS
B.1 DETAILS FOR LEARNING A AND PREDICTORS PE, PN
One-step predictors and disagreement metric. We train two one-step predictors on replay. In the environment with discrete state spaces (e.g., the grid-world with obstacles), the predictors param- eterize the transition kernel (s 8, a) for E (E,N) implemented as a categorical distribution over the possible next states. In environments with continuous state spaces, the predictors output the increment to the next state, P. (s, a) ($, a). intended approximate the transition dynamics of ME and MN. respectively.
In the discrete case, each predictor is trained via cross-entropy loss transitions (s,a, a,,,ss).. s').
((s,,,)) log P,(s') | a)], ie [E,N).
pred
The discrepancy measure D PE, Px) is defined as the total-variation distance, consistent with the definition of ep(s, a) in Eq. [I]
d(s a) = Σ
PPr(d s, - PE(6 88 s.a)--
SESS
In the continuous case, each predictor is optimized by minimizing mean squared error on the state increment As == S:
(e.,,,'")~ (| (s, 1x i (F,
pred
The disagreement is then defined as the squared difference between predicted increments:
d(s, a) | (s.
One-step reward prediction and disagreement metric. In the variants of Grid-World experi- ments (Fig. 4p. we additionally equip each predictor with reward head R, a), E NE,N), implemented extra head on top of the shared predictor trunk P; In this case, the transition and reward predictors are trained jointly on a,, 8') with the combined loss
log Pi(s' a))a) + a) - -11]
(E,
pred
The reward component the disagreement defined as an é distance, consistent with (R(s,)) Eq. I and the overall disagreement combines transition and reward terms:
d(s,a)
Σ (Nr 18 a. Pr(ss 8, s.a) + [tx(ss)) R e'ES
Disagreement thresholding and label generation. We maintain running statistics (Me, σx) of d(s, a) via the Welford algorithm (Chan et al. 1983). form raw threshold f; = nr + or (re- flecting the assumption that symmetry-breaking is sporadic), and then apply exponential smoothing:
Tt
1T 3) fr.
17

---

Published as a conference paper at ICLR 2026
Binary supervision is given by y(s. a) L[d(s, a) T.). Here, is a hyperparameter that controls what fraction the disagreement distribution treated as symmetry-breakiee We find that exhaus- tive tuning is unnecessary: rough estimate of whether symmetry violations sparse dense is sufficient to choose a robust K. (Fig.
For more challenging dynamies (e.g., Grid-World environment with 20,30, 30, 40 obstacles), found slight performance gain from a batchwise quantile-base variant: within each minibatch B, treat {d(ss a): (s,a) E B} an empirical distribution and batchwise threshold
Qa ((d(s, a) (s,a) E (B)),
TB
where Qa denotes the upper a-quantile. Binary labels are then y(8, a) = 1 (d(s, a) > TB).
Gating function training and stochastic gating. We train the gating network A. : S to estimate the likelihood of symmetry-breakien using the binary cross-entropy loss (Eq. ( on minibatches from the replay buffer DD During each RL update of the value function and policy, recompute and cache (s) the sampled minibatch and obtain a stochastic hard gate by Bernoulli sampling:
10.
p(8,a) == (8) i(x,a) (ernoull((pssaa))
We then form Qo (1 x)QE + XQN for the critic update and use Å for the actor-side alignment (Sec. 6.2). Gradients from Q/" do not flow into w (stop-gradient through X).
Target gate to reduce variance. To reduce non-stationarity and variance induced by stochastic gating, we maintain exponential moving average (EMA) of the gating probabilities for use in the target Quupdates:
- T (1 (1-Tx) Tx)
where p = (s, a) denotes the current gate probability. The hard RL gate is then sampled from the smoothed probability rather than from p.
İ(s,a) Bernoull(p)).
Warm-start. To avoid noisy labels before the dynamics predictors stabilize, warm-up period W steps during which the gate loss is disabled (i.e., = 0 and w not updated). A small prior routing is used by clamping x=1 with probability Pwarm during warm-up.
B.2 DETAILS FOR LEARNING Aç(s)
train a state-only actor gate d + [0, to conservatively aggregate the action-dependent critic gate via
(x(x) max (s,
To do adopt expectile regression with high expectile level 7 1, which approximates the max while remaining stable on in-distribution actions. Concretely, for each state we draw candidate actions MM (from current policies; sampling details below) and minimize
M
(x( EAND
1
Σ
L.
L,(u) = 1 < 0f//u² 2
M
This objective encourages state-level gate.
(s) to match the upper tail of (s o;) M. leading to conservative
Bernoulli actor gating (training & inference). At both training and inference, binary actor gating sampled from the probability >c(s);
{s(s) (ernoull((A{(s))
For RL updates cache X₂ per minibatch and apply stop-gradient through the sample.
18

---

Published as a conference paper at ICLR 2026
Candidate action sampling for expectiles. We form the candidate set {a per state by draw- ing from a mixture of current policies (e.g., TE. nN). This increases the chance including symmetry-breakee actions. We use the same M across tasks (see the hyperparameters Table
To avoid noisy supervision before dy stabilizes, we apply short warm-up period W steps where (x(C) is disabled; We use a small prior bias by clamping x==1 with probability Pwarn during warm-up.
Warm-start.
Gradient isolation. Gradients from the RL losses do not flow into Ae; the gate is updated only via the expectile objective above.
B.3 NETWORKS
For the Grid-World experiments, we implemented all equivariant networks from MDP- Homomorphic Networks (Van der Pol et al. 2020). In continuous control tasks, we used EMLP layers (Finzi et al. 2021b). The remaining networks, including nN, PN. Aw Aç. and the critics, were implemented as standard MLPs.
B.4 IMPLEMENTATION FRAMEWORK
Our implementation builds the Residual Pathway Prior (RPP) codebase (Finzi et al. 202 which provides flexible infrastructure for combining equivariant and non-equivariant components. We extend this framework with gated Q-networks, gated policies, and disagreement- baaee À supervision, while keeping the training loops and optimization settings consistent with RPP.
C ABLATION STUDIES
We present additional experiments on alternative actor- eetor-gating schemes, hard vs. soft gating, and shared trunks for Q. policy, and predictor networks for parameter efficiency. All curves averaged over five random seeds, and shaded regions indicate standard error.
Ant
FetchReach
Average Return
6000
Average Roturn
TRAINABLE LAM
4000
-4
MAX K4
MAX KB
2000
-10
0.25 Steps
Steps
x10
x10
Figure 6: Ablation on actor-gating schemes. As described in Sec. 6.2 our main method uses trainable state gate >s(s) to approximate the conservative aggregation max, (8, a), so that byrmeiny-breaking can be detected even when it occurs only for a small subset of actions. Here compare this gate to simpler sampled-max variant that estimates max, (s a) by taking the maximum over K actions, obtained by sampling K/2 actions from each of TE and Ty. In the legend, RRAINABLE LAM denotes the (s(s), and MAX K4 and MAX K8 denote these sampled- max schemes with K E {4, 8}. All three options achieve similar performance and sample efficiency, indicating that the sampled-max gate is reasonable alternative when architectural simplicity is prioritized.
19

---

Published as a conference paper at ICLR 2026
Ant
FetchReach
Average Return
6000
Average Return
-2.5-
HARD GATE
4000
-5.0
SOFT GATE
FIXED GATE
2000
RPP
-10.0
0.5
Steps
Steps
x10
Figure 7: Hard vs. soft gating in practice. The theory permits any measurable gate X(s, a) [0. 1]. as long as symmetry-breakinn remains tight. In our implementation, however, A is trained separately from the critics, using as a soft mixing weight inside the Bellman backup can affect stability. We compare three variants: HARD GATE, our default, which samples a binary gate and routes entirely QE or QN: SOFT GATE, which uses (s. a) E [) directly as a convex convex-combination weight; FIXED GATE, which uses a constant mixture ((s.a)
regions are routed to the true MDP so that the bound in Corollary
0.5 (analogous to fixed blend of equivariant and non- equivariant Q-heads); and RPP (Finzi et al. 2021a). a baseline that combines equivariant and non- equivariant features within the linear layers. On Ant, FIXED GATE exhibits noisy learning but eventually reaches reasonably good performance, while SOFT GATE is more stable but remains below HARD GATE (and RPP). On Fetch Reach, FIXED GATE performs comparably to HARD GATE and outperforms RPP, whereas SOFT GATE fails to learn, supporting our choice of hard gating in the practical algorithm.
Ant
FetchReach
6000
Averagee Return
Average Retur
-2
SEPARATE
4000
-4-
Q, Palicy
2000
PREDICTOR
ALL
-10
1.0
0.25
0.5
Steps
x10
Steps
x10*
Figure 8: Shared separate trunks for Q, policy, and predictors. We compare four archi- tectures. SEPARATE our default, with distinct networks for the critics, policy, and one-step predictors. Q, POLICY shares an equivariant trunk between the critics and the policy, followed by cquivariant and non-equivariant heads for (QE, QN) and (T)) PREDICTOR shares an equiv- ariant trunk between the predictors PE and Px. ALL combines both sharing schemes, using shared trunks for Q/policy and for the predictors simultaneously. On Ant and Fetch Reach, sharing only between predictors is mostly benign (with mild instability Ant), whereas sharing the Q/policy trunk or combining both sharing schemes (Q, POLICY and ALL) significantly harms performance or leads to failed training. These results support our choice of fully separate networks in the main experiments.
20

---

Published as a conference paper at ICLR 2026
Ant
FetchReach
Average Retum
8000
Average Retum
-2
k= 0.5
k= 0
0.5
2000
-10
Steps
x10*
Steps
Figure 9: Ablation study on sensitivity. We evaluated the sensitivity of method the choice of к in two environments. In our setup, A is trained using state- aatteeaction pairs identified as outliers in the online disagreement distribution d(s,, a), with g controlling the z-score threshold for labeling symmetry-breaking samples. In Ant, performance remains stable and even improves к increases (N 0), suggesting that symmetry breaking relatively rare and only small upper tail d(s, a) needs to be treated as breaking. In Fetch Reach, lower thresholds (k < 0), which label a larger fraction of pairs as symmetry--reaking, yield better and more stable performance, while higher thresholds lead to late-stage degradation. Overall, the method is not overly sensitive to the exact value re; a coarse estimate of whether symmetry violations are sparse or common is sufficient to choose robust threshold.
21

---

Published as a conference paper at ICLR 2026
D EXPERIMENTAL DETAILS
In this section, we summarize baseline models, the group symmetries, environment details, and the hyperparameters used in each environment. The implemented group symmetries used in each environment are summarized in Table | and the corresponding group representations for each state and action space summarized in Table for DQN used in Grid- World, including those for PE- PE-DQN Table] 7 For SAC, we use the default hyperparameters (Haarnoja et al., 2018), which are listed in Table including those for PE-SAC. All the experiments were run on NVIDIA RTX 4090 GPUs.
6 We summarize the common hyperparameters
Table 1: Symmetries of environments used in the experiments.
Env
Implemented Symmetries
Grid-World
C1
Hopper
Z2
Ant
Z4
Swimmer
Zz
Fetch
S0(3)
UR5e
S0(3)
Baselines. We evaluate all baselines under unified experimental setup. For the frrd-World en- vironment, the Q-nttwork is built using the MDP- Hommmoommoipiiic architecture (Van der Pol et al.) 2020 For continuous-control tasks, every method employs the same EMLP backbone (Finzi al. b021bb to ensure architectural consistency.
(1) Vanilla RI. A standard non-equivariant MIP.
Exact-Equie Equivariii.... An exactly equivariant EMLP implemented following the RPP codebase
Finzi et al. 202 but without the GAN-style Adam 3 hyperparameters introduced in Miyiato et al. 2018) (we keep the default Adam betas).
(3) RPP (Finzi et al., 2021a). We use the official RPP implementation with both value and policy RPP-based, as in the authors' main script. We do not use the non-equive non-equivariant value networks in- troduced in their ablations. We enable the GAN-style Adam 3 settings for stability, mirroring their recommended configuration.
(4) Approximately Equivariant RL (Prkk et al., 2025). We incorporate the relaxation mecha- nism of Park et al. (2025) into either an MDP-Homee Homomorphic network (Van der Pol et al.) (2020) or EMLP backbone (Finzi al. (021b), rather than using the original esenn stack (Weiler & Cesa 2019). This ensures that all baselines share an identical architecture and training pipeline. In
practice, the DQN configuration closely follows the original design, while the SAC version applies at 1 1-to the outputs of each equivariant linear layer. This constitutes a mechanism-lewe adaptation rather than full reimplementation of their approach.
learned per-channel
Grid-World. The symmetry used in Grid-World the Cyclic group Ca. summarized in Table It consists of a 15 x 15 grid, with observations given by the concatenated agent and goal positions [Fagent- Vagent Foool gg The action space is {t, +, }). Group representations are implemented as two concatenated 2D rotation matrices the state space and a 4 x 4 permutation matrix on the action space. Rewards are defined as +1 for reaching the goal and -0.01 per step otherwise.
In additional Grid- World experiments, we consider two variants of this base environment. For the reward-level symmetry-breaking setting, we convert half the (randomly selected) obstacles into passable cells that incur reward penalty of -0.5 instead of blocking the agent. For the complex- dynamics variant, inspired by Gym's FrozenLake environment (Brockman et al. 2016), we intro- duce stochasticity in the action execution: when an action is issued, it is applied to the intended direction with probability 0.65, and with probability 0.35, the agent is moved to different adjacent cell.
22

---

Published as a conference paper at ICLR 2026
Locomotion. The symmetries used in each environment summarized in Table The state and action space representations, as well as the hyperparameters, are adopted from RPP (Finzi et al, 2021a). For Swimmer-v2, Finzi et al. (2021a) reports using the approximate symmetry 22 Zo (left-right, front-back symmetries), but the official code does not provide a correct implementation of this. Therefore, instead the exact symmetry Z2 (left-right symmetries) in our Swimmer experiments.
Manipulation. The symmetries used in each environment are summarized in Table [ In Fetch Reach, the agent is trained to move the end-effector to randomly sampled target position in each episode. The corresponding state and action spaces, together with their representations of the exploited symmetries, the negative Euclidean distance between the current end-effector and the goal position. UR5e Reach, the agent is trained to reach randomly sampled SE(3) target pose in each episode. The corresponding state and action spaces with the representations of the exploited symmetries are provided in Table A dense reward is given every timestep as the negative weighted sum of the Euclidean distance (translational error) and the geodesic distance (rotational error) between the current end-effector and the goal poses. A weight of 11999621461 is applied the geodesic distance term so that 15" rotational error is treated as equivalent to 0.05m translational error. We scale action of translation by 0.05 for both tasks, and rotation by 0.2618 rad (159) for UR5e Reach task.
provided in Table 5 A dense reward is given every timestep as
Overall. The state and action representations used for the equivariant networks in each environ- ment except Grid-World are shown Table 2 3 5 6 (last column). In these tables, V denotes n-dimensional base representation, transformed by permutations for Zn and by rotation matrices for SO(3). denotes a 1-dimensional scalar representation which invariant under these group actions. P' denotes I-dimensional pseudoscalar representation, which transformed by the sign of the permutation. (e.g., for Swimmer-v2, P flips sign under left-right reflection of the body.) Note that powered representations such as //a indicate the direct sum of n instances of the representation; this is given here as an example:
V
V.
Hyperparameters used for locomotion and manipulator (SAC) experiments shown in Table Those are shared across tasks, unless specified in the table.
Table 2: Hopper-v2 state and action spaces with their representations
Name
Description
Dim
Rep
Torso
z-coordinate of the torso
R
Orientation
Torso pitch angle
I
P
Thigh angle
Thigh joint angle
1
P
State
Leg angle
Leg joint angle
P
Foot angle
Foot joint angle
P
Torso velx
Linear velocity of torso (x)
P
Torso velz
Linear velocity of torso (z)
R
Torso angvel
Angular velocity of torso (y)
P
Thigh angvel
Angular velocity thigh hinge
P
PPPD
Leg angvel
Angular velocity of leg hinge Angular velocity foot hinge
Foot angvel
P
Thigh
Torque applied on thigh joint
P
Action
Leg
Torque applied on leg joint
I
P
Foot
Torque applied on foot joint
P
23

---

Published as a conference paper at ICLR 2026
Table 3: Ant-v2 state and action spaces with their representations
Name
Description
Dim Rep
Torso
z-coordinate the
R R
Torso quat
Orientation of the torso (quaternion)
4
Hip I angle
Angle between torso and front-left link
Hip 2 angle
Angle between torso and front-right link
Hip 3 angle
Angle between torso and back-left link
Hip 4 angle
Angle between torso and back-right link
Ankle angle
Angle between two front-left links
Ankle 2 angle
Angle between two front-right links Angle between two back-left links Angle between two back-right links
V
Ankle 3 angle
State
Ankle angle
Torso vel
Linear velocity of torso (x, y, 2)
IR3
Torso angvel
Angular velocity of torso (x, y, 2)
RR3
Hip angvel
Angular velocity of front-left hip joint
Hip 2 angvel
Angular velocity of front-right hip joint Angular velocity of back-left hip joint
Hip 3 angvel
Hip 4 angvel
Angular velocity of back hip joint
Ankle angvel
Angular velocity of front-left ankle joint
Ankle 2 angvel
Angular velocity of front-right ankle joint
Ankle 3 angvel
Angular velocity of back-left ankle joint
Ankle 4 angvel
Angular velocity of back back-right ankle joint
Hip 1
Torque on front-left hip joint Torque on front-right hip joint Torque back-left hip joint Torque on back-right hip joint
Hip 2
v
Hip 3
Hip 4
Action
Ankle
Torque on front-left ankle joint Torque front-right ankle joint
Ankle 2
V
Ankle 3
Torque on back-left ankle joint
Ankle 4
Torque back-right ankle joint
Table Swimmer-v2 state and action spaces with their representations
Name
Description
Dim
Rep
Orientation angle
Front tip angle
1
P
Head joint angle Tail joint angle
First rotor angle
1
P
Second rotor angle
1
P
State
y velocities
Tip velocities along x,
2
RR2
Orientation angvel
Front tip angular velocity First rotor angular velocity Second rotor angular velocity
1
P
Head joint angvel
1
P
Tail joint angvel
1
P
Head joint
Torque on first rotor Torque second rotor
1
P
Action
Tail joint
1
P
24

---

Published as a conference paper at ICLR 2026
Table 5: Fetch Reach state and action spaces with their representations
Name
Description
Dim
Rep
EE pos
End-effector position (1,
State
EE vel
End-effector velocity (Uzy Myy Uz)
V
Goal pos
Goal position (z, V. 2)
3
EE rel trans
Relative translation (Az, Ay. Az)
3
V
Action
Gripper cmd
Gripper open/close control
R
Table 6: URSe Reach state and action spaces with their representations
Name
Description
Dim
Rep
EE pos
End-effector position (x. y, 2)
3
V
EE rot6d
End-effector orientation (6D rep.)
6
V2
EE velp
End-effector linear velocity (Var uy Uz)
3
V
State
EE velr
End-effector angular velocity (wax wys WysWz)
3
V
Goal pos
Goal position (2r, y)
3
V
Goal rot6d
Goal orientation (6l) rep.)
6
V2
EE rel trans
Relative translation (Ar, Ay, Az)
3
R3
Action
EE rel
Relative rotation exxis-angle) (azz Oy. 0z)
3
R3
Table 7: Hyperparameters used in Grid-World (DQN) experiments.
Hyperparameter
Value
Optimizer
Adam Kingma & Ba 2015
Learning rate
3 x 10
Hidden size
[256, 256]
Batch size
256
Discount factor ?
0.99
Target network update rate
0.005
Replay buffer size
1 x 10
E-greedy schedule
1.0 + 0.05 (5 10* steps)
X, Pg. Px batch size
256
PE. PN learning rate
10-4
À learning rate
x 10
#A warm-start steps
x 10 (~~30 obstacles), x 10 (40 obstacles)
À prior bias
0.5
hidden size
[256, 256]
À gradient clipping
1.0
PE, Pv hidden size
[256, 256]
PE, Pv gradient clipping
1.0
PE. PN gradient steps per update Disagreement coefficient (0/ 10 / 20 obstacles)
20
1.5
Quantile coefficient (30 / 40 obstacles)
0.6 0.3
# Threshold update interval steps
200
Threshold EMA |
0.2
25

---

Published as a conference paper at ICLR 2026
Table 8: Hyperparameters used locomotion and manipulation (SAC) experiments.
Hyperparameter
Value
Optimizer
Adam Kingma & Ba 2015
Actor learning rate
3 x 10
Critic learning rate
3 x 10-4
Temperature learning rate
3 x 10
Entropy coefficient
auto-adjust (Haarnoja et al. 2018
Batch size
256
Discount factor
0.99
Target network update rate
0.005 (0.004 for RPP Swimmer-v2)
Target entropy
-0.5 x dim (action)
Hidden size
[256. 256] 0.5
Gradient clipping
Awr Ag hidden size
[128, 128]
PE, PN hidden size
2566, 256]
Aw Aç₁ PE, PN batch size
256
# PE, PN gradient steps
2
N dę learning rate
1 x 10
Au Àg gradient clipping
0.5
No; X5 prior bias
0.7685
# Threshold update interval steps
100
Threshold EMA 3
0.1
Expectile regression coefficient Texp
0.8
# Expectile action samples M
26

---

Published as a conference paper at ICLR 2026
E EQUIVARIANCE ERROR AND ITS PROPAGATION UNDER SYMMETRY-BREAKING
Goal
1.0
Rotate
Goal 90"
0.9
0.7
0.6
Fixed Obstacle (Symmetry-beeaking)
(a) V
(b) V with rotated goal
Stabilized Goal
Local Equivariance Error
0.15
Equivariance Error
Propagation
0.05
Greedy action Suboptimal action
0.00
(c) Relative V Error
(d) Trained Policy
Figure 10: Equivariance error under symmetry-breaking. We assess rotational equivariance by comparing the base optimal value function V with the value obtained after rotating the goal by 90 (red star) while keeping obstacles fixed (black cells), thereby breaking the symmetry. To intensify the effect symmetry-breaking, we additionally incorporated stochasticity into the transition kernel in this figure. (a) Baseline optimal value V* (b) V with the goal rotated by 90 while obstacles (black) remain fixed. (c) Per-state relative equivariance error (sV*(s) V (gs) [/VV((())))))))))(////)) with the goal stabilization. The sky-blue cells bordered by red line coincide with the overlap between the original obstacle and its image under g. creating large local errors. The error then propagates outward, reflected by the surrounding regions whose shading gradually darkens. This non-local propagation occurs for all E G and has broader implications for equivariant RL training. (d) Greedy actions from an equivariant DQN, Red arrows denote suboptimal moves, illustrating that the leamed policy inherits errors in symmetry- broken regions.
THE USE OF LARGE LANGUAGE MODELS
In this paper, we used LLMMs solely for text polishing and generating code snippets. Study design, theoretical results, algorithmic contributions, and all experiments/analyes were conceived and im- plemented by the authors. All code generated with LLM assistance was reviewed and verified by the authors.
27


Reviews:
[{"review_id": "VrMkAT9ZYv", "reviewer": "Reviewer_8FGx", "summary": "This paper proposes incorporating equivariance with reinforcement learning only for certain state-action pairs. The authors propose a partially group-invariant MDP that uses a gating mechanism to switch between a group-invariant or a standard MDP. They bound the optimal Q value so that errors are not propagated when the gating function routes correctly. Experiments on a gridworld domain and continuous control and robotics domains show improved performance over baselines.\n\nOverall, this paper proposes a good way to handle symmetry violations in RL with a gating mechanism, such that one does not have to resort to global notions of exact or approximate equivariance and can use local definitions.", "questions": "- The theory is assumed for strict binary gating, while the experiments seem to use a soft gating mechanism for $\\lambda_\\omega$. Did you find that soft gating works better empirically over strict gating?\n- I didn't fully understand why a separate state-only gating function was needed. Can you not just use the Q-gating function $\\lambda_\\omega$ for the policy? Is this only necessary for continuous actions? I feel like making $\\lambda_\\zeta$ learnable can potentially lead to mismatch between $\\lambda_\\omega$ and $\\lambda_\\zeta$.\n- One weakness I can see is that since the PI-MDP framework requires separate critics and policies, using separate networks can require more samples than with a single MDP. For example, if the gating is routed to $M_E$, then only the $Q_E, \\pi_E$ networks are optimized and vice versa, leading to lower sample efficiency. This is perhaps why a soft gating mechanism is preferable as well.\n- Another question is that if we assume that symmetry violations in certain state-action pairs, then there are large regions where $M_E$ and $M_N$ coincide and thus their respective $Q$ and $\\pi$ are the same/similar. Using separate networks can be somewhat parameter inefficient in this case. Have you tried sharing portions of the networks (such as a common trunk) among the critics and policies, and maybe also for the dynamics models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The authors consider an important problem in equivariance and RL, specifically that symmetry violations are often local in practice. The proposed gating mechanism seems like a good and straightforward approach.\n- The PI-MDP formulation is well-motivated and the authors also provide theoretical analysis on the error of the optimal Q function, given the optimal gating.\n- The use of disagreement to discover symmetry violations seems like a good choice.\n- Experiments are carried out on various domains and the results show that the PI-MDP framework often outperforms exactly equivariant RL.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BaA48tLInL", "reviewer": "Reviewer_tm3G", "summary": "This paper tackles the problem of symmetry-breaking in equivariant reinforcement learning, where standard equivariant methods fail when environment symmetries are not held. The authors propose a novel framework called the Partially group-Invariant MDP (PI-MDP) and practical algorithms (PE-DQN, PE-SAC) that use a learned gating function to dynamically switch between an equivariant and a standard network, applying equivariance only where symmetries are satisfied. Experiment results demonstrate that the method successfully combines the sample efficiency of equivariance with robustness to symmetry-breaking, outperforming the baselines.", "questions": "1. Was an ablation or sensitivity analysis performed on the parameters used in the proposed method?\n2. The paper uses a hard gating strategy for simplicity. Did the authors experiment with a soft, probabilistic gating mechanism?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is generally well-written. The problem setup is clear, and the main idea is easy to follow.\n2.  The core idea of using a local and learned gating mechanism to handle symmetry-breaking is well-motivated.\n2. The paper provides a theoretical analysis of the error propagation from local symmetry-breaking.", "weaknesses": "1. The method introduces significant additional complexity: training two dynamics models, with corresponding two policy/value networks and two gating functions. The paper does not conduct the ablation analysis on these components, which would strengthen the impact.\n\n2. The current approach relies on dynamic disagreement to detect symmetry-breaking. This might be less effective for environments where symmetry is broken primarily in the reward function rather than the dynamics. The paper notes this limitation but does not explore alternatives or the performance impact in such scenarios.\n\n3. In lines 262~263, the author states that \" We assume those symmetry-breaking disagreements as outliers in the online distribution of $d(s, a)$. We label outliers with $y(s, a) ∈ {0, 1}$ using an online detector.\" Such a core assumption that its outliers reliably indicate symmetry violations is potentially fragile. For example, in stochastic environments, high variance in transitions could inflate $d(s, a)$ values even in symmetric regions, while in cases where the dynamics models are also inaccurate, leading to an unreliable gating function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HddGK8t3d4", "reviewer": "Reviewer_Gw6k", "summary": "This paper introduced a novel partially equivariant reinforcement learning formulation and a novel algorithm to solve RL in imperfect equivariant MDP scenarios. The proposed partially equivariant reinforcement learning formulation, i.e., Partially group-Invariant MDP (PI-MDP), tackles symmetry-broken MDP by decomposition the MDP into an euqivariant MDP and a standard MDP. Later, authors proposed to identify symmetry breaking scenarios (state-action pairs) by dynamic-disagreements. Lastly, two algorithms, PE-DQN and PE-SAC are introduced to solve discrete and continuous PI-MDPs respectively. Experiments show the proposed method outperforms various baselines.", "questions": "Beside questions in weakness, I have following questions:\na. what is the alpha in equation9?\nb. Could authors explain the detailed setup of the grid-world task? Especially how it break symmetry.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This paper Addressed partially invariant MDP by decomposing it into group-invariant MDP and a standard MDP. Then the two MDPs are merged by a measurable gating function. Beside theory, the paper introduced a forward disagreement measurement to practically estimate the gating. Experiments demonstrated the advantage of the proposed method.", "weaknesses": "Firstly, the paper do not explain well what is a symmetry breaking MDP. Authors gave an equation for group invariant MDP, could authors give equation for symmetry breaking MDP? Furthermore, Figure 1 showed an example of symmetry breaking MDP, I assume the obstacle is observable. Nevertheless, this example to me is more like an extrinsic equivariance [1], where the right subfigure is a new data, rather than a group-transformed seen observation (since the obstacle is not transformed). An equation of symmetry breaking MDP would clarify it.\nThe forward disagreement measurement seems relied on how good the unconstrained regressor can learn the forward dynamics. If the underlying MDP is group invariant, and the unconstrained regressor learns a bad dynamic, then this regressor will learn to large disagreement with the constrained regressor. In this case, the disagreement measurement fails to measure the equivariance of the MDP. Could authors explain more on this?\n\n[1] A general theory of correct, incorrect, and extrinsic equivariance. D Wang, X Zhu, JY Park, M Jia, G Su, R Platt, R Walters - Advances in Neural Information Processing Systems, 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)