How to Lose Inherent Counterfactuality in Reinforcement Learning
Authors: Ezgi Korkmaz
Venue: ICLR 2026 Poster (2026)
Keywords: reasoning
generalization
counterfactuality
alignment
AI safety
robust
inherent skills
reinforcement learning
Primary area: reinforcement learning

Abstract: Learning in high-dimensional MDPs with complex state dynamics became possible with the progress achieved in reinforcement learning research. 
At the same time, deep neural policies have been observed to be highly unstable with respect to the minor variations in their state space, causing volatile and unpredictable behaviour. 
To alleviate these volatilities, a line of work suggested techniques to cope with this problem via explicitly regularizing the temporal difference loss to ensure local $\epsilon$-invariance in the state space.
In this paper,  we provide theoretical foundations on the impact of 
robust, i.e. adversarial,
training on reinforcement learning.
Our comprehensive theoretical and experimental analysis reveals that standard reinforcement learning inherently learns counterfactual values while recent training techniques that focus on explicitly enforcing $\epsilon$-local invariance cause policies to lose counterfactuality, and further result in learning misaligned and inconsistent values. 
In connection to this analysis, we further highlight that this line of training methods breaks the core intuition and the true biological inspiration of reinforcement learning, sacrifices essential inherent skills that enable reasoning and generalization, and introduces an intrinsic gap between how natural intelligence understands and interacts with an environment in contrast to AI agents trained via $\epsilon$-local invariance methods.
The misalignment, inaccuracy and the loss of counterfactuality revealed in our paper further demonstrate the need to rethink the approach in establishing truly reliable and generalizable reinforcement learning policies.

Published as a conference paper at ICLR 2026
How TO LOSE INHERENT COUNTERFACTUALITY IN REINFORCEMENT LEARNING
Ezgi Korkmaz
ABSTRACT
Learning in high-dimensional MDPs with complex state dynamics became pos- sible with the progress achieved in reinforcement learning research. At the same time, deep neural policies have been observed to be highly unstable with respect the minor variations in their state space, causing volatile and unpredictable behaviour. To alleviate these volatilities, a line of work suggested techniques to cope with this problem via explicitly regularizing the temporal difference loss ensure local e-invariance in the state space. In this paper, provide theoretical foundations on the impact of robust, i.e. adversarial, training on reinforcement learning. Our comprehensive theoretical and experimental analysis reveals that standard reinforcement learning inherently learns counterfactual values while re- cent training techniques that focus explicitly enforcing e-local invariance cause policies to lose counterfactuality and further result learning misaligned and inconsistent values. In connection to this analysis, further highlight that this line of training methods breaks the core intuition and the true biological inspiration of reinforcement learning, sacrifices essential inherent skills that enable reasoning and generalization, and introduces an intrinsic gap between how natural intelligence understands and interacts with an environment contrast Al agents trained via e-local invariance methods. The misalignment, inaccuracy and the loss of counterfactuality revealed in our paper further demonstrate the need to rethink the approach in establishing truly reliable and generalizable reinforcement learning policies.
INTRODUCTION
Inspired by the learning dynamics and cognitive abilities of natural intelligence (Watkins) 1989
Sutton 1984 Kehoe et al. 1987: Romo Schultz, 1990; Montague al.. 1996 Schmidhuber 1999; Schultz et al.) 1993. Pan et al.] 2005). reinforcement learning research has been the focal point of immense research progress (Mnih et al. 2015) Hasselt et al., 2016 Korkmaz, 2026). Deep reinforcement learning has become emerging field the past decade with the introduction of deep neural networks as function approximators leading to learning policies that can surpass human cognitive abilities in highly complicated tasks by solely interacting with given environment through trial and error without any supervision, consequently resulted in building Al agents that can reason and strategize (Mnih al. (2015) Kapturowski al., [2023; Krishnamurthy al. 2024 Korkmaz 2025). In parallel, advances in neuroscience revealed the precise structures and neural circuitry dedicated the computation of counterfactual state-action values in the human brain, and how these values are later compared to make decisions. In the specialized neural circuitry that underpins ceeiisoonmmainn a compelling functional divide has been identified: while the prefrontal cortex encodes the expected values of executed actions, the dorsomedial frontal cortex plays a critical role in the analysis of counterfactual decisions providing the mechanisms for learning that can reason and generalize (Wunderlich al. 2009 Lau & Glimcher 2007 Klein-Flügge al. 2016).
Beyond the initial neuroscientific inspiration, reinforcement learning further offers strong, mathemati- cally provable, asymptotic guarantees on its ability to learn policies for solving complex problems via trial and error (Sutton) 1984] Watkins & Dayan 1992). Nonetheless, a recent body work exposed
critical safety concerns of reinforcement learning (Krrkmaz) 2022 2024), and consequently, new class of algorithms has emerged that modify standard reinforcement learning algorithms to ensure
reliability and safety in deep reinforcement learning (Madry al. 2018 Huan et al. 2020 Li 2024).

---

Published as a conference paper at ICLR 2026
In this paper, investigate the core principles of reinforcement learning, and analyze the theoret- ical underpinnings of counterfactuality and alignment in connection to the neuroscientific analysis of natural intelligence, and the consequential effects of trying to ensure safety in reinforcement learning. Our analysis discovers that the line of research focused on safety fails to deliver the guaranteed outcomes, and further risks potentially significant changes to the behavior and semanties of the trained policies that disrupts the foundations of reinforcement learning and its inherent capabilities. Essentially in this paper we aim seek answers for the following questions: (i) What are the consequences of efforts to explicitly impose safety on reinforcement learning? (ii) What are the underlying reasons for preserving the core intuitive principles, neuroscientific foundations, and inherent capabilities of reinforcement learning? To be able to answer these questions we focus on the foundations of reinforcement learning and its alignment with natural intelligence, and make the following contributions:
Contributions. We first provide a theoretically well-founded rigorous analysis of the state-action value function learnt by methods explicitly enforcing e-local invariance and standard reinforcement learning in Section 3 Our analysis uncovers fundamental insights into how c-local invariance im- position alters the very fabric of an agent's learned value judgments. Our paper is the first one that demonstrates, both theoretically and empirically, that methods explicitly enforcing robustness in fact fundamentally disrupt the inherent learning processes of standard reinforcement learning, consequentially leading to the subversion and loss of essential skills. Our analysis reveals that reinforcement learning possesses an inherent ability for counterfactual reasoning and is naturally aligned with human recision-making processes, while a recent line of work focusing on enforcing reinforcement learning to be explicitly robust causes RL policies to lose the inherent counterfactual ability and results in learning policies that are inaccurate, inconsistent and misaligned. We then conduct experiments in MDPs with nigh-dimensional state spaces from the Arcade Learning En- vironment (ALE) in Section 4 and our comprehensive study verifies the theoretical analysis and demonstrates a critical trade-off. Our findings reveal that reinforcement learning naturally retains core skills that align with the value assignment of natural intelligence which allows them to reason and generalize. However, subjecting them to robust training shatters this elegant relationship and eradicates this intrinsic counterfactual ability and alignment. Our paper establishes the foundational principle of an intrinsic trade-off between counterfactuality and robustness and further uncovers the core mechanisms driving this fundamental trade-off direct result of explicit imposition of safety, which fundamentally sacrifices the core attributes allowing reasoning and generalization.
2 BACKGROUND AND PRELIMINARIES
Markov Decision Process. An MDP is represented by a tuple M (S, Po: where S is a set of continuous states, A is a discrete set of actions, P is a transition probability distribution on Ax A + IR: is a reward function, Po is the initial state distribution, and 7 is the discount factor. The objective in reinforcement learning is to learn policy क S - A(A) which maps states to probability distributions on actions in order to maximize the expected cumulative reward
1989) the goal is to learn the optimal state-action value function 2 (* a) R(s,a) + +7 > P a) maxaç Q' (s',a) a). Thus, the optimal policy is determined by choosing the action a" (s) max, @* a) in state s.
R R EEE 7'r(st, ax) where ag ~ (Tt.). ). In Q-learning (Watkins
Adversarial Crafting and Training, Concerns regarding e-invariance start with the work of Goodfel low et al.) (2015), who observed that perturbations that are imperceptible to natural intelligence can in fact change the decision of deep neural network and further suggested a fast method to produce such perturbations based on the linearization of the cost function used in training the network. Kurakin 2016) proposed the iterative version of the fast gradient sign method proposed by Goodfellow ct al. 2015 inside an e-ball
et al.
(1) in which J(x, y) represents the cost function used to train the deep neural network, x represents the input, and y represents the output labels. While several other methods have been proposed Korkmaz (2024) using nomentum-based extension of the iterative fast gradient sign method,
.N+1 x'ay
clip + assin( 2I B)))
U++1 p Ut +
V Eata J (S + f4- v2,a) a)
++1 adv
Syn. +
U141
| es J (s + + Ue, al
T0 ll2
2

---

Published as a conference paper at ICLR 2026
robust training, ie. e-invariance training, has mostly been conducted with perturbations computed by projected gradient descent, i.c. PGD. proposed by Madry et al. (2018) (i.e. Equation D.
Neuroscientifie Results and Alignment with Natural Intelligence Decisions Making, A no- table aspect of human cognitive decision-making is the assignment of counterfactual values to alternative, unchosen decisions (Wunderlich et al. 2009 Lee et al.] Phillipss et al. 2019). This mechanism serves to inform future decision-making by preserving a clear ordering of both factual and counterfactual outcomes and is key attribute of the decision-making process that enables generalization and reasoning (Hoeck et al. [015 Phillips et al.] 2019; Grabenhorst & Rolls 201T). Notably, the results in Figure D report analysis of fMRI seans of human brains during a decision-making task to identify a neural structure that compares the values of chosen and unchosen options for a particular decision. The results demonstrate that the value of each option was encoded in this structure, and that the actual decisions made were accurately predicted by these values (Klein-Fliggge et al. 2016).
Figure 1: Human decision making and value assign- ment for options (Klein) Flügge et al. 2016).
Concerns on Reliability of Deep Neural Policies. The initial investigation volatilities of deep neural policies was conducted based on testing e-invariance via the utilization of the fast gradient sign method proposed by Goodfellow et al. (2015). Robustness and reliability of reinforcement learning policies has been further analyzed, and some studies argued the existence of shared c- variant directions across MDPs and algorithms can be due an underlying linear structure learnt by policies (Korkmaz 2022). While several studies focused on improving optimization techniques for computing optimal perturbations, line of research focused on making deep neural policies resilient to these perturbations. In particular, Pinto et al. (2017) proposed to model the dynamics between the adversary and the deep neural policy as zero-sum game where the goal of the adversary is to minimize expected cumulative rewards of the deep neural policy. Gleave et al. (2020) approached this problem with an adversary model which restricted to take natural actions the MDP instead of modifying the observations with lp fp-norm bounded perturbations. The authors model this dynamic as a zero-sum Markov game and solve it via self-play. Recently, HHaan et al. (0200) proposed to sateeaddvesarii MDP, and further claimed that their proposed algorithm SA-Double Deep Q-Network (SA-DDQN) learns theoretically certified robust c-invariant policies against both natural noise and perturbations. Recent studies surprisingly revealed that robust, i.e. adversarial, training exhibits generalization issues with unpredictable behaviour and larger oscillations compared to standard reinforcement learning (Korkmaz 2023 2024). Despite these observations on generalization, currently still a large body of work is produced on explicitly optimizing variants of e-invariance training, without any foundational analysis and explanation on how precisely this class of algorithms affect the inherent abilities of standard reinforcement learning and why this line of approach might not be the way of achieving true safety and reliability.
model this interaction
3 THE CORE INTUITION OF REINFORCEMENT LEARNING: THE INHERENT COUNTERFACTUALITY
Our extensive analysis and results discover that c-invariance training methods break the core intuitive principles of reinforcement learning and erode the inherent skills of RL policies up to the level of learning random values for counterfactual actions. Our analysis reveals that a key and strong attribute of reinforcement learning that allows generalization and reasoning lost when subjected to robust, i.e. adversarial, training. Our study provides evidence that the application of e-invariance training fails to fully address the critical issues of robustness and safety in modern AI. The observed misalignment between this training methodology and desirable system behavior reveals a key dichotomy: while certified training aims for provable guarantees, appears to diverge from the inherent principles that underpin the core attributes of natural intelligence with regard to reasoning and generalization.
The theoretically motivated e-locally invariant (e-LI) training techniques achieve certified defense against perturbations inside the e-ball, (s) {8: ||s م|ا <e} <).. forming the current foundations of the robust reinforcement learning. However, we provide foundational evidence that this approach induces significant changes in the Q-function where the state-action value function no longer accu- rately represents the MDP. In particular, e-locally invariant (c-LI) training causes deep neural policies to learn misaligned, inaccurate, overestimated state-action value functions while causing standard
3

---

Published as a conference paper at ICLR 2026
reinforcement learning to lose it's inherent counterfactuality. Furthermore, we connect and highlight the neural processing of decision making of natural intelligence, core intuition of reinforcement learning and certified training (Wanderlich al. 2009: Lau & Glimcher 2007| Grabenhorst & Rolls 2011). Our results reveal that certified training disrupts the core intuition of reinforcement learning and leads to learning policies that are disjoint and orthogonal to natural intelligence decision making, i.c. the true biological inspiration of reinforcement learning (Romo & Schultz 1990) Montague et al. 1996). In the remainder of this section we will provide the theoretical foundations on: i. Why we need to preserve the core intuition of reinforcement learning, and ii. What factors precisely disrupts the essential core skills learnt by reinforcement learning. The theoretical underpinning of e-invariance training methods is derived from Danskin's 's theorem.
Theorem 3.1 (Danskin 1967)). ) Let X be a compact topological space f: R" x X + R. f(,xz) is differentiable for every x E X, (0) {or E arg max [(0,, x)} and (0 x)) is continuous on R" X Then the max function x(()) = max f(0, x) is locally Lipschitz continuous, directionally differentiable, and its directional derivatives satisfy x' h) supzes>(0) (0) h (z, 8). Further- more, if the set (0) has size one i.e. there is a unique maximizer z3 then V Don(()) (of x%))
Danskin's 's theorem provides a method to compute the gradient of a function that is defined in terms of maximization over a set. With this theoretically well-motivated start, a line of algorithms have been proposed to make models reliable. The approach of e-invariance training techniques is based on editing the standard Q-learning update. This change made the update is designed penalize Q-functions for which perturbed state & E (s) can change the identity of the highest Q-value action. Formally, the canonical definition used in the literature is
Definition 3.2 (Robust reinforcement learning). Within an c-ball De(s) = {8 : reinforcement learning policy is robust, i.e. the @ function is e-locally invariant if Vs E S, e D. arg max, ((s, a) arg max, @(8,a).
||s Ellp S cc the
Now we will prove that there is fundamental trade-off between accurate estimation of Q-values and robustness. In particular, the optimal state-action value function Q is not e-invariant, but there is e-invariant state-action value function Qo that overestimates the optimal state- action values.
Theorem 3.3 (Inherent trade-off between estimation and robustness). Let O. In the linear function approximation setring, there is an MDP such that all linear-state action value functions matching the optimal state-action values 2 are not e-invariant. Furthermore, there is linear state-action value function Qo that is e-invariant, but overestimates the optimal state-action values while maintaining the correct optimal action.
Proof. Let there be two states 81 and 82 such that 1/s1-821: = 1. Further suppose that the optimal state-action values satisfy Q (81,a1) = =/10, (*1 a2)
Q*((2, = 0.8, and (*2, 42) = 1.0. Next let Q0(8, a) be any linearly parameterized state-action value function that agrees with 2" (s, a) on the states s1 and 32. Consider the one-dimensional functions (E - {) (1((1-
s1 + ę 82, a1) and V2(E) = 2s((1- E)- $1 + { -22, a2) which are the restriction of 2o(s, a) the line segment from $1 to $2. By linearity of 2g we also have that both v and T are linear. Furthermore, since Qo agrees with Q' at $1 and 82, we know the values of both functions at two points i.e. ,
("(s),a)). (1) e' (52. V (0 (s1 a2), and (2(1)= Q (82, a2). As V and 2/2 are linear functions on IP, the values at two points are sufficient to uniquely determine the functions. particular we have
L (E) = (0.8 - c/10)ę + e/10 and 22(E)= =
Note that these two lines intersect at the point f = 24c Let = (1 (1- t1 Since the lines of V
<
and V2 intersect E. conclude that Q0 a2) 2 Qo(5, a1). However, Q0 a1) > (o (()))a)) Furthermore, ||s| - $|| = ズサく < c. Thus, 2a is not c-invariant. However, if we instead choose new parameters 8' for the state-action value function so that @s = 0.8 and (2)) = 0.7 one can easily check that Qo e-invariant for all e < 0.1. Furthermore, observe that Qo gives the correct ranking of actions in state 81, but overestimates the optimal state-action value by a factor of 8/c.
The results reported in Section 4 verify the theoretical analysis the fundamental trade-off between estimation and e-invariance in eeural--etwoor approximation of the Q-function. Now we will further theoretically analyze the effects of canonical e-invariance training techniques (Huan al. 2020)
4

---

Published as a conference paper at ICLR 2026
Definition 3.4 (Baseline Certified e-invariance Training). The regularizer that achieves certified robustness, i.c. e-invariance, inside the e-ball (s = {s: [ $ll0c I <) for 2s(s, is
R(0)= E( max RED (8) asaarg max 2o(s.a)
max
Q0(8, Qo(8, arg max (0(8, a))).
The certified training algorithm proceeds by adding R(0) to the standard temporal difference loss CH (r(s, a) + max, Qlargelss (s', d') ) (s(s, a)) + R0)..
Now we will show that changing the standard Q-update will cause losing counterfactuality Va E A where A!' == =aala * arg max; ((s, [{s.a)}. and overestimation of the state-action values. For this now let us look at the MDP M where two states parametrized by feature vectors $1, E R", with three possible actions {at) in each state where taking any of the actions in state 81 leading to a transition to state #2 and vice versa. Let ? be the discount factor, and let ō 1j > 0 be small constants with 7 > 6. The rewards for each action are as follows: r(81,41) = - %, r(81, 7, and r(s2, 03)
7 - Clearly, the optimal policy is to always take action @1 in state 81, and action @2 in state 52 as these are the only actions giving positive reward. Thus the optimal state-action values are given by: Q (*(s1 = (" (2) = 21=0(1 - >- = 1, Q" (21.02) Q" 41) = - - tt= - >>> and (*1 (g)) = (x2, 0g) =
r(s1, 0z) 6 - 7, r(32,a) 01) = 1) - 7, a2)
7 7)? = 6. Let the Q-function be linearly parametrized by / (01, 02, 0z) so that Co(s, a.) (0,, s). Finally, let D for i {1, 33 be three orthonormal vectors, and let the state feature vectors satisfy:
1. 81 = Φ1 + 803 7 and 2. 82 = 42 + 0 3- + 1741
Then it follows that the optimal O-function is parametrized by 0- = (87.0% 05) where 07 o; iee Q0- (8, a) = 2 (s, a) for all 8 and a. Thus, according to the function 26- (8, a). for 81 the best action is 01, for 82 the best action is a2, and in all states the second-best action is ag. Next we identify the optimal perturbations used in the computation of the regularizer R(0") for this setting.
Proposition 3.5. In the MDP M for any E > O.
1. For s = 81:84
2 (0g
0Ι)
arg max max sED.(s) as)a*(s
Q0- (8, Qa- 2 (e,a"(s))
2. For 8 88:
(0%
02) = arg max max BED.(s) (s)
Qa (8, - 2x (8, (s))
2
Proof. We will prove item 1, and item 2 will follow from an identical argument with roles of 0j and 07 swapped. Let s $1. Since a" (s) 1, there are two case consider for the maximum over * a" (s)), either a = 2 or a = 3. In the case that a = 2 we have
($, a" (s)) = max max Q8- (8,aa) a) - C-- 28, (+)
(02, 5) (0) 8).
(2)
(e)
This is the maximum a ball of radius around s of the linear function (0; - 07 8). Therefore the maximum is achieved by & = + S/ (02 0¹). The corresponding maximum value is
max (07, 8) (0) 8) (0 07. + + (E).
filla = - -
(3)
In the case that 3 an identical argument implies that the maximum is achieved by $ v (6z - 81). with corresponding maximum value
max (0z, 8) (0) 8) = (05 - 07 +c100 01 = 11 +ev2. (e)
(4)
Because 8 >> we conclude that the value achieved in 4 larger than that in 3 Thus the maximizer is 3 = /2 (0% 0%) desired.
In words, the optimal direction to perturb the state 81 in order to have a (s) * a (s) is toward 85 87. Similarly for the state $2, the optimal perturbation is toward 0 85 Next use this fact to show that in order to decrease the regularizer is sufficient to simply increase the magnitude 0 and 0g, and decrease the magnitude of 0z-
5

---

Published as a conference paper at ICLR 2026
800
130
France
144
Figure 2: Up: Performance drop (2( with respect to action modification 02 for the c-invariance and vanilla trained deep reinforcement learning policies. Down: Performance drop (w. with respect action modification aw Left: BankHeist. Center: RoadRunner. Right: Freeway.
Proposition 3.6. In the MDP M let À > 0, 1/v2 > e > 0, and suppose that (1-A)8 < (1+ <
Let 0 (01, 02 0z) be given by 0₃ (1 + X)0 02 (1 + 2)02 and 0z (1 2)03. Then
R()) < (0")..
The proof of Proposition 3.6 is provided in the supplementary material. Combining Proposition 3.6 and Proposition 3.5 we can prove the main result of this section on the effects e-invariance training.
Theorem 3.7 (Existence of Overestimation, Misalignment and Losing Counterfactualitt). There is an MDP with linearty parameterized sate-action values, optimal state-action value parameters 8", and a parameter vector 0 such that: [(0) < L()0), and the parameter vector 0 overestimates the optimal state-action value and re-orders the sub-optima ones.
The proof of Theorem 3.7 is provided in the supplementary material. The results reported in Section 4 verify the fundamental trade-off and the theoretical predictions of Section 3 In particular, across a diverse portfolio of statc-of-the-art e-invariance training techniques that aim to obtain safe and reliable reinforcement learning, our results demonstrate that certified c-invariance trained policies learn misaligned, inaccurate and inconsistent values while further losing counterfactuality compared to standard reinforcement learning.
4 EMPIRICAL ANALYSIS IN HICH-DIMENSIONAL MDPs
The empirical analysis is conducted in high dimensional state representation MDPs of the Arcade Learning Environment (ALE). The standard reinforcement learning policy is trained via DDQN (Wang et al., 2016) initially proposed (van Hasselt 2010) with prioritized experience replay proposed by Schaul et al. 2016). and the e-invariance reinforcement learning policies are trained via SA-MDP RL (State Adversarial MDP, see Section 2). RADIAL (Robust Adversarial Loss-RL) OOikarinen et al. 2021), and Optimal Robust Policy (ORP) (L et al. 2024 where all of these influential studies were recognized with oral and spotlight presentations at NeurIPS and ICML respectively. The standard error of the mean is included for all of the figures and tables. Performance drop P is given by P (Scorcnase Scoreacmea) /SSorrease - Scoremin), where Scorebase represent of the game without modification, Scoremin represents the minimum score available
the baseline for a given game, and Scorescmod represents the run of the game where the actions of the agent are modified for a fraction of the state observations. To measure the accuracy for the state-action value estimates formally, let a; be the Thh best action decided by the deep neural policy in given state (i.e. Q(8, a) is sorted in decreasing order, and a; is the action corresponding to 1t largest Q-value). For trained agent, the value of ((s, aa) should represent the expected cumulative rewards obtained by taking action a; in state s, and then taking the highest Q-value action (i.e. a1) in every subsequent state. Thus, natural test to perform would be: for random state s the policy should take action a, in state s, and the highest Q-value action for the rest of the states. By comparing the relative performance drop P in this test clean run where the agent always takes the highest Q-value action, one can measure the decline in rewards caused by taking action a, Further, we can provide measure of accuracy for the state-action value function by comparing the results of the test for

---

Published as a conference paper at ICLR 2026
Pyd01
Viadi
302
Figure 3: Loss of counterfactuality: P2((2) and (Q) results with respect to d2 and aw for ORP
(Optimal Robust Policy) e-invariance reinforcement learning and vanilla reinforcement learning.
each i E [1.2...[A[], and checking that the relative performance drops P; are in the correct order ie. 0= Pi P: PlAl. We take this one step further and analyze the performance drop with Q-fraction of the states in the episode uniformly at random, and making the policy execute action in each of the sampled states. We then record the relative performance drop as function of Ω yielding performance drop curve (S()). More formally, the performance curve
Definition 4.1 (Performance Drop Curve). Let M be an MDP and ( a) be a state-action value function for M. In each state label the actions d [|A] in order that Q(s, a1) 2 Q(s, a2) 2 @(s, a. The performance drop curve P(( is the expected performance drop of an agent in M which takes action a; in a randomly sampled Q-fraction of states, and executes di all other states.
Using these performance drop curves one can confirm whether P:( lies above (2) whenever i j. Yet to be precise we will quantify the relative ordering of the performance drop curves.
Definition 4.2 (r-domination). Let [0. 1] [0, 1] and G : [0, 1] [0, 1]. For any 0, say that r-dominates if J (F(2) - (()) d > T.
To compare the accuracy state-action values for vanilla versus e-invariance trained agents, we can thus perform the above test, and check the relative ordering of the curves P(( using Definition 4.2 for each agent type. In addition, we can also directly compare for each the curve P:-in (52) for the c-invariance trained agent with the curve pyyaillla (S2) of the vanilla trained agent. This is possible because (S2) measures the performance drop of the agent relative to clean run, and thus always takes values on a normalized scale from 0 to 1. Hence, an observation of P.-in (S2) r-dominating Pranilla (S2) for some r > O, would lead to the conclusion that the state-action value function of the vanilla trained agent can accurately represent the counterfactual actions while the e-invariance trained agents cannot.
4.1 LOSING INHERENT COUNTERFACTUALITYY
In Section 3 we provided thcoretical analysis on how e-invariance training affects the core principles of reinforcement learning. In this section, we demonstrate that reinforcement learning is inherently counterfactual and certified c-invariance training causes the policy to lose counterfactuallit. Figure 2 and Figure 3 report the performance drop (2?) and Pwr as function of the fraction of states f in which the action modification is applied for e-invariance and vanilla trained reinforcement learning policies. In particular, the action modification is for the second best action d2 decided by the state- action value function ((s, a). As the fraction of states for P2 (I2) increases, vanilla trained deep neural policies experience lower performance drops compared observe that the performance drop does not exceed 0.55 even when the action modification is applied for a large fraction of the visited states for the standard reinforcement learning policies. This gap in the performance drop between the c-invariance and vanilla trained deep neural policies indicates that the state-action value function learnt by standard reinforcement learning has a better estimate for the state-action values. We further investigate the effects of a,, arg min, @(s. a), i.e. worst possible action in given state, modification on the deep neural policy. Intriguingly, Figure 2 and 3 report that the performance drop P(.(S)) is higher in the vanilla trained deep neural policies compared to e-invariance trained ones when the action modification set to a,.. This again further verifies the theoretical predictions in Section 3 and demonstrates that standard reinforcement learning learns
e-invariance. Especially in BankHeist we
7

---

Published as a conference paper at ICLR 2026
20
8
12
Lo
Figure 4: Q values of arg ma ((8, a) for e-invariance and vanilla trained deep neural policies.
gmour
MDPs
BankHeist
RoadRuaner
Freeway
Method
e-lavariacce
Vanilla
c-Invariance
Vanilla
e-Invariance
Vanilla
AM
0.44920.007
0.191=0.04
0,41420.015
0.247-0.009
0.351 10.009
0.302=0.007
AM ow
0.311 0.111
0.398=0.011
0.345=0.011
0.393=0.002
0.201
0.311=0.010
Table 1: Area under the curve performance drop under action modifi- cation (AM) a2 and a, for the state-of of the-art e- e-invariance trained deep neural policies and vanilla trained deep neural policies.
Figure 5: P2 and P... of certified c-invariance train- ing.
counterfactual and accurate state-action value functions while e-invariance policies lose core inherent skills reinforcement learning.
Reinforcement learning has inherent counterfactual ability and intrinsically learns aligned values.
The progression of AI, from foundational work in perception advanced decision-making, has been marked by key milestones driven by concepts drawn from biological inspiration (Treisman &
Gelade 1980; Snowden et al. 1991; Rao & Ballard 1999] Parthasarathy et al.) 2024; Newell
1992] Imaizumi et al. [2022] Hassabis et al.) (017)) Reinforcement learning is founded on the
1987 Romo & Schultz 1990: Montague et al. 1996) providing further theoretical guarantees on its limitations
inspiration drawn from natural intelligence (ichi Amari & Arbib 1982, Kehoe et al.
and capabilities (Watkins & Dayan 1992; Barto al. 1995). Our results show that c-invariance
training compromises the foundational intuition of reinforcement learning, leading inherent counterfactuality and creating significant value misalignment. Our analysis and results demonstrate that extensive recent line of work myopically focusing safety in fact diverts the main contributions and the tight core connection of reinforcement learning with neuroscience while producing policies that are both in fact unsafe and misaligned. In particular, Figure 5 demonstrates that choosing the worst action leads to a smaller performance drop than choosing the second best action i.e. Pu (12 < P2( for all Ω. Notably, these results reveal that e-invariance training methods assign random values to the counterfactual actions which is a direct misalignment with natural intelligence decision making. The results reported in Figure demonstrate the clear juxtaposition between standard reinforeement learning and reliability-concenned concerned reinforcement learning, i.e. invariance. Intriguingly, these findings reveal that standard reinforcement learning successfully learns aligned values and possesses an inherent capacity for counterfactual reasoning. Imposing reinforcement learning to be e-invariant strips out these intrinsic skills. While learning inconsistent and misaligned values can cause vulnerability problems from security point of view, our analysis further highlights the foundational loss information in the state-action value function as a novel fundamental trade-off intrinsic to e-invariance training.
loss of the
Imposing e-invariance causes misalignment and the loss of the inherent counterfactuality of RL.
Biased Q-values in e-invariance Trained Deep Reinforcement Learning Policies. In this section we investigate state-action value estimates of e-invariance trained and vanilla trained deep reinforce- ment learning policies. The results demonstrate that e-invariance training leads to overestimation in Q-values which verifies the theoretical analysis provided in Section 3 In particular, Tigure@report the overestimation bias the state-action values learned by the e-invariance trained deep neural policies. Note that the fact that e-invariance trained policies assign higher state-action values than the vanilla trained deep reinforcement learning policies while performing similarly, i.e. obtaining similar expected cumulative rewards, reveals that the e-invariance training techniques, on top of the misalignment, counterfactuality and the inaccuracy issues, learn explicitly biased state-action values.
8

---

Published as a conference paper at ICLR 2026
125
0.75
033
U4
023
-021
www
032
3.4
113
0281
04
553
1908
1930
2063
1900
22.3
122.
322
Banthea:
Dousharrer
242
2.24
218-
322
114
66
Figure 6:
Normalized state-action values for the best action a", second best action d2 and worst action Ow over states. Up: Vanilla trained. Down: State-of-the-art e-local invariance trained
Action Gap Phenomenon. The action gap is defined as the difference Q- valuus,, i.e. g(0,s) A @(s,a) c.A. Q(s, a). A connection between the action gap and the approximation errors has been discussed in prior studies and it has been hypothesized that increasing the action gap of the learned value function causes a decrease in overestimation of Q-values. Following this study, several papers built on the hypothesis that increasing the action gap causes reduction in bias. However, our results reveal that targeting increase the action gap must be upper-bounded by preserving the order of the counterfactual actions to obtain truly reliable and safe policies. Once this upperbound is passed the policy forms values that are misaligned and without the ability think counterfactual. To preserve the core principles of reinforcement learning and its neuroscientific foundations that allow them reason and generalize, we must preserve the approaches that targeted learning methods that align and matc the foundational inspiration of reinforcement learning (Baird & Moore 1993
Watkins & Dayan 1992 Averbeck & Costa 2017).
A Transparent Discussion and Call for Reconsideration. While these certified training algorithms have attracted significant level of attention from the research community, including several spotlight and oral presentations at NeurIPS and ICML., to encourage more efforts on this line of research com- mitting to development of responsible policies, it is more significant than ever to discuss principled investigation of these approaches. If these issues are not openly and transparently discussed, it will harm the progress towards achieving true reliability and safety while influencing future research directions and significantly pivoting research efforts. Without the principled knowledge of the actual costs and drawbacks of these algorithms a significant level of research effort might be misdirected. The results reported in Section 4.1 and Figure reveal concrete problems of the e-invariance training techniques and how they erode reinforcement learning core skills including inherent counterfactuality and generalization. Our results reveal fundamental trade-off between safety and alignment, and call for an urgent reconsideration of safety-concerned training of reinforcement learning and what constitutes true robustness.
CONCLUSION
In this paper, we focus on the core principles of reinforcement learning and how the inherent capabilities of RL policies are impacted by the efforts on explicit imposition of safety and robustness. We provide an extensive theoretical analysis on the fundamental effects of e-invariance training of reinforcement learning. Both our theoretical analysis and empirical analysis conducted in high- dimensional state representation MDPs reveal that standard reinforcement learning is inherently counterfactual and aligned with the human decision making process, while techniques focused on imposing c-invariance erodes core skills of reinforcement learning. Moreover, our theoretical analysis reveals that there is a fundamental trade-off, and our empirical results demonstrate invariance training breaks the core principles of reinforcement learning and causes policies to lose counterfactuality and learn misaligned and inaccurate state-action value functions. Our paper highlights transparent progress and calls for reconsideration of robustness and safety. Our analysis is critical in understanding the true capabilities of standard reinforcement learning, and opens avenue for more principled approaches for designing algorithms to improve reliability without sacrificing the inherent essential skills of reinforcement learning that allow reasoning and generalization.
9

---

Published as a conference paper at ICLR 2026
REFERENCES
Bruno B Averbeck and Vincent D Costa. Motivational neural circuits underlying reinforcement learning. Nature Neuroscience, 2017.
Leemon Baird and Andrew Moore. Reinforcement learning through gradient descent. In Conference Neural Information Processing Systems, NeurIPS, 1993.
Andrew G. Barto, Steven Bradtke, and Satinder P. Singh. Learning to act using real-time dynamic programming. In Artificial Intelligence, 1995.
John M Danskin. A method for solving a convex programming problem with convergence rate o (1/k2). New York: Springer, 1967.
Adam Gleave, Michael Dennis, Cody Wild, Kant Neel, Sergey Levine, and Stuart Russell. Adver- sarial policies: Attacking deep reinforcement learning. International Conference on Learning Representations ICLR, 2020.
Ian Goodfellow, Jonathan Shelens, and Christian Szegedy, Explaining and harnessing adversarial examples. International Conference on Learning Representations, 2015.
Fabian Grabenhorst and Edmund Rolls. Value, pleasure and choice in the ventral prefrontal cortex.
Trends in Cognitive Sciences, 2011.
Demis Hassabis, Dharshan Kumaran, Christopher Summerfield, and Matthew Botvinick. Neuroscience-inspired artificial intelligence. Neuron, 2017.
Hado van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In Thirtieth AAAI conference on artificial intelligence, 2016.
Nicole Van Hoeck, Patrick D. Watson, and Aron K. Barbey. Cognitive neuroscience of human counterfactual reasoning. Frontiers in Human Neuroscience 2015.
Zhang Huan, Chen Hongge, Xiao Chaowei, Bo Li, Mingyan Boning, Duane Liu, and ChoJui Hsiesh. Robust deep reinforcement learning against adversarial perturbations on state observations. NeurIPS Spotlight Presentation, 2020.
Shun ichi Amari and Michael A. Arbib. Competition and cooperation neural nets. Systems neuroscience, 1982.
Yuri Imaizumi, Agnieszka Tymula, Yasuhiro Tsubo, Masayuki Matsumoto, and Hiroshi Yamada. neuronal prospect theory model in the brain reward circuitry. Nature, 2022.
Steven Kapturowski, Victor Campos, Ray Jiang, Nemanja Rakicevic, Hado Hasselt, Charles Blundell, and Adrià Puigdomênech Badia. Human-level atari 200x faster. In The Eleventh International Conference Learning Representations, ICLR 2023. OpenReview.net, 2023.
E. James Kehoe, Bernard G. Schreurs, and Peita Graham. Temporal primacy overrides prior training in serial compound conditioning of the rabbit's 's nictitating membrane response. Animal Learning and Behavior, 1987.
Miriam Klein-Flügge, Steven W. Kennerley, Karl Friston, and Sven Bestmann. Neural signatures of value comparison in human cingulate cortex during decisions requiring an effort-reward trade-off. In The Journal of Neuroscience, 2016.
Ezgi Korkmaz. Deep reinforcement learning policies learn shared adversarial features across mdps. AAAI Conference on Artificial Intelligence, 2022.
Ezgi Korkmaz. Adversarial robust deep reinforcement learning requires redefining robustness. AAAI Conference on Artificial Intelligence, 2023.
Figure rreports that e- e-invariance training increases the action gap, still learns biased values. Due to the fact that the e-invariance trained policy has hiased Q-values, the results are reported in the normalized form in order compare the action gaps of e-invariance and vanilla trained policies in the same graph.
10

---

Published as a conference paper at ICLR 2026
Ezgi Korkmaz. Understanding and Diagnosing Deep Reinforcement Learning Decision Making, In International Conference on Machine Learning. ICML 2024, 2024.
Ezgi Korkmaz. Counteractive RL: Rethinking core principles for efficient and scalable deep rein- forcement learning. Conference on Neural Information Processing Systems, NeurIPS 2025, 2025.
Ezgi Korkmaz. Principled analysis of deep reinforcement learning design and evaluation paradigms. In The 40th Annual AAAI Conference on Artificial Intelligence, AAAI 2026, 2026.
Akshay Krishnamurthy, Keegan Harris, Dylan J. Foster, Cyril Zhang, and Aleksandrs Slivkins. Can large language models explore in-context? Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 2016.
Brian Lau and Paul W. Glimcher. Action and outcome encoding in the primate caudate nucleus. In Journal of Neuroscience, volume 27, pp. 14502-14514, 2007.
Daeyeol Lee, Hyojung Seo, and Min Whan Jung. Neural basis of reinforcement learning and decision making. The Annual Review of Neuroscience, 2012.
Haoran Li. Towards optimal ar q-learning with bellman infinity-error. International Conference on Machine Learning (ICML) [Oral Presentation]. 2024.
Haoran Li, Zicheng Zhang, Wang Luo, Congying Han, Yudong Hu, Tiande Guo, and Shichen Liao. Towards optimal adversarial robust q-learning with bellman infinity-crror. In Forty-first International Conference on Machine Learning, ICML 2024, [Oral Presentation], 2024.
Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver; BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, are G Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518: 529-533, 2015.
Read Montague, Peter Dayan, and Terrence Sejnowski. A framework for mesencephalic dopamine systems based on predictive hebbian learning. Journal Neuroscience, 1996.
Allen Newell. Unified theories of cognition. Harvard University Press, 1992.
Tuomas P. Oikarinen, Wang Zhang, Alexandre Megretski, Luca Daniel, and Tsui- Wei Weng. Robust deep reinforcement learning through adversarial loss. In Advances in Neural Information Process- ing Systems 34: Annual Conference Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 26156-26167, 2021.
Wei-Xing Pan, Robert Schmidt, Jeffery R. Wickens, and Brian I. Hyland. Dopamine cells respond to predicted events during classical conditioning: Evidence for cligibility traces in the reward- leward-laanning network. Journal of Neuroscience. 2005.
Nikhil Parthasarathy, Olivier Hénaff, and Eero Simoncelli. Layerwise caaplexity-matchee learning yields an improved model of cortical area V2. Transactions in Machine Learning Research, 2024.
Jonathan Phillips, Adam Morris, and Fiery Cushman. How we know what not to think. Trends in Cognitive Sciences 2019.
11

---

Published as a conference paper at ICLR 2026
Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforce-
ment learning. International Conference on Learning Representations ICLR, 2017.
Rajesh Rao and Dana Ballard. Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects. Nature Neuroscience, 1999.
Ranulfo Romo and Wolfram Schultz. Dopamine neurons of the monkey midbrain: contingencies of responses to active touch during self-initiated arm movements. Journal of Neurophysiology, 1990.
Tom Schaul, John Quan, loannis Antonogloua, and David Silver. Prioritized experience replay. International Conference on Learning Representations (ICLR), 2016.
Juergen Schmidhuber. Artificial curiosity based on discovering novel algorithmic predictability through coevolution. In Congress Evolutionary Computation, IEEE Press, 1999.
Wolfram Schultz, Paul Apicella, and Tomas Ljungberg. Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning delayed response task. Journal of Neuroscience, 1993.
RJ Snowden, S Treue, RG Erickson, and RA Andersen. The response of area mt and vl neurons to transparent motion. Journal of Neuroscience, 1991.
Richard Sutton. Temporal credit assignment in reinforcement learning. PhD Thesis University of Massachusetts Amherst, 1984.
Anne M. Treisman and Garry Gelade. A feature-integration theory of attention. Cognitive Psychology, 1980.
Hado van Hasselt. Double q-learning. Conference on Neural Information Processing Systems (NeurIPS), 2010.
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Mare Lanctot, and Nando. De Freitas. Dueling network architectures for deep reinforcement learning. International Conference on Machine Learning ICML, pp. 1995-2003, 2016.
Chris Watkins. Learning from delayed rewards. In PhD thesis, Cambridge. King's College, 1989.
Christopher C. H. Watkins and Peter Dayan. Q-learning. 1992.
Klaus Wunderlich, Antonio Rangel, and John P. O'Dohrtty. Neural computations underlying action- based decision making in the human brain. Proceedings of the National Academy of Sciences (PNAS) 2009.
12


Reviews:
[{"review_id": "8AYMoNzmhO", "reviewer": "Reviewer_vbm4", "summary": "The authors study recently developed methods for RL training that enforce ϵ-local invariance using theoretical and empirical analysis. They show that such methods have significant problems when compared to more traditional methods.", "questions": "The abstract notes that “…this line of [ϵ-local invariance] training methods break the core intuition and the true biological inspiration of reinforcement learning...” Why is this a problem? You outline many other problems, but I don’t see why these are a particular problem.\n\nThe introduction notes that “Our analysis reveals that reinforcement learning possesses an inherent ability for counterfactual reasoning and is naturally aligned with human decision-making processes…” Concretely, what does this mean?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "**Clear organization and writing**: While there are some improvements to be made (see below), the overall organization and writing of the paper is clear.\n\n**Important topic**: This topic is clearly important. Safety, reliability, trustworthiness, and robustness have been a major theme of research in RL over the past several years, and this paper is an important re-examination of some of the consequences of current solutions.\n\n**Both theoretical and experimental evidence**: The authors provide both theoretical and more concrete and realistic empirical evidence. This should be more strongly leveraged in the paper — for example, by more clearly identifying intuitions and mechanisms (see below) — but the theory and experiments are there to do that.\n\n**Strong results**: The results, particularly in the experimental sections, are fairly strong. It’s clear that there are major differences between the training methods that enforce ϵ-local invariance and more traditional methods. Ultimately, this is what matters.", "weaknesses": "**Insufficiently explained impacts**: Many of the effects of enforcing ϵ-local invariance that are noted by the authors are sufficiently tied to impacts in both the abstract and introduction. For example, in the paper’s abstract, the authors say that such training “cause policies to lose counterfactuality”, “result in learning misaligned and inconsistent values”, “break the core intuition and the true biological inspiration of reinforcement learning”, and “introduce an intrinsic gap between how natural intelligence understands and interacts with an environment in contrast to AI agents”. These sound bad, but what are the impacts on performance? That is, under what circumstances and by how much does this training approach reduce concrete performance measures such as long-term reward across some set of environments? Note that the performance impacts are made clear in the experimental section, but that should be carried through into the abstract and introduction.\n\n**Hyperbolic language**: The authors use hyperbolic language throughout the paper. For example, in the paragraph on contributions, the authors refer to “our comprehensive study” (when “our study” would be adequate), refer to “a theoretically well-founded rigorous analysis” (when “a theoretical analysis” would be adequate), and note that “ϵ-local invariance training shatters this elegant relationship” (when “ϵ-local invariance training disrupts this relationship” would be adequate). The authors should use more neutral language throughout the paper. \n\n**Lack of clear intuition and mechanisms**: The authors state high-level ideas in the introduction and elsewhere (e.g., “Our extensive analysis and results discover that ϵ-invariance training methods break the core intuitive principles of reinforcement learning…”), and they provide low-level theorems and experiments. However, the intuitions behind those theorems and the mechanisms to that lead to the experimental results are not made clear. The paper would be greatly improved by more mid-level intutions and mechanisms for how ϵ-local invariance affects policies (and, thus, performance).\n\n**Overly broad conclusions**: The paper includes statements such as “Reinforcement learning has inherent counterfactual ability…” However, the paper clearly cannot provide evidence for *all* RL methods, so statements such as this should be limited to some set of methods. Additionally, the paper does not define “inherent counterfacual ability.” The conclusions should be stated more narrowly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hHEVVYomZS", "reviewer": "Reviewer_t3Jn", "summary": "This paper shows that enforcing ϵ-local invariance—a common technique to stabilize deep reinforcement learning policies—undermines the natural counterfactuality of value estimation, leading to inconsistent and misaligned policies. The authors argue this approach contradicts the biological principles underlying reinforcement learning, creating a gap between artificial and natural intelligence. Their findings urge a rethinking of regularization methods to preserve counterfactual reasoning while ensuring robustness.", "questions": "- What is an action modification exactly?\n- What is the definition of $\\mathcal{P}_{w}$? What is $w$ in it?\n- I don't understand Figure 4. What is the x-axis? What does the color represent exactly?\n- What is the definition of counterfactuality under RL settings? How do you measure it exactly?", "limitations": "", "rating": 4, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "I'm not an expert in this area and don't understand much about this work.", "weaknesses": "See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Se34plvRL7", "reviewer": "Reviewer_WJm3", "summary": "The paper claims that $\\varepsilon$-locally invariant ($\\varepsilon$-LI) training—defined as keeping the optimal action unchanged for any perturbed state $\\bar{s}$ inside an $\\varepsilon$-ball—breaks RL’s inherent counterfactuality and yields misaligned or over-smoothed $Q$-functions.  \nHowever, its reading of prior “observation-robust” RL (notably Huan et al., NeurIPS 2020) appears stronger than what that work actually proves.  \nHuan formalizes a state-adversarial MDP with a perturbation set $B(s)$, analyzes value functions under an optimal adversary, and introduces regularizers or certificates that bound changes, not strict invariance of $\\arg\\max_a Q(s,a)$ for all $\\bar{s}\\in B(s)$.  \nBy building its theory on strict $\\varepsilon$-invariance (argmax stability over the entire $\\varepsilon$-ball), the paper’s counterfactuality-loss claims do not directly follow from Huan’s bounded-difference / certification framework.", "questions": "- Definition consistency (core issue).\n\nThe paper’s Definition ($\\varepsilon$-LI: argmax invariance within the $\\varepsilon$-ball) and Huan et al.’s bounded-change/certification results differ in strength.\nWhich prior methods actually satisfy your definition of $\\varepsilon$-LI?\nIf none, wouldn’t it be more appropriate to describe your work as a limit-case analysis—a theoretical exploration of what would happen under strong invariance—rather than as a critique of existing algorithms?\n\n- Replacement with bounded-difference assumption.\n\nIf the invariance condition were replaced by Huan’s bounded-difference assumption—e.g.,\n$|Q(s,a)-Q(\\bar{s},a)|$ or $D_{TV}(\\pi(\\cdot|s),\\pi(\\cdot|\\bar{s}))$ being bounded—\nhow would your counterfactuality-loss theorem change?\nWould the same qualitative result hold, or would the strength of the conclusion weaken?\n\n- Applicability across robustness categories.\n\nBeyond observation-robust settings, does the proposed “loss of counterfactuality” also occur in action-robust, transition probability-robust, or reward-robust RL algorithms?\nSince the definition of robustness (what is being regularized or perturbed) varies across these classes, the effect on $Q$-ordering may also differ.\nCould you clarify this distinction theoretically or empirically?\n\n- Supplementary/Appendix clarification.\n\nSupplementary material(L229) references Appendix, which is not included in the submission.\nInstead of adding it to the supplementary, it may be clearer and more appropriate to include those contents directly as a Main Paper Appendix, so that key theoretical or experimental details are accessible without external files.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Raises a timely and interesting question: Can robustness training erase counterfactual structure?\n- Clear toy-MDP reasoning (e.g., linear function arguments around $\\varepsilon$-balls) clarifies a plausible accuracy–invariance tension.\n- Provides a useful critique that encourages more precise definitions of “robustness” targets in RL (invariance vs bounded sensitivity).", "weaknesses": "- Over-interpreting prior work as $\\varepsilon$-invariance.\n\nThe paper states that $\\varepsilon$-LI methods “form the foundations of robust RL” and implies that Huan et al. (2020) learn $\\varepsilon$-invariant policies.\nHowever, Huan formalizes $B(s)$, analyzes SA-MDP values under an optimal adversary, and proposes regularizers/certificates that bound changes —not a guarantee that $\\arg\\max_a Q(s,a)$ is identical for all $\\bar{s}\\in B(s)$.\n\n- Assumption-driven logical dependency.\n\nThe paper’s core theoretical framework (Def. 3.2—$\\varepsilon$-LI and subsequent theorems) assumes strong argmax invariance.\nHuan’s mathematics, however, deals with continuity/boundedness and partial certification (fractional guarantees).\nTherefore, the paper’s counterfactuality-loss conclusion, which depends on full invariance, cannot be directly derived from the premises.\n\n- Ambiguity in generalization scope.\n\nThe paper mainly targets the observation-robust line of research but does not rigorously distinguish whether the same “counterfactuality loss” applies to action-robust, transition-robust, or reward-robust settings.\nSince each robustness category defines perturbation over different MDP components, it is unclear whether the claimed phenomenon generalizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yALZzp11qP", "reviewer": "Reviewer_LRJj", "summary": "This paper provides empirical and theoretical evidence challenging the prevalence of $\\epsilon$-invariant adversarial retraining in robust reinforcement learning (RL), and provides a new dimension of analysis to the performance-robustness tradeoff in adversarial RL. Specifically, the paper provides proofs that policies can either be $\\epsilon$-invariant or optimally estimate state-action values, but not both. The empirical results follow this conclusion, showing that when $\\epsilon$-invariant policies are forced to choose actions based on counterfactual observations, performance degradation is worse than would otherwise be seen with nominal policies.", "questions": "**Questions**\n- Is counterfactuality well-defined?\n- Why is counterfactuality important in Q functions, beyond conceptual links to neuroscience? Policy gradient methods, such as PPO, do not use the Q/Value function after training, so it would seem that counterfactuality is not meaningfully leveraged for its generalizability.\n- Some more recent robust RL methods optimize other objectives, such as maximin value [2] and minimum regret [3], instead of value-optimal invariant actions. Can the conclusions in this paper be extended to have implications for those methods as well? \n- Theorem 3.3 proves the inherent tradeoff between accurate Q functions and robustness. If $\\epsilon$-invariance is enforced in the policy objective only, is this a potential solution to the described problem?\n- Also in Theorem 3.3, it is stated that the two functions $Q^*$ and $Q_\\theta$ 'agree' on certain states. Is the agreement an equivalence of value, or is it referring to the ordinal rank of actions?\n\nLastly, a comment: Section 4.1 describes that the performance drop of vanilla policies is larger than that of robust policies for the worst-case action. It could be argued that the robust methods are succeeding in their objective here.\n\n[2] Yongyuan Liang, Yanchao Sun, Ruijie Zheng, Furong Huang: Efficient Adversarial Training without Attacking: Worst-Case-Aware Robust Reinforcement Learning. NeurIPS 2022\n\n[3] Roman Belaire, Arunesh Sinha, Pradeep Varakantham: On Minimizing Adversarial Counterfactual Error in Adversarial Reinforcement Learning. ICLR 2025", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper provides strong theoretical and intuitive results showcasing the downsides of commonly used robust RL frameworks.\n- The work is very insightful and gives strong justifications for where the field of adversarial robust RL should be looking.", "weaknesses": "#### Preliminaries\nBy reading the paper, the term _counterfactuality_ can be implicitly understood as the accuracy of the value function for non-optimal actions. The paper would benefit from an explicit definition, as it would help understand the implications of losing counterfactuality.\n\n#### Minor formatting:\n- Figure 6 caption, \"Lipschitz\" has a \"c\".\n- In section 2, concerns on reliability, the citation for [1] has the author's first and last name switched, should be Zhang et. al. (dblp: https://dblp.org/rec/conf/nips/0001CX0LBH20.html)\n\n[1] Huan Zhang, Hongge Chen, Chaowei Xiao, Bo Li, Mingyan Liu, Duane S. Boning, Cho-Jui Hsieh: Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations. NeurIPS 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)