Robust Optimization for Mitigating Reward Hacking with Correlated Proxies
Authors: Zixuan Liu
Xiaolin Sun
Zizhan Zheng
Venue: ICLR 2026 Poster (2026)
Keywords: Reward hacking
Robust Reinforcement Learning
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Designing robust reinforcement learning (RL) agents in the presence of imperfect reward signals remains a core challenge. In practice, agents are often trained with proxy rewards that only approximate the true objective, leaving them vulnerable to reward hacking, where high proxy returns arise from unintended or exploitative behaviors. Recent work formalizes this issue using 
r-correlation between proxy and true rewards, but existing methods like occupancy-regularized policy optimization (ORPO) optimize against a fixed proxy and do not provide strong guarantees against broader classes of correlated proxies. In this work, we formulate reward hacking as a robust policy optimization problem over the space of all 
r-correlated proxy rewards. We derive a tractable max-min formulation, where the agent maximizes performance under the worst-case proxy consistent with the correlation constraint. We further show that when the reward is a linear function of known features, our approach can be adapted to incorporate this prior knowledge, yielding both improved policies and interpretable worst-case rewards. Experiments across several environments show that our algorithms consistently outperform ORPO in worst-case returns, and offer improved robustness and stability across different levels of proxy–true reward correlation. These results show that our approach provides both robustness and transparency in settings where reward design is inherently uncertain.

Published as a conference paper at ICLR 2026
ROBUST OPTIMIZATION FOR MITIGATING REWARD HACKING WITH CORRELATED PROXIES
Zixuan Liu, Xiaolin Sun & Zizhan Zheng
Department of Computer Science
Tulane University
New Orleans, LA 70118, USA
{z {zliu41, xsun12, 2eeeunnn,,zzzereeee zzheng3)
ABSTRACT
Designing robust reinforcement learning (RL) agents the presence of imper- feet reward signals remains a core challenge. In practice, agents are often trained with proxy rewards that only approximate the true objective, leaving them vul- nerable to reward hacking, where high proxy returns arise from unintended exploitative behaviors. Recent work formalizes this issuc using r-correlation be- tween proxy and true rewards, but existing methods like occupancy-ree regularized policy optimization (ORPO) optimize against a fixed proxy and do not provide strong guarantees against broader classes of correlated proxies. In this work, we formulate reward hacking as robust policy optimization problem over the space of all r-correlated proxy rewards. We derive tractable max-min formu- lation, where the agent maximizes performance under the worst-case proxy con- sistent with the correlation constraint. We further show that when the reward is a linear function of known features, our approach can be adapted to incor- porate this prior knowledge, yielding both improved policies and interpretable worst-case rewards. Experiments across several environments show that our algo- rithms consistently outperform ORPO in worst-case returns, and offer improved robustness and stability across different levels of proxy-true reward correlation. These results show that our approach provides both robustness and transparency in settings where reward design is inherently uncertain. The code is available at https: / hhu....huh.... com http:/ixxxnnniidueae rwwaadd hacking
INTRODUCTION
Real-world reinforcement learning (RL) systems often struggle with reward specification: it is no- toriously difficult to craft reward function that perfectly captures the intended goals in all sce- narios (Amodei et al.] 2016; Ibarz et al., 2018 Stray et al.] 2024). In practice, designers rely on proxy rewards that approximate the true objective (Tien et al. (023). However, agents optimizing these imperfect proxies can lead to unintended exploitative behaviors, achieving high proxy returns while yielding poor true outcomes, phenomenon known reward hacking (Leike al. 2017 Everitt et al. 2017 2021 Koch et al. (2021). Such reward hacking behaviors are not merely hy- pothetical; they have led to undesirable or even catastrophic consequences in safety-critical settings (e.g., autonomous driving) (Krakovna et al. 2018 Knox et al. 2023) and are alarmingly common in real-world deployments (Kleinberg et 2024 Franchi et al. 2023 Milli al.) 2021 Obermeyer ct al. 2019). Beyond reward hacking, interpretability and transparency of RL policies are increas- ingly recognized as critical requirements for real-world acceptance (Vouros 2022, Puiutta & Veith, 2020 Tyer et al.) 2018). Policymakers and practitioners in safety-critical domains require systems not only to be robust but also interpretable; they must understand which specific decision-making criteria lead to undesirable outcomes to effectively mitigate risks and ensure compliance with safety regulations [Rudin| 2019; Druce et al. 2021 Doshi- Velez Kim 2017). These challenges high- light the need for RL algorithms to address
fundamental challenges: robustness to uncertain or poorly-spe -ppeiiidee rewards, and interpretability to facilitate oversight and compliance by human stakeholders, especially in high-stakes, real-world environments like traffic control (Vinitsky et al.,

---

Published as a conference paper at ICLR 2026
2018), healthcare decision- making (Fox al. 2020 Holzinger et al. 2017). and pandemic response strategies (Kompella et al., 2020).
Recent work has begun to formalize reward hacking and develop principled mitigations. (Laidlaw ct al. 2 define a proxy reward to be r-correlated with the true reward if it maintains a correla- tion coefficient > 0 state-action pairs encountered by certain reference policy. Notably, their definition permits the proxy and true reward to diverge arbitrarily in parts of the state-action space not visited by the reference policy, precisely the regions an RL agent might exploit under intensive optimization. Using this framework, reward hacking is formalized as the situation in which optimiz- ing an r-correlated proxy yields a policy with lower true reward than that of the reference policy. Building on this definition, they propose Secupancy-Regularigg mitigation strategy. ORPO augments the standard RI. objective with a regularization term that pe- nalizes deviations between the learned policy's 's occupancy measure and that of the reference policy.
Policy Optimization (ORPO) as a
Despite significant progress, existing solutions to reward hacking show several limitations. First, their effectiveness relies heavily on the choice of the specific proxy reward. However, designing perfect proxies is challenging, and in real-world scenarios, reward proxies are often derived heuris- tically or empirically from noisy or limited data (Jeon et al., 2020; [Sadigh et al.) (017)) leading to uncertainty or variability in the exact correlation with true rewards. Therefore, robustness to varia- tions in proxy rewards is crucial for dependable deployment. While the regularization method used by ORPO provides a lower bound on improvement in true reward, its guarantee on the worst-case performance against adversarially chosen proxy is weak. Second, current methods like ORPO typically treat reward function as a black box and learn complex policy with no easily inter- pretable structure, making it hard understand why the resulting policy avoids reward hacking or to trust its behavior in novel situations. Further, they cannot be easily adapted to incorporate prior knowledge of the true reward. These shortcomings underscore the need for a more robust and transparent approach to reward hacking in RL.
In this work, we formalize reward hacking as a robust RL problem under proxy reward uncertainty and develop new algorithms to address the above gaps. The key idea is to optimize against an adversarial proxy reward rather than trusting a single proxy. We assume the true reward could be any function that remains r-correlated with the proxy, and we train the agent to perform well against the worst-case such proxy. This approach explicitly accounts for uncertainty in proxy design and guards against unintended exploitative behaviors. Concretely, we propose max-min formulation in which the policy chooses its strategy to maximize its guaranteed true return while an adversary minimizes the true return by selecting reward function from the set of all r-correlated proxies. By solving this problem, the agent learns a policy that is robust to all plausible deviations of the proxy reward within the correlation bound. We derive closed-form solution for the adversary's 's worst-case reward assignment given any candidate policy, which allows efficient evaluation of the inner minimization and provides insight into how proxy reward flaws are most damaging. Building on this result, we introduce a practical algorithm for Max-Min Policy Optimization that iteratively updates the policy against this worst-case reward signal. Moreover, to improve the tractability and transparency of the inner optimization, we introduce Linear Max-Min variant of our method. In this variant, we assume the true reward lies in class of linear functions over known features, an assumption that has been extensively studied in prior work on successor representations and successor features (Dayan 1993 Barreto et al., 2017 7 2018), and which allows us to characterize the worst-case proxy reward as sparse linear combination of those features. While the policy itself remains parameterized by general neural networks, the learned worst-case reward function becomes interpretable in terms of its feature weights. This provides insight into which aspects of the proxy reward space the policy is robust to or vulnerable against, making it valuable for applications where understanding the failure modes of the reward design is important.
Finally, we empirically evaluate the proposed approaches on several challenging environments. Across all domains, our Max-Min and Linear Max-Min policies outperform ORPO in terms of worst-case reward, indicating substantially improved robustness. Moreover, under a large range of proxy-true correlation scenarios, our methods exhibit higher average reward and lower variance compared to ORPO, meaning the performance of our policies remains more consistent and reliable. These findings demonstrate the practical significance of our robust formulation, paving the way for safer and more trustworthy RL deployment in real-world applications.
2

---

Published as a conference paper at ICLR 2026
Our main contributions can be summarized follows: 1) We propose novel robust RL formulation that explicitly models reward hacking as a max- optimization problem over proxy rewards con- strained by correlation with the true rewards. 2) We develop practical algorithm for the max-min problem, which is further extended to linear rewards with improved robustness and interpretabil- ity. 3) We provide theoretical convergence guarantee for the max-min objective with sample- complexity bound for the occupancy estimation. We also show that accurate occupancy estimation pivotal for robustness. 4) Experiment results demonstrate improved robustness and worst-case rewards across five real-world inspired reward hacking environments.
2 PRELIMINARIES
Reinforcement Learning. A reinforcement learning (RL) problem can be formulated an infinite- horizon Markov Decision Process (MDP) defined by the tuple (S,A, p. μo: R, 7), where S and A denote the state and action spaces, p(s' s, a) is the transition probability from state to s' given action a, Ho is the initial state distribution and E [0, 1) is the discounted factor. The agent interacts with the environment over discrete time steps = 1,
At each time step, it selects an action Oe E A based on the current state 8 E S according to a policy T(a s), which defines a distribution over actions conditioned on the state. Upon taking action ae, the agent receives a reward R(st, a.) E R and transitions to the next state 81+11 according to p($1+1 St, az). The goal of the agent is maximize the expected cumulative discounted return:
J ( (1 ()-7)BB EN
Σ y (((ss (e)
(1)
where y [0, 1) the discount factor, and the expectation is taken over trajectories generated by following policy æ. We define the state-action occupancy measure Msg of a policy as: H (*xssa)
(1 - 7) Ex IEi= LL = 5, ag = a)). which represents the discounted visitation frequency of each state-action pair under policy æ. Using the occupancy measure, the return can be equivalently expressed as: J(x, R) = - (R).
Correlated Proxies and Reward Hacking. Below give overview of the recently proposed r-correlated proxy framework proposed in (Laidlaw et al. 2025) for detecting and mitigating reward hacking, which our work is built upon. A detailed discussion of related work reward hacking and robust RL is given in Appendix D In particular, they consider a setting where the agent is given a reference policy nef and a proxy reward Ryroxy. while the true reward is hidden. They further assume that the proxy reward is r-correlated with the true reward under the reference policy, that is:
Ept ( (Reuny
Repoxy
[Trett Rpeoxy)
Ruue
[ Ruve)
]
=
(2)
Bageay
Ruse
where 07 Myynny Ea [Rprocy [Treets Rerocy)) ))² and o = Eyp. [Ruree J [Trett Rave))
the variances of the proxy and true rewards, respectively, under the reference policy. Reward hacking is said to occur when a policy π optimized for an r-correlated proxy reward achieves lower true reward than the reference policy: J(T, Rame) < [(Tret, Rume). To mitigate reward hacking. Policy Optimization (ORPO) to optimize a regularized policy
they propose Sacupancy-Regulare objective given below, which shown to provide lower bound on improvement in true reward:
max JTT, Rproxy) - X (2 Px.e).
(3)
where x (p(xx || (mt)) denotes the >ssuuareed divergence between the occupancy measures of π and Tiefi and the regularization strength X = o Rymay vI पo. This encourages the learned policy to stay close to the reference distribution when the proxy reward is weakly correlated with the true reward.
3 METHOD
In this section, discuss robust policy optimization approach for mitigating reward hacking. In contrast to regularization-bat methods such as ORPO, we consider a max-min formulation that identifies robust policy with respect to the worst-case reward across all reward functions that
3

---

Published as a conference paper at ICLR 2026
r-correlated with the proxy reward. We further extend our framework to settings where the reward function is a linear combination of known features with unknown weights. Our approach effectively leverages this structural information, when known priori, to improve both robustness and interpretability, a task that is particularly challenging for regularization-basel techniques.
3.1 MAX-MIN POLICY OPTIMIZATION
Similar to ORPO, we assume that the agent is given a proxy reward Raesy and a reference policy Tref, while the true reward is hidden. Rather than regularizing the policy under fixed proxy reward, we consider the entire space of rewards Rewr that satisfy the correlation constraint with respect to a known proxy reward, as defined in Equation 4
Recerr { R: (s, + Ep
R M.
Rpoxyy = ( R) M, o v3].
(4)
M and V denote the fixed mean and standard deviation of the reward function R. under the refer- ence policy Trel. For simplicity, we define Rpoxy to be the normalized proxy reward Rgnxy () ==
Rynay (8,0)_J(sie)) Ryyss) where Remaxy is the original (unnormalized) proxy reward. After normaliza-
"Ryuns tion, have J(Tret Reruxy) ) 0 and Var (Rpocy) = 1, which simplifies the correlation con- The hyperparameter controls the degree of alignment between the proxy and true reward. allows us to interpolate between strong robustness (small r) and high proxy fidelity (large r), enabling a principled generality to consider fixed M and V, which we will further elaborate on later.
straint in Equation [
maassteessaccuuras trade-off. We remark that it is without loss of
We propose worst-case optimization framework where the policy is trained maximize expected performance under the least favorable reward within Re Assuming that the true reward lies some- where within this set, this approach improves robustness by ensuring that the policy does not overfit to any single optimistic interpretation of the proxy reward. Formally, the objective becomes:
max min J(Z, R) = max min ((e,a))p[[[<s.. [R(s. a)].
(5)
RER
RER
However, a challenge arises: the objective (R(s, a)] depends on the state-action occupancy
Mm,, whereas the constraints defining Reur expressed in terms of PTa This mismatch com- plicates direct optimization. To resolve this, we apply a change-of-we ofea-ure technique (Hu & Hong.
2013 Lam 2016) to rewrite the expectation under Pr Specifically, let L(s, a) denote the Radon-
Nikodym derivative: L(s,a) a) = M=(s.a)
Prad (s.0)- By definition, L(s,a) 2 0 and E, [L(s, a)] =
1. Applying the change-of-measure formula, we can express the return as: E,,, [R(s, a)]
SSxA Ma (s, a) R(s, a) d(s, a) = SSxA (s, a) (x(8.0) "saaasa) R(s, a) d(s. a) = E [L(s, a) R(,())((sooooo a)]. Thus, both the objective and the constraints can be rewritten as expectations with respect to MTm
For notational simplicity, we will suppress variables (s. and write for example, L as L(s, a). Under this reparameterization, the inner minimization in Equation can be reformulated as:
min RE
Ey [L- RJ.
Although the feasible set in Problem not convex due to the equality constraint on the variance, still derive an optimal solution using a Lagrangian formulation. Our approach leverages tools from duality theory, commonly used in robust optimization (Delage & Yel 2010 Goh & Sim 2010). We further justify the validity of our solution in Appendix E.2 Specifically, the Lagrangian functional associated with this problem is defined as: lo(A, A22 A3, R2) Eyp [L-R~ >1 " Romay -22R-
\t + + A2 - >g (M2 +V2), where '₁ A2, A3 are the Lagrange multipliers corresponding to the correlation constraint, mean constraint, and variance constraint, respectively. Then the original problem in Equation 6 is equivalent to the following problem:
max min lo(A,, X2, A3, R).
As RER
We now solve the inner minimization problem in Equation by finding the optimal R for fixed dual variables (A1, X2, As). Taking the functional derivative of the Lagrangian lo with respect to R(s,a) Olp gives: BR
Mr, (8, [) > Rywss
A >2) 2Ag R?. When (s, a) > 0, setting the derivative
4

---

Published as a conference paper at ICLR 2026
of the Lagrangian to zero yields the optimal adversarial reward function:
(= a) L(s,a) A1 Remex
V A2
(8)
2X3
However, for state-action pairs where (T (5, = O, i.e. those not visited under the reference policy, the correlation and moment constraints become vacuous. In these regions, the adversarial reward R (s, a) can be driven arbitrarily poor, reflecting that no constraint prevents the adversary from assigning highly penalizing values to rarely visited or unobserved state-action pairs. Neverthe- substitute the optimal R from Equation 8 into
less, consider the case where (s, a) > O, the Lagrangian lo and get the dual objective. After some process detailed in Appendix the optimal solution to problem (6), the original max-min problem (5) reduces to:
E. we get
max r V Ey,, [Rprayy V v r2 V (1 (m E2 [Possy) M.
(9)
Thus, the final policy optimization objective becomes maximizing the proxy reward, regularized by penalty that depends on the distributional shift between Jly and p and the expectation of the current policy under proxy reward E,,, [Rposy], and the correlation strength r. We observe that the constants M and V do not affect the optimal policy: while they influence the absolute value of the worst-case reward for a given policy æ. they only apply linear transformation (scal- ing by V and shifting by M) and do not change the relative ordering of policies. Therefore, for simplicity, we set V = and M = 0 in implementation. This also provides fair way to compare the worst-case rewards of different policies. Notice that the optimization objective in Equation
3 However, there are two key differences: (1) our regularization strength is -I-r" instead of Mynny vI-vV --- and (2) our penalty term is x²( ll Pagg)) - E22 .prxyy] rather than simply x²(xx Prg)). The proof that x²(xz |I MTT)) E2 [Rpasy] 2 0 holds can be found in Appendix E.3 A detailed comparison between our policy gradient and that of ORPO is provided Appendix E.8
| closely resembles the ORPO objective proposed in Equation
To further illustrate how our framework in Equation 9 helps prevent reward hacking, i.e., how op- timizing a proxy reward can translate into an improvement in the true reward over the reference policy, as discussed in Section 2, we formalize the following theorem:
Theorem 1. Suppose that the true reward function Rycue lies in the correlation uncer- tainty set Reor.. Then, for any policy such that Fe & Pr (i.e., Un (8,a) 0 > pr(s, 0), we have
J(z Rinue) [TTr. Ryyue) 2 r Eycr [Rpoovy -1- ((2 MmTT))
BZ [R Rpmray]
Proof can be found in Appendix E.9 From Theorem I we see that our objective optimizes a pes- simistic lower bound on the true improvement over the reference policy. By Definition 4.2 in (Laid- law et al. 2025), reward hacking occurs when J(TT, Ruse) < J (Treff ). Although this is precisely cannot do directly because the true reward is unob-
the quantity would like to maximize, served, and therefore we must instead optimize the max- max-mmin objective in Equation [ Theorem shows that objective is always lower than (but anchored to) the true improvement, which plains why our framework can promote robustness against potential reward hacking: improving our surrogate objective necessarily improves a conservative bound on (T, Rire) J (Tref, Rame).
||
Remark: Our optimization problem in Equation Sjis standard in distributionally robust optimization (DRO). However, it remains underexplored in the context of RL, with only one relevant work that considers uncertainty sets based on the first and second moments of the reward distribution (Nguyen al. 2022). While their formulation appears similar, their results are not directly applicable to our max-min framework, and we still need to explicitly solve our formulation. We also note that under certain assumptions, the ORPO objective (Equation 3) can be reinterpreted as a special case of the max-min formulation in (Nguyen et al.) 2022) (Theorem 1), providing a complementary view of the connection between these approaches. Nevertheless, our optimization objective remains structurally different. Moreover, in the pessimism offline RL setting, where distribution shift is the central challenge, the x2 regularization together with maxmin formulation has also been explored (Zhan et al. 2022 Huang et al.) 2024) from a perspective different from ours. However, frameworks such as XPO (Huang et al. (022)) are not applicable in our setting because they require the regularizer to be/ f-divergence. The square-root term in Equation 9 does not satisfy this requirement.
5

---

Published as a conference paper at ICLR 2026
3.2 STRUCTURED REWARD SPACES VIA FEATURE LINEARIZATION
A natural concern with worst-case optimization is over-conservatisw: if the reward uncertainty set Roo is too broad, the resulting policy may become overly cautious deviate from realis- tic task objectives. Additionally, the learned worst-case rewards may themselves be implausible or uninterpretable. To address these issues, we introduce structure into the reward space by as- suming that all rewards are linear combinations of known features, an assumption that has been widely adopted in prior work (Dayan 1993 Barreto et al. 2017 2018). Specifically, we assume: R(s,a) a) 0 Φ(s, a), where ((s,a) a) tor M known or engineered feature functions, and 0 [01, 02................... MM] E I represents the uncertain feature weights. The linearization yields two key benefits: Realism and Inter- pretability: In many real-world tasks, reward functions are naturally approximated as linear com- binations over interpretable features. For example, in traffic control environment, features might include total commute time, vehicle speed, acceleration, and inter-vehicle headway distances. 2) ConttreCons Robustness: By restricting uncertainty to structured, feature-based rewards, the worst-case optimization problem becomes more grounded and avoids pathological, unrealistic reward functions.
(5,a), (2 ..... (M a)] E IR- denotes vec-
In this section, we assume that the agent is aware of the set of features but not their true weights. We show that our robust optimization framework can naturally extended to incorporate the structure in rewards to improve robustness. In our experiments, we further demonstrate that linear rewards help interpret policy's performance even when it is trained without such prior knowledge. Under our assumption, the uncertainty set reduces to the set of feature weights A E RRM satisfying:
Rlin
{0 {0E IR
Bura 10
Rproxy] p [TTT] = 0, E p 100 (0T6)9=1
cb]
(10)
To simplify the analysis, we assume without loss of generality that the worst-case reward R(s, a) = 0 d'( a) is normalized to have zero mean and unit variance under the reference policy Tref- This corresponds to setting M = and V = 1, which, as shown in our earlier derivation, does not affect the resulting optimal policy. As before, Rproxy denotes the normalized proxy reward under Tet. satisfying E, Ey.,, [Raoyy] = 0 and Var,-,, [Rproyy] =1
We now derive the corresponding max-min optimization under the structured reward assumption:
max
min Re 020
[0T (pssa))
(11)
Similar to previous steps, we introduce the Radon-Nikodym derivative T(s,a) = Pr (r(s.a)
Me [s.a]* use
change-of-measure, and define the Lagrangian functional for the inner minimization Equation [1 O Q0 Arr Às where ux,
as: 1, (A1, A2, As 0) _ (Esnn) MAy,An (s, a){((, a) (s)
Hn >I Rproay Aup Q -(ee,)) Pma a) (s, a)Φ(s, TTT. A detailed derivation can be found in Appendix F.4 Then solving the inner minimization over in Equation 11 equivalent to:
max min A2, 020
12,1,
A2, = T [Σ Eurinse)) 030 Q0 + Air X3
(12)
Notice that 2 (A1. X2, As, 0) is a convex quadratic function of O (assuming Ag S 0) subject to linear inequality constraints 0 > 0. Thus, is a standard convex quadratic program (QP) with non-negativity constraints (Boyd & Vandenberghe, 2004). However, it is not possible to derive universal closed-form solution for the optimal 0 under arbitrary Q. To further simplify the problem and obtain a dlosed-form solution, we transform the feature vector & into a whitened version ø such that the matrix Q becomes the identity matrix I and we formally show this in Appendix [E.5] Specifically, we perform a whitening transformation using the Cholesky decomposition (Boyd & Vandenberghe, 2004). Let W Q Ö(s, a) = Wo((s, (s,,,, a), where Q- denotes a matrix square root of Q= (which exists since Q is positive semi-definite and non-singular, which detailed in Appendix E.5 b. Then the original problem in Equation 12 can be further simplified into:
max min 4A1, A2, As. 0) -
Σ (ur.Asss (s,a)
- Air + Ag. (13)
Ag 020

---

Published as a conference paper at ICLR 2026
where we now optimize over the parameter & using the transformed features Øb. For notational sim- plicity, we will drop the tilde and henceforth use & to represent the whitened feature ø. and 0 to represent 0. Then we can get closed-form solution (we detail the steps in Appendix E.6) for opti- mal 0 as: 0- = max 0. _i.w) KA (8.2)))()))
where the max(-, 0) is applied elementwise.
2As
Details for solving the outer maximization in Equation 13 can be found in Appendix [E.7 After obtaining the optimal dual variables (Λī, Ag Az), we can substitute them back into the optimal 0 to construct the worst-case reward, which is the optimal solution of the inner problem of Equation 11 given Then we can solve the outer maximization over the policy и using standard RL algorithms.
ORPO with Linear Rewards. While ORPO provides a general guarantee based on occupancy measure regularization, it does not exploit any structural assumptions about the reward function. In particular, even when the true reward is linear in a set of features, ORPO does not explicitly incorporate this structure into its policy optimization or theoretical analysis. While the lower bound (Theorem 5.1 in (Laidlaw et al. (025 continues to hold, it is unclear how to leverage this structure to obtain tighter lower bound or to guide policy updates more effectively. This suggests a missed opportunity: by explicitly modeling the reward as linear function, it becomes possible to derive stronger guarantees, interpret worst-case reward directions, and efficiently optimize against them. Our Linear Maxmin method fills this gap by parameterizing reward uncertainty directly in the space of reward weights, enabling both robustness and greater transparency.
3.3 OCCUPANCY ESTIMATION AND CONVERGENCE
A core step in both our algorithms and ORPO is to estimate the Radon-Nikodym derivative L(s, a). To this end, following prior works (Laidlaw et al. 2025 Kang et al. 2018 Ho & Ermon (016)), we
fit a discriminator network (s, a) with L (s. a) exp dg. a). We learn Φ by minimizing:
= arg min E Eps, [log(1 + e do(e,)))) )] + Ey,, [log(1 + e --do(.)))))
(14)
is known that the optimal discriminator satisfies d' (s, a) = log (a((,,) and we estimate Lo a) (e.a)
as Lø(s, a) exp ds a) with ds(s,a) x (s) As discussed in Section 3.1 if the policy visits state-action pairs that the reference policy Tiet rarely or never visits, the adversarial reward can be arbitrarily poor. In theory, the estimated L(s, a) is expected to grow arbitrarily large in this case, which should discourage the learned policy from exploiting such regions. However, empirically (Section[42) that the ORPO policy still visits some of these low-coverage regions under Tet. This is because in the original ORPO implementation, the discriminator is not fully optimized during policy learning. Specifically, the discriminator receives only a small number of gradient updates per RL iteration, resulting in underfitting and inaccurate estimates of the Radon-Nikodym derivative i(ssa). To address this, we substantially increase the number of gradient updates per iteration and carefully tune the learning rate. Our goal is to strike a practical balance between training time and discriminator quality, which we discuss in Appendix F.I We further show that the following theorem, which establishes that the discriminator estimation achieves a sample complexity of On -1/),, where n denotes the sample size.
observe
Theorem 2 (Occupancy ratio Ls error bound). Under assumptions, let L == ed be the empirical estimation and L = ed be the true ratio. Then, with probability at least 1 - 8,
log(M/ 1/2 1/2
Ep [I--L-)] S c c(''+
where C, of and M are some constants.
The full argument is presented in Appendix G.I where we adopt the optimistic cover notion (Def- inition 3) from (Huang et al., 2023) technical tool and establish the new concentration analysis as well as the resulting complexity bounds specific for estimating the loss in Equation 14
To compute the final objective for our Max-Min policy in Equation 9 we estimate the x² divergence, the normalized proxy reward Raoxyy- and the first and second moments E,- [Rpmoy) and E? [Rpooyy] These components together define the robust optimization objective used to update the policy. A
7

---

Published as a conference paper at ICLR 2026
simplified Max-Min policy optimization procedure is outlined in Algorithm We provide detailed descriptions of each estimation step. as well as the complete algorithmie implementation for both Max-Min and Linear Max-Min in Appendices F2 and FS We further obtain convergence bound of O1/TT + 1/N + n -1144) for our Max-Min algorithm, by viewing (9) maximizing general
utility considered in (Zhang et al. 2022 Barakat et al. 2024). Here 7 is the number of iterations and N is the batch size for policy update. Detailed proofs and the convergence analysis for Linear Max-Min are in Appendix [
Algorithm 1 Max-Min Policy Optimization (Simplified)
1: Initialize policy parameters 0
2: Initialize reference policy Tel and collect trajectories
3: Estimate mean and variance of the proxy reward under Tref
4: for each iteration do
5: Collect trajectories from current policy me
6: Normalize the proxy rewards for state-action pairs in the collected trajectories
7: Estimate the expected proxy reward and its second moment under the current policy
8: Estimate the discriminator using Equation [4]and x² divergence between μ*x and Mx
9: Update the policy using PPO maximize the Max-Min objective in Equation 9
10: end for
4
EXPERIMENT
4.1 EXPERIMENT SETUP
We evaluate method across five realistic benchmark environments: Traffic, Pandemic, Glucose Monitoring, Tomato Watering GridWorld and RLHF. These environments were originally proposed in (Pan al. 2022 Leike et al., 2017) and represent diverse forms of proxy reward hacking, includ- ing misweighting, ontological mismatch, and scope misalignment (Pan et al. (022). A detailed de- scription of the environments and their respective reward structures is provided in Appendix [F44] In each of the five environments, we train policies using both our Max-Min and Linear Max Minn algorithms for random seeds. For baselines, we compare against the ORPO policy. To isolate the impact of discriminator training, we also include an ablation: ORPO,, where we train the ORPO policy using the same full discriminator training schedule as in our algorithms. This variant shares the same architecture and optimization settings as the original ORPO, differing only in the extent of discriminator training. Including this baseline allows us to evaluate the specific contribution of dis- criminator optimization to policy robustness. For the RLHF environment, we additionally include the Ensemble baseline (Eisenstein et al. 2023 a reward-centric approach designed to mitigate reward hacking in RLHF. We include more detailed experimental settings in Appendix F. and discussion of training time and complexity of all algorithms Appendix[F.6
Since the correlation may only be approximately estimated and there is currently principled method for selecting its optimal value. We adopted similar approach used by ORPO (Laidlaw ett al. 2025). For each environment, we first performed grid search of r, and for each fixed r, we trained the policy using our algorithm. We then selected the r value that leads to the policy with the best expected worst-case return (detailed in Appendix (.22). which is 0.3 for Traffic, 0.7 for Pandemic, 0.9 for Glucose, 0.4 for Tomato, and 0.4 for RLHI. Results on all searched can be found Appendix H.5 Notice that ORPO selects the optimal that yields the best expected return under the true reward, which is infeasible in practice when the true reward is unknown during training. On the other hand, when the exact correlation is unknown, our approach also raises a concern about how to interpret which worst-case reward is actually meaningful. We include detailed discussion about how to choose r in practice Appendix[]
several different values
As for evaluation metrics, we report both the expected proxy and true rewards, along with the ex- pected worst-case reward as described Section 3.1 Note that some policies may visit state-action pairs that are not covered by the reference policy Meff- In such cases, exclude those trajectories and report the occupancy measure of the unseen state-action pairs. Additionally, we evaluate each policy using two variants of the expected linear worst-case reward introduced in Section 3.2 The first uses only the features present in the proxy reward, while the second variant, denoted Linear Worst* leverages features from the true reward, some of which remain unseen during training. This
8

---

Published as a conference paper at ICLR 2026
Table I: Evaluation results on Traffic, Pandemic, Glucose, and RLHF environments. All policies are trained using only the proxy reward. In Traffic, the proxy reward is based on vel, accel, headway (1. 1, 0.1). while the true reward uses commute, accel, headway (1. 1. 0.1). In Pandemic, the proxy reward includes infection, lower stage, smooth changes (10, 0.1, 0.01), while the true reward additionally includes political with weight 10 after infection. In Glucose, the proxy uses expected patient cost, and the true reward uses magni bg. In RL.H, the proxy uses a 70M LLM,, and the true reward uses 8B L.M. We report in the same order as feature weights. Occ denotes total occupancy over state-action pairs unseen by Twf, where discriminator outputs infinity.
Eay
Tralicc
Method
True
Froxy
Worst
Linear Worst (0)
Laneur Worst* (0)
Ore
CRFF
169120112
34130.13
0.09 (0.71, 0.21, 0.69)
-0.3320U2 (0.43, 1.12, 0.97)
3.82e-04 20 13
DBRC+
10.2600.99
135+0.09
-1 350e000-00000k
-0.44-0.22 (0 46, 0,18, 0.85)
-0.4++0011 (0.58, 0.06 0.51)
1844-0400 0--0200
Max Min
12 12.000006
3635009
268311 1183114441
000K-00.01 (0 0.02, 6.96)
40 0006H000 00.001, 0.02. 0.99)
0,00+00000
Sinear Nax-Min
16.1610.10
2.101.111
-119e+01
0.2010.01 (0.64, 0,07, 0.76)
0.1210011 0.01, 0.67)
0,0010.00
Fundcmule
Procy
Linear Worst
Linear Worst"
GRPO
175
-241-002
0355 0:17)
2.65-602 005 (:08)
CRBC+
11811.19
11810.19
4.66=006
1,366.0001 (0.25, 0.97, 0.13)
-1.3610.01 (0.25. 0. 0.97, 0.13)
1:2520.18
125
63 2913
-1.11 (0. 0.99, 0.01)
-1,1120001 10.14, 6, 0.59, 0.01)
sinear Naa-Mtin
3.65:30111
70010113
0,08100011 100001, 0.23, (U00)
0,17:00:02 (0.01, 0.22, 0.09)
Eay
Glucuse
RLIFF
Method
True x ItP)
Prony
Wurst
True
Prony
Worst
6.0:1011
100 48
27 54333333
830 N/A
06330021
-1844-1.00
6.3402
11 36-0.56
N/A
N/A
6.3+0.1
66-0.8
-1.710025
5:88+0922
114:0111
+0:00=0.01
3nsemblle
2.111123
-1,70.0,04
0.3
0.0
-1.0
11.0
-1.0
2.0
Pandemic
-2.0
-3
-3.0
3.00
-4.0
2.5
55.0
2000
2.00
1.00
-3.0
1.55
22.0
2.0
-2..0
-4.0
2.5
-26.0
24.00
5.0
-28.0
26.0
6.0
3.5
ORPO
oRpo-
May
UntM
ORPO oRpo
AMM
LinMi
7.0
Buat
ORPO
ORPO-
MM
Lindt
ORPO ORPO-
MM
Linm
Figure I: Mean reward and standard deviation under sampled and true reward features different proxy true reward correlation levels for the Traffic and Pandemic environments. Our methods MMax-Min and Linear Max-Min) yield more stable and higher average performance across all choices of r.
setup mimics a more realistic real-world scenario in which the true reward function may depend on features not explicitly modeled at training time. Comparing performance under this setting allows us to assess the robustness of each policy to unseen or misaligned reward structures. All rewards are normalized with respect the reference policy गet to ensure consistent scale across metrics, enabling fair and meaningful comparisons. Note that all worst-case rewards are reported using the fixed correlation level r specified during training.
4.2 RESULTS
Worst-Case Performance. Table presents the evaluation results on the Traffic, Pandemic, Glu- cose, and RLHF environments. Additional results for Tomato are provided in Appendix H Note that we omit the Linear Max-Min policy from the Glucose and RLHF environments for the fol- lowing reasons. In the Glucose and RLHF environment, both the proxy and true rewards used in prior work (Laidlaw et al. 2025) [aker et al. 2025) are based on single feature, making the lin- ear reward formulation trivial. Although the original Glucose simulator provides multiple candidate features related to patient health, selecting an appropriate feature combination without prior knowl- edge of clinical intent is nontrivial. Therefore, in both settings, we report only the results for the Max-Min policy alongside the baselines. Our Max-Min and Linear Max-Min policies achieve better expected worst-case performance under both general and linear adversarial rewards, while remaining competitive with baselines in terms of expected true and proxy rewards. Notably, the Max-Min policy attains the highest expected worst-case return, followed by I. near Max-Min,

---

Published as a conference paper at ICLR 2026
Conversely, Linear Max-Min yields the highest expected linear worst-case reward, followed by Max-Min, demonstrating the robustness of both approaches under worst-case scenarios. For the Linear Worst* evaluation, which uses reward features unseen during training, we observe minimal degradation in Max-Min policy's 's performance, indicating its strong robustness to feature variation. In contrast, the performance of L, near r Max-Mir declines in this case, suggesting its advantage diminishes when prior assumptions about feature structure are inaccurate. We also find that ORPO* exhibits better worst-case performance than the original ORPO. In particular, training the discrimina- tor more thoroughly significantly reduces the occupancy of state-action pairs that are not visited by the reference policy, indicating that more accurate estimation of the Radon- -oidoo-- derivative leads to improved policy robustness. Notably, in the Pandemic and Glucose environment, we observe no such unvisited state-action pairs, and the discriminator outputs remain small across all policies. This could be due to either the discriminator network not being fully optimized or its inability to capture rare events that fall outside the support of Tie-- Developing more reliable techniques for handling such rare unseen state-action pairs remains open direction for future work. We also report the adversarial weight vectors I for each policy. These weights reveal which features are most vulnera- ble proxy exploitation under the learned policy and can be used to diagnose and revise the proxy reward function, thereby improving robustness. This highlights the interpretability benefits of our framework. Moreover, several patterns emerge from the results, which is detailed in Appendix H.. We also notice that the En Ensemble baseline in the RLHF setting achieves only limited improve- ment in expected true return over the reference policy and attains a lower expected worst-case return than our method. These results indicate that using reward ensembles alone is insufficient to effec- tively mitigate reward hacking compared to our approach. However, such reward-centrie methods, including InfoRM (Miao et al.) (024) and RRM (Liu et al.) (2024a), can be easily integrated into our framework. In particular, these approaches can be used to construct a stronger proxy reward, which can then be plugged into our method to further improve performance.
Robustness Across Correlation Levels. To further assess the robustness of each policy across a broader range of proxy-true correlation scenarios, also compute the Linear Worst* for each pol- icy under varying r values. Specifically, for each r. we sample 1000 vectors O such that 0 E Rlin and report the average return and variance achieved by each policy over these sampled rewards. Im- portantly, the variation in r is applied only during evaluation; all policies are fixed and trained using the specific values reported in Appendix [F.]] Unlike evaluations that only consider several reward functions, this approach evaluates policy performance across the entire reward set Riie providing a more comprehensive measure of robustness and better reflecting nol-world scenarios where the true reward and correlation r are unknown. Figure I shows the average reward and variance achieved by each method under different levels of proxy-true reward correlation r. As expected, the reference policy Ter (blue) performs the worst across all correlation levels in both environments. In Traffic, its variance is relatively small, suggesting consistently poor but stable behavior. In contrast, variance highest in the Pandemic environment, indicating increased policy fragility. Notably, ORPO* (purple) consistently achieves lower variance than ORPO (red) across both environments and outperforms in terms of average reward at 2 0.9 and R 0.7 in Traffic, and across nearly all values in Pandemic. This underscores the importance of accurate discriminator training for improving both stability and robustness. Max-Min (green) demonstrates the highest average reward and lowest variance across a wide range of r values in both environments, showing strong resilience to reward misspecification. While Linear Max-Min n (orange) achieves the best performance at specific correlation levels, particularly r ss 0.3 in Traffic and r x 0.7-0.9 in Pandemic. As r decreases and the proxy becomes less informative, differences in average reward among methods shrink, while variance increases. These results highlight the significance of variance control in low-correlation regimes and demonstrate that Max-Min and Linear Max-Min offer robust and stable perfor- mance under high uncertainty.
5 CONCLUSION
In this work, we formalize reward hacking as a robust optimization problem and introduce both a Max-Min formulation with a closed-form adversarial reward and Linear Max-Min variant that further improves interpretability and tractability. We develop efficient algorithms and empirically validate that both Max-Min and Linear Max-Min policies achieve stronger worst-case performance and improved stability compared to prior baselines such as ORPO across diverse environments. We
further discuss limitations and broader impacts of our method in Appendices B and C
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported in part by NSF grant CNS-2146548 and a grant from the Louisiana Board of Regents. We thank the anonymous reviewers for their insightful and constructive feedback.
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine learning (ICML), 2004.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in ai safety. arXiv preprint 16X.vv:606.06550 2016.
J Andrew Bagnell, Andrew Y Ng, and Jeff G Schneider. Solving uncertain markov decision pro- cesses. 2001.
Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojeiech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint 2rXiv:2503.1/922 2025.
Anas Barakat, Souradip Chakraborty, Peihong Yu, Pratap Tokekar, and Amrit Singh Bedi. Towards scalable general utility reinforcement learning: Occupancy approximation, sample complexity and global optimality. arXiv preprint arXiv:2410.04108, 2024.
André Barreto, Will Dabney, Rémi Munos, Jonathan J Hunt, Tom Schaul, Hado P van Hasselt, and David Silver. Successor features for transfer in reinforcement learning. Advances in neural information processing systems, 30, 2017.
Andre Barreto, Diana Borsa, John Quan, Tom Schaul, David Silver, Mattco Hessel, Daniel Mankowitz, Augustin Zidek, and Remi Munos. Transfer in deep reinforcement learning using successor features and generalised policy improvement. In International Conference on Machine Learning, pp. 501-510 PMLR, 2018
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O'Brien Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp. 2397-2430. PMLR, 2023.
Serena Booth, W Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi. The perils of trial-and-erro reward design: misdesign through overfitting and invalid task spec- ifications. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp- 5920-5929, 2023.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge University Press, 2004.
Ronen Brafman, Giuseppe De Giacomo, and Fabio Patrizi. Ltff/llf aa--makovvian rewards. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
Alberto Camacho, Oscar Chen, Scott Sanner, and Sheila Mcllraith. Non-markovian rewards ex- pressed in Itl: guiding search via reward shaping. In Proceedings of the International Symposium on Combinatorial Search, volume 8, pp. 159-160, 2017.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
Peter Dayan. Improving generalization for temporal difference learning: The successor representa- tion. Neural computation, 5(4):613-624, 1993.
Erick Delage and Yinyu Ye. Distributionally robust optimization under moment uncertainty with application to data-driven problems. Operations rexearch, 58(3)595-612, 2010.

---

Published as a conference paper at ICLR 2026
Esther Derman, Matthieu Geist, and Shie Mannor. Twice regularized mdps and the equivalence between robustness and regularization. Advances in Neural Information Processing Systems, 34: 22274-22288 2021.
Dotan Di Castro, Aviv Tamar, and Shie Mannor. Policy gradients with variance related risk criteria. In International Conference on Machine learning (ICML), 2012.
Finale Doshi-Veles and Been Kim. Towards rigorous science of interpretable machine learning. star, 1050:2, 2017.
Jeff Druce, Michael Harradon, and James Tittle. Explainable artificial intelligence (xai) for in- creasing user trust in deep reinforcement learning driven autonomous systems. arXiv preprint 03f:::200600377,, 2021.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 2023.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herd- ing? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244 2023.
Tom Everitt, Victoria Krakovna, Laurent Orseau, Marcus Hutter, and Shane Legg. Reinforcement learning with corrupted reward channel. In International Joint Conference on Artificial Intelli- gence (ICCAI) 2017.
Tom Everitt, Marcus Hutter, Ramana Kumar, and Victoria Krakovna. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. Synthese, 198 (Suppl 27):6435-6467, 2021.
Benjamin Eysenbach and Sergey Levine. Maximum entropy rl (provably) solves some robust rl problems. In International Conference on Learning Representations.
Alan Femm, Sriraam Natarajan, Kshitij Judah, and Prasad Tadepalli. A decision-theoretic model of assistance. Journal of Artificial Intelligence Research, 50:71 2014.
Ian Fox, Joyce Lee, Rodica Pop-Busui, and Jenna Wiens. Deep reinforcement learning for closed- loop blood glucose control. In Machine Learning for Healthcare Conference, pp. 508-536. PMLR, 2020.
Matt Franchi, JD Zamfirescu-Pert Periera, Wendy Ju, and Emma Pierson. Detecting disparities in police deployments using dashcam data. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp. 534-544, 2023.
Clement A Gehring. Approximate linear successor repressnaatot Reinforcement Learning Decision Making, 2015.
Joel Goh and Melvyn Sim. Distributionally robust optimization and its tractable approximations. Operations research, 58(4-part-1):90)
2010.
Charles AE Goodhart and CAE Goodhart. Problems of monetary management: the UK experience. Springer, 1984.
Vineet Goyal and Julien Grand-Clement. Robust markov decision process: Beyond rectangularity. arXiv preprint arXiv: 181100011,. 2018.
Julien
and Christian Kroer. Scalable first-order methods for robust mdps. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 12086-12094, 2021.
Dylan Maadfeld-Menell Smitha Milli, Pieter Abbeel. Stuart J Russell, and Anca Dragan. Inverse reward design. In Advances Neural Information Processing Systems, 2017.
12

---

Published as a conference paper at ICLR 2026
Yuval Noah Harari. Nexus: A history of information networks from the stone age Al. Signal, 2024.
Alexander Havrilla, Maksym Zhuravinskyi, Duy Phung, Aman Tiwari, Jonathan Tow, Stella Bider- man, Quentin Anthony, and Louis Castricato. trlx: framework for large scale reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 8578-8595, 2023.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, 2016.
Andreas Holzinger, Chris Biemann, Constantinos Pattichis, and Douglas B Kell. What do we need to build explainable ai systems for the medical domain? arXiv preprint arXiv:1712.099233 2017.
Zhaolin Hu and L Jeff Hong. Kullback-leibler divergence constrained distributionally robust opti- mization. Available at Optimization Online, 1(2)9, 2013.
Audrey Huang, Jinglin Chen, and Nan Jiang. Reinforcement learning in low-rank mdps with density features. In International Conference on Machine Learning. pp. 13710-13752. PMLR, 2023.
Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of kl-regularization: Direct alignment without overopti- mization via chi-squared preference optimization. arXiv preprint urXiv:2407.1339% 2024.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems, 2018.
Rodrigo Toro Icarte, Toryn Q Klassen, Richard Valenzano, and Sheila A Mcllraith. Reward ma- chines: Exploiting reward function structure in reinforcement learning. Journal of Artificial telligence Research, 73:173-2000 2022.
Hamish Ivison. Yizhong Wang, Jiacheng Liu, Zeqiu Wu, Valentina Pyatkin, Nathan Lambert, Noah A Smith, Yejin Choi, and Hanna Hajishirzi. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. Advances neural information processing sys- tems, 37:36602-366333 2024.
Garud Iyengar. Robust dynamic programming. Mathematics of Operations Research, 30(2):257- 280, 2005.
Rahul Iyer, Yuezhang Li, Huao Li, Michael Lewis, Ramitha Sundar, and Katia Sycara. Transparency and explanation in deep reinforcement learning neural networks. In Proceedings of the 2018 AAAI/ACM Conference AI, Ethics, and Society, pp. 144-150, 2018.
Hong Jun Jeon, Smitha Milli, and Anea Dragan. Reward-rational (implicit) choice: A unifying formalism for reward learning. In Advances in Neural Information Processing Systems, 2020.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In International Conference on Machine Learning, pp. 4940-4950, PMLR, 2021.
Chi Jin, Praneeth. Netrapalli, and Michael Jordan. What is local optimality in nonconvex-nonconccays minimax optimization? In International conference on machine learning, pp. 4884-488488 PMLR, 2020.
Bingyi Kang, Zequn Jie, and Jiashi Feng. Policy optimization with demonstrations. International Conference on Machine Learning (ICML), 2018.
David L Kaufman and Andrew J Schaefer. Robust modified policy iteration. INFORMS Journal on Computing. 25(3):396-410, 2013.
Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. The challenge of understanding what users want: Inconsistent preferences and engagement optimization. Management Science, 70(9): 6336-6355, 2024.
13

---

Published as a conference paper at ICLR 2026
W Bradley Knox and James MacGlashan. How to specify reinforcement learning objectives. In Finding the Frame: An RLC Workshop for Examining Conceptual Frameworks, 2024.
W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. Artificial Intelligence, 316:103829, 2023.
Jack Koch, Lauro Langosco, Jacob Pfau, James Le, and Lee Sharkey, Objective robustness in deep reinforcement learning. arXiv preprint arXiv:2 14/11,11111111 2, 2021.
V Kompella, R Capobianco, S Jong, J Browne, S Fox, I. Meyers, P Wurman, P Stone, et al. Re- inforcement learning for optimization of covid- 19 mitigation policies. In CEUR WORKSHOP PROCEEDINGS, volume 2884, CEUR-WS, 2020.
Victoria
Krakovna.
URL Specification gaming examples in ai. April
2018.
waraaa.wrrddss...o wwt.comm//m//// 2amm22118 /02/ speci cation- ng new-eeaplls---eeeeeeeeeee Blog post.
Victoria Krakovna.
Classifying specificatiio problems variants of goodhart's law,
August 2019.
URL http:///.caaakooo.oooooo......./
Blog post.
Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg. Penalizing side effects using stepwise relative reachability. arXiv preprint arXiv:/806.07 186, 2018
Cassidy Laidlaw, Eli Bronstein, Timothy Guo, Dylan Feng, Lukas Berglund, Justin Svegliato, Stuart Russell, and Anca Dragan. Scalably solving assistance games. In ICLR 2025 Workshop Bidirectional Human-Al Alignment, 2024.
Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and im- proved mitigation for reward hacking. In International Conference on Learning Representations, 2025.
Henry Lam. Robust sensitivity analysis for stochastic systems. Mathematics of Operations Re- search, 4484) 2016.
Jan Leike, Miljan Martic, Victoria Krakovna, Pedro Ortega, Tom Everitt, Ryan Lefrancq. Laurent Orseau, and Shane Legg. AI safety gridworlds. arXiv preprint arXiv:1711.09853 2017.
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph E Gonzalez, Michael I Jordan, and lon Stoica. Rllib: Abstractions for distributed reinforcement learning. International Conference on Machine Learning (ICML), 2018.
Fei Liu al. Learning summarize from human feedback. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 583-592, 2020.
Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. Rrm: Robust reward model training mitigates reward hacking. arXiv 2024a.
preprint arXiv:2409.1311
Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. Enhancing Ilm safety via constrained direct preference optimization. Xrviv preprint arXiv:2403.02475 2024b.
Chiara Dalla Man, Francesco Micheletto, Dayu Lv. Mare Breton, Boris Kovatchev, and Claudio Cobelli. The uva/padova type diabetes simulator; new features. Journal of diabetes science and technology, 8(1):26-34, 2014.
Jeremy Memahan, Giovanni Artiglio, and Qiaomin Xie. Roping in uncertainty: Robustness and regularization in markov games. In International Conference on Machine Learning, pp. 35267- 35295. PMLR, 2024.
Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mit- igating reward hacking in rlhf via information-theorett reward modeling. Advances in Neural Information Processing Systems, 37:134387-134429, 2024.
14

---

Published as a conference paper at ICLR 2026
Smitha Milli, Luca Belli, and Moritz Hardt. From optimizing engagement to measuring value. In ACM Conference on Fairness, Accountabiliry; and Transparency ( (FAccT), pp. 714-722, 2021.
Nathan Monette, Alistair Letcher, Michael Beukman, Matthew T Jackson, Alexander Rutherford, Alexander D Goldie, and Jakob N Foerster. An optimisation framework for unsupervised envi- ronment design. arXiv preprint (eXiv:2505520653 2025.
Jorge J. Moré. The twrebbeg-marquaree algorithm: Implementation and theory. Technical Report ANL-80-20, Argonne National Laboratory, Argonne. IL, 1978. Lecture Notes in Mathematics, vol. 630.
Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. Rule based rewards for language model safety. Advances in Neural Information Processing Systems, 37:108877-108801, 2024.
Calarina Muslimani, Kerrick Johnstonbaugh, Suyog Chandramouli, Serena Booth, W Bradley Knox, and Matthew Taylor. Towards improving reward design in rl: reward alignment metric for practitioners. arXiv preprint wwxiv:2503.05990 2025.
Hoang Nam Nguyen, Abdel Lisser, and Vikas Vikram Singh. Distributionally robust chance- constrained markov decision processes. arXiv preprint arXiv:2212.08) 2022.
Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780-798, 2005.
Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366((464):447-4555 2019.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022.
A Paszke. Pytorch: An imperative style, high nigh-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019.
Erika Puiutta and Eric MSP Veith. Explainable reinforcement learning: A survey. In International cross-domain conference for machine learning and knowledge extraction, pp. 77-95. Springer, 2020.
Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12188 2024.
Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206 2256,22151111 2019.
Karim Abdel Sadek, Matthew Farrugia-Roberts, Usman Anwar, Hannah Eriebach, Chris- tian Schroeder de Witt, David Krueger, and Michael Dennis. Mitigating goal misgeneralizatioon via minimax regret. arXiv preprint arXiv:2507.03066 2025.
Dorsa Sadigh, Anca Dragan, Shankar Sastry, and Sanjit Seshia. Active preference-basee learning of reward functions. Robotics: Science and Systems, 2017.
Andreas Schlaginhaufen and Maryam Kamgarpour. Identifiability and generalizability in con- strained inverse reinforcement learning. International Conference on Muchine Learning (ICML), 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. In arXiv preprint arXiv:1707.06344 2017.
Rohin Shah, Pedro Freire, Neel Alex, Rachel Freedman, Dmitrii Krasheninnikov, Lawrence Chan, Michael D Dennis, Pieter Abbeel, Anca Dragan, and Stuart Russell. Benefits of assistance over reward learning. 2020.
15

---

Published as a conference paper at ICLR 2026
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9477 2022.
Garry M Steil. Algorithms for a closed-loop artificial pancreas: the case for proportional-interaa derivative control. Journal of diabetes science and technology, 7(6):6221-161,, 2013.
Jonathan Stray, Alon Halevy, Parisa Assar, Dylan Hadfeeld-Menell, Craig Boutilier, Amar Ashar, Chloe Bakalar, Lex Beattie, Michael Ekstrand, Claire Leibowiez, et al. Building human values into recommender systems: An interdisciplinary synthesis. ACM Transactions Recommender Systems, 233):1-57, 2024.
Richard $ Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. In International Conference on Learning Representations, 2023.
Martin Treiber, Ansgar Hennecke, and Dirk Helbing. Congested traffic states in empirical observa- tions and microscopic simulations. Physical Review E, 62(2): 1805, 2000.
Eugene Vinitsky, Aboudy Kreidieh, Luc Le Flem, Nishant Kheterpal, Kathy Jang, Cathy Wu, Fangyu Wu, Richard Liaw, Eric Liang, and Alexandre M Bayen. Benchmarks for reinforcement learning in axxed-autonomy traffic. In Conference Robot Learning, pp. 399-409. PMLR, 2018.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, C. Carey, ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental algorithms for scientific computing in python. Nature Methods, 17:261-272, 2020. doi: 10. 00188//41522-19--000
George A Vouros. Explainable deep reinforcement learning: state of the art and challenges. ACM Computing Surveys, 55(5):1-39, 2022.
Cathy Wu, Abdul Rahman Kreidich, Kanaad Parvate, Eugene Vinitsky, and Alexandre M Bayen. Flow: A modular learning framework for mixed autonomy traffic. IEEE Transactions on Robotics, 38(():127001 1286, 2021.
Tengyang Xie, Bo Liu, Yangyang Xu, Mohammad Ghavamzadeh, Yinlam Chow, Daoming Lyu, and Dacsub Yoon. A block coordinate ascent algorithm for mean-variance optimization. In Advances in Neural Information Processing Systems, 2018.
Rui Yu, Shenghua Wan, Yucen Wang, Chen-Xiao Gao, Le Gan, Zongzhang Zhang, and De- Chuan Zhan. Reward models in deep reinforcement learning: A survey. arXiv preprint M.Xiv:2506.15421,, 2025.
Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory, pp. 2730-2775. PMLR, 2022.
Junyu Zhang, Amrit Singh Bedi, Mengdi Wang, and Alec Koppel. Multi-agent reinforcement learn- ing with general utilities via decentralized shadow reward actor-critic. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 9031-9039, 2022.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
LLM Usage
19
Limitations and Future Work
19
Broader Impacts
20
D Related Work
20
D.I Reward Hacking
20
D.2
Reward Modeling in Reinforcement Learning
21
D.3 Robust Reinforcement Learning
21
D.4
Successor Representations in Reinforcement Learning
22
Proofs and Additional Theoretical Results
22
Solve the Max-Min Objective
22
Proof of Optimality
26
................................
E,3
Proof that (²
(
Rproxy
28
E,4
Derive Lagrangian Functional for Linear Max-Min Objective
................................
29
E.5
Proof for Whitening Transformation
30
E,6
Derive Optimal Primal Variable for Linear Max-Min Objective
................................
31
E.7
Solve the Dual Objective for Linear Max-Min Objective
32
E.8
Policy Gradient Derivation
33
Proof of Theorem
35
Additional Implementation Details
35
F1
Training Discriminator Network
................................
35
Derivation of Max-Min Policy Optimization
37
Derivation of Linear Max-Min Policy Optimization
................................
38
F4
Environment Description and Reward Hacking Types
40
F5
Additional Experiment Setup
42
F66
Training Time and Complexity
44
Convergence Analysis
45
G.I
Sample Complexity Occupancy Measure Estimation
45
G.2
Guarantees for Max-Min with Occupancy Measure Approximation
51
G.3
Convergences for Linear Max-Min algorithm
52
H
Additional Experiment Results
53
Feature Weights in Linear Max-Min Mix Optimization during Training
53
H.2
Additional Worst-Case Performance Results
54
17

---

Published as a conference paper at ICLR 2026
H.3
Additional Results for Robustness Across Correlation Levels
56
Additonal Unnormalized Results
57
5 Results for all
57
How to choose practice?
58
18

---

Published as a conference paper at ICLR 2026
A LLM USAGE
In this work, we used ChatGPT to improve grammar, wording, and paragraph flow throughout the paper after completing an initial draft. We also used ChatGPT's research capabilities help sur- face potentially relevant prior work for the related work and introduction sections. All references were independently verified by the authors. No algorithms, proofs, or experimental results were generated by ChatGPT, and no proprietary or sensitive data were shared with the tool. All technical contributions and analyses are solely the authors' work.
B LIMITATIONS AND FUTURE WORK
Despite the effectiveness of our framework, several limitations remain. First, although our Max-Min formulation can be extended naturally beyond linear reward structures, incorporating more expres- sive representations such as neural networks makes the inner optimization problem significantly harder. In such cases, the inner minimization may longer admit a closed-form solution, necessi- tating iterative training between the policy and adversary. This increases computational complexity and undermines the efficiency advantages of our current formulation. Developing scalable solutions for general reward representations remains an open direction.
Second, in our Linear Max-Min algorithm, computing Q results in Odd² space and O(d3) time complexity. While low-rank approximations could potentially reduce computational cost, such methods often discard small eigenvalues. However, in our setting, these small eigenvalues be- come critical due to the inversion in the whitening step, and removing them may severely distort the worst-case reward direction. Therefore, naive low-rank approximations may not be applicable in our setting, and we emphasize the need for principled, scalable extensions when applying method to settings with very high-dimensional feature spaces.
Third, for complex environments, constructing effective features for the reward function is often challenging without prior domain knowledge. For example, in the Glucose environment, a large number of healthhrelated indices are provided. However, without medical expertise or knowledge of glucose monitoring, it is difficult to determine which combination of indices best captures patient health blood glucose trends. Using arbitrarily selected features in such cases can lead proxy rewards that exhibit little or no correlation with the true reward. While our max-min formulation can still offer robustness under such misspeciffcation, the resulting policy nevertheless expected to perform poorly due to the fundamental misalignment between the proxy and true objectives. There- fore, designing meaningful reward features remains a fundamental and unresolved challenge, and we will include this as limitation of our method in the main text. Moreover, in some environ- ments, such as Tomato, the reward function is not explicitly feature-based. Although our general max-min algorithm still applies in this setting, incorporating non feature-based reward structure into the uncertainty set remains an open problem.
Fourth, like ORPO, our framework assumes access fixed proxy reward, reference policy, and a pre-specified correlation parameter r. all provided offline. This setup limits the ability of the algorithm to adapt refine its reward model based new information. However, observe that the adversarial rewards generated by our method, particularly the structured linear ones, can serve as diagnostic tools identify vulnerable reward features. These insights could be leveraged to guide human-in-the-loop refinement or adaptive querying of stronger feedback models (c.g., large language models). Extending our framework close the loop between diagnostic robustness and reward learning is an exciting direction for future work.
Fifth, while our experimental results demonstrate that the proposed method improves robustness across a range of proxy-true reward correlation levels, an alternative and perhaps more direct strategy would be train the policy against multiple proxy rewards sampled at varying levels of correlation In principle, optimizing the average performance across a diverse set of proxies could yield policy that is robust to a wider distribution of potential reward misspecifications. However, this approach presents several practical challenges. First, there is trade-off between computational cost and coverage: sampling too few proxies may fail to represent the full space of plausible reward deviations, while sampling many proxies significantly increases training time. Second, efficiently generating reward functions that satisfy a fixed correlation constraint with the proxy reward becomes non-trivial in high-dimensional or continuous state-action spaces. Designing scalable and effective
19

---

Published as a conference paper at ICLR 2026
reward sampling mechanisms (such as leveraging diffusion models) under correlation constraints remains an open problem and a promising direction for future research.
C BROADER IMPACTS
Designing reward functions that faithfully reflect designer intent remains a fundamental challenge in deploying reinforcement learning (RL) systems in the real world. When reward misspecification occurs, agents can behave in undesirable or even dangerous ways. Our work addresses this issue by proposing a robust policy optimization framework that explicitly accounts for uncertainty in the reward function, improving worst-case performance across a range of plausible reward proxies. This approach has the potential increase the safety and reliability of RL systems in safety-critical ap- plications such as healthcare, autonomous driving, and digital infrastructure, where poorly specified incentives can lead unintended consequences. In addition robustness, our linear variant con- tributes to policy interpretability by yielding explicit weightings over features that can be inspected and audited. This can help practitioners identify vulnerable components in their reward specification and make better-informed decisions when refining proxies. However, while our method is primarily intended to prevent reward exploitation, one could conceivably use adversarial reward modeling to stress-test attack policies. We believe the benefits of improved safety and robustness outweigh this risk, especially when combined with interpretability. Overall, this work contributes to the safe and trustworthy deployment of RL by equipping practitioners with more robust and explainable optimization tools.
D RELATED WORK
D.1 REWARD HACKING
Early work in AI safety underscored the pitfalls of optimizing an imperfect proxy reward. Amodei et al. (Amodei et al. 2016) famously illustrate how agent "game" its reward function: for example, a cleaning robot rewarded for not seeing any messes might simply close its cameras or create messes to clean up, maximizing the proxy reward while betraying the designer's intent. Other examples of such reward hacking include an agent in a racing game that spins in circles to collect points instead of completing the race (Skalse al. 2022), social media recommendation systems that promote emotionally extreme content to increase engagement (Harari, (2024), and Large Lan- guage Models (LLMss) that generate trivial or hard-coded solutions to pass unit tests rather than producing general, correct code (Baker et al. 2025).. Krakovna et al. (Krakovna) 2018) have cata- logued many such failure cases across diverse domains. Several studies have analyzed the causes of reward hacking (Amodei et al., 2016 Krakovnal 2019 Skalse et al. (022), often interpreting it as a manifestation Goodhart's s Law (Goodhart & Goodhart 1984): when a proxy metric becomes a target for optimization, it ceases to be good measure. In reinforeement learning, this risk is partic- ularly acute because agents can exploit even small imperfections in the reward specification. Pan et al. (Pan al. 2022) further propose a taxonomy of proxy reward misspecification into three types: misweighting, ontological, and scope errors.
To mitigate such risks, several reward-centric methods have been proposed (Hadfiel-Mennll al., 2017 Ramé et al., (2024). Inverse Reward Design (Hadfield-Menell et al. 2017) aims to infer the in- tended true objective from a given proxy and its training context, helping agents generalize without exploiting flawed signals. Recent work by Rame et al. (Ramé et al. 2024) averages the parameters of multiple reward models to smooth out idiosyneratic errors, reduce the impact of individual proxy biases, and demonstrate reduced reward hacking on held-out tests. Another line of defense focuses on regularizing policy behavior to reduce sensitivity to reward flaws. Common approaches include penalizing divergence from a reference policy using KL-regularization (Liu et al.) (020). Recent research by Laidlaw et al. (Laidlaw et al. 2025) proposes Occupancy-Regulaaiize Policy Optimiza- tion (ORPO), which applies a x penalty the state-action distribution to constrain deviation from a baseline policy and reduce exploitative behaviors. Another complementary paradigm is assistance games (Fern et al. 2014 Shah et al. 2020). in which human users remain actively involved and the agent's ' actions complement the user's 's to achieve optimal joint performance. Assistance games can mitigate reward hacking by removing incentives for deception since the agent's performance
20

---

Published as a conference paper at ICLR 2026
depends the human's latent (true) reward. Recent work has developed scalable assistance- game approaches in practice (Laidlaw et al. 2024).
Overall, existing approaches either attempt correct the reward specification, regularize against fixed proxy, or explicitly involve human interaction. In contrast, our method trains policies against an entire set of plausible proxy rewards, those that remain sufficiently correlated with the true reward, offering robustness to a broader range of misspecifications. Moreover, we show that this robust training objective can be reformulated an equivalent regularized optimization problem, providing both theoretical and practical benefits.
D.2 REWARD MODELING IN REINFORCEMENT LEARNING
In standard RL benchmarks, the reward is usually assumed to be given, but real-world applications rarely offer a well-defined reward signal upfront. Therefore, designing an effective reward func- tion, often referred to reward modeling, is critical yet challenging aspect of RL (Skalse et al.) Liu et al. 2024b Booth et al. 2023 Knox & MacGlashan (024). Moreover, evaluating whether designed reward truly captures the designer's intent is non-trivial. Recent work (Musli- mani et al. 2025) has proposed to measure reward alignment via a "trajectory alignment coefficient," which quantifies how closely the rankings of trajectories induced by given reward match a human stakeholder's preferences. Such efforts underscore the importance of conceptual frameworks that go beyond treating the reward as a black box, instead focusing on principled reward design and evaluation.
2022
To incorporate domain knowledge and improve interpretability, researchers have explored structured or rulc-based reward modeling frameworks (Icarte et al. 2022 Brafman et al. 20188 Camacho et al., 2017]. One prominent example the use of reward machines (caarte et al. 2022) that explicitly rep- resent the reward function's 's logic. A reward machine exposes the internal structure of the reward (e.g. different sub-goal phases or conditions) to the agent, enabling techniques like automated re- ward shaping and task decomposition for more sample-efficient learning. Even before the advent of reward machines, prior works had leveraged logical task specifications to design rewards. For instance, translating Linear Temporal Logic (LTL) formulas into automata and rewarding the agent upon reaching designated accepting states (Camacho et al., 2017 Bratman et al. 2018). By defin- ing rewards through such rules logical templates, the intended behavior is encoded transparently, making the reward function more interpretable. Along similar lines, many approaches assume a structured parametric form for the reward function itself to aid transparency (Yu et al. 2025| Mu eet al. 2024]. In particular, it is common to model the reward as a linear combination of feature functions, simplification used in inverse RL and preference based reward learning to make reward inference tractable and explainable (Yu et al. (2025). Recent work in RL from human feedback also implements rule-based reward signals as linear models over interpretable features (Mu et al. 2024). Our approach follows this tradition: by assuming the reward is linear in a set of human-interpretablle features, we improve the interpretability of the learned policies and reveal which feature components robust vulnerable.
D.3 ROBUST REINFORCEMENT LEARNING
Our work is also related to robust reinforcement learning, where the agent assumes the reward function (and/or transition dynamics) lies within a given uncertainty set, and it seeks to maximize performance against the worst-case realization from that set. This can be formulated as a zero-sum dynamic game between the agent and an adversary who selects the most adverse reward or dynamics; solving the robust MDP thus involves a challenging max-min optimization (Iyengar, 2005 Nilim & El Ghaoui 2005). To alleviate the computational complexity, early works in this vein rely on a rectangularity assumption that is crucial for traceability. Thus, classical robust RL. formulation typically considers rectangular uncertainty sets rewards or transition probabilities, which lead to conservative solutions but permit efficient algorithms such as robust value iteration (Bagnell et al,
2001 Grand-Clément & Kroer 2021 modified policy iteration (MPI) (Kaufman & Schaefer
2013
Recent theoretical work has revealed intimate connection between adversarial robustness and pol- icy regularization in the context of rectangular uncertainty sets. Several researchers have shown that solving robust MDP is equivalent to solving certain regularized RL. problem (Derman al. 2021
21

---

Published as a conference paper at ICLR 2026
Eysenbach & Levine Mcmahan et al. 2024). In particular, the worst-case effect of the adversary can often be captured via an additional penalty term in the agent's 's objective. Derman et al. (Der-) man et al. (021) prove that any entropy- or L'-regularized MDP can be interpreted MDP with uncertain rewards- - in fact, a regularized MDP a special case of a reward-robust MDP. Their analysis establishes duality between max-min reward-robust objective and a single-agent maximization of expected reward plus regularization term. Eysenbach and Levine (Eysenbach & Levine) show that the optimal policy from maximum entropy RL formulation is provably robust to some adversarial reward perturbations. More recently, these insights have been extended and for- malized for general MDPs and even multi-agent settings. McMMaan al. (Memahan et al., 2024) study robust Markov games with (s, u)-rectangular uncertainty, and they prove that computing robust equilibrium polynomial-time equivalent to computing an equilibrium in a corresponding regularized game. In their framework, the added regularization term is exactly the support func- tion of the uncertainty set, effectively the dual representation of the adversary's worst-case reward selection. This means that for common uncertainty sets (e.g., those inducing entropy lp-norm regularizers), one can replace the inner minimization over rewards with an explicit regularization term in the objective.
robust
The setting in our work departs from the above literature by considering non-rectangular reward uncertainty. In particular, tion, meaning that the adversary's permissible deviations reward are coupled across states. This structure can mitigate the conservativeness of the worst-case solution (the adversary cannot simul- tancously push all state rewards to their extreme worst values) (Goyal & Grand-Clement (0188), but it also means that the neat robustnes--rguulaaization duality from the rectangular-case no longer applies and the robust optimization must be solved (or approximated) directly. In summary, our work tackles a form of reward uncertainty which lies beyond the scope of existing robustness-as- regularization analysis.
assume correlation-constrainee uncertainty set for the reward func-
D.4 SUCCESSOR REPRESENTATIONS IN REINFORCEMENT LEARNING
The linear reward assumption and the use discounted feature expectations are also closely re- lated to the literature on successor representaations/suceesss features. Successor representations and successor features represent values as inner products between reward weights and discounted occu- pancies features of future states and actions. It was introduced a generalization of the value function (Dayan 1993). This idea was later generalized by (Gehring 2015 Barreto et al. 2017; 2018) to handle nigh-dimensional, continious state spaces and to use the method for transfer learn- ing. Specifically, (Barreto et al. 2017) formalize this idea into the successor features (SF) framework for transfer learning, assuming that tasks share dynamics but differ only in their reward functions parameterized as linear combinations of features. This yields a value function representation that effectively decouples the environment's 's transition dynamics from the reward parameters. (Barreto) et al. 2018) further extend successor features to deep reinforcement learning and introduce gener- alized policy improvement over multiple tasks, demonstrating effective transfer by reusing learned successor features across a family of related tasks.
E PROOFS AND ADDITIONAL THEORETICAL RESULTS
E. SOLVE THE MAX-MIN OBJECTIVE
In this section, we show the complete proof for solving the following max-min problem:
max min J(n, R) max min (R(e)),,[[(ss.) a)).
(15)
x RER
x RER
where Reon is the entire space of rewards that satisfy the correlation constraint with respect to known proxy reward, as defined below:
R - M V
Runo
R: as, +
Eprie
Rpraxy r, [ R) M, o V2
(16)
R R-a M
R: (s, + Eppaa
Roroay] By [R]
M, E Ey R² V2 + M
V
22

---

Published as a conference paper at ICLR 2026
M and V denote the fixed mean and standard deviation of the reward function R. under the reference policy Tet. Rproxy is the normalized proxy reward
Retoxy (s, a) = Rproxy (s,oxy(3.0) a) [ Rgroxy)
where Riiexy is the original (unnormalized) proxy reward. J [TTrrt Rproxy) = O and Var (Rroxy)
After normalization, have
To solve the challenge that the objective Ep, [R(s, a)] depends on the state-action occupancy Pas whereas the constraints defining Rever expressed in terms of Prrer We apply change-of-
technique (Hu & Hong 2013 Lam 2016) to rewrite the expectation under Paar Specifically, let
L(s, a) denote the Radon-Nikodyee derivative:
1xx (5,
L(s a)
P (8, 0.)
By definition, L(s, a) 2 and Ep [[s = 1. Applying the change-of- measure formula, we can express the return as:
1 Mx ((ss d(s,a)
Bp, [R(s,
SSX
=JScA -SXA M a) (z( a(s
(),
Eppes [Tss, a) R(s,a)
Thus, both the objective and the constraints can be rewritten as expectations with respect to the reference distribution Maw. For notational simplicity, we will suppress the variables (s, a) when necessary. Under this reparameterization, the max-min objective in Equation [5 reformulated as:
max min Ep [L R]. RER
(17)
Solve Inner Minimization Problem. mization problem of 17 is defined as:
The Lagrangian functional associated with the inner mini-
R- M V
4o(A, A2, Ag. R) = Eur IL.
A
Raroxy - + + Ag V2)
where Å1 A2, λ are the Lagrange multipliers corresponding to the correlation constraint, mean con- straint, and variance constraint, respectively. Then the inner minimization problem Equation 17 is equivalent to the following problem:
max min 4o(A,, A2, Aa, Rt)
(18)
A Ag, RER
We now solve the inner minimization problem in Equation by finding the optimal for fixed dual variables (21,d2;As), Taking the functional derivative of the Lagrangian lo with respect to R(s, a) gives:
Ol
R Repoxy (s,a)
N (s, as,a) -A1
X2) 223R]
0R
When Maa (s) > 0, setting the derivative of the Lagrangian to zero yields the optimal adversarial
reward function:
R² (s, a) L(8, a) X1
Rynoc (s,a)) X2
(19)
2)g
23

---

Published as a conference paper at ICLR 2026
However, for state-action pairs where (s.aa = 0, i.e., those not visited under the reference policy, the correlation and moment constraints become vacuous. In these regions, the adversarial reward R (s, a)) can be driven arbitrarily poor, reflecting that no constraint prevents the adversary from assigning highly penalizing values to rarely visited or unobserved state-action pairs. Never- theless, consider the case where Mr ( (8,a) > 0, after substituting the optimal R" from Equation 19
into the Lagrangian lo in Equation 18 and simplifying, we obtain the following dual objective:
(L(s,a) A1 Runayys.d
22)²
max 4o(A1, A2, A3, R2)) A1,d2,A3
+2
4A3
(20)
We now compute the gradients of the dual objective with respect the dual variables:
Olp
[L. X1 (p
X2)
Pipnay (s.a)
-Eptat
+r
aλ1
2)3
Olg
L A1 Rypay
A2
- EEptat
+
0A2
2
Ølp
(L A1 Rs
A²)
Epret
+ M + V2
(21)
0
412
Setting these gradients to zero yields the system of equations: (e.((
A2) (8.0)
E
(22)
213
R Rpoa(o.a)
L \1
A2
E Eppa
M,
(23)
2A3
(L A1 B (8,a)
(s,a)
)2)
B
= M + V2.
(24)
4X3
Expanding and simplifying each condition:
Solving for Az=
Starting with Equation 23
Rproxy (5,a) a)
Eptrng [L] X E praee
- d2 2A
V
Recall from our normalization that E, E [Rpaxxy) = Thus,
A2 Eut,ar [L] 2\3M
Since L(s, a) Mr (s,a) and using properties of Radon-Nikodyn derivatives, we have: (s.a]
E pl [L]
Thus, find:
A2 1 2\3M
Solving for A1= Substituting d2 22gM M into Equation 22
P
Rproxy
+ 2A3M) Rpooxy
2r = EBuraa [[L A1
-Epee [L.
Reroxy
V A1 1/2
Proxy
2 M) E
Ryroxy
V
Again, using normalization, Epr, [Rrrxyy] O and Eppr [RPraxy)]
1, so get:
Rproxy
A1
2r = Epprar [T.
V
V2
which rearranges to:
\1 Vi [L Rproxy] 2r V2
24

---

Published as a conference paper at ICLR 2026
Solving for As:
Substituting X2 22xM into Equation 24
Rproxy
415(M² + V2) = Byrne
(1. \ι
1 + 22
V
R2 Airoxy
R Rprixy
Bu [1,2] + Ai jet
+ (1 223 1/2
22 D [
Rproxy
2(1 22 RR ao [T]] 2A1(1 223 BuT
Again, using normalization (E Rproxyy = 0, IE| RR proxy] = 1, [R[T] = 1), this simplifies to:
AP
Rpxxy
4M3 V2) E E [L,]+
22 E [1.
422M2
1/2
V
A
Rprxx
423V² Eppral
[I +
22 Epa [L.
V22
V
Now substitute A1 V [L. Reroayl Ar 222 into this expression. After rearrangement and simplification, we obtain:
41 - -22 Bytrae [L2]
E² Hen
IL. Rprovy]
Thus,
A3 = + V Bue [I.]] 2
E2 Mxxe
[L.
Repwayl
VV
We argue that A3 <
0 yields the optimal dual variable. To determine which root maximizes the
above dual objective in Equation 20. we compute the second derivative from Equation 21
8
(L X1 Rymay
x22
EE E
V
a
2X3
Since the numerator is always non-negative and when A3 < O, we have 8 < O, which implies ox. the dual objective is concave in A around this root. Thus, selecting the negative root yields local maximum of the dual objective.
Recognizing that Er [12]] 1 corresponds to the x² divergence between the occupancy measures:
x2 (px (itee) EE [L
and noting that:
E B [L. Rproxy] = Eyp [Rprxyy
we can express the solution for 03 as:
vx ( ll (ma) - E2 [Rrray)
A
(25)
2V
Solve Outer Maximization Problem. Now that we have solved for the optimal primal variable R and dual variables Å1. Xg. and A3, we plug them back into the original max-min objective in Equation
Minel LL max Bunne L(s,a)- L(s, A1 Ryony (s,0) X2
max 18
(26)
2
Using the earlier substitutions:
\ V. -,, [Prooyy 22
A = 22yM,
1. Vx²(Hs (Pamm E2 [R Pproxy
Ag =-2
VV1 -
25

---

Published as a conference paper at ICLR 2026
We simplify the expression:
Rproxy
max E [LR] = max
Burar [1:2] A2
L.
22 per [1]
2ds
Recall the identities:
E Ey [T.2] = x (14r ll (Tal + 1,
Epera [L Rproxy] = Byp (Rroxy]:
E paot
[1] = 1,
We substitute these and get:
1
Eyg, Mproxy) x2
max E o2
[.
x (1T MITee) +1 A1
2d3
V
Now substitute the expressions for A1 and A2:
1
x² + 1- (Byrr [Rpaay] 1 2rAsV) Ey,, Rprnay] -(1 2daM1
max Bprrd [LR] = max
2dx
1
E2 Mi [Rroay)
max
+ 2rr [Rprux) + 223
2Ag
Now cancel 2\g in numerator and denominator:
E2 [Rrroy]]
max E LR = max E,,, RRroxyy
- x2) +
2As
V
Now plug in the expression for Ag:
x2 E2 Rproxy
1
A
2 VVI
This gives the final outer problem for the original max- max-min objective in Equation 15
V. E,,, [Rpoxyy - V. v1 -1-/" 22. v x2 |l Mana)) E2 [Rprosy] M
(27)
E.2 PROOF OF OPTIMALITY
Recall the inner minimization problem of max-min objective in Equation 17
min
E
[L R
RE
where L pzz (8, a) Mr (8,a) is treated as fixed, and the feasible set is:
Roo = { R (8, + Eu [[R - (R-M))- Reroxy] = r V,
E Mend [R]
M,
E Miat [R2]] = V2 + M
The feasible region not convex due the quadratic equality constraint Ep R2] V2 M This defines the boundary of an L ball (a hypersphere) function space, which is not convex. Therefore, traditional convex programming tools and strong duality do not directly apply.
However, we still claim that the resulting R. derived in Appendix E. is globally optimal. This is supported by the following facts:
26

---

Published as a conference paper at ICLR 2026
Stationarity. When considering R for any fixed dual variables Å1 A2, As, looking the inner minimization problem in Equation 18 as follows:
min 1o(A1; A2, Λ3, R) RER
The term with R(s, a) in lo is:
R R- M
En LL-R---- R-
Rproxy A2Rl AzR2
For this quadratic in R to have minimum (since it is minimization problem for R), the coefficient of I² must be positive.
our case, the coefficient is -A3. Therefore, for the minimization problem to be well-posed and have finite minimum, quadratic term in R is concave upward parabola, which means that minimum exists. Moreover, in Appendix E.1 we explicitly state that R (s, a) is derived by setting the derivative of the Lagrangian function to in Equation 18 to zero with respect to R(s,a) a). Thus, R2 (s, a) is indeed the optimal value for the minimization problem for fixed Λ1.d2, A2 Å3 where X3 < O. The Stationarity in this context implies that R lies within the domain where the Lagrangian is well-defined and differentiable, which it does.
must have X < O. This condition ensures that the
Feasibility. We also argue that the closed-form primal solution (X*),, where X denotes the optimal dual solution, is feasible the original sense, that is, satisfies the three equality constraints in the feasible set Reor- Specifically, detailed in Appendix [E.I] we substitute R back into the dual objective lo and compute the gradient with respect to each dual variable. We then solve:
Olo(A1, X2 X3, R' (Az, A2 As))
O, for 1,2,3,
to find the optimal values ii, X2, Ag.
By the chain rule, we have:
Alold1, 22, 23, (21, As))
Öly OR" 0x,
Olp
+
OR
where the first term vanishes because R chosen to minimize lu for fixed À (i.e 0l/OR at R*). Therefore, the derivative simplifies to:
Ölg Ma(AA.A2.
A2, Ag R³)
Og 0);
Setting these derivatives to zero yields:
Olg
-r ax Ming (RR" -M) Rproxy] rV = 0,
Ølp
Qgg Mew [R" + M =0
Ølo
Eyraa [(+*))]+ V2 M2 =0
which exactly recover the original feasibility constraints. Hence, the solution R2 (X*) is feasible by construction.
Therefore, Rº satisfies both stationarity and feasibility. In general, stationarity and feasibility are sufficient for global optimality when the feasible set nonconvex. our case, however, global optimality does hold, relying on the specific structure of the inner problem.
Recall the inner minimization problem discussed above, and work in the Hilbert space H. L (P Using the normalization assumptions: Epzu [Rprxy]] O and Ep [Rproxy] = 1, the constraints can be rewritten as inner products H:
RR 1) M (mean constraint)
(R, Rpmoxy) rV (correlation constraint)
27

---

Published as a conference paper at ICLR 2026
| V2 M² (norm constraint).
Let fev, e c.....
} be an orthonormal basis of 7/, where
eg is proportional to the constant function of 1
E1 Reroxy.
and f22, spans the orthogonal complement of span (1, Rproxy)
en and ej orthonormal because Epinas [proxy]
0. Expanding
R apea + 0161 +
Σ
ag.k
R22
Notice that the mean constraint and correlation constraints uniquely fix og and 01. The norm con- straint then forces:
Σaz=1
for some constant radius p 0. Hence the remaining degrees of freedom lie sphere in the subspace orthogonal to and Rprexy- This is to say, although Reor is not convex in the ambient space, it is a spherical manifold (the boundary of an -all intersected with an affine subspace), which is compact and smooth. Moreover, the objective is linear in R:
Ex [L. R] (L, R) const (L',
where the projection of Fx onto the subspace spanned by {ex) and R' Exzr agek. Therefore the optimization reduces to
Mew
min RL 2=p
This is simply minimizing linear function over Euclidean sphere. In this setting, is well-known that the only stationary points of a linear functional sphere are its global maximum and global minimum. There are no other local minima or saddle points. Thus, on this particular nonconvex feasible set, any feasible stationary point is automatically global optimizer.
In summary, our previous analysis shows that:
1. For fixed (A1, A2; Aς) with λ < O, the Lagrangian strictly convex quadratic in R, so its stationary point R (A) is the unique global minimizer of the inner problem with those multipliers.
2. Solving the dual and enforcing feasibility recovers the specific choice of multipliers X for which R' (X*) lies on the sphere defined by the norm constraint.
3. Because the reduced problem is linear over sphere, this feasible stationary point Rt (X*) must be the global minimizer of the original inner problem.
E.3 PROOF THAT (p || (T ) 2
Hr [PRRxyy]
To ensure that the inner term of the square root in Equation 25 remains non-negative, need to show that
x² (1r (1 2 E [Rproxy]
Proof. Recall that
E [Rproyy] = E E we [L. Rpeoyy,
where L(s Ms (s.0) is the Radon-Nikodie derivative. Since Rproxy is normalized to have zero Me (s,a)
mean under Preee we have:
E. [Rroayy
0.
28

---

Published as a conference paper at ICLR 2026
Thus,
Bym [Rroxy] Ry (Rroay (8, ()) -1]
Σ
Rproxy a) (8, 1)
(0,0)
Applying the Cauchy-Schwarz inequality:
Rexoxy( Rpoxy(s, (5) (s:a)
Σ
Σ
Hsw (8, a) R (Proay((,,))
Σ
Msner (I) 1)2
(0,0)
(o,a)
By the assumptions: Ep [1 = 1, Eisa) Mard (8, (L,,))((ssa) 1)2 = xx2(ux Mmmd)
We obtain:
E M4 (roocy) S x2 (1 (aa))
as desired.
E.4 DERIVE LAGRANGIAN FUNCTIONAL FOR LINEAR MAX-MII MIN OBJECTIVE
Recall that our max-min optimization under the structured reward assumption as follows:
max
min OE
E (ssa))= (T)((s.))
(28)
020
where Rlin is the uncertainty set defined as follow:
{0 -R-M
Br 0 TT
Φb Rymxxy] T, E pe 10 [T] = 0, E pn [00 [b)) - =1].
(29)
We assume without loss of generality that the worst-case reward R(s, = 0 TT (fs a) is normalized setting M
have zero mean and unit variance under the reference policy Tref- This corresponds 0 and V = 1, which, shown in our earlier derivation, does not affect the resulting optimal policy. As before, Rproxy denotes the normalized proxy reward under Tie, satisfying E,,,.. [Rprosy] = 0 and Varp= p [Rprxyy] =1 1.
Similar to previous steps, we introduce the Radon-Nikodym derivative
(8, a) PT 0.)
M88 a)
We use a -f-aauur, and define the Lagrangian functional for the inner minimization in Equation 28 as:
(A1, A2, As, 0) Eppnr OTT ~11 (Berhe [Reruyy [ -r)
X2E [ØT - >a EEpre ((TT)) -1
Eppay [[L d1 Rproxy - \2) Φ} >3g p= [T]} + Air + Aa
Σ Mana (s, [(T. >1 Rproxy - X2) Φ (0 [TT) Air + Aa
(s.a)
Define the following terms for simplicity:
u(s, a) = ( (s, a)
D(s, a) Pro (s,a) Reroxy (s,a)
C(s, a) = (s. a.
1,,)2 (8, v(s, a) A1 D(s, 22
29

---

Published as a conference paper at ICLR 2026
Then the Lagrangian function simplifies to:
(1i,, A2, A3, 0) = Σ
(8, Φ(s,a) (s0)),,,00(s(00)
AaC(s,,)(0' ((s,a))² + Air A
(5,0)
TT
Σ
(s, - O30
C(,.0)0(s. + - (8.0)
Σ
(3:a)
where expand the quadratic term:
Σ C(s a(s,a))
Σ
((s
(8.a)
(s,a)
(s,a)
T
Σ
C(s..))( 0
(8,a)
Let
Q= Σ ((s(.))(s.0)(((..))
(30)
(s,a)
then write the Lagrangian function
4 (A1,,22, 0) = TT
Σ (s.0)
>30T00 + Air + Aa
And the inner minimization problem in Equation 28 becomes:
max min 4 (Ai, X2 X3, R) A1, 020
(31)
E.5 PROOF FOR WHITENING TRANSFORMATION
To simplify the problem associated with the Lagrangian function above, we transform the feature vector into whitened version & such that the matrix defined in Equation 30 becomes the identity matrix I. Specifically, we perform a whitening transformation using the Cholesky decom- position (Boyd & Vandenherghe 2004). Let
W =
Q
((. a) = a))
where Q denotes matrix square root of Q Then have:
Σ (3,0)
((s,))((s.
cls.))
(+.0)
C(s, Cls
(x.0)
w
((s)s))(ss.)))(s.)) WT
(s.a)
WQWT
Q-iQQ-+
desired.
30

---

Published as a conference paper at ICLR 2026
Note that the whitening step requires to be invertible that Q-1/2 (and hence Q=1) exists. holds when Q is positive remi-definite and non-singular. Q is positive semi-definite since it is a sum of outer products d( a)o(s, a) weighted by non-negative coefficients (occupancy measure of Tee 2 0). For Q to he non-singular, it is necessary that the span of {Ø(8, a) (s, a) 0} covers R" i.e., the features associated with state-action pairs visited by aer must span the full feature space. To achieve these conditions, the reference policy should visit a diverse and representative subset of the state-action space with non-trivial occupancy. This is more likely when Tee derived from ei- ther expert demonstrations that exhibit rich behavior or from stochastic exploratory policies (e.g., entropy- reguua-ergul policies or policies trained with exploration bonuses). Moreover, the feature mapping Ø(s, a) must exhibit sufficient variation across the visited state-action pairs. This typically holds when @ encodes task-relevant dynamics (e.g., learned embeddings or expressive hand-crafted features) and when Tre does not collapse to trivial or deterministic behavior. In our experiments (Appendix F.44) the reference policies for the Traffic and Pandemic environments are trained via behavioral cloning trollers. The feature representations used in these environments, such as velocity, acceleration, and headway in Traffic, and infection level, disease stage, and smooth transitions in Pandemic, encode meaningful task-relevant dynamics. These demonstrations cover a wide range of task-relevant be- haviors, and the induced occupancy over state-action pairs spans a high-dimensional subspace the feature space. We empirically verified that the resulting Q matrices in our experiments are full-rank and numerically well-conditioned. Though ensuring sufficient coverage of the feature space by the reference policy is generally challenging in practice.
1/2
large, diverse trajectories generated by human experts or hand-crafted con-
E.6 DERIVE OPTIMAL PRIMAL VARIABLE FOR LINEAR MAX-MIN OBJECTIVE
After whitening transformation as discussed in Appendix E.5
the problem in Equation
31 becomes:
max min 42, 0) = ØT As 820
Σ (s.a)
(xn__((.0)((ss.0) A Air + Ag.
(32)
where we now optimize over the parameter Ô using the transformed features C. For notational simplicity, we will drop the tilde and henceforth use Φ to represent the whitened feature C. and @ to represent the whitened weights O.
Separable Structure. In the whitened feature space, the objective becomes separable across ordinates of 0. Thus, the inner minimization problem in Equation 32 decouples into M independent one-dimensional convex minimization problems, one for each feature coordinate E {1, ................................ M
min 220
Σ ux (8, (s) (8.0)
Xa09
Let us solve the i-th subproblem. Assuming Àg < 0. the objective is a convex quadratic function in 0, (an upward-opening parabola), The unconstrained minimum occurs at:
0, E(s.a) uA Aa (s, (s,
2
Considering the constraint 0, 2 0, we have two cases:
If the unconstrained minimum 0: 2 0, then also the solution to the constrained prob- lem.
If 0 < 0. then the constrained minimum occurs at the boundary 0, 0.
Thus, the final optimal 07 is:
E(ea) uA (8, a))
0 max
0,
2)g
31

---

Published as a conference paper at ICLR 2026
Collecting across all we express the final optimal solution 0 as:
E(s.a) (s, (ss,))
max 0,
(33)
2)3
where the max(:;0) 0) is applied elementwise.
E.7 SOLVE THE DUAL OBJECTIVE FOR LINEAR MAX-MIN OBJECTIVE
Let the outer objective in Equation 32 be:
g 22, As) 1 (A1, A2, A3, 0*)
Then we want to solve the following dual objective:
max g(A1. A3) A1.A2:AB
(34)
Let
(A A2) = (v(s,a)- A1D(s, - x2)(s, (5)
(0,0)
denote the linear coefficient for each feature {!....; The optimal 0j is:
(A()) max 0, g,(A2, A2)
2
Now, we compute the gradients:
Gradient with respect to A2:
0g
0h
(A)
(A, (A))
ax
Oxi
Σ
(DD(s,)) o(s,
(s:a)
=r--
Σ 1=1
Das (A)
where
Doj = Σ
D(s,a)os(s,0)
(+.0)
Gradient with respect to Az:
0g (A)
0h
(X, 02))
O22
Σ
(-Ci(s, (-C(ss ((s,a))
(a.)
M
Ca.. (7(
where
Coj Σ
C(s, ((s,a)
(s.a)
32

---

Published as a conference paper at ICLR 2026
Gradient with respect to Az:
åg
01
(A)
(x, (2))
aλ3
0g3
Σ ((()s,)((0"((ss,)))
(8,0)
M
Σ (1))
j=1
where we use the whitening assumption (s,,) ((s.a)o(s.c))ps.d) =
Thus, the full gradients are:
0g
M
Dan max (, (,(A1,22)
(A) = Σ
0λ1
213
j=1
0g (A) = Σ
(2),,
Co.
max
0λ2
2d3
j=1
0g
4 (A1. A2)
(A) = 1
max 0,
2d3
We solve for the optimal dual variables A2, As) using standard first-order optimization methods. Since g(A) is concave (under the condition A3 < 0), optimization is well-behaved and converges reliably. After obtaining the optimal primal variables 0* and dual variables (Ai, Aÿ. Aï). we can substitute them back into Equation 28 and solve the outer maximization over the policy using standard reinforcement learning algorithms, such PPO (Schulman al. 2017
E.8 POLICY GRADIENT DERIVATION
We now derive the gradient of the robust objective equation 27 with respect to the policy parameters Recall that the robust objective is:
I ((PTo) = r- V EEar, [Rprayy] - V. V p2 V (2= ll (maa - (Eyaao (Rpocsss) )² + M
2
-Ep Rproxy]
x² (1 ll (mma)
(Eyiro [proce)
where we M = and V 1 without loss of generality. We also divide the entire objective by r, which is assumed to be positive (r > 0), so this rescaling preserves the optimization direction and does not affect the final policy solution. The x² divergence is defined as:
Mi (8,a)² -1
x2 (M ll (1m)) = Σ
Ma (s,a)
(s.a)
Applying the chain rule, we compute:
VI
VoJ = VoE [Rroay]]
Vo (v (((MA)))
(35)
where we define:
h(Mao)) = =x=2=xxxx (Hoo |l (ara) - [Ep [Rpoxxs) )
Using the chain rule again:
1
V Vov h(Hma))
V voh(Pos)
2 2/hhlux
33

---

Published as a conference paper at ICLR 2026
Now compute Voh(Hzx)):
Voh(u=)) = Vox 2 (Wis ll Maxe) Vo (Eung (Rpoxyy?)
Mro (8, a)² -
V
Σ
Bun (proxy) Vo Ey Bproxy
Movel (8, (i)
(x,0)
The individual terms are:
Umo (8, aa2
Uso (8.
V
Σ
-1
Vopuro (s, a)
We (8, a)
Man ( (3,a)
(8,0)
(x.0)
Vele Bon (Rooxyy
V Bll (8, a) Rproxy(s
(s,a)
Thus:
Volta (8, a) ( Mao (8, a)
Vah(pm)) =
2Ey Rgrosy Reroxy(s,)
M (8.
(5.0)
Then we compute V Bypug RRpoxy]:
Vo Rpruxy] = Vo Σ
Mxo (s, a) Reanxyss a)
(s.0)
VollTo (s) Rynuuy (8, a)
(s.a)
Put them together, we get the final gradient in Equation 35 as:
VoJ = V VOlp pig [Rroxy]
p
Vo (Vh(μmo)
VollTo (8, a) Rgnoay (8,a) v1
p2
1
Voh(HTT)
2 (Vh(um))
(e.a)
V ol (8, a)) Rproxy (8, ())
(e.0)
v1 pd
V Vol (8,0) 2 Moo (8. a) (Mtar(800)
21 [Rroxy] Rerouy a)
2 2/h (((MTT)
(s.a)
pd
Mou MTo(s,
offora (s)
Rproxy
EEHr, [Raaxyy Reroxy (s. (s.
Vh(uar))
Mora (5, a)
(8.0)
(36)
The full policy gradient for the ORPO algorithm, as presented in Appendix of (Laidlaw et al. 2025), is given by:
Muo (8, a) (37)
Volto a) Rymny(( V ( (mma)
Pm (s, ()
(x.0)
Interpretation.
The policy gradient consists of two terms:
A standard term encouraging the policy to increase Rproxy a).
A correction term that penalizes deviations from the reference occupancy Mnar while also adjusting for alignment with the proxy reward.
34

---

Published as a conference paper at ICLR 2026
This correction enforces robustness to potential reward hacking by optimizing against adversarially misaligned interpretations of the proxy reward.
Notice that derived policy gradient in Equation 36 shares structural similarities with ORPO but rooted in formal robust optimization framework. Unlike ORPO, our formulation introduces an additional correction term involving both the occupancy ratio and the expected proxy reward, capturing how the proxy aligned with the current policy's 's behavior. This structure more explicitly penalizes the combination of distributional shift and proxy overoptimization, discouraging policies from exploiting proxy-specific artifacts. Both methods share the goal of improving robustness, but our approach is derived from first principles by directly optimizing for worst-case performance over correlationconstrainge constrained uncertainty set.
a correlation
E.9
PROOF OF THEOREM
Proof. For any reward function R. define the performance difference
AJ J R) [ R).
By definition of the correlated uncertainty set, our distributionally robust objective considers
F(T) == min RER.
AJ(
Under the assumptions the correlation, mean, and variance of rewards in Reors, Equation 27 provides closed-form expression for this inner minimum. In particular, for any policy л with (r Man Equation 27 gives
F(r) = r pr, [Rrrxy] v1 - Vxe(mn | Mxxe - Min [Rprxy]]-
Now assume that the true reward Riree lies in Rear- Since Rinee is one feasible element of the uncertainty set, we must have
F(x) min AJ(T, R) S AT, Rmme) J [TT, Rume) J (Tref, Rune). RER.om
Rearranging yields
J(r Ruse) [[Tert Rucee
F(T),
and substituting the explicit form of F(T) from Equation 27 gives the claimed inequality.
F ADDITIONAL IMPLEMENTATION DETAILS
F.1 TRAINING DISCRIMINATOR NETWORK
core step our Max-Min optimization algorithm and ORPO is to estimate the Madon-Ni
derivative L(s, a), which is critical for computing the x2 divergence, as detailed in Appendix F2 To this end, we follow prior works (Laidlaw et al., 2025 Kang et al. 2018 Ho & Ermon 2016 and train discriminator network. Specifically, reference policy ret and another batch D, from the current policy æ. The batch sizes used for each discriminator architecture identical that in (Laidlaw et al. 2025). denoted by (8 a), which is optimized according to:
sample batch of trajectories D from the
specified in Table - And then
arg min Eppn log(1 1 + e dal(sa)) E,c log((1 + e -d(()a))
arg min ED [lo AelleeA4ooe (o(,)))) +ED, llg([ + do(2))))]
(38)
is known that the optimal discriminator satisfies d' (s, a) log
Pa (s.a) and we estimate L(s, a) pr (5,0)
as i(s,a) = dols.a) with (s) x d' (s. However, in the original ORPO implementation
35

---

Published as a conference paper at ICLR 2026
the discriminator is not fully optimized during policy learning. Specifically, the discriminator ceives only a small number gradient updates per reinforcement learning iteration, resulting in underfitting and inaccurate estimates of the Radon- Radon-Nikodym derivative L(s
This undertraining is evident in Figure 2 which shows the discriminator loss across RL iterations. The loss remains nearly constant (e.g., around 1.4 in the Traffic environment, which is the initial loss value as shown in Figure indicating that the discriminator is not learning effectively. This limits its ability distinguish between and reef especially for state-action pairs where their occupancy distributions diverge.
To address this, substantially increase the number of gradient updates per iteration and carefully tune the learning rate. Our goal is to strike practical balance between training time and discrimi- nator quality: while fully training the discriminator to convergence each iteration computationally expensive, insufficient training leads to inaccurate divergence estimates and unstable optimization.
Figure 3 shows that in our implementation, the discriminator loss consistently decreases within each iteration, e.g., from an initial value around 1.4 to below 0.2 in the Traffic environment, indicating effective optimization and more accurate occupancy-ratic estimation. In the Glucose and Pandemic environments, however, we observe that training the discriminator for too long leads to slower con- vergence and little improvement in loss. In these cases, we apply early stopping to limit training time. The specific training schedules provided in Table 2
Discriminator
1.4
12:
12-
N
1.-
11
0.6
10-
044
0.4
(a) Traffic
(b) Glucose
(c) Pandemic
Figure 2: Discriminator loss across RL iterations the original ORPO implementation, The loss stays flat and high (~1.4 1.4 for the traffic environment), indicating the discriminator is not adequately trained. This undermines the accuracy of the estimated occupancy ratios.
opoch
one epoch
epoch
1.4
12
1.382
1.56
0.8
1.38
0.6
0.4
3.376
1.8
(a) Traffic
(b) Glucose
(c) Pandemic
Figure 3: Discriminator loss over training steps within each RL iteration our implementation. The loss decreases rapidly from its initial value (e.g., 1.4 to values near 0.2 in the Traffic environment), indicating successful training and improved accuracy of occupancy-ratio estimates.
As for the discriminator network architecture, we follow the same structure described in (Laidlaw et al. 2025). For each environment, employ fully connected neural network with two hidden layers, each consisting 256 units and ReLU activations. Table summarizes the hyperparameters used for discriminator training across different environments.
36

---

Published as a conference paper at ICLR 2026
Table 2: Hyperparameters used for discriminator network training across different environments.
Hyperparameter
Traffic
Glucose
Pandemic
Learning rate
5 x 10
x 10-2
5 x 10-4
SGD epochs per iteration
200
20
15
Batch size
40000
100000
3860
SGD minibatch size
16384
1024
64
F.2 DERIVATION OF MAX-MIN POLICY OPTIMIZATION
Using the estimated (s, a) from trained disciminator as discussed Appendix F.1 pute the x² divergence via:
we can com-
Me (s a) Maa (8.a)
(2 (TT)) Ey,,
~ En,
(da(ssa)_
-
(39)
For environments where both state and action spaces are discrete, we directly estimate the occupancy measure via empirical sampling. Specifically, given the same batch of trajectories collected from policy ı (as used for training the discriminator), approximate the discounted occupancy measure as follows (Schlaginhaufen Kamgarpour 2023; Abbeel & Ng 2004):
(=
ΣΣ
(T( 8, = a),
(40)
where I{-} is the indicator function. Using this empirical estimate, we can compute the Radon- Nikodym derivative and x² divergence without training discriminator.
In our formulation, we assume that the proxy reward is normalized with respect to the reference policy Tet. To achieve this, we reuse the same batch of trajectories D sampled from Ted to estimate the expected return J [TTtt, Rpeoxy) using:
[Tref Rproxy) = (1 7)
Σ Rproxy (7)))
(41)
where each (8) = (su, ag: 81, a1 8T) ~ DTTer a trajectory sampled from Tee is the number of sampled trajectories, and (00) Σt 7" (s)) (4) (())). This estimation is unbiased
when using trajectories generated by the policy л. To estimate the empirical variance of the proxy reward, use:
E2
Rproxy]
Mm [R
(42)
Mazt
estimate É By [R using:
Paint R2
Piroxy = -
N Re (7())
where R2 (T = Σt P
MTm [Reowyy directly from single batch introduces bias, because the square of an empirical mean not an unbiased estimator of
( (8 di (.) However, estimating B2
the square of the true mean. To obtain an unbiased estimate of E2 of [Rproy], we apply the double-
sampling technique (Di Castro et al. 2012
Xie et al. 2018). Specifically, we independently sample from the reference policy Neei and compute:
another batch of trajectories, denoted D "iet"
22 [Rprxy]]
D Paol
[Rproxy] x E Ep [Rroxy]
Di
Fad [Rproxy] denote the empirical returns computed from the two inde- pendent batches using Equation [4] This ensures an unbiased estimation of E,, [Proxy]s which is critical for correctly computing the regularization term in the objective 27
Dr RRposy] and E where Ey
37

---

Published as a conference paper at ICLR 2026
We then normalize the proxy reward for each state-action pair in Drr as:
Reroxy (s, a)
[TTee R Rproxy)
Aprayy (s, a)
(43)
For notational simplicity, we will continue to use Rerexy to denote the normalized proxy reward throughout the remainder of this section.
We use the same batch of sampled trajectories Dx from current policy ı to estimate E,,, [ Rpoxy] using:
E, [prxsy] J(T, Rproxy) (1 1
Reroxy (T(1)
(44)
where each 7(1) (So, du. $1, 01, S1) Dx is trajectory sampled from To estimate E2 [Raoay], apply the same double-sampling technique. Specifically, independently sam- ple another batches trajectories, denoted D from the current policy x, and compute:
22
He Rprosy] = (R [Rpxxy] x Epp [Rpuxy)
(45)
[ where D= [Rroyy and Bp: R Ryyyy] denote the empirical returns computed from the two indepen- dent batches using Equation
Putting all the steps together, the maxmin algorithm is in Algorithm 2
Algorithm Max-Min Policy Optimization
1: Initialize policy parameters
2: Initialize reference policy uet and collect trajectories Dint
3: Estimate J (Tret, Aprexy) using Equation 41 and o
using Equation 42
4: for each iteration do
5: Sample trajectories Dar from current policy T
6: if discrete environment then
7: Estimate occupancy measure using Equation
40
8: else
9:
Train discriminator dæ by minimizing Equation end if
38
10:
11:
Estimate x² divergence using Equation 39
Normalize proxy reward for each state-action pair in D, using Equation 43
13:
Estimate proxy reward expectation E,,, [/proyy] using Equation 44
14:
Estimate E2 [Rpray] via double-sampling using Equation 45
15:
Update policy ωe using PPO to maximize robust objective Equation 27
16:
end for
F.3 DERIVATION OF LINEAR MAX-MIN POLICY OPTIMIZATION
As for the Linear Max-Min optimization problem, following the discussion in E.4 E.7 we first need to estimate Q:
Q Σ ((s.)))(s.ajó(s.a))
(s,a)
where C(s, (s. ((s, a) is vector of known feature functions sampled under the current policy To recover the reference occupancy Mees, define
MT (8, a)
ds(s,a) log
Mb (s)
we can rewrite Q via importance sampling (Sutton & Barto 2018):
Q = E,,,
c do((s.a) (),,5))
38

---

Published as a conference paper at ICLR 2026
Note that dø(s,a) a) differs slightly from dø(s, a) log (rgg(s (5,0) used in the Max-Min optimization
ne (=(0)))
algorithm discussed in Appendix F.2 where the numerator and denominator are reversed. To es-
timate ds,sa)). a). we again train similar discriminator network described in Appendix F.1 by minimizing:
arg min Bp log(1 + do(o,a))) + E,t- [o + edo().0))]
arg min ED [oo te ) + ED, [log(1 + cdo(sa))]
(46)
At optimality, the discriminator satisfies:
Mm (s, a)
(s, log M4 (8,
And we use (s( a) Re d' (s, We then estimate Q:
Q (1 - ED,
(47)
We then perform feature whitening by applying a linear transformation:
((,,a) a) a)
where W- Q-1
1/2 is the matrix square root inverse of Q
All subsequent quantities are computed using the transformed features Ø. As before, we also nor- malize the proxy reward for each state-action pair in D using Equation 43
After whitening, need to estimate the gradient of each dual variables (A1 A2, Xy) as derived in Appendix E.7
Estimating Co. and Do.j Recall that
Caj Σ ((s,a)o,(s,a)
(s,a)
which can be rewritten via importance sampling as:
Co.j = Eys, (
and then can be approximated via:
2
Čo. (1-
l 2
Similarly, recall that
Do.j E (s,,5)
(#.0)
where D(s, a) prm (s) Rproxy (s, a). Using importance sampling, we can write:
D&j Eps de Rpooxy (8, (s,)
and can be approximated in practice by:
Da., (1 - (1-7)Ep,
* dalonae) Raroxy (st, as (5) ax)
1=0
39

---

Published as a conference paper at ICLR 2026
Estimating 93 (A1: A2).
Recall that
(;(A1,
Σ
(o(8,a) x D8,,a) (
(s.a)
Σ ((s,,)) - A1 E ((s,a) - X2 E
((s,0)0,(+,0))
(s.a)
(s,a)
(s.a)
E (oy a)] DD X=Co.
where u(s.a) (s a) and Ey, (0),(s, a)] is the discounted feature expectation under the policy и. We can estimate the first term using:
1 N Ep (1- N
= r)
(F
where each =(1) (so, dg, 81, 01 o,,, ST) D is trajectory sampled from n. and oy(+(1)
Elzo (5()
(1) (0)) Given the above estimates, we can finally compute:
(A1, A2) = pr, (8, Do - A2Co..
With the above estimation, we can compute the gradient and solve for the optimal dual variables = (A1, A2, Aς) using the Wyvenberg-Marquaal method designed for solving nonlinear systems of equations. Specifically,
algorithm (Moré. 1978), a damped least-squares the root solver in
SciPy (Virtanen et al. 2020) to find the stationary point of the gradient AAg(A) = 0. We initialize the optimization with A 0, A2 O, and λ -1, and enforce A 0 throughout training to ensure concavity of the dual objective g()). To enforce the non-negativity constraint O 2
as required by the analytical form Equation 33 we manually clip each 0 ensure it remains non-negative. Future work may explore alternative solvers better suited to constrained optimization.
Recall that the optimal primal variable 07 is:
(A) max (0,
Li (A1,
2)x
After optimizing for the dual variables, we can substitute the optimal 12 back into the above equation and get:
(A( max (o. T (2)
2X
for all features. Then we can substitute the optimal 0- back in the robust reward objective in Equa- tion 28 and train the policy æ to maximize the outer problem using the standard reinforcement learning algorithm proximal policy optimization (PPO) (Schulman et al. 2017).
Putting all the steps together, the linear maxmin algorithm is in Algorithm
F.4 ENVIRONMENT DESCRIPTION AND REWARD HACKING TYPES
Traffic. This environment simulates highway merging scenario, adapted from (Pan al. 2022 Wu et al.) 2021] Vinitsky et al.) 2018). where group autonomous vehicles (AVs) controlled by an RL agent must merge into human-driven traffic. Each AV observes its own state (position and velocity) and those of nearby vehicles, and outputs continuous acceleration actions. The true reward designed to ensure smooth and efficient traffic flow, encouraging low commute times and gentle accelerations. The reference policy Nee is behavioral cloning (BC) policy trained on demonstrations generated by the Intelligent Driver Model (IDM) (Treiber et al.) (2000).
Pandemic. Based on the PandemicSimulator (Kompella et al. 2020). this environment models infection dynamics using an extended SEIR model. At each timestep, the agent selects a lockdown policy to control the spread of disease while minimizing societal costs. The true reward balances infection severity, political disruption, and policy smoothness over time. The reference policy is trained via behavioral cloning mixture of realistic and hand-crafted policy trajectories.
40

---

Published as a conference paper at ICLR 2026
Algorithm 3 Linear Max-Min Policy Optimization
1: Initialize policy parameters 0
2: Initialize reference policy wer and collect trajectories Dat
3: Estimate J (Tttet Rerexy) using Equation 41 and B using Equation 42
4: for each iteration do
5: Sample trajectories D from current policy Tg
6: if discrete environment then
7:
Estimate occupancy measure using Equation 40
8: else
9:
Train discriminator dø by minimizing Equation 46
end if
11:
Normalize proxy reward for each state-action pair in D using Equation 43
12:
Estimate Q using Equation 47
13:
Compute feature transformation W = Q- and transform features (6s a) Wo(ssa)
14:
Estimate Co.j. Deg using transformed features
15:
Compute & (A1, A2) for all features
16:
Solve for optimal dual variables (21, by maximizing the dual objective (Equation 34
17:
Compute the optimal primal variable 07 for all features
18:
Update policy xe using PPO to maximize the robust objective in Equation 28
19:
end for
Glucose Monitoring. This environment uses the SimGlucose simulator (Man et al. 2014 Fox et al. 2020), where RL agent administers insulin doses to simulated patient with Type I Dia- betes. The goal is to maintain safe blood glucose levels and minimize long-term health risk. The reference policy is trained via behavioral cloning using data generated by PID controller with clin- ically tuned parameters (Steil 2013). Proxy rewards in this setting often reflect surrogate objectives such treatment cost or patient burden.
Tomato Watering GridWorld. This environment presents a simple spatial grid where the agent waters tomato plants. The true reward corresponds to the number of tomatoes correctly watered. However, the proxy reward includes artificially high bonus at specific grid location (a "sprinkler state"), which causes the agent to overfit by remaining in that region despite little actual benefit to overall tomato growth. The reference policy follows (Laidlaw al. 2025), with 10% random actions added to allow for policy improvement.
RLHE This environment builds on prior work (Laidlaw et al. 2025 Coste et al.) that studies overoptimization of LLM-bbsed reward models. The proxy reward function is derived from reward model fine-tuned on the AlpacaFarm preference dataset (Dubois et al. 2023). using the Pythia-70M model (Biderman al. 2023), comparatively small model. For the true reward signal, adopt the Llama 3 Tulu V2 8B reward model released by A12 (Ivison et al., 2024). The reference policy corresponds the supervised fine-tuned (SFT) model from (Laidlaw et al. 2025 Coste et al.), which was trained on the AlpacaFam SFT dataset using Pythia-1.4B.
Types of Reward Hacking. We adopt the taxonomy proposed in (Pan et al. kinds of proxy reward misalignments that lead to reward hacking. Our selected environments span all three major categories:
2022 ) to classify the
Misweighting:
The proxy reward includes all relevant objectives but uses incorrect relative weights. Our Linear Max-Min method specifically seeks the most adversarial weighting in this space.
Ontological:
The proxy captures the correct high-level goal using different or incomplete features. In the Traffic environment, the true reward combines commute time, acceleration, and headway, whereas the proxy replaces commute time with velocity. In the Pandemic environment, the true reward penalizes infections, political cost, lower stage changes, and non-smooth policies, while the proxy omits the political cost entirely. Similarly, in Glu-
41

---

Published as a conference paper at ICLR 2026
cose, the proxy reward only considers the expected patient costs while the true reward only measures the health risk.
The proxy evaluates behavior over a limited domain. In the Tomato environment, the true reward reflects the number of tomatoes successfully watered. However, the proxy introduces a large bonus at a specific state (the sprinkler), incentivizing the agent to pursue this location at the expense of fulfilling the intended watering task. In the RLHF environ- ment, the proxy reward is produced by a comparatively small model with limited evaluative capacity, whereas the true reward is derived from much larger, stronger model. Conse- quently, the proxy reward provides a less reliable evaluation signal.
Scope:
F.5 ADDITIONAL EXPERIMENT SETUP
Non-LLM Experiments. For the policy networks, follow the architectures described in (Laid- law et al. 2025). In the Pandemic, Traffic, and Tomato environments, we use fully connected neural networks with 2 layers of 128 units, 4 layers of 512 units, and 4 layers of 512 units, respectively. For the Glucose environment, we employ a three-layer LSTM network, where each LSTM layer has 64 units. We use the pre-trained policies provided in the ORPO repository as the reference policies Tee. We initialize the policy network with the corresponding pre-trained checkpoint for the Traffic, Glucose, and Pandemic environments, and initialize a random policy for the Tomato environment. Table[3ssumman the hyperparameters used for PPO training across all models and environments.
Table 3: Hyperparameters used for PPO training across different environments.
Hyperparameter
Traffic
Glucose
Pandemic
Tomato
Training iterations
250
500
260
500
Batch size
40000
100000
3860
3000
Optimizer
Adam
Adam
Adam
Adam
Learning
5 10
1
10-5
0.0003
1
10-3
Gradient clipping
N/A
10
10
0.1
Discount factor (y)
0.99
0.99
0.99
0.99
Random seed
0
0
0
GAE coefficient (A)
0.97
0.98
0.95
0.98
Entropy coefficient (start)
0.01
0.01
0.1
0.01
Entropy coefficient (end)
0.01
0.01
0.01
0.01
KL target
0.02
1 x 10
0.01
1 1 10-3
Value function loss clipping
10000
100
20
10
Value function loss coefficient
0.5
0.0001
0.5
0.1
Share value function layers
True
True
True
False
As for the reward used during training and evaluation, we follow the same setup as ORPO (Laidlaw et al. 2025). ) All policies trained using only the proxy reward, while both the true and proxy rewards are used for evaluation.
In the Traffic environment, the proxy reward is weighted combination of velocity, acceleration, and headway, with weights 1, 1, 0.1, respectively. The true reward, on the other hand, uses commute time, acceleration, and headway, also weighted 1, 1, and 0.1.
In the Pandemic environment, the proxy reward is composed of infection summary absolute, lower stage, and smooth stage changes, with weights 10, 0.1, and 0.01. The true reward adds a political component to these three features and is weighted with 10.
For the Glucose environment, the proxy reward includes only feature: expected patient cost.
The true reward is based on mgg which measures the health risk of the patient.
/ / wttp:ttth......cco..ooooooooooee http::/:hhhhhaaaooaaaaaaaaaattte checkpoints nts
42

---

Published as a conference paper at ICLR 2026
In the Tomato environment, the true reward counts the number of watered tomato. The proxy reward adds a large bonus at a specific state (sprinkler), incentivizing the agent to reach that location regardless of its impact on the primary task.
LLM Experiments. We adopt the common formulation for RLHF as a contextual bandit problem, where the environment is modeled as a Markov Decision Process (MDP) with a discount factor 0. In this setup, the return of a policy π under a given reward function R simplifies to:
[ R) Earpo
[R(s, a)];
where Ao denotes the distribution over initial states. In the context of RLHF, each state corresponds to prompt sampled from a dataset, and the action is the model's s generated response. The reward then computed based on this prompt-response pair.
Under this contextual bandit assumption, the x² divergence between occupancy measures reduces to the divergence between action distributions conditioned on prompts:
x2 (M (Txe) Es--u
(x(((s) <ra(-|s))]=
established in Lemma A.6 of (Laidlaw et al., 2025). This allows to avoid discriminator-base estimation of occupancy ratios in this setting. Instead, we directly estimate the x² divergence using the following estimator:
π T(aas) s)
Tar(aas) 2
X MTxe) = Ess-pprann
[s)r(a
M (als)
as proposed in (Laidlaw et al. 2025).
For policy optimization, we apply the same Max-Min training algorithm described in Appendix 12 adapting it to the contextual bandit structure without discriminator training. We evaluate our ap- proach in the RLHF setting using setup consistent with prior work (Laidlaw al. 2025 Coste iet al.). The proxy reward function is derived from a reward model fine-tuned on the AlpacaFaam preference dataset (Dubois al. 2023), using the Pythia-70M model (Biderman al. 2023) For the true reward signal, we adopt the Llama 3 Tulu V2 8B reward model released by A12 [Ivison ct al.) 2024). The reference policy corresponds to the supervised fine-tuned (SFT) model from (Laidlaw et al. 2025 Coste et al.), which was trained on the AlpacaFarm SFT dataset using Pythia-I.4pp All policy evaluations, both proxy and true rewards, are conducted on the same set of prompts used in [Laddlaw et al. 2025).
To further strengthen our experimental evaluation regarding the RLHF setting, we also compare against the Reward Ensemble method (referred to as Ensemble for brevity) (Fisenstein et al. 2023). Specifically, we adopt their finetune ensembles setting: we fine-tune five reward models on the AlpacaFarm preference dataset, all initialized from the same pre-trained Pythia-70M model but using five different random seeds, and aggregate their outputs using the mean rule. This setup is directly comparable to our RLHF configuration for ORPO and our methods, where both ORPO and our approach use a single fine-tuned Pythia-700 70M reward model.
Selection of For our Max-Min and L, near Max-Min policy optimization algorithms, the correlation parameter r serves as an additional hyperparameter. In practice, as with ORPO (Laidlaw) al. 2025), r may only be approximately estimated, and there currently no principled method for selecting its optimal value. To address this, we perform a grid search over r E {0.1, 0.2,... 0.9} for each environment and measure the resulting Max- -inn and Linear Max-Mixg policy expected returns under the worst-case or linear worst-case reward. Results on all searched can be found in Appendix(H.5) Additional analysis of how different training values of affect robustness under vary- ing evaluation values is provided in Appendix H.2 Unless otherwise noted, we use the following r' values for training and evaluation throughout our experiments: r = 0.3 for Traffic, r = 0.7 for Pandemic, 0.9 for Glucose, 0.4 for Tomato, and 0.4 for RLHF. As for ORPO policy, we trained with occupancy-measur x2 regularization, using the official implementation from (Laid- set as recommended ensure optimal performance. The ORPO+ shares the exact same setting as the ORPO policy with the full discriminator training schedule as in our algorithms.
law et al. 2025). All hyperparameters
Evaluation of the worst-case performance. Theoretically, in the absence of structural constraints on the reward function, as opposed to the case of linear rewards, the worst-case reward of policy
43

---

Published as a conference paper at ICLR 2026
in atatenaction pairs unvisited by Tie be arbitrarily negative without violating the correlation constraint. However, assigming extremely negative values is impractical in real-world scenarios due to domain constraints. Moreover, doing so would render all policies with at least one unseen state-action pair equally poor in terms of worst-case reward, obscuring meaningful comparisons. To address this, we define minimal feasible reward value Rmin and assign it to all unseen state-action pairs, The actual expected worst-case reward (Worst*) is thus calculated as:
Σ
1 a) Rworst
Σ
p=(s's aa)) Renin
where the first part derived from the adversarial reward function given by our inner minimization solution, and the second part applies to state-action pairs unvisited by Trel-
In practice, however, environments like Traffic, Pandemic, and Glucose are continuous with large state-action spaces, making it difficult to reliably estimate frr (s, a) and Ma (s. a) from limited number of trajectories. As result, identifying unvisited or low-density regions these environ- ments is far more ambiguous. Therefore, for these continuous environments, we rely on the output of the discriminator as a signal for detecting unseen state-action pairs. Specifically, if the discrimi- nator outputs infinity (or diverges numerically) for given state-action, we treat this that the state-action was never visited by the reference policy Tel We approximate the total occu- pancy (Oee) over such state-action pairs by computing their frequency in the sampled trajectories, and use the expected worst-case reward (Worst) of a policy π over the remaining state-action pairs the default worst-case performance metrie: [(s.a):d, co fx (8, a) Rwora (s, a). In contrast, for the discrete Tomato environment, we directly estimate the occupancy measure by sampling state- action pairs and then compute Worst accordingly. Further details on this procedure are provided in Appendix [H.2]
indication
To compare the worst-case performance of different policies, we sample 200 trajectories in the Traffic and Glucose environments, 20 trajectories in Pandemic, 1000 trajectories Tomato, and answers per prompt in RLHF to estimate the worst-case performance of a policy.
Evaluation of policy robustness. To evaluate robustness across different correlation levels, we uniformly sample candidate vectors 0, where each component 0, is drawn from the interval [0, 1]. We use the same number of trajectories sampled from the reference policy Tee to determine whether it satisfies the correlation constraint:
0 O M. V
Rproxy = r,
where M and V denote the mean and standard deviation of 0T under the reference policy.
Note: For our worst-case performance evaluation, we explicitly normalize the reward to have zero mean and unit variance under the reference policy (enforcing M 0 and V 1). In contrast, for the robustness evaluation across correlation levels, we do not apply such normalization. This allows us report the average reward under each 0 in its original scale, reflecting variability more comparable to the original true reward landscape.
F.6 TRAINING TIME AND COMPLEXITY
Table 4
reports the total training time for each algorithm across different environments. All experi- ments were conducted on a single NVIDIA RTX 4090 GPU (24GB memory) and a 13th Gen Intel Core 19-1390KKF CPU (32 threads). We implemented all methods in Python 3.9 using PyTorch Paszke 2019). RLlib (Liang et al. 2018) and trIX (Havrilla et al.) 2023).
2.6.0
The training times and memory usages across different environments are summarized in Table and T Since the training durations for ORPO*, Max-Min, and Linear Max-Min differ by less than one hour each setting, and the memory usages differ by less than 10MB, we group them together for brevity. Since the RLHF environment does not require training discriminator, the training times and memory usages for ORPO and our Max-Min are identical. We therefore exclude RLHF from the runtime analysis. As shown in Table 4 all three methods require more training time compared to the original ORPO implementation. The icreassed training time primarily results from additional gradient steps used to more thoroughly train the discriminator network. Specifically, the
44

---

Published as a conference paper at ICLR 2026
per-iteration training time is approximately 2.5 minutes for Traffic, 4.6 minutes for Pandemic, and 8.9 minutes for Glucose. This leads to a total training time increase from roughly 7 hours to 37 hours in Glucose. However, the added cost is environment-dependen and remains moderate in simpler settings, for example, increasing from 5 to 10 hours in Traffic. In contrast, the memory footprint of methods is very close to that of ORPO: the peak CPU memory usage differs by less than 30-50MB across environments (within a few percent of ORPO in all cases). Overall, these results indicate that our methods introduce modest runtime overhead and negligible memory overhead, achieving practical trade-off between computational cost and the improved quality of divergence estimation.
Table 4: Approximate training time for each algorithm across different environments.
Algorithm
Traffic
Glucose
Pandemic
Tomato
ORPO
R55h
77h
s 114h
11h
ORPO+
Max-Min
10h
=37h
119h
2 1h
Table Approximate memory usage for each algorithm across different environments.
Algorithm
Traffic
Glucose
Pandemic
Tomato
ORPO
ss =1679MB
ss *1662MB
s *181MM
2= 214MMB
ORPO Max-Min / Linear Max-Min
11706MB
~1674MB
~ =164MBB
2 1903MB
Complexity. At first glance, regularization-basee approaches like ORPO may appear more com- putationally efficient than max-min optimization, which often involves iterative procedures to solve both inner and outer objectives. However, in practice, ORPO requires repeatedly estimating the x² divergence between policy distributions during each policy update step. This estimation is done by training discriminator network, which itself involves multiple optimization steps per iteration. In contrast, our Max-Min formulation admits a closed-form solution for the inner minimization over reward functions. This allows variant, although a closed-form expression for the dual variables is not avail- able, the corresponding dual optimization problem smooth and well-posed, and can be solved efficiently using standard gradient-based methods. Therefore, despite the max-min structure, our method does not incur higher practical complexity compared ORPO. In fact, both approaches rely on discriminator-baa iteration. The main difference lies the structure of the objective, not in the asymptotic or em- pirical complexity. In summary, ORPO does not inherently enjoy complexity advantage over our Max-Mix or Linear Max-Min algorithms.
avoid iterative solving in the inner loop entirely. For the
Linear
divergence estimation and perform comparable amounts of computation per
G CONVERGENCE ANALYSIS
In this section, we study the convergence of our Max-Min and Linear Max-Min algorithms. As both methods rely on accurately estimating the occupancy measure, we begin by analyzing the sample complexity of this estimation via the discriminator described in Appendix F.
G.1 SAMPLE COMPLEXITY OF OCCUPANCY MEASURE ESTIMATION
In this section, we analyze the sample complexity of estimating the occupancy measure via the discriminator described in Appendix(F.1 Our argument adapts techniques (uang et al. (2023); Barakat al. (2024) to our setting. To start with, we define the following notations for convenience. Let (s, range over the space X' of all possible state-action pairs. We consider two reference distributions on X: Pa and pr. The (true) density ratio of ft with respect MAx is
(x(x)
(x)x)= =
on the support of Psat
Ma (x)
45

---

Published as a conference paper at ICLR 2026
and its log-ratio is d² dx) = log (z).
We work with a parametric log-ratio class D = {do ;x + R} and the induced ratio class =
{Lg = = Following Equation 38 in Appendix F.I we learn do by minimizing the following loss:
R(d) = Bewaeal log(1 + ) + 1 Ez log +e e=)())).
(48)
2
Given Dref i.i.d. samples {xr pendent, we can minimize the empirical loss as follows in practice:
Mover and n, i.i.d. samples {x; و Max, which are inde-
Rize
R(d)
1
log [ + )((a
Σ log t =-d(z))).
(49)
Thyof
nx
j=1
Let the true loss minizer be
d' E arg min R(d), dED
==== Ea"
Let the empirical loss minimizer
ā E arg min R(d),
:== e
dED
For convenience, we also define the mixture distribution Mmix := 3 Mas + 2 Ma, which will be used later. We make the following assumption throughout the analysis. Assumption (Modeling, boundedness and cover). The following conditions hold throughout the analysis:
1. Realizability. The true log-ratio belongs to the model class: d' E D (equivalently, L E C).
2. Bounded. There exist constants S < co such that
S Lø(x) 8 for all E X, d,
and hence a S (x) S 8 as well. Equivalently, d2(x) E [log or log 3].
3. L, optimistic cover (Definition 3 of (Huang et al. (023 There exists finite set C
(0, oo)x with cardinality [2] M and scale
O such that for every T. E L there is
I. E L with
I(x)) 2 L(x) for all x,
Egoeppan [[(x)= - [(x)]] sm. a [(x)) SB.
We denote D = (d = log I: I)
Assumption I collects the conditions used throughout our analysis. First, realizability is standard in likelihood-basee occupancy estimation and allows to control the estimation error through the complexity of the parametric class rather than the size of X. In practice, sufficiently expres- sive neural discriminator makes this assumption reasonable. Second, boundedness guarantees well- posedness on the support of Paat and prevents divisions by zero. It can be enforced by restricting attention to the support of Mr or by applying ratio clipping during training. Finally, the L1 opti- mistic cover (adopted from Definition 3 of Huang et al. (2023) is the technical device that enables uniform concentration and converts control of the loss in Equation 48 into an L₁ error with clean constants. We instantiate this cover for our discriminator class later in the proof.
We begin by stating some auxiliary lemmas that formalize the structural claims used later.
Lemma (Strong convexity of R(d).. Let Assumption 7 hold true. Define
min(a, B)
A
0.
(1 max (ax, 3})2
Then for any measurable d E D and for the unique minimizer d' to Equation 48 we have
X
R(d) R(d") 2
Eurpaull (d(x) d'(x)) )
2
46

---

Published as a conference paper at ICLR 2026
Proof. Fix x and define the pointwise loss
pn (x)
Th (d) == (1 - ((=) log) + e*) + (((r) e nTz) ==
M1x (x) Mar (x)
Its derivatives w.r.t. the scalar d are (d) = o(d) (d(()--i 7xx) and rd) = (d) odd) > 0, where
(d) I+ed We notice that (d) is independent of ()()). Therefore, at every v; the pointwise loss Ta is strictly convex in d.
Now estimate the lower bound for rd))). On the range d [log a, log, 3] (boundedness from Assumption 1. let y = E [a, then
(d) = (1-)(d)) = (1 (1+y) y)
y
=: f(y)
1-y Let's consider the monotonicity of f(y). f'(y) = (I-y)3 so increases on (0, 1] and decreases on [L, 00). Therefore,
min f(v) = min (1+ (1+ 8)2 5E(a,
a
We can consider a slightly more conservative but simpler bound:
minfa, (8}
min f(y), (E|o,9)
+1+ max 3))2
Thus, for all and all d E [log a, log B], have (d)
A. Notice that strong convexity (with
parameter A) of a C univariate function g satisfies:
g(u) 2 g(v) + g'(u) (u v) + x
2 (u u)²
for all u, v.
Applying this with g(-) T-(+), d(z), and v d (z), where dt the unique pointwise
minimizer (so r' (d* (x)) 0). We get for every
To (d(x)) - -T (d (x)) 2 (d(x) - d' (x))))
Recall that Mmix
1 2 F + 2 ² and
1
R(d) = 1/2 Ferrsppall log(11 + e {()) + 2 Ra [lo((1 edd(z))] Exroypi [d)(d)
Taking expectation with respect the mixture Mmix gives
Rdd)= R(d)) = R. (d(x)) - rr (dx
2 2
E Be -d)())
which is the desired inequality.
We now establish three Lipschitz bounds that will be used repeatedly in the analysis.
Lemma 2 (Lipschitz bounds). Let I.+ = I+3 and I._ = Ita following hold:
1 For all d, d E [loga, log 3). the
1. log(1 ed) - log(1 + اd)ه S Le Id d).
2. log(1 + ed) log(1 + e s I._ ld d).
3. ed - edl < 3 ld d|.
Proof. (1) Define f+( log(1 + e"). Then f1 (u) o(u)
- On E [log a, log 3] we
have e [a,, hence
[u)) lte" eu
e"
sup DEla,
y
B
L+.
1 1+ y
1 B
47

---

Published as a conference paper at ICLR 2026
By the mean value theorem, (f+ f+(d)] - L+ ld d].
(2) Define f-(u) log(1 + t) Then (u( -0(-
I+e" ITee For [log oe, log 3].
1
[f-(w)] = 1t E sup
yelu,0] 1 1+ 1+a L...
Again by the mean value theorem, fd [d) -- ld -d.. d|.
(3) For g(u) = ex we have g'(u) = e". On llog a, log B), E" S 3, hence (g S B. The mean
value theorem yields |ee ed S B|d d.
Lemma 3 (Uniform deviation over the finite cover). Let Assumption 7 hold true. Define
B = 1/2 log(1 + 3) + 1 log (1 4)
Let 'D be finite cover (I1 optimistic cover from Assumption 1 with cardinality [D] M. Define log(b//0) log( (M/6) for any E (0, 1). Then, with probability least 6,
== (nref n and 17 == V
72
sup dED
R(d)) R(d)) | I nB.
(50)
Proof. Fix d D. Define
Ther
Amr(d) =
log(1 + [()))]] - Eppre log(1 +c)))))
i=1
nhe
Am
log(1 + -e-0(z)]] E.pl log(1 +e e- -a(z))]
Then
R(()) R(d)) = 1 Aer (d) (d),
R ((d)) S (d) (d)
By the boundedness of Assumption 1 we have d(x) [log a, log 3], each summand satisfies
S log(1 + ) S log(1 + 3),
0 S log(1 + e d(wr) s log + d).
Hence, by the Hoeffding's ss inequality, for any t > 0.
2nrett?
2n,t² t
P()Are((d) 2 t) = exx
(())(()) 2 t) s 2
(Tlog() 3)2
log(1 + _
Choose
log(2M/5)
log(2M/5)
:= log(1 + B)
1 : log(1 2)
2nref
2n
Then P( [d) 2 free ) < 8/M and P(IA, (d) 2 t S 8/M. Taking union bound over all d E D yields, with probability at least 1 - 6,
R(d) R(d)
1 < 2 fref + t
sup ZED
Finally, since n = min min(n nref, n ), we have
log (28
log(M/8)
log (2M/)
log( (M/M)
2nref
n
2n
n
up to benign constant factors that we absorb into the front constant.
Using the definition of and
setting 1) /l8g(M/8)/n gives
sup
R(d) R(d) s 2Bn n
7
which is Equation 50
48

---

Published as a conference paper at ICLR 2026
We now prove the transfer bounds that move deviations the cover element d back an arbitrary d, measured either in the original loss or the empirical loss. These inequalities will let us relate risk differences to L1 discrepancies between ratio functions.
Lemma (Transfer bounds from a cover element to an arbitrary discriminator). Let Assumption hold true. Let L2 = T-3 L_ = Ita and define
L4 + BL
C2
2a
Then:
R(d) R(d)
$ Co %
(51)
(R(T) R(d) 5 Ca 7.
(52)
Proof. Start from the definition
R(d) E Eut... log(1 ed)] + Ep log(1 + t]
By the triangle inequality,
((())-R(d) < 1/2 Eu-yar
E,r - |. (53)
Recall the Lipschitz bounds derived in Lemma
2
log(1+ e*) - log(1 + e²) < L+ Iu o).
log(1 + +e- - log(1 (1 + t s L 1u la
yield from Equation 53
(d) R(d) S 1 I.4 R. E d + 1 I. E, ld dl)
(54)
Since 7 L Prat and L < 3 by boundedness from Assumption |
B ld - d|] Bu Lid dl S 3 E Boouee d - d]]
(55)
Plug Equation 1atiiioon55abacce b5ck into Equation 54
R(d) R(d) 5 L4 + 33
E Penee dd ].
(56)
2
Next, need to conver E Eur,or [|d - -d] to E But,pa on [a, R).
[ - L1].
Use the mean value theorem for log
1 1L- ---L L1p
d - d - |lo L log Li = {
11 L1
between L and L.
Therefore, we have
E E jd d < - E Manel
I - [[-
(57)
Combining Equation 56 and Equation 57
(R()) - Rdd)) S CA Ex [T(x) - [(r]],,
CA CA= = (I+ + (2.))
Finally, according to the L1 optimistic cover of Assumption
we have
Epp, [IL LI] <>
We get
R(d)) - R(d) < CA
The same derivation holds if we replace expectations by empirical averages (sample means). Every inequality we used above (triangle inequality and the Lipschitz bounds) is pointwise and hence holds averaging over a finite sample instead of the distribution. Concretely,
Ruef
((() - R(d) Co
Σ [I((x) (((())-(aa)) L(x)] S Co
where the last inequality uses the empirical L1 closeness of L and L.
49

---

Published as a conference paper at ICLR 2026
We control the excess loss of the empirical minimizer by combining the uniform deviation bound over the optimistic cover with the transfer inequalities.
Lemma 5 (Excess-loss bound for the empirical minimizer). Let Axsumption true. Let = log( (+ (og(1++/// CA (I4 (/2)))/(2o)) with IT+ = -3/(1 and T._ = 1 + and 1 = //og(M//)) where n = min ( nx defined as before. Then, with probability at least 1 6,
R(d) R(d') S 3C - -4n..
(58)
Proof. Since d E mingee R(d,, have:
R()) R(d) for all d,
particular for = dr and for d d' (the cover of dr, d' log Lt). Start with standard add-subtract trick:
R(d) (R))-R(d"))
(59)
20
For the first difference in Equation 59 insert the cover element d log of d and apply the transfer bounds (Lemma 4 and the uniform deviation bound over the finite cover (Lemmal 3
(d) R(d) = (R( R(d)) + (R(d) R(d)) + (R(d) -((ā) - [R(d) - R(d) + [R(d) - R(d)) + ( - R(())
s Cay + 2Bn + Ca7-
The first term uses the transfer bounds from d to its cover d. The middle term uses the uniform deviation bound the finite cover. applies directly because df D. The third term the empirical transfer bounds. Thus, we have
R(d) - R(d)= S 20 + 2Bn.
(60)
Returning to Equation 59 the middle term is nonpositive by optimality, and the remaining two terms are bounded by the same two lemmas:
(e(d)) - R(d)) & 2Bn,
R((d) - R(d") & Cay.
Combining with Equation 60 yields
R R(d) - R((d)) < (2C + 2Bn) + +BB + CaT = 3C + 4B4.
which is Equation 58
Finally, we derive the occupancy ratio error bound.
Theorem (Occupancy ratio L1 error bound). Let Assumption hold true. Let
3
1 = 2 log(1 + + (1 + 1/a),
L = 1+
I_ =
1+a'
L + +L
minfa, 33
CA =
A
0,
2a
(1 max(ax 8})2
and n == mn }. J. 17 = Vlog(M/8) for any E (0, 1). Let d arg minged R(d) be the empirical minimizer, T= e², and T e² be the ratio. Then, with probability at least 6,
4
Enr -(x) $ 3 3
/CCA + 4Bn
(61)
50

---

Published as a conference paper at ICLR 2026
Proof. By Lemma (excess-risk bound for the empirical minimizer),
R(d) R(d") I 3C2y + 4Bn with probability least 8.
From Lemma I Te have,
2 (3
Egrnpia (d(xx) dz (x))) - R((d)) - d
where Mmis 2 Mover + 2 Pr. Since Mmnix 2 2/2 Mxyer have Epee.tt 21 and
4 (3C
E Ee dx
Finally, by the third point of Lemma 2 (the exponential map is 8 B-Lipschitz [log a, log 8))
and S B B Bu [ld-a'] S 3 B [ppaac(a-d d-)3] S $ 3CA7+4Bm,
which is Equation 61
G.2 GUARANTEES FOR MAX-MIN WITH OCCUPANCY MEASURE APPROXIMATION
In this section, we establish convergence guarantees for our Max-Min Algorithm Our analysis follows the Reinforcement Learning with General Utility (RLGU) (Zhang al. 2022 Barakat
et al. 2024). where given utility function F()), 0 + F(Px,) over the policy-induced occupancy measure Ma the goal of RLGU is to find policy Th such that Th arg max, F ((uTr): In RLGU, there is no reward function. Instead, we can view VgF(H)) as pseudo-reward depending on the unknown occupancy measure induced by the policy. The procedure for solving RLGU follows three steps: (i) estimate the occupancy May (c.g., by MLE), (ii) form the pseudo-reward from this estimate, and (iii) update the policy. Our Max-Min algorithm mirrors this pipeline. Specifically, we first estimate the occupancy ratio by training discriminator. Then we construct the worst-case reward using Equation 19 from the estimation. Finally, we perform a policy update. Consequently, the general RLGU sample complexity guarantees apply to our algorithm after replacing the pseudo- reward V F(()) with our worst-case reward and substituting their occupancy occupancy-estimation estimmaton error with occupancy-ratic error obtained above. We formalize this correspondence and state the resulting bounds below.
For each iteration of our Max-Min algorithm 1 1., let the pseudo- ra (s, is defined as in Equation 19 Let's define
F(y)) = 4hx /
(e(5, a)
- (ls.))
M (8, d(s,a)
(62)
MaTer
where c(s,a)= = 1 (A1 Rproxy(s.a)) 22). By construction, have (s) (,ee( 2A9
which means the utility gradient in μ is exactly the pscudo-reward. Let's fi L, Mamt be the occupancy estimator, F((s,a) a) = (,)(())(ss,))) be the estimated pseudo-reward and E max; F F(Um)) be the maximum.
We next introduce some assumptions that are required for our results, which are adapted from (Barakat al. 2024).
Assumption 2 (Policy parametrization, Assumption 6 from Barakat et al.
2024 For every
as E x A and every 0 E Rd, the policy has full support, Lee, ae(a 8) mapping 0 + Ta(a 8) is continuously differentiable, and the score function O V log (a(a 8) uniformly bounded:
0. Moreover, the
|| log Te(a |)|l bx for some constant by 0 and all (s, (s,a), 0.
This assumption typically holds in practice, for instance, with the standard softmax policy param- cterization. Next, we make smoothness assumption the utility function, which is crucial in deriving the final convergence bound. We also verify that the defined utility function in Equation 62 satisfies the smoothness assumption.
51

---

Published as a conference paper at ICLR 2026
Assumption 3 (General utility smoothness, Assumption 7 from (Barakat al. 2024)). For utility
function F(.),8 8 + F(HTo), there exist constants يμ > 0 such that for all pl P2 E..
V P(m))l s Ey
and [[m)(()) V
Ly | palle-
Notice that Assumption B holds in our setting since Hessian is diagonal with entries most V (())(s a) 12x3 Mazer (8, a)). Thus, if Mn,er (8, a) 2 Pmin > 0 on the support of all (s,a). which we assume it holds, then we have that V F(1) is -ipsschti with
1
[[F VnF(e'l12 S Ln l1l 1l12.
2|Aa| Pmin
the utility function F(JTT) is Lg-smooth. Using these properties, our Max-Min algorithm admits the following first-order stationarity guarantee:
Under Assumptions 2 and 3
Theorem 4 (Guarantee for the Max-Min update). Assume Assumptions 2 and hold. Let N be the batch size for estimating the policy gradient at each iteration, a the stepsizes satisfying ag $ 1/(2L)), Keonv = || (B a) and
EL == 3 / V3C2++4Bn 2 E Bg [1((x) [x))
Then we have:
16 16 - -FFF-EBF(A F[F( 31) C1 + Keoav EL.
E VoF )1²] 2
(63)
TT
where T is drawn uniformly from {1....,Ty T), expectation is w.r.t all randomness (in (0,) and T), 81 12 [1-50p with r be the discount factor in RL.
8/. C [1-7) and C2
Proof. Since we already verified that Assumptions 2 and 3 hold in our setting, according to Theo-
rem 8 in (Barakat et al. 2024)), ) we directly have
T
16 (F* E (M
Ci N
T
T
E E[IVo |
T
+
+
C2
T
Σ
E [[]T
To control the last term via the ratio error derived in Theorem
3 note that with The Tex piver and
[4 = La Mrter
[i Hell2 Σ
M a)² (Lu(s.a) < || lloo (B (8-a)
Σ Mane (s, a) 1t Lel,
8.a
using x² < (3 axl for [x|| < 3- and 2 5 $|lttees oooo Mireer Therefore,
E[ 14 [12] Keonv Ezzpp [[(((r)) Ix(x))] < Kenny E1,-
Averaging and drawing uniformly from {1...., T} yields Equation 63
G.3 CONVERGENCES FOR LINEAR MAX-MIN ALGORITHM
For our Linear Max-Min algorithm, challenging derive a convergence bound directly. How- ever, as discussed in Appendix [..6] the inner optimization problem, i.e., finding the worst-case reward for a given policy, admits a globally optimal closed-form solution under our formulation in the tabular setting. Therefore, for any given policy π. have access to oracle that outputs the optimal worst-case reward R*, and our Linear Max-Min algorithm can be viewed as alternating between gradient ascent
and the optimal minimization R As shown in Section 4 of (Jin jet al., 2020). our algorithm converges, and the resulting policy π corresponds to an approximate stationary point of the outer optimization problem.
52

---

Published as a conference paper at ICLR 2026
ovtrr spachs
spochs
1.00
0.3
6 0-15
0.00
0:2
0.00
0:2
0.0
000
30
300
350
200
250
50
220
356
200
(a) Traffic environment: Oo (velocity), O, (accelera- tion), 02 (headway).
(b) Pandemic environment: Og (infection summary),
0 (early-stage), 02 (smoothness).
Figure 4: Evolution of adversarial reward weights 0 over training epochs for different environments
using the Linear Maxmin method.
H ADDITIONAL EXPERIMENT RESULTS
H.1 FEATURE WEIGHTS IN LINEAR MAX-MIN OPTIMIZATION DURING TRAINING
Figure visualizes the evolution of each component of the linear worst-case reward weight vector during training in the Traffic and Pandemic environments. We observe distinct behaviors in the dynamics of 0 across tasks.
In the Traffic environment (Figure 4ap. we observe that the three 0 parameters vary significantly in scale. Specifically, 01 (accelcration) exhibits the largest magnitude, ranging from 0 to 2, while 8g (headway) has the smallest scale, ranging from 0.05. This highlights how the linear max min algorithm assigns different levels of penalization to each feature. Moreover, we also observe distinct phases in the dynamics of over the course of training. the early epochs (<50), all components, especially 02 (acceleration) and 02 (headway), exhibit nigh-frequency fluctuations. At this point, the dual optimization problem is not yet well-conditioned, and the adversarial reward is highly sensitive to small changes in occupancy or feature values. As training progresses (~epochs 100-250), the parameters begin to stabilize. Most notably, 02 (headway) converges close to zero and remains suppressed, indicating that the worst-case reward does not emphasize this feature. This may suggest that headway is less harmful under adversarial reweighting compared to others (velocity or accel) or is already well aligned with the reference policy Tref- Meanwhile, 0, (acceleration) consistently exhibits higher values and sharper spikes than the other components. This indicates that acceleration plays a dominant role in the adversarial reward, likely because policies that optimize for the proxy reward tend to exploit aggressive acceleration patterns that diverge significantly from the behavior of Tee- contrast, O (velocity) remains small and relatively stable throughout training, suggesting that speed alone is not strongly penalized under adversarial interpretations.
Overall, the observed pattern reflects the interpretability and sparsity benefits of the linear max-mi min formulation. The model is able to selectively emphasize features that are most vulnerable to reward hacking, while suppressing those that are either irrelevant or well-aligned. This structured behavior supports the practical value of using linearly parameterized worst-case rewards to improve policy robustness.
In the Pandemic environment (Figure 4b) unlike the Traffic environment, where 0 converged to a sparse and interpretable solution. we observe high variability across all components throughout training. In particular, find the following pattern:
Persistent fluctuations. All three components exhibit frequent oscillations over the course of 260 epochs. This ongoing instability suggests that the adversarial reward continually
53

---

Published as a conference paper at ICLR 2026
adapts the policy changes, likely due to the environment's temporal sensitivity and com- plex dynamics.
2. 0g (smoothness) remains active. The smoothness-relate component 0g is frequently non- zero and relatively stable compared to the others. This indicates that the worst-case reward consistently emphasizes penalizing erratic or unstable responses in the infection trajectory a behavior often neglected by naive proxy metrics.
3. 01 (early-stage transitions) is highly volatile. The component associated with early in- fection stage changes spikes intermittently. This suggests that early-stage mismanagement is recurring vulnerability in the learned policy that the adversarial reward seeks to exploit.
4. 8n (overall infection) activates intermittently. Although Og sometimes spikes, it does not dominate the adversarial reward. This may indicate that the learned policy already accounts for infection magnitude reasonably well, or that smoothness and early-stage control offer more leverage for reward hacking under the proxy constraint.
Overall, this pattern highlights that in more dynamic and temporally complex environments like Pandemic, the worst-case reward remains non-sparse and adapts to different policy weaknesses throughout training. In contrast to the Traffic environment, adversarial emphasis here is broader and more reactive.
H.2 ADDITIONAL. WORST-CASE PERFORMANCE RESULTS
Table 6: Evaluation results on Traffic, Pandemic, Glucose, and RLHF environments. All policies are trained using only the proxy reward. In Traffic, the proxy reward is based on vel, accel, headway (1. 1, 0.1). while the true reward uses commute, accel, headway (1. 1, 0.1). In Pandemic, the proxy reward includes infection, lower stage, smooth changes (10, 0.1, 0.01), while the true reward additionally includes political with weight 10 after infection. Glucose, the proxy uses expected patient cost, and the true reward uses magni.be In RLHE, the proxy uses a 70M LLM, and the true reward uses a 8B LLM. We report in the same order as feature weights. Occ denotes total occupancy over state-action pairs unseen by Tnf, where discriminator outputs infinity.
Eav Metlond
Traffic
Truc
Frogy
Wurst
Linear Worst (0)
Linear Worst (0)
Oce
GIUPO
1601-612
541
-I -150040
-.60-1100 (0 0221, (.699)
-0AT-02 (0.63, E.TZ (:32)
3826-04 +i 13G
CR2C+
10.26
13310009
-1.35e=0L 10.02=001
0.641.0.2 (0.46, 0.18, 0.85)
-0.411001 (0.58, 0.05, 0.81)
1.84e 154e-0410000 07
12.20:0006
36320099
26833
0.060000 (0.01. 0.96)
4100:0001 U.00, (159)
0,005:0000
16.66-0.10
2.40-.111
1.166044000000
0.30+0001 (0.64,
-0.12+000 (0.91,
0,00+0000
Pandemik
Metlind
Proxy
Lincar Worst
Linear Worst" (8)
ORPO
-1081:211
12520119
5 5.3eeo0-
2.44::0..02 (0.23, 0.95, 0.17)
2260520012 (1,02, a.95, 0.92. 0.88)
CRPC+
1.1810.19
1181:119
1.56:10.01 (0.25, 0.97, 0.13)
-1.36.00.01 10.25, 0, 0.97, 0.13)
Max Kin
1:254018
1254018
-63 20++ 35
-1.11=0.011 (0 14,14.000 0.99, 0.01)
-1.11+n0l (0.14, 0. 0.59 0.01)
Sinear Max-Min
365.10.11
76010.13
6 6.820+06 1.00eee-
0.65 (0.00), 0.23, 0.02)
0.11..0:02 (0.01. 0.97, 0.2.0.2.20000 0.09)
Eay
Glucose
KLHE
Method
True
Proxy
Worst
True
Prony
Worst
10048=0054
54
T07
0635021
-1,841100 N/A
6.3102
..79100.7
N/A
N/A
n
6,320011
06:00.
-1.11:02
5,38
01430.11
00100=0.001
Adversarial Weight Analysis. In Table 6 we also report the adversarial weight vectors for each policy. These weights reveal which features are most vulnerable to proxy exploitation under the learned policy and can be used to diagnose and revise the proxy reward function, thereby im- proving robustness. This highlights the interpretability benefits of our framework. Moreover, several patterns emerge from the results. In the Traffic environment, first, we observe a clear dominance of the headway feature, with all methods assigning it the highest weight. This suggests that headway is the most critical component exposed to reward hacking under correlation constraints. Second, the acceleration feature consistently downweighted across all methods. This indicates that accel- eration may be less prone to exploitation or already well aligned with the reference policy. Third, the velocity feature is moderately emphasized by Linear Max-Min and ORPO (e-g.. 0.64 and 0.71), while Max-Min nearly suppresses it (0.01). This contrast suggests that Linear Max-Min anticipates some vulnerability from velocity deviations, while Max -Min focuses almost entirely on headway. In the Pandemic environment, first, both ORPO* and Max Min assign zero weight to the political feature. This occurs because the expected feature value under their policies is exactly zero, making the correlation constraint inactive for that dimension. Interestingly, this feature plays a significant role in the adversarial rewards for both ORPO and Linear Max-Min, with their cor- responding 0 assigning non-negligible weight to it (e.g., 0.95 and 0.97 respectively). This suggests
54

---

Published as a conference paper at ICLR 2026
that these policies expose themselves to vulnerability in feature dimensions that entirely ignored by Max-Min and ORPO+. Second, the lower stage feature consistently receives the highest weight across all methods, indicating it is the most sensitive component under proxy misalignment.
Table 7: Evaluation results on Tomato environments. All policies are trained using only the proxy reward. In Tomato, the proxy includes number of watered tomatoes plus bonus specific state (sprinkler), while the true reward only measures watered tomatoes. Occ in the Tomato environment denotes total occupancy over state-action pairs unseen by #ref based on 1000 sampled trajectories. Worst refers to the expected worst-case reward computed while excluding those unseen state-action pairs. Worst denotes the actual expected worst-case reward, while Rmin represents the minimum possible reward of any state-actin sate-action pair. All rewards normalized according to the reference policy net.
Method
Tomato
True
Proxy
Worst
Occ
Worst"
ORPO ORFO
6.28+0.22 4.00::0.18
6.83+0.28
-1.51+0.09
2.50e-04-0.63e-1
-1.51+ 2.50c-04
3.98=0.23
-1.09=0.10
3.09c-55=0.59e-05
-1.09 3.09c---5
Max-Min
4.56
4.68 10255
-1.37
1.01e-05 -1.37+ 1.01e-00
Worst- Worst-Case Performance in Tomato Environment. Table 7 reports worst-case performance sults for the Tomato environment. We omit the Linear Max-Min n policy from these experiments for the following reasons. In the Tomato environment, the reward structure is difficult express in a clean feature-based form suitable for linear modeling. Therefore, we report only the results for the Max-Min policy alongside the baselines.
The results for the Tomato environments exhibit trends similar to those observed in other environ- ments (Section 4.2) In particular, ORPO* appears outperform others in the Tomato environment in terms of worst-case performance. Recall that these results are reported under Worst, the expected worst-case reward restricted state-action pairs observed under weef- Since the Tomato environment is discrete, we can explicitly identify which state-action pairs are unseen through sampling, enabling clearer interpretation of their physical meaning as well as the evaluation of the actual worst-case per- formance Worst* The latter corresponds to the Worst value plus the product of the occupancy in unseen regions (Occ) and Rmin- Because Max -in exhibits the lowest occupancy among all meth- ods, it demonstrates greater robustness under varying assumptions about Rmin-
Nevertheless, ORPO* still shows marked improvement over ORPO, both in worst-case return and in reducing occupancy over unseen state-action pairs. As previously noted, in the Glucose environ- ment, the discriminator fails to detect any state-action pairs missed by the reference policy. This reinforces earlier concern that the current discriminator training procedures may have limited capacity to identify rare or oft-o--distibbttio events.
Impact of Correlation Parameter Selection Robustness. In this section, present addi- tional experiment results to examine how the proxy-true reward correlation parameter r used during training affects the robustness under varying evaluation
Table 8: Evaluation of robustness in the Tomato environment across different training-time corre- lation levels r. Oce denotes total occupancy over state-action pairs unseen by Tees based on 1000 sampled trajectories. Worst refers to the expected worst-case reward computed while excluding those unseen state-action pairs.
Oce
Worst (r = 0.1)
Worst (r = 0.4)
Worst (r = 0.7)
Worst (r = 0.9)
0.1
1.36G-03++0.222 12--03
-1.344+.005
-1.12++0.04
-0.74+0.03
-0.27+0.02
0.4
1.044-05:1::3e-055
-1.6641.007
-1.3740.06
-0.70=10.03
-0.05=10.2
0.7
105e-02-10.10e-02
-2.10
-1.82-10.07
-1.33-10.06
-0.66.10.04
0.9
1.290-055-0.41-055
-..10+0.20
-8.92+0.18
-7.60++0.15
-5.49-0.12
Table and Table report the robustness evaluation results in the Tomato and Traffic environ- ments under different training-ting time values of the correlation parameter r. Several consistent patterns emerge across both environments.
55

---

Published as a conference paper at ICLR 2026
Table 9: Evaluation of robustness in the Traffic environment across different training-time corre- lation levels r. Occ denotes total occupancy over state-action pairs unseen by Tel based on 200 sampled trajectories. Worst refers to the expected worst-case reward computed while excluding those unseen state-action pairs.
Occ
Worst (r = 0.1)
Worst (r = 0.3)
Worst (r = 0.5)
Worst (r 0.9)
0.3
0.00-10.00
-2794.63:-2.10
1268.3114.14
-82.07::2.04
-22.03::..88
0.5
0.00:10.00
-777ee-04#1.20e-000
-1.95c40413. 10c+02
-6:68.40::124.75
-1350.22:::2.95
0.9
9.66e-055:1.44-005
-300c=00566.55++03
-9.5e+404 -2.73=+04 -9.33c+03 1888+002
First, for any fixed policy (i.e., fixed training r), we observe that the expected worst-case return monotonically increases as the evaluation r' increases. This aligns with intuition: higher correlation levels correspond to smaller uncertainty sets over rewards, meaning the worst-case reward functions are less adversarial. In contrast, low r values expand the reward uncertainty set, allowing more pathological implausible reward functions, and thus lead to more pessimistic evaluations. How- ever, this does not hold universally. For a fixed policy, the expected worst-case return in Equation 27 is monotone in r only when the policy has a positive expected proxy return (which is the case here). If the policy's expected proxy return is negative, this monotonicity no longer holds.
Second, we find that training with a moderate correlation level, particularly around 0.3 0.4, yields better robustness across a wide range of evaluation r values. In contrast, training with overly small (e.g.. r = 0.1 for Tomato) large (e.g.. 0.9 for Tomato and Traffic) correlation levels degrades robustness. A small r leads to overly conservative training, anticipating extreme forms of reward hacking and thus hurting general performance. On the other hand, a high r overly trusts the proxy reward and fails to hedge against potential deviations, resulting in poor worst-case behavior under reward misspecification. This trade-off highlights that intermediate values of may strike a better balance between conservativeness and optimism, enabling the policy to generalize to broader and more plausible spectrum of reward functions. Therefore, in the absence of prior knowledge of starting with moderate is practical heuristic.
H.3 ADDITIONAL RESULTS FOR ROBUSTNESS ACROSS CORRELATION LEVELS
As discussed previously (Appendix H.2). we do not include linear worst-case evaluation for the Glucose and Tomato environments. Consequently, we cannot perform a uniform search over 0 as we do for the Traffic and Pandemic environments (Appendix F.5). As noted in Appendix B it is generally difficult, and often infeasible, to sample full reward function over all state-action pairs, particularly in high-dimensional or continuous environments, such as the Glucose environment. To approximate this process for the Tomato environment, which a discrete environment, we instead sample 1000 trajectories using the reference policy Teef- We then restrict the search to the visited state-action pairs. For each such pair, we perturb the original proxy reward by adding Gaussian noise with zero mean and variance sampled uniformly from the interval [0.001, 1]. We then check whether the resulting perturbed reward f satisfies the constraint Rt E Re As in previous evaluations, do not explicitly constrain M and V. We sample 20 perturbed reward functions and then use them to evaluate each policy. Note: Some policies, such as ORPO, may visit state-action pairs that are not included in the sampled set from Tee.. For these unseen state-action pairs, the proxy-true correlation constraint does not apply, as no corresponding reference data is available. such cases, we default to using the original proxy reward to evaluate those portions of the trajectory.
We emphasize that this procedure is neither optimal nor efficient, and is employed solely for eval- uation purposes in the Tomato environment. Designing more principled and scalable methods for reward sampling under correlation constraints remains an important direction for future work.
Figure 3 presents the average reward and standard deviation across varying correlation levels r for the Tomato environment. As expected, the reference policy म (blue) consistently underperforms across all values of though it exhibits the lowest variance-inddictting stable yet suboptimal be- havior. Interestingly, ORPO* (purple) performs worse than ORPO (red) throughout, suggesting that improving the accuracy of occupancy measure estimation does not necessarily enhance robustness in this environment. In contrast, our Max-Min method (green) achieves the highest average reward across all correlation levels, highlighting its better robustness under reward uncertainty.
56

---

Published as a conference paper at ICLR 2026
re 0.9
r 0.7
7he 0.5
re 0.3
2-
ORPO
ORPO
Max-Min
Figure 5: Mean reward and standard deviation under sampled reward functions at different proxy- true reward correlation levels for the Tomato environment. Our methods (Max-Mine yicld higher average performance across all choices of r.
H.4 ADDITONAL UNNORMALIZED RESULTS
To ensure fair comparison with prior work (Laidlaw et al. 2025). which reports results in the un- normalized reward scale, we also include the raw (unnormalin ized) expected proxy and true rewards. However, for worst-case reward metrics, it is nontrivial to reverse the normalization transformation, as formulation explicitly constrains the reward to have zero mean and unit variance under the reference policy. Therefore, we omit worst-case results in the unnormalized setting.
Table 10: Unnormalized performance comparison across all environments.
Method
Traffic
Fandemic
Glucuse
Tomato
BLI
True (x10")
Truc 3.962:0000
0B20
12.8119117
10.611:0.5
9,001011
16.112007
11.0610.15
-1100
-88.240.5
46.4260229
8.2400100
8:400010
16.320006
.0.31:0011
Linear Maxe 201.
-675.124:004
1522.34::219
9,905009
-.98=0010
NIA
MIA
NDA
Ensemble
Table 10 presents the unnormalized performance results across all environments. We observe that both our -aax and Linear Max-Min policies achieve comparable performance to ORPO on most tasks. Interestingly, the ORPO* variant (with fully trained discriminator) outperforms the original ORPO in some environments (e.g.. Pandemic and Glucose), but performs worse others, such Traffic and Tomato. While our earlier analysis (Section 4.2) shows that better discriminator training generally improves worst-case robustness, these results suggest that accurate discriminator estimation does not always translate to improved performance for every specific reward function. Understanding the nuanced effects of discriminator optimization on various reward metrics is be- yond the scope of this paper and remains important direction for future research.
H.5 RESULTS FOR ALL r
Here, we report the results of a uniform grid search over r 0.9) for our Max-Min algorithm across all training- time correlation levels the Traffic, Tomato, Pandemic and Glucose envi- ronments Tables II] 12 13 and 14] Each table presents the mean and standard deviation of the expected true reward, expected worst-case reward, and occupancy measure achieved by the learned policy over five random seeds. Here, the "true reward"" refers to the original (unnormalized) true re- ward, while the "worst reward" refers to the normalized reward. We also include the corresponding ORPO results for fair comparison.
Note that the original ORPO algorithm performs a grid search over X, where = o Rynny VI and 0 B is the standard deviation of Ramoxy under meet which is generally unknown and must be
57

---

Published as a conference paper at ICLR 2026
estimated. Thus, ORPO effectively searches over X, whereas method searches directly r. For a fair comparison, we estimate Byny and use it to map each value of r in our grid to a corresponding À for ORPO. However, across all environments we find that the resulting À values occupy much narrower scale than r: while spans the full range from 0.1 to 0.9, the induced values confined small interval (e.g., approximately 0.021-0.05 in Tomato and 0.035-0.08 in Pandemic). As consequence, the ORPO policies change only marginally across the mapped r values, and their expected and worst-case returns appear similar in the tables. In practice, ORPO would need to search over a broader range of X values. By contrast, the expected true and worst-case returns for our Max method vary meaningfully across the full span of r. This highlights that, in practice, our method and ORPO naturally operate on different hyperparameter scales when tuning their respective robustness parameters.
Table 11: Evaluation in the Tomato environment across training-time correlation levels for ORPO and Max-Min. À denotes ORPO's coefficient, with X Bgory v1 - ,2, We use Rymo 0.05 in this environment, consistent with the ORPO setting. Oce denotes the total occupancy over state- action pairs unseen by Ted. Worst refers the expected worst-case reward computed under the training r while excluding those unseen state-action pairs.
ORPO
Max-Min
True
Worst
True
Worst
Occ
0.1 0.050
0.46
6.008+0.07
8.22+-03=:0.7e-00
0.13+00.15
-1.34++0005
0..e-033=0122-000
0.2 0.049
0.66
6.888+0.08
2.8e-03:10:1ee-03
7.7910.10
-0.72::0.55
2.18e-33=0.25e-00
0.3 0.048
0.70
-7.5510.04
1666-00311.11e-03
7.6810.11
-0.96110.04
035e-03=0.16e-00
0.4 0.046
0.16
9..2010.06
1.1e--33-0.13e-03
8.24:1.10
-1.37-1.006
1.0le-05=0.43e-05
0.5 0.043
0.52+0008
6.53++0.05
0.028+0.0010
7.38+0.12
-1,21+0.04
2.15--03-0.37e-03
0.6 0.040
0.51
-7.4400.06
0.028++0.001
6.65:10.17
-1.22++0.06
1.18e-03=0.32e-00
0.7 0.035
0.8410.12
-5.8510.07
0.02710.0010
0.1610008
-1.3310.06
0.0-002=0.10e-02
0,8 0.030
0.16
7..0100.06
2.88e-33:10119-0:
1,02:1013
2.86110.04
5.77e-04=0.35e04
0.9 0.021
0.1110.09
-7.3010.08
0.032:10.0009
0.3710.13
-5.4910.12
1.29--05=0411-05
Table 12: Evaluation in the Traffic environment across training-time correlation levels for ORPO and Max-Min. À denotes ORPO's coefficient, with = Ryong ví r We use d Rynny = 2e in this environment, consistent with the ORPO setting. Occ denotes the total occupancy over state- action pairs unseen by Tel. Worst refers to the expected worst-case reward computed under the training r while excluding those unseen state-action pairs.
ORPO
Truc
Worst
Oce
Troe
Worst
Occ
1.99p-
1063.75=1.008
1. 1220=005+0002403
-1428-21=5.34
-3.43c404
2.29c-04:t0 36c-04
1.95e-4
-775.79=1.15
1.20=00110.Me+01
8.71e-0110..3e-01
-1312.6729.14
-2 2.833e+0
119c-0441.422-0
0.3
1.91c-
-689.31=1.12
5.13c40F200044000
3.98-04420022--00
-750.32=1122
-268.31 31=4,114
0.00=0.00
-1109.4121.59
4.94c-0310.055-00
-732.8611.19
-314.14=5.377
0.00:0.00
0.5
1.73e.4
1.73.45=1.36
6.85e.04:0000ee
.1044.242..22
-6168.40 124.75
0.00=00.0
0.6
1.60e4
7.5.0411.55
165e=00410.0Me-h
5.43e-03:1.022e-0
-1322.7413.92
.6.734403
4 1.50e05511.27ee0
1,43e.4
..93es044
+..04::44
7.60e-02-00.6ee-k
-1398.61+2.73
-3.82es044
1.74ex04
2.38e-04
-..35--44
0.8
1,20e-4
-782.01
8.74e=04
19.
5.97e-03-10.02e
.1359.232..08
4..4c+0411
101-0011144455 1.45++04
3.48e (eAkee04... (4.10.255e-0
8.726-5
-669.89-1.00
4.11e-04++0.56-00
-1337.41+245
-9.3e403-+
1.88e422
9.6fie-05
1.84-005
I How TO CHOOSE r IN PRACTICE?
When r is unknown, both method and ORPO lack principled mechanism for selecting an appropriate value. Besides the simple heuristics derived from our experiments as discussed in Ap- pendix H.2 outline two potential upproaches to this important problem below.
Statistical inference of r. we have access the true reward on subset of actton pairs, or if such labels can be acquired through active learning, we can estimate using the definition:
Eppsat (
[Ramayy
[ Rpeoxy)
Rime
J [ Rime))
)] =r;
F
(64)
O Bigoyy
a Bime
58

---

Published as a conference paper at ICLR 2026
Evaluation in the Pandemic environment across training-time correlation levels for ORPO and Max-Min. À denotes ORPO's coefficient, with À 0 Baaay v1 r². We o Rawy 0.08 in this environment, consistent with the ORPO setting. Worst refers to the expected worst-case reward computed under the training r while excluding those unseen state-action pairs.
Table 13:
ORPO
Max-Min
A
True
Worst
True
Worst
0.1
0.080
-12.22+0.14
7.29e+06+0.05e+006
0.37
-7 0..44e+04+00119 19++004
0.2
0.078
-11.77+0.11
-1.70=+07=001100 10e+07
-16.31+-049
3.03e=004=0.12+0000
0.3
0.076
-12.4940.20
-2.49e=06=10.55eHtt
-19.2711.38
-7004=+0420.05+404
0.4
0.073
-12.17.0.117
-19.211:.15
-2.03e+03
0.5
0.069
-12.26:#0.24
1.07c+06:40.0c<-mm
-13.75=0014
-.58c++03=0155+000
0.6
0.064
-12.08
2.65c+00=10.00c4000
-13.15=0.24
-104.00=0.22
0.7
0.057
-1..45++0.33
2.92=+05+0010e+000
-11.01+0.14
-63.29+3.35
0.8
0.048
-12.22++0.14
9.37c=05500006+000
-11.20+:..2
-133.65+0.15
0.9
0.035
-12.02.1..00
3.29:+0411.09e+444
-11.0510.13
-77.20
Table 14: Evaluation in the Glucose environment across training-time correlation levels r for ORPO and Max-Min. À denotes ORPO's coefficient, with d Ruary v1 n2. We use Rymo = 0.05 in this environment, consistent with the ORPO setting. Worst refers to the expected worst-case reward computed under the training r while excluding those unscen state-action pairs.
ORPO
Max-Min
r
True(x10*)
Worst
True( 109)
Worst
0.1
0.050
-90.210..7
-3.0.9410.37
-169.340.66
-317.97
0.2
0.049
-88.1+0.8
-199.26=0059
-1.0.2+0.6
-304.15+0.34
0.3 0.048
-7..110.66
-2.5.71=0.45
-118.310.4
-139.47=0.46
0.4 0.046
-72.210.44
-20..4010.27
-113.540.8
-123.11+0.32
0.5 0.043
.4..4+0.4
-2.5.00=0.27
-125.120.7
-126.41=00.3
0.6 0.040
-68.0+0.9
-266,50=0:47
-95.9
-84.67=0.17
0.7 0.035
-7..6.0..5
-3.2.
-51.740.8
-18.84:10.43
0.8 0.030
-5..540.6
-227.99
-3..3+0.4
-11.25+10.77
0.9 0.021
-5..9.0.5
-255.07
48.240.5
-1.71
In fact, Equation 64 defines the Pearson correlation coefficient between the true reward Rime and the proxy reward Rynoxy under the occupancy measure Mage Given batch of state-action pairs {(s, a,));= sampled from nee for which we have both R and plf) we can estimate this correlation using the sample correlation coefficient:
n
(R(()
Rime) Ramee (R Rproxy)
8
Li=1 (1 Ranu) EE
(Rpidxy
Rproxy)
We can then use Fisher's z-transformation to compute the confidence intervals for After getting this bounded range, we can plug this bound into our framework to define a tighter reward uncer- tainty set. For example, correlation constraint in Equation 64 to be bounded by both Tlower and I Fupper The optimal solution under this new constraint can be similarly obtained using the approach in the paper.
use Flower for more pessimistic robustness. Or we can redefine the
min-max regret approach. more principled approach to addressing the uncertainty in r may come from regret-based perspective. Let J, (,(T) denote the worst-case return for given policy under a specific correlation level r, i.e., (x(TT = minne J , Rt.. The regret can then be defined as Reg(T, r) = max,- JT()" - (T), which quantifies the performance gap between the optimal policy under and the current policy. With this formulation, robust objective
59

---

Published as a conference paper at ICLR 2026
be expressed ming max, Reg(I,, (T, r), aiming to find policy that minimizes the worst-case regret across all possible values of r. This framework enables us to train policies that are robust to uncertainty in the correlation parameter We think this is promising future direction, especially for cases where r may be misspecified during training. As studied in (Sadek et al. (025). minimax- regret may provide strong robustness guarantees under distribution shifts for In such settings, methods like Prioritized Level Replay (Jiang et al. (021) and recent progress in (Monette et al., 2025) could be adapted solve the problem by sampling multiple and solving Equation 27 in our paper. We should note that the reason these frameworks are potentially applicable is that our formulation admits closed-form solution for the inner minimization. However, the main challenge lies in estimiating the occupancy measure. An interesting direction for future work is investigate whether policy gradients can be approximated without explicitly oceupancy estimation.
60


Reviews:
[{"review_id": "kO7uIsye8q", "reviewer": "Reviewer_b81V", "summary": "This paper addresses the issue of reward hacking in reinforcement learning, where an agent may optimize for an imperfect proxy of the reward function which could lead the agent to diverge from the intended true objective. To tackle this, the authors take a robust approach to optimize for the feature weights of the true underlying reward from an uncertainty set if feature weights defined by $\\chi^2$-divergence. This allows them to model this as a max-min optimization problem over the proxy rewards which are constrained via correlation with the true rewards of the system, which the authors then extend to linear rewards. Ultimately, they prove convergence of their algorithm and sample complexity bounds on the occupancy estimation.", "questions": "How could this extend to other uncertainty sets?\n\nMinor things:\n- $\\gamma$ should be defined prior to equation 1 where you define the objects in the MDP tuple.\n- It would be nice to see one of the algorithms appear in the main text as well as the convergence bounds stated formally in section 3.3 to help highlight the contribution of this work.\n- Missing space between \"latent\" and \"(true)\" on line 1026.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is relevant and highly motivated by practical implementation. It builds upon the state of the art to improve upon interpretability and derive a tractable objective for the non-convex robust optimization problem. The paper is very clear and progresses fluidly.\n- A closed-form solution is proposed and derived for the worst-case reward feature vector of the adversary by transforming this vector into a whitened version of itself, $\\tilde{\\phi}$, such that the $Q$-function becomes the identity matrix.\n- From what I could determine, the proposed method is backed up by a strong theoretical analysis. Subsequently, the authors provided much empirical validation of this theory to verify their claims as well as sufficient detail to reproduce these experiments.", "weaknesses": "- In practice the correlation between a proxy and rewards, $r$, is unknown. The authors briefly mention this in the appendix and use a grid search to find this. However, as the author's mention, there is not a principled method for selecting the optimal $r$ and thus it may not scale.\n- An assumption is made that the true rewards lie within the defined uncertainty set. In practice, this may not always be the case. The proposed reference policy may not provide sufficient coverage of the feature space which could lead to the problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5MOCqALGbS", "reviewer": "Reviewer_CXL3", "summary": "Building off Laidlaw et al (2025)'s definition of \"r-correlated\" proxy reward functions, the authors introduce a robust optimization framework aimed at reducing reward hacking. Their method focuses on maximizing the minimum reward achievable across all possible r-correlated reward functions. This leads to a loss function that, while similar to that of Laidlaw et al, is distinct in its exact regularization term. Across several environments, they show that their approach achieves higher worst-case reward, compared to ORPO from Laidlaw et al, while typically maintaining similar expected reward.", "questions": "- Why is the same grid for r not used in all environments?\n- Why did the authors not define R_corr as all reward functions that are *at least* r-correlated to the proxy, rather than *exactly* r-correlated to the proxy?\n- How would one obtain the \"reference policy\" in practice?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has several strengths:\n- Intuitive conceptual contribution of optimizing worst-case reward over all reward functions that are r-correlated to the given proxy reward function. This seems like the natural progression from Laidlaw et al (2025).\n- Theoretically sound framework. They apply several tools from robust optimization to derive a tractable loss function, which though ultimately similar to Laidlaw et al (2025), differs in the exact regularization term, and they show this leads to improved worst-case rewards in their experiments\n- Improvement for linear reward setting. They are able to derive a refinement for the case where the reward function is known to be linear.\n- Better occupancy measure estimates lead to also improving ORPO. Both this work and Laidlaw et al (2025) use a discriminator network to measure the occupancy measure divergence between a given policy and the reference policy. The authors note that in the original ORPO implementation, the discriminator network is not fully optimized, and thus, the resultant policies still end up visiting states that are low frequency under the original policy. By training the discriminator for longer, they also obtain improved results compared to the original ORPO implementation.\n- Extensive experiments and ablations. The authors test their method on five environments that were designed for studying reward hacking, and include several additional results in the appendices (e.g. results with unnormalized rewards, results on robustness to \"r\", etc)", "weaknesses": "One paragraph that I thought was a bit strange was:\n> \"We adopted a similar approach used by ORPO (Laidlaw et al., 2025). For each environment, we first performed a grid search over several different values of r, and for each fixed r, we trained the policy using our algorithm. We then selected the rvalue that leads to the policy with the best expected worst-case return ... Notice that ORPO selects the optimal r that yields the best expected return under the true reward, which is infeasible in practice when the true reward is unknown during training. In contrast, our approach for choosing r is more practical.\"\n\nIt's true that the authors' approach is computable without the true reward unlike the process used in ORPO, but it's not much more practical, as without knowing the \"correct\" value of r, it's not clear which measure of the worst-case reward is most meaningful.  Therefore, I would hesitate to describe this approach as more \"practical\" than ORPO. Instead, it might be more helpful for the authors to directly reference the two practical methods for selecting r discussed in Appendix I.\n\nBy the way, the authors say they include results for all r that they considered in H.5, but then H5 only includes results for the Traffic and Tomato environments. Also strangely, they consider r \\in {0.3, 0.5, 0.9} for Traffic and r \\in {0.1, 0.4, 0.7, 0.9} for Tomato. Why do they not use the same grid for all environments? Also, it is somewhat strange that the grid for Traffic is not uniform either. Can the authors please include results for all environments?\n\nThe authors' process for selecting \"r\" also highlights one aspect of their framework that I found counter-intuitive. The set R_corr(r) as defined in Eq 4 includes all reward functions that are *exactly* r-correlated with the proxy. Apriori, I would have expected them to define this set as including all reward functions that are *at least* r-correlated with the proxy. Then, increasing r would monotonically increase the maximal worst-case reward over the functions in R_corr(r). As I understand it, with the current defn, this kind of monotonicity does not necessarily hold? And that is also why in the selection process for r that they use in the experiments (noted above), the highest r is not always picked (though, noise in the occupancy measure estimation and policy training process may also affect this empirically)? It seems odd that, under this framework, selecting r = 0.3 makes the policy robust to reward functions with exactly 0.3 correlation, but not to those with higher correlation, such as 0.6. Could the authors clarify this design choice?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oMNrBH5xdQ", "reviewer": "Reviewer_uNZH", "summary": "The paper studies robustness to reward misspecification by assuming access to a proxy reward $R_{\\text{proxy}}$ and a reference policy $\\pi_{\\text{ref}}$.\nIt defines an uncertainty set of true rewards whose Pearson correlation with the proxy reward under $\\pi_{\\text{ref}}$ equals $r$, and trains a policy to maximize worst-case return over that set.\nA change-of-measure via the Radon--Nikodym derivative (density ratio) was used, which results in a regularized optimization objective the authors note \"closely resembles\" occupancy-regularized policy optimization (ORPO) objective, differing by a regularization scaling and an extra penalty term inside the square root.\nConvergence guarantee of the proposed method is provided, and empirical evidence on four \"real-world inspired\" reward hacking environments is given.", "questions": "In its current form, the paper offers a tidy derivation and a small geometric refinement of ORPO, but it does not resolve the central mismatch between reward-space uncertainty and behavior-space objectives, and it relies on an unintuitive correlation hyperparameter and a reference-policy anchor.\nI encourage the authors to\n- clarify the relationship between reward function correlation under a reference policy and behavior difference;\n- clarify the implications of off-support vacuity and provide coverage-aware constraints; and\n- justify or estimate $r$ more principledly.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper gives a clean derivation of a max-min robust RL objective under a correlation-constrained reward set and reduces it to a tractable form via duality and change of measure.\nThe resulting penalty can be read as an \"orthogonalized\" $\\chi^2$ term, penalizing the component of occupancy shift not aligned with proxy improvement, which is a neat geometric refinement relative to ORPO's regularization.\nThe implementation details for ratio estimation and the comparison to ORPO will be useful to practitioners.", "weaknesses": "## Reward-space vs. behavior-space\n\nFirst, the core uncertainty model is reward-space, not behavior-space.\nCorrelation under $\\mu_{\\pi_{\\text{ref}}}$ ignores the well-known fact that many behaviorally equivalent rewards (e.g., potential-based shapings) can have arbitrary correlation with the proxy, while tiny changes in reward can flip the optimal policy near decision boundaries.\nAs a result, the method may penalize or ignore the wrong directions in practice; a behavior-centric formulation (e.g., safe improvement stated in terms of behaviorial metrics) would better match the problem that matters.\n\nThe paper repeatedly motivates \"guarding against reward hacking,\" yet provides no behavioral safe-improvement guarantee and even asserts that the correction term \"enforces robustness to potential reward hacking,\" which is not supported by a theorem.\nOver-pessimistic and over-optimistic rewards can both lead to suboptimal behaviors, and the agent may just learn a different reward hacking behavior.\n\n## Reference policy\n\nSecond, the entire construction is anchored to a reference policy.\nHow's it chosen? Does it need to be a good policy?\nThe correlation constraints are imposed under $\\mu_{\\pi_{\\text{ref}}}$, and are vacuous off its support; the adversary may set arbitrarily bad rewards in unvisited regions.\nConsequently, the guarantee weakens precisely where a newly learned policy might go.\nThe paper assumes $\\pi_{\\text{ref}}$ is given and does not offer a principled selection rule, beyond practical choices.\n\n## Correlation hyperparameter\n\nThird, $r$ is a hyperparameter that must be known or tuned. The authors state there is \"no principled method\" to pick it and resort to grid search over $r \\in \\\\{0.1,\\dots,0.9\\\\}$, which is unintuitive for a Pearson correlation meant to represent true-proxy alignment.\nResults also depend on the training/evaluation $r$ mismatch.\n\n## Succesor representation\n\nFinally, the linear-reward assumption $R(s,a)=\\theta^\\top\\phi(s,a)$ is long-established (**successor representations/features/measures**); the section omits direct attribution even though the subsequent optimization uses discounted feature expectations exactly as in that literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rOhd8W15vj", "reviewer": "Reviewer_46uB", "summary": "This paper aims to design reinforcement learning (RL) agents that remain robust to imperfect or proxy reward signals. The authors formulate reward hacking as a robust policy optimization problem over all proxy rewards that are $r$-correlated with the true reward, deriving a tractable max–min formulation. When rewards are linear in known features, the method incorporates this structure to yield more interpretable and robust policies. Experiments across several environments show improved worst-case performance compared to existing approaches.", "questions": "Please see the weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and clearly presented.\n\nIt addresses an important problem in reinforcement learning (RL) where the reward function is imperfect. There have also been recent papers on similar settings in language models, such as:\n- [1] https://arxiv.org/abs/2504.03784\n- [2] https://arxiv.org/abs/2405.11204\nConsidering the growing popularity of LLMs, it would be interesting to discuss the potential applications of this approach.\n\nThe proposed approach appears principled, and the theoretical results effectively support the empirical findings.", "weaknesses": "- Equations (9) and (13) do not include the importance sampling ratio, whereas Section 3.3 does. I think this is because the chi-square divergence involves this ratio—if so, the authors could clarify this more explicitly. \n\n- The authors claim to have developed efficient algorithms, but the computational complexity of the proposed method is not discussed. Including empirical comparisons of runtime and memory usage against benchmarks would make the paper more complete. \n\n- How are the feature mappings selected? Are the theoretical/empirical results sensitive to the choice or number of mappings?\n\n- The experiments could be strengthened by comparing with recent related works, such as:\n  - InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modeling\n  - RRM: Robust Reward Model Training Mitigates Reward Hacking\n  - Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)