Robustness in the Face of Partial Identifiability in Reward Learning
Authors: Filippo Lazzati
Alberto Maria Metelli
Venue: ICLR 2026 Poster (2026)
Keywords: Inverse Reinforcement Learning
Reward Learning
Preference Based Reinforcement Learning
Theory
Primary area: reinforcement learning
TL;DR: We propose to tackle the identifiability problem in reward learning with a robust approach.

Abstract: In Reward Learning (ReL), we are given feedback on an unknown target reward, and the goal is to use this information to recover it in order to carry out some downstream application, e.g., planning. When the feedback is not informative enough, the target reward is only partially identifiable, i.e., there exists a set of rewards, called the feasible set, that are equally plausible candidates for the target reward. In these cases, the ReL algorithm might recover a reward function different from the target reward, possibly leading to a failure in the application. In this paper, we introduce a general ReL framework that permits to quantify the drop in "performance" suffered in the considered application because of identifiability issues. Building on this, we propose a robust approach to address the identifiability problem in a principled way, by maximizing the "performance" with respect to the worst-case reward in the feasible set. We then develop Rob-ReL, a ReL algorithm that applies this robust approach to the subset of ReL problems aimed at assessing a preference between two policies, and we provide theoretical guarantees on sample and iteration complexity for Rob-ReL. We conclude with some numerical simulations to illustrate the setting and empirically characterize Rob-ReL.

Published as a conference paper at ICLR 2026
ROBUSTNESS IN THE FACE OF PARTIAL IDENTIFIABILITY IN REWARD LEARNING
Filippo Lazzati
Alberto Maria Metelli*
Politecnico di Milano
Politecnico di Milano
Milan, Italy
Milan, Italy
filippo. aazzata zat aaazzttzzipiii
a bertoma r iiaaaaa. www.mmeeeeepo.......m imi itt
ABSTRACT
In Reward Learning (ReL), we are given feedback on an unknown target reward, and the goal is to use this information to recover it in order to carry out some downstream application. e.g., planning. When the feedback is informative enough, the target reward is only partially identifiable, i.c., there exists a set of rewards, called the feasible set, that are equally plausible candidates for the target reward. In these cases, the Rel. algorithm might recover a reward function different from the target reward, possibly leading to a failure in the application. In this paper, we introduce general ReL framework that permits to quantify the drop in "performance" suffered in the considered application because of identifiability issues. Building this, we propose robust approach to address the identifiability problem in principled way. by maximizing the "performance" with respect to the worst-case reward in the feasible set. We then develop Rob-ReL, ReL. algorithm that applies this robust approach to the subset of Rel. problems aimed at assessing a preference between two policies. and we provide theoretical guarantees on sample and iteration complexity for nob-ReL. We conclude with some numerical simulations to illustrate the setting and empirically characterize Rob-ReL.
INTRODUCTION
Reward Learning (ReL) is the problem of learning reward function from data (Jeon et al. 2020). When the data are demonstrations, ReL is known as Inverse Reinforcement Learning (IRL) (Russell 1998), whereas when the data are (pairwise) comparisons of trajectories, Rel. is usually called Preference-basee Reinforcement Learning (PbRL) (Wirth et al. (017)) or Reinforcement Learning from Human Feedback (Kaufmann al. 2024).
The main strength of ReL is that the reward function that it aims to learn, referred to as the target reward, corresponds "a succinct and transferable representation of the preferences of an agent"
(Russell 1998 Arora & Doshi (021). As such, ideally, ReL allows the use of datasets of demonstra- tions and comparisons for variety of important applications, such as reward design (Hadfield Menell et al. 2017). Imitation Learning (IL) (Abbeel & Ng. 2004), risk-sensitive IL (Lacotie et al. 2019). preference inference (Hadfield-Menl
al. 2016), behavior transfer across environments (Fu et al.. 2017), behavior improvement (Syed & Schapire 2007). and, more generally, any task that can be carried out using a reward.
However, in practice, ReL has been successfully applied mainly to IL (Finn et al. 2016) and reward design (Christiano al., 2017). The primary obstacle to the adoption Rel. algorithms for other applications is partial identifabilityy (Cao et al., |021 Kim et al.) 2021 Skalse et al.) 2023). This arises when the available feedback (demonstrations, comparisons, or otherwise) does not allow to uniquely identify the target reward, but instead leads
of rewards (referred to the feasible set candidates for the target reward. As a
Metelli et al. [202T] 2023)) that represent equally-plausibe result, the recovered reward might differ from the target reward, potentially causing failure in the downstream application. As noted by several works (Cao et al. 2021 (Skalse et al.
2023 Finn et al.
2016), most existing ReL methods, including (Ng & Russell 2000) Ziebart et al. 2008
Boularias
et al.
2011 Wulfmeier et al. 2016 Christiano et al. 2017). are sensitive to this issue.
Corresponding author.

---

Published as a conference paper at ICLR 2026
The standard solution in the literature is to try to ensure that the feasible set contains (almost) only the target reward. This is typically achieved by collecting additional feedback (Amin & Singh 2016)
Cao et al., 2021 Schlaginhauten & Kamgarpour 20044] Lazzati & &Metelli] 2024) or by imposing
further assumptions on the available feedback (Kim et al. 2021). However, in practice, additional
feedback may not be available, and the added assumptions may be too strong.
A more powerful and general approach was recently proposed by Skalse et al. (2023). 3. Rather than requiring the target reward to be uniquely identifiable, they instead ask that the feasible set contains only reward functions that are "equivalent" to the target reward w.rl the considered application. Intuitively, this milder condition applicable only when we have prior knowledge of the intended
use (i.e., the application) of the learned reward, which is almost always the case (Ng & Russell 2000
Ziebart et al. 2008 Fu et al. 2017 Christiano et al. 2017).
However, this approach has two drawbacks. First, it is difficult to apply in practice because, except for simple feedback and applications. it non-trivial to verify whether the equivalence condition holds. Second, it is qualitative: if the feasible set contains a reward that is not equivalent to all others, then Skalse et al. (2023) classify the ReL problem as prone to failure, without quantifying how severe the difference is. Intuitively, if this is small, the downstream application might still be carried out nearly successfully.
In this paper, we present a novel general framework for ReL that enables quantitative considerations. Based on this, we propose an easy-to-apply robust approach for addressing the identifiability problem.
Contributions. The contributions of this paper are summarized as follows.
We introduce a new quantitative framework for ReL. (Section 3).
We propose a robust approach for tackling the identifiability problem (Section
We present Rob-ReI, an efficient algorithm implementing the robust approach (Section 5
We conduct some simulations to characterize the problem setting and the method (Section
All results are proved in Appendix while additional related work is discussed in Appendix
2 PRELIMINARIES
Notation. Given Ne N, we denote [M] == NJ.. Given a finite set X. we denote by |*|] its cardinality and by X the probability simplex on X. Given two sets X' and V denote the set of Siiplysssiiiiep.e
conditional distributions as A = (q: A² We use R to denote the non negative orthant in k dimensions. A vector DE R is a subgradient for a function h: - R at WE RR if, for all we Ie in the domain of h, it holds that h(w) i h(()) + c + u). Sometimes, we use (v, w)= oTw for the dot product of vectors v, w RR We say that a function d: x X - RR is premetric if, for all re X, we have d(xr,a) =0. O. Moreover, for any xE X, we denote the f2-projection onto set y as any point such that: [1)(z) arg minleey || w/12.
Markoy Decision Processes (MDPs). A finite-horizon Markov decision process (MDP) (Puterman 1994) is defined as a tuple M = (S,
H, so.p.r), r). where S is the finite state space (S:= S)), is the finite action space (A : [A), so E the initial state, H N is the horizon, pe _ is the
transition model, and r e P = Sr:S x A [H] - [0, 1]} is the reward. A policy is a mapping re II A==A5< We let P,, denote the probability distribution induced by π in M starting from 8o (we omit so p for simplicity). and E, denote the expectation w.r.t PT. The visitation distribution induced by π in M is defined as di (s. == P= 0, = a) for all s, a, h, so that (s..)) x di (s. for every h E [F]] We denote the set of all state-action trajectories as Ω == (S x A) trajectory = ($1, (8. (4 Ω. we define the "visitation distribution" d of w at each s, a, h as d: (s, a) = 1 a == ). Moreover, we let G(w;r): = >ne [#] nh (5 ax)) be the return of w under reward r R, and note that G(w:r)= (d=, r). We denote the expected return of a policy ı in MDP M as J (r:p) E,, [heeA (sh an)) ] = =(d", .)). the optimal policy ** as any policy in arg max, J (r:p). and the optimal expected return as (+:rr:p)) p) == max, J (r((r p). Finally, for any 3 >>0 and stochastic policy म, let J (r:p) == E Σ
Given a
(7h (hh ax) 3 foog TA ( [a)]ss))] be the entropy-regularized return (Ziebart 2010 Haarnoja et al. 2017).
2

---

Published as a conference paper at ICLR 2026
Feehack
Feedback type and Q7
Feasible set R₁
optimal expert Ng. & Russell 2000
Ir: Jn (r:p) - J (r;r))
S-MCE expert (Ziebart 2010
demonstrations R
Ire of = arg umxx J3(rz)) (e: J (r:p) i a (m: -
t-suboptimal expert (Poiani et al.
2024
BTL with (Christian et al.
2017 7
trajectory comparison (w* w2))
Fr : (ow'r)/ / ellunr)) In: G(uttr) & Gluftr))
hard preference (Jeon el al.
2020
BTL with q (new)
policy comparisun (+²,
(r 1 g ol" (rom) Dn o" (ran)
hard preference (new)
(rr s (mpp) a J (rp))
Table 1: A list of some feedback considered in literature. For simplicity, we have grouped different feedback f based on the quantity Q, obtaining the three categories in the first column. Note that the policy comparison feedback are introduced in this paper for the first time and capture the situation in which we are given a preference on the behavior of two other agents (more in Appendix B). MCE stands for "Maximum Causal Entropy". while BTL. abbreviates the "Bradley-Terry-L.ue
model.
Reward Learning (ReL). In the literature [Russell| 1998 Jeon et al. 2020 Skalse et al. 2023)
ReL is defined as the problem of learning unknown target reward r" from certain amount of feedback, Le., data, like demonstrations (Ng & R 2000) or trajectory comparisons (Wirth et al. 2017). that "leak information" about r". The ultimate goal is use the recovered reward for some downstream application (Skalse et al., 2023), such as finding the optimal policy (planning). The concept of partial identifiability (Cao al. [202]] Kim et al.) [202T] Skalse et al.) (023) refers the existence of multiple rewards that are equally plausible candidates for the target reward with respect
to the given feedback. This set of rewards is called the feasible set (Metelli et al. 2021 2023)
3 A QUANTITATIVE FRAMEWORK FOR REWARD LEARNING
In this section, we present a new framework for studying ReL problems. Beyond modeling feedback in simple yet flexible way, our framework crucially models applications in a quantitative manner paving the way to new approaches to Rel. (e.g.. see Section
In our framework, we define a ReL. problem (Russell 1998 Jeon et al. 2020 as pair (F,g), where F F= {ffi;s is set of feedback and g an application. Informally, the feedback F represent what we know about the unknown target reward r", while the application g represents what we want to do with it. In the following two sections, we formalize these important concepts.
3.1 FEEDBACK
A feedback f relates a known quantity 2, with the unknown target reward r". We consider as feedback only those statements that can be translated into a constraint on r" of the type r" E Ry, where R2 C R is some set of rewards associated with feedback f. that we call feasible set. See Table T for a list of popular feedback and their corresponding feasible sets.
For instance, saying that "policy Q, π is optimal for (i.e., the "optimal expert" (Ng & Russell 2000) entry in Table [T is an example of feedback drawn from the IRL literature, and it is equivalent to saying that r* E R, = fr E R J=" (r:p) = (r:z)). Another example of feedback, taken from the PbRL literature, is "trajectories Q5 = (W², ,w²)) are such that the return under wf is no more than that w (i.e., the "hard preference" (Jeon et al. (2020) entry in Table II. and corresponds to r" E R2 = fr E R: G(w";r) & G(w²;r)). Note that our formulation is very flexible and allows us to work with almost any feedback we desire, such as "given Q, (s, a, w), the reward r" of the pair (s, a) is 80% of the return of w",, corresponding to r' R2 {r E R r(s, a) = 0.8 (8-G(wr))..
If we are given multiple feedback F (ftt. then we can combine them to obtain a smaller feasible set R, of candidates for r". Formally, we define the feasible set of J as the intersection R₂= == N. R, of the feasible sets of all the feedback in F. Note that R, = R, for every i, meaning that combining multiple feedback permits to reduce our "uncertainty" on r- If R5 # {r*), then we suffer from partial identifiability.
In Appendix A.2 we provide a comparison of our framework with the qualitative framework of Skalse al. (2023), while in Appendix D we provide quantitative discussion on model selection through our new framework.
3

---

Published as a conference paper at ICLR 2026
Application
Set X.
Loss Lg(r,))
Imitation of Abbeel Np 2004
(r:p) (r(pp))
Planning in (Christianio
et
2017
II
J (rrpp') (m(ppp')
Constrained planning with c (Schlaginhuuien & Kamgarpour 2023
II
max,ell,a J (*(r))
A Assessing trajectory preference wi ,0
B
|a - (Glw!:r) (r))
Assessing policy preference m!.m²
R
|a (J (r:p) J (r:n))
Learning a reward (Ramachandran & Amir 2007
x r12
Table 2: A list of some applications considered in literature. Note that used He { J (=(c:) & k). where e is the cost and k the threshold.
3.2 APPLICATIONS
We define an application g as pair (xg, Lg), where X, is a set, and C. : R x X, - R, is "loss" function. An application g is carried our by choosing x Xg. which results in suffering from a loss (r"",, 2)) (see examples in Table 2) To solve a Rel. problem (F,g), we must carry out the application g while incurring the minimum possible loss, i... we must select an object E X, for deployment such that the loss Cg (r",, xx) is as small as possible. Thus, ideally, the goal is to output:
x² E arg min (g(r",,')
x'ex
However, a Rel. problem not an optimization problem because the target reward r", and therefore the loss Cr(r",.), :), are unknown. For this reason, the function Lg is defined over all rewards r R, with the meaning that (r,-) quantifies the loss we would suffer if r were the target reward r*.
An example of application g is the well-known IL problem (Ho et al. 2016 Osa et al. 2018 (see "imitation of .t Table D). where we aim to output a policy x that "imitates" some given policy ie.. that matches its expected return under the unknown r* Thus, (r,z)) (r:p) - J (r:p)), with the intuition that, if r-rr, then the error of x is (g((r.))
set Xg II= and
Another example of a ReL application g is planning (see "planning in p"" Table 2. which arises in many contexts, including reward design (Christiano al. 2017) and transferring behavior (Fu et al. 2017). Here, we aim to find a policy x with the largest possible expected return under r- in some environment with different dynamics p'. Thus, we have Xg II and Ly x) J (r:
As final example, consider the problem of assessing how much trajectory wt is preferred to w² by some agent (see Table - Assuming that r" models the agent's preferences, we can view this problem as an application g where X, = R and (r,r,) = | - (G(w":r) G(W2:r))).
Note that our framework can also be used in scenarios where the ultimate goal is learning r" (e.g.. because the application g is not known yet) by setting X, R. and using some distance between rewards for f.g. e.g., C, (r, = | rll2 (Ramachandran & Amir 2007) (see more Appendix C
Remark 3.1. In this section, we considered feedback and applications g that are fully known. in the sense that all the quantities (e.g. policies, transition models and other parameters) involved in the definitions R. Ny. by are known exactly. However, in practice, these quantities are unknown and must be estimated from finite samples. We will consider the snite-sample regime in Section 5
4 A ROBUST APPROACH TO TACKLE PARTIAL IDENTIFIABILITY
In the previous section, presented a framework for formalizing ReL problems. In this section, introduce a novel, principled way to solve a ReL problem g). i.e., to select the object x e X, to deploy. We begin by reviewing the existing approaches adopted in the literature.
Existing approaches. In the literature, the majority of existing ReL methods, including the most
popular IRL (Ziebart et al. 20088] Boularias et al. 2011 Wulfmeier et al. 2016 Finn et al. 2016) and PbRL (Christiano et al. [011]] et al.) 2018 Jeon et al.) 2020) algorithms, solve a ReL. problem (F,g) by first drawing an arbitrary reward from the feasible set R₂, and then deploying the object T that minimizes the "loss" (F, w.r.t. the recovered reward 7, as if were the true target reward
T arg min CF,FF x²). Z'Ex
(1)
4

---

Published as a conference paper at ICLR 2026
However, there are two main problems with this approach: (t) there is no clear motivation for why this choice of œ is a "good" way for solving the ReL problem, because in general 7 # r* (Amin & Singh 2016 Cao al. 2021 Kim al. 2021), and so the minima of (, -) might incur very large value of the true loss Lg(T", : (12) none of the aforementioned works provides an estimate of the true loss (r 3) incurred by selecting ã. thus providing information whether the chosen I can be safely deployed or not. In the next two paragraphs, we present our approach to overcome these limitations (i) and (zi).
Our approach. We propose robust approach that arises quite naturally once the framework introduced in Section 3 is adopted. Specifically, the idea is that, whatever choice x' E Xg we make, since we know that the target reward r' belongs to the feasible set Ryy then, in the worst-case, the true loss (r(r", x²) incurred by deploying x' is upper bounded as:
(r ,x) < max (T,
reR
For this reason, we propose to deploy the object Fg E Xg that minimizes the loss associated with the worst possible value that r" can take:
FF E arg min max (r,
x'ex reRs
Some observations are in order. First, whether the optimization problem in Eq. ( can be solved efficiently depends on the specific application g and feedback F in question. Next, we choose to be robust (minimax) because our problem setting is not Bayesian (Ramachandran & Amir 2007). ). i.e.,
we do not have distribution over the set of rewards, but we only know that 7 R,=. Finally, note that many IL algorithms (e.g., be seen as adopting our robust approach (see Appendix A).
(e.g.,[Abeeel Abbeel & Ng (2004); (Syed & Schapire (2007); Ho et al. 200 can
Quantifying the error. Eq. represents principled way to solve ReL problems that, unlike the approach commonly adopted in the literature (see Eq. D provides worst-case guarantees. However, we cannor solve every ReL problem (F. by merely outputting Fy.,g because although #y.g y is the choice with the smallest worst-case loss, the loss associated with FF, might still be too large in the worst case. In other words, there are ReL problems (F,g) that cannot be solved even robustly, because we cannot guarantee that, in the worst case, with the information available, the true loss falls below some pre-specified threshold. Intuitively, this happens when the application g requires significant knowledge about r", which is not sufficiently provided by the feedback F. In such cases, we must collect additional feedback if available; otherwise, we must tolerate weaker guarantees than those worst-case.
For these reasons, is important to quantify the loss suffered in the worst case by FF Thanks to our new framework, we can compute it as:
IF max Lg(r, min max (r, rER
I'ER, reRz
Since IF.g measures how wninformative is F for we call it the uninformativeness of F for g-
A special case. We conclude this section with some observations on the special case where X₂ = R. ie.. when we aim to output reward. Here, the robust choice 2F,g in Eq. (2) can be interpreted as the Chebyshev center (Alimov & Tsar' kov 2019) of the feasible set R, in the premetric space (R, Lgy. Building on the properties the Chebyshev center, it possible derive interesting results. The main of these is that the robust reward F. E R does not necessarily belong to the feasible set RR,, which is rather counterintuitive, especially because the entire Rel. literature has focused on recovering reward function from the feasible set R, (moredetails in Appendix
ROB-REL: A ROBUST ALGORITHM FOR REL
The goal of this section is introduce an algorithm for solving ReL problems using the robust approach presented in Section 4 To this aim, we make two important observations.
Solving ReL problems using the robust approach is not merely an optimization problem. In fact, F, g usually have to be estimated from finite data (see Remark 3.1.
5

---

Published as a conference paper at ICLR 2026
No single and simple algorithm solve robustly all ReL problems. Indeed, with infinite data, depending on F and g. Eq. (2 exhibits different properties from the optimization viewpoint.
For these reasons, now focus specific subset of ReL problems in the finie-sample regime and we present Rob-ReL, a provably efficient algorithm for solving this subclass of ReL. problems.
5.1 THE FAMILY OF REL PROBLEMS SOLVABLE BY ROB-REL
We consider an interesting and explanatory family of ReL problems (F.g) that have received limited attention in the literature. Specifically, we let g be the application of "assessing policy preference" (see Table 2 between policies π target MDP without reward M (S,A, A H, 80, p), i.e., we aim to output a scalar J. as close as possible to the difference in their expected returns:
X =
- (J (r:p) - (rip))|-
In addition, we require that the set of feedback F= FD u FrC U FPc contains only demonstrations FD. trajectory comparisons Fre and policy comparisons FPc feedback (see Table I. of the following kind.
We allow for mp > demonstrations feedback FD {fD.t) mm where each fb.; a "t,-suboptimal expert" feedback (Table D in some MDP without reward MD (S, so,,,, with t E [0, JI,, corresponding to the feasible set:
Rí... = fr f :: J (r:pp)) > J (r:pp.) - tJh. vi [mp]]
(4)
Moreover, we allow for mec e 0 trajectory comparison feedback Frc = - where each frc, is "hard preference" feedback (Table D in some MDP without reward Mrc. (S, A. H, So,rC,i, Prc,i), with feasible set:
R/c, {r R: G(wle,AT) & G(wic,Ar)),
Vie [mrc].
(5)
Finally, we allow for mpc 3 0 policy comparison feedback FFC = (SpC.)) where each SPc, is "hard preference" feedback (Table in some MDP without reward MPc, = (S, H1, s0, PPC.)) with feasible set:
R fr E R: J (uees(r:Pp,) (PiP)),)) & (ries(;;
Vi [mpc]]
(6)
Finite data. To keep things realistic, we assume that the policies n² r1,=² WD.i. m/c, Tie and the transition models p. PD.i. Prc,i, prc,i are not known and must instead be estimated from data. We adopt a mixed offline-online setting that is common in the literature (e.g., see GAIL Ho & Ermon 2016)). To estimate the policies ,², ADi T TPC. we assume access to batch datasets of trajectories D1, D22 DD,s, Dic, Dic, obtained by executing the policies the corresponding environments M, MD.i, MMCc, for n² _,,, nD,i, "Cc.. nic., trajectories, respectively. To estimate the transition models p. PD. PTC, Pec.s, we assume access forward sampling mode F for each MDP without reward M, M,MD. MICii Mrc, from which we can collect N, ND. NTc, Nrc. trajectories, respectively.
5.2 ROB-REL
We now present Rob-ReL (Robustness for Reward Learning, Algorithm I. ReL algorithm for solving this family of ReL problems using the robust approach from Section Specifically, this setting, even with infinite data available, the robust approach (Eq. 2 requires solving the following optimization problem, where the constraints define the feasible set Ryt
FFFg E arg min max lx'' (d d== .!
E'ER
s.t.: max J (r: PD.:) (d t
Vi [mp]]
d re r)
Vi e [mrc]]
All the considered environments share same A.A. H because we work with rewards on this domain. A forward model (Dann& & Brunskill 2015 Kakade 2003) of an MDP M' permits to collect trajectories from M' by exploring will.

---

Published as a conference paper at ICLR 2026
Algorithm 1: Rob-ReL
Algorithm 2: PDSM-MTN
Input iterations K, feedback data
Input :objective î., iterations K
// Est imation:
Estimate d= _d ,dmu. . Te via Eq.
(8)
Ay +
2 ro - 0
Estimate Po., RF-Express Menard et al.
D+
[>> 0: |A]2 & s) 0.1..... K do
2021)
for k
17 Optimization:
Tx+1 1l ad, (rk,
mx- PDSM-MIN(, K)
5
Aa41 Iall (Au adx ((rks
Mx + PDSM-MAX(L, K)
// Targets:
7 end
5k 8 FK < K -k=0 Tt
KK (Mx + mx) /2
mg (d -d' ,Px)
Ix (MK mxk) )/2
Return Sx. Ix
Return mk
(d d' di =0
Vi [mpc]]
However, with finite data, the quantities highlighted in blue are known. Therefore, Rob-ReL instead solves the optimization problem obtained by replacing these quantities with their estimates. The next two paragraphs describe these estimates are computed by Rob-ReL and the optimization method it employs.
Given dataset D of .f....,,,,, Sy z +1+1)) of trajectories collected by some policy n, estimate the visit distribution of π at all (s, h) e S x A x [H]
Estimation.
d a)
Σ sc[n]
11 s. a}.
(8)
Namely, with its empirical estimate on D. In this way, d= from the corresponding datasets D D22 DD. Dic. D Pcs (Line l). Next, to estimate pD. Rob-ReL executes RF-Express (Menard et al. (202I). Line 2). a minimax opptima reward-free exploration algorithm (Jin al. 2020a). In short, RF-Express collects ND,; trajectories from each MD.i, and then uses the resulting data to estimate PD.;. Note that, since p. Prc.i, and prc.; do not appear in Eq. 77. then Rob-ReL does need estimate them.
estimates d= d== d d'ic..,
Optimization. Let R₂= and E, be the empirical counterparts of R2= and L, obtained by replacing the quantities in blue in Eq. (7) with the estimates described above. Then, Rob-ReL addresses the optimization problem:
Fr.g arg min max (r,rxx)). r'RR rER,
This is a minimax problem with non-trivial constraints. Nevertheless, we can simplify it. Define M, f as the largest and smallest values of (āx²
F²² .>) over RS:
M == max
r), m min (a² d=² :)). R
(10)
reR
Then, we can rewrite Eq. 99) in more convenient form:
Proposition 5.1. It holds that: FF.g (M + m2/2.
As result, instead of directly solving Eq. (9) Rob-ReL computes Fy by first computing M and via Eq. (IO) (see Lines 3/4) and then combining the results through Proposition 5.] (see Line 5). Observe that the optimization problems in Eq. (10) convex, since both the objective functions and constraints are linear or convex (the pointwise maximum of linear functions convex (Boyd & Vandenberghe, 2004)). To solve them, Rob-ReL finds saddle points of the Lagrangian function using the primal dual subgradient method (PDSM, Nedić & Ozdaglar (2009). which alternates subgradient updates for the primal and dual variables. Specifically, for any 8i and
7

---

Published as a conference paper at ICLR 2026
A (A No. [((rA)) (d d E [m(m X (max, J (r:PD.)) <d= .r) ti)
NFC) 1 R mue the Lagrangian L of both problems in Eq. (10
is:
(11)
= iefmell dd d r) Ema (f a r))
Then, the subroutines PDSM-MIN (Algorithm and PDSM-MAX (Algorithm B1 Appendix E invoked by Rob-ReL at Lines
aim to compute maxx30 millyeer L(r, A) and minAço maXre [r, A) by alternating between one subgradient step for r and one for A. Here,
0 the step size, is hyperparameter (see Theorem 5.3 for principled choices of a and s). The expressions 8, (rr, rk: AR) and 0 Zrr,, >ạ)) denote subgradients of I. w.rt. r and X evaluated at (rk, A&) (see Appendix E.2 for their formulas). To compute max, J=: PD..). ), both subroutines make use of the backward induction algorithm (Puterman (1994), see Appendix
Finally, recall from Section 4 that we are also interested in quantifying the worst-case loss IF.,, that might be incurred when solving problem (F,g). This computed Ix by Rob-ReL Line based on the following result:
Proposition 5.2. holds that: Ir.g max, nRR, (r, = (M - -m)/2
5.3 THEORETICAL ANALYSIS
We now show that Rob-ReL is both computationally and sample efficient. To this aim, make the assumption that the feasible set R5= contains a strictly feasible reward F, which is common in both the optimization (Nedić & Ozdaglar 2009) and the RL (Ding al. 2020 literature:
Assumption 5.1 (Slater's condition). There exist & > 0 and F R such that:
max, J (F: - (d - t4 < -
Vi e [mp]
(d d 7) & f (d dri ,r), & -
Vi [mrc]]
Vi [
Then, prove the following result for Rob-ReL:
Theorem 5.3. Let (F,g) be Rel. problem described in Section 5./ for which Assumption 5.1 holds. Let (0,2H) and (0, 1). If we set s 4H1/E + v +[ll/E)=+SAII)) and (1(16((1 + symm mrr + ), then, with probability 1 8, Rob-ReL satisfies:
C,(r", & IIa+t and [F - -īxl E
with number of samples:
n. n. up..s npc,s) m/c,s < 3 =0(5 OSSAHS
6202 log )
No. < δ
SAH
S log z)) N. NTC,s, Nrc.1 = O.
252
and number of iterations:
1312
VH { 7)
(1
H H H(mp mrc mpc) ( (.A)
2)
K KEO
VSA
E22
Simply put, Theorem 5.3 tells us that coobReLL enjoys sample and iteration complexities that are polynomial in the quantities of interest S, A, 11, 1, log 1. g mi, mTrc, mpc. In Appendix E.6 we other forms of applications and feedback. while in Appendix E.5
show how extend Rob-Ret. we discuss possible extension to non-tabulat environments. Finally, we observe that Rob-Rell can also be used for estimating the worst-case loss max,eR, (r, 2) incurred by deploying an arbitrary object x X, (see Appendix F.4.1 for details).
Proof Sketch. Define M := max, (d= d=² r) and m == min,eR, (d= - d= r). Then, after having shown that FF. M + mm/2 and Ir. (M m)/2, the result can be proved by upper
8

---

Published as a conference paper at ICLR 2026
iN
fm
1.0
0.8
0.4
0.2
0.0
Figure 1: (Left) The target environment considered in the experiment. (Right) The feasible set with r" and f, fM, Fm. Axis B, S, I' refer to the reward values of B, S and T.
bounding the estimation error M - MM + m ml and the iteration error MM - Mx Im m upper bound the estimation error in Lemma [.2] by first bounding the error estimating the visitation distribution of the policies (using Hoeffding's 's inequality) and the transition models (using the results in Menard et al. (2021)), and then showing that, under Assumption 5.1 all M, M, m, m are saddle points with bounded optimal Lagrange multipliers (using Lemma 3 Nedic & Ozdaglar (2009)). Regarding the iteration error (see Lemma E.6p, we exploit the theoretical guarantees of the PDSM (Proposition of Nedić & Ozdaglar (2009))
NUMERICAL SIMULATIONS
In this section, we present an illustrative experiment aimed at (i) exemplifying the class of problems solvable by Rob-ReL and (ti) providing a graphical intuition the robust approach. For these reasons, we focus low-dimensional problem. We refer the reader to Appendix F2 for a more extensive empirical analysis of Rob-ReL
Environment. We use the following state-action space (S. A. H) model the road depicted on the left of Figure There are 12 states E S obtained by combining three lanes (left (L), center (C), right (R)) with four items ball (B), a square (S), a triangle (T), or nothing (N)). The action space contains three actions A faz, ac, an). that aim to bring agent to, respectively, the left, keep the current lane, or go to the right, respectively. The horizon is H 5.
ReL. problem. We consider the ReL. problem (F.g) in which agent, Alice (e.g., human), has a preference over the items B, and T, that can be modeled through a three-dimensional reward r* [rr rss r where each component represents the value that Alice associates with visiting states containing items B. S and T (r* is zero for N). r" is unknown to our learner, and for the experiment we set
[0.7, 0.1, 0.2]. The application consists of assessing how much prefers the policy T² that always takes action an, w.r.t. policy π², that always selects environment constructed using (S, H) and some dynamics 80. P (see Appendix FT. This value AJ((") = J (r":p) (r":p) 0.39, i.e., Al iice's preference for xI over x. has an "intensity" of 0.39. To simulate some feedback JF from Alica Ae, we randomly generated some demonstrations, policy comparisons and trajectory comparisons feedback consistent with r" (1 + 2 + 3 feedback total). The resulting feasible set R.F, along with r* is plotted in Figure (right).
in an
Simulation and results. Our goal is output scalar R that is close possible to AJ((") = 0.39 without knowing r", but only that r* belongs to the yellow set R, in Figure We have executed Rob-ReL for K 1200 iterations with step size 0.01 using for simplicity the exact values of policies and transition models, obtaining fx = 0..2 Mr 1.02, corresponding to the rewards in RR that provide the smallest and largest values of AJ (see the blue f.. and green
f. rewards Figure The output of Rob-ReL is, therefore, 2 F 0.2 (corresponding to
'The code for running simulations can be found at httpst//tgptiiteee com/ fili pppo 1 izzati/
Rob- Rell

---

Published as a conference paper at ICLR 2026
the black reward fr.g in Figure and Σ5.g 0.82. Thus, know that the distance between Fy.g 0.2 and AJ(r") ) 0.39 most Zy.g 0.82 the worst case, and that can be reduced by collecting more feedback. Moreover, looking these numbers, approach had not been adopted, i.e., an arbitrary reward r in R,= had been used for prediction, then the worst-case error might have been doubled approximately 1.64 (e.g., if far and r* fm).
realize that, if our robust
7 CONCLUSION
In this paper, we presented unifying and quantitative framework for studying ReL problems. We then introduced a principled approach for solving ReL problems in general and we described Rob-ReL, an algorithm tailored specific subset of ReL problems that uses our approach.
Limitations and future work. The main limitation of this work is that the proposed algorithm, Rob-ReL, does not address all ReL problems. Therefore, we believe that future research should focus on developing additional algorithms that adopt the robust approach to tackle settings not covered by Rob-ReL. this context, we also note that there may be ReL problems for which our robust approach leads to an intractable optimization problem complex estimation problem. Therefore, it will be important to develop meaningful approximations in such cases.
ETHICS STATEMENT
This paper presents work whose goal is advance the field of Machine Learning. There many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
REPRODUCIBILITY STATEMENT
The proof of Theorem appendix also includes several additional theoretical insights, each accompanied by its own proof. For the numerical simulations, further details are given Appendix and we include with the submission the code necessary to replicate the experiments.
outlined in Section 5.3 and provided in full in Appendix E.3 The
LLM USAGE
We relied on LLMs during paper writing solely to correct grammar and enhance clarity.
ACKNOWLEDGEMENTS
This publication was funded with the contribution of Ministero dell' Università e della ricerca pursuant to D.D. 7206 of 17 April 2025 - BANDO FIS 2. Project FIS-2023-02599 (Starting Grant), title: "Unified Learning from Diverse Human Feedback" (HUmLm)). CUP: D53C25000710001.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference Machine Learning 2/ (ICML), 2004.
Alekh Agarwal, Nan Jiang, Sham M. Kakade, and Wen Sun. Reinforcement Learning: Theory and Algorithms. 2021. URLLLtts:://Etteeoryyoooo
qithub.
A. R. Alimov and I. G. Tsar'kov Chebyshev centres, jung constants, and their applications. Russian Mathematical Surveys, 74, 2019.
Kareem Amin and Satinder Singh. Towards resolving unidentifiability in inverse reinforcement learning. 2016.
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297: 103500, 2021.
Abdeslam Boularias, Jens Kober, and Jan Peters. Relative entropy inverse reinforcement learning. In International Conference on Artificial Intelligence and Statistics 14 (AISTAT' 2011), pp. 182-189, 2011.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Stephen Boyd, Jhon Duchi, Mert Pilanci, and Lieven Vandenberghe. Subgradients, 2022. Notes for EE364b, Stanford University.
Daniel Brown, Scott Niekum, and Marek Petrik. Bayesian robust optimization for imitation learning. In Advances in Neural Information Processing Systems 33 (NeurlPS), pp. 2479-2491, 2020.
Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learning. In Advances in Neural Information Processing Systems 34 (NeurIPS), pp. 12362- 2021.
Jie Cheng, Gang Xiong, Xingyuan Dai, Qinghai Miao, Yisheng Lv, and Fui-Yue Wang. Rime: Robust preference-basee reinforcement learning with noisy preferences. International Conference on Machine Learning 41 (ICML), 2024.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30 (NeurIPS), 2017.
Fabrizio Dabbene, Mario Sznaier, and Roberto Tempo. Probabilistic optimal estimation with uni- formly distributed noise. IEEE Transactions on Automatic Control, 59:2113-2127, 2014.
Christoph Dann and Emma Brunskill. Sample complexity of episodic fixed-horizon reinforcement learning. In Advances in Neural Information Processing Systems 28 (NeurIPS), 2015.
L. Danzer, B. Grünbaum, and V. Klee. Helly's Theorem and Its Relatives. Proceedings of symposia in pure mathematics: Convexity. American Mathematical Society, 1963.
Dongsheng Ding, Kaiqing Zhang, Tamer Basar, and Mihailo Jovanovie. Natural policy gradient primal-dual method for constrained markov decision processes. In Advances in Neural Information Processing Systems 33 (NeurIPS). pp. 8378-8338 -0998, 2020.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning 33 (ICML), pp. 49-58, 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. In International Conference on Learning Representations 5 (ICLR), 2017.
Uri Gadot, Esther Derman, Navdeep Kumar, Maxence Mohamed Elfatihi, Kfir Levy, and Shie Mannor. Solving non-rectangular reward-robust mdps via frequency regularization, 2024.
Tuomas Haarnoja, Haoran Tang. Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based based policies. In International Conference on Machine Learning 34 (ICML), volume 70, pp. 1352-1361, 2017.
11

---

Published as a conference paper at ICLR 2026
Dylan HadfieldMMenell Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In Advances in Neural Information Processing Systems 29 (NeurIPS), 2016.
Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems 30 (NeurIPS), 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Newral Information Processing Systems 29 (NeurIPS), 2016.
Jonathan Ho, Jayesh K. Gupta, and Stefano Ermon. Model-free imitation learning with policy optimization, 2016.
Jessie Huang, Fa Wu, Doina Precup, and Yang Cai. Learning safe policies with expert guidance. In Advances in Neural Information Processing Systems 31 (NeurIPS), 2018.
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems 32 (NeurIPS), pp. 8022-8034, 2018.
Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: A unifying for- malism for reward learning. Advances in Neural Information Processing Systems 33 (NeurIPS), pp. 4415-4426, 2020.
Chi Jin, Akshay Krishnamurthy, Max Simchowitz, and Tiancheng Yu. Reward- exploration for reinforcement learning. In International Conference on Machine Learning 37 (ICML), volume 119, pp. 4870-4879, 2020a.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory 33 (COLT), volume 125, pp. 2137-2143, 2020b.
Heinrich Jung. Ueber die kleinste kugel. die eine räumliche figur einschliesst. Journal für die reine und angewandte Mathematik, 123:241-257, 1901.
Sham Machandranath Kakade. On the sample complexity of reinforcement learning. University of London, University College London (United Kingdom), 2003.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback, 2024.
Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in inverse reinforcement learning, In International Conference on Machine Learning 38 (ICML), pp. 5496-5505, 2021.
Jonathan Lacotte, Mohammad Ghavamzadeh, Yinlam Chow, and Marco Pavone. Risk-sensitive generative adversarial imitation learning. In International Conference on Artificial Intelligence and Statistics 22 (AISTATS), volume 89, pp. 2154-2163, 2019.
Lcon Lang, Davis Foote, Stuart Russell, Anca Dragan, Erik Jenner, and Scott Emmons. When your ais deceive you: Challenges of partial observability in reinforcement learning from human feedback. In Advances Neural Information Processing Systems 37 (Neur!PS), pp. 93240-93299 2024.
Filippo Lazzati and Alberto Maria Metelli. Learning utilities from demonstrations in markov decision processes, 2024.
Filippo Lazzati, Mirco Mutti, and Alberto Maria Metelli. How does inverse rl scale to large state spaces? a provably efficient approach. In Advances Neural Information Processing Systems 37 (NeurIPS), pp. 54820-54871, 2024a.
Filippo Lazzati, Mirco Mutti, and Alberto Maria Metelli. Offline inverse rl: New solution concepts and provably efficient algorithms. In International Conference on Machine Learning 41 (ICML), 2024b.
12

---

Published as a conference paper at ICLR 2026
David Lindner, Andreas Krause, and Giorgia Ramponi. Active exploration for inverse reinforcement learning. In Advances Neural Information Processing Systems 35 (NeurIPS), pp. 2022.
Manuel Lopes, Francisco Melo, and Luis Montesano. Active learning for reward estimation in inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases (ECML PKDD), pp. 31-46, 2009.
Pierre Menard, Omar Darwiche Domingues, Anders Jonsson, Emilie Kaufmann, Edouard Leurent, and Michal Valko. Fast active learning for pure exploration in reinforcement learning. In Interna- tional Conference on Machine Learning 38 (ICML), volume 139, pp. 7599-7608, 2021.
Alberto Maria Metelli, Giorgia Ramponi, Alessandro Concetti, and Marcello Restelli. Provably efficient learning of transferable rewards. In International Conference on Machine Learning 38 (ICML), volume 139, pp. 7665-7676, 2021.
Alberto Maria Metelli, Filippo Lazzati, and Marcello Restelli. Towards theoretical understanding of inverse reinforcement learning. In International Conference on Machine Learning 40 (ICML), pp. 24555-24591, 2023.
A. Nedié and A. Ozdaglar. Subgradient methods for saddle-point problems. Journal of Optimization Theory and Applications, 112:205-228, 2009.
Andrew Y. Ng and Stuart J. Russell. Algorithms for inverse reinforcement learning. In International Conference on Machine Learning (ICMI. 2000). pp. 663-670 2000.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J. Andrew Bagnell, Pieter Abbeel, and Jan Peters. An algorithmic perspective imitation learning. Foundations and Trends@ in Robotics, 7(1-2): 1-179, 2018.
Riccardo Poiani, Gabriele Curti, Alberto Maria Metelli, and Marcello Restelli. Inverse reinforcement learning with sub-optimal experts. 2024.
Martin Lee Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In International 2007.
Joint Conference on Artifical Intelligence 20 (IJCAI), pp.
Nathan D. Ratliff, J. Andrew Bagnell, and Martin A. Zinkevich. Maximum margin planning, In International Conference on Machine Learning 23 (ICML 2006), pp. 729-736, 2006.
Paul Rolland, Luca Viano, Norman Schürhoff, Boris Nikolov, and Volkan Cevher. Identifiability and generalizability from multiple experts in inverse reinforcement learning. In Advances in Neural Information Processing Systems 35 (NeurIPS), pp. 550-564, 2022.
Stuart Russell, Learning agents for uncertain environments (extended abstract). In Proceedings of the Eleventh Annual Conference on Computational Learning Theory 11 (COLT), pp. 101-103, 1998.
Andreas Schlaginhaufen and Maryam Kamgarpour. Identifiability and generalizability in constrained inverse reinforcement learning. In International Conference on Machine Learning 40 (ICML), volume 202. pp. 30224-30251, 2023.
Andreas Schlaginhaufen and Maryam Kamgarpour. Towards the transferability of rewards recovered via regularized inverse reinforcement learning, 2024.
P.R. Scott. An extension of jung's 's theorem. The Quarterly Journal of Mathematics, 42:209-212. 1991.
Lior Shani, Tom Zahavy, and Shie Mannor. Online apprenticeship learning. In AAAI Conference on Artificial Intelligence 36 (AAAI), pp. 8240-8248, 2022.
Jour Skalse and Alessandro Abate. Quantifying the sensitivity of inverse reinforcement learning to misspecification. In International Conference on Learning Representations 12 (ICLR), 2024.
13

---

Published as a conference paper at ICLR 2026
Joar Skalse, Matthew Farrugia-Robess Roberts, Stuart Russell, Alessandro Abate, and Adam Gleave. In- variance in policy optimisation and partial identifiability in reward learning. In International Conference Machine Learning 40 (ICML), volume 202, pp. 30033-22003533 -2008,,, 2023.
Umar Syed and Robert E Schapire. A game-theoretic approach apprenticeship learning. In Advances in Neural Information Processing System 20 (NeurIPS), 2007.
Luca Viano, Yu- Yu-Ting Huang, Parameswaran Kamalaruban, Adrian Weller, and Volkan Cevher. Robust inverse reinforcement learning under transition dynamics mismatch. Advances in Neural Information Processing Systems 34 (NeurIPS), pp. 25917-25931, 2021.
Andrew J Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson. Reward-free RL is no harder than reward-aware RL in linear Markov decision processes. In International Conference Machine Learning 39 (ICML), volume 162, pp. 12430-22435 2022.
Wolfram Wiesemann, Daniel Kuhn, and Breç Rustem. Robust markov decision processes. Mathe- matics of Operations Research, 38(1):153-18) 183, 2013.
Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference- based reinforcement learning methods. Journal of Machine Learning Research, 18:4945 4990,
2017.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce- ment learning, 2016
Lei Zhao, Mengdi Wang, and Yu Bai. Is inverse reinforcement learning harder than standard reinforcement learning? In International Conference on Machine Learning 41 (ICML), 2024.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human
feedback from pairwise or k-wise comparisons. In International Conference Machine Learning 40 (ICML), volume 202, pp. 43037-43067, 2023.
Brian D. Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy, 2010.
Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence 23 (AAAI), volume 3, pp. 1433-1453 2008
14

---

Published as a conference paper at ICLR 2026
A ADDITIONAL RELATED WORK
In this appendix, we provide comprehensive presentation of the main related work of this paper. We group the related work into four categories: papers that "ignore" partial identifiability because it does not create issues in the applications that they consider, papers that explicitly address partial identifiability by looking for sufficient conditions that guarantee that it does not create issues, papers that aim estimate the feasible set in a provably-efficien manner, and a miscellaneous of other papers. Next, in Appendix A.1] we mention that some IL algorithms can be seen as adopting our robust approach. Finally, in Appendix [A2 we provide a thorough comparison of our quantitative framework introduced in Section 3 with the qualitative framework provided by [Skalse et al. (2023).
Works that "ignore" partial identifiability. The partial identifiability in ReL. is topic that dates back to the seminal works of [usssll| ( (1998); Ng & Russell (2000). However, in many existing Rel. works Ng & Russell (2000); Rathiff et al. (2006); Ziebart et al. (2008); Ziebart (0100); Boularias et al.) (2011): Wulfmeeir et al. (2016); Finn et al. (2016 Christiano et al. 2). partial identifiability is not considered. The reason that these works consider ReL. problems Fig) where partial identifiability does not create issues, because all the rewards in the feasible set R, behave as the unknown target reward p* w.r.t application g. Many of the mentioned papers are IRL works where the application g is II, and the set of feedback F { contains a single demonstrations feedback f able to provide enough information for II.. In particular, Ng & Russell 2000); |atliff et al. (2006) introduce the heuristic of "margin maximization" to extract a reward from the feasible set that facilitates the IL task, and then perform planning on it to find the imitating policy to deploy. Ziebart et al.] (2008); Ziebart (2010) introduce new Rel. feedback consisting in demonstrations collected from maximum (causal) entropy policy and present an algorithm that recovers an arbitrary reward from the corresponding feasible set (the authors let the inner optimization algorithm break the ties). [Bolarria et al. (201
Wulfmeier et al. (2016); Finn et al. (2016) adopt the model Ziebart et al. (2008) and extend its algorithm by, respectively, adopting a model-free approach, using neural networks for parameterizing the reward function to learn, and approximating the objective to speed-up the learning process under unknown dynamics in high- -igge-imensional continuous systems. Finally, we mention Christiano et al. (2177) that, in the context of PbRI., adopts maximum likelihood approach to recover an arbitrary reward from the corresponding feasible set (also Christiano et al. (207)) let the inner optimization algorithm break the ties) from finite data.
Works on partial identifiability. When the aforementioned algorithms have been applied to other applications (i.e., by using the corresponding recovered reward function), then partial identifiability turned out to be problem, as observed empirically for instance by Finn et al. (2016). who tried to transfer the recovered reward to a new environment. Consequently, many works addressing the identifiability problem have appeared Amin & Singh| (2016); Fu et al. (2017); Cao et al. (2021):
Skalse et al. (2023); Schlaginhaufen & Kim et al. (202T): Viano et al. (202T); Rolland et al. g
Kamgarpour (2023) 2024): Lang et al. (2024). In particular, Amin & Singh (2016) consider access to demonstrations feedback from multiple environments to reduce the size of the feasible set. Similarly,
Cao et al. ()2TI) consider demonstrations feedback from multiple environments using multiple
discount factors to improve the identifiability of the target reward, while Kim et al. (202T) focuses on the properties of the considered MDPs. Fu et al. (2017) assume that the target reward state-only and that the environment satisfies certain additional properties to guarantee that the feasible set contains only rewards that can be transferred new environments in the same way the true target reward. Viano et al. (202T) propose a robust approach for the reward transfer application that is rather different from ours. Specifically, the authors propose to deploy the policy that minimizes the loss w.r.t. the worst possible transition model in a certain rectangular uncertainty set Wiesemann et al. (2013) centered in the transition model of the target environment. [Rolland et al. (2022) studies the identifiability problem using as feedback demonstrations from multiple experts, and focuses on the application of reward transfer. |Schlaginhaufen & Kamgarpour 20033 2024) provide a similar study but use, respectively, different kinds of feedback and different method for studying the similarity of the environments, [kkals et al. (2023) present a general study of the identifiability problem in both IRL and PbRL, and we provide a detailed analysis of this work in Appendix A.2] Finally, [ang et al. (2024) study the identifiability problem in PbRL assuming that the feedback is based only on partial observations of the environment.
15

---

Published as a conference paper at ICLR 2026
Works that estimate the feasible set. A recent line of research has focused on studying the sample
complexity of estimating the feasible set Metelli et al. (2021): Lindner et al. (2022); Metelli et al.]
(2023)
Zhao et al.] (2024); Lazzati et al.] (2024b a): Poiani et al. 2024). All these papers consider the
IRL problem settings where the feedback consists in demonstrations from an optimal or e-optimal MataallllyyMMiiiiii
Poiani et al. (2024) expert's policy. Specifically, MMtelll et al. (202T) is the seminal work in this context. It assumes availability of a generative sampling model of the environment in the tabular setting, and provides an upper bound the sample complexity for estimating the feasible set. This result is completed by Metelli et al. (2023) who present a lower bound in similar setting. Lindner al. (2022) and Zhao al. (2024) focus on a forward sampling model in tabular MDPs, and provide upper bounds to the sample complexity. [Lzzati et al. (2024a) extend these theoretical results to the Linear MDPs Jin et al. (2020) setting. Instead, Lazzati et al. (2024a) and Zhao et al. (2024) consider the offline setting where only batch datasets of demon are available, and provide results under some concentrability assumptions. Poiani et al. (2024) present both lower and upper bounds for estimating the feasible set in tabular settings assuming suboptimal expert.
2024) represent a problem setting in which the true reward is unknown, but known to belong to a given uncertainty sel R. There are some similarities and differences between robust MDPs and our robust Rel. formulation. In particular, in both geward-Robust MDPs and our Eq. (2). the goal is to be robust against the missing knowledge of the true reward, which is known to belong to certain set of rewards, that is called fcasible set in our ReL setting, while it is called uncertainty set in the context of Robust MDPs. However, there three crucial differences. First, in robust MDPs, there is single application, i.e., finding a good policy under the unknown reward, while in our ReL setting there can be a variety of different applications. Second, and most importantly, in robust MDPs the uncertainty and known, while in ReL. the feasible set must be estimated from finite data (e.g., demonstrations or trajectory/policy comparisons). Third, in the literature, the uncertainty set in Robust MDPs is almost always rectangular, while in ReL the shape of the feasible set can be different and more complex.
Robust MDPs. Robust MDPs (Wiesemann et al. 2013 Gadot et al.
is given
Others. Ramachandran & Amir (207)) adopt a Bayesian approach in the context of IRL. It assumes prior the set of rewards R, and assumes that the feedback provide likelihood. The authors focus on two specific applications. i.e., IL and learning r", and propose to minimize the expected loss. In some way. |Ramachandran & Amir (2007) be seen as deploying the object that minimizes the average loss w.r.t. some known distribution over r' (instead of our worst-case loss), and Brown et al. (2020) that generalizes this approach using risk measures. We mention also Zhu al. (2023). that in the context of PbRL make a "pessimistic" choice of policy given "confidence" set of rewards. From a high-level perspective, their pessimistic approach is very close to our robust approach. However, their proposal aims to address estimation issues, while ours concerns identifiability issues. Jeon et al. (2020) introduce a framework for combining multiple and various ReL feedback, that are all modeled using a Boltzmann distribution constructed indirectly through the target reward. [Cheng et al. (2024) adopt a robust approach to PbRL, but the considered robustness aims to address a noisy feedback, and not partial identifiability. Finally, mention Huang et al. (2018) that adopt robust approach for policy optimization using multiple demonstrations.
A.1 A NOTE ON IL WORKS
In the IL literature Osa et al. 2018). there are some algorithms that can be interpreted as adopting the robust approach presented in Section 4 Specifically, such algorithms aim solve the following optimization problem:
E arg min max (r:(p:)) JT (rip)). r'lll rR
(12)
where से is the expert's policy and R is some set of reward functions Ho et al. (2016), e.g., rewards linear Abbeel & Ng (2004) or convex Syed & Schapire (2007) in some known feature map ø S A + R. If we look Eq. (12) through the lens of our framework, then i can be seen as the robust choice of policy that minimizes the worst-case loss Cg = (r: J (r:p) over the feasible set RS R, and Xg IT.
16

---

Published as a conference paper at ICLR 2026
A.2 COMPARISON WITH SKALSE ET AL.. (2023)
The work of Skaase et al. (2023) deserves its own section because it introduces a ReL. framework that shares similarities with ours. Thus, it is important to remark the differences.
Similarly to our framework, also the framework of Skalsse et al. (2023) provides a model of feedback that always corresponds applications g. Specifically, [Skalse et al. (2023) model an application g as a function g : R - Xg. and, for every reward e H, they let g(r) be the object that should be deployed in case reward were the target reward r". Crucially, this model is qualitative, namely, it does not contemplate the possibility that another object pair of rewards r, r' E Ry in the feasible set, it must hold that g(r) = g(r') (since r* E R,, this guarantees that no object other than g(r*) can be deployed). Instead, in our quantitatiie framework, we allow for this to happen as long as the loss (r"",,2) xz) of x is sufficiently small.
feasible set R S R. The difference lies in how they model the
*g(r") can be deployed, and so the authors require that, for every
In other words, the framework of [Skalse al. (2023) and ours give birth to different sufficient conditions on whether a set of feedback JF can be used for carrying out an application In particular, the framework of [Skalse et al. (2023) requires that:
(13) i.e. that all the rewards in the feasible set Ry prescribe the same object x g(r"), the object prescribed also by r". Clearly, this is qualitative, and if we add single reward r' with g(r') # g(r") to Rr,, then the framework Skalse et al. (2023) concludes that we cannot use for g anymore. Our framework, instead, provides more general quantitative sufficient condition. Specifically, for some threshold A > 0, our framework requires that there exists at least an item x E Xg such that:
RS S 9 (9(r)),
(r, & A
Vr E Ry-
(14)
Intuitively, if we set A O, then we recover Eq. (3). because we would be requiring that all the rewards r in the feasible set have the same minima of (r, -), and so, that they all prescribe the same item. Instead, by enforcing Eq. (I4). we are basically asking that it is possible to find at least one item (i.e., in Eq. 14). that suffers from true loss upper bounded by A, i.e., Eq. (4) guarantees that:
Cg(r" A
since r" E Ry.
A simple example can help in recognizing the advantages of our framework. Consider MDP with a single state s, three actions 41, a2 03 and horizon H = 1. Assume that we have received some feedback F that results in the feasible set R, (ri, containing only two rewards, and that the application g consists in outputting the optimal policy under r*. Then, Then,[Skalse et al. (2023) tell us that we can solve this problem the optimal policies induced by "1 and r2 coincide (in other words, if it is irrelevant whether r" is or r2). However, what about the situation in which, e.g.:
if ai
A
if ai if a2
as)
if 02
r2(8,a)
0 if a
03
0
if = a3
Clearly, Y11 makes a optimal, while 12 makes a2 optimal. However, as long as A is small, o1 is an almost-optimal policy also for 72, thus, intuitively, it "will not problem for g if we deploy a even if r" = r2".. Note that our framework allows for this situation, by telling us that, as long A is sufficiently small, then partial identifiability is not an issue for this problem, and we can solve it (by deploying the policy that plays 11 better, based on the robust approach presented in Section 4 by deploying the policy that plays a mixture of a1, 02 with equal probabilities). In addition, note that A corresponds to the uninformativeness I F and, thus, can be computed.
To sum up, our framework has the advantage of allowing for quantitative considerations, while the framework of Skalse et al. (2023) cannot.
17

---

Published as a conference paper at ICLR 2026
B MORE ON THE Policy Comparison FEEDBACK
In this appendix, we provide additional details on the new policy comparison feedback (see Table introduced in Section 3.
In the IRL. literature, we are given a dataset of demonstrations, i.e., state-action trajectories D [w"), d= collected by executing some (expert) policy πτ. This setting models the situation in which we observe an agent doing a task many times, and the assumption is that the agent behavior is guided by the reward
In the PbRL literature, we are given two trajectories and a preference signal between them. This setting models the situation in which agent expresses preference between two trajectories, i.e., the agent observes two trajectories and says which one it prefers. In doing so, the choice is guided by the target reward r".
The policy comparison feedback introduced in Section 3.1 concerns with the scenario in which we have an expert agent (i.e., the agent with r" in mind) that expresses a preference (or, more generally, any statement) between two datasets of demonstrations collected by (but not necessarily) two other agents. Simply put, it can be seen a mix of the demonstrations and the trajectory comparisons feedback types.
For example, assume that we observe two agents A2 demonstrating the task of driving a car, and they provide two datasets of demonstrations D2 = {w] policy of A₁ and x² is the policy of A2. Assume that we do not want to learn the reward function that guides the behavior of A₁ nor A2, but we aim to learn the reward of a third agent E (i.e., r" is with preference signal (or, more generally, any
}; ~ d= D2 = =w"]; ~ d=² where n² the
the reward of E). Thus, can ask E to provide statement between the demonstrated olicies TI', r² between the behavior of A1, A2. For instance, can show to E the video of how A1, A2 drive, and we can ask him who drives better. Then, can use this feedback to infer r' and carry out any downstream application
C WHEN THE APPLICATION IS TO DEPLOY A REWARD
In this appendix, we provide additional insights on the class of ReL problems in which the application g requires the deployment of a reward function, i.e., Xg = R. Before that, we need some additional notation.
x X - R. be a premetric in X. If, in addition, d satisfies (i) d(x, y) 0 if and only if x = y (identity of indiscernibles), (fi) d(a,, y) d(y, .)) Vx, E X (simmetry), (zii) d(æ, y) & d(<x + d(z, y) Vx, =,=E E X (triangle inequality). then we say that d metric. Let d be premetrie in set X. The Chebyshev center Alimov & Tsar'kor kov (2019) of a set y S X' is any of the points in arg min maXzey d(z ([x v). The Chebyshev radius of y defined as minzex maxysy d(xr y), while the diameter of y is max d(x,, y). Moreover, given reward and an environment with dynamics p. we denote by II (T+(p)) ; arg max, J" (r:p) p) the set of optimal policies under r in p.
Additional notation. Let X' be a set, and let
Setting. There are situations in which are interested in using some given feedback J for learning a reward function X, R. In these cases, the loss by R. x R. - R_ can be seen as a premetric in the set of rewards R. These settings can happen for various reasons, like:
The ultimate goal is to learn r". Then, of rewards, like that induced by the 2-norm (see Ramachandran & Amir 2007
as loss some "standard" metric in the set
L2(r, r') == |r-irri r'12
(15)
or by the max norm:
(r(r,)) = |r - -er-rlla 1l == max (nn(s, r (s, 8.0,
(16)
The true application g is unknown revealed a posteriori, and just know that it belongs to a given set of applications. For instance, in case we know that we want to use r" to assess the return of some trajectory w', but we do not know w' yet, then solving all the
18

---

Published as a conference paper at ICLR 2026
ReL problems corresponding to every possible trajectory may be inefficient (there is an exponential number trajectories). Thus, we can simply learn a reward and then use it at a later time when will observe w'. In such setting, we can use loss:
(TR( max (((w;)) G(w;r")) wESt
(17)
which quantifies the worst-case error among all possible trajectories.
For some reason, we aim to pass through a reward function solve the true application. For instance, in case the application is planning in some MDP with transition model p (see Table 2. then we can add an intermediate step of passing through a reward function by using as loss:
Cet,p(r,r)) =- J= (r':))
min ('('':p), (el)((r:))
(18)
which considers, among all the optimal policies of r. the one that maximizes the subopti- mality under r'. Another similar example is the application of computing the greedy policy (::r) e arg max A r(:, a) in stationary environment Zhhu al. (2023). Here, for some distribution E we can use:
LOR r') = Eg-pl max - ('s,, xF (s;
(19)
aeA
Note that we introduced the notion of premetric because not all of the introduced dissimilarity functions are metrics:
Proposition C.1. For any p. p. then LpL.ss fco.as CoR.D are premetrics. Moreover, there are some p, P such that:
Cco.: lack the identity of indiscernibles; CrL.ps LGR.p are not simmetric:
LPL.ps LGR.p lack the triangle inequality.
Proof. It immediate that, for any transition model e As x A x [H] and distribution e AS, the distances LPI.p: Lco.p. LGR,P are non-negative for any pair of rewards r, e R, and also that they are all 0 when = Thus, we conclude that they are premetrics.
Now, consider the identity of indiscernibles property defined earlier. Given distance L. it does not hold if there exist r * r' s.t. Cg(r,r'= O. Concerning CpL.pp we see that any pair of rewards r, r' such that (r; = 11 (r':p) satisfies (r..p(r,r') = O. Thus, we can take r' to be, e.g., a multiple of to get ( there exists at least a (s, a, h) E S A [H] s.t., for all A [ (s, 0. Then, such triple does not contribute to the performance of any policy, and therefore any pair of rewards r, that coincide everywhere except for s, a, h satisfy (ro,p(r,r) = O. Concerning LGR,0- we can take r,r for which there exists a state & E S where maxas. r(s, a) = maxas (8, a), but the maximum is achieved by different actions arg max r(s.u) * arg max r' (8, a). Clearly, r # r', but LoR,p =0.
0. Concerning Lco.p. simply consider as p transition model for which
Consider now the simmetry property. For CPL.py take two rewards r,r' such that 11 (r:p) { III (r':p) = {
2 for some policies T n²2 Then:
- Jr (r'sp) = 0. (r.p((,)r) = - (r':p) min ("(r':)) J" = (r'sp) (rpp)(p)))))
Crpp(r ,r) = J (r:p)
min (ril*(r'))
("(r:p) = J (r:p) J (r: # 0.
Thus, (rt.p(r,r)) r²) t (r r), so simmetry does not hold. For Lco.p the simmetry property holds for any pair of rewards r, r' A:
(co,p(r:)) == max (r':p) - (r:p)
7
= =max
- (J"((:))- - (=((:p))
=max
-max||F((rpp)
Lcoop(r'
19

---

Published as a conference paper at ICLR 2026
Concerning La the simmetry property does not hold. To see it, consider a problem with single state s and two actions 41, a2, and let r,r" be two rewards such that:
if d1, r(s.a) {
5o
if a 01,
r((s
t0 if 02,
0.5
if 02.
Then, we have that:
aEA (r, r/) == Baupl max r' (s, - r'(,, 15 (8:r)) = r' r' = 0.5,
(r,,p(r' Es-pl
max maaxr(s, r(s, a) - r(s, n. (s: = r(). - rsss, = 1.
aeA
Finally, let us consider the triangle inequality property. First, we consider CpL Let r, r', r" E R be three rewards such that II (r: = (r* (I':p) = {T²). and take r" 0.5r. Then:
(r r)) J (r:p) J (r:p).
(rt.plt',r" +====== J (r": J= (r" = 0.5JJ* (r:p) 0.5.J* (r:p) = (r CrL,p("" rr) = _ (rip) - J= (r:p) =
Therefore, we have that:
( r)) > C r²) + (r
which proves that triangle inequality does not hold. Concerning Lco.p. triangle inequality holds for any p and r,r', ,,r',"" ,r"" E R:
(r' max | (r:p) J" (r"
max (r":)) J" (r':p)) + max J (T:pp) (p):
-co,p(r,,")) + (co.p(r", ,r),
where we have applied the triangle inequality of the absolute value and the fact that the maximum of sum is smaller than the sum of maxima. As far as LaR,p is concerned, we note that it lacks the triangle inequality property with a counterexample analogous to that for CPL.p. Consider a problem with single state (or supported only it) and two actions 01, d2, and consider the rewards r, r y" E R such that:
r(s,a) {
i ai.
[0 (0
ifa ai if a 0z
0.5 if d
((ss = {1
((s,
if 02
if 02
Then, we have that:
(F r) r(s,01) r(8,01) 1 La r²)) r"(,, 01) (s,
= 0.5 0.5LcR,
(a,, r) r(s,01) r(s,a))= =0
Thus:
LOR, 1>> LoR,p(r',
( ,r") + CoR,p(1" 0.5 1.5+0 O.
This concludes the proof.
Robust approach and Chebyshev center. In these scenarios, the robust choice FF presented in Eq. (2) corresponds to the Chebyshev center of the feasible set R, in the premetric space (R, L. Since Z,F.g E R. then we denote by ry.g. Moreover, observe that the informativeness Is.g (Eq. 3) can be interpreted as the Chebyshev radius of the feasible set Ry, and that, in the worst-case, the ReL procedure carried out by most algorithms in literature (see Section 4 Eq. [ can be seen as the diameter of the feasible set, that we denote by Dr.g- Finally, given arbitrary reward r E R = X we can define its worst-case loss as:
Cr.(rr max (r(r,r').
(20)
rER,
See Figure 2 for simple graphical intuition of all these quantities.
20

---

Published as a conference paper at ICLR 2026
R
R
I
ry.9
D
CF (r
Figure 2:
Illustration of the quantities of interest. is any reward.
In the following, we provide some interesting results on these quantities.
Some results. One of the most interesting results that, depending on F. the Chebyshev center might lie outside of the feasible set RRy. This is interesting because no Rel. work in literature has looked for reward function outside R, as far we know. Formally, we show that this can happen for LPL.p and Lx:
Proposition C.2. If the loss is Ly GrL.p. then there exists a (convex) set Ry for which rF.g g Ryy.
Proof. Consider simple MDP without reward with a single state 8, three actions 01 02 ag and horizon H = 1. Let nt , n be, respectively, the deterministic policies that play actions a d2 ag. In this context, take R, = (r. r'), where:
1
if a 01
0 if a 01
r(s,a)
if 02
r((s
1
a 02:
0.5 if 03.
0.5 a = ag
Since 11:(p;)) {m²} and II ()):p) {. then it is immediate that:
(P..D(T,r') = CPL =
The robust reward choice is any reward FF.9 s.t. (rF.g))))) = {-²)). Indeed, in this manner:
(rL.p(FF.s)) = LT = 0.5.
Thus, FFg $ R.F.
Note that hull of r'. - In this way, it is simple to see that no reward in this new feasible set can make n3 be the unique optimal policy. As such, the Chebyshev center still external R,F.
provide counterexample with convex feasible by using R, the convex
This concludes the proof.
Proposition C.3. Let the feasible set Ry be the 3-dimensional convex hull of rewards [1, [,,, 11,11,11, [1,1,-1] Then, if we take Lg Lov, the Chebyshev center is FF.g 00
which lies outside Rx.
Proof. See footnote in Dabbene et al. (2014).
To quantify how convenient our reward choice TF.99 w.r.t. what is done in literature (see Eq. have to understand how small is the ratio T Dy.g- However, since coincides with the ratio between the Chebyshev radius and the diameter of the feasible set R, in premetric space (R, Lg), then the answer depends on the specific R, and Eg at stake.
If C. is metric, then the error reduction is at most 50%:
Proposition C.4. If Ly satisfies simmetry and triangle inequality, then:
If 3 /
Proof. Let (11, arg max,, (r, r'), i.e., be the points in the diameter of the feasible Rx. thus Dr.g (g), Because of triangle inequality, for any Ri including the Chebyshev
21

---

Published as a conference paper at ICLR 2026
center rg.g. it holds that:
D & Lg(r1,)) (g(T)
If we take 18,9 and apply simmetry, then we see that fg(T1,1F.9) (o & max,RR, Cg(FF.g.T) = Ir.g. and also that (2) & max (r(.ggg))) -I.g. from which:
DF (g(r & 2I
Lg is induced by the 2-norm, we get an error reduction of least 1/v2:
Proposition C.5. If Cg = C2. then:
If & +
As an example of why it is not exactly 50%, think to the equilateral triangle or the simplex.
Proof. The result is immediate by an application of the Jung's 's theorem (ung 1901 Danzer et al. 1963 Scott 199T), that states that in every Euclidean space with dimensions, the Chebyshev radius r of a set with diameter D satisfies:
D.
2(n + 1)
It should be noted that, in absence of the triangle inequality, Proposition C.4 does not hold, and the radius IF.g can be zero when the diameter D is not, providing an "infinite" increase in performance: Proposition C.6. There exist P, P and feedback F such that the premetrics PL LoR, satisfy: IF((P..)) = 0 and (F((PL.)) H, IF((RR,)) O and DF.(GR.0) 1.
Proof. Let us begin with LpL.p- Consider a problem with horizon H where R, {r Di | [r:p)) the feasible set contains all the rewards that make least क, optimal policy. Consider transition model for which there exists a reward F E R s.t. (*:pp)) = J (F:p) = H, and for some other policy 7² it holds that: _=² (F: 0 (for instance, if we construct to be deterministic, then, this is possible). Then, since R,z contains also the reward r' that makes all the policies optimal (in particular 7r,), we have that:
DF.(P 2 LPL,D (T) J² -" = H.
Instead, we consider any reward r" s.t. II" (r" {Λ²) 3 (as our robust reward choice), then is clear that:
(P.PPL.))) max (rL, (r", = J= (r:p)) =. PER,
The other claim of the proposition can be proved with an analagous construction.
D SOME INSIGHTS ON MODEL SELECTION AND ACTIVE LEARNING
In this appendix, we provide some considerations on model selection and active learning that make use of the framework and robust approach introduced in the main paper. Here, for simplicity of presentation, we introduce new symbol Ay: will refer to feedback f as pair (Q,, Af), where A denotes the assertioonstatemmett that describes the relation between Q2 and r* and, so, gives birth the feasible set Ry. It is convenient to introduce & because it permits reason different feedback f.f f of the same type, i.e., sharing the same quantity Q = Q p, in a simple and intuitive way. We will call Ay an "assertion".
22

---

Published as a conference paper at ICLR 2026
D.1 DISCUSSION ON MODEL SELECTION
For simplicity of presentation, we consider a single feedback F = (ʃ3, with f = (Qç: (.Q.AA), A), and we assume that infinite data available.
Modelling feedback. In practical applications, we observe the data Q,, and we have to select the specific assertion y based on our knowledge on how Q2 has been generated.
Example D.1 (Feedback of driving safely). Let the unknown larget reward r" represent the lask of driving safely. Let 2 = {wi]s be a dataset of demonstrations collected by an agent N that "aims to demonstrate how to drive safely in certain environment", What assertion do we adopt?
We can decide to model the problem in different ways, i.c., there are always multiple assertions & that can be associated to the given data Q to try to capture the "true" relationship between Q and the underlying unknown r".
Example D.1 (continue). We can model Th as optimal policy for the "driving safely" task by using assertion OPT, corresponding to the "optimal expert" entry in Table we think that the demonstrated π sometimes takes suboptimal actions, then we might prefer using A MCF ("3-MCE
expert" in Table 7). Alternatively, we can simply assume that T at least e-oplimal for some 0, ie., J (r*:p) &2 (p) - C. We call this assertion ASU ("t-suboptimal expert" in Table L)
We incur in misspecification if f does not correctly describe the relationship between r" and Q (see Skalse & Abate (2024) for an analysis of misspecification in IRL).
Example D.1 (continue). we model data Q using assertion MCE, but, actually, xB is optimal (i.e., it follows & ),T),, then our feedback is misspecified.
Choosing the correct modelling assertion. The crucial question is: what is the best modelling assertion for the given data 2 Intuitively, any assertion that permits carry out the downstream ReL application g effectively is satisfactory. In other words, we can tolerate some misspecification error as long as the final outcome is acceptable.
Thanks to our framework, we can make these considerations more quantitative. As explained in Section 4 the uninformativeness Iif).g represents the minimum error that can be achieved in the worst-case for doing 9 using f (Q₃, f. Ay). Under the model of feedback f (i.e., the assertion A considered, the worst-case error cannot be smaller than I()),- Therefore, this is unsatisfactory if we aim carry out g with an error A I(/).9-
To solve this issue, we have to reduce the size of the feasible set R, so that its Chebyshev radius
I(f), reduces too. There are two ways for doing this. The best one (i) consists in collecting additional feedback f' obtain R2 n R2 S R, (e.g., through active learning, see Appendix D.2). However, additional feedback might not be available in practice. The other way (it) consists in imposing "more structure" to the problem by changing the assertion A to &p to restrict the feasible set a1)) to A.).
Example D.2. If we model Q, using A OPT we obtain a feasible set that is strictly contained into the feasible set obtained using assertion A Thus, whatever the application g at stake, A003, & I100,
By adopting more restrictive (potentially less realistic) model Ar, we reduce the "measurable" error I[Q, (hy at the price of a larger unknown misspecification error. This is fine as long as Ar is perceived as sufficiently realistic. If there is no realistic assertion with small enough value of informativeness, then we conclude that the application g cannot be carried out effectively with the only data Q,- In other words, the Rel. problemm cannot be solved with the desired accuracy.
D.2 ACTIVE LEARNING
In the Active Learning setting (Lopes et al., 2009). in addition to a given set of feedback F, we can choose to receive new feedback f' from set F {ff). Crucially, for any f E F.
f' = (Q ft, Ar), we consider the assumption Agr to be known, while the actual data Qp is revealed after our choice.
Example D.3. We might choose between feedback fi = (Q,, AOT). consisting of demonstrations in environment Mi (i.e., Q1).fom an optimal expert (i.e., A. OPT see Example 121 f2 (Q2, A MCE).
23

---

Published as a conference paper at ICLR 2026
made of demonstrations in environment M2 (i.e., Q2) from a maximal causal entropy expert (i.e., A M. After our choice, the expert demonstrates only one policy (i.e., either Q1 or Q2).
What the "best" choice of feedback f to receive without knowing its data O₁² Thanks to our framework, and specifically the notion of uninformativeness defined in Section the immediate choice is the feedback that, whatever its true data, is the most "informative":
Definition D.1 (Information gain). We define the information gain (ff of feedback f w.E.t. given as:
[GF = IF - Iyulfts
In words, (J) represents the reduction of uninformativeness (worst-case error) of the ReL. problem. Observe that 1G (f) since Iru(f).g & IF.g. i... the more feedback the less error.
Formally, the feedback that reduces the most the uninformativeness is the feedback fj from set F' (M), S (Qi, Az), that maximizes the worst-case information gain w.r.t. the data Q;:
arg max min IG ((Qi, Ar)))
Q:
Simply put, select the feedback that, whatever the true data will receive, we know that it will bring the larger reduction of informativeness, i.e., of error, for the Rel. problem.
E ADDITIONAL RESULTS ON SECTION 5
In this appendix, we provide additional results for Section 5 We begin by reporting the subroutine PDSM-MAX in Appendix F..] which is not present in the main paper. Then, we provide explicit computation of the subgradients of the Lagrangian function i for the estimated problem (Appendix We prove Theorem [5.3] in Appendix E.3 we provide miscellancous of other results in Appendix E.4 and we conclude with a discussion of possible extensions of Rob-ReL to continuous
E.2
environments (Appendix E.5 and by showing that Rob-ReI. ReL can be easily extended other Rel.
problems (Appendix E.6
E. PDSM-MAX
The pseudocode of the PDSM MAX subroutine reported in Algorithm 3 Note that analogous to that of PDSM-MIN (Algorithm(2) 2), with the difference that set D is mirrored w.r.t. D+, and that the signs of the subgradients update are reversed.
We remark that operators 1199, II._ IIl. Indeed, as explained in Boyd & Vandenberghe (2004), given vector RRd, we can implement the projection onto set ) := [k,, k2 (with arbitrary k < k2) as:
be implemented in ((d) time for vector in Rd.
hj
if &
[y(x))
ha
if > ka
otherwise
where [-]y denotes the j-th component.
Finally, we remark also that the explicit computation of the subgradients is explained in Appendix
E.2 SUBGRADIENTS OF THE LAGRANGIAN
Both subroutines PDSM-MA MIN and PDSM EDSM-MAX require the computation of the subgradients 2, eg of the estimated Lagrangian L (Bq. 11I w.r.t. r, A. We report Eq. (11) below for arbitrary r, A: i(r,A) (d=' Fr' .)) Σ, 1b (max, J (r: PD..) <d ,,) ts)
Σ, Noc(wties i r) + Σ, ( g
24

---

Published as a conference paper at ICLR 2026
Algorithm 3: PDSM-MAX
Input :objective I., iterations K
Ào -0
ro +-0 -0
3 D - { :: |A]a & s}
for k = =0,1..... K do
Tx+1 - (rx + ad, (ree Nu))
A++1 - lll (Ae - ad, ((rn)da)
end
FK
5 K Lk= Tk Return (d d=² .xk)
Then, the following quantities are subgradients of 7. evaluated at any r E R and A == Dre, dc, Ap]
arī(r, = (d" - -da) + E, Ab )
(21)
+ E. X (de(tdee dric)) + ", A (dFlee amic.:)) 02 [(r' ,X) m (r: (ā=ois ti
Vie [mD]
me P(r,, X) = (dttt () e [
xxp L X) = (aTte. g
Vi e [mre]]
where DD E arg max, J" PD.s represents an arbitrary optimal policy in the estimated MDP (S, A, H, PD.i,, for all [md] and d represents the visit distribution of TD in the corresponding estimated MDP without reward (S, H, So,D. PD.1).
an*
To that this is actually the subgradient of Z. simply note that 7 is linear in A, thus the subgradients simply correspond the gradients. Concerning ê, L. observe that L is differentiable in because contains the pointwise maximum of differentiable functions max, (r; :...). Thus, the subgradient
Zrr A) can be obtained as any convex combination of the gradients d= of the functions that attain the maximum (Boyd et al. 2022).
Ar: We remark that the computation of max, J Pb.)) and "D.i arg max, (F((::p:) can be done
exactly using the backward induction algorithm Puterman (1994.. Then, also d can be obtained exactly given $ and (S, A, H, S0,D,5 PD..) by simply starting from:
A. (s. = #b. [a]s)](s so,D.).
and then constructing the visit distribution for h 2.... ... H recursively using the relation Puterman (1994):
de (s,a) - (a|s) Σ
3ot (s
E.3 PROOF OF THEOREM 5.3
Theorem 5.3. Let (F,g) be a Rel. problem as described Section 5.1 for which Assumption 5.1 holds. Let e E (0,2H) and (0, 1). If we set s = 4H/ +
and
e/(16(( + symmp MTC + )) then, with probability
Rob-ReL satisfies:
C,(r*, & IF and IF Ix & e,
with number of samples:
n¹ x² ap nPC,3 nics δ SAHS
2222 log ))
25

---

Published as a conference paper at ICLR 2026
No. & Ö
SAH.
(s (st log z)) N, Nrc,,, Nrc. = O,
E²
and number iterations:
115/2
VH
H H Hmmp mre mpc) x2 $S.)
2) )
K 0
VSA+
E22
Proof. Define M, m as:
M == max r
(dt
d ,
min (d d .7) rR
(22)
Then, we can write:
,,(r",5mr0 IF & max rRR
Tx) -IF.g
M - m
max
kx m, M -āx
M + m
M + m
max
2
Mr + mK
+ m
M + m
MR + fik }
max
2
2
1
|M M + |mx - -ml)
2
1
MM M + M M + ||m m
+ mm ml
2
where at (1) we used that r* E Rr by hypothesis, at (2) we apply Lemma E.8 twice using the
definitions of IF and Cg. (3) we insert the definitions of Mr. fx computed by Rob-ReL, at (4) we apply triangle's inequality, and at (5) we add and subtract M, m and apply again triangle's inequality.
Similarly, note that:
IF
Fz,0
M m
Mr
mi
2
1
MK
M
I.
M
ml + Im ml)
Observe that MM M +
lū m represents the iteration error, while J. M| + Im m the
estimation error. Let EEST (0, 1), and define the good event & as the intersection of the following events:
Eg
sup redi
r) J (r:p) & cBST/2 Vi (1.21).
ED
sup reDt
D. r) T (r: po.)) EEST/2 vie [mol]
Ep..
sup rEM
max, J (r: PD.)) - J (r: (r:))).))
csst/2
Vi [mol]
(23)
Eyc,1
sup reat
(a r) j (r: PPc.))
Vi E [mvc]]. CEST/2
Epc,2
sup reat
( ,r) J (r: (r:
cEST/2
Vi [mpc]]-
Then, conditioning E, Lemma E.2 allows bound the estimation error, while Lemma E.6 permits to bound the iteration error.
The result follows by applying Lemma E. with GEST
26

---

Published as a conference paper at ICLR 2026
E.3.1 CONCENTRATION
Lemma E.1 (Concentration). Let EEST E (0, 1), ò E (0, 1). Then, the good event & Eg n Ep n
ED P E E,C,, where these events are defined in Eq. (3). holds with probability at least 1 8, with at most:
n n ,DD, ".D.i,,
3
SAHs
log
mpc + mp + 2
+2)
LEST
(24)
No. & Ö SAH 2
S log
mpc
mp + 2
2) 2)
5
EST
Proof. Rob-ReL estimates the visit distribution of the considered policies through their empirical estimates, defined in Eq. (8). Thus, events Eg. &D. Epc,1 Epc. can be shown to hold using the same
proof in Theorem 5.1 Lazzati et al. (2024a) (see also (Shani et (2022)) and union bound.
Regarding event & we can use the guarantees in Menard et al. (021) (see also Lazzati et al. (2024a))
The result follows by an application of the union bound.
We remark that, of course, in case myc mp 0, then the good event holds with a number of samples:
D Pcc i Pic.i: ND. = O,
2
SAH3 1 EST
n n² Ö
log
8
Nevertheless, for simplicity of presentation, we will mention only the bound in Eq. 24
E.3.2 BOUNDING THE Estimation Error
Lemma E.2 (Estimation error). Let E (0, (0]2H],6 8 E (0, 1). Make Assumption 5.1 and assume that event E holds with CEST < T0H
Ee
Then, we have that:
M
M + Im
ml <
Proof. We proceed for M, . (the proof for m, m is completely analogous):
M
M
min max L(r, A)
min max D(r, X) regt
AE
rest
XE
max L(r, A) max L(r. A) P6
max
X6D
rESt
(3)
L(r, X) -(rr, 2)
max max reat
XED
(ā=' 2r² .r) (dr' d=² .r)
max max reat
De
Σ X (max+ J P.o.s) (dto.;,)))
max J (=:(r: (r::)))) (dTToss
Apc (amte: - Joikee, (dite.. d=ic.s,)))
(4)
+ max | AED
< EEST
(5)
5H
ESST
where at (1) have applied Lemma E.3 and Lemma E.4 at (2) and at (3) the Lipschitzianity of the maximum operator. At (4) we use triangle's 's inequality. that event & holds and we upper bound with the 1 1-norm of X, and at (5) use the size of set D provided by Lemma R.4
27

---

Published as a conference paper at ICLR 2026
By choosing:
ÇE
ESS
10H
which is smaller than E/2 for e (0, (H) as required by Lemma E.4 we get the result.
Lemma E.3. Let I be the Lagrangian of the problem:
A(r,)) (d d" r) + Σ. N (max J (rirppn,) (d 1i)
(25)
E. A diies + Σ, Apo(aricc dric,e,))
Then, under Assumption 5.1 it holds that:
M = min max L(r, A), XeD
rest
m max min L(T; A), reDt
AED)
where D- correspond to 4H in Algorithms 2
and namely, D+ {} >
0 ||la|| &
(E) D- () & |||| & 4H/C).
Proof. Under Assumption 5.1 we have that Slater's constraint qualification holds. Thanks also to the linearity of the objective function and to the convexity of Ryy, we have that strong duality holds Boyd & Vandenberghe (204)), thus:
M min max L(r, A), rEDt
772 max min [rr, A).
A30 re
To prove the boundedness of the Lagrange multipliers corresponding to the saddle point, let (r* X*) be any saddle point for problem M (the proof for m is analogous). Under Assumption 5.1 we can apply Lemma 3 of Nedić & Ozdaglar (2009) to obtain that, for the reward F in Assumption 5.1
<d d= T) M
|
E
(d=²
d=² .T) +
2H
Thus, the values of the Lagrange multipliers in saddle points can be found in bounded sets D + Note that we upper bound again with 4//C to comply with the bound for the estimated problem in Lemma E.4
D_.
Lemma E.4. Make Assumption 5.1
Under the good event & with eEST E (0, E/2), we have:
M min max L(r, A),
XED
rEDt
ne max min [(r, A),
where D_
& 0| 4H/S) and D+ (1 0 | 4H/S).
Proof. The proof is analogous to that of Lemma
E.3 as long as we use Lemma E.5
Lemma E.5 (Slater's Condition). Make Assumption 5.1
Under the good event with EEST (0, E/2).
for the reward F in Assumption 5.7 we have:
max J (n.:)
amnn P) - 4 44-5
Wie [mp]
(d=to- d=her, ,)) <
Vie [mvc]] [[/c]]
(d Fe. .T) &
28

---

Published as a conference paper at ICLR 2026
Proof. The result follows directly by using the conditions in the good event &. For instance, concern- ing the policy comparison feedback, we have:
(grt. Toe,, (ārte. a (ori,,,,)+ r) ((dries dricss,)
< FEST (fdries- - -dhi:,,) ,))
EEST
By using that EEST (0,E/2) we get the result.
E.3.3 BOUNDING THE Iteration Error
Lemma E.6 (Iteration error). Let (0, 2H). make Assumption 5.1 and assume that event & with
EEST E (0, (/2) holds. Then, if we choose:
AH/E (AHH/)) SAH/4,
a c/(16H(1 8V mp
symp + mre + mpc)
we have that:
|M 1-MM + Im mxl cc
with number of iterations:
& 0 H
VH )1 +H/V/ HH(m + mrc mpc)( + SA) )
E (VSA+
Proof. We prove the result by imposing that both the terms JM MRl and Im - f k are smaller
than c/2. We present the proof only for |M Mxl, because the proof for Imm mx is completely analogous. For simplicity, define as:
4H
SAH
+
4
and note that, in this way, for the choice of s in the statement, we have:
4H E
+ l.
Thanks Lemma E.7 using this choice of obtain the following upper bound to the norm of the subgradients, that we call U::
2 + +
4H
U 2= 2 +
Observe that the choice of a corresponds to:
4
Using Lemma E.5 and Lemma (recall that we start with ro
E.7
we can apply Proposition of Nedic Ozdaglar 2009 obtain
0,
0):
| 2Ka
Mr M
aU2
(26)
M Mr
4H
4H
+ e)²
1f
aU²
(27)
g
Kal
E
2Kal
28
We now bound each of these terms.
Concerning the first term, i.e., Eq.
26 using that Px E Ri and the choice a = =<((AU2) we have:
SAH
Mx - M <
2ka + aU2
25 ASAHU2
cK
4
29

---

Published as a conference paper at ICLR 2026
which is smaller than c/4 if:
K SSAHU2
e²
Regarding the second term, namely, Eq.
27
we write:
4H
2
4H
2
4H SAH
4H aU2
M Mr <
+
+
Kal
6
E 2Kal
5 2
4H
2
4H
+
1)
2
4H SAH
E
Ç Kal
5
E 2Kal
4
2
4H
2
4H
2
4H VSAH
6
+
Kal
5
Ka
4
3 4H
16H
VSAH
4H VSAH
+
&
Ka
E
2
Ka
4H
(16H
VSAH
E Ka
8H3/2
16 &
+ VSA
+
{Ka
4
22U2H31
(16
6 4'
+ v SSA
+
KKe &
where at (1) we use that l > 2H/ and the choice of a = e/ )40 at (2) we use that / >
SAH
1 at
(3) we use that
a 2) /e Hil1
VSAH and at (4) we use that e/(AU3).
The resulting quantity is smaller than e/2 if:
3//2
16VH H + SA
128H
(VSA + 16V H
< c
4 =
K
ŞKe
&
E22
E
By inserting the definition of U, we get:
1
SA + 16
K
(VSA+
EE2
128H3/22
(SAA+ 16
2V + { +1 [H//F
14H
2
Ee2
VH
(1+ (1+ + SA) ) ))
O
(SSA+ +
EE
The result follows by noting that this quantity is larger than the upper bound derived for Eq. (26).
Lemma E.7. we execute PDSM-MIN PDSM-MAX for K iterations using any choice of
$ > O, then it holds that:
max max
{ |12,, 26) 10 L(r,
((), Awllla < 522 max / + (
RSK
where we denoted [F| := mp + mrc + mpc-
Proof. We prove the result for PDSM-MAX only, because for PDSM-MIN the proof is analogous.
Consider the subgradient expressions in Eq. (1.. We have (the dependence on is implicit in D_):
max (0, 2x)12 & max max 0 (r, A)ll2 K
rest AED
(d=
J + Σ, Ni (b )
max max
rESt XED
30

---

Published as a conference paper at ICLR 2026
+ Σ, Mic(diic dfics) + Σ APc (d - =fre:)) )12
C
- 122 " |bbl d... -2.0.111
& max maxX rest AED
|irel doics - dcic
111 2 + E. | Aivell@Atee - ::ic.
(2)
$ 2V H max (1+ 1111)
AED
< 2v H + MDD + mec + mpc max
A|[2
AE
2v H 1 symp + mrc + mpc
where at (I) we use that the PDSM projects onto those sets, thus rke A cannot lie outside, at (2) we use the fact that the 2-norm of every visit distribution at most v H, at (3) we use the relationship between the 1-norm and the 2-norm, recalling that À is (mp mrc + macc)-dimensiona.
Regarding the subgradients w.r.t. X, we can write:
(4)
max [rxī(rk. 2x) 12 $ max max (r, reat XED
Reg K
0xp (rr A), ...,, C. Ar,,)), ..., O..,Oxe (r:
max reg
2H1, ..... H, H
2H VMD + meC Mpc.
where at (4) that the PDSM projects onto those sets, thus Tk. A cannot lie outside, at (5) note that the subgradient vector is (mp + mTc + mpcc)-dmmensional. and write down its components, noting also that there is dependence on A. At (6) we use the formulas in Eq. (2T) and note that the difference between returns and expected returns is bounded in [-H, H, + H] because r R, and also that t; e fO, H] for all i [mp]] by hypothesis.
The result follows by joining the two upper bounds.
E.4 OTHER PROOFS AND RESULTS
Lemma E.8. Let d E N. and let y be subset of R.. Let h y - R be any function that attains both minimum and maximum in Y. and define Mr. == max tuyy) and my mingey h(y). Then:
Mr mh
min max k h(v) keR
2
yEy
MR + m.
arg min max h(v)). RE
2
yey
k: - M(v) = max { k mh, Mr - -ak-k kt
max yEy
wk e R.
Proof. For any E R. we can write:
max
h(y)
max max k h(y). h(y)) k3 yEy
ye
max
max k: yE
h(v). max h(y) k.
vE
mmxx
min h(y), max h(y) k
yE
yEy
max
l: mh, Mn - k
31

---

Published as a conference paper at ICLR 2026
Then, know that the maximum between two items is minimized when the two items coincide. Thus, the argmin must satisfy:
Mn. mp
mh
MA
Substituting this value into the previous expression, we get:
Mn
ma
My mh
MA. mh
max
mh. Md.
2
This concludes the proof.
Proposition 5.1. It holds that: FF.9 (M + mm/2
Proof. Apply Lemma E.8
Proposition 5.2. It holds that: Ty.g max max (r(55 (M -m)/2.
Proof. Apply Lemma E.8
E.4.1
USING ROB-REL FOR ESTIMATING THE WORST-CASE LOSS
Thanks Lemma H.8 cquivalently computed as:
realize that the worst-case loss suffered by any object Ng can be
max (r, max m. M r
This suggests that we can use Rob-ReL to quantify the error suffered in the worst case by arbitrary Xy. potentially the output provided by other Rel. algorithms. To do so, the estimate:
max x - mg. Mr- - z).
and note that: max r mK Mr 2 max m, m, M =z)| & max Im m IM - Mxl}
< m n + JM M7
(Im- ml |M M)) (lAR ml+ [M MD)
namely, the error be upper bounded by the estimation and iteration errors considered by Theorem 5.3 Therefore, we have the following corollary:
Corollary E.9. Let Xg be arbitrary. Under the setting of Theorem 5.3 with the number of samples and iterations specified in the statement of the theorem, with probabiliit 1 it holds that:
max Lg(r,x) max mx, Mx - 23) 2c. r
E.5 EXTENSION TO CONTINUOUS ENVIRONMENTS
Our algorithm, Rob-ReL, is designed for the "vanilla" tabular setting, where no structural assumption made the MDP the target reward. In this setting, a polynomial dependence the size of state space (see Theorem 5.3) is almost always present in RI. theory is and also unavoidable (e.g., see Agarwal al. (202I))
Scaling Rob-ReL to large or continuous state spaces while preserving theoretical guarantees is possible as long as additional structural assumptions the dynamics of the MDP and on the target reward are introduced, and if some changes are made to the algorithm. To make an example, assume that the MDP has an infinite/comtinouss state space and that a Linear MDP (Jin et al.) (2020)). i.e., the transition model and the reward r* are linear in some given d' d-dimensional feature map S [1], Then, robust Rel. can be performed with a variant Rob-ReL that:
Replaces the occupancy measures with the feature expectations (Arora & Doshi 2021
32

---

Published as a conference paper at ICLR 2026
Replaces RF-Express with any reward-free algorithm for linear MDPs (e.g.. Wagenmaker ct al. )2022);;
Applies the primal-dual subgradient method in the d-dimensional space of reward parame- ters.
Crucially, we believe that, through a proof analogous to that of our Theorem 5.3, should be possible derive sample and time complexity bounds that replace the dependence the size of the state space with dependence on the feature dimension d.
If instead wants scale Rob-ReI without preserving the theoretical guarantees, then do even need reward-free free exploration subroutine nor the structure of Linear MDPs, but assuming access to an online planner subroutine and assuming the target reward is linear in some known Φ suffice. Indeed, in such case, we could work with feature expectations instead of occupancy measures, and the online planner would allow us to estimate the feature expectations of the optimal policy in the demonstration feedback.
E.6 OTHER REL PROBLEMS
The algorithm presented in Section c can be straightforwardly extended, along with its theoretical guarantees, consider other kinds of feedback and applications that preserve the convexity of the problem.
For instance, if replace the application g with that of assessing a preference between two trajectories (see Table then it is clear that the scheme of the algorithm does not change, but we just have to modify the computation of the subgradients.
As another example, we might consider demonstrations from "bad" policies i.c., demonstrations from policies whose performance almost the worst possible:
("(rip)) ip) & min J (r";p) +
for some > Note that the reward-free exploration algorithm of Menard et al. (2021) works also in this setting, and its theoretical guarantees can be easily adapted to this setting.
We mention that we could also consider feedback a fractional comparison of policies n! 2 (or trajectories w ..)::
J= (rt:p) 2 aJ (r
for some a E (0, 1].
F EXPERIMENTAL DETAILS AND ADDITIONAL SIMULATIONS
We provide here additional simulations (Appendix F2 and details on the simulation in Section (Appendix F.Tp. Note that all the experiments have been conducted on server with processor Intel(R) Xeon(R) Gold 6418H", and required dozens of minutes for execution.
F.1 ILLUSTRATIVE SIMULATION
We describe here precisely the illustrative simulation reported in Section 6
F.1.1 TARGET ENVIRONMENT, TARGET REWARD r", APPLICATION
For the application g, we considered environment reported in Figure (left) and described in Section 5 with initial state the C lane, and the stationary transition model described below. Note that p depends only on the lane and the action played, thus, with abuse of notation, we write as:
1, if L 0, C
00.6, if L
0, L
pn pn(-LLar))
pn(-IC,
0.4, if C
pn pn(-)
0.6, if C
0, R
Of = R
0.4, =
33

---

Published as a conference paper at ICLR 2026
hed Sw4
h-5
f++2
hed
Swe
hw3
Figure 3: (Left) The target environment considered in the experiment. (Right) Representation of r" for the target environment.
1.00
0:07
Figure 4: (Left) Plot of d='
(Right) Plot of dd
2
Gloee 1100
h-4
h-5
h=1
Glse 1180
h-5
Figure 5: The trajectories compared in the first feedback. w on the left, and w/ on the right.
0. L
if Pr(-)Raac)
0,
ph((|L,ac)- =
0.45, if C
pn(+IC,ac)
0., = C
0.45, if C
0,
0., R
0.55, if R
0.3, if L
0,
0, if
(n(-)L,aR)
0.7,
Pa(-IC,aR)
0.3, if =
pn(-(R,aR)
0, if =
0f R
0f = R
1, if R
Intuitively, action az moves to the left w.p. 0.6, and keeps the lane w.p. 0.4, except when it is the left lane, where it keeps the lane. Action a is analogous but with different bias. Instead, action ag keeps the lane w.p. 0.4, and moves the left cannot move in certain direction, thus the remaining probability is splitted equally in the other two lanes. The target reward r* considered is described in Section 5 and is shown in Figure 3 on the right. Finally, the occupancy measures d= dd² describing the application y arise from policies described in Section and the transition model presented earlier, and are shown in Figure 4
the right w.p. 0.3. When it is the borders,
F.1.2 FEEDBACK F
Trajectory comparisons. We construct three trajectory comparison feedback. The first pair is in Figure and we associated 4 = 0.3 to it. The second pair of trajectories is in Figure 6 and we set to Finally, the third pair is in Figure 7 and has i3 -0.5.
Comparisons. Concerning the comparisons feedback, we considered the new environment shown FigureBJor the left, keeping the same transition model p described earlier but using the left L lane as initial state. We compared the two occupancy measures in Figure using t 0, and the two occupancy measures in Figure 10 using l2 = 0.5.
Demonstrations. For the demonstrations feedback, adopted the map in Figure the right, preserving the transition model p. but using lane R as initial state. We considered only one feedback, whose policy has the occupancy measure in Figure 1T which associated l 1.
34

---

Published as a conference paper at ICLR 2026
h-3
Glsg.
Swe
hw3
Figure 6: The trajectories compared in the second feedback. wi is on the left, and w on the right.
hw1
h+2
Glua 7 1- 0.00
hw5
h=1
hw2
Glog
1
had
ha5
Figure The trajectories compared in the third feedback, wi is on the left, and wg on the right.
h=
h-3
h-5
h=]
hw2
hw3
h
Figure (Left) The new map considered for the comparisons feedback. (Right) The new map considered for the demonstrations feedback.
012
0.22
0.28
0.32
Figure 9: The occupancy measures compared in the first comparisons feedback. a the left, and лf on the right.
f=2
hw3
h-5
h=1
f+2
hw3
Swd
h-5
1.00
0.14
Figure 10: The occupancy measures compared in the second comparisons feedback. T.. is on the left, and n on the right.
0.23
Figure 11: The occupancy measure of the expert's policy for the demonstrations feedback.
35

---

Published as a conference paper at ICLR 2026
0.0
1.0
002
0.8
A((-)
-0.3
0.6
-0.4
0.4
0.0
1.0
-0.6
1.0
0.6
0.8
0
200
400
600
800
1000
1200
0.4
0.4
Iteration
0.2
0.2
0.0
Figure 12: (Left) The sequence rewards fm,, computed by our algorithm. (Right) The corresponding values of the objective function.
0.8
0.8
A/(-)
0.6
0.4
0.4
1.0
0.0
1.0
0.8
0.6
200
400
800
1000
1200
0.4
0.4
Iteration k
0.2
0.0
0.0
Figure 13: (Left) The sequence of rewards PMR computed by our algorithm. (Right) The corresponding values of the objective function.
F.1.3 SIMULATION
The execution of Rob-Rel. generated the sequence of reward functions fm, for finding in Figure 12 on the left, while on the right we plotted the corresponding value the objective function (f
The analogous plots for are in Figure 13
F.1.4 VALIDATION THROUGH DISCRETIZATION
To understand the values of FM Fms F.gg Mr. my computed by the execution of Rob-ReL (see Figure [ right) make sense, i.e., are close the true values T'M Tm. FFF.g we have computed them also through an "exact" method, by computing a discretization of the feasible set and then taken the rewards that maximize/minimize the objective function (for approximating F'M Fm), and then averaged them. The results of this computation are reported in Figure [4] Clearly, these values are close to those in Figure 1 thus the computation carried out Rob-ReL makes sense.
36

---

Published as a conference paper at ICLR 2026
1.0
0.8
0.6
0.4
0.2
0.0
10
0.8
0.8
0.6
0.4
0.4
0.6
5
0.2
0.2
0.0
Figure 14: The rewards TM, 7m, FF.9 computed through a discretization of the feasible set.
S, A,A, H
We, mec. meCc
71
err æ
er
3.3.5
1,2.
50, 100
0.07 0.04
0.09 0.05
3.
1,
500, 1000
0.02 0.01
0.02 0.02
3,
5, 15, 15
50, 100
0.07 + 0.04
0.08 = 0.06
3,
5, 15,
500, 1000
0.02 = 0.01
0.03 0.02
100, I0, 15
1,2,
50, 100
0.13 0.07
0.33 = 0.19
100, 10. 15
1,
500, 1000
0.04 + 0.03
0.06 0.05
100, 10, 15
5, 15, 15
50, 100
0.14 4 0.12
0.17 a 0.15
100, 10, 15
5. 15, 15
500, 1000
0.04 0.03
0.06 0.07
Table 3: Results of experiment (i).
F.2 ADDITIONAL SIMULATIONS
The simulation presented in Section olis illustrative, i.e., aims to clarify the importance of our robust approach. Here, we conduct additional simulations to better characterize how Rob-ReL scales to larger problems.
To this aim, we generated at random large and various number of Rel. problems analogous to that in Section 6 i.e., where the state space can be interpreted as road with some objects, the actions allow to change lane (with some noise), and the reward is stationary and state-only, and in particular depends only on the object in the considered state. We generated two kinds of these problems, one (i) where the number of objects is small (specifically, 4), and so it is possible to compute FF. by simply discretizing the space of rewards and keeping the one that optimizes the minimax objective, and another (ii) where the number of objects is large (20 and 50), and so the optimal solution cannot be approximated through discretization. Both simulations have been conducted with values of hyperparameters
0.01 (the step size), K 3000 (number of iterations), and 1000 (the bound on the Lagrange multipliers).
Regarding (i), we conducted 8 simulations with the values specified in the first three columns Table 3 where n denotes the number of trajectories provided by any comparisons feedback, and N the number of exploration episodes allowed by any demonstrations feedback. Each simulation consists of generating 20 different problem instances with the specified features (size S, A, H, number of feedback mp, mpc. MTc- and number data n, NN), for each problem instance "finding" the optimum FF IF.gIIF IF using the discretization approach, computing the outputs εx. IIx of Rob-ReL, and then compute the mean absolute errors between F.F.g and kK and between Ir.g and Îx. The results are reported in the last two columns of Table 3 Clearly, the error reduces as the number of samples increases. Morcover, larger problem instances require more data expected also by theory (Theorem 5.3). We mention that as increases the time required for running RF-Express increases not trivially, execution cannot be vectorized (Menard al. 2021).
Regarding (ti), we conducted 4 simulations all with S, A, H = 20, 5, 12 and mp, myc. Mcc = 3, 7, where changed the number of objects and the number of data (see Table 4h. As aforementioned. here we cannot assess performance using the solution outputted through discretization. To this aim, for each problem instance, compute (r"))) and compare it with the output Gx of Rob-ReL
37

---

Published as a conference paper at ICLR 2026
objects
N
etr 8
20
50, 100
0.11
20
500, 1000
50
50, 100
0.07 + 0.05
50
500, 1000
0.11 # 0.08
Table 4: Results of experiment (ii).
have idea of the error. Results in the last column of Table We mention that some cannot be avoided because F does not directly estimate AJ(r"), and also that increasing the number of objects from 4 to 20 50 basically reduces performing gradient descent 20mm0)-diman space, which is still very efficient.
38


Reviews:
[{"review_id": "PElcKsswMW", "reviewer": "Reviewer_PqUe", "summary": "This paper develops a new framework, robust reward learning, that tackles the known problem of partial identifiability in reward learning. The framework incorporates several important forms of data for reward learning (including trajectories from an optimal policy, trajectory comparisons, and policy comparisons) and considers several potential downstream applications (including imitation, finding an optimal policy, and learning to compare preferences). The paper proposes to use worst-case performance in a downstream application, across all rewards compatible with the training data, as a metric for success.\n\nThe paper provides an algorithm for solving robust reward learning problems and provides theoretical guarantees on the sample and computational complexities. They also run some experiments in a small RL setting to demonstrate their algorithm works.", "questions": "Can the authors provide any insight about whether worst-case reward learning can scale as well as alternative approaches (such as cooperative inverse RL, or reinforcement learning from human feedback)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "I think the paper has a number of strengths:\n1. The paper addresses an *important problem*:  improving the safety of reward learning, which is a salient problem for modern AI deployments.\n2. The *novel formulation* of robustness for reward learning adds upon prior work addressing this problem. The uninformativeness measure is an interesting way to quantify the difficulty of doing reward learning.\n3. The formulation is *very general* and explicitly considers three important kinds of reward learning feedback. (Prior work, e.g. Skalse et al. (2023) only considered two kinds).\n4. A *(somewhat) tractable algorithm* for solving the novel problem is introduced (Rob-Rel), and theoretical guarantees on sample complexity and time complexity (at least in terms of the number of iterations) are given. Further, by using duality, the paper avoided any dependence of time complexity on the size of the feasible reward set.\n5. The paper *thoroughly, clearly, and fairly considers existing work*. In particular, tables 1 and 2 give clear relationships to broad prior work, and appendix *A.2* gives a thorough comparison to Skalse et al. (2023) which helped me to understand its contribution.", "weaknesses": "### Scalability\nThe proposed algorithm is polynomial in the size of the state space. This is a limited weakness, as it is common to some of the prior literature.\n\nHowever, some reward learning methods have been shown to work in realistic and large or continuous state spaces. For example, Christiano et al.'s (2017) reward learning has been applied to text settings (large, discrete state spaces) and physics simulations (continuous state spaces). Laidlaw et al. (2025) applied CIRL to a large game.\n\nI think the impact of this work would be significantly improved if evidence was given to suggest the framework and algorithm were scalable. If the algorithm is not scalable, it would be helpful to describe these limitations.\n\n### Practicality of worst-case return\nThe proposed algorithm maximizes worst-case robustness relative to a feasible set. As the authors themselves suggest on line 241, there may be some reward learning problems where it is infeasible to get an acceptable worst-case reward. (In these cases, I think the uninformativeness could be a helpful metric for measuring this infeasibility, as noted in the strengths above). I expect that, in many real-world applications with large feasible reward spaces, many reward learning problems will be infeasible.\nWhile it is not reasonable for the paper to solve this problem, I do believe it is a notable limitation, relative to e.g. a Bayesian approach that only tries to minimize expected loss over a posterior reward distribution.  Evidence that robust reward learning is feasible in real-world environments might improve the paper.\n\n\n### Clarity\nI found the paper to be quite difficult to parse in a number of places, although I think the paper is relatively clear given its technical density. Three potential points for improvement might be:\n1. Reduce the introduction of abbreviations: \"ReL\", \"IL\", \"PBIRL\", and other abbreviations could be far easier to parse if set out plainly.\n2. Reduce the introduction of notation, or where possible, explain in plain English what terms mean. Section 5, in particular, introduces a whole range of new notation that is hard to keep track of and could be better supported with natural language explanations.\n3. (Minor) Throughout the paper, citations are given without bracketing, where bracketing would be much clearer. For example, lines 214, 121 and 105.\n\n\n### References\n* Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017). Deep reinforcement learning from human preferences. arXiv preprint arXiv:1706.03741.\n* Laidlaw, C., Bronstein, E., Guo, T., Feng, D., Berglund, L., Svegliato, J., Russell, S., & Dragan, A. (2025). AssistanceZero: Scalably solving assistance games. arXiv preprint arXiv:2504.07091.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YtAMG1YT2x", "reviewer": "Reviewer_mwRB", "summary": "This paper addresses the fundamental problem of partial identifiability in Reward Learning (ReL), where limited feedback makes it impossible to uniquely recover the target reward function. The authors propose a quantitative framework that allows measuring performance degradation due to identifiability issues, moving beyond prior qualitative approaches. They introduce a robust minimax approach that optimizes for the worst-case reward in the feasible set and develop Rob-ReL, an algorithm for policy preference assessment problems with provable sample and iteration complexity guarantees. The work combines demonstrations, trajectory comparisons, and newly introduced policy comparison feedback in a mixed offline-online setting. Theoretical analysis shows polynomial complexity in relevant problem parameters, and numerical experiments illustrate the approach on a low-dimensional navigation task.", "questions": "How does the method scale to continuous state-action spaces with function approximation? \nIs there a path toward extending Rob-ReL to deep RL settings, perhaps using neural network reward parameterization?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "**Rigorous Theoretical Analysis.** Theorem 5.3 provides polynomial sample and iteration complexity bounds under reasonable assumptions (Slater's condition). The proof technique combining visitation distribution estimation errors with primal-dual subgradient convergence is sound. The use of RF-Express for minimax-optimal reward-free exploration is appropriate.\n\n**Clear Presentation and Organization.** The paper is well-structured with motivation, framework, approach, algorithm, and theory presented logically. The illustrative example in Section 6 effectively conveys the main ideas visually.", "weaknesses": "**Limited treatment of function approximation.** The tabular setting with explicit state-action representation limits applicability to high-dimensional problems. While the authors mention neural network parameterization in related work, Rob-ReL does not incorporate function approximation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "30fTHsbEcp", "reviewer": "Reviewer_tEva", "summary": "This paper addresses the issue of partial identifiability in reward learning (ReL), a consequence of when the provided feedback is insufficient and does not allow for the identification of the target reward $r^\\*$. Instead of traditional approaches, the authors propose a new framework that incorporates robust optimization to minimize the loss tied to the worst possible value for $r^*$ within some feasible set.", "questions": "- Can you expand on what you mean on lines 240-241 and/or provide a reference speaking towards a practical example?\n- You make the assumption \"that the feasible set $\\mathcal{R}_\\mathcal{F}$ contains a strictly feasible reward $\\bar{r}.$ How realistic is this in practice?\n- You make the assumption \"that the feasible set $\\mathcal{R}_\\mathcal{F}$ contains a strictly feasible reward $\\bar{r}.$ How realistic is this in practice?\n- Consider changing the colored text in equation 7 so that it matches the blue in Lemma E.2.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The formalization, explanation, and clarity of the ReL problem as a pair $(\\mathcal{F},g)$ is very well written, general, relevant, and well positioned against related work. A key piece is how the ReL problem is reformulated to finding the optimal object $x\\in\\mathcal{X}_g$ for some application $g$, when given the uncertainty set of rewards $\\mathcal{R_F}$.\n- The metric used to calculate the loss, $\\mathcal{I}_{\\mathcal{F},g}$, or how \"uninformative\" $\\mathcal{F}$ is for application $g$, is very useful as it effectively allows for the principled comparison of various feedback sets and/or applications.\n- The algorithm instantiation and associated analysis is novel. A key piece that facilitates this is in Proposition 5.1 where the minimax problem is simplified into two convex problem.", "weaknesses": "- While the authors present a powerful, general framework, they do not address tractability concerns. The limitations section significantly downplays this fact and speaks broadly on it.\n- The experiments, though useful in demonstrating/verifying the author's theoretical, do not have any baselines to compare against. For example, in section 4, existing approaches are discussed. It would be interesting to see how the proposed algorithm compares against these non-robust baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VVtwIkmSTd", "reviewer": "Reviewer_kR8k", "summary": "This paper introduces a framework for reward learning from preferences and demonstrations. The authors try to quantify the effect of  identifiability issues of the reward, and propose a minimax approach to optimize the worst-case scenario over the feasible reward set. They provide Rob-ReL, an algorithm for policy preference assessment, providing theoretical guarantees on sample and computation complexity.", "questions": "Please see the weaknesses above", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Clear formalization of the problem. \n- The paper is well written and organized in a clear progression\n- Clear sample-complexity results for their algorithm with illustrative numerical results", "weaknesses": "- A big limitation of this work is the limited number of applications. The authors provide a general framework, but then they limit themselves to a single scenario. This leaves the overall framework not properly tested, and severely limits the contribution of the paper.\n- Another limitation, is that while the authors provide a general framework, the theoretical insights seem to be limited. For example, when solving a minimax game the equilibrium may not be a pure equilibrium; however, the authors do not seem to discuss this issue. Depending on the properties of ${\\cal X}_g$, the loss, etc...we may have different situations. In the application they propose the solution they get seems a pure one (because it's estimating a scalar), but i would expect mixed solutions to appear depending on the problem (e.g., when ${\\cal X}_g=\\Pi$). While the authors do an effort at quantifying the error (eq 3), it is not very clear what are the properties of this minimax problem (which depends on the set of rewards, loss, etc.). \n\n- Theorem 5.3: the dependency on $\\xi$ seems quite large\n- It is not clear why the chosen application (estimating policy-value differences) is an interesting one\n\n- While the paper is well written, there is still lot of notation and it is hard to follow the proofs.\n\n- The method is only tested on a simple problem, while larger problems are untested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "en6J66Bc4B", "reviewer": "Reviewer_Kgr5", "summary": "The paper proposes a unified framework for robust reward learning in partially identifiable settings. Given some form of feedback (e.g., preferences, demonstrations), the method constructs a feasible set of reward functions consistent with observed data and then optimizes the worst case over this set (for a downstream decision such as a policy). The formulation aims to generalize across multiple feedback modalities and tasks.", "questions": "1) How does (2) differ theoretically or algorithmically from robust IRL or minimax RLHF formulations?\n2) How sensitive is performance to the feasible reward set?\n3) Can the approach handle partially observed dynamics or limited coverage?\n4) What is the empirical complexity compared to baselines? how does it scale with state or trajectory length?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The framework unifies various feedback types (e.g., pairwise comparisons, demonstrations) and downstream objectives under one minmax formalism.\n+ The worst-case formulation in (7) is intuitive and could be more realistic than assuming a point estimate of the reward.", "weaknesses": "- The general minmax structure resembles many existing approaches in IRL, robust MDPs, and RLHF (e.g., distributionally robust RL, reward uncertainty sets). The paper does not sufficiently delineate what is new in (2) beyond framing and unification.\n\n- The paper needs sharper theoretical distinctions (e.g., identifiability analysis, sample complexity bounds, or formal generalization of previous frameworks). Without this the contribution risks being incremental.\n\n- The approach requires estimating or approximating policy-induced dynamics and state-action visitation measures, which can be computationally prohibitive and may not be scalable to realistic domains.\n\n- The proposed method replaces one set of unknowns (the reward) with multiple estimated quantities (transition probabilities, occupancy measures, feasible reward bounds), which may not improve robustness in practice.\n\n- The experiments are limited to small toy domains (even with the ones in the appendix). The paper needs results on more standard RL or preference-learning benchmarks (e.g., MuJoCo or D4RL tasks) to demonstrate practical value. The lack of scalability experiments weakens the claimed generality.\n\n- It is unclear whether ablation or sensitivity studies were conducted to assess dependence on feasible set or uncertainty size.\n\n- The paper does not discuss how many trajectories or feedback samples are required to characterize the feasible reward set. It remains also unclear how the method handles poor coverage or unobserved states which is an important concern when identifiability is partial.\n\n- The framework implicitly assumes access to accurate simulators or transition estimates, which limits realism in practical RLHF settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)