Understanding and Adopting Rational Behavior by Bellman Score Estimation
Authors: Kuno Kim
Stefano Ermon
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Inverse Reinforcement Learning

Abstract: We are interested in solving a class of problems that seek to understand and adopt rational behavior from demonstrations. We may broadly classify these problems into four categories of reward identification, counterfactual analysis, behavior imitation, and behavior transfer. In this work, we make a key observation that knowing how changes in the underlying rewards affect the optimal behavior allows one to solve a variety of aforementioned problems. To a local approximation, this quantity is precisely captured by what we term the Bellman score, i.e gradient of log probabilities of the optimal policy with respect to the reward. We introduce the Bellman score operator which provably converges to the gradient of the infinite-horizon optimal Q-values with respect to the reward which can then be used to directly estimate the score. Guided by our theory, we derive a practical score-learning algorithm which can be used for score estimation in high-dimensional state-actions spaces. We show that score-learning can be used to reliably identify rewards, perform counterfactual predictions, achieve state-of-the-art behavior imitation, and transfer policies across environments. 

Published as a conference paper at ICLR 2023
UNDERSTANDING AND ADOPTING RATIONAL BEHAV- IOR BY BELLMAN SCORE ESTIMATION
Kuno Kim
Stefano Ermon
Department of Computer Science Stanford University Stanford, CA 94305 khkimêcs, stanford.edu
Department of Computer Science Stanford University
Stanford, CA 94305 ermonêcs. stanford.edu
ABSTRACT
We are interested in solving class of problems that seek to understand and adopt rational behavior from demonstrations. We may broadly classify these problems into four categories of reward identification, counterfactual analysis, behavior imitation, and behavior transfer. In this work, make key observation that knowing how changes in the underlying rewards affect the optimal behavior allows one to solve variety of aforementioned problems. To a local approximation, this quantity is precisely captured by what term the Bellman score, i.e the gradient of the log probabilities of the optimal policy with respect to the reward. We introduce the Bellman score operator which provably converges to the gradient of the infinite-horizon optimal Q-values with respect the reward which can then be used to directly estimate the score. Guided by our theory, we derive practical score-learning algorithm which can be used for score estimation in dineeiiiensional state-actions spaces. We show that lcore-learning can be used reliably identify rewards, perform counterfactual predictions, achieve state-of-the- art behavior imitation, and transfer policies across environments.
INTRODUCTION
A hallmark of intelligence is the ability to achieve goals with rational behavior. For sequential decision making problems, rational behavior is often formalized policy that is optimal with respect to a Markov Decision Process (MDP).
other words, intelligent agents are postulated to learn rational behavior for reaching goals by maximizing the reward of an underlying MDP (Doyal 2007 Nettci& & Averbeck 2019 Niv 2009). The reward covers information about the goal while the remaindr of the MDP charactenizes the interllay between the agent's 's decisions and the environment.
In this work, we are interested in solving class of problems that seek to understand and adopt rational (i.e optimal) behavior from demonstrations of sequential decisions. We may broadly classify them into categories of reward identification, counterfactual analysis, behavior imitation, and behavior transfer. (see Appendix for detailed definitions) These four problems of understanding and adopting rationality arise across a wide spectrum science fields. For example, in econometrics, the field of Dynamic Discrete Choice (DDC) seeks algorithms for fitting reward functions to human decision making behavior observed in labor (Keane & Wolpin 1994) financial markets (Arcidiacono & Miller 201T). The identified utilities are leveraged to garner deeper insight into people's 's decision strategies (Arcidiacono & Miller 2020), make counterfactual predictions about how their choices will change in response market interventions that alter the reward function (Keane & Wolpin 1997). and train machine learning models that can make rational decisions like humans (Kalouptsidi et al. 2015). In animal psychology and neuroscience, practitioners model the decision strategies of animals by fitting reward models their observed behavior. The fitted reward is analyzed and then used to train Al models that simulate animal movements (Yamaguchi et al. 2018, |Schater et al. 2022) in order to gain a better understanding of ecologically and evolutionarily significant phenomena such as habitat selection and migration (Hirakawa et al. 2018). In robot learning, Inverse Reinforcement Learning (IRI) is used to infer reward functions from teleoperated robots and the learned rewards are fed through an RL algorithm
teach robot how perform the same task without human controls (Fu et al. 2018 Finn et al. 2016b) Chan & van der Schaar (02T). As rewards for a task are often invariant to perturbations in the environment, (c.g walking can be described by the same reward function which encourages forward velocity and stability regardless of whether the agent is walking

---

Published as a conference paper at ICLR 2023
on ice grass) the same task can be learned in various environmental conditions by optimizing the same inferred reward (Fu et al. 2018 Zhang et al. 2018).
We make a key observation that knowing how changes in the underlying rewards affect the optimal behavior (policy) allows one solve variety of aforementioned problems. To local approximation, this quantity is precisely captured by what we term the Bellman score, i.e gradient of log probabilities of the optimal policy with respect to the reward. Prior works which study related quantities to the Bellman score are scarce and relied on having full-knowledge of the environment dynamics small discrete state-action spaces (Neu & Szepesvári 2012 |roman 2014, Li et al. 2017). We introduce the Bellman score operator that provably converges to the gradient of the infinit-hhrrizon optimal Q- values with respect to the reward. This Q-gradient can then be used estimate the score. We further show that the Q- graddien is equivalent to the conditional state-action visitation counts under the optimal policy. With these results, we derive the gradient of the Maximum Entropy IRI. (Ziebart et al. 2008 Finn et al., 2016b; Fu et al. 2018) objective in the general setting with stochastic dynamics case and non-linear reward models. Guided by theory, we propose a powerful score-learning algorithm that can be used for model-freee score estimation in continuous high-dimensional state-action spaces and an effective IRI. algorithm, named Gradient actor-Critic (GAC). Our experiments demonstrate that score-learning can be used to reliably identify rewards, make counterfactual predictions, imitate behaviors, and transfer policies across environments.
2 PRELIMINARIES
A Markov Decision Process (MDP) MM& E Q. where f2 is the set of all MDPs, is a tuple M = (X, A, P. Po, re. 7) where X' is the discrete F state space, A is the discrete action (decision) space, P E [l is the transition probability matrix, Po E Rla is the initial state distribution, ro E RRI is the (stationary) parametric reward with parameters 0 E O and y E [0, 1] is the discount factor. θ is the parameter space, typically set to a siiteeddmensionall real vector space Rdim((), We will use T 2 0 to denote the time horizon for finite-horizon problems. A domain d is an MDP without the reward M M 'r. Moving forward we will alternate between vector and function notation, e.g P(a'Ia |r. a), ro(2,, a) denote the value of vector rg at the dimension for the state-action (z, a) and the value of matrix P at the location for ((z, a), x²). Furthermore, one-dimensional vectors are treated as row vectors, e.g R AX = RII|X*XAA,, A (stationary) policy is a vector т E R x that represents distributions over actions, i. E.EA (a(zz)
1 for all x. A non-stationary policy for time horizon T is a sequence of policies मे = (TT, (mT)) E (IR xxAIxITT where ne the policy used when there are environment steps remaining and T = ( for k < T denotes subsequence. When there is no confusion we will use म to denote to both stationary and non-stationary policies.
Next, Pa E [XXXA|XXXXXX denotes the transition matrix of the Markov chain on statc-action pairs induced by policy π,, i. P= (x', a' Ix, a) = P(x'Iz, a) (aa)) (z'). Furthermore, Pr denotes powers of the transition matrix with π = I where I E x A is the identity. For non-stationary policies, we define P = PAr_11 Pr P Pr..,...Prr Pay_.. for n 2 1 and P0 = I.
Let La,a E g|*xAl denote the indicator vector which has value 1 at the dimension corresponding to (xr, a) and 0 elsewhere. The conditional marginal distribution for both stationary and non-stationary policy т after environment steps is P-r,(((ar, a) Le.a R|IxAl The (unnormalized) t-step conditional occupancy measure of x is the discounted sum of conditional marginals:
(-1. a) = Σ P' (.2z.))
Σ 7thh Dr.aa Pm
n=0
n==
Intuitively, Pr,e(a', a'|x, a) quantifies the visitation frequency for (x', a') when an agent starts from (z. a) and runs for f environment steps, with more weight on earlier visits. The infinite horizon conditional occupancy exists for 7 < 1 and we will denote it by simply omitting the subscript I, Le Pn (-|2z, a) = lim (-1x, a). The (unconditional) occupancy measure can be recovered by Pr,a(x', d') = Σ. a Po(x) (a)=) (2))) (x', a |0r, a). The t-step Q-values QT, E TR X'xA for both stationary and non-stationary policy π are defined as Qn,e(2, a) = E Ex''a'mpppt ( (a) [re(z', a')] pa. (-12,, a)-r, i. the conditional expectation of discounted reward sums when there are t environment
We use discrete spaces for notational brevity later on, but our results can be extended continuous spaces 2 Bacckards indexing is standard in dynamic programming RL algorithms (Bertsekas & Tsitsiklis 1995) Step refers to the number of environment transitions and the number of decisions made
2

---

Published as a conference paper at ICLR 2023
steps remaining. The theory in this section will be derived in the Maximum Entropy RL (MaxEntRL) setting (Haarnoja et al. 2017, Yarats et al.) (02T). Let us define the t-step discounted conditional entropy of a policy as HT,e(x, a)
Σn=1 ? Pn (-12; a) log ø--n- In MaxEntRI, the soft Q-values are defined as Q (æ,a) = Qx,(x, a) + Hn, a), i.e the conditional expectation of discounted reward and policy entropy sums.
For reward ra, let Q [RIxAA denote the t-step optimal Q-values defined iteratively by the soft Bellman optimality operator TH XXA + R xxAI as
Q (I,a) a) (Tr (2,, re (z( + 7 (( [log Σ exp Q Q0.-1 (a', (z',a') a')]
(2)
a'E.A
for all t 0 and Q7,0 r. Hence the optimal Q-values for decision problem with horizon t can be obtained by sequentially computing the optimal Q-values for 1,... - step decision
problems using dynamic programming (Bellman 1957), i.e value iteration. The operator Th is
contraction in the max-norn nox- (Bertsekas & Tsitsikli 1995) whose unique fixed point is the infinite- horizon optimal Q-values Q satisfying Q The Qroft 8,00 For finite horizon problems, the (non- stationary) optimal policy = ( )i=o soft 2 (a.a), /Ea ea, (2.0) and, similarly, the infinite-horizon (stationary) optimal policy sofft
is derived by
= "0,t (a|z) = softmax (āat (x, a)
softmax (Qaoft) We may now define the key quantity of interest, the Bellman score.
Definition The finite-horizon Bellman score stoft ( yr+ is the gradient of the log probabilities of the (non-stationary) optimal policy with respect to the reward,
5g s0ft
(50.1 T = (Ve log soft YT=
Similarly the infmite-horizon Bellman score
$0,00
soft
E I I xdim(() is
soft 8,00
= Vo log 78,00
soft
sofft is the Jacobian of the log optimal policy vector with respect In words, the lnite-horizon score S0, to the reward parameters, i.c soott (z, a) log wooft (a|x) is the direction in which the reward parameters should perturbed in order to increase the log probability of action at state r under the optimal policy for time t. To a local linear approximation around 8, the score conveys how the optimal behavior changes function of the underlying reward parameters. Next, we will prove useful properties of the score as well as derive dynamic programming algorithm compute it.
3 SCORE ITERATION
Since soft (a/x)= = softmax (Qooft (a., a), the Bellman score can be written as a difference of the n0,
gradients optimal Q Q-values with respect the reward parameters:
soft 1,8 (x, a) a log T8,t (alz) VoQu (x,a) a) Eernngg (-lx) (Ha)) Qo (z, d')]
(3)
We will refer VoQuoh the Q-gradient or value gradient. Eq. shows that an unbiased sample estimate of the score can be obtained from the Q-gradient and the optimal policy. Thus, will derive a dynamic programming algorithm termed score iteration which efficiently computes the Q-gradients. Similar to how value iteration proceeds by repeated application of the Bellman optimality operator (Eq. score iteration will rely on the Bellman score operator which we define now.
The Bellman score operator Gr.0 R [ixxxx(x R XxA xdim(0) for a policy and reward ro is defined on an inpul J E 1IA'xA|/xdim@e as
Definition 2
Gr,oJ J = V JP.J.
To gain intuition about the score operator, consider the tabular reward (0 a) = = a) where gl(xxAA In this setting, the score operator simply computes the conditional occupancies for π (Eq. I when applied to the identity matrix 7. For example, the values of Gr Varo + 7 the for state-action (z, is 2 + PP, (- (z, a) which corresponds to the one-step conditional occupancy Pr. a). While the score operator has a pleasantly simple form, Theorem will show that the Q Q-gradients can be computed by its repeated application and, as a result. that the Q-gradieet is the conditional occupancy measure of the optimal policy for the underfying reward.
+
3

---

Published as a conference paper at ICLR 2023
Algorithm 1 Score Iteration: Computes the infinite-horizon Bellman score via dynamic programming
M = AA P. Po, re, r)) Markov Decision Process g: randomly initialized Q-gradient vector procedure SCORELTERATIONAAS
For M. learn the optimal policy soft while is not converged do for a E X, a E A do
Update Q Q-gradient: g(2, Vora(z, + t Papee (1),
for X, A do Compute Bellman score: + g(x, - "o (o
soft return 6,000 9,
Theorem For all t= 1,,,, T and any matrix J (2Ixxx)xddm(0))
VoQa. 9 a....9 Iroot of V VoQ2.0 )
V V Irze al V9Q20,00 k0000
lim
m00 0
where, VoQ0.0 Verg. Furthermore, the Q-gradient satisfies:
VoQu.t (Ir, a) Egy Ba Pagt (- V Voro(s',n)) B,d
V VoQunt (z,a) Eg pa (12,a) Vore(r',
Theorem shows that both finite and infinite horizon Q Q-gradients can be computed by repeatedly applying the score operator of the current reward Ta and its optimal policy x
zoft "T,000 Similar to how the Bellman optimality operator (Eq. converges any starting vector in A to the infinite- horizon optimal Q-values (Bertsekas & Tsitsiklis 1995), the Bellman score operator converges any starting matrix in A)xmmt()) the -oriiiee Q-gradient.
Furthermore, we see that the Q-gradient is in fact the expected reward gradient under the conditional occupancy measure. To gain more intuition, once again consider the tabular reward re a) 0(x, a). Then, VoQ2of (I,
Ex -e'p--gg , (-Ir,a) [1lx aa Przeft. We see that the Q-gradient is in fact
equivalent to the conditional occupancy of the optimal policy, which is consistent with our previous analysis of the score operator. Thus, in order increase the optimal Q-values at (x. a). the rewards of state-actions (x', a') should be increased proportional to their discounted visitation frequencies when starting from (I,a) and following the optimal policy. As alternative to the operator perspective, one may consider Linear Programming (LP) interpretation. The optimal Q-values are the solution to an LP optimization problem over the set of conditional occupancies D that are feasible under the Markovian environment dynamics, ie Q (z, a) = maXpeD r9 p. Loosely speaking, invoking the supremum rule (Boyd et al., 2003) allows one to show that the gradient of the LP solution, i.e V,, max,eD is in fact the conditional occupancy that solves the LP.
The main practical contribution of Theorem I that the operator convergence results enable designing a powerful dynamic programming method for score computation as shown in Algorithm 7 (Finite-horizon version in Appendix D) The algorithm computes the optimal policy T8, then proceeds by repeatedly applying the score operator Grzes .0. While highly effective in countable
state-action spaces, score iteration has limited applicability to continuous state-action spaces and requires full knowledge of the environment dynamics. Similar to how Q-learning is a model-free instantiation of the value iteration algorithm, will proceed propose score-learning which is model-free instantiation the score iteration algorithm.
SCALING UP SCORE ESTIMATION
In order scale score estimation the model-free settings with (potentially) high-dimensional continuous state-action spaces, we start by parameterizing the policy, Q-gradient network, and score network as neural networks To x A - R. go X' - di)) and 8 X A (d Just as Q-learning (Silver et al.) 2016) approximates an application of the Bellman optimality operator by learning Q-network that minimizes boot-strapped regression loss, score-leariing approximates
4

---

Published as a conference paper at ICLR 2023
Algorithm 2 Score-learning; model-free instantiation of SCORELTERATION
(8, d, 1,, w): Weights reward. policy, Q-gradient, and score network
B: Buffer transitions (can be either online or offline data)
Ng: Total number algorithm iterations, number Q-gradient update steps per policy update
ne Ml: Q-gradient learning rate, score-learning rate
Og: Target Q-gradient mixing rate
RL: One policy update step in an RL algorithm, e.g one policy gradient step in SAC (Haarnoja et al. 2018
procedure SCORELEARNINGI o, U, N)
for E (1,...,NN N do
# +Update policy TT if iteis is not None then Update policy to maximize reward re: ø + (L(0,
# Update Q-grudient Gv for j { [I....,N, No do Sample batch: (z, a, =0)~~ B if v is not None then
Sample next action from current policy: a' ~ (Ta((a²))
else
Sample next action from buffer a' ~ B
Update Q-gradient (Eq. & +- 1b no & (Vora(z, a) (a', go (2, a))²
Update Target Q Q-gradient Ty with soft mixing rate og (Haamoja ct al. 2017
Update Bellman score 8y
Sample batch and contrastive action: (æ, ~ B,  (-(-))))
Update score network (Eq. 5 w no Vo (0(x, a') sw (w,
return 47, w
an application of the score operator by learning Q-gradient network that minimizes the boot-strapped regression loss in Eq. g The score network then minimizes the regression loss in Eq. 5hihch directly follows by replacing the Q-gradient with our estimate Go in Eq.
Lg = (Voro (z, a) + En ot - (1 I (1', (I',a') ( a))²
(4)
L = (0 a) - Eu'na ( (.1a) los (Ir, a')] (x, a))²
(5)
Note that Ge is a target Q-gradient network used for stabilizing optimization analogous to target Q-networks that are used Q-learning (Haamoja et al. (017). Algorithm shows the full execution flow of score-learning. Instead of sequentially learning each model until convergence, score-learning alternates between making small improvements to the policy (red), regressing to the Q-gradient with the current estimate of the optimal policy (blue), and updating the score (yellow) with the current estimate of the Q gradient. This enables the algorithm to output approximate estimates of all three components when run for few-steps, which will be useful for downstream algorithms that use approximate score-learning in the inner-loop. If the buffer B already contains samples from the optimal policy, simply not passing & disables policy updates. Note that score-learning can operate with both online and offline data B depending on the choice of the RL algorithm used to update the policy. We now show how the score-learingg algorithm can be used for a variety of downstream tasks.
4.1 MAXIMUM ENTROPY INVERSE REINFORCEMENT LEARNING WITH SCORE-LEARNING
In this section we will show how score-learning can be used for maximum entropy inverse reinforce- ment learning (MaxEnilRL). Let 7 = (Io, ag. .... TT, ar) denote a trajectory of state-actions, and (T( log (o(x)) I T,, (ar (2)) IT t== P( [22,, ax) denote the trajectory distribution of the MaxEnt optimal policy soft for reward re and similarly let p" (r) be the trajectory distribution of the expert policy π (x; >o from which the demonstrations are sampled. The goal in MaxEntIRL is to find reward parameters 0 that maximize the log-likelihood of the expert trajectories:
max Eypop [log po(
The gradient of the objective in Eq. is highly useful it enables direct gradient ascent solve the maximization. Previously, the EnXLL gradient has been derived in either the limited setting of deterministic dynamics Ziebart et al., 2008) or in the general stochastic dynamics setting for the special case of linear reward models Ziebart et 2010). Here, leverage Theorem T to derive the general MaxEntIRL gradient in the stochastic dynamics setting with non-linear rewards.
5

---

Published as a conference paper at ICLR 2023
Algorithm 3 Gradient Actor-Critic (GAC) ): Inverse RL via Q-gradient Estimation
(8, 6, Li, (E): Weights for reward, policy, learner Q-gradient, and expert Q-gradient network
D: Demonstrations of expert behavior
M: Total number GAC iterations
N, No: Number of score iteration steps per GAC iteration, Reward update interval
Mr Reward learning rate
procedure GAC(B, o, Φ. M. () for E { ,, .MM do
Take step in environment with T3, add transition (I, s',)) to buffer B Update learner Q Q-gradient and learner policy:
Φ, th + ECORELEARNING(C Φ VL, N)
Update expert Q --gaddient:
thg < SSCORELEARNING(0, (O-NNG(,,O-Nnne, VE, D)
if mod No == 0 then
Sample expert hatch of initial state-actions (No, ao) ~ Ddemo
Sample learner batch of initial state-actions du B, āo To (āo)
Update reward with Q-gradient: 0 4- + 1- V)- (gr (20, uo) gr (To. oo)) return 0,0
(o)
Theorem
The gradient of the MaxEntIRL objective of Eq.
5 is
V Vo (log po (T)) = E2- (-Ir), (a), W. (-1z) VoQ r (1, voQur(u,a))
= Ezzarrps-. Vorra (z. a)] - Eg Euyaar gm Voro(x', a')l
(8)
Theorem shows that the reward should be updated to increase the Q-values of the expert and decrease the optimal Q-values for the learner (Eq, 7 which is equivalent to increasing the reward on state-actions sampled from the expert and decreasing the reward of state-actions sampled from
the learner (Eq. 8 With deterministic dynamics. Eq. g simply reduces the known contrastive
divergence gradient (Finn et al. (016b) for energy-based models. When To is linear, Eq. 3 reduces to
(2010). (see Appendix A for extended discussion) the expected feature gap derived in Ziebart al.
Many prior works in MaxEntRL (Finn al., 201 2016b; Fu et al. 2018) take Monte Carlo samples to estimate Eq. aadd perform stochastic gradient updates on o. The key challenges with this approach are, first, the difficulty of reusing old samples (2r, a) ~ T8..a from policies for older rewards Oold and, second, the high variance of the Monte Carlo estimator. The main practical significance of Theorem comes from Eq. which reveals that we may instead learn Q-gradient networks with score- leare-ea to approximate the expectations in Eq. g At the cost of introducing bias to the gradient estimates, this approach resolves the two aforementioned challenges; score-learning works with offline data, which enables old sample reuse, and the Q-gradient network outputs the mean reward gradient, which reduces variance. We name our algorithm Gradient Actor-Critic (GAC) (see Algorithm 3 which uses score-learning to estimate the expert Q-gradient VoQ,. (2, a) and learner Q-gradient VOQ.. a) with parametric functions gegg and 90z To reduce computational cost, GAC alternates between approximate Q-gradient estimation, i.c running score-learning for N steps, and using the rough gradients for reward updates. Our method is analogous to actor-critic methods (Haarnoja et al. 2017) that learn a Q-network to estimate mean policy returns which are then used for policy gradient updates. These methods similarly introduce bias into the policy gradients in exchange for the ability to reuse old policy data and perform lower variance gradient updates. Just as how actor-critic typically outperforms Monte-Carde policy gradient methods, our experiments will demonstrate similar usefulness trading off bias for lower variance.
snft
4.2 COUNTERFACTUAL PREDICTIONS WITH SCORE-LEARNING
Here we describe how score-learning can be used for counterfactual predictions. The goal is to predict how the optimal policy changes when the original reward ra is perturbed to a counterfactual reward rær. With access to efficient environment simulator, problem rgg. However, this is difficult in many domains where solving the RL problem expensive, particularly when we wish to make many counterfacutal predictions for multiple reward alterations. Instead, we may use the score sofft for the original reward ra to estimate of the optimal policy at the $,000
natural solution is to simply resolve the RL
new reward rer by simply using the first order taylor approximation of the log optimal policy:
log soft (xr, a) x log soft (ar, a) + soft (zx, a) (6' 0)
Eq. shows that log mgr.t (a/z) increases proportional to how well the direction of change in reward 00' 0) aligns with the score so(x, a). We use Algorithm 2 to estimate 8p with gø and apply Eq.

---

Published as a conference paper at ICLR 2023
White
Hiltary
0.00
Prosectty Eiectunn
0.96
0
004
0.00
25
I00 nn kub
100 bue sub 2000
e.. tree Sup 5000
pred suC 2000
pred sub 8000
Blue
coller
Home
Hilttay
Probuaatyy or Decaaion
322-
CA
0.4
3.5
3:1-
0.15
0.10
0.00
25
Timestep
-.: n0 sub
true SUb 2000
true 8O00
pred EUB 2000
prod Sub 800D
Figure I: Counterfactual Prediction Performance for two scenarios:
education subsidy (top row) and military enrollment incentives (bottom row). "no sub" corresponds to the initial reward from which the scores are estimated, "true sub" shows the probability of actions under the MaxEnt optimal policy obtained by resolving the RL model for perturbed rewards, and "pred sub" shows the score- estimated optimal policy probabilities. 2000, 8000 denote the magnitude of the perturbation to the reward parameters which corresponds the amount of subsidy or incentives.
5
EXPERIMENTS
5.1 REWARD IDENTIFICATION AND COUNTERFACTUAL PREDICTIONS
We experiment with the Keane and Wolpin (KW) environment (Keane & Wolpin 1994, 1997) which is a widely used benchmark in the DDC literature. The KW environment simulates life-time of occupational decisions where each time-step an agent selects among five actions: white collar work, blue collar work, military, school, and staying home. The state is 25 dimensional vector consisting of various features about the agent such as the time spent in each occupation, and their current job. The dynamics captures how their current job decision alters the state. Agents are required to be forward looking and maximize the rewards summed over a life-time of occupational choices. The reward parameter 0 is a 63 dimensional vector where each parameter dimension corresponds to either a cost or benefit, e.g the cost attending college or the benefit of transitioning from a white collar job to the military. The specific structure of the reward is omitted for brevity, but full details about the environment can be found in (Keane & Wolpin 1994 1997.
Environment:
We solve the forward MaxEntRL prob- lem in the KW environment and use the optimal policy to generate 100 trajectories of occupa- tional choices for two different reward parame- ters: Pedu and 0mat The first reward simulates scenario with lower education costs and the lat- ter captures scenario with high benefits for mil- itary enrollment (Keane & Wolpin| 1994 1997).
Results:
Table I: Reward Identification on KW dataset.
Mean squared error between true reward and IRL estimated reward evaluated on the demonstrations.
GCL
MSM
GAC
Pedu 6mi
12.1 + 5.3
0.14 0.4
0.16
0.3
20.5
6.8
0.14 0.3
0.08
0.2
We test the reward identification performance by computing the mean squared error between the ground truth reward Tecau Tem.t and the estimated rewards on Te : T88 T evaluated on the demon- strations. As rewards are only identifiable up to additive constant shifts (Kim et al. 202T). we search this equivalence class for the parameters that minimize the mean squared error. Table I shows that GAC is able to identify the rewards to similar precision as the Method of Simulated Moments (MSM) baseline which is the state-of-the-are method for identification in the DDC literature. While GAC and GCL optimizing the same objective, i.e Eq. 8, posit that trading off bias for lower variance (as explained in section accounts for GAC's superior performance.
Next we test the counterfactual prediction performance using the method in section We simulate two counterfactual scenarios where the govermment provides subsidy for college tuition and military participation. The scenarios are simulated by reducing the cost parameters in the reward for attending college and joining the military. Figure [ shows how our score-estimated counterfactual predictions of policy changes compare fully re-solving the RI. problem for the perturbed rewards. We are able
7

---

Published as a conference paper at ICLR 2023
Table 2: State Control Performance when provided with one demonstration.
BC
DAC
AIRL
GCL
GAC (OURS)
BALL IN CUP CATCH
39.2 15.2
291.9 - 29.1
124.3 25.0
502.2 : 104.2
949.9 + 42.4
CARIPOLE SWING-UP
106.0 50.1
709 = 10.9
391.4
134.2
320.1 105.8
970.1 + 26.1
FINGER SPIN
159.1 60.1
203.5 = 91.9
105.0
0 22.8
304.1 50.6
984.0
+ 18.4
SWIM
205.9
10.8
10.5
85.1
203.0 1
22.5
HOPPER STAND
10.6 50.2
809.1 - 98.4
592.2 101.2
900.1 100.1=150: 150.2
917.1
15.9
WALKER
143.5 22.8
191.
990.0
WALKER WALK
40.1 - 39.1
125.9
78.1
496. 77.2
102.4 30.0
943.1
- 56.7
CHEETAH RUN
230.1 1 99.8
206.10
75.1
102.4 29.5
169.1 1 20.3
807.0
1 17.5
QUADRUPED RUN
10.45
1: 91.9
316.3 4 33.6
195.4
91.3
33.5 1: 98.2
961.3
50.1
cartpolc swingup
finger spin
catch
Fish upright
1000
1000
750
750
750
750
300
530
500
500
250
250
walker stand
Walker walk
cheotah, run
hopper stand
1000
1000
1000
1000
750
750
750
900
500
500
250
250
250
250
5
10
10
10
Num Demonstrations
GIRL PARR
P-DAC
Expert
Figure 2: Visual Control Performance when varying number of demonstrations.
to accurately estimate the increase in time spent school and the military. Particularly, when the reward perturbations are smaller, the score-based counterfactual predictions are more accurate. For significantly larger perturbations, we would expect our method's accuracy to drop as the first order approximation is no longer valid.
5.2 BEHAVIOR IMITATION
Environment and Baselines: We experiment with high-dimensional robotics control environments from the Dedd Control Suite (DMC) (Tassa al. 2018). which widely used benchmark for RL and Imitation Learning (IL). For state control problems the agent has access to its proprioceptive states. For visual control the agent
three consecutive observation images of size 84 84 x 3 and uses action-repeat following the standard DMC protocol for visual RL (Yarats et al. 2022 We
compare GAC against Behavioral Cloning (BC) (Pomerleau) 1991). Adversarial IRL (AIRL) (Fu
al. 2018), Guided Cost Learning (GCL) Fiin al. (016b)). and Discriminator Actor- Critii (DAC)
(Kostrikov et al. 2019 Cohen et al. 2021). We report the mean performance and standard deviation
across ten random seeds where performance of each seed is measured as the mean reward of twenty
trajectories sampled from the final policy. (see Appendix for experiment details)
For state control, we experiment in the extreme low-data regime where only one demon- stration trajectory (consisting of 1000 state-actions) is provided. The experts for all tasks obtain rewards near 1000. Table shows that GAC outperforms all baselines while achieving expert level performance. For visual control, Sshoww that GAC outperforms baselines while achieving expert performance we found that the IRL baselines GCL and AIRL are incapable of performing well, which suggests that trading off bias for lower variance is beneficial in practice. While DAC achieves competitive performance some tasks such HOPPER STAND, it does not have the benefit of learning reward. Some limitations of GAC include longer compute time, potential optimization instabilities from the deadly triad (aan Hasselt et al. 2018), and double-sampling (Zhu & Ying 2020). A more complete discussion of the limitations well hyper-parameter ablation studies in Appendix
Results:
many hard visual control tasks with just one demonstration. In particular,
8

---

Published as a conference paper at ICLR 2023
Table 3: Reward Transfer when provided with len demonstrations in the original environment
DIRECT
AIRL
GCL
GAC (OURS)
SLIPPERY WALKER
302.5 + 29.5
493.2 + 49.3
201.9 + 89.6
789.3 + 93.3
STIP WALKER
205.2 - 98.3
455.2
+ 99.3
301.3 = 89.4
668.9 2
SLIPPERY CHEETAH
366.5 + 29.3
591.9
- 65.7
93.1 - 00.1
803.9
+ 98.9
STIFF CHEETAH
363.2
1 39.6
430.3
1 68.2
325.0
44.6
650.3
92.3
5.3 BEHAVIOR TRANSFER
Next we show that that learned rewards from GAC robustly transfer behaviors to perturbed environ- ments (Fu et al. 2018). We create variations of the bate-based DMC environments by altering the surface and joint Irictions to simulate realistic scenarios for robot learning where we may want to learn a single reward that can teach robots of different joint mechanics to mobilize in different walking surfaces. For example, the SLIPPERY WALKER environment has about 80% less friction the walking surface while STIFFCHEETAH has roughly 150% higher friction on the robot joints. Rewards are first learned on the non-perturbed environments using 10 demonstrations, then re-optimized via RL on the perturbed environments. As sanity check, also include the DIRECT baseline which simply uses policy learned in the non-pete cen-perturbed environment. Table 3 shows that GAC is able to outperform the baselines. However, GAC is not able to attain perfect expert level performance, which we believe is due the perturbed environments requiring slightly different state visitation strategies.
6 RELATED WORKS
Inverse Reinforcement Learning and Dynamic Discrete Choice: The field of IRL was pioneered by (gg et al.] 2000) and has since accumulated a rich history starting with early works that rely on linear reward models and hand-crafted features (Abbeel & Ng. 2004 Ramachandran & Amir
2007 Choi & Kim
Neu & Szepesvári, 2012). Since the introduction of the Maximum Entropy 20T0) framework, the field has gravitated towards learning more flexible rewards parametenzee by deep neural networks (Wulfmeier et al., 2013) paired with adversarial training Fuu et al. 2018: Ho & Ermon 2016 Finn et al. a) while some works attempt scale
2011
IRL (Ziebart et al. 2008
more classical methods such as Bayesian IRL Chan & van der Schaar 2021 Mandyam et al. (021).
Most related our work is gradient- gradient-based apprenticeship learning (Neu & Szepesvári 20122) which showed that the gradient of the infinite horizon Q-function in the standard RL. setting satisties a linear fixed point equation. Bellman gradient iteration (Li et al. 2017: 2019) and maximum sorkiimumlikelihowe likelihood IRL. VVommn 2014) take a computational approach to estimating approximate Q-gradients. Unlike these prior works, we generalize results to the MaxEntRL setting where we prove new properties of the Bellman score, and derive a dynamic programming algorithm that provably converges the infinite-horizow Q-gradient. Most significantly, we propose a model-free score estimation algorithm which scales to nigh-dimensional environments and apply it to various applications beyond IRI., The field of econometrics has a rich body of work on identifying Dynamic Discrete Choice (DDC) (Rust) 1994. |994||Acciiaaa AAcidiacono & Miller [2011: | 2020; Abbring & Daljord (2200) models which is equivalent to the problem of MaxEntIRI (Ziebart et al. 2008). DDC literature more so focuses on counter factual predictions (Christensen & Connault, 2019) which has various application areas such as in labor markets (Keane & Wolpin]] 1194), health care (Heckman & Navarro 2007), and retail competition
AAridiacono & Miller 2011).
Imitation Learning A related field of Imitation Learning (IL.) seeks learn policies from demon- strations (Pomerleau [991] Ho & Ermon 2016: Zhang et al. 2020 Rajaraman et al. 2020 Xu et al. 2020) which is viable alternative when the sole goal is behavior adoption. In recent years, IL has shown superior results than IRL for policy performance, particularly in the visual space. (Samak et al. 202T; Young et al., 202T) Techniques such as data augmentation and encoder sharing that have boosted performance in visual RL (Yarats al.) 2022) have been combined with adversarial IL. methods to solve challenging control environments (Cohen et al. 202T
7 CONCLUSION
We have studied the theoretical properties of the Bellman score which allowed us to derive a model- free algorithm, score-leaming, for score-estimation. We showed that score-learning has various applications in IRI,, behavior transfer, reward identification, and counterfactual analysis. We are looking forward to seeing future works that apply score-learning to other problems such as reward design (Hadfeld-Menell et al. (017) and explainable RI. (Puiutta & Veith 2020

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This research was supported by NSF (#1651565), ARO (W91INF-21-1-0125)) ONR N00001-23-11 2159), CZ Biohub, and HAL We would like to acknowledge Bertsekas & Tsitsikis (9955) for their descriptions of the Value Iteration algorithm which sparked erucial intuitions for the theory in this work.
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning. pp. 2004.
Jaap H Abbring and Øystein Daljord. Identifying the discount factor in dynamic discrete choice models. Quantitative Economics, 11(2)471-501, 2020.
Peter Arcidiacono and Robert A Miller. Conditional choice probability estimation of dynamic discrete choice models with unobserved heterogeneity. Econometrica, 19(6):1823-188 1867,,18777 2011.
Peter Arcidiacono and Robert A Miller. Identifying dynamic discrete choice models off short panels. Journal of Econometrics, 215(())473-48) 2020.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural
Information Processing Systems, 32, 2019.
Richard Bellman. Dynamic Programming. Dover Publications, 1957. ISBN 9780486428093.
Dimitri P Bertsekas and John N Tsitsiklis. Neuro-dynamic programming: an overview. In Proceedings of 1995 34th IEEE conference on decision and control, volume 1, pp. 560-564. IEEE, 1995.
Stephen Boyd, Lin Xiao, and Almir Mutapcic. Subgradient methods. lecture notes of EE392o. Stanford University, Autumn Quarter, 2004:2004-2005 2003.
Alex James Chan and Mihaela der Schaar. Scalable bayesian inverse reinforcement learning. In International Conference on Learning Representations, 2021. URL httptttt/pppeneeeeeee
net Eorum
4qR3coiNaIv
Jaedeug Choi and Kee-Eung Kim. Map inference for bayesian inverse reinforcement learning. Advances in Neural Information Processing Systems, 24, 2011.
Timothy Christensen and Benjamin Connault. Counterfactual sensitivity and robustness. arXiv
preprint arXiv:/904 00989, 2019.
Samuel Cohen, Brandon Amos, Mare Peter Deisenroth, Mikael Henaff, Eugene Vinitsky, and Denis Yarats. Imitation learning from pixel observations for continuous control. In Deep RL Workshop NeurlPS 2021, 2021.
Kenji Doya. Reinforcement learning: Computational theory and biological mechanisms. HFSP journal, 1(1):30, 2007.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852, 2016a.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pp. 49-58, June 2016b.
Justin Fu, Katic Luo, and Sergey Levine. Learning robust rewards with adverserial inverse re- inforcement learning. In International Conference on Learning Representations, 2018. URL
https://openreviie
net/ wtw.:fommmmmdddill------
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International conference on machine learning, pp. 1352-1361. PMLR, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critie: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference machine learning, pp. 1861-1870. PMLRR, 2018.
10

---

Published as a conference paper at ICLR 2023
Dylan Hadfield-Menell Meneel, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
James J Heckman and Salvador Navarro. Dynamic discrete choice and dynamic treatment effects. Journal of Econometrics, (56(2):3413990) 2007.
Tsubasa Hirakawa, Takayoshi Yamashita, Toru Tamaki, Hironobu Fujiyoshi, Yuta Umezu, Ichiro Takeuchi, Sakiko Matsumoto, and Ken Yoda. Can ai predict animal movements? filling gaps in animal trajectories using inverse reinforcement learning. Ecosphere, 910))e02447, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp.
2016.
Myrto Kalouptsidi, Paul T Scott, and Eduardo Souza-Ro nuzza-Rorrigue. Identification of counterfactuals in dynamic discrete choice models. Technical report, National Bureau of Economic Research, 2015.
Michael P Keane and Kenneth I Wolpin. The solution and estimation of discrete choice dynamic programming models by simulation and interpolation: Monte carlo evidence. The review of economics and statistics, pp. 648-672. 672, 1994.
Michael P Keane and Kenneth I Wolpin. The career decisions of young men. Journal of political Economy, 105(3):473-522, 1997.
Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in inverse reinforcement learning. In International Conference on Machine Learning, pp. 1496-5505 PMLR, 2021.
lyya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Discriminator-accor-rrittt Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019. URL https:/ openreview n forum? Per2iddiit Hk4-HkkffpAA5km
Bai Li, Changyou Chen, Hao Liu, and Lawrence Carin. On connecting stochastic gradient meme and differential privacy. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 557-566. PMLR, 2019.
Kun Li, Yanan Sui, and Joel W Burdick. Bellman gradient iteration for inverse reinforcement learning. arXiv preprint arXiv: /707 777700776, 2017.
Aishwarya Mandyam, Didong Li, Diana Cai, Andrew Jones, and Barbara Engelhardt. Efficient bayesian inverse reinforcement learning via conditional kernel density estimation. In Fourth Symposium on Advances in Approximate Bayesian Inference, 2021.
Emre Neftei and Bruno B Averbeck. Reinforcement learning in artificial and biological systems. Nature Machine Intelligence, (3):1133-433, 2019.
Gergely Neu and Csaba Szepesvári. Apprenticeship learning using inverse reinforcement learning and gradient methods. arXiv preprini arXiv: 1206.526, 2012.
Andrew Y Ng, Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, pp. 663-670, 2000.
Yael Niv. Reinforcement learning the brain. Journal of Mathematical Psychology, 53(3):139-154, 2009.
Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation, 3(1):88-97, 1991. ISSN 0899-7667.
Erika Puiutta and Eric Veith. Explainable reinforcement learning: A survey. In International cross-domain conference for machine learning and knowledge extraction, pp. 77-95. Springer, 2020.
Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. Advances in Neural Information Processing Systems, 33:2914-299 2020.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7,
pp. 2586-2591 2007.
11

---

Published as a conference paper at ICLR 2023
John Rust. Structural estimation of markov decision processes. Handbook of econometrics, 4: 3081-3143, 1994.
Tanmay Vilas Samak, Chinmay Vilas Samak, and Sivanathan Kandhasamy. Robust behavioral
cloning for autonomous vehicles using end-to-end imitation learning. SAE' International Journal of Connected and Automated Vehicles, 4(3):279-22 295, aug 2021. ISSN 2574-0741. doi: 10.4271/
12-04-03-0023. URLpttps: / /ttp://ioooorooo//ooooooo 10. 1111111122-4000-00000000
Toryn LJ Schafer, Christopher K Wikle, and Mevin B Hooten. Bayesian inverse reinforcement learning for collective animal movement. The Annals of Applied Statistics, 16(2):99--001, 2022.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panncershelvam,, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484, 2016.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy Lillicrap, and Martin Riedmiller. DeepMind control suite. arXiv preprint 1rXiv:1801.00690 January 2018.
Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint 18220488,, 2018.
Monica C. Vroman. Maximum likelihood inverse reinforcement learning. Rutgers The State University of New Jersey-New Brunswick, 2014.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce- ment learning. arXiv preprint arXiv: 04888, July 2015.
Tian Xu, Ziniu Li, and Yang Yu. Error bounds of imitating policies and environments. Advances in Neural Information Processing Systems, 33: 33:15737715749, 2020.
Shoichiro Yamaguchi, Honda Naoki, Muneki Ikeda, Yuki Tsukada, Shunji Nakano, Ikue Mori, and Shin Ishii. Identification of animal behavioral strategies by inverse reinforcement learning. PLoS computational biology, 14(5)=e100612) 2018
Denis Yarats and Ilya Kostrikov. Soft actor-critic (sac) implementation in pytorch. https:/ github. com http:///essyaaattttttrrrooo _sac 2020.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International Conference on Learning Representations, 2021. URL
net for http://oommmmmm-omeoeee 6sTvaa
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved duta-nueeented reinforcement learning. In International Conference on Learning Representations, 2022.
yyees
Sarah Young. Dhiraj Gandhi, Shubham Tulsiani, Abhinav Gupta, Pieter Abbeel, and Lerrel Pinto. Visual imitation made easy. In Conference on Robot Learning, pp. 1992-2005. PMLR, 2021.
Amy Zhang, Harsh Satija, and Joelle Pineau. Decoupling dynamics and reward for transfer learning. 2018. URL httls::////// http//ppwweeeeeeeeeeeeerrrrnaraa
Yufeng Zhang, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Generative adversarial imitation learning with neural network parameterization: Global optimality and convergence rate. In International Conference on Machine Learning, pp. 1 1004-11 1054. PMLR, 2020.
Yuhua Zhu and Lexing Ying. Borrowing from the future: An attempt to address double sampling. In Mathematical and scientific machine learning, pp. 246-268. PMLR, 2020.
Brian D Ziebart, Andrew Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433-1438, 2008.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In ICML, 2010.
12


Reviews:
[{"review_id": "jBuNl96nLfD", "reviewer": "Reviewer_DpqQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces an algorithm that computes two utile quantities, that they call Q-gradient (g(x, a)) and Bellman-score (s(x, a)), given transitions from a replay buffer, .\nThe Q-gradient can be used to update a reward function in IRL, and the Bellman-score to predict the adaptation of a policy to a change of the reward function.\nIn both cases, the experiments shows that both applications provides strong results, outperforming the baselines used for comparison.\n", "summary_of_the_review": "Overall, although the paper could be more impactful and would benefit some writing clarifications, I think it already desserve communications and will benefit the RL community. I rate for a marginal accept, but may increase my score.\n\nMinor comments:\n\n- Preliminaries, in the unconditional occupancy measure two lines after Eq1, I think Pi_T should be replaced by Pi_0.\n\n- At the end of page 2, I am pretty sure that the entropy as used in the MaxEntRL setting is only a function of the state, and is the entropy of the policy at a given state. Maybe a comment is missing here.\n\n- Page 3 Line 1: \"the the\"\n\n- Algo 2: \"if \\psi is not None\" --> \"if \\phi is not None\"\n\n- A related work for IRL that also have strong results with single demonstrations: Primal Wasserstein Imitation Learning by Dadashi and Al.\n\n", "strength_and_weaknesses": "As far as I understand the maths are Ok (I haven't been through the appendix), and the approach definitely makes sense. \nThe results are impressive, especially for the IRL experiments with a single demonstration. The experimental protocol is correct (they used 10 random seeds for each baseline and averaged scores of policies over 20 sampled trajectories).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "iHuegsXOKC6", "reviewer": "Reviewer_9iGT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the problems of understanding and learning optimal policies from demonstrations, including reward identification, counterfactual analysis, behavior imitation and behavior transfer. Under the  framework of MaxEntRL, they first propose the concept of Bellman score which captures how changes in the reward functions influence the optimal policies, and further show that Bellman score can be calculated as a difference of the gradients of optimal Q-values w.r.t the reward parameters. Then they introduce the Bellman score operator and prove that applying this operator repeatedly yields the true Q-gradients. Based on this operator, they develop a model-free algorithm Score-learning to learn Bellman score with finite samples. Next, they apply the Score-learning method on tasks of IRL and counterfactual predictions. In experiments, they evaluate that applying score-learning can achieve SOTA performance on the mentioned four tasks.", "summary_of_the_review": "In this work, the authors propose the concept of Bellman score, which is novel. To learn Bellman score with finite samples, they develop the algorithm score learning based on DQN. The Score-learning method has a broad applicability on tasks of IRL and counterfactual predictions. However, the benefits of the developed IRL method GAC over existing IRL methods are not clear. Besides, GAC suffers the computational issue. Some claims in this paper are not well supported. In conclusion, I recommend the score of 5. \n", "strength_and_weaknesses": "\nStrength:\n\n1. The authors propose the concepts of Bellman score and Q-gradient, which are novel to my knowledge. The proposed Bellman score and Q-gradient have an important role on the problems of understanding and learning optimal policies from demonstrations, including inverse reinforcement learning (IRL) and counterfactual analysis.\n\n\n\n\n\nWeakness:\n\n1. In section 4.1, they derive a new formulation for the gradient of the MaxEntIRL (i.e., Eq. (7)) and propose a new MaxEntIRL algorithm GAC based on Eq. (7) and Score-learning. However, I cannot see the benefits of GAC over existing IRL methods like GAIL, DAC. \n\n   The authors claim that existing IRL methods utilize the formulation of Eq. (8) and thus suffer the issues of data reusing and high variance. First, for the data reusing issue, DAC has already addressed this issue by using data in replay buffer to update rewards. \n\n   Second, I think there is no high variance issue in performing SGD updates based on Eq. (8) and the analogy between Eq.(8) and MC-PG method made by the authors does not make sense. In Eq. (8), we use the sample average of reward gradients to estimate the expectation. Unlike the policy value (a multi-step value with a scale of 0 to T) estimated in MC-PG method, the reward gradient is a one-step value and thus using MC estimates does not have a high variance. Instead, GAC based on Eq. (8) could have a high variance since it only uses initial state-action pairs for estimating.             \n\n2. The computational cost of GAC is large. Compared with existing IRL methods like GAIL, AIRL and DAC, GAC additionally maintains the Q-gradient network, target Q-gradient network and score network. As pointed in Appendix C, GAC requires roughly 2+ more compute time than the baselines.   \n\n3. The authors claim that score learning can be applied in behavior transfer. In experiments, they show that learned rewards from GAC can transfer behaviors to perturbed environments. As far as I am concerned, unlike Adversarial IRL, GAC does not contain special algorithmic designs for handling transfer. It remains a question why GAC can transfer rewards to new environments.    \n\n4. The theory in this paper is derived within the scope of MaxEntRL. The authors claim that similar results for the standard RL can be found in the Appendix. However, I do not see these results in the Appendix. \n\n5. This paper contains the review of algorithm progress in IRL and IL. However, some basic works on the theory of IRL and IL are missing, see e.g., [1, 2, 3].\n\nReferences:\n\n[1] Y. Zhang, Q. Cai, Z. Yang, and Z. Wang. Generative adversarial imitation learning with neural network parameterization: Global optimality and convergence rate. ICML, 2020.\n\n[2] N. Rajaraman, L. F. Yang, J. Jiao, and K. Ramchandran. Toward the fundamental limits of imitation learning. NeurIPS, 2020.\n\n[3] T. Xu, Z. Li, and Y. Yu. Error bounds of imitating policies and environments. NeurIPS, 2020.\n\nMinor issues:\n\n1. The first line in page 3: \"the the soft Q-values\"\n2. In definition 3, the shape of the Bellman score should be d \\times |X \\times A|? \n\n3. In the line of \"update Q-gradient\" in Algorithm I, g (x, a) should be g (x', a') under the transition probability of P (x', a' |x, a).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "hd_4DjY4k_7", "reviewer": "Reviewer_9PVy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper contributes to the study of gradient-based inverse reinforcement learning, in particular, learning a reward function by gradient ascent, under which an observed policy is optimal. The main problem of interest is the estimation of the gradient of the policy return with respect to the parameters of the reward function. It is shown that this gradient follows a dynamic-programming Bellman-type equation similar to the actual policy return, which can then be exploited to derive gradient estimation algorithms based existing RL algorithms. Substantially improved performance is demonstrated on benchmarks.", "summary_of_the_review": "This paper makes an important, well-argued contribution to a relevant problem. Novelty and significance are relatively high.", "strength_and_weaknesses": "This paper has a strong basic insight, which is utilized well. There are no obvious flaws that struck me while reading.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XFb69ad5JS", "reviewer": "Reviewer_gbZN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "To recover the parameters of an unknown reward function, this work introduces the notion of the Bellman score - corresponding to the (vector-valued) gradients of the log probabilities of the true optimal policy with respect to the true unknown reward. Hence, it proposes an algorithm to estimate this quantity via approximate dynamic programming procedures. The authors show improvements over prior literature performing inverse reinforcement learning and transfer learning in a continuous control benchmark. They also show their algorithm is effective for reward estimation and predicting changes in optimal behavior in an econometrics-based simulation environment.", "summary_of_the_review": "I found the contribution of this work to be novel and relevant, with the potential to have significant implications. However, in its current form, I believe there are several flaws in the paper, undermining its quality, clarity, and potential fairness. Moreover, given the nature of the contribution, I think that providing a way to reproduce and verify the experiments is of particular importance. Hence, I believe this paper is currently borderline, but I am very open to updating my score based on the rebuttal.", "strength_and_weaknesses": "**Strengths**\n\n1. The novelty and contribution of this work are significant.\n2. The reward identification and IRL problems are very relevant and have several downstream implications.\n3. The evaluation is performed for different problems, highlighting the generality of the procedure.\n\n**Weaknesses/Areas of improvement**\n\n1. I believe the paper does not appropriately address the limitations of the proposed algorithm in continuous/high-dimensional environments. I would imagine challenges related to the deadly triad [1] would provide even greater instabilities when applied to an estimator of a high-dimensional vector. Moreover, using a backup operator with a buffer of stored behavior will incur the 'double-sampling' issue in stochastic environments (e.g. [1, 2]). Currently, there only is a brief mention of the limitations in the Appendix, solely listing minor and non-specific computation-time considerations.\n\n2. I have some concerns about the empirical IRL results. In particular, the authors mention in the Appendix they perform a hyper-parameter search for their method and run it until convergence. In contrast, the authors mention that the baselines use off-the-shelf implementations that were already tuned for the considered environments. However, since they mainly consider evaluating given unusually low amounts of data, I believe that also the IRL baselines' hyper-parameters should be re-tuned for this specific experimental setting and be allowed to run until convergence (of the reward/agent behavior).\n\n3. Some of the stated contributions sound a bit exaggerated without proper context. For instance, the authors claim \"[We] derive, for the first time, the gradient of the Maximum Entropy IRL (Ziebart et al., 2008; Finn et al., 2016b; Fu et al., 2017) objective in the fully general setting with stochastic dynamics case and non-linear reward models.\" I believe this result (Theorem 2) already follows quite intuitively from the work of Ziebart et al. 2010 [3], which would be important to explicitly specify.\n\n4. The authors state \"Hyperparameter ablation studies as well as compute time comparisons are in Appendix C.\" However, Appendix C does not contain any table/performance curve showing empirical results, but only two paragraphs with some general considerations and recommendations. Such missing information would be very important to get an intuition for the sensitivity and generality of the proposed algorithm. Since the authors state \"More detailed ablation results will be added shortly.\", do they plan to add this data for the rebuttal revision?\n\n4. I believe this work provides far too little information to allow the reproduction of the results. I find this concern of particular relevance for correctly assessing the contribution, given its significant novelty. As the authors mention that \"The implementation code will be released shortly.\", I would appreciate it if this code could be anonymously shared with the reviewers for the rebuttal phase.\n\n5. I think there is significant room to improve the paper's writing. For instance, I encountered repetitions of sentences even within the same paragraph, e.g. \"The theory in this section will be derived in the Maximum Entropy RL (MaxEntRL) setting (Haarnoja et al., 2017; Kostrikov et al., 2020) but similar results for the vanilla RL setting without entropy regularization can be found in the Appendix [...] The theory in the remaining sections will be derived in the MaxEntRL setting for readability. Similar results for the standard RL setting without entropy regularization can be found in the Appendix.\" Moreover, I found several typos, e.g.: \"i.e gradient of\", \"Gradients of the objective in Eq. 6 is highly\", \"the jacobian\", \"The key challenges with this approach are the difficulty of\",... I would encourage the authors to perform additional proofreading of the text to improve clarity.\n\n*References*\n\n[1] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.\n\n[2] Zhu, Yuhua, and Lexing Ying. \"Borrowing from the future: An attempt to address double sampling.\" Mathematical and scientific machine learning. PMLR, 2020.\n\n[3] Ziebart, Brian D. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. Carnegie Mellon University, 2010.\n\n**Additional questions**\n\n1. The underlying assumption of the proposed method is that the observed behavior was generated by a MaxEnt policy. What would the authors empirically expect to observe if this assumption was broken? Would there be any way to verify if this were the case?\n\n2. Given we might be interested in imitating some demonstrations observing an external agent, without access to the agent's actions, can we leverage the same procedure to recover the score of a hypothetical reward function only dependent on the visited states?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WxS9rWLqh05", "reviewer": "Reviewer_VmMx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors of this study make the crucial point that by understanding how variations in the underlying rewards impact optimal behavior, several discussed issues on imitation learning and inverse reinforcement learning may be resolved. This amount can be approximated locally by the Bellman score, which is the gradient of the log probability of the best strategy with respect to the reward. The authors present such a Bellman score operator, which can be used to estimate the score directly and is shown to converge to the gradient of the infinite-horizon optimal Q-values with respect to the reward. Using their theory as a guide, the authors construct an effective score-learning technique that can be applied for score estimation in high-dimensional state-actions spaces. The authors demonstrated that score-learning might be used to transfer policies between environments, accomplish state-of-the-art behavior imitation, make counterfactual predictions, and accurately identify rewards.", "summary_of_the_review": "Given my previous detailed comments, I believe this work is consistent in terms of theory, the presentation of the introduced approach, and empirical studies. However, there are still issues with the paper. Specifically, I wouldn't say that this paper is reproducible, as no detailed experimental setup or code is given. Furthermore, the quality and novelty are slightly below the standards for a venue like ICLR. In particular, the theory and background are heavily based on prior work. Only a set of trivial calculus extends the previous approaches. Overall, I couldn't say this work is acceptable unless my concerns are addressed through straightforward comments or references.", "strength_and_weaknesses": "**Strengths:**\n- The paper is well written, and the flow is excellent.\n- The motivation is described well.\n- Empirical studies are promising and seem to be representative of a successful comparative evaluation. For example, the proposed method achieves near-optimal performance for the IRL experiments with a single demonstration, and the use of 10 random seeds follows the widely-studied deep RL benchmarking standards.\n- The approach has some novelty in it. \n- The adaptations of the introduced algorithm to the considered downstream tasks seem credible.\n\n\n**Weaknesses:**\n- The proposed method heavily relies on the _Maximum Entropy RL (MaxEntRL)_ algorithm and the [work](https://arxiv.org/abs/1702.08165) by Haarnoja et al. (regarding notation, background, theory, etc.). While the authors claim to advance the prior art, the additional novelties are unclear and decrease the soundness of the presented work due to such an issue.\n- The authors constantly discuss \"model-free settings with high-dimensional continuous state-action spaces.\" However, the experiments are performed on a set of discrete action spaces. I couldn't understand the connection until I looked at the supplementary material in which the SAC algorithm is used for a part of the RL learner, which remains unclear in the main text. \n- Can the proposed method be extended to standard continuous control (e.g., OpenAI Gym benchmarks)? I know it can't (mostly), yet the authors should address that. \n- The preparation of the paper is untidy. This reduces the quality and demonstrativeness of the proposed method. See my detailed comments below. \n- There are many references (maybe some belong to the supplementary text); consider reducing them. For example, the ones in the introduction section, i.e., references to the studies on economics. In addition, most of the references, e.g., the SAC algorithm, point to the pre-print versions. Please replace them with the published versions.\n- I suggest the authors define the concepts of reward identification, counterfactual analysis, behavior imitation, and behavior\ntransfer, rather than only giving examples, as many other readers not be familiar with them.\n- Although I believe that the proofs of the theorems are correct, not giving them in the main text prevents the persuasiveness of the math in the paper, which is the backbone of the proposed dynamic programming algorithm. I suggest authors provide at least a single proof in the main text that they believe is the most important.\n- Although an extensive set of experiments are performed to evaluate the proposed method and compare it against the baselines, the discussion essentially translates the results in the Figures or Tables to words. Please discuss your comparative evaluations in-depth, such as why your approach outperforms baselines, what they lack, what are the limitations of your approach and, how they can be overcome, etc.\n\n**(Very) Detailed Comments:**\n- Last paragraph of section 3: There is a sentence \"(We omit the finite-horizon version for brevity, but it is very similar to the infinite-horizon case)\". Why didn't the authors show this (or give a reference) instead of briefly mentioning it?\n- First paragraph of the introduction: The reward in an MDP may not always capture the goal. Rewards can sometimes represent the intermediate goals that are mandatory to obtain the final goal (or aim of the task). Consider changing the corresponding sentence with something like: \"The reward covers the information about the goal...\"\n- Second paragraph of the introduction: \"an RL algorithm...\" not \"a RL algorithm\".\n- Preliminaries: An MDP does not necessarily consist of a discrete state and action space. If such a case is considered, please explain why.\n- First line on page 3: Duplicated \"the\". \n- Second paragraph of Preliminaries: Consider referring to Bellman, Dynamic Programming (Dover publications) while mentioning dynamic programming. \n- Line 5 of the second paragraph on page 4: Put a comma after (x, a).\n- Second sentence of the first paragraph in Section 4.1: Remove \"from\" before the dot.\n- Put floats on the same page or later when they are first mentioned. For instance, Algorithm 3 is first mentioned on page 6, but the pseudocode is given on page 5. \n- Give a cross-reference to Table 1; what does it tell (e.g., \"Table 1 shows the results for...\")?\n- I suggest the authors not highlight the highest scores in Table 2; make them bold instead.\n- Put punctuation after equations if they are used in conjunction with sentences.\n- Algorithm 2: \"if \\psi is not None\" should be \"if \\phi is not None\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7aiejurjMOF", "reviewer": "Reviewer_1Fw8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a dynamic programming algorithm (\"score iteration\") which can efficiently compute Q-gradients, and thereafter, \"Bellman scores\". This Bellman score is useful because it directly gives information about how changes in reward affects policy. The authors then demonstrate its utility for doing behaviour imitation, policy transfer, and counterfactual predictions. ", "summary_of_the_review": "A clearminded paper that would be of interest to the community and should hopefully attract further investigation into the benefits of Bellman score in RL, and even better ways to efficiently compute it. ", "strength_and_weaknesses": "Strengths: this paper demonstrates strong benefits of using the Bellman score, in particular for maximum entropy IRL and counterfactual predictions. And most crucially, they discover a dynamic programming algorithm that enables its efficient computation. The algorithm is conceptually solid and reasonable, and well motivated by classic ideas in deep RL (esp DQN, bootstrapping, etc). The demonstrations on IRL and counterfactuals were strong. Relevant baselines were compared, and a good variety of tasks were used. \n\nOverall, for its purpose, it is pretty solid. Develops theory and shows relevant experiments to make a central coherent point about the benefits of efficiently computing Bellman score. \n\n\nWeaknesses: More details on exact implementation would be helpful, so that readers can have a chance to replicate and further study their phenomena. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "3bPRBf3eVzu", "reviewer": "Reviewer_nKZi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces the Bellman score, which is the gradient of the log-probabilities of the optimal policy with respect to the reward parameters, and a corresponding operator to learn the gradient of the Q-function (which is related to this quantity) via fixed-point iteration.\nThe authors show that score-learning is a versatile approach and derive an actor-critic algorithm for IRL (GAC).  \nThey empirically study several applications: imitation learning, inverse RL, reward transfer, counterfactual estimation.", "summary_of_the_review": "This is an exciting paper with a lot of potential impact.\n\nI have made a small number of questions/comments that could be addressed by the authors.\n\nI recommend acceptance.\n", "strength_and_weaknesses": "### Strengths\n\nThis is as far as I can tell a novel, neat idea that is quite versatile.\n\nNo knowledge of the environment dynamics is required.\n\nThe paper is overall well-written, clear, and to the point. It is a bit dense though, and might benefit from additional figures.\n\nEmpirical results on various tasks (IRL, imitation learning, reward transfer) are good.\n\n### Weaknesses\n\nI would advocate for using optimal (or near-optimal) instead of the term rational that I find confusing, e.g. one could think that it means sensible according to human evaluation. Unless this is related to the fact of using demonstrations, in which case it should be better explained.\n\nNit: Algorithm 1 misses the initialization of the Bellman score\n\nIt is not clear to me how the gradient of the reward appears in the Bellman score operator (definition 2). Could authors expand on that and add a clarification in the text?\n\nAlgorithm 2 leverages the gradient of the reward. Does that mean that score learning only works in tasks where a reward parameterization exists and is known? In that case, I think the current state of the paper is not clear enough about this. I do get that this is not a limitation for IRL and IL tasks since a reward is to be learned anyway.\n\nI encourage the authors to open-source their code to maximize the impact of their contribution.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%