In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before an Ongoing Trajectory Terminates
Authors: Shicheng Liu
Minghui Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: inverse reinforcement learning
Primary area: reinforcement_learning

Abstract: Inverse reinforcement learning (IRL) aims to learn a reward function and a corresponding policy that best fit the demonstrated trajectories of an expert. However, current IRL works cannot learn incrementally from an ongoing trajectory because they have to wait to collect at least one complete trajectory to learn. To bridge the gap, this paper considers the problem of learning a reward function and a corresponding policy while observing the initial state-action pair of an ongoing trajectory and keeping updating the learned reward and policy when new state-action pairs of the ongoing trajectory are observed. We formulate this problem as an online bi-level optimization problem where the upper level dynamically adjusts the learned reward according to the newly observed state-action pairs with the help of a meta-regularization term, and the lower level learns the corresponding policy. We propose a novel algorithm to solve this problem and guarantee that the algorithm achieves sub-linear local regret $O(\sqrt{T}+\log T+\sqrt{T}\log T)$. If the reward function is linear, we prove that the proposed algorithm achieves sub-linear regret $O(\log T)$. Experiments are used to validate the proposed algorithm.

In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before an Ongoing Trajectory Terminates
Shicheng Liu & Minghui Zhu School of Electrical Engineering and Computer Science
Pennsylvania State University
University Park, PA 16802, USA
{sf15539,muzz mwz16) 0psu. edu
Abstract
Inverse reinforcement learning (IRL) aims to learn a reward function and a corre- sponding policy that best fit the demonstrated trajectories of expert. However, current IRL works cannot learn incrementally from an ongoing trajectory because they have to wait to collect least one complete trajectory to learn. To bridge the gap, this paper considers the problem of learning a reward function and a corresponding policy while observing the initial state-action pair of ongoing trajectory and keeping updating the learned reward and policy when new state- action pairs of the ongoing trajectory observed. We formulate this problem an online bi-level optimization problem where the upper level dynamically adjusts the learned reward according to the newly observed state-action pairs with the help of a meta-regularization term, and the lower level learns the corresponding policy. We propose novel algorithm solve this problem and guarantee that the algorithm achieves sub-linear local regret O + log T + VT T)). If the reward function is linear, we prove that the proposed algorithm achieves sub-linear regret (log T'). Experiments are used to validate the proposed algorithm.
Introduction
Inverse reinforcement learning (IRL) aims to learn reward function and a corresponding policy that are consistent with the demonstrated trajectories of an expert. recent years, several IRL methods are proposed to help learn the reward and policy, including maximum margin methods [ 2
maximum entropy methods 3 4) maximum likelihood methods [.6]. and Bayesian methods
The aforementioned IRL. works learn from pre-collected demonstration sets and do not improve the learned model during deployment. Online IRL 2 IC [T]] instead can learn from sequentially arrived demonstrated trajectories and continuously improve the learned reward and policy from the newly observed complete trajectories. However, recent applications of IRL motivate the need to learn incrementally from ongoing trajectory before it terminates. For example, inferring moving shooter's 's intention from its ongoing movement in order to evacuate the hiding victims [2 before the shooter finds them. In this case, need to quickly update the inference about the shooter's intention once a new movement of the shooter is observed, so that we can use the latest inference to plan a rescue strategy soon possible. We cannot wait until the shooter trajectory ends, in the shooter has found the victims. Another example is learning a target customer's 's investment preference from its daily updated investment trajectory in stock market [13] in order to recommend appropriate stocks [4 [5] before other competitors get this customer. However, current IRL works cannot learn from an ongoing trajectory because they have wait collect at least one complete trajectory to learn from. To bridge the gap, this paper proposes in-trajectory IRL, new type of IRL that learns a
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

reward function and a corresponding policy at the initial state-action pair of an ongoing trajectory, and keeps updating the learned reward and policy once a new state-action pair of the ongoing trajectory is observed. We summarize our contributions as follows:
Contribution statement. This paper proposes the first in-trajectory IRI. framework, termed meta- regularized in-trajectory inverse reinforcement learning" (MERIT-IRL), to learn a reward function and corresponding policy from an ongoing trajectory. Our contributions are twofold.
First, we formulate this in-trajectory learning problem as an online bi-level optimization problem where the upper level continuously updates the learned reward according to the newly observed state-action pairs and the lower level computes the corresponding policy. We develop a novel online learning algorithm (MERIT-IRL) to solve this problem. The major novelty of the proposed algorithm is that we propose a novel reward update mechanism specially designed for the in-trajectory learning setting. This special reward update not only aims to explain the expert trajectory observed so far, but also aims to consider for the future. Moreover, since the data is lacking as there is only one ongoing trajectory, we introduce meta-regularization term to embed prior knowledge and avoid overfitting
Second, we theoretically guarantee that MERIT-IRL achieves sub-linear local regret O(VT + +oo T VT log T). If the reward funetion is linear, prove that MERIT-IRL achieves sub-linear regret Olog T). The major novelty of the theoretical analysis is to address the difficulty that the input data is not identically independent distributed (i.i.d.) but temporally correlated, i.e., the data (st, ae) at each time is affected by the data (s;-1, a1- at last time.
2 Related works
Due to the space limit, we only discuss the related works on learning from incomplete trajectories here, and we include the discussion more related works in Appendix A.
Papers [6 use partial trajectories update the learned reward function by comparing the expert trajectories after the expert states and the trajectories starting from those expert states rolled out by the learned policy. However, given the current expert state, they use expert trajectory suffix (i.e., future trajectory) to compare while we can only access expert trajectory prefix (i.e., previous trajectory) from an ongoing trajectory. Papers [8
19 20 use imitation learning to learn from incomplete demonstrations. In specific, paper [18] uses a discounted sum along the future trajectory as the
weight for weighted behavior cloning and works effectively even if only portions of trajectories are observed. Papers [19] 20 extend GAIL (2] to solve for the case where the action sequences are not complete. However, these works all require a pre-collected set of demonstrations so that they are not in-trajectory learning since the trajectory in their cases is not ongoing.
3 Problem Formulation
In this section, we formulate the problem of in-trajectory IRI.. In in-trajectory IRI., there is an expert whose decision makinn based on a Markov decision process (MDP). An MDP tuple (S, AS,A, TE, Po. P) which consists of a state set S, an action set A, a discount factor 7 E [0. 1). a reward function rg: S A + R. and the initial state distribution Po(.). The state transition probability (density) function is denoted by -(-I--) such that P(s'|s, ss,a) denotes the probability (density) of state transition to s' from s by taking action a. The expert is using its policy me to demonstrate an ongoing trajectory Ç = S A Sf ...... and at each time t, only the state- action pair ( (S. A is observed. We want to learn a reward function and a corresponding policy from the ongoing trajectory and update the learned reward function and policy at each time t.
employ bi-level learning structure. In this structure, the upper level learns a reward function while the lower level aims to find an associated policy by solving an RL problem under the current learned reward function. Inspired by their bi-level learning structure, we formulate the in-trajectory IRL. problem as an online non-convex bi-level optimization problem. In specific, we aim to learn a reward function To (parameterized by 0) in the upper level and policy corresponding to re in the lower level. The loss function at time ł is defined as:
Many IRL algorithms [5] 6 22 23 24
2xy 110 2
(0: (S, AA5) A --"10p log To (AF15 +
011
To arg max Jo(TT) + HT)). (1)

---

Note that (0;;; (S )) is defined using Ra and f is the parameter of the reward function re, here the policy Ty also parameterized by 0 because it is computed by solving an RL problem (in the lower level) under the reward function re. and thus is indirectly parameterized by 0. Maximum likelihood IRI. (MI.-IRI.) [5] has a similar bi-level formulation with ( however, ML.-IRL only solves an offline optimization problem and its analysis does not hold for non i.i.d. input data and continuous state-action space. We discuss our distinctions from ML-IRL in Appendix A.I]
The upper-level loss function L, has two terms. The first term -7" log To (Af ( is the discounted negative log- likeelklioood of the state-action pair (S. A.f) at time and the second term A= |10 0| is the discounted rettarreuulrrzztioon term [25] where À is a hyper-parameter. The likelihood function commonly used in IRL. S G to learn reward function. Basically, the upper-level loss function at time & encourages to find a reward function ro that makes the observed state-action pair (S AA)) most likely and meanwhile, the reward parameter 0 should not be too far from the prior experience, i.e., the meta-prior 0. Note that 0 is pre-trained meta-prior that embeds the information of "relevant experience". We will introduce the training of 0 in Subsection 4.3 and Appendix C
The lower-level problem is used to compute Te using the current reward function r/g- It proposes to find policy सe that maximizes the entropy-regularized cumulative reward Jø(TT) + H (TT) The cumulative reward of a policy under the reward function re is (T) - E ( ( At)l where the initial state is drawn from Po- The causal entropy of a policy ı is defined as H(T) & 13 Ei 7² log T [A
Since the expert demonstrates ((SF,,)"))) )/20 sequentially, we have a sequence of loss functions {L:(0; (SF, ))/20 We use this sequence of loss functions ( to formulate an online learning problem. A typical online learning problem is to minimize the regret: Σi [0, (SF, A4)) ming Σ L,(0: (S A.)). However, it is too challenging to minimize the regret in our case because the loss function L, could be non-convex. Therefore, we aim to minimize the local regret which is widely adopted in online non-convex optimization [26] 27 and online IRI. [T]. The local regret quantifies the general stationarity of a sequence of loss functions under the learned parameters. In specific, given a sequence of loss functions (f((z)) the local regret
26 27 at time
is defined as ll | Ei f((=a))) which quantifies the gradient norms of the average of all the previous loss functions under the current learned parameter Zt. The total local regret is defined as the sum of the local regret at each time i, i.e., 2o= [ Σ fi(x))]] In our case, we replace {ffe J220 with the loss function h2 defined in (I) and thus formulate the local regret which has a bi-level formulation. We aim to minimize the following local regret:
Σ
(L, (S. (. AF))I
E (((S) (((S AA))> (.)0 (.))
(2)
s.t. To. arg max Ja. (T) H(T),
(3)
where (SF AAF) P7 (,)), and PT (T. is the state-action distribution induced by π at time f in the MDP
IPT means that ASf,AA is drawn from the state-action distribution
Difficulties of solving problem 33.. We want to design fast algorithm to solve problem ( since we need to finish the update of and ı before the next state-action pair is observed and the time between two consecutive state-action pairs can be short. However, designing and analyzing such a fast algorithm difficult due to the following challenges:
(i) First and foremost, current state-of-the-arts 26 27 on online non-convex optimization use follow- the-leader-based algorithms which solve ming Σ L,( (S. ))) to near stationarity at each time t. This is moeecconumiig because they require multiple gradient descent updates of O. One way to alleviate this problem is to use online gradient descent (OGD) which only updates 0 by one gradient descent step at each time However, since OGD does not solve the problem to near stationarity at any time f, it is extremely difficult to quantify the overall stationarity after T iterations. While ceteeeetttiiiiiiii optimization, it is rarely studied online non-convex optimization. The recent work [I] uses OGD to quantify the local regret, however, its analysis can only hold when the input data is i.i.d. In contrast, the input data our problem is not i.i.d. In specific, the input data at time t (i.c., (S. AA) ) is actually affected by the input data at last step (i.c.,
OGD has been well studied in online convex
3

---

15 4,11). .)) This correlation between any two consecutive input data makes it difficult to analyze the growth rate of the local regret.
(ii) Second, it is time-consuming if we fully solve the lower-level problem (3) to get my because this requires multiple policy updates to solve an RL problem. Therefore, we "single-lopp" method which only requires one-step policy update for given rg. However, since the policy is only updated once, the updated policy can be far from me and thus making the analysis difficult. Single-loop methods are widely adopted to solve hierarchical problems, including bi-level optimization [28] (29].
game theory 30 BT,, min-ma min-max problems 32 etc. Recently, single-loop methods are applied to IRL [5], however, the paper [5] only solves an offline optimization problem and its analysis does not hold for non i.i.d. input data and continuous state-action space. We include section in Appendix
A.1 to discuss our distinctions from [
4 Algorithm and Theoretical Analysis
This section has three parts. The first part presents a novel online learning algorithm that solves the problem (2)-3) and tackles the aforementioned two difficulties. The second part proves that Algorithm] I aachievss sub-linear local regret. If the reward function is linear, we prove that Algorithm achieves sub-linear regret. The third part introduces a meta-learning method to get the meta-prior f.
4.1 The proposed algorithm
In practice, the expert will demonstrate specific trajectory sh ag .1 o ..... For distinction, use the capital letters (c.g., S) to represent random variables and the lower-case letters (c.g., s) to represent specific values. To design fast algorithm, single-loop algorithm. In specific, at each time t, the algorithm updates both policy π and reward parameter 9 only once. The policy update is solve the lower-level problem (3) and the reward update is to solve the upper-level problem (2). In the following, we elaborate the procedure of policy update and reward update.
..........
propose online-gres -reeeeetteeteeeeee--t------ besee---bbbtteeeddd
Algorithm 1 Meta- regelaaegul In in-trajectory Inverse Reinforcement Learning (MERIT-IRL)
Input: Initialized policy To, the streaming input data {(F), a.)) }f))22 Output: Learned reward parameter 0r and policy गт
1: Compute 9 using the reetaaregulai in Section 4.3 and Appendix
C and set 0g 0
2: for = 0, 1, T do
3: Compute the soft f-function Q (defined in Appendix B.1 under the current reward
function ræ, and policy Ti
4: Update (als) x exp(Qe. a)) for any (s,a) S
5: Roll out policy T++ starting from (sf aa) to get s4411
twice: one starting from of to get str do s di and the other
6: Compute St. = -i= 7 V'Vora (5%, at) - o d/ ) X1
1-y (0, 0) where
S/ = st' and (s a²" = Noft,af) for 0 <ist
7: Update 9.+1 0, 049t
8: end for
Policy update (lines 3-4 of Algorithm D. At each time I, we only partially solve the lower-level problem (3) via one-step soft policy iteration 5 34 In specific, the soft policy iteration contains two steps: policy evaluation and policy improvement. Policy evaluation aims to compute the soft Q-function Q (see the expression in Appendix B. under the current learned reward function re, and learned policy ne. Policy improvement aims update policy according to (1.1 a) xx exp(Quen (s, a)) for any (s, a) E S x A. In practical implementations, T++1 can be obtained by one-step policy update in soft Q-learning (34 or one-step actor update in soft actor-critic [35].
Reward update (lines 5-7 of Algorithm At each time t, the algorithm observes (sf ,af') and aims to leverage all the previously observed data to update the reward parameter. In specific, as (0: (&f af)) -y log (a
(a (
A= 0|12 is the meta-regularized negative log- likeliood

---

of (sf .a the algorithm can formulate Σi=o L,(0; (s.", a.5)) at time t using all the previously collected data (i.e., {(o,. of ) .t). To update the reward parameter, the algorithm partially minimizes Ei=o L,(0; (s... a) via one-step gradient descent.
Lemma 1. The gradient of Ei=o Li (5." a..))
can be calculated as follows:
V pooo As) - =s + 1(1- (1ggtt))
L:(0; (5t () = -E
(0-
S,A
1 1-7
E=
1=0
V Vora(s; (s, ( E.S.A
Voooo A.) = 88 A. = of
(4)
=
4=++1
Note that the gradient 4 holds for continuous state-action space. Since the gradient (4) has expecta- tion terms under the policy Ta. we can only approximate it. In specific, we roll out the policy T twice: one starting from s6 to get a trajectory [(4,, a()) where % =s0- and the other one starting from (sF .a)) to get a trajectory {(s"", af) N220 where (s"", a²)) ) = (sf a.)) for 0
S - t. Then we
use the empirical estimate 9: = Ei= 7 Voro, (8/, (4) Ei= 7 Voro, (s", a XI ++1 (0, 0)
1-y
to approximate V Ei= L, (s. af With the gradient approximation gt. we utilize stochastic online gradient descent 0.+1 = 0r arge to update the reward parameter.
special design of the reward update. The right subfigure in Figure 1 visualizes our reward update (modulo the meta-regularization term). The green trajectory (i.e., {( a ))20)) is the expert trajectory, and the red trajectories (i.e., ((8/, ((,. r20 and {(s", a"))es>)) are the trajectories generated by the learned policy. Given the expert trajectory prefix (i.e., the incomplete trajectory {(of, aa)) } observed so far), our method completes the expert trajectory by rolling out the learned policy starting from (sf.,ff)) and filling the trajectory suffix {ss ) The combined complete trajectory includes the expert trajectory prefix {( ( and the learner-filles trajectory suftix {(o",, a!") h22.. We update the reward function by comparing this combined trajectory to a complete trajectory [(s), d())22 generated by the learned policy starting from the expert's initial state off
Discussion
A more straightforward way for the reward update is to directly compare the trajectory prefixes (visualized in the middle of Figure I at each time f. However, this naive method can be problematic. We explain the issue of this naive method and the advantage of our method in the following context.
Figure I visualizes the reward up- date for standard IRL. (left), the naive method (i.e., directly run standard IRL algorithms on the expert trajec- tory prefix) (middle), and our method (right). The standard IRL. (left) up- dates the reward function by com- paring the complete expert trajec- tory and the complete trajectory gen- erated by the learned policy. This case is ideal, however, is infeasible when the trajectory is ongoing and we can only observe an incomplete expert trajectory [(s." H at each time l. The naive method (mid- dle) updates the reward function by simply comparing the expert trajectory prefix {(*f. of { and the incomplete trajectory {($] at) a J with the same length generated by the learned policy starting from the expert's 's initial state sf. This kind of reward update myopic as it does not consider for the future. Since it runs standard IRL on the trajectory prefix observed so far, it will always regard the current acate-action pair as the terminal state-action pair and thus has no ability to consider the state-action pairs in the future. In contrast, our special design gives the algorithm the ability to consider for the future. Instead of only comparing incomplete trajectory prefixes, our method compares complete trajectories just as the standard IRL. In specific, we compare the combined complete trajectory {[(sf, at) ) {(<], (("":"")) ]=x) and the complete trajectory {(s). a()) solely
(5 a5)
(56. mlo
(5f ag)
(sf ab)
(s ag)
(56.
(sf.af
(5 (ii,at)
Cff Caffeeosss
(
D (sf (ff D
(sf.af) yum (st, y
(sf jaw (.
(sf.af) Ce (st
Vara
(o of 1 J (1 (
Fe ol D Dene (5 d
(5f af) Pore Ir + +:00
(5) af))
(5f
(s af)
+ +ool
T oo
T +00
Standard IRL
Naive method
Our method
Figure 1: Standard IRL (left). the naive method for in- trajectory learning (middle), and our method (right).
generated by the learned policy. The comparison between the learner prefix {5/, df ) Jiz and expert
5

---

prefix {s," a Ji=n encourages the learned reward function to explain the expert's demonstrated be- haviors so far, and the comparison between the suffixes (fs/, d' and {s", o hise) encourages that we are learning reward function that is useful for predicting the future. Note that as / increases, the expert trajectory prefix weights more and more in the combined complete trajectory, and eventually we will recover the standard IRL reward update when goes to infinity. In Theorems [ | we theoretically guarantee that the proposed reward update can achieve sub-linear (local) regret. This shows the perfect consistency between the intuition and theory.
4.2 Theoretical analysis
To quantify the local regret of Algorithm I we have two challenges: (i) Since we only update by one step at each time i, we have t++1 instead of the optimal solution To, of the lower-level problem (3) The policy T1+1 can be far away from Te, and thus the empirical gradient estimate St can be bad approximation of the gradient (4) (ii) Since we only update once at each time f instead of finding& a near-stationary point O' such that |I Ei=o Li(0'; (sf, of ))ll S as in [26 277. the gradient norm Ei VL, (s.". (o,"))]] is not stabilized under the threshold at every time Therefore the local regret (i.e., Ei= 11 Ei (0, (sf,af))) 1)1?) hard to quantify and may not be sub-linear in T. What's 's worse, the input data is not i.i.d. but correlated, i.e., the input data (s. a at time is affected by the input data (s 1) at last step. This correlation makes it even more
difficult to quantify the local regret.
To solve the first challenge, we adopt the idea of two-timescale stochastic approximation 28 where the lower level updates in a faster timescale and the upper level updates in a slower timescale. The policy update is faster because it converges linearly under a fixed reward function (36) while the reward update is slower given that we choose O2 x (t + 1) -1/2, Intuitively, since the policy update is faster than the reward update, the reward parameter is "relatively fixed" compared to the policy. It
is expected that 1 shall stay close to Ta. and at last converges to Ty, when & increases.
To solve the second challenge, we divide our analysis into two steps: (i) We quantify the difference of the gradient norms between the current correlated state-action distribution p (..') and stationary state-action distribution for any loss function L,, i 2 O. Note that 1 (..)) is affected by P. (.-)). (ii) We quantify the local regret under the stationary distribution. The benefit of doing is that the input data is i..dd under the stationary distribution, and thus we can cast the online gradient descent method as stochastic gradient descent method and quantify its local regret. Finally, we can quantify the local regret under the current correlated distribution P (..-) by combining (i) and (ii).
We start our analysis with the definitions of stationary state distribution and stationary state-action distribution. For given policy т, the corresponding stationary state distribution is (s) 2 (1 7) Er PP (s) and the stationary state-action distribution is n²' (s, & (1 (1-17 7) E "IP? (s, a). Assumption 1. The parameterized reward function ro satisfies (ro, (s, a) ro, (s, a)) I C, - 02 and [[Vora, (s,a) - Vare, (s, a)ll < Č, 02r any (01, 02) and any (8,a) E S x A where Cr and C, are positive constants.
Assumption 2 (Ergodicity). There exist constants Css O and p E (0,1) such that for any policy x and any t 2 0, the following holds for the Markov chain induced by the policy a and the state transition function P: supso~P, drv(P? (.) (.)) S C'sp'' where Sees [P{(() 1 -P(()) is the total variation distance between the two state distributions P1 and P2. So is the initial state, and P. (.) is the state distribution induced by the policy n at time t.
&
Assumptions aare are common in RL 337 38 39 40 Assumption Zholdd for any time-homogeneous Markov chain with finite state space or any uniformly ergodic Markov chain with general state space.
Proposition 1. Suppose Assumptions
7 hold and as E (0, 1=²), we have the following relation for
any Z O and any 0,, 1 Z 0:
P1SF.AR)EEP (..) [I (S DPP - Eis E(8F AB [[] (SF,
2 7
< 80 02
p
1 q
where (S.5 .A.) ~ P. (...) means that (SFF AF) is drawn from the correlated distribution IP. (.))
and (Sf AAB) (
means that (S AP) ) is drawn from the stationary distribution 4 (.)))

---

Proposition I quantifies the gap of gradient norms between the current correlated distribution P, and the stationary distribution μ= We next quantify the local regret under the stationary distribution f= with the following lemma:
2 hold and choose at (1->)(+-1) 1/2
Lemma 2. Suppose Assumptions
it holds that:
PI(ST (((SPAA)) AA) ((.))s
Σ E
Σ
VI, (0;; , 1
==
S Dlog + D2VT+ + D3 D1)
where Di, D2, and D3 are positive constants whose expressions can be found in Appendix B.4
Lemma 2 quantifies the local regret under the stationary distribution μ²* With Proposition
1 and
Lemma 2
we can quantify the local regret under the current correlated distribution.
hold and choose at (1-))(+1)-1>
Theorem 1. Suppose Assumptions
we have that:
X
E((SF (1)PP. (.)))10
(L,(O.: AF))IP]
1=0
8CM 12 -2
< Di (1 (1-m> (1 7)
(log T +1)+ D2VT + DaVT((ogT +
Theorem T is based Proposition 1 and Lemma It shows that Algorithm T achieves sub-linear local regret. Moreover, if the reward function is linear, Algorithm I achieves sub-linear regret: Theorem Suppose the expert reward function TE and the parameterized reward to are linear, and Assumptions hold. Choose at
x(++)]( 311 we have that:
(o: APDI
E (((S ( (.)))22
-min (iss (1SS..AA)e a (1) L. .A.)). s S + D5(log 1),
where D4 and D3 are positive constants whose expressions are in Appendix B.6
4.3 Meta- Retta-Reularization
Since there is only one training trajectory and this trajectory is not complete during the learning process, we need to add a regularization term to avoid overfitting, Inspired by humans' using relevant experience to help do inference, we introduce the meta-regularizztion 110 0112 where is the hyper-parameter and the meta-prior 0 is learned from "relevant experience". In specific, we introduce set of relevant tasks {T)}==P where each task T, is an IRL problem and Py is the implicit task distribution. The tasks bTo are relevant in the sense that they share the components (S, 7, Po. P) of the MDP with our in-trajectory learning problem. However, the expert's 's reward functions of different tasks are difffernnt and are drawn from an unknown reward function distribution. For example, in the stock market case mentioned in the introduction, the experts of different tasks invest in the same stock market but may have different preferences. As standard in meta-learning we assume that the expert's reward function rgg of our in in-trajectory learning problem also drawn from the same unknown reward function distribution. Note that the reward functions of the relevant tasks (T)) are different from TE even they are drawn from the same unknown reward function distribution.
41 42 43
For each task Ty, there is a batch of trajectories and we divide this batch into two sets, i.e., D and D The training set D only has one trajectory, just as our in-trajectory learning problem, and the evaluation set Deval has abundant trajectories. Define the loss function on certain data set D A m as (0, D) A Em= Ei 7¹ log no (a. (sr). The goal of each task T₁ to learn a task-specific adaptation oi using the training set Dj. such that @, can minimize the test
7

---

loss L(Ø,, D on the evaluation set Diul The goal of meta meta- regularization is to find a meta-
8, from which such task-specific adaptations Φ3 can be adapted all tasks (T In specific, metulregulaiizztton [25] proposes a bi-level optimization problem ( The lower-level problem uses only one trajectory Djj to find the task-specific adaptation Φj such that the meta-regularized loss function [Ø,, DI)) + 2(17)
A 1 0| is minimized. The upper-level problem is to find meta-prior
0 such that the corresponding task task-specific adaptations p can minimize the expected loss function L(é,, D over the evaluation sets of all tasks {-i
A s.t. Φj arg min L(ø, Dj)) + 2(1 7) 1 01.
mjn PT~ [L(6 D)
(5)
The lower-level loss function in 5 is the offline version of our in-trajectory loss function ([ (i.e., L( Df) + 2リーーح Ø 0| = Ei= Lr(ds (sf a)))) where (sf af) E Dj. Our in inttraecctory learning problem can also be regarded as to find a task-specific adaptation. Note that the in-trajectory learning problem is online while the lower-level problem in (5) is offline because the in-trajectory problem is ongoing where we keep observing new state-action pairs. In contrast, the lower-level problem in (5) based on "experience" that has already happened. Due to the space limit, we include the algorithm and theoretical guarantees of solving the problem (5) in Appendix C
Experiments
We present three experiments show the effectiveness of MERIT-IRL. We use four baselines for comparisons. (i) IT-IRL: this method is MERIT-IRL without meta-regularizztion. (ii) Naive MERIT- IRL: this method has the meta-regulariiztioon term but uses the naive way (in the middle of Figure T to update reward. (iii) Naive IT-IRL: this method uses the naive way to update the learned reward and does not have the reuuaregulaiizztionn term. (iv) Hindsight: this method is meta reguareegua ML-IRL [5] which can access the complete expert trajectory and uses the standard IRL (visualized in the left of Figure TT with meta-regularizztton to update the learned reward. The experiment details are in Appendix D
5.1 MuJoCo experiment
In this subsection, we consider the target velocity problem for three MuloCo robots: HalfCheetah, Walker, and Hopper. The target velocity problem is widely used in meta-RL [44] and meta- (4). In specific, the robots aim to maintain target velocity in each task and the target velocity of different tasks different. To test the performance of MERIT-IRL, we use 10 test tasks whose target velocity is randomly between 1.5 and 2.0. In the test tasks, there is only one expert trajectory and the state-action pairs of this trajectory are sequentially revealed (to MERIT-IRL, IT-IRL. Naive MERIT-IRL, and Naive IT-IRL) in online fashion. The baseline Hindsight uses the complete expert trajectory to learn a reward function. The ground truth reward is designed as -Iv Uuagel (as in 44 where is the current robot velocity and Uarret is the target velocity. To learn the meta-prior 8, 50 relevant tasks whose target velocity is randomly between 0 and 3.
show the in-rrajectory learning performance where the r-axis is the time step t of the expert trajectory and the y-axis is the cumulative reward of the learned policy n. when only the first / steps of the expert trajectory are observed. The z-limit is 1, 000 because the trajectory length in MuloCo is 1, 000. Note that the baseline "Hindsight" is not in-trajectory learning since it learns from a complete expert trajectory. For comparison, we use two horizontal lines (close to each other) to show the performance of Hindsight and the expert in the figures. Figure 2a shows that MERIT-IRL achieves similar performance with the expert when only 40% of the complete expert trajectory (t 400) is observed while IT-IRL can only achieve performance close to the expert after observing more than 90% of the complete expert trajectory (t = 900). This shows the effectiveness of the regularaauaii Naive MERIT-IRL and Naive IT-IRL. fail to imitate the expert even if the complete expert trajectory is observed (t = 1. 000).. This shows the effectiveness of our special design of the reward update. The discussions on Figures 2b and 2c are in Appendix (D.2
Figures 2a
Table T shows the results after observing the complete expert trajectory. MERIT-IRL performs much better than IT-IRL, Naive MERIT-IRL, and Naive IT-IRL. MERIT-IRL achieves similar performance
8

---

Table 1: Experiment results. The mean and standard deviation are calculated from 10 test tasks.
MIRITTIRL
Naive MERIY-IRI
Naive
Hindsight
Expert
HallChheetah
-214063 1: 53.96
-380.78 2: 152.65
-538 22 1 00.51
766.27 2 104.41
-211874
21 37.23
-181.51 a 28.35
Walker
-U51.77
102.50
-891.99
-202.32
1319.25
157.92
-031
123..77
Hopper
-476,72 + 32.00
-009.8 = 03-63
-601 103 93.35
1112.0 = 74:33
-455 706.700-0 74.03
-121.74
84.30
Stock market
386.70 62.55
256.81
68.61
192.49 + 76.34
12.33 16.73
300.30
77.37
403.15 -1.
-300
-750
300
1750
-000
Chanred time (ip.1))
(a) HalfCheetah
(b) Walker
(c) Hopper
(d) Stock Market
MERIT-RL
MENITIR
Hindsight
Figure 2: In-trajectory learning performance.
with Hindsight and expert. Note that is not expected that MERIT-IRL outperforms Hindsight since Hindsight uses the complete expert trajectory to learn.
5.2 Stock market experiment
RL to train a stock trading agent has been widely studied in Al for finance 446 48 In this experiment, we use IRL to learn the reward function (i.e., investing preference) of the target investor in stock market scenario. In specific, we use the real-world data of 30 constituent stocks in Dow Jones Industrial Average from 2021-01-01 to 2022-01-01. We use benchmark called "FinRI." [8] to configure the real-world stock data into an MDP environment. The target investor (i.c., expert) has an initial asset of $1, 000 and trades stocks on every stock market opening day. The stock market opens 252 days between 2021-01-01 and 2022-01-01, and thus the trajectory length is 252. The reward function of the target investor is defined P1 - P2 where p1 is the investor's 's profit which is the money earned from trading stocks subtracting the transaction cost, and P2 models the investor's preference of whether willing to take risks. In specific, P2 is positive if the investor buys stocks whose turbulence indices are larger than a certain turbulence threshold, and zero otherwise. The value of pa depends on the type and amount of the trading stocks. The turbulence thresholds of different investors are different. The turbulence index measures the price fluctuation of a stock. If the turbulence index high, the corresponding stock has a high fluctuating price and thus is risky buy (48). Therefore, an investor unwilling to take risks has relatively low turbulence threshold. We include experiment details in Appendix D.3 To test performance, we use 10 test tasks whose turbulence thresholds are randomly between 45 and 50. To learn 0, we use 50 relevant tasks whose turbulence thresholds are randomly between 30 and 60.
Figure 2d shows that MERIT-IRL achieves similar cumulative reward with the expert at = 140 which is less than 60% of the whole trajectory, while the three in in-rrajectory baselines fail to imitate the expert before the ongoing trajectory terminates. The last row in T shows that MERIT-IRL achieves similar performance with Hindsight and the expert. More discussions the results are in Appendix D33
5.3 Learning from a shooter's ongoing trajectory
This part presents the experiment of learning from an ongoing shooter trajectory. Following 12] we model the shooter's ' movement as navigation problem. We build simulator in Gazebo (Figure Ba) where the shooter moves from the door (lower left corner) to the red target (upper right corner). The learner observes the ongoing trajectory of the shooter and keeps updating the learned reward and policy. our case, the complete shooter trajectory has the length of 140. Figures 3b 3g show our in-trajectory learning performance where the heat maps visualize the learned reward. We normalize the learned reward to [0, 1). We can observe that as the ongoing trajectory is expanding. the learned reward function becomes more and more precise to locate the goal area. When & = 40, cannot tell

---

the goal area from the heat map (Figure 3bj. However, as the time grows, we can almost locate the goal area when 60 (Figure 3cb and precisely locate the goal area when 80 (Figure 3d..
(a) Gazebo environment
(b) = 40
(c) 60
(d) / 80
Face
68
180 142
(c) 100
(f) t 120
(g) 140
(h) Success rate
Figure 3: Learning performance on the active shooting scenario.
Figure 3h shows the policy learning performance. Since there is ground truth reward in this problem, we use "success rate" to quantify the performance of the learned policy. The success rate the rate that the learned policy successfully reaches the goal. From 33 we see that MERIT-IRL outperforms the other baselines and can achieve 100% success rate when f = 80 (i.e., only observing 57% of the complete trajectory). Note that do not include Hindsight and Expert in Figure since they both achieve 100% success rate.
Conclusion
This paper proposes MERIT-IRL, the first in-trajectory inverse reinforcement learning theoretical framework that learns reward function while observing an initial portion of a trajectory and keeps updating the learned reward function when extended portions (i.e., new state-action pairs) of trajectory are observed. Experiments show that MERIT-IRL can imitate the expert from the ongoing expert trajectory before terminates.
Acknowledgements
This work is partially supported by the National Science Foundation through grants ECCS 1846706 and FCCS 2140175. We would like to thank the reviewers for their insightful and constructive suggestions.
10

---

References
[1] P. Abbeel and A. Y. Ng. "Apprenticeship learning via inverse reinforcement learning," in International Conference Machine Learning, pp. 1-8, 2004.
[2] N. D. Ratliff, A. Bagnell, and M. Zinkevich, "Maximum margin planning," in International Conference on Machine Learning, pp. 729-736, 2006.
[3] B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey, "Maximum entropy inverse reinforce- ment learning," in National Conference on Artificial intelligence, pp. 1433-1438, 2008.
[4] B. D. Ziebart, J. A. Bagnell, and A. K. Dey, "Modeling interaction via the principle of maximum causal entropy," in International Conference on Machine Leurning, pp. 1255-1262, 2010.
[5] S. Zeng, C. Li, A. Garcia, and M. Hong. "Maximun-lkkelhood inverse reinforcement learning with finite-time guarantees," in Advances in Neural Information Processing Systems, 2022.
[6] S. Liu and M. Zhu, "Distributed inverse constrained reinforcement learning for multi-agent systems," Advances in Neural Information Processing Systems, vol. 35, pp. 33444-33456, 2022.
[7] D. Ramachandran and E. Amir, "Bayesian inverse reinforcement learning.." " in International Joint Conference on Artificial Intelligence, pp. 2586-2599 2007.
[8] A. J. Chan and M. van der Schaar, "Scalable bayesian inverse reinforcement learning," in International Conference on Learning Representations, 2021.
(9] N. Rhinehart and K. M. Kitani, "First-person activity forecasting with online inverse rein- forcement learning," in IEEE International Conference on Computer Vision, pp. 3696-3705, 2017.
[10] S. Arora, P. Doshi, and B. Banerjee, "Online inverse reinforcement learning under occlusion." in International Conference on Autonomous Agents and Multiagent Systems, pp. 1170-1178, 2019.
[11] Liu and M. Zhu, "Learning agett behaviors from distributed and streaming demonstra- tions," Advances in Neural Information Processing Systems, vol. 36, 2024.
[12] A. Aghalari, N. Morshedlou, M. Marufuzzaman, and D. Carruth, "Inverse reinforcement learning to assess safety of a workplace under an active shooter incident," " IISE Transactions, vol. 53, no. 12, pp. 1337-1350, 2021.
[13] Q. Sun, X. Gong, and Y.-W. Si. "Transaction-awarr inverse reinforcement learning for trading in stock markets." Applied Intelligence, vol. 53, no. 23, pp. 1831862882000 2023.
[14] J. Chang and W. Tu, "A stock-movement aware approach for discovering investors' personalized preferences in stock markets," in International Conference on Tools with Artificial Intelligence, pp. 275-280. 2018.
[15] X. Hu, Y. Chen, Ren, and Z. Xu, "Investor preference analysis: An online optimization approach with missing information," Information Sciences, vol. 633, pp. 27-40. 2023.
[16] Y. Xu, W. Gao, and D. Hsu, "Receding horizon inverse reinforcement learning." Advances in Neural Information Processing Systems, vol. 35, pp. 27880-27892, 2022.
[17) G. Swamy, D. Wu, S. Choudhury, D. Bagnell, and S. Wu, "Inverse reinforcement learning without reinforcement learning," in International Conference on Machine Learning, pp. 33299- 33318, 2023.
[18] K. Yan, A. Schwing, and Y.X.. Wang, "A simple solution for offline imitation from observations and examples with possibly incomplete trajectories," Advances in Neural Information Processing Systems, 2024.
[19] M. Sun and X. Ma, "Adversarial imitation learning from incomplete demonstrations," in International Joint Conference Artificial Intelligence, pp. 3513-3519, 2019.
11

---

[20] D. Xu, F. Zhu, Q. Liu, and P. Zhao, "Arail: Learning to rank from incomplete demonstrations." Information Sciences, vol. 565, pp. 422 422-437, 2021.
[21] J. Ho and S. Ermon, "Generative adversarial imitation learning," " in Advances in Neural Infor- mation Processing Systems, pp. 4572-4580, 2016.
[22] G. Qiao, G. Liu, Poupart, and Z. Xu, "Multi-modal inverse constrained reinforcement learning from a mixture of demonstrations, Advances in Neural Information Processing Systems, vol. 36, 2024.
[23] S. Liu and M. Zhu, "Meta inverse constrained reinforcement learning: Convergence guarantee and generalization analysis," in International Conference on Learning Representations, 2023,
[24] G. Qiao, G. Quan, R. Qu, and G. Liu, "Modelling competitive behaviors in autonomous driving under generative world model," in European Conference on Computer Vision, 2024.
[25] A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine, "Meta- Meetanle with implicit gradients." Advances in Neural Information Processing Systems, pp. 113-124, 2019.
[26] E. Hazan, K. Singh, and C. Zhang, "Efficient regret minimization in non-convex games," in International Conference Machine Learning, pp. 1433-144, 2017.
[27] N. Hallak, P. Mertikopoulos, and V. Cevher, "Regret minimization in stochastic non-convex learning via a -ogdietl-rrddent approach," in International Conference on Machine Learning, pp. 4008-4017, 2021.
[28] M. Hong, H.-T. Wai, Z. Wang, and Z. Yang, "A two-timescale framework for bilevel optimiza- tion: Complexity analysis and application to actor-critie," arXiv preprint arXiv:2007.05170 2020.
[29] T. Chen, Y. Sun, Q. Xiao, and W. Yin, "A single-timescale method for stochastic bilevel optimization," in International Conference Artificial Intelligence and Statistics, pp. 2466- 2488, 2022.
[30] F. Schäfer and A. Anandkumar, "Competitive gradient descent," Advances in Neural Information Processing Systems, pp. 7625-7635, 2019.
[31] V. S. Varma, J. Vectaseveera, R. Postoyan, and I.-C. Morärescu, "Distributed gradient methods to reach nash equilibrium in potential games," in IEEE Conference on Decision and Control, pp. 3098-3199, 2021.
[32] S. Boyd and L. Vandenberghe, Convex Optimization. Cambridge university press, 2004.
[33] J. Zhang, P. Xiao, R. Sun, and Z. Luo, "A single-loop smoothed gradient descent-ascent algo- rithm for nonconvex-coneave min-max problems," Advances in Neurul Information Processing Systems, pp. 7377-7389, 2020.
[34] T. Haarojja, H. Tang, P. Abbeel, and S. Levine, "Reinforcement learning with deep mnergy-based policies," International Conference on Machine Learning, pp. 1352-1361, 2017.
[35] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor," International Conference on Machine Learning, pp. 1861-18777 2018.
[36] S. Cen, C. Cheng, Y. Chen, Y. Wei, and Y. Chi, "Fast global convergence of natural policy gradient methods with entropy regularization," Operations Research, vol. 70, no. 4, pp. 2563- 2578, 2022.
[37] Z. Zheng, F. Gao, L. Xue, and J. Yang. "Federated Q-learning: Linear regret speedup with low communication cost," arXiv preprint arXiv:2312.1500 150223 2023.
[38] H. Gong and M. Wang, "A duality approach for regret minimization in average-award ergodic markov decision processes," " Conference on Learning for Dynamics and Control, vol. 120, pp. 862-883, 2020.
12

---

[39] Z. Zheng. H. Zhang, and L. Xue, "Federated Qllearning with reference reference-advanttug de- composition: Almost optimal regret and logarithmic communication cost," arXiv preprint arXiv:2405. 18795, 2024.
[40] Z. Zheng, H. Zhang. and L. Xue, "Gap-dependee bounds for Q-learning using reference- advantage decomposition," arXiv preprint arXiv:2410.07574, 2024.
[41] Xu and M. Zhu, "Efficient gradient approximation method for constrained bilevel optimiza- tion," in AAAI Conference on Artificial Intelligence, vol. 37, pp. 12509-12517, 2023.
[42] S. Xu and M. Zhu, "Meta value learning for fast policy-centric optimal motion planning," in Robotics science and systems, 2022.
[43] Xu and M. Zhu, "Online constrained meta-learning; provable guarantees for generalization." Advances in Neural Information Processing Systems, vol. 36, 2024.
[44] C. Finn, P. Abbeel, and S. Levine, "Model-agnostie meta-learning for fast adaptation of deep networks," in International Conference on Machine Learning. pp. 1 1126-11355 2017.
[45] S. K. Seyed Ghasemipour, S. Gu, and R. Zemel, "Smile: Scalable meta inverse reinforcement learning through context-conditione policies," in Advances in Neural Information Processing Systems, pp. 7881-7891, 2019.
[46] Y. Deng, Bao, Y. Kong. Z. Ren, and Q. Dai. "Deep direct reinforcement learning for financial signal representation and trading," IEEE Transactions on Neural Networks and Learning Systems, vol. 28, no. 3, pp. 653-664, 2016.
[47] Z. Zhang, S. Zohren, and Roberts, "Deep reinforcement learning for trading," " The Journal of Financial Data Science, vol. 2, no. 2, pp. 40, 2020.
[48] X.-Y. Liu, H. Yang. J. Gao, and C. D. Wang. "Finrl: Deep reinforcement learning framework to automate trading quantitative finance," " in ACM International Conference on Al Finance, pp. 1-9,2021,
[49] B. D. Ziebart, A. I.. Maas, J. A. Bagnell, and A. K. Dey, "Human behavior modeling with maximum entropy inverse optimal control.," in AAAI Spring Symposium: Human Behavior Modeling, vol. 92, 2009.
[50] M. Monfort, A. Liu, and B. D. Ziebart, "Intent prediction and trajectory forecasting via predictive inverse quaara-ua regulation," AAAI Conference on Artificial Intelligence, pp. 3672- 3678, 2015.
[51] S. Gaurav and B. Ziebart, "Discriminatively learning inverse optimal control models for pre- dicting human intentions," in International Conference on Autonomous Agents and MultiAgent Systems, pp. 1368-1376, 2019.
[52] W. Krichene, M. Balandat, C. Tomlin, and A. Bayen, "The hedge algorithm on continuum," in International Conference on Machine Learning, pp. 824-832, 2015.
[53] N. Agarwal, A. Gonen, and F. Hazan, "Learning non-convex games with an optimization oracle," in Conference Learning Theory, pp. 18-29, 2019.
[54] D. Garg. S. Chakraborty, C. Cundy, J. Song, and S. Ermon, "Iq-learn: Inverse soft-q learning for imitation," Advances in Neural Information Processing Systems, vol. 34, pp. 4028 4028-4039, 2021.
[55] M.-F. Balcan, M. Khodak, and Talwalkar, "Provable guarantees for gradient-based meta- learning," in International Conference on Machine Learning, pp. 424-433, 2019.
[56] K. Xu, E. Ratner, A. Dragan, S. Levine, and C. Finn, "Learning a prior over intent via meta- inverse reinforcement learning," " in International Conference on Machine Learning, pp. 6952- 6962, 2019.
[57] T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey, "Meta-learning in neural networks: survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. pp. 5149-5169, 2021.
13

---

[58] X.-Y. Liu, Z. Xia, Rui, J. Gao, H. Yang, M. Zhu, C. Wang, Z. Wang, and J. Guo, "FinRL- Meta: Market environments and benchmarks for data-driven financial reinforcement learning." " Advances in Newral Information Processing Systems, vol. 35, pp. 1835-1849, 2022.
14

---

This appendix consists of four parts: related works, proof, meta-regularizztion algorithm and conver- gence guarantee, and Experiment details.
A Related works
Applying IRL to predict from ongoing trajectories. Papers 49
51 use standard IRL to 50
predict goals of incomplete (or ongoing) trajectories. In specific, they first learn the reward function corresponding to each potential goal candidate from complete trajectories in the training phase and then use Bayesian methods to pick the most likely goal candidate of incomplete trajectories in the testing phase. However, these works are not in-trajectory learning since they do not learn a reward function from an incomplete ongoing trajectory.
Online non-convex optimization. This paper casts the in-trajectory learning problem as an online non-convex bi-level optimization problem where at each online iteration, a new state-action pair is input. Current literature online non-convex optimization has two major categories. The first one is to use the regret originally defined in online convex optimization [52 53 However, it assumes to find a global optimal solution of non-convex optimization problem each online iteration. Therefore, the second category studies "local regret" 126 27 and uses bollow-the-leader-oe based methods to minimize the local regret. However, the whwoowtthh-eeddr-bbee non-convex optimization problem to obtain a -taa-nntai point at each online iteration, which can be computationally expensive and coneucon..,ing. If the streaming data arrives at fast speed, the computation at each online iteration may not be finished before the next data arrives. The computational burden of each online iteration can be mitigated by online gradient descent (OGI) methods where we only partially solve the non-convex optimization problem by one-step gradient descent each online iteration. While OGD widely studied in online convex optimization, is rarely studied in online non-convex optimization. [1l] uses OGD to quantify the local regret, however, its analysis only holds when the input data is identically independent distributed (i.i.d)., In contrast, the input data in problem is not i.i.d. In specific, the input data at time t (i.e., (SF Aff)
methods need solve a
affected by the input data at last step (i.e., A 1) This temporal correlation between any two consecutive input data makes it difficult to analyze the growth rate of the local regret.
Regularization and meta-learning in IRL. Moreover, the data of in in-trajectory learning is extremely lacking since there is only one demonstrated trajectory and this trajectory is not complete during the learning process. The lack of data can easily lead overfitting and a common way to alleviate this problem is to use regularizers 21 54 Inspired by humans' using relevant experience to help the inference, we introduce a novel regularization method called metu-regularizztion [25] 55). Compared to the regularizers commonly used in IRL 54), the meta-regularizer provides human- experience-like prior information which helps recover the reward function from few data. Similar to the meta-initialization method [55] 44 commonly used in IRL (56), meta-regularizatioo provides an initialization that the algorithm starts at. However, more importantly, regulaagguai also provides a regularization term to avoid overfitting.
A.1 Distinction from Maximum-likelioon inverse reinforcement learning (ML-IRLL) S
We discuss our distinctions from ML-IRL from the following three aspects: problem setting, algorithm design, and theoretical analysis.
Distinction in problem setting. We study in-trajectory IRL and formulate online optimization problem, while ML.IRI studies standard IRI. and formulates an offline optimization problem.
Distinctions in algorithm design. ML-IRL and our algorithm both update policy and reward in a single loop. However, we propose a novel reward update mechanism specially designed for the in-trajectory learning case. This special design requires to use the current learned policy to complete the expert trajectory, which gives the algorithm the ability to consider for the future. This special design of reward update is novel compared to ML-IRL.
Distinctions theoretical analysis. The analysis in our paper is substantially different from that in ML-IRL due to three facts: (1) The input data in our paper is not i.i.d., while the input data in MI-IRL i.i.d. (2) We solve an online optimization problem, while MI.III solves an offline optimization problem. (3) Our analysis holds for continuous state-action space, while the analysis of ML-IRL is
15

---

limited to finite aate-action space. We now discuss the distinctions in theoretical analysis caused by the three facts detail.
In our case, the input data is not i.i.d. but temporally correlated, i.e., the input data (st, a²) is affected by the input data (st-1, at last time step. In contrast, the input data in ML.IRLL is i.i.d. sampled from pre-collected data set. To solve this non i..d. issue of the input data, we propose novel theoretical technique that has three steps (detailed in Subsection 4.2p. Step We propose the stationary distribution nT and quantify the gradient norm difference between the real distribution P7 and this stationary distribution =x in Proposition [ Step 2: We quantify the local regret over the stationary distribution in Lemma [ The benefit of doing this is that the input data can be regarded as i.idd sampled from this stationary distribution. Step 3: We combine step and step 2, and quantify the (local) regret over the real distribution, where the data is not i.idd., in Theorem [ and Theorem 2 We can see that step and step are to solve the non i.i.d. issue of the input data, so that the corresponding theorem statements (Proposition [ Theorem [ and Theorem 2 are novel compared to ML.IRL, because ML-IRLL does not have this non i.i.d. issue. The only theorem statement relevant to 2 in step 2 where we both analyze the algorithm over stationary distribution,
ML-IRL is Lemma
and the data is i.idd sampled from the stationary distribution.
However, Lemma in step 2 still has significant distinctions from ML-IRL because Lemma quantifies the local regret in the context of online optimization, while ML-IRL quantifies convergence in the context of offline optimization. First, the objective function is dynamically changing in the online setting because the learner observes a new state-action pair at each online iteration, while the objective function is fixed in ML-IRI.. Second, the local regret contains the term L. (0,; (sE of af)), however, O, is computed before the learner knows (s of This makes it more difficult to quantify the local regret because the learner does not know L, when it computes 0, These two difficulties do not appear in the offline optimization in ML-IRL. To solve these two issues, we need to additionally construct new time-invariant function L in Appendix and and quantify the convergence of the new function L. Then, in order to quantify the local regret of Lee ] we need to quantify the difference between the real loss function (Le)e2 and the constructed loss function L.
Moreover, our theoretical analysis holds for continuous state-action space while the theoretical analysis in ML-IRL is limited to finite nate-action space. The extension to continuous state-action space brings new difficulties and requires significant novel analysis. In general, the difficulties stem from two aspects: (I) The constants in ML-IRL, e.g., the smoothness constant of the loss function L and the coefficient of convergence rate, include the term [S] x [A]. When the state-action space is continuous, those constants are not finite because |S| x |A| is now infinite. To address this issue, we propose new methods to bound those constants. For example, in order to show that the loss function L smooth, rather than using ((11)) (f2))l to find the smoothness constant as in ML-IRL, we aim to show that || V [())] || is upper bounded by constant C₁, in Lemma A.2 and this constant C₁. does not rely on (S] x |A]. Given that [|V2T. S CL, the loss function T. is Cz.-smooth. (2) Since the action space A is finite in ML-IRI,, their proved properties of the Q-function Qsoft (e.g.. Lipschitz continuity, contraction property, monotonic improvement, and smoothness) can be easily extended to the value function / by summing over different actions E A. When the action space becomes continuous, summing over infinitely many different actions does not preserve those properties. Thus we have to propose new methods to prove those propoerties of the value function ysolt In specific, we prove the Lipschitz continuity, contraction property, monotonic improvement, and smoothness of the value function srofft in Claims 3 in Appendix B.4
B Proof
This section provides the proof of all the proposition, lemmas, and theorems in the paper. To start with, we first introduce the expression of soft Q-function and soft Bellman policy.
B.1
Notions
The soft Q-function and soft value function are:
P(s' a)) (s)dds' /
Q (s, a) 2 re(s,a) a) +
Jes
16

---

Vent(s) (s)
_ P.S.A
y (ro(Se, A2) log (A.(s.)) So 00
0,
1=0
The soft Bellman policy is follows:
To(a|s) exp( (s, a))
expp(V
28th (8, a) ro(s, + / WES
P(s' VV
v80th (s) = log (f exp(Q7 (1 (s, exp(Q0"(s,a))
a))aa)
It has been proved 34 that the soft Bellman policy ny is the optimal solution of the lower-level problem (4). We define Jo(s) & ES 1 (rro(Sr. A0)S = s] as the expected cumulative
reward of policy we starting from state and Js(s,a) & L. IEr 7rr A.) = s, Ao = a).
Lemma 3. We huve the gradient Vo (a(ss) E 1= 7 (S,,,, So 8, Ao
a] S. Ei o (Vrre(S Ar)
s). s].
Proof. Define Zo(s, a) exp( Qa a)) and [s(s) (V therefore Zo is smooth in 0 given that is a composition of logarithmic, exponential, and linear functions of ro and ro is smooth in 0 (Assumption 1).
V log (s) = SOEA VoZo(s,
Zo(s)
Zo(s,a)
Vo log a)
AEA
Z*(x)
/ Ta(a|s)
Voro(s, +7 / P(dlls,) log Zo(s')dd da,
JaCA
YeS
Voro(s, a) + I P(s'(s.a) I SWEA (Voro(st,a)
Tolals)
JEEA
WES
+7 I SEES
18 a Vo log Zo("")do" da'ds' ds' da.
Keep the expansion, we can get V log Z = FE Ver A,)(SO s] and similarly
we can get Vo log Zo(s.u) = ES 1E/0 (V, Ar) = 8, Ao = a). Thus we have the gradient Vo log To (als)) = Vo log Zo Zo a) Volog (s))
B.2 Proof of Lemma 1
Recall that Eizo [2(0; (5 ,af)))))) = El > (og (s istic, we have that
When the dynamics determin-
Li(0; (3F,a a.)) ((1)==------ - Σ Log
F))
i=u
VoQuo" a5) Vo 1070f(
==0
Verrllff F
Σ
of) + Vo (5 Vo (s
+1
'orro a
(o +
7 V Viol
17

---

Voro(sf - vo Vo 17 (s
Voro(s -ESA
(Vrr = +
#=1+1
200 S,A
Vora A:)1 = s
(oor -FSSA
Σ
Vor = = A. a.
1=0
1=t+1
S.A
Vera (S, A,) = sp
1-00
where equality (a) follows from the proof Lemma 3
When the dynamics P is stochastic, we can prove that the above gradient is an unbiased estimate of
VE (F)),, 2.)60)00
(SF,
Li(0; (S AA)1)
Vo log No (A11s
1=0
E ARPE A (1)66)02
(S AA) P( (8++))
1=0
(Vooro AA9 VV VoV (5 (
AB) 2f)
"Voro(SP - VoVo0 (S
+ S.A
V Vora(S,, A.) S
t=u
Elise (F) (.)6)02
n" (oro A
"orro A.)SS. = Sp. A. A + E
"Var A:) = 500
S.A.
imt+1
0
where equality (b) follows from the fact that
F (1
Σ
3 VorooS A.)|S SF, A = A4]
S.A
1=++1
E S.A
Voro(S, A,)( = S -
1mt+1
because PTT (-) = 17 (SF, PFIS( A and SE
P(-ISP, AF).
Since we quantify the local regret expectation Theorem this unbiased estimate can be used when the dynamics is stochastic.
18

---

B.3 Proof of Proposition !
From Assumption 2, we know that drv ( (=))))(()))))))))))))))))))))( Sass -f I Cmp' where the initial state is 80- For any state-action pair (s, a) S x A, we know that I2, (8,a) a) p (s)s)(als) and (8, 1x (s)Tr(a)s) Therefore, we have that:
dIv(F (.
1 I
[P] (s)TE((a))))) - " (s)
2 JOSS DEA
1
PPT -50 (s)
2 JSES DEEA
1
(PT ((1))~p'' ~
2 SES
S Cmp'.
Claim 1. The trajectory of 02 is bounded, i.e., 110. 811 S 2cr
X for any t 2 O.
Proof.
10s+1 0|| = ||0, 049 - -l1
20
2
Voro + A(1 211-1-1)1
0r 0 a1
oTo, (s),
1-7 (0: 0)
and(1 +
)0: 0) 04
7 VoVoo,
oro, (6% (e
1-7
(1 ag\(1 - -2441/ 1- 7
1100 011 + +
Voooo, (8)
Voro, (8% a??
1=0
(6)
$ (1 a+1(1 )
Cr,Cr
1110 81 +
y 7
1
an 100 01
Cr
< (1
+
1-7'
where (a) follows triangle inequality and (b) uses the upper bound of Vere in Assumption 1. Therefore, we have the following relation:
20,
(110:
20,
|10+1 - 01
A11
A
1 1-
A
ai)
20,
20,
110, -01 S (1
)6
1100 011
7
X
(e) 2C, 1 (1 x,2 )
(d) 20,
where (c) follows the fact that 0, = 0 and (d) follows the fact that a₁ < 1z7
Recall that the loss function (sf ,a (L (sf,a!)) a) -2 "V (s .a -V + \0 8). From Lemma 3
lo sf) + Ar' 22' 10 012 and thus
we know that VoVinn (s ET 1E ('Voro A.)S of and VoQu (s of)
FS. DE I' Voro S, B Ao
a Then, /V (sf') I and
I[V (sf of S T
Now that
LL i (5," a. +.))11 = -0l1=>\1VVol 22. (s,,af) VoVa. (5.²) +(0, -0)1,
19

---

S 12 VoVa. (b + 2(0, - -011 s/ (
(1-; + +2C,)
(2
2C,
1 (1-7) (2-3)-
2 2
IIV (st a 11 & ,0
Therefore, we have that
Eis Ap) [1V (
V /
PT (s)
JAEA -f I
(s. a)) 2
JOEA
< /
- (s, 01) (s,
(0 (s, a))
JOSS JOEA
S 220MP' 402720 2 (²=2)= 80 2 - (2-2) ( p¹y23
2
Lemma 4. Suppose Assumptions 1-2 hold, the we have the following for any (s, E SXx A and any
01,02,t t: [|VL, VLr (s, a))l S Cz|10 021 and Q0 (s, a)
02.50, (s,a) S
Cqll0: 02ll, where CL 1-7
20, + (1 (1-7)+ + A and Cq l-y
403
C,
Proof. Note that Q = Qiai and V VoQT (s,a) a) = D S.A IE 7 Vorr(S., = 8, Ao = =a (proof of Lemma 3 Therefore, we have that
Cr A Co 1-7
(8, V
We know from Lemma that LO, (sf - V (of (s))
To find the smoothness constant of Lt, we need to compute the Hessian of L, First, we have that
2 2oo (s. (s) V.E
A,), Ao =],
/
Voore(s,a)+>
P(s'ls,oVVoL.)) (Sr, s'dds,,
NE8
t=
= -
+7 I p(
/ To(d'ls") ESA
? (or = AA
= =''Id''d ds',
S.A
YES
a'EA
1=
P(2l(s,0) / Voro(al'ss) E Σ
= Vagrols, a) +7
o 'v Vore(SS; 5s,
PES
A'EAL
1=0
Ag a'l + xo(als')))
E.
(oro(S,,AA)66 =s A =dd] dal ds'
S.A
t=
Keep the expansion, we can get
Vog Q (s)
E
(S, Ac)|So 80, Ao aol
+ E
VoTo(A,
E A
Voro(S, 131S% S, Ao = A.) So so. Ao =
1=0
20

---

Now we take a look at the second term in the above equality:
Voto(als) ES.
Voro(Si. = s, Au = a],
1=
= Vo
E.AA
(Sr = A, = =d],
t=0
- VoQ2 (s,a) VoV2 (s)
S
7
Voroo Ar) s, Ao a).
1=0
VoTo(als) E
V Voro( Sr., A.) = 8, Ao = al
t=0
C. + Cr
C
2C3
1-7 1-7
1 1-7 7
(1 7)3
Therefore, we have that
V Vo
27
(s,
a)1
E
(St, =so Ao = aol
1=0
+
Ext
Σ
V'V ( (A)(5)
F.S.A
7 Voro(S, AJ1So S% Ao A4]
1=0
t=0
C
20
Õ 20
Σ
1 7
(1 7)3
1-9 (1 71-
1=0
Similarly, we can get | (oof()))
1 c.
+ 20
Therefore, we have that
1- (1-9)s-
(5t a {f)] 7/ | (s. (s
E a
)|| +
| Va (s5)]1++ +2).
+1))
S
I- (1 (1- + +*)
20r +
403
s
20,
+
40
- Cl.
(6)
1 n
7)-
B.4 Proof of Lemma
This proof is based on the proof in ML.IRL (5). The differences are: (i) their proof only holds for finite state-action space while extend to continuous state-action space; (ii) their analysis is for offline settings while we extend to online settings to quantify the local regret. We first introduce the following claims which serve as building blocks this subsection.
Claim 2. For any given policy π and state-action pair (s, a), it holds that 1 (s,
Qao (s, S Coll01 0211 and [V (s) Va (s)l S Coll01 02]1.
Proof.
Q01t (s.
Qsoft (s, a),
=ES.
Σ
?' [ro, (St, Ar) - -log T (Ar[S.)]
So 8, Ao
- E
7'(ros (St, AA) log (A, St)]
So s, Ao
E
Mra, (St, A.) - ra, (S, A+)]
So = 8, Ao =
21

---

C1,110 - 021l = C
= (8) (8, s Σ
1 1-9 1-1110, 110 - 021l Coll0, 0211-
1=0
Similarly, we can get that
20
Vor (8) - Vo (6)] S D.A
IT., At) - rro (Sr, Ao
So =
Cr 110 021 -Col101 -2211-
Σ /0,110, -0211 1-7
Claim 3. The soft Bellman operator Tg
(Te Q)))))(s a) - ro(s; +7 I P(b'ls,))oog // explQ(s', a'))da do' Q)(s)
JUSS
WoEEd
(T ()s) & log LaaA exp (r((s,a) +7 P( da
s'ES
contraction map with constant 7,
Proof. It has been proved that T is a contraction map with constant y (Appendix A.2 in Here we show that T is contraction map with constant Define norm of V as V2ll = sup.es (V)(s) (x) and suppose [[V - -/2l| = e. Then we have that
34
/ P(s'ls, (s'))) )ds
T = log
exp ra(s,a) + +
da
aEA
WES
(a(s,a)+7/1 / P(s P(s'(s,a)\V2(s)) + elds' da
< log
exp
AEA
WES
exp ro(ssa) + I P(s'ls, P(55)s. + da
log
AEA
WES
/ aEA
log
exp(1e) exp ro ro(s,a)+ +7
P(slss, (22ss) )ds' da
YES
-To0 (2()) Tc.
Similarly, we can get (s)) 2 V2(s) - ye. Therefore, || V1 - Taun S 7E >/lV/ -V2l.
Claim 4. It holds that Qaat V (s, 2 To. ( (s, a) and 0e, (8) 2 To. TV. (Vorsa,) (s) for any (8, a).
Proof.
Q (s) (1)
(s, +7 f P(s'ss, a) Eor 10 (5', -log T 1 (a') s')d's
jues
(2)
2 ro (3 + SES
EA (A 10tm (8', A') - log Te+1 (A'ls')
(s, + / Sses
/ exp(Q (s', a d')a ds',
P(s'|s, a) log
d'EA
TTt0t (Qaft as,
where (i) follow equations (2)-(3) in (35] and (ii) follows policy improvement theorem (Theorem
in 34 ). Similarly, we can get that
softt
Beseets (s) EA
(s 1
log T (Als))
22

---

2 E EANE (1s))Q01r (s, A) - log Te+1 (A[s)),
log
exp(Q0rr (s,a))da
DEA / cxp
log
re (s) +7
P(s ass'ss,aVVarl a) (6')ds' da
DEA (Varit (V,05, (s))
eEES
Claim 5. The following holds for any (s, a) S and 0g 0z
Ve. (s, a) Var (s, a)l - Coll0; 02l1.
Proof. Note that V VV = E Di= Vero (S. Ar] = s) (proof of Lemma 3 therefore
C
VoV Noft (s)ll - 1-7
= Cq.
We first show the convergence of n, in Algorithm 1. For any (s, a) E A, we know that
log (alss log ne. { & 1Q (s,a) a) Q [s, + IV (s) Vor (s)].
Now we take a look at the term 1 Be (s, a) - (i, a)l:
10,r (s, a) - Q5aft( (s, a)),
$ 10mm (s,a) Qu. 1. (s, + 1 (s, Qu: (s, as) 12th (s. a) Q2, (s,a)),
Coll0; 01-1/1 + IQ 1 (s,
200t (s, a)) + CQ 10: 01-1l
2Cql 10: 81-1 + Q00 m. (8, a) Q (s, a));
@ 20ol 00 1--111 Q (s, a)
Q Q (s, a);
3 20o/10: 0-111 2 (s, a)
To (0 (s,
@ 20o| 0--1/| || + T (Q (s, a) Tooth (Qoeft (401-11 as,
(9)
< 2Co|10, - 0,-1 + +1 (s, a) Q 1. a))
(7)
where (a) follows Lemma 2 and Claim 2 (b) follows the fact that mo the optimal solution, (c)
follows Claim 4 (d) follows the fact that 4 is fixed point of T (Theorem 2 in (34),, and (e)
follows Claim
Similarly we can bound the term |rooft 0.,
(s)
//sott (s)l:
Vioott (s) 1/s0f Pe,
(s)l,
< Vonn
Ve,ne (s) - V/ Vor-1.TT (s)l + 1/ 1 (s) Va (s)l + |V (s) Vara Vo--1, (s)l
(
Col1o, -6-111 + [
1.m2 (8)
V-all (s)l Collo: 6:-1l1
(9)
2CQ 10: 0,-1 + 1/soft (s) Tagot (Vaoot ( (s),
$ 2Col - 0.-1/1 + soft (s) Vasit (s)],
where (f) follows Claim and claim 5 and (e) follows Claim
Now we take a look at the term 100 - 0 0--1/|:
00: 0-11 ---1lle-1l1
23

---

Vora (6
+ Σ
V Voro (6 at)
+ 1(1 71-1) )
0 - 0
1 1-7
1=0
1=
(1)
Cr
C. 1(1 1(1-9(11) 20r)
S a1-1
1-7
1-7
1-7
A
(8)
1-7
where (f) follows claim Recall that
log (als) - log me, (a|=)] S 10 (8, -8 (s, (s,,a]= + |V (6) - Vorn (+)]-
Summing from = 0 to t, we get
1 (s -2000 (s, a)l + Ve (s) ViJ.0t (s)
ACQ110: 11: - -0--1l1 + Q (s, (0,,) - -o 1 (8, + |
P (8) 1/sont
0:-1 (8)1 (61)
1-1
+ (1 - 7)
Σ
Quit (s, -Q (s. a)l + Di,T, (s) - V2 (s)
im
( 16
ai-1+ + (10 (s, a) - -Qoa (5, a)l + |100, (s) - -Vo0= (s)
1=1
Q8 (s, (s, a) - Qo0t (s, a)l - |V (s) Vooo((s))
Ioaen (8, a) - (s, a,) + |V (8)
Vo, s/ [s)]
C ((1 7)² i=1
Σ
a + 8(1-7) (190)(())))) -Q00 (5, a)l + |1 Voo.T0 (s) V/500t (s)l
1
(s,
12 (s,
Quaa (s, (s, - Voii (8) Vo. (s))
Dj D2
Vt t
1, (1-7)² and D. = - (1000(s) --1(1100018
where (1) follows 8. D
a) -Qaa a)) + |V (s)
Vio0 [s)] -12 (s, a)
Q (s,, a)l
Iraoft
(s) VaL (5))
Therefore, we can see that
De,m.
D Vt
D2 t
Σ 1=0
log n,- (als) log ne, (a|s) <
+
(9)
Define the loss function L(0)= 2 ()s.A)) [- log TO (A|S) + 1|1 -0]3], then we can see that E(S5,A (.) (1. (S. A )) = 7 [()). Moreover, we have that
(1)
(B A5)
<
7s 1 Đ|
00
7 B
Vero, (S,, As 5.5 ESA
Voro, (Sk. Ab) = S 1A A
k=0
(N)
Cr
2 2r'O, +
(10)
1-7¹
24

---

where (j) follows Lemma and (k) follows Claim Therefore, we have that
!
V (I, (S A,5) VL(0011 (1)
20, + 20, 2
Eis ((sF,AF)mura (F)
-
= Fis AA.) Af)
I (0,; (S, A.))) - -'((0))1
1--2") (1-x) Bisp,
(0(:: (5FFA))) I(0.)))²,
1 (2--)
(11)
7 (1 71)
where (I') follows the fact that bounded variable X E [-a, a] has bounded variance at most a².
Now take look at the term 9t 1 VL(O,):
((S.A)m)"E(.)) [Bt 1- 7-41
Vi.
1-7
t
(S, .AA))
= (
gr
1=
1-1
VI(6))
+ EE(is (((ss,af))pp (()))0
(0:; ,
1 7
V (0;; (S AB1
= (5)
100
= EESss (1
Σ
"orr (6/, a))
ESAA
Verro (S,, A.) -
+ +Fi((S (f)
Σ
/ or,, ( a!)
1=1+1
-Esh
Voro (Si, A.)|S, S A. -A7]
1=+1
AB -B A
E.S.A
V gro (Si, A,) St
Varo, (Si, A:)[So -S + S,A
Σ
Voooo A.) = SF. A. -
i=t+1
E
?" V'Vo A.)SS. = S A, = 141
S.A
1=++1
From equation (64) 5 we know that
8
20
E S.A
Voro, (St, A.)[S so] F.s.A
Voro, (S,, Ar){So sol
20,
/
Quall (8, Quan (s, a) (dads,
1- JOSS J
CC,Ca
{1Q8oft(s, (s,a) a) - Q (s,a)): }.
sup
1 ?
(s,))Sxx
25

---

where Co is the product of the area of S and the area of A. Therefore, can get that
B(S.A)=pr
9t
1 V
VL(0.)
<
Ca sup
1100. (s,
Q8 (5, a)). (12)
?
1-7 (s.))SxxA
From and 8 we know that
80; 803-ICoC,
Q0 a) Q0.t (s, S
Q8 [s,a)]+
1 1-7
| CoCr
+ ai Q8i (0) 28,
(x, a)l S a0-i17100 a
(8,0) - Q (s, +
1-7
CoCr
as 1i (s, a) - 22,00,00 a)l s
20,, Q2 (s,a)l
+
1- ?
2m0
1
(1
as Qoot (5, - Q8, (s. a)l
==0
8a CoC
S 200 (s (s, a)l at 1000 (s, -Q0, (s,a)) a))
(13)
7
and similarly we can see that
8a,CoCr
Σ 1 (s) Q8.0 (s, $ Σ
12 (s, - Q8, -0. (s,
1 1-7
==0
(1 Quan () Q (s, a)l
1-1 Sa, Ca,C
180 (5, a) Qual (s, [s)a)l= Qore a) (8, (s,
(14)
- 7
Telescoping from i= to t - 1, we get
1 1- 1- ++1
ai ((
1 7 V ((.))
9i
②
Cd
1-1
0: 1 (S, A) Q (S, A)1,
7
1=0
(m)
AC,Ca 7)2
t-1
80,
or Soof (S, A) Q20l (S, A)I orl 1 (S. A) 1 A) +
Σ
1 7
1=0
(15)
where (1) follows 12 and (m) follows 13 Similarly, we can see that
1 1- 8+1
E(s.A)=ar
9:
v(
1 -
48,C
Qiat (8, A) - Qi (S, (S, A)},
1 7 1=0
CC,Ca
8a CoC
Q (S, A) - -28fft (S, A)] 1Q (S Q80 (S, A)) +
(1 (1-
7
(16)
26

---

Now, we start to quantify the local regret:
CL
I(0(+1) 2 I(0,) + [VL(0.]] (0.41 - 81)
0:+1 - -0:113
2
(n)
2 [(0,) + a [V..) 9: (1 (1-7)²²
L(0.) + at 1- 1-
1 70t1 VL(0.))
8a
|i(00)11 + a. (i((() (g:
1-
-
(1-7)2
++1
1 1-20t1 7441
8a3020
2 L(O,) + 0i 1 y
|V(00)]12 O; - VL(00)) 119:
VL(0)II (1 (1-))²)
1 -
- [(0.) +
1 1- 1 141
22, - 7)
- 7
vI(00)] 80,0;0
|[VI(0))) - 05
11::
1 1-7
1-9
y
(1 7)2
[-1
1 1- 7 1+1
Σ
VL(0))11
+ ((155.A))/("8(:)1122
1=0
1 1- 1-
< L(00.) - I(00)= +
0i BiS ((5F 21.020
|19:
A V((0:)]]
1 1-7
1 1-9
1=0
Σ 8a2
1=0 (1-7)3
(P)
I L(O))- I(00)= + 80,Ca(2-11 7)
E(LSP (aPe,A)) (1))120
au Qulo
1 - Qi0 (S.. AA)))
(1-7)3
8C,C1
1-1 a
- a (S. AP) Q (S,F AAA)] +
Σag
(1 7)3
1=0
where (n) follows 8 (0) follows 10 (p) follows (55) Therefore, we have that
GT VI(0.)1
(17)
T
Ol g-
< ((ss
1=0
$ <L(0r) - I(Ob)+ + 8C²C2(2 -7) 7)
E [((SF A. (.))((..))))))))))))))
ao 0
(Se. .40)
1 AA)
(1-7)3
-ar120m (SF. Af) - Quoft Afi1| + (320g 8C²C2,
T-1
(3(1-7))
Σ a
i= (1 7)2
= E(is (a) Σ ||VL()))1 S DDVT + Dgv DgVT T + +1),
(18)
where D2 = [(Or)= I(Hb) + 8C3Ca(2-1) (F))
Ou QulQuoot
(SF A/)
(1-7)3
Q5at(SE - (SF:AR) -2aottttt Af)1 and D3 2(1-2) Then we can see that
32Co (1-7)
80,02 (1-7)²
X
S 22E (B((SFFAAF>)) (.))02
Σ
V. (0:; (S+, -
27

---

(1+ 1+11(1-7) (1 V
+
< 2E (1S5
(L,(0:; (5. (At) ---)))'
+ 2E [(SF (((
[VL(OJ]P)
( T-1 82 - 7)
Σ
(t + 1)(1 - -7)1 (E((SP.AF)) ( (-.))20
VI(())1
< D1 (log T + + + D2VT + D3 DsVT T + 1),
where D1 = (- The (q) follows 11I Note that to achieve this local regret rate, we actually need to take an extra expectation over the dynamics P because need to roll out to formulate gt- Here, we omit the expectation over the dynamics.
(1-
B.5
Proof of Theorem 1
We know that
[L (S.. .PP))111
S ELIIS ((
1 (L())))
IVL (SF, ,4)) - -'VI(0))1
+ (((S5,AF)~
<
(i(0))]
1=0
+ (((55,A)))
(B;; (S ,4)) - ~V
(0)
x + + 1)) +
2i 40
1 1
2 2
i=0
EqSS AP) t
[VL, (SF _A>)112
50(
1
log(t + 1)
+
1
Vt+1
vt+1
*++
(19)
where (a) follows
18 and
11
Therefore,
V (a, (S APysIP]
E t+1 t=
V (0 (SF A5)D1
[[V: AA)))
P((sFLAF))p
[+1 1=0
28

---

(((SP::))=PT (s)))oo (L: (S,, (
-(((SF (1)))/=((())))aa)).._))))) (0: (sP ..44) )1
(9)
T-1
2 (2-7)
I Do T + D2VT Do TVT +
80 02
(1- p'y2i
1=0
1=
Dologg++ + D2VT Do T TVT
Σ
C2
(2 (2-7)²
1
t+1
1 1-
1 py
t=0
82 7)2 s <(Di+ (1 - (1-))1-7))) log D2VT + D3 log TV,
where (s) follows (19 and Proposition 1 Note that to achieve this local regret rate, we actually need to take an extra expectation over the dynamics P because we need to roll out ne to formulate gt- Here, we omit the expectation over the dynamics.
B.6 Proof of Theorem 2
Suppose the expert reward function rE and the parameterized reward function ra are both linear, i.e., rE 0g and ra 0 Φ, where d:
A RR is an n-dimensional feature vector such that [lo(s, a)ll S C for any (s, a) E S A. The proof follows the similar idea with that of Theorem where follow two-step process: step (i) quantifies the regret under the stationary distribution μ= (-:)) and step (ii) quantifies the difference between the correlated distribution P and the stationary distribution _²==. We start our proof with the following claim:
Claim 6. If the parameterized reward function re is linear, the function L(0) A-strongly convex any 0.
Proof. Recall that [(0) = E(s. "), log No (A1S)] + 1 01F Define L(0; (5,A)) A
log (AIS) 211 0112, from Lemma we can see that
20
r *o, A.) So= 3.Ao A +X(0-B).
VI(0: A)) F (SS A,)
So 5
FS.A
Σ
Therefore, we have that
V2O A)),
VOE
ros Ao = V.E
to A.) So 5, Ao A + x.
S.A
1=0
i=0
Now, we take look at
VOESA
Σ
E Ar)
Ao) Su = 55 = - = Vo vofol (5,,A)
S,
1=0
I
((
/
To(a S.A
Σ
N Ai) S1 =s1,A1 aj dajdsi }
JAES
JaylIEA
-/
Mss A.) 8 -1,
P(si(S,A)
E VoTo(ai(s1)
JAES
JAEA
1=1
+ (o(a1s)))
V
Ege S.A
2 A:) Si = 81, A1 = a1
dondds.
Keep the expansion, we can see that
Va T
Σ
r'o(S, A.) So = 3, Ao A A)
1=0
29

---

Y (S, 4:) Si = Si, A, = 11 So 56= S, 5,A0 - =]..
F
V o A115)) F.g Σ
(' 44) S0 = S
V8 E.A. S,A
8
8
5/ $). (5 A.) So = S, Ao - 16
E
Vozo ( (At1S)) ESSAA
Σ
1=0
Thus we have that
E(s,A)-u
{
VOLLE
y H A.) = 5
- VoLiS.
Σ -=0
7 (0 A.)
= A,) So
Ao =1}}
S.A
S.A
1=0
(A) -E.gA))-a
ES.A
Σ
20
HSS A.)
So = S. Ao = -Abl 5 S0}
HoS, A1/s: S/: A1 11 So = 5}
VoTo (A(1S)) F
S1 = S1 = 1/ So = 35 = }}}
EES. (OTT(AA1))) ESS.A
Ao(S, A,)
(E(
ES ,'' VOTO(Ab (t) 5,A
Σ
YO A.) So = Sy Ao = Ab So So
1=0
(4) (( D A To(A6 [x - EX = $ $
= =Cov(X) > O,
(20)
where (a) follows Lemma X E (S, = 5, Ao = A] is random vector, EX A E SSAlE (S, A. S] is the expectation of X over the action distribution, and Cov(X) is the covariance matrix of the random vector X with itself, which is always positive definite because the policy Ty is always stochastic.
Therefore, we can see that
( 1 [|[[)]|||[CCo())) +AI1 _ X
where (b) follows 20
Step (i). We first quantify the regret under the stationary distribution 1= (t,-). Suppose 0* E arg min T.(0)- = arg min 1-2" T.(0)= = arg min F(s .A E [ET-1 (0; (SF, AF))..
|10+11 - 112 = 10,g - anst - 0²112 = 110 - 1- + az 19 (9r O: - 0*),
+ Eill0+1 111,
< El110, 13 + 16a C2
1 - ++
1 - x-1
20r
VL((.), 0, - 0-) 2or (g:
VL(O.), - 00), (21)
(1 7)2
1 1-7
1 7
where (c) follows 8
Since T.(0) X-strongly convex, we have that
X 1 1-1,
T((0') Z T.(O.) + (Vi(0),, 0,) +
+ L(B.) _((r) < (VL().), 0²) 2 10. 112
>
30

---

(1) 7
|10, - 1--1--1-111000-
- -110+1 8-113
- yet1 (9)
7
1 - +1
VL( 01 - 84)
2ay (1 p(+1)
1 - 7
8ay, C7
+ (1 7)(1 7(11)
2010, 0-12
+ E(SE Af) 88 (1,)) (L(0,) ((07)).
1 7 - Xa -/
FiiST A Af -001]]]
20 ge+1) F(S A/ ( -0
7
(0 - )1+1
(1 -]((111111111111 - r(t1) +
C7
1 get (18P.A8))p0TE11
1 1-7
|19.
1 g0t1
vi(0,)lI 110, - 1110:-01|
1 - y
(0) 7 - (1 - 7
Fiss A (c)
(-)1100-00111)
-0-11]
7 7
(1)1000
-01-1-00113 0-1]3]
20 1)+1)
20 (1 - g(+1
Ö(1-7) ga+1 E(sp (a5F,11))ma (()))))))))))))))))))))))))))))) 119.
1- 71t11
1 7 VI(O
(22)
(1 71-7)(1- - 11+1)
where (d) follows (2I). and (e) follows Claim which shows that the trajectory of O2 is bounded and thus there is positive constant & such that 110, 0-1 S Õ.
Telescoping (2) from = t T 1, we can see that
T-1
E(SE (F)
Σ Li AA))
Σ
(SF,
AF1)
1=0
T-1 Σ 7' ((O)) Σ r2L(OM)
= -E(SF.AP)=a
1=0
T-1 Σ T((07
Σ I(O.)
1=0
t=0
( 1 1-7- -100 A00
1-7
E(SF (FFF.A)))TB [100,
1-
20T 1 - 7²²) E(s AA -f) 1-
200
4C,C,C'
1-1
8a,CoC,
10 (S, A) Q A) Qo my (8, A) -2a ( (S, A) +
(1 71-
1-
1=0
D 32CqCaCCC
(log T + 1),
(23)
X(1 731-7)3
where (ʃ) follows
16
and D4
1- Aoa
E(s -
0-|
200
1-y 207-1(1-77] Bis ,A ( 1 (S, Q (S, A)]]-
1-1 + C.C [1Quan 10 (S, A) - Qual (S, (S,A)) A)]
(1-7)"
Step (ii). We now quantify the difference between the stationary distribution pM (t..) and the correlated distribution (-,-). Note that ||0:]| is bounded (proved in Claim [ the soft Bellman policy Ty continuous in and (s, a), and we assume that the state-action space is bounded, there is a positive constant C₂ such that log no, (a|s)]l < C₂ for any (s,a) E S x A. We start with the following relation:
Fis (ASP.AA)aNT Al) (T((0): (S8 AA))
Fiss A (F)
(S: A.))
= SES V
=
(s, -M (s. (s) || n.. (als)]ldads
S SCCoMr'''
T-1
+ EISP
Σ L. (S .AD1]
Bise (1) (v)
Σ
L. (SF .AD)
31

---

5 C2OM
(24)
1-7p 7P
Therefore, we have that
(SF,AP)>P?
Σ Lo (5 .A11
-iin List y
(S((((FF
0
S (SP
Le(Or;
.4511 (.S.sAA)))))))))))))))) - -E(SF,AF)mm AF)
LS, .431
+ BBss (F)
:0(0r(ss' AA1))
t=
+ +F
AF
(8F, (AF)
min Eiss (1sf.,1)=03 y
Σ Ls ABD)
S ES(SP
Lo (S, ,A-1)
EsSE AF 1 Σ (S .A8)) ,
T-1
+ E(SF,AF)~PE
(0 (SP ,A))
Σ (S
(S AA1)
t=
Σ L,(0"; (SPFAA)))))) .ATI)
+ BB(
L,(0"; (5F AF))
E
1=0
(5) C.CM
CNCM + -D4+ + X(1 7) (log 7 +1) 1-7p"
32CoCrCCe
1-7p
D4+ + DssllgT++1)
2C.CM, + D4, and Ds = C2C,C,0022 1-70
where (f) follows
23 and 24
D4
X(1-3)
C Meta-Regularization Algorithm and Convergence Guarantee
To solve problem (5) we use double-loop method, i.e., we first solve the lower-level problem for K iterations to get an approximate áj.K of Φ, and then use the approximate øj. to solve the upper-level problem. We do not use a single-loop method to solve (5) because we need to get the task-specific adaptation Φ for each task T,. The single-loop method only partially solves the lower-level problem by one-step gradient descent and thus the obtained parameter can be far away from Øy.
Lemma 5. The gradient of the lower-level problem in (5) is ES 2 (S,, =
$|] Ee E'vor'V o(r) + A_ 1 ( 0) where (s"",at) E D.Y.
We roll out the policy मæ from sy to get a trajectory sn ag: ... where sy s and approximate the X (cb 0). We lower-level gradient by JS = Ei y' V (s @b Ei '' (or((s (s af)) + 1-
Ver
update 0j,++1 = Φ - Be for K times to get Φj where Bg is the step size and Φb is the parameter at time k, and use $j.K to calculate the approximate of the hyper-gradiens d L(oz Di
Lemma 6. The aepar-gradientoo (i.e., gradient of the upper-level problem in ()) is d L Djval) I++ x V2 L(g.. " VoL(os, Drval) ), where
IDinll
V VoL(O,, = =|Dgval
/ To (S,, A:) So050/a0 ~ Po
Σ
TT, (s),
S.A
1=
120 D.y) = E
7 222 20 To (St, Ar) So = s/
Σ
+e.
32

---

The [Drrll is the number of trajectories in Dy (s) a,) E Dj and e is an extra term whose expression can be found Appendix C.2
To approximate the hyper-gradient, first roll out Toy to get two trajectories 6o S... 00 d5
and
80 300 K where 80 d is drawn from Po and $0" =. sy. We estimate V VoL(Øj, Doval) via Ippral
V VoLLos, K, Di = [Dirall Et 7 a S.N a a.s.
2m Ei=0 V VoTon (5) ae)
and estimate the term 2 (o,, Dj via og (0),K, D - Vo 0o øy (5f ar x. Ei= 2'V To (sf a."). Therefore, via h = + dΦ [. L Dr We omit the extra term e in the approxi- mate h, and its impact on the convergence can be bounded (proved in Appendix C.4).
V2
approximate the hyper-gradient term [ DD
To solve the upper-level problem in (5), at each iteration n, we sample batch of B tasks, and compute the specifc adaptation Øj.K and the hyper- apper-gradient// hy for each task. The update law to solve the upper-level problem is: 0n+1 On B Σ3 hy where is the step size. Note that the time index à for the lower-level problem and n is for the upper-level problem.
Algorithm 2 Meta- reta-regulatizzatoon
Input: Initialized meta-prion d and task-specific adaptation 0y.0 Output: Learned prior 0N
for = 0, 1.... . 1 do
2: Sample batch of B tasks (Ti) PT
3: for each task T, do
4: for 1 = 1, 1 do
5:
Compute the soft Bellman policy Tdya via soft Q-learning or soft actor-critic Roll out the policy Tes.k to get trajectory so Ly.k
6:
7:
Compute the gradient I0,,, Ei V (5 a - V do ag)
+ +1-, (0j,k Bn)
8:
Update 0J 03,a BKBes. end for
9:
10:
Compute the soft Bellman policy To,.x via soft Q-learning or soft actor-critic
ag 12: Compute the dyper-gradiem, hy = + 1-2 22 Lo,, VoL Deval)
Roll out the policy TØy. get two trajectories so dy.x
0.00 d5.K
and
$y di
end for
14:
Update 0n+1 = 0, I Σド=1 h2
15:
end for
Lemma 7 (Convergence of the lower-level problem). Suppose Assumptions 1-2 hold and À 2 Ct where 1) (0, S.). Let Ba 1m3, then we have
+1
2
Billos, 1511
Assumption 3. The parameterized reward function T9 has bounded thirl-order gradient, iee.
|| V a)ll < č, for any (s, a) where Č, is positive constant.
Theorem (Convergence of the upper-level problem). Suppose Assumption 3 and the condition in Lemma 7 hold. Let Tn (n 1) -1/2 and define F F(0) as Ej-Py [L D Then we have the following convergence:
Drval under the meta-prior
N-1
Σ
log N 1 +
Ct
N
N
and the expression of C1 can be found in 27
33

---

C.1
Proof of Lemma 5
The proof is similar to that of Lemma | We first prove the case of deterministic dynamics and the corresponding result can be an unbiased estimate in cases of stochastic dynamics (proved in Subsection B.2
VoL(O,D))
V v log (a))))))
t=
8
V Q (5 V (3
1=0
06
/ Vorg(o), a.) +
V
Vo
V Voon
(s8)
V Varo($H
20
(a)
(S A.) So = o
- 7 V
-
1=0
where (a) follows the proof of Lemma
C.2 Proof of Lemma
Since the lower-level problem of 5 is unconstrained and Φ; is the optimal solution, we know that
Vol DJ) +
bb 0 = 0
(25)
We further take derivative of both sides in 25 with respect to and we get that
V2 Loj, DJ)
X
1 2 Vaol(on,
0
Vodg
I
1
Therefore, have that
1 1 1-21-22 Vz 20 L(bs, DD),
d0 L(b,, Dy = (Vids) V L(j Doval)
Voblo,,
=
Similar to Lemma
we can know that
ID
VoLloj, D)
V oT (S,, A1)
58)
rn af),
t==
1=
(26)
where (si, E n" and Ç" E Doyal
Since there usually abundant data in Dr and S. Po. we can reformulate (26) as the following:
VoL(o,, Dyyal) |D EE,
V O Vorr (s)
So Po
Voro (s),
Claim 7. The second-order information v2 2%0 (s,a) = A(s,a) + E Ei o' Cor((s))ss Ap a] and 22
Φ
2Φ (- (s) = ((s) + S.A EEe (Cor(S))) = s) where A(s,a)
1E 2'22 T0 A0)S A, a). (ss) D VV ( Ar)SS = s).
and Cov(s) SaEA [ ((s) da is the covariance matrix of A -) at state
34

---

The proof of Claim Claim(7ollow follows the proof of Lemma
Now take look at the term V2 [1bj. D2) and consider the case of deterministic dynamics:
pd Lob, DJ)
V22 op To,
Toy (a"1
V
bp 05 ( a.) - V 60 u (s!5)
1=0
20
?1
2
Ap og (5 a.)) + + p by
v (5
V2 oo
129 (8))
1=0
V
Vama (5))
V
po (5)
(a)
S.A
2 Po (S., Ar)|So = 5h] - Σ
7
2
To (5h
t=
t=
+ ES
(oo((S)) = =s]l.
1m0
where (a) follows Claim 7 As proved in Subsection B.2 we can still use this expression in the case st)) However, e is intractable to compute because it requires to compute CS,(S)) at every visited state. To approximate the covariance matrix CS)(S.). we need to empirically roll out the policy Te, from St for enough times to get enough samples. We need to do these policy roll-outs at every state St. For example, suppose we roll out the policy T6, ten times to get ten samples at each state S.. Empirically, we need to do these roll-outs 10x x T where 7 is a very large integer that we regard as infinity because the trajectory horizon is infinite. This is intractable because we need to roll out the policy for too many times. Moreover, we cannot guarantee that we can approximate COV(S,) well given that we only use ten samples to approximate it.
of stochastic dynamics. We define
F.g.A Ei= 2Cov
C.3 Proof of Lemma 7
We know that 2/ 20 [(o, DJ) + AA I|V da [(0, DD)1 E (IV2I,(0)]++>))
C..C.. where (a) follows ( Therefore, the lower-level objective function in (8) is -moohh. Moreover, since X 2 2 + n7. then Lo D1)) S Cu-2y. Therefore, the lower-level objective 2g
2(1-))*
function in (8) is I-y -tronnly convex. Following the standard result for strongly-convex and smooth stochastic optimization, we can reach the result in Lemma 7
C.4 Proof of Theorem 3
In this proof, we first bound the and then prove the convergence. Define h, & [1 + V2 ())) V (0,5 Dr where
approximation error (i.e., d [0,, Dy h2ID
e ESS. (()
1E 2V (S,, Ar)SS = xg] E 7" oe To (5) a.)) Therefore, V2
we have that
11g
[0b,, DDyal)) |||
d0
de [0),, ((()),,))) D. [T +
1 F2 ed
L(bj, D2)11
|| [(b)g ( Do
A
V 2n L(0s, D.)
1
1| ( )1 v2 80 [ 139
[ +
+
[ +
A
x
21
2/D C
21 21 - -C, +
4 C₁ > 0,
(27)
x 2
1-7
35

---

where (a) follows the fact that || +
V
X de Lo, 2
11 L Dj)1 2 1
Cr-2y given that V 00 L6, Dj)l < 2(1-7)
Cr-2n (proved in the proof of Lemma 7.. Therefore ||| +
2A
L(o,, D2 - 2 Similarly, we can bound ||[I + 1 V [0j, Dj)I '11 =
21
X de
2- given that Log. DIDl S
Note that A > CL and Cr. > C, (proved in ( I-
I
therefore C1 is a positive constant.
Now, we bound the term 11g - hgll- We define As, = I+ -
00 L0j, Dj) and Aes =
I + - V 2 (0LK0, Thus we have || = x 20, (follows (27 and similarly || |1 S
A 20 Therefore,
Ellib- - hulll = E 2 ( o, V VoL( Disa))
E | x V VoL( Diyall A م V ss (6 DISIA
<
A Dol V VeLl VoL( Di - م V VoLlog. DiSa))
E 11 A
|| V VoL(O), K Dg - V Volleg,
(F Di || + 114 0..
- -للمو -5Vo-11Voo(0 Dy
(0)
Cl 201 11, xاوه + E 0iN
20,
A -22 |Dggut|
2(1 - 7)
ال ||| 1elll-1p3 IDy
1-
20,
s <0( K x) ) + = x,,xx1 || |لر |1 - -,,111 ID.yall
1-7'
50 1 R
+ )
1 CLEE111 25.x 11 114 || ||| -D.y
"1 110 - -5 S x
20,
1
(28)
where 1I $00 [0d, D1Il S Čt. The expression of Ö2 can be derived following the proof
of Lemma 4 by notifying Assumption The (b) holds because (i) 11V 2 L Di
(D
dd ((6, DI)II S (Di
C-22 and (ii) (lV, S |D 2(17))
2c r
(see the ex-
pression of V ((bj Diyal) in Lemma 5).
d Elllh d0 [0,, Dy ||| < Elih, - h + [1,
S Ci c t )
(29)
d0
where (c) follows
Define F(0) as Ej ((0), Dr under the meta-prior 0. Note that
||bs]| $ 114 x ||- di L( : Deyal))l ||| S x-20; ID
20, Therefore,
1-7
F(O.+1) 2 F(θ.) + (VF(O.)) (((0))))(((+1+ (0n+1 - 0m) |Deval| (C2 - -211) 4(1 - 7)
10 - -0,112
[Deeal (C - 21))7² 1
2 F(O.)) B
(FF( h2
h
4(1 71)
B
2 F(0.)) + Tn
B
(VF(0.)) hg
[Deval| (Cz, - 21))
X
|iDgyal
20,
B
77) 7 2C,
1 1-
j=1
B
DDval P(C - 2n)
2((w)++ +
Tn
(VF(O.)) hy
B
4(1 - 2) -2Ćr)
j=1
+
E(F((n+1)] >
+
Tn
B Σ
d
((or,
B
db
j=1
+
Tz
E
d
-
Deval 2(Cr, - 21)
B
4(1 - 22 - -2O,)
36

---

(1 + 01 2AC, DDeal (Derala -21))
Th
+
4(1 7-) 2(2,
NN FF
Σ In Bn
< [((x)) F(06))
1 N
N-1 Σ
Tu (1 + 0 マ +
N
2xC,
(Devl Deeal
(2 -21)
N
4(1 7)² -2C,)
log
E B[|VF(O0)11 < 0
+ Ci2
VN
K
0=0
where |Deval| A sup {IDgral } and (d) follows
29
D Experiment details
The code was running on laptop whose processor is AMD Ryzen 7 4700U with Radeon Graphics, 2.00GHz, and the installed RAM is 20.0GB. The operating system is Ubuntu 18.04. We use a neural network to parameterize the learned reward function. The neural network has two hidden layers where each hidden layer has 64 neurons. The activation functions are respectively ReLU and Tanh.
D.1 Baselines
Here we provide the update rule for each baseline. Given a learned reward function rgg the policy updates of the four baselines are the same with that of MERIT-IRL, i.e., one-step policy iteration. The diffcrence is the reward update.
IT-IRL: IT-IRL is MERIT-IRL without the meta-reguuariizatinn term. Therefore, the reward update of IT-IRL is 81+1 = 0₁ a.g' where gf = Σis 7 Vero, (s). Ei= > Vore,, (s", af"). Recall from Subsection 4.1 that {(s,, a() b220 is generated by the learned policy no, starting from the initial state 5/ = E and {s" a.") J:20 is generated by the learned policy тe, starting from (s"', a()) where (ol', a²) (s a.) for S Sist t.
Naive MERIT-IRL: This method has the meta-regularizztion term, however, it uses the naive way (depicted in the middle of Figure [ to update the reward function. In specific, it only compares the partial expert trajectory {s," a,"
" Ji=o and partial learner trajectory {s Mi Therefore, the
reward update of Naive MERIT-IRL is 0.41 Oe a.gl where g o' (5), d()
Ei V' Voro, (s aF) + A(1-3(+1) (0-0). 0).
1-7
Naive IT-IRL: This method does not have the meta-regularizztion term and uses the naive way to update the reward function. Therefore, the reward update of Naive IT-IRI. 0.+1 = 0g a.g.
where g." = -i 7 Vero, (s', a²)) - > "oro,, a.
Hindsight: This method is standard IRL method with the regulaagglaiizzttoo term where the complete expert trajectory {oft a, J:20 and the complete learner trajectory (5/; a/}220 are compared to update the reward function. Therefore, the reward update of Hindsight is 0++1 = 0r orgi an where
g!" - Vero, (5/ d)) - -
(o af)) A(1-3(+))
(0 0).
1-7
D.2
MuJoCo
D.2.1
Walker
Figure [b] that MERIT can achieve similar performance with the expert after 600 while the other three in-trajectory learning baselines fail to imitate the expert before the ongoing trajectory terminates. Note that the naive methods (i.e., Naive MERIT-IRL and Naive IT-IRL) have much smaller improvement from / = 0 compared to MERIT-IRI and IT-IRL. The reason is that the naive reward update method flawed. Intuitively, the reward update mechanism of these two baselines are myopic as explained in Subsection 4.1 Theoretically, the gradients gi of Naive MERIT-IRI and g!"
37

---

of Naive ITT-RRL are biased estimate of (4) even if Ny approaches TØ, since suffix t) terms while ge and g only include the trajectory prefix
4 includes the trajectory
terms.
MERIT-IRL performs much better than IT-IRL. The reason is that the meta-regularivztionn term restricts the learned reward parameter within a certain neighborhood of the meta-prior 0 (proved in Appendix B.3). Given that 0 is trained over family of relevant tasks, it is expected that the actual reward function parameter of our task shall be "close" 0 [25 44 57], i.e., inside this neighborhood. Therefore, MERIT-IRL can efficiently learn the expert's 's reward function. On the contrary, IT--RL does not have the meta-prior 0 guidance and thus has to search over the whole parameter space, which is extremely difficult to learn the expert's ' reward function when the data is lacking. Note that MERIT-IRL and Naive MERIT-IRL. have better initial performance than IT-IRL and Naive IT-IRL since MERIT-IRL and Naive MERIT-IRL starts at the meta-prior 0 while IT-IRL and Naive IT-IRL initializes randomly.
D.2.2 Hopper
Figure 2c] shows that MERIT can achieve similar performance with the expert after 500 while the other three in-trajectory learning baselines fail to imitate the expert before the ongoing trajectory terminates.
D.3 Stock Market
We use the real-world data of 30 constitute stocks in Dow Jones Industrial Average from 2021-01-01 to 2022-01-01. The 30 stocks are respectively: 'AXP', 'AMGN', 'AAPL', 'BA', 'CAT'. 'CSCO', 'CVX', 'GS', 'HD', 'HON', 'IMM', 'INTC', 'JNJ' 'KO', 'JPM', 'MCD', 'MMM', 'MRK', 'MSFT. 'NKE", 'PG', "TRV", "UNH', 'CRM', 'VZ', -V', 'WBA', "WMT", 'DIS', 'DOW'.
The state of the stock market MDP is the perception of the stock market, including the open/close price of each stock, the current asset, and some technical indices (48). The action has the same dimension as the number of stocks where each dimension represents the amount of buying/selling the corresponding stock. The detailed formulation of the MDP can be found in FinRL 48 58
The turbulence index is technical index of stock market and is included as dimension of the state [48] (8). The function P2 is defined as the amount of buying the stocks whose turbulence index is larger than the turbulence threshold. Therefore, the more the target investor buys the stocks whose turbulence index is larger than the turbulence threshold, the larger P2 will be and thus the smaller reward the target investor will receive.
Discussion on the experiment results. Figure 2d] MERIT-IRL can achieve the similar cumulative reward with the expert when only the first 60% of the trajectory is observed while IT-IRL can achieve performance close to the expert after & = 220. This shows that the meta-regularizatio can help imitate the expert faster. In contrast, Naive MERIT-IRL and Naive IT-IRL barely improves because the naive reward update method is flawed. Intuitively, the reward update mechanism of these two baselines are myopic as explained in Subsection 4.1 Theoretically. the gradients of of Naive MERIT- IRL and g." of Naive IT-IRL are biased estimate of (4p even if ग. approaches Te, since (4) includes the trajectory suffix (i f) terms while 9 and yt only include the trajectory prefix (i {)) terms.
The last row in Table 1 shows the final results of the algorithms. We can see that MERIT-IRL achieves much better performance than the other in-trajectory learning baselines (i.e., IT-IRL, Naive MERIT-IRL, and Naive IT-IRL). MERIT-IRL achieves comparable performance with Hindsight and the expert. Note that it is not expected that MERIT-IRL outperforms Hindsight since Hindsight has the complete expert trajectory to learn.
E Potential negative societal impact
Since MERIT-IRL can infer the reward function of the expert, potential negative societal impact may occur when the learner malicious. Take the stock market experiment as an example, private information like preferences or habits of the investors may be leaked by using MERIT-IRI. To avoid this situation, the investors needs to take additional strategies such as protecting its investment data from unsecure resources.
38

---

Limitations
From the objective stration, i.e., finding reward function such that its corresponding policy makes the expert trajectory most likely. An ideal case that we can also directly quantify the reward learning performance and study the reward identifiability issue. Thus, a future work is to study the reward identifiability issue the context of in-trajectory IRL.
can see that the goal of MERIT-IRL to align with the expert demon-
39

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract summarizes our contributions, and the introduction has a "con- tribution statement" part which elaborates our contributions and mentions the consistency with theoretical and experiment results. The assumptions and limitations are included in "theoretical analysis" (Subsection 4.2) and Appendix
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitation is mentioned Appendix F Under the assumptions, i.e., Assumptions and 2 we have justification of either the assumption is widely used in [
literature or the assumption can be satisfied by real scenarios.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
40

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We clearly state our assumptions in Assumptions and 2 The theoretical state- ments are also clearly stated Subsection 4.2 and the correct proof is included in Appendix
B
All the assumptions and theoretical statements are numbered and cross-referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results,
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: This paper provides pseudocode and elaborates each step of the algorithm in Subsection 4.1 and Appendix 4.3 We also include the experiment details in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibilityy can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
41

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code is submitted in the supplementary materials and we include document in the code folder to describe how to run the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
guidddes/ggude////eeeeeoes CodeSubmiee ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Appendix Dinnluddes the experiment details, and we also submit the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In the experiment results (i.e., Table Table ??, Table ??). we include both the mean and standard deviation. Figure we also plot the mean and standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
42

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include the type of operating system, CPU, and RAM used in the first paragraph in Appendix D
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///nrripsp.ccuuule/eeiiiiie
Answer: [Yes]
Justification:
This paper follows NeurIPS Code of Ethnics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:[Yes)
Justification:
We discuss the potential negative societal impact in Appendix E
Guidelines:
The answer NA means that there is no societal impact of the work performed.
43

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks because we do not have data nor model to relcase.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
We use the SAC code from Github repository, and we clearly state it at the top of that code script.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
44

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The submitted code can be considered as an asset, and we provide a file along with the code document the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
45

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
46


Reviews:
[{"review_id": "gEQT3UKXcq", "reviewer": "Reviewer_MWk1", "summary": "The authors consider a new problem setting, called in-trajectory IRL, where a reward function and a corresponding policy need to be learned from an ongoing trajectory. The authors propose a novel reward update mechanism specially designed for this scenario and incorporate a meta-regularization strategy to embed prior knowledge and avoid overfitting. Theoretical analysis is provided for the convergence of the algorithm towards stationary points (and global under linear reward parametrization). MuJoCo benchmark and a real-world stock market example are used for experiments.", "questions": "How the authors collect data for the meta-regularization in the experiment?\n\nOther questions please refer to the Weaknesses.", "limitations": "The authors have discussed the limitations in Appendix F.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe topic is novel to IRL community. The new problem setting of learning the reward function and policy on the fly aligns with many real-world scenarios. In this new situation, standard IRL algorithms are not suitable anymore.\n2.\tThe special reward update design of predicting the future trajectory is reasonable and interesting for this ongoing trajectory setting.\n3.\tThe meta-regularization is novel to IRL, which helps to better find a reward from limited data.\n4.\tThe theoretical analysis is solid and tackles the issue of non i.i.d. input data. The theoretical distinction from literature is well clarified. The experiment includes real-world stock market data.", "weaknesses": "1.\tAssumption 1 assumes that the parameterized reward is smooth, which can be a strong assumption, especially for neural networks with non-smooth activation functions such as ReLU.\n2.\tThe meta-regularization part seems to be important to the performance. However, the meta-learning needs to pre-collect data for many related tasks, which can be difficult in real world.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive review. We believe that our discussion will lead to a stronger paper. We address your comments below:\n\n**Weakness 1**: Assumption 1 assumes that the parameterized reward is smooth, which can be a strong assumption, especially for neural networks with non-smooth activation functions such as ReLU.  \n**Answer**: Thanks for mentioning Assumption 1. We agree that this assumption could be strong when the neural networks use ReLU activation. However, the smoothness assumption can be satisfied by using some smooth activation functions such as tanh and using linearly parameterized models. In our experiment, we use tanh as the activation function. Moreover, as mentioned in the paragraph under Assumption 1 (i.e., line 255), this assumption is widely adopted in RL [A1] and IRL [A2,A3]. It is interesting to relax this assumption, and we will explore it in the future.\n\n**Weakness 2**: The meta-regularization part seems to be important to the performance. However, the meta-learning needs to pre-collect data for many related tasks, which can be difficult in real world.  \n**Answer**: Meta-learning requires to collect data for related tasks before training, however, data collection is not difficult in many real-world problems. In fact, meta-learning has been applied to many real-world problems, such as velocity modelling for self-driving where the data of related tasks (i.e., velocity trajectories in different roads) can be obtained from public data sets [A4], and object grasping of robotic arms where the data of related tasks (i.e., demonstrations of grasping objects from different locations) is generated by solving RL problems to grasp the objects [A5].  \nIn Section 5.2, a real-world stock market example is used to evaluate our algorithm. Data collection is not difficult for this application. In particular, the demonstration data of related tasks is the investment trajectories of other investors with different preferences of taking risks. We can collect such information from public records [A6]. Moreover, our method can also be applied to [A4,A5]. Take [A5] as an example, we can first use their data sets to learn the meta-prior. Then, given a new demonstration trajectory of grasping an object from a new location, we can sequantially reveal this trajectory to the learner in an ongoing fashion.\n\n**Question 1**: How the authors collect data for the meta-regularization in the experiment?  \n**Answer**: For the MuJoCo experiment, the different training tasks have different target velocity from 0 to 3 (line 328) and the  reward function is $-|v-v_{\\text{target}}|$ where $v$ is the current velocity and $v_{\\text{target}}$ is the target velocity (line 326). Therefore, for each training task, we randomly sample the target velocity from 0 to 3, design the reward function correspondingly, and use the reward function to train an RL policy to generate demonstration trajectories. For the stock market environment, different tasks have different preferences of taking risks, which is captured by different turbulence threshold (line 362). The reward function is $p_1-p_2$ where $p_1$ quantifies the profit and $p_2$ quantifies the amount of trading stocks that are above the turbulence threshold. Therefore, for each training task, we randomly sample turbulence threshold from 30 to 60 (line 365), design the corresponding $p_2$, and train an RL agent under the current $p_1-p_2$ to generate investment demonstrations. For the active shooting experiment in Appendix D.5, different training tasks have different goal locations. For each training task, we randomly sample an $1\\times 1$ goal area from the state space and train an RL algorithm to reach the goal to generate demonstrations.\n\n[A1] Kaiqing Zhang ,Alec Koppel, Hao Zhu, and Tamer Basar. \"Global convergence of policy gradient methods to (almost) locally optimal policies.\" SIAM Journal on Control and Optimization, 2020.\n\n[A2] Siliang Zeng, Chenliang Li, Alfredo Garcia, and Mingyi Hong, “Maximum-likelihood inverse reinforcement learning with finite-time guarantees,” in Advances in Neural Information Processing Systems, 2022.\n\n[A3] Ziwei Guan, Tengyu Xu, and Yingbin Liang. \"When will generative adversarial imitation learning algorithms attain global convergence.\" In International Conference on Artificial Intelligence and Statistics, 2021.\n\n[A4] Bo Yu, Xiangyu Feng, You Kong, Yuren Chen, Zeyang Cheng, and Shan Bao. \"Using meta-learning to establish a highly transferable driving speed prediction model from the visual road environment.\" Engineering Applications of Artificial Intelligence, 2024.\n\n[A5] Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. \"One-shot visual imitation learning via meta-learning.\" In Conference on robot learning, 2017.\n\n[A6] Xiaoyang Liu et al. \"FinRL-Meta: Market environments and benchmarks for data-driven financial reinforcement learning.\" Advances in Neural Information Processing Systems, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "50GtwLi7Ya", "reviewer": "Reviewer_cmJ6", "summary": "The paper \"Learn Incrementally from An Ongoing Trajectory: A Provable In-Trajectory Inverse Reinforcement Learning Framework\" proposes an innovative approach to inverse reinforcement learning. The authors introduce an online learning algorithm to address the IRL problem with incomplete expert demonstrations. A thorough theoretical analysis guarantees convergence, and a comprehensive experimental section validates the approach.", "questions": "1. Can the author explain why comparing the suffixes \\(\\{s_i', a_i'\\}\\) and \\(\\{s_i'', a_i''\\}\\) encourages the reward function to predict the future? Since you are completing the incomplete trajectory using the same policy, can reward function can learn something even if both are generated by the same policy.\n\n\n2. Just wondering how you complete the incomplete trajectories in MuJoCo using saved states, since the state in MuJoCo only includes partial information about the simulator.", "limitations": "The authors address the limitations of this work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.Very interesting problem and strong theoretical analysis.\n\n\n2.The paper introduces a practical algorithm that appears easy to reproduce and yields good results.", "weaknesses": "1.In the experiments, I don’t understand why different algorithms have different initial points? When t=0, are you using a same random reward function or the prior-meta reward function to train your policy?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful reviews. We believe that our discussion will lead a better paper. We address your comments below:\n\n**Weakness 1**: In the experiments, I don’t understand why different algorithms have different initial points? When t=0, are you using a same random reward function or the prior-meta reward function to train your policy?  \n**Answer**: Thanks for mentioning the different initial points. There are two different initial points in the experiment. One initial point is the meta-prior reward and the other initial point is random reward. There are four algorithms where the algorithms with meta-regularization (i.e., MERIT-IRL and naive MERIT-IRL) use the same meta-prior reward function as the initialization and the algorithms without meta-regularization (i.e., IT-IRL and naive IT-IRL) use the same random reward function as the initialization. The different initial points show the benefit of meta-regularization where the algorithms with meta-regularization have higher initial cumulative reward than the algorithms without meta-regularization. \n\n**Question 1**: Can the author explain why comparing the suffixes $\\\\{s_i', a_i'\\\\}$ and $\\\\{s_i'', a_i''\\\\}$ encourages the reward function to predict the future? Since you are completing the incomplete trajectory using the same policy, can reward function learn something even if both are generated by the same policy.  \n**Answer**: The suffices $\\\\{s\\_i',a\\_i'\\\\}\\_{i\\geq t}$ and $\\\\{s\\_i'',a\\_i''\\\\}\\_{i\\geq t}$ are both generated by the learned policy but they start from different initial state-action pairs where $\\\\{s\\_i'',a\\_i''\\\\}\\_{i\\geq t}$ starts from the expert state-action pair $(s_t^E,a_t^E)$ and $\\\\{s\\_i',a\\_i'\\\\}\\_{i\\geq t}$ starts from the learner state-action pair $(s_t',a_t')$. Consider the standard IRL where we have a complete expert trajectory $\\\\{s\\_i^E,a\\_i^E\\\\}\\_{i\\geq 0}$ to guide the learned trajectory $\\\\{s\\_i',a\\_i'\\\\}\\_{i\\geq 0}$. The best option to predict the future is to use the expert future (i.e., expert suffix $\\\\{s\\_i^E,a\\_i^E\\\\}\\_{i\\geq t}$) to guide the learner suffix $\\\\{s\\_i',a\\_i'\\\\}\\_{i\\geq t}$. However, in our setting, we do not have the future expert trajectory $\\\\{s\\_i^E,a\\_i^E\\\\}\\_{i\\geq t}$ so that we use the suffix $\\\\{s\\_i'',a\\_i''\\\\}\\_{i\\geq t}$ as an approximation of the future expert trajectory $\\\\{s\\_i^E,a\\_i^E\\\\}\\_{i\\geq t}$ to guide the learner suffix $\\\\{s\\_i',a\\_i'\\\\}\\_{i\\geq t}$. If the approximation suffix $\\\\{s\\_i'',a\\_i''\\\\}\\_{i\\geq t}$ is a good approximation of the expert suffix $\\\\{s\\_i^E,a\\_i^E\\\\}\\_{i\\geq t}$, the learned reward can be accurate for the entire horizon.  \n\nNow the question left is whether the approximation suffix $\\\\{s\\_i'',a\\_i''\\\\}\\_{i\\geq t}$ is a good approximation of $\\\\{s\\_i^E,a\\_i^E\\\\}\\_{i\\geq t}$. The answer is that the approximation error between $\\\\{s\\_i'',a\\_i''\\\\}\\_{i\\geq t}$ and $\\\\{s\\_i^E,a\\_i^E\\\\}\\_{i\\geq t}$ diminishes when $t$ increases. This is mathematically justified by the sub-linear regret in Theorems 1 and 2 because the regret will be at least linear if the approximation error is always above a certain constant. The intuition behind this is that when $t$ increases, we observe more portion of the expert trajectory and thus the learned reward $r_{\\theta_t}$ improves. A better $r_{\\theta_t}$ will lead to a better learned policy $\\pi_t$ that generates trajectories closer to the expert trajectory, and thus the new approximation suffix $\\\\{s\\_i'',a\\_i''\\\\}\\_{i\\geq t+1}$ will be closer to the expert suffix $\\\\{s\\_i^E,a\\_i^E\\\\}\\_{i\\geq t+1}$, i.e., the approximation error becomes smaller. Therefore, by comparing the learner suffix $\\\\{s\\_i',a\\_i'\\\\}\\_{i\\geq t+1}$ and the approximation suffix $\\\\{s\\_i'',a\\_i''\\\\}\\_{i\\geq t+1}$, the learner suffix $\\\\{s\\_i',a\\_i'\\\\}\\_{i\\geq t+1}$ gets closer to the expert suffix $\\\\{s\\_i^E,a\\_i^E\\\\}\\_{i\\geq t+1}$. \n\n**Question 2**: Just wondering how you complete the incomplete trajectories in MuJoCo using saved states, since the state in MuJoCo only includes partial information about the simulator.  \n**Answer**: In MuJoCo, the learned policy usually maps the observation to a distribution of actions. Given the current observation and current action, the simulator tells us the next observation. We agree that the observation only includes partial information about the simulator. However, the observation is not the physical state of the simulated robot. The physical state in MuJoCo is characterized by two vectors, called \"qpos\" and \"qvel\". The vector \"qpos\" stores all the current positions and orientations of the joints and bodies and the vector \"qvel\" stores all the current linear and angular velocities of the joints and bodies. The vectors \"qpos\" and \"qvel\" include the complete information of the simulated robot. Therefore, we save \"qpos\" and \"qvel\" instead of the observation to save a physical state of the simulated robot. In practice, we use the command \"env.data.qpos\" and \"env.data.qvel\" to obtain the \"qpos\" and \"qvel\" vectors of the current simulated robot and save these vectors. When we want to set the robot's physical state to a saved (qpos,qvel), we use the command \"env.set\\_state(qpos,qvel)\". These commands work for gym-v4.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "enGb36xd8N", "reviewer": "Reviewer_L1zi", "summary": "In this paper, a novel framework for performing Inverse Reinforcement Learning (IRL) from an ongoing trajectory, i.e., to learn a reward function that induces an optimal policy that best explains the expert's demonstrations sequentially, without waiting to having observed the expert's trajectory entirely. Authors propose an online learning algorithm for solving the problem, and this algorithm has the added feature of meta-regularization. Both a theoretical and empirical analysis of the algorithm are provided to validate the results.", "questions": "- What reward are you extracting among all the rewards that explain the observed demonstrations? What can it be used for?\n- Since in RL what matters is the cumulative sum of rewards, how can you be sure that you extract a meaningful reward even though you have not observed the expert's policy for the entire horizon?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The framework of learning from ongoing trajectories is interesting and finds some applications.", "weaknesses": "- There is no clear discussion and intuition of what kind of reward function is extracted by the algorithm and what it can be used for. For instance, various works about identifiability or the feasible set concern what we can do with the extracted rewards, because in most cases we cannot just extract a single reward explaining the demonstrations. However, this work does not analyse this fact.\n- The presentation of the contents and contributions is poor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive reviews. We believe that our discussion will lead to a better paper. Before addressing your comments, we would like to clarify the goal of IRL and what role the reward unidentifiability issue plays. IRL aims to learn a reward function that can explain the expert demonstrations. Explaining expert demonstrations means that the optimal policy of the learned reward generates the same trajectories as the expert demonstrations [C1,C2]. Current IRL methods can already learn a single reward function to explain expert demonstrations [C1,C7,C8] but cannot solve the reward unidentifiability issue in general cases. Reward unidentifiability means that even if we can learn a single reward to explain the demonstrations, we cannot guarantee that the learned reward is the expert reward because there are many different rewards that can explain the expert demonstrations. There are some papers studying reward identifiability, however, they all impose additional assumptions. For example, [C3] shows that the reward can be recovered up to a constant if we have demonstrations under different discount factors or different environments. [C4] quantifies the reward distance between the learned reward and the ground truth reward under the assumption that the MDP is linear. [C5,C6] study feasible reward set to identify a set of rewards that can explain the demonstrations, however, they do not solve the identifiability issue because the feasible reward set still includes multiple reward functions that can explain the expert demonstrations. In summary, if we want to learn a single reward to explain expert demonstrations, it is not necessary to solve reward unidentifiability issue nor study reward feasible sets. In fact, the reward unidentifiability issue in general IRL cases is still unsolved to IRL community.\n\n**Weakness 1**: There is no clear ... this fact.  \n**Answer**: Thanks for mentioning (1) reward identifiability and feasible reward set, (2) what kind of reward is extracted, and (3) what the learned reward can be used for. We address these three comments one by one.\n\nFor the reward identifiability and feasible reward set, please refer to the first paragraph in this response that (i) it is sufficient for IRL to learn a single reward function to explain the demonstrations, (ii) reward identifiability and learning a feasible reward set are sufficient but not necessary for reward learning. Note that reward unidentifiability in general IRL cases is still unsolved to IRL community. In this paper, we empirically compare the learned reward and the expert reward in Appendix D.2, and the results show that the learned reward is close to the expert reward. As mentioned in Appendix F, we would like to explore the reward identifiability in future works.\n\nWe now discuss what kind of reward we  extract. Lines 89-94 mention that the kind of reward we aim to learn is the reward with maximum likelihood. Maximum likelihood IRL [C8,C9] is a standard IRL approach to learn a reward function to explain the expert demonstrations. The intuition behind this is that we aim to learn a reward function whose corresponding optimal policy makes the expert demonstrations most likely (line 94), i.e., the corresponding optimal policy can generate trajectories that are same with the expert demonstrations. Therefore, the learned reward with maximum likelihood explains the expert demonstrations [C8,C9].\n\nThe learned reward can be used to explain the demonstrations. Moreover, compared to imitation learning that directly learns an imitating policy, the benefit of learning a reward is that the learned reward can be used for counterfactual analysis such as the estimation of optimal policies under different environment dynamics [C8].\n\n**Weakness 2**: The presentation ... is poor.  \n**Answer**: Thanks for mentioning the presentation of the contents and contributions. We have a contribution statement in lines 38-53 that clearly summarizes our contributions: (1) we study a novel problem setting of in-trajectory learning, formulate it as a bi-level optimization problem, and propose a novel reward update mechanism to solve the problem; (2) we provide solid theoretical analysis to guarantee that the algorithm achieves sub-linear (local) regret where the input data is not i.i.d.\n\nTo improve the presentation of other contents, we will (i) add more explanations about problem (1) to highlight the kind of reward we learn, (ii) add the discussion on related works about reward identifiability and feasible reward set, (iii) add more discussions on the technical assumptions and the theoretical statements.\n\n**Question 1**: What reward ... used for?  \n**Answer**: Please refer to the answer to weakness 1.\n\n**Question 2**: Since in RL ... the entire horizon?  \n**Weakness**: Please refer to the global response.\n\n[C1] Pieter Abbeel, and Andrew Y. Ng. \"Apprenticeship learning via inverse reinforcement learning.\" ICML, 2004.\n\n[C2] Saurabh Arora, and Prashant Doshi, \"A survey of inverse reinforcement learning: Challenges, methods and progress.\" Artificial Intelligence, 2021.\n\n[C3] Haoyang Cao, et al. \"Identifiability in inverse reinforcement learning\", NeurIPS, 2021.\n\n[C4] Zihao Li, et al. \"Reinforcement learning with human feedback: Learning dynamic choices via pessimism\", arXiv preprint arXiv:2305.18438, 2023.\n\n[C5] Alberto Maria Metelli, et al. \"Towards theoretical understanding of inverse reinforcement learning\", ICML, 2023.\n\n[C6] Filippo Lazzati, et al. \"Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms\", arXiv preprint, arXiv:2402.15392, 2024.\n\n[C7] Brian D. Ziebart, et al. \"Maximum entropy inverse reinforcement learning.\" AAAI, 2008.\n\n[C8] Siliang Zeng, et al. \"Maximum-likelihood inverse reinforcement learning with finite-time guarantees.\" NeurIPS, 2022.\n\n[C9] Shicheng Liu and Minghui Zhu. \"Learning multi-agent behaviors from distributed and streaming demonstrations.\" NeurIPS, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**How can we extract a meaningful reward even though we have not observed the expert’s policy for
the entire horizon?**

The reason is that our reward update approximates the entire expert horizon and learns from the approximate entire expert trajectory (detailed in lines 188-219). We include a figure (Figure 4) in the uploaded one-page PDF to visualize this process.

Consider the standard IRL (visualized in Figure 4a) where we can compare the entire expert trajectory $\\{s\_i^E, a\_i^E\\}\_{i \geq 0}$ (the green one in Figure 4) and entire learner trajectory $\\{s\_i', a\_i'\\}\_{i \geq 0}$ (the red one in Figure 4). In this standard IRL case, the learned reward considers the entire horizon. In our case, at time $t$, we only observe expert trajectory prefix $\\{s\_i^E, a\_i^E\\}\_{0 \leq i \leq t}$. To ensure that the learned reward still considers the entire horizon, we use the learned policy to complete the expert trajectory by generating the suffix $\\{s\_i'', a\_i''\\}\_{i \geq t}$ that starts from $(s\_t^E, a\_t^E)$ (the yellow one in Figure 4b). Now we have a combined entire trajectory whose prefix is $\\{s\_i^E, a\_i^E\\}\_{0 \leq i \leq t}$ and suffix is $\\{s\_i'', a\_i''\\}\_{i \geq t}$. This combined trajectory approximates the entire expert trajectory $\\{s\_i^E, a\_i^E\\}\_{i \geq 0}$ where the suffix $\\{s\_i'', a\_i''\\}\_{i \geq t}$ approximates the expert suffix $\\{s\_i^E, a\_i^E\\}\_{i \geq t}$. We compare this combined entire trajectory with the entire learner trajectory $\\{s\_i', a\_i'\\}\_{i \geq 0}$ to enable the learned reward to consider the entire horizon. Moreover, this comparison encourages the learner suffix $\\{s\_i', a\_i'\\}\_{i \geq t}$ to approach the approximation suffix $\\{s\_i'', a\_i''\\}\_{i \geq t}$. If the approximation suffix $\\{s\_i'', a\_i''\\}\_{i \geq t}$ is a good approximation of the expert suffix $\\{s\_i^E, a\_i^E\\}\_{i \geq t}$, the learned reward can be accurate for the entire horizon.

Now the question left is whether the approximation suffix $\\{s\_i'', a\_i''\\}\_{i \geq t}$ is a good approximation of $\\{s_i^E, a_i^E\\}\_{i \geq t}$. The answer is that the approximation error between $\\{s\_i'',a\_i''\\}\_{i\geq t}$ and $\\{s\_i^E,a\_i^E\\}\_{i\geq t}$ diminishes when $t$ increases. This is mathematically justified by the sub-linear regret in Theorems 1 and 2 because the regret will be at least linear if the approximation error does not diminish or is always above a certain constant. The intuition behind this is that when $t$ increases, we observe more portion of the expert trajectory and thus the learned reward $r_{\theta_t}$ improves. A better $r_{\theta_t}$ will lead to a better learned policy $\pi_t$ that generates trajectories closer to the expert trajectory, and thus the new approximation suffix $\\{s\_i'',a\_i''\\}\_{i\geq t+1}$ will be closer to the expert suffix $\\{s\_i^E,a\_i^E\\}\_{i\geq t+1}$, i.e., the approximation error becomes smaller (visualized in Figure 4c). Therefore, by comparing the learner suffix $\\{s\_i',a\_i'\\}\_{i\geq t+1}$ and the approximation suffix $\\{s\_i'',a\_i''\\}\_{i\geq t+1}$, the learner suffix $\\{s\_i',a\_i'\\}\_{i\geq t+1}$ gets closer to the expert suffix $\\{s\_i^E,a\_i^E\\}\_{i\geq t+1}$.

Meta-review: This paper considers in-trajectory IRL, where a reward function and a corresponding policy need to be learned from an ongoing trajectory. The authors introduce an online learning algorithm to address this IRL problem with incomplete expert demonstrations. The paper includes a thorough theoretical analysis of the proposed solution, as well as a convincing empirical analysis on simulated data and some toy problems. A reviewer raised the issue regarding the identifiability of reward functions, but other reviewers think that this is a minor issue in the context of the other contributions of this work.
Decision: Accept (poster)