Title: In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before an Ongoing Trajectory Terminates
Venue: NeurIPS 2024
Accepted: yes
Keywords: inverse reinforcement learning
Abstract: Inverse reinforcement learning (IRL) aims to learn a reward function and a corresponding policy that best fit the demonstrated trajectories of an expert. However, current IRL works cannot learn incrementally from an ongoing trajectory because they have to wait to collect at least one complete trajectory to learn. To bridge the gap, this paper considers the problem of learning a reward function and a corresponding policy while observing the initial state-action pair of an ongoing trajectory and keeping updating the learned reward and policy when new state-action pairs of the ongoing trajectory are observed. We formulate this problem as an online bi-level optimization problem where the upper level dynamically adjusts the learned reward according to the newly observed state-action pairs with the help of a meta-regularization term, and the lower level learns the corresponding policy. We propose a novel algorithm to solve this problem and guarantee that the algorithm achieves sub-linear local regret $O(\sqrt{T}+\log T+\sqrt{T}\log T)$. If the reward function is linear, we prove that the proposed algorithm achieves sub-linear regret $O(\log T)$. Experiments are used to validate the proposed algorithm.

In-Trajectory Inverse Reinforcement Learning: Learn Incrementally Before an Ongoing Trajectory Terminates

Abstract

Inverse reinforcement learning (IRL) aims to learn a reward function and a corresponding policy that best fit the demonstrated trajectories of an expert. However, current IRL works cannot learn incrementally from an ongoing trajectory because they have to wait to collect at least one complete trajectory to learn. To bridge the gap, this paper considers the problem of learning a reward function and a corresponding policy while observing the initial state-action pair of an ongoing trajectory and keeping updating the learned reward and policy when new stateaction pairs of the ongoing trajectory are observed. We formulate this problem as an online bi-level optimization problem where the upper level dynamically adjusts the learned reward according to the newly observed state-action pairs with the help of a meta-regularization term, and the lower level learns the corresponding policy. We propose a novel algorithm to solve this problem and guarantee that the algorithm achieves sub-linear local regret O ( √ T +log T + √ T log T ) . If the reward function is linear, we prove that the proposed algorithm achieves sub-linear regret O (log T ) . Experiments are used to validate the proposed algorithm.

1 Introduction

Inverse reinforcement learning (IRL) aims to learn a reward function and a corresponding policy that are consistent with the demonstrated trajectories of an expert. In recent years, several IRL methods are proposed to help learn the reward and policy, including maximum margin methods [1, 2], maximum entropy methods [3, 4], maximum likelihood methods [5, 6], and Bayesian methods [7, 8].

The aforementioned IRL works learn from pre-collected demonstration sets and do not improve the learned model during deployment. Online IRL [9, 10, 11] instead can learn from sequentially arrived demonstrated trajectories and continuously improve the learned reward and policy from the newly observed complete trajectories. However, recent applications of IRL motivate the need to learn incrementally from an ongoing trajectory before it terminates. For example, inferring a moving shooter's intention from its ongoing movement in order to evacuate the hiding victims [12] before the shooter finds them. In this case, we need to quickly update the inference about the shooter's intention once a new movement of the shooter is observed, so that we can use the latest inference to plan a rescue strategy as soon as possible. We cannot wait until the shooter trajectory ends, in case the shooter has found the victims. Another example is learning a target customer's investment preference from its daily updated investment trajectory in a stock market [13] in order to recommend appropriate stocks [14, 15] before other competitors get this customer. However, current IRL works cannot learn from an ongoing trajectory because they have to wait to collect at least one complete trajectory to learn from. To bridge the gap, this paper proposes in-trajectory IRL, a new type of IRL that learns a

reward function and a corresponding policy at the initial state-action pair of an ongoing trajectory, and keeps updating the learned reward and policy once a new state-action pair of the ongoing trajectory is observed. We summarize our contributions as follows:

Contribution statement . This paper proposes the first in-trajectory IRL framework, termed me tar egularized i nt rajectory inverse reinforcement learning" (MERIT-IRL), to learn a reward function and a corresponding policy from an ongoing trajectory. Our contributions are twofold.

First, we formulate this in-trajectory learning problem as an online bi-level optimization problem where the upper level continuously updates the learned reward according to the newly observed state-action pairs and the lower level computes the corresponding policy. We develop a novel online learning algorithm (MERIT-IRL) to solve this problem. The major novelty of the proposed algorithm is that we propose a novel reward update mechanism specially designed for the in-trajectory learning setting. This special reward update not only aims to explain the expert trajectory observed so far, but also aims to consider for the future. Moreover, since the data is lacking as there is only one ongoing trajectory, we introduce a meta-regularization term to embed prior knowledge and avoid overfitting.

Second, we theoretically guarantee that MERIT-IRL achieves sub-linear local regret O ( √ T +log T + √ T log T ) . If the reward function is linear, we prove that MERIT-IRL achieves sub-linear regret O (log T ) . The major novelty of the theoretical analysis is to address the difficulty that the input data is not identically independent distributed (i.i.d.) but temporally correlated, i.e., the data ( s t , a t ) at each time t is affected by the data ( s t -1 , a t -1 ) at last time.

2 Related works

Due to the space limit, we only discuss the related works on learning from incomplete trajectories here, and we include the discussion on more related works in Appendix A.

Papers [16, 17] use partial trajectories to update the learned reward function by comparing the expert trajectories after the expert states and the trajectories starting from those expert states rolled out by the learned policy. However, given the current expert state, they use expert trajectory suffix (i.e., future trajectory) to compare while we can only access expert trajectory prefix (i.e., previous trajectory) from an ongoing trajectory. Papers [18, 19, 20] use imitation learning to learn from incomplete demonstrations. In specific, paper [18] uses a discounted sum along the future trajectory as the weight for weighted behavior cloning and works effectively even if only portions of trajectories are observed. Papers [19, 20] extend GAIL [21] to solve for the case where the action sequences are not complete. However, these works all require a pre-collected set of demonstrations so that they are not in-trajectory learning since the trajectory in their cases is not ongoing.

3 Problem Formulation

In this section, we formulate the problem of in-trajectory IRL. In in-trajectory IRL, there is an expert whose decision-making is based on a Markov decision process (MDP). An MDP is a tuple ( S , A , γ, r E , P 0 , P ) which consists of a state set S , an action set A , a discount factor γ ∈ [0 , 1) , a reward function r E : S × A → R , and the initial state distribution P 0 ( · ) . The state transition probability (density) function is denoted by P ( ·|· , · ) such that P ( s ′ | s, a ) denotes the probability (density) of state transition to s ′ from s by taking action a . The expert is using its policy π E to demonstrate an ongoing trajectory ζ E = S E 0 , A E 0 , S E 1 , A E 1 , · · · and at each time t , only the stateaction pair ( S E t , A E t ) is observed. We want to learn a reward function and a corresponding policy from the ongoing trajectory and update the learned reward function and policy at each time t .

Many IRL algorithms [5, 6, 11, 22, 23, 24] employ a bi-level learning structure. In this structure, the upper level learns a reward function while the lower level aims to find an associated policy by solving an RL problem under the current learned reward function. Inspired by their bi-level learning structure, we formulate the in-trajectory IRL problem as an online non-convex bi-level optimization problem. In specific, we aim to learn a reward function r θ (parameterized by θ ) in the upper level and a policy corresponding to r θ in the lower level. The loss function at time t is defined as:

Note that L t ( θ ; ( S E t , A E t )) is defined using π θ and θ is the parameter of the reward function r θ , here the policy π θ is also parameterized by θ because it is computed by solving an RL problem (in the lower level) under the reward function r θ , and thus is indirectly parameterized by θ . Maximum likelihood IRL (ML-IRL) [5] has a similar bi-level formulation with (1), however, ML-IRL only solves an offline optimization problem and its analysis does not hold for non i.i.d. input data and continuous state-action space. We discuss our distinctions from ML-IRL in Appendix A.1.

The upper-level loss function L t has two terms. The first term -γ t log π θ ( A E t | S E t ) is the discounted negative log-likelihood of the state-action pair ( S E t , A E t ) at time t and the second term λγ t 2 || θ -¯ θ || 2 is the discounted meta-regularization term [25] where λ is a hyper-parameter. The likelihood function is commonly used in IRL [5, 6] to learn a reward function. Basically, the upper-level loss function at time t encourages to find a reward function r θ that makes the observed state-action pair ( S E t , A E t ) most likely and meanwhile, the reward parameter θ should not be too far from the prior experience, i.e., the meta-prior ¯ θ . Note that ¯ θ is a pre-trained meta-prior that embeds the information of 'relevant experience". We will introduce the training of ¯ θ in Subsection 4.3 and Appendix C.

The lower-level problem is used to compute π θ using the current reward function r θ . It proposes to find a policy π θ that maximizes the entropy-regularized cumulative reward J θ ( π ) + H ( π ) . The cumulative reward of a policy π under the reward function r θ is J θ ( π ) ≜ E π S,A [ ∑ ∞ t =0 γ t r θ ( S t , A t )] where the initial state is drawn from P 0 . The causal entropy of a policy π is defined as H ( π ) ≜ E π S,A [ -∑ ∞ t =0 γ t log π ( A t | S t )] .

Since the expert demonstrates { ( S E t , A E t ) } t ≥ 0 sequentially, we have a sequence of loss functions { L t ( θ ; ( S E t , A E t )) } t ≥ 0 . We use this sequence of loss functions (1) to formulate an online learning problem. A typical online learning problem is to minimize the regret: ∑ T -1 t =0 L t ( θ t ; ( S E t , A E t )) -min θ ∑ T -1 t =0 L t ( θ ; ( S E t , A E t )) . However, it is too challenging to minimize the regret in our case because the loss function L t could be non-convex. Therefore, we aim to minimize the local regret which is widely adopted in online non-convex optimization [26, 27] and online IRL [11]. The local regret quantifies the general stationarity of a sequence of loss functions under the learned parameters. In specific, given a sequence of loss functions { f t ( x ) } t ≥ 0 , the local regret [11, 26, 27] at time t is defined as || 1 t +1 ∑ t i =0 ∇ f i ( x t ) || 2 which quantifies the gradient norms of the average of all the previous loss functions under the current learned parameter x t . The total local regret is defined as the sum of the local regret at each time t , i.e., ∑ T -1 t =0 || 1 t +1 ∑ t i =0 ∇ f i ( x t ) || 2 . In our case, we replace { f t } t ≥ 0 with the loss function { L t } t ≥ 0 defined in (1) and thus formulate the local regret (2)-(3) which has a bi-level formulation. We aim to minimize the following local regret:

where ( S E t , A E t ) ∼ P π E t ( · , · ) means that ( S E t , A E t ) is drawn from the state-action distribution P π E t ( · , · ) , and P π t ( · , · ) is the state-action distribution induced by π at time t in the MDP.

Difficulties of solving problem (2) -(3). We want to design a fast algorithm to solve problem (2)-(3) since we need to finish the update of θ and π before the next state-action pair is observed and the time between two consecutive state-action pairs can be short. However, designing and analyzing such a fast algorithm is difficult due to the following challenges:

(i) First and foremost, current state-of-the-arts [26, 27] on online non-convex optimization use followthe-leader-based algorithms which solve min θ ∑ t i =0 L i ( θ ; ( S E i , A E i )) to near stationarity at each time t . This is time-consuming because they require multiple gradient descent updates of θ . One way to alleviate this problem is to use online gradient descent (OGD) which only updates θ by one gradient descent step at each time t . However, since OGD does not solve the problem to near stationarity at any time t , it is extremely difficult to quantify the overall stationarity after T iterations. While OGD has been well studied in online convex optimization, it is rarely studied in online non-convex optimization. The recent work [11] uses OGD to quantify the local regret, however, its analysis can only hold when the input data is i.i.d. In contrast, the input data in our problem is not i.i.d. In specific, the input data at time t (i.e., ( S E t , A E t ) ) is actually affected by the input data at last step (i.e.,

( S E t -1 , A E t -1 ) ). This correlation between any two consecutive input data makes it difficult to analyze the growth rate of the local regret.

(ii) Second, it is time-consuming if we fully solve the lower-level problem (3) to get π θ because this requires multiple policy updates to solve an RL problem. Therefore, we use a 'single-loop" method which only requires one-step policy update for a given r θ . However, since the policy is only updated once, the updated policy can be far from π θ and thus making the analysis difficult. Single-loop methods are widely adopted to solve hierarchical problems, including bi-level optimization [28, 29], game theory [30, 31], min-max problems [32, 33], etc. Recently, single-loop methods are applied to IRL [5], however, the paper [5] only solves an offline optimization problem and its analysis does not hold for non i.i.d. input data and continuous state-action space. We include a section in Appendix A.1 to discuss our distinctions from [5].

4 Algorithm and Theoretical Analysis

This section has three parts. The first part presents a novel online learning algorithm that solves the problem (2)-(3) and tackles the aforementioned two difficulties. The second part proves that Algorithm 1 achieves sub-linear local regret. If the reward function is linear, we prove that Algorithm 1 achieves sub-linear regret. The third part introduces a meta-learning method to get the meta-prior ¯ θ .

4.1 The proposed algorithm

In practice, the expert will demonstrate a specific trajectory s E 0 , a E 0 , s E 1 , a E 1 , · · · . For distinction, we use the capital letters (e.g., S ) to represent random variables and the lower-case letters (e.g., s ) to represent specific values. To design a fast algorithm, we propose an online-gradient-descent-based single-loop algorithm. In specific, at each time t , the algorithm updates both policy π and reward parameter θ only once. The policy update is to solve the lower-level problem (3) and the reward update is to solve the upper-level problem (2). In the following, we elaborate the procedure of policy update and reward update.

Algorithm 1 Meta-regularized In-trajectory Inverse Reinforcement Learning (MERIT-IRL)

Input : Initialized policy π 0 , the streaming input data { ( s E t , a E t ) } t ≥ 0 Output : Learned reward parameter θ T and policy π T

1: Compute ¯ θ using the meta-regularization in Section 4.3 and Appendix C, and set θ 0 = ¯ θ

2: for t = 0 , 1 , · · · , T -1 do

3: Compute the soft Q-function Q soft θ t ,π t (defined in Appendix B.1) under the current reward function r θ t and policy π t

4: Update π t +1 ( a | s ) ∝ exp( Q soft θ t ,π t ( s, a )) for any ( s, a ) ∈ S × A

5: Roll out policy π t +1 twice: one starting from s E 0 to get s E 0 , a ′ 0 , s ′ 1 , a ′ 1 , · · · , and the other starting from ( s E t , a E t ) to get s ′′ t +1 , a ′′ t +1 , · · ·

6: Compute g t = ∑ ∞ i =0 γ i ∇ θ r θ t ( s ′ i , a ′ i ) -∑ ∞ i =0 γ i ∇ θ r θ t ( s ′′ i , a ′′ i ) + λ (1 -γ t +1 ) 1 -γ ( θ t -¯ θ ) where s ′ 0 = s E 0 and ( s ′′ i , a ′′ i ) = ( s E i , a E i ) for 0 ≤ i ≤ t

7: Update θ t +1 = θ t -α t g t

8: end for

Policy update (lines 3-4 of Algorithm 1). At each time t , we only partially solve the lower-level problem (3) via one-step soft policy iteration [5, 34]. In specific, the soft policy iteration contains two steps: policy evaluation and policy improvement. Policy evaluation aims to compute the soft Q -function Q soft θ t ,π t (see the expression in Appendix B.1) under the current learned reward function r θ t and learned policy π t . Policy improvement aims to update policy according to π t +1 ( s, a ) ∝ exp( Q soft θ t ,π t ( s, a )) for any ( s, a ) ∈ S × A . In practical implementations, π t +1 can be obtained by one-step policy update in soft Q-learning [34] or one-step actor update in soft actor-critic [35].

Reward update (lines 5-7 of Algorithm 1). At each time t , the algorithm observes ( s E t , a E t ) and aims to leverage all the previously observed data to update the reward parameter. In specific, as L t ( θ ; ( s E t , a E t )) = -γ t log π θ ( a E t | s E t )+ λγ t 2 || θ -¯ θ || 2 is the meta-regularized negative log-likelihood

of ( s E t , a E t ) , the algorithm can formulate ∑ t i =0 L i ( θ ; ( s E i , a E i )) at time t using all the previously collected data (i.e., { ( s E i , a E i ) } i =0 , ··· ,t ). To update the reward parameter, the algorithm partially minimizes ∑ t i =0 L i ( θ ; ( s E i , a E i )) via one-step gradient descent.

Lemma 1. The gradient of ∑ t i =0 L i ( θ ; ( s E i , a E i )) can be calculated as follows:

Note that the gradient (4) holds for continuous state-action space. Since the gradient (4) has expectation terms under the policy π θ , we can only approximate it. In specific, we roll out the policy π t +1 twice: one starting from s E 0 to get a trajectory { ( s ′ i , a ′ i ) } i ≥ 0 where s ′ 0 = s E 0 , and the other one starting from ( s E t , a E t ) to get a trajectory { ( s ′′ i , a ′′ i ) } i ≥ 0 where ( s ′′ i , a ′′ i ) = ( s E i , a E i ) for 0 ≤ i ≤ t . Then we use the empirical estimate g t = ∑ ∞ i =0 γ i ∇ θ r θ t ( s ′ i , a ′ i ) -∑ ∞ i =0 γ i ∇ θ r θ t ( s ′′ i , a ′′ i ) + λ (1 -γ t +1 ) 1 -γ ( θ t -¯ θ ) to approximate ∇ ∑ t i =0 L i ( θ t ; ( s E i , a E i )) . With the gradient approximation g t , we utilize stochastic online gradient descent θ t +1 = θ t -α t g t to update the reward parameter.

Discussion on our special design of the reward update . The right subfigure in Figure 1 visualizes our reward update (modulo the meta-regularization term). The green trajectory (i.e., { ( s E t , a E t ) } t ≥ 0 ) is the expert trajectory, and the red trajectories (i.e., { ( s ′ t , a ′ t ) } t ≥ 0 and { ( s ′′ i , a ′′ i ) } i>t ) are the trajectories generated by the learned policy. Given the expert trajectory prefix (i.e., the incomplete trajectory { ( s E i , a E i ) } t i =0 observed so far), our method completes the expert trajectory by rolling out the learned policy starting from ( s E t , a E t ) and filling the trajectory suffix { ( s ′′ i , a ′′ i ) } i>t . The combined complete trajectory includes the expert trajectory prefix { ( s E i , a E i ) } t i =0 and the learner-filled trajectory suffix { ( s ′′ i , a ′′ i ) } i ≥ t . We update the reward function by comparing this combined trajectory to a complete trajectory { ( s ′ t , a ′ i ) } t ≥ 0 generated by the learned policy starting from the expert's initial state s E 0 .

A more straightforward way for the reward update is to directly compare the trajectory prefixes (visualized in the middle of Figure 1) at each time t . However, this naive method can be problematic. We explain the issue of this naive method and the advantage of our method in the following context.

(s,a) (s,a)

(s,a) (s,a)

(s,a) (s,a)

(sE,a)(s,a)

(s,a)(s,a')

(s,a)(s,a')

全

(sE,aE)(st,at)

(sE,aE)+(st,at)

(s+1,at+1)+($t+1,at+1)

($+1, a+1)→($t+1,at+1)

(s,a)(s,a)

(s",a")  (s,aT)

ore

T→+8[

1T→+8

StandardIRL

Naivemethod

Ourmethod

Figure 1 visualizes the reward update for standard IRL (left), the naive method (i.e., directly run standard IRL algorithms on the expert trajectory prefix) (middle), and our method (right). The standard IRL (left) updates the reward function by comparing the complete expert trajectory and the complete trajectory generated by the learned policy. This case is ideal, however, it is infeasible when the trajectory is ongoing and we can only observe an incomplete expert trajectory { ( s E i , a E i ) } t i =0 at each time t . The naive method (mid- dle) updates the reward function by simply comparing the expert trajectory prefix { ( s E i , a E i ) } t i =0 and the incomplete trajectory { ( s ′ i , a ′ i ) } t i =0 with the same length generated by the learned policy starting from the expert's initial state s E 0 . This kind of reward update is myopic as it does not consider for the future. Since it runs standard IRL on the trajectory prefix observed so far, it will always regard the current state-action pair as the terminal state-action pair and thus has no ability to consider the state-action pairs in the future. In contrast, our special design gives the algorithm the ability to consider for the future. Instead of only comparing incomplete trajectory prefixes, our method compares complete trajectories just as the standard IRL. In specific, we compare the combined complete trajectory {{ ( s E i , a E i ) } t i =0 , { ( s ′′ i , a ′′ i ) } i>t } and the complete trajectory { ( s ′ i , a ′ i ) } i ≥ 0 solely generated by the learned policy. The comparison between the learner prefix { s ′ i , a ′ i } t i =0 and expert

prefix { s E i , a E i } t i =0 encourages the learned reward function to explain the expert's demonstrated behaviors so far, and the comparison between the suffixes ( { s ′ i , a ′ i } i>t and { s ′′ i , a ′′ i } i>t ) encourages that we are learning a reward function that is useful for predicting the future. Note that as t increases, the expert trajectory prefix weights more and more in the combined complete trajectory, and eventually we will recover the standard IRL reward update when t goes to infinity. In Theorems 1 and 2, we theoretically guarantee that the proposed reward update can achieve sub-linear (local) regret. This shows the perfect consistency between the intuition and theory.

4.2 Theoretical analysis

To quantify the local regret of Algorithm 1, we have two challenges: (i) Since we only update π by one step at each time t , we have π t +1 instead of the optimal solution π θ t of the lower-level problem (3). The policy π t +1 can be far away from π θ t and thus the empirical gradient estimate g t can be a bad approximation of the gradient (4). (ii) Since we only update θ once at each time t instead of finding a near-stationary point θ ′ such that || ∑ t i =0 L i ( θ ′ ; ( s E i , a E i )) || ≤ ϵ as in [26, 27], the gradient norm || ∑ t i =0 ∇ L i ( θ t ; ( s E i , a E i )) || is not stabilized under the threshold ϵ at every time t . Therefore the local regret (i.e., ∑ T -1 t =0 || 1 t +1 ∑ t i =0 ∇ L i ( θ t ; ( s E i , a E i )) || 2 ) is hard to quantify and may not be sub-linear in T . What's worse, the input data is not i.i.d. but correlated, i.e., the input data ( s E t , a E t ) at time t is affected by the input data ( s E t -1 , a E t -1 ) at last step. This correlation makes it even more difficult to quantify the local regret.

To solve the first challenge, we adopt the idea of two-timescale stochastic approximation [28] where the lower level updates in a faster timescale and the upper level updates in a slower timescale. The policy update is faster because it converges linearly under a fixed reward function [36] while the reward update is slower given that we choose α t ∝ ( t +1) -1 / 2 . Intuitively, since the policy update is faster than the reward update, the reward parameter is 'relatively fixed" compared to the policy. It is expected that π t +1 shall stay close to π θ t and at last converges to π θ t when t increases.

To solve the second challenge, we divide our analysis into two steps: (i) We quantify the difference of the gradient norms between the current correlated state-action distribution P π E t ( · , · ) and a stationary state-action distribution for any loss function L i , i ≥ 0 . Note that P π E t +1 ( · , · ) is affected by P π E t ( · , · ) . (ii) We quantify the local regret under the stationary distribution. The benefit of doing so is that the input data is i.i.d. under the stationary distribution, and thus we can cast the online gradient descent method as a stochastic gradient descent method and quantify its local regret. Finally, we can quantify the local regret under the current correlated distribution P π E t ( · , · ) by combining (i) and (ii).

We start our analysis with the definitions of stationary state distribution and stationary state-action distribution. For a given policy π , the corresponding stationary state distribution is µ π ( s ) ≜ (1 -γ ) ∑ ∞ t =0 γ t P π t ( s ) and the stationary state-action distribution is µ π ( s, a ) ≜ (1 -γ ) ∑ ∞ t =0 γ t P π t ( s, a ) . Assumption 1. The parameterized reward function r θ satisfies | r θ 1 ( s, a ) -r θ 2 ( s, a ) | ≤ ¯ C r || θ 1 -θ 2 || and ||∇ θ r θ 1 ( s, a ) -∇ θ r θ 2 ( s, a ) || ≤ ˜ C r || θ 1 -θ 2 || for any ( θ 1 , θ 2 ) and any ( s, a ) ∈ S × A where ¯ C r and ˜ C r are positive constants.

Assumption 2 (Ergodicity) . There exist constants C M > 0 and ρ ∈ (0 , 1) such that for any policy π and any t ≥ 0 , the following holds for the Markov chain induced by the policy π and the state transition function P : sup S 0 ∼ P 0 d TV ( P π t ( · ) , µ π ( · )) ≤ C M ρ t where d TV ( P 1 ( · ) , P 2 ( · )) ≜ 1 2 ∫ s ∈S | P 1 ( s ) -P 2 ( s ) | ds is the total variation distance between the two state distributions P 1 and P 2 , S 0 is the initial state, and P π t ( · ) is the state distribution induced by the policy π at time t .

Assumptions 1-2 are common in RL [37, 38, 39, 40]. Assumption 2 holds for any time-homogeneous Markov chain with finite state space or any uniformly ergodic Markov chain with general state space.

Proposition 1. Suppose Assumptions 1-2 hold and α t ∈ (0 , 1 -γ λ ) , we have the following relation for any i ≥ 0 and any θ t , t ≥ 0 :

where ( S E i , A E i ) ∼ P π E i ( · , · ) means that ( S E i , A E i ) is drawn from the correlated distribution P π E i ( · , · ) and ( S E i , A E i ) ∼ µ π E ( · , · ) means that ( S E i , A E i ) is drawn from the stationary distribution µ π E ( · , · ) .

Proposition 1 quantifies the gap of gradient norms between the current correlated distribution P π E i and the stationary distribution µ π E . We next quantify the local regret under the stationary distribution µ π E with the following lemma:

Lemma 2. Suppose Assumptions 1-2 hold and choose α t = (1 -γ )( t +1) -1 / 2 λ , it holds that:

where D 1 , D 2 , and D 3 are positive constants whose expressions can be found in Appendix B.4.

Lemma 2 quantifies the local regret under the stationary distribution µ π E . With Proposition 1 and Lemma 2, we can quantify the local regret under the current correlated distribution.

Theorem 1. Suppose Assumptions 1-2 hold and choose α t = (1 -γ )( t +1) -1 / 2 λ , we have that:

Theorem 1 is based on Proposition 1 and Lemma 2. It shows that Algorithm 1 achieves sub-linear local regret. Moreover, if the reward function is linear, Algorithm 1 achieves sub-linear regret:

Theorem 2. Suppose the expert reward function r E and the parameterized reward r θ are linear, and Assumptions 1-2 hold. Choose α t = 1 -γ λ ( t +1)(1 -γ t +1 ) we have that:

where D 4 and D 5 are positive constants whose expressions are in Appendix B.6.

4.3 Meta-Regularization

Since there is only one training trajectory and this trajectory is not complete during the learning process, we need to add a regularization term to avoid overfitting. Inspired by humans' using relevant experience to help do inference, we introduce the meta-regularization λ 2 || θ -¯ θ || 2 where λ is the hyper-parameter and the meta-prior ¯ θ is learned from 'relevant experience". In specific, we introduce a set of relevant tasks {T j } j ∼ P T where each task T j is an IRL problem and P T is the implicit task distribution. The tasks {T j } j ∼ P T are relevant in the sense that they share the components ( S , A , γ, P 0 , P ) of the MDP with our in-trajectory learning problem. However, the expert's reward functions of different tasks are different and are drawn from an unknown reward function distribution. For example, in the stock market case mentioned in the introduction, the experts of different tasks invest in the same stock market but may have different preferences. As standard in meta-learning [41, 42, 43], we assume that the expert's reward function r E of our in-trajectory learning problem is also drawn from the same unknown reward function distribution. Note that the reward functions of the relevant tasks {T j } j ∼ P T are different from r E even if they are drawn from the same unknown reward function distribution.

For each task T j , there is a batch of trajectories and we divide this batch into two sets, i.e., D tr j and D eval j . The training set D tr j only has one trajectory, just as our in-trajectory learning problem, and the evaluation set D eval j has abundant trajectories. Define the loss function on a certain data set D ≜ { ζ v } m v =1 as L ( θ, D ) ≜ -∑ m v =1 ∑ ∞ t =0 γ t log π θ ( a v t | s v t ) . The goal of each task T j is to learn a task-specific adaptation ϕ j using the training set D tr j , such that ϕ j can minimize the test

loss L ( ϕ j , D eval j ) on the evaluation set D eval j . The goal of meta-regularization is to find a meta-prior ¯ θ , from which such task-specific adaptations ϕ j can be adapted to all tasks {T j } j ∼ P T . In specific, meta-regularization [25] proposes a bi-level optimization problem (5). The lower-level problem uses only one trajectory D tr j to find the task-specific adaptation ϕ j such that the meta-regularized loss function L ( ϕ, D tr j ) + λ 2(1 -γ ) || ϕ -¯ θ || 2 is minimized. The upper-level problem is to find a meta-prior ¯ θ such that the corresponding task-specific adaptations { ϕ j } j ∼ P T can minimize the expected loss function L ( ϕ j , D eval j ) over the evaluation sets of all tasks {T i } j ∼ P T .

The lower-level loss function in (5) is the offline version of our in-trajectory loss function (1) (i.e., L ( ϕ, D tr j ) + λ 2(1 -γ ) || ϕ -¯ θ || 2 = ∑ ∞ t =0 L t ( ϕ ; ( s tr t , a tr t )) ) where ( s tr t , a tr t ) ∈ D tr j . Our in-trajectory learning problem can also be regarded as to find a task-specific adaptation. Note that the in-trajectory learning problem is online while the lower-level problem in (5) is offline because the in-trajectory problem is ongoing where we keep observing new state-action pairs. In contrast, the lower-level problem in (5) is based on 'experience" that has already happened. Due to the space limit, we include the algorithm and theoretical guarantees of solving the problem (5) in Appendix C.

5 Experiments

We present three experiments to show the effectiveness of MERIT-IRL. We use four baselines for comparisons. (i) IT-IRL : this method is MERIT-IRL without meta-regularization. (ii) Naive MERITIRL : this method has the meta-regularization term but uses the naive way (in the middle of Figure 1) to update reward. (iii) Naive IT-IRL : this method uses the naive way to update the learned reward and does not have the meta-regularization term. (iv) Hindsight : this method is meta-regularized ML-IRL [5] which can access the complete expert trajectory and uses the standard IRL (visualized in the left of Figure 1) with meta-regularization to update the learned reward. The experiment details are in Appendix D.

5.1 MuJoCo experiment

In this subsection, we consider the target velocity problem for three MuJoCo robots: HalfCheetah, Walker, and Hopper. The target velocity problem is widely used in meta-RL [44] and meta-IRL [45]. In specific, the robots aim to maintain a target velocity in each task and the target velocity of different tasks is different. To test the performance of MERIT-IRL, we use 10 test tasks whose target velocity is randomly between 1 . 5 and 2 . 0 . In the test tasks, there is only one expert trajectory and the state-action pairs of this trajectory are sequentially revealed (to MERIT-IRL, IT-IRL, Naive MERIT-IRL, and Naive IT-IRL) in an online fashion. The baseline Hindsight uses the complete expert trajectory to learn a reward function. The ground truth reward is designed as -| v -v target | (as in [44]) where v is the current robot velocity and v target is the target velocity. To learn the meta-prior ¯ θ , we use 50 relevant tasks whose target velocity is randomly between 0 and 3 .

Figures 2a-2c show the in-trajectory learning performance where the x -axis is the time step t of the expert trajectory and the y -axis is the cumulative reward of the learned policy π t when only the first t steps of the expert trajectory are observed. The x -limit is 1 , 000 because the trajectory length in MuJoCo is 1 , 000 . Note that the baseline 'Hindsight" is not in-trajectory learning since it learns from a complete expert trajectory. For comparison, we use two horizontal lines (close to each other) to show the performance of Hindsight and the expert in the figures. Figure 2a shows that MERIT-IRL achieves similar performance with the expert when only 40% of the complete expert trajectory ( t = 400 ) is observed while IT-IRL can only achieve performance close to the expert after observing more than 90% of the complete expert trajectory ( t = 900 ). This shows the effectiveness of the meta-regularization. Naive MERIT-IRL and Naive IT-IRL fail to imitate the expert even if the complete expert trajectory is observed ( t = 1 , 000 ). This shows the effectiveness of our special design of the reward update. The discussions on Figures 2b and 2c are in Appendix D.2.

Table 1 shows the results after observing the complete expert trajectory. MERIT-IRL performs much better than IT-IRL, Naive MERIT-IRL, and Naive IT-IRL. MERIT-IRL achieves similar performance

(a) HalfCheetah

(b) Walker

(c) Hopper

(d) Stock Market

500

400

0

-600

250

400

-500

-800

500

300

-1000

-1000

1200

-1000

750

200

-1500

1400

1250

100

-1600

1500

-2000

-1800

1750

0

-2000

-2000

-100

0

200

400

600

800

1000

0

200

400

600

800

1000

0

200

400

600

800

1000

0

40

08

120

160

200

240

Observed time steps (t)

Observed time steps (t)

Observed time steps (t)

Observed time steps (t)

MERIT-IRL

IT-IRL

naive MERIT-IRL

naive IT-IRL

Hindsight

Expert

with Hindsight and expert. Note that it is not expected that MERIT-IRL outperforms Hindsight since Hindsight uses the complete expert trajectory to learn.

5.2 Stock market experiment

RL to train a stock trading agent has been widely studied in AI for finance [46, 47, 48]. In this experiment, we use IRL to learn the reward function (i.e., investing preference) of the target investor in a stock market scenario. In specific, we use the real-world data of 30 constituent stocks in Dow Jones Industrial Average from 2021-01-01 to 2022-01-01. We use a benchmark called 'FinRL" [48] to configure the real-world stock data into an MDP environment. The target investor (i.e., expert) has an initial asset of $1 , 000 and trades stocks on every stock market opening day. The stock market opens 252 days between 2021-01-01 and 2022-01-01, and thus the trajectory length is 252. The reward function of the target investor is defined as p 1 -p 2 where p 1 is the investor's profit which is the money earned from trading stocks subtracting the transaction cost, and p 2 models the investor's preference of whether willing to take risks. In specific, p 2 is positive if the investor buys stocks whose turbulence indices are larger than a certain turbulence threshold, and zero otherwise. The value of p 2 depends on the type and amount of the trading stocks. The turbulence thresholds of different investors are different. The turbulence index measures the price fluctuation of a stock. If the turbulence index is high, the corresponding stock has a high fluctuating price and thus is risky to buy [48]. Therefore, an investor unwilling to take risks has a relatively low turbulence threshold. We include experiment details in Appendix D.3. To test performance, we use 10 test tasks whose turbulence thresholds are randomly between 45 and 50 . To learn ¯ θ , we use 50 relevant tasks whose turbulence thresholds are randomly between 30 and 60 .

Figure 2d shows that MERIT-IRL achieves similar cumulative reward with the expert at t = 140 which is less than 60% of the whole trajectory, while the three in-trajectory baselines fail to imitate the expert before the ongoing trajectory terminates. The last row in Table 1 shows that MERIT-IRL achieves similar performance with Hindsight and the expert. More discussions on the results are in Appendix D.3.

5.3 Learning from a shooter's ongoing trajectory

This part presents the experiment of learning from an ongoing shooter trajectory. Following [12], we model the shooter's movement as a navigation problem. We build a simulator in Gazebo (Figure 3a) where the shooter moves from the door (lower left corner) to the red target (upper right corner). The learner observes the ongoing trajectory of the shooter and keeps updating the learned reward and policy. In our case, the complete shooter trajectory has the length of 140 . Figures 3b-3g show our in-trajectory learning performance where the heat maps visualize the learned reward. We normalize the learned reward to [0 , 1] . We can observe that as the ongoing trajectory is expanding, the learned reward function becomes more and more precise to locate the goal area. When t = 40 , we cannot tell

the goal area from the heat map (Figure 3b). However, as the time t grows, we can almost locate the goal area when t = 60 (Figure 3c) and precisely locate the goal area when t = 80 (Figure 3d).

(a) Gazebo environment

0.0

0.2

0.4

0.6

0.8

1.0

(b)

t

= 40

(c)

t

= 60

(d)

t

= 80

(e)

t

= 100

(f)

t

= 120

(g)

t

= 140

(h) Success rate

1.2

MERIT-IRL

1.0

IT-IRL

naive MERIT-IRL

0.8

naive IT-IRL

0.6

0.4

0.2

0.0

0.2

0

20

40

60

08

100

120140

Observed time steps (t)

Figure 3h shows the policy learning performance. Since there is no ground truth reward in this problem, we use 'success rate" to quantify the performance of the learned policy. The success rate is the rate that the learned policy successfully reaches the goal. From 3h, we can see that MERIT-IRL outperforms the other baselines and can achieve 100% success rate when t = 80 (i.e., only observing 57% of the complete trajectory). Note that we do not include Hindsight and Expert in Figure 3 since they both achieve 100% success rate.

6 Conclusion

This paper proposes MERIT-IRL, the first in-trajectory inverse reinforcement learning theoretical framework that learns a reward function while observing an initial portion of a trajectory and keeps updating the learned reward function when extended portions (i.e., new state-action pairs) of the trajectory are observed. Experiments show that MERIT-IRL can imitate the expert from the ongoing expert trajectory before it terminates.

References

P. Abbeel and A. Y. Ng, 'Apprenticeship learning via inverse reinforcement learning,' in International Conference on Machine Learning , pp. 1-8, 2004.

N. D. Ratliff, J. A. Bagnell, and M. A. Zinkevich, 'Maximum margin planning,' in International Conference on Machine Learning , pp. 729-736, 2006.

B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey, 'Maximum entropy inverse reinforcement learning,' in National Conference on Artificial intelligence , pp. 1433-1438, 2008.

B. D. Ziebart, J. A. Bagnell, and A. K. Dey, 'Modeling interaction via the principle of maximum causal entropy,' in International Conference on Machine Learning , pp. 1255-1262, 2010.

S. Zeng, C. Li, A. Garcia, and M. Hong, 'Maximum-likelihood inverse reinforcement learning with finite-time guarantees,' in Advances in Neural Information Processing Systems , 2022.

S. Liu and M. Zhu, 'Distributed inverse constrained reinforcement learning for multi-agent systems,' Advances in Neural Information Processing Systems , vol. 35, pp. 33444-33456, 2022.

D. Ramachandran and E. Amir, 'Bayesian inverse reinforcement learning.,' in International Joint Conference on Artificial Intelligence , pp. 2586-2591, 2007.

A. J. Chan and M. van der Schaar, 'Scalable bayesian inverse reinforcement learning,' in International Conference on Learning Representations , 2021.

N. Rhinehart and K. M. Kitani, 'First-person activity forecasting with online inverse reinforcement learning,' in IEEE International Conference on Computer Vision , pp. 3696-3705, 2017.

S. Arora, P. Doshi, and B. Banerjee, 'Online inverse reinforcement learning under occlusion,' in International Conference on Autonomous Agents and Multiagent Systems , pp. 1170-1178, 2019.

S. Liu and M. Zhu, 'Learning multi-agent behaviors from distributed and streaming demonstrations,' Advances in Neural Information Processing Systems , vol. 36, 2024.

A. Aghalari, N. Morshedlou, M. Marufuzzaman, and D. Carruth, 'Inverse reinforcement learning to assess safety of a workplace under an active shooter incident,' IISE Transactions , vol. 53, no. 12, pp. 1337-1350, 2021.

Q. Sun, X. Gong, and Y.-W. Si, 'Transaction-aware inverse reinforcement learning for trading in stock markets,' Applied Intelligence , vol. 53, no. 23, pp. 28186-28206, 2023.

J. Chang and W. Tu, 'A stock-movement aware approach for discovering investors' personalized preferences in stock markets,' in International Conference on Tools with Artificial Intelligence , pp. 275-280, 2018.

X. Hu, Y. Chen, L. Ren, and Z. Xu, 'Investor preference analysis: An online optimization approach with missing information,' Information Sciences , vol. 633, pp. 27-40, 2023.

Y. Xu, W. Gao, and D. Hsu, 'Receding horizon inverse reinforcement learning,' Advances in Neural Information Processing Systems , vol. 35, pp. 27880-27892, 2022.

G. Swamy, D. Wu, S. Choudhury, D. Bagnell, and S. Wu, 'Inverse reinforcement learning without reinforcement learning,' in International Conference on Machine Learning , pp. 3329933318, 2023.

K. Yan, A. Schwing, and Y.-X. Wang, 'A simple solution for offline imitation from observations and examples with possibly incomplete trajectories,' Advances in Neural Information Processing Systems , 2024.

M. Sun and X. Ma, 'Adversarial imitation learning from incomplete demonstrations,' in International Joint Conference on Artificial Intelligence , pp. 3513-3519, 2019.

D. Xu, F. Zhu, Q. Liu, and P. Zhao, 'Arail: Learning to rank from incomplete demonstrations,' Information Sciences , vol. 565, pp. 422-437, 2021.

J. Ho and S. Ermon, 'Generative adversarial imitation learning,' in Advances in Neural Information Processing Systems , pp. 4572-4580, 2016.

G. Qiao, G. Liu, P. Poupart, and Z. Xu, 'Multi-modal inverse constrained reinforcement learning from a mixture of demonstrations,' Advances in Neural Information Processing Systems , vol. 36, 2024.

S. Liu and M. Zhu, 'Meta inverse constrained reinforcement learning: Convergence guarantee and generalization analysis,' in International Conference on Learning Representations , 2023.

G. Qiao, G. Quan, R. Qu, and G. Liu, 'Modelling competitive behaviors in autonomous driving under generative world model,' in European Conference on Computer Vision , 2024.

A. Rajeswaran, C. Finn, S. M. Kakade, and S. Levine, 'Meta-learning with implicit gradients,' in Advances in Neural Information Processing Systems , pp. 113-124, 2019.

E. Hazan, K. Singh, and C. Zhang, 'Efficient regret minimization in non-convex games,' in International Conference on Machine Learning , pp. 1433-1441, 2017.

N. Hallak, P. Mertikopoulos, and V. Cevher, 'Regret minimization in stochastic non-convex learning via a proximal-gradient approach,' in International Conference on Machine Learning , pp. 4008-4017, 2021.

M. Hong, H.-T. Wai, Z. Wang, and Z. Yang, 'A two-timescale framework for bilevel optimization: Complexity analysis and application to actor-critic,' arXiv preprint arXiv:2007.05170 , 2020.

T. Chen, Y. Sun, Q. Xiao, and W. Yin, 'A single-timescale method for stochastic bilevel optimization,' in International Conference on Artificial Intelligence and Statistics , pp. 24662488, 2022.

F. Schäfer and A. Anandkumar, 'Competitive gradient descent,' Advances in Neural Information Processing Systems , pp. 7625-7635, 2019.

V. S. Varma, J. Veetaseveera, R. Postoyan, and I.-C. Mor˘ arescu, 'Distributed gradient methods to reach a nash equilibrium in potential games,' in IEEE Conference on Decision and Control , pp. 3098-3103, 2021.

S. Boyd and L. Vandenberghe, Convex Optimization . Cambridge university press, 2004.

J. Zhang, P. Xiao, R. Sun, and Z. Luo, 'A single-loop smoothed gradient descent-ascent algorithm for nonconvex-concave min-max problems,' Advances in Neural Information Processing Systems , pp. 7377-7389, 2020.

T. Haarnoja, H. Tang, P. Abbeel, and S. Levine, 'Reinforcement learning with deep energy-based policies,' in International Conference on Machine Learning , pp. 1352-1361, 2017.

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, 'Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,' in International Conference on Machine Learning , pp. 1861-1870, 2018.

S. Cen, C. Cheng, Y. Chen, Y. Wei, and Y. Chi, 'Fast global convergence of natural policy gradient methods with entropy regularization,' Operations Research , vol. 70, no. 4, pp. 25632578, 2022.

Z. Zheng, F. Gao, L. Xue, and J. Yang, 'Federated Q-learning: Linear regret speedup with low communication cost,' arXiv preprint arXiv:2312.15023 , 2023.

H. Gong and M. Wang, 'A duality approach for regret minimization in average-award ergodic markov decision processes,' in Conference on Learning for Dynamics and Control , vol. 120, pp. 862-883, 2020.

Z. Zheng, H. Zhang, and L. Xue, 'Federated Q-learning with reference-advantage decomposition: Almost optimal regret and logarithmic communication cost,' arXiv preprint arXiv:2405.18795 , 2024.

Z. Zheng, H. Zhang, and L. Xue, 'Gap-dependent bounds for Q-learning using referenceadvantage decomposition,' arXiv preprint arXiv:2410.07574 , 2024.

S. Xu and M. Zhu, 'Efficient gradient approximation method for constrained bilevel optimization,' in AAAI Conference on Artificial Intelligence , vol. 37, pp. 12509-12517, 2023.

S. Xu and M. Zhu, 'Meta value learning for fast policy-centric optimal motion planning,' in Robotics science and systems , 2022.

S. Xu and M. Zhu, 'Online constrained meta-learning: provable guarantees for generalization,' Advances in Neural Information Processing Systems , vol. 36, 2024.

C. Finn, P. Abbeel, and S. Levine, 'Model-agnostic meta-learning for fast adaptation of deep networks,' in International Conference on Machine Learning , pp. 1126-1135, 2017.

S. K. Seyed Ghasemipour, S. S. Gu, and R. Zemel, 'Smile: Scalable meta inverse reinforcement learning through context-conditional policies,' in Advances in Neural Information Processing Systems , pp. 7881-7891, 2019.

Y. Deng, F. Bao, Y. Kong, Z. Ren, and Q. Dai, 'Deep direct reinforcement learning for financial signal representation and trading,' IEEE Transactions on Neural Networks and Learning Systems , vol. 28, no. 3, pp. 653-664, 2016.

Z. Zhang, S. Zohren, and S. Roberts, 'Deep reinforcement learning for trading,' The Journal of Financial Data Science , vol. 2, no. 2, pp. 25-40, 2020.

X.-Y. Liu, H. Yang, J. Gao, and C. D. Wang, 'Finrl: Deep reinforcement learning framework to automate trading in quantitative finance,' in ACM International Conference on AI in Finance , pp. 1-9, 2021.

B. D. Ziebart, A. L. Maas, J. A. Bagnell, and A. K. Dey, 'Human behavior modeling with maximum entropy inverse optimal control.,' in AAAI Spring Symposium: Human Behavior Modeling , vol. 92, 2009.

M. Monfort, A. Liu, and B. D. Ziebart, 'Intent prediction and trajectory forecasting via predictive inverse linear-quadratic regulation,' in AAAI Conference on Artificial Intelligence , pp. 36723678, 2015.

S. Gaurav and B. Ziebart, 'Discriminatively learning inverse optimal control models for predicting human intentions,' in International Conference on Autonomous Agents and MultiAgent Systems , pp. 1368-1376, 2019.

W. Krichene, M. Balandat, C. Tomlin, and A. Bayen, 'The hedge algorithm on a continuum,' in International Conference on Machine Learning , pp. 824-832, 2015.

N. Agarwal, A. Gonen, and E. Hazan, 'Learning in non-convex games with an optimization oracle,' in Conference on Learning Theory , pp. 18-29, 2019.

D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon, 'Iq-learn: Inverse soft-q learning for imitation,' Advances in Neural Information Processing Systems , vol. 34, pp. 4028-4039, 2021.

M.-F. Balcan, M. Khodak, and A. Talwalkar, 'Provable guarantees for gradient-based metalearning,' in International Conference on Machine Learning , pp. 424-433, 2019.

K. Xu, E. Ratner, A. Dragan, S. Levine, and C. Finn, 'Learning a prior over intent via metainverse reinforcement learning,' in International Conference on Machine Learning , pp. 69526962, 2019.

T. Hospedales, A. Antoniou, P. Micaelli, and A. Storkey, 'Meta-learning in neural networks: A survey,' IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 44, no. 9, pp. 5149-5169, 2021.

X.-Y. Liu, Z. Xia, J. Rui, J. Gao, H. Yang, M. Zhu, C. Wang, Z. Wang, and J. Guo, 'FinRLMeta: Market environments and benchmarks for data-driven financial reinforcement learning,' Advances in Neural Information Processing Systems , vol. 35, pp. 1835-1849, 2022.

This appendix consists of four parts: related works, proof, meta-regularization algorithm and convergence guarantee, and Experiment details.

A Related works

Applying IRL to predict from ongoing trajectories . Papers [49, 50, 51] use standard IRL to predict goals of incomplete (or ongoing) trajectories. In specific, they first learn the reward function corresponding to each potential goal candidate from complete trajectories in the training phase and then use Bayesian methods to pick the most likely goal candidate of incomplete trajectories in the testing phase. However, these works are not in-trajectory learning since they do not learn a reward function from an incomplete ongoing trajectory.

Online non-convex optimization . This paper casts the in-trajectory learning problem as an online non-convex bi-level optimization problem where at each online iteration, a new state-action pair is input. Current literature on online non-convex optimization has two major categories. The first one is to use the regret originally defined in online convex optimization [52, 53]. However, it assumes to find a global optimal solution of a non-convex optimization problem at each online iteration. Therefore, the second category studies 'local regret" [26, 27] and uses follow-the-leader-based methods to minimize the local regret. However, the follow-the-leader-based methods need to solve a non-convex optimization problem to obtain a near-stationary point at each online iteration, which can be computationally expensive and time-consuming. If the streaming data arrives at a fast speed, the computation at each online iteration may not be finished before the next data arrives. The computational burden of each online iteration can be mitigated by online gradient descent (OGD) methods where we only partially solve the non-convex optimization problem by one-step gradient descent at each online iteration. While OGD is widely studied in online convex optimization, it is rarely studied in online non-convex optimization. [11] uses OGD to quantify the local regret, however, its analysis only holds when the input data is identically independent distributed (i.i.d.). In contrast, the input data in our problem is not i.i.d. In specific, the input data at time t (i.e., ( S E t , A E t ) ) is affected by the input data at last step (i.e., ( S E t -1 , A E t -1 ) ). This temporal correlation between any two consecutive input data makes it difficult to analyze the growth rate of the local regret.

Regularization and meta-learning in IRL . Moreover, the data of in-trajectory learning is extremely lacking since there is only one demonstrated trajectory and this trajectory is not complete during the learning process. The lack of data can easily lead to overfitting and a common way to alleviate this problem is to use regularizers [21, 54]. Inspired by humans' using relevant experience to help the inference, we introduce a novel regularization method called meta-regularization [25, 55]. Compared to the regularizers commonly used in IRL [21, 54], the meta-regularizer provides humanexperience-like prior information which helps recover the reward function from few data. Similar to the meta-initialization method [55, 44] commonly used in IRL [56], meta-regularization provides an initialization that the algorithm starts at. However, more importantly, meta-regularization also provides a regularization term to avoid overfitting.

A.1 Distinction from Maximum-likelihood inverse reinforcement learning (ML-IRL) [5]

Wediscuss our distinctions from ML-IRL from the following three aspects: problem setting, algorithm design, and theoretical analysis.

Distinction in problem setting . We study in-trajectory IRL and formulate an online optimization problem, while ML-IRL studies standard IRL and formulates an offline optimization problem.

Distinctions in algorithm design . ML-IRL and our algorithm both update policy and reward in a single loop. However, we propose a novel reward update mechanism specially designed for the in-trajectory learning case. This special design requires to use the current learned policy to complete the expert trajectory, which gives the algorithm the ability to consider for the future. This special design of reward update is novel compared to ML-IRL.

Distinctions in theoretical analysis . The analysis in our paper is substantially different from that in ML-IRL due to three facts: (1) The input data in our paper is not i.i.d., while the input data in ML-IRL is i.i.d. (2) We solve an online optimization problem, while ML-IRL solves an offline optimization problem. (3) Our analysis holds for continuous state-action space, while the analysis of ML-IRL is

limited to finite state-action space. We now discuss the distinctions in theoretical analysis caused by the three facts in detail.

In our case, the input data is not i.i.d. but temporally correlated, i.e., the input data ( s t , a t ) is affected by the input data ( s t -1 , a t -1 ) at last time step. In contrast, the input data in ML-IRL is i.i.d. sampled from a pre-collected data set. To solve this non i.i.d. issue of the input data, we propose a novel theoretical technique that has three steps (detailed in Subsection 4.2). Step 1: We propose the stationary distribution µ π E and quantify the gradient norm difference between the real distribution P π E t and this stationary distribution µ π E in Proposition 1. Step 2: We quantify the local regret over the stationary distribution in Lemma 2. The benefit of doing this is that the input data can be regarded as i.i.d. sampled from this stationary distribution. Step 3: We combine step 1 and step 2, and quantify the (local) regret over the real distribution, where the data is not i.i.d., in Theorem 1 and Theorem 2. We can see that step 1 and step 3 are to solve the non i.i.d. issue of the input data, so that the corresponding theorem statements (Proposition 1, Theorem 1, and Theorem 2) are novel compared to ML-IRL because ML-IRL does not have this non i.i.d. issue. The only theorem statement relevant to ML-IRL is Lemma 2 in step 2 where we both analyze the algorithm over a stationary distribution, and the data is i.i.d. sampled from the stationary distribution.

However, Lemma 2 in step 2 still has significant distinctions from ML-IRL because Lemma 2 quantifies the local regret in the context of online optimization, while ML-IRL quantifies convergence in the context of offline optimization. First, the objective function is dynamically changing in the online setting because the learner observes a new state-action pair at each online iteration, while the objective function is fixed in ML-IRL. Second, the local regret contains the term L t ( θ t ; ( s E t , a E t )) , however, θ t is computed before the learner knows ( s E t , a E t ) . This makes it more difficult to quantify the local regret because the learner does not know L t when it computes θ t . These two difficulties do not appear in the offline optimization in ML-IRL. To solve these two issues, we need to additionally construct a new time-invariant function ¯ L in Appendix B.4 and quantify the convergence of the new function ¯ L . Then, in order to quantify the local regret of { L t } t ≥ 0 , we need to quantify the difference between the real loss function { L t } t ≥ 0 and the constructed loss function ¯ L .

Moreover, our theoretical analysis holds for continuous state-action space while the theoretical analysis in ML-IRL is limited to finite state-action space. The extension to continuous state-action space brings new difficulties and requires significant novel analysis. In general, the difficulties stem from two aspects: (1) The constants in ML-IRL, e.g., the smoothness constant of the loss function L and the coefficient of convergence rate, include the term |S| × |A| . When the state-action space is continuous, those constants are not finite because |S| × |A| is now infinite. To address this issue, we propose new methods to bound those constants. For example, in order to show that the loss function L is smooth, rather than using ||∇ L ( θ 1 ) -∇ L ( θ 2 ) || to find the smoothness constant as in ML-IRL, we aim to show that ||∇ 2 L ( θ ) || is upper bounded by a constant C L in Lemma A.2 and this constant C L does not rely on |S| × | A | . Given that ||∇ 2 L ( θ ) || ≤ C L , the loss function L is C L -smooth. (2) Since the action space A is finite in ML-IRL, their proved properties of the Q -function Q soft (e.g., Lipschitz continuity, contraction property, monotonic improvement, and smoothness) can be easily extended to the value function V soft by summing over different actions a ∈ A . When the action space becomes continuous, summing over infinitely many different actions does not preserve those properties. Thus we have to propose new methods to prove those propoerties of the value function V soft . In specific, we prove the Lipschitz continuity, contraction property, monotonic improvement, and smoothness of the value function V soft in Claims 3-5 in Appendix B.4.

B Proof

This section provides the proof of all the proposition, lemmas, and theorems in the paper. To start with, we first introduce the expression of soft Q -function and soft Bellman policy.

B.1 Notions

The soft Q -function and soft value function are:

The soft Bellman policy is as follows:

It has been proved [34] that the soft Bellman policy π θ is the optimal solution of the lower-level problem (4). We define J θ ( s ) ≜ E π θ S,A [ ∑ ∞ t =0 γ t r θ ( S t , A t ) | S 0 = s ] as the expected cumulative reward of policy π θ starting from state s and J θ ( s, a ) ≜ E π θ S,A [ ∑ ∞ t =0 γ t r θ ( S t , A t ) | S 0 = s, A 0 = a ] . Lemma 3. We have the gradient ∇ θ log π θ ( a | s ) = E π θ S,A [ ∑ ∞ t =0 γ t ∇ θ r θ ( S t , A t ) | S 0 = s, A 0 = a ] -E π θ S,A [ ∑ ∞ t =0 γ t ∇ θ r θ ( S t , A t ) | S 0 = s ] .

Proof. Define Z θ ( s, a ) ≜ exp( Q soft θ ( s, a )) and Z θ ( s ) ≜ exp( V soft θ ( s )) , therefore Z θ is smooth in θ given that it is a composition of logarithmic, exponential, and linear functions of r θ and r θ is smooth in θ (Assumption 1).

Keep the expansion, we can get ∇ θ log Z θ ( s ) = E π θ S,A [ ∑ ∞ t =0 γ t ∇ θ r θ ( S t , A t ) | S 0 = s ] and similarly we can get ∇ θ log Z θ ( s, a ) = E π θ S,A [ ∑ ∞ t =0 γ t ∇ θ r θ ( S t , A t ) | S 0 = s, A 0 = a ] . Thus we have the gradient ∇ θ log π θ ( a | s ) = ∇ θ log Z θ ( s, a ) -∇ θ log Z θ ( s ) .

B.2 Proof of Lemma 1

Recall that ∑ t i =0 L i ( θ ; ( s E i , a E i )) = -∑ t i =0 γ i log π θ ( a E i | s E i ) . When the dynamics P is deterministic, we have that

where equality ( a ) follows from the proof of Lemma 3.

When the dynamics P is stochastic, we can prove that the above gradient is an unbiased estimate of

where equality ( b ) follows from the fact that because

Since we quantify the local regret in expectation in Theorem 1, this unbiased estimate can be used when the dynamics is stochastic.

B.3 Proof of Proposition 1

From Assumption 2, we know that d TV ( P π E t ( · ) , µ π E ( · )) = 1 2 ∫ s ∈S | P π E t ( s ) -µ π E ( s ) | ds ≤ C M ρ t where the initial state is s 0 . For any state-action pair ( s, a ) ∈ S × A , we know that P π E t ( s, a ) = P π E t ( s ) π E ( a | s ) and µ π E ( s, a ) = µ π E ( s ) π E ( a | s ) . Therefore, we have that:

Claim 1. The trajectory of θ t is bounded, i.e., || θ t -¯ θ || ≤ 2 ¯ C r λ for any t ≥ 0 .

Proof.

where ( a ) follows triangle inequality and ( b ) uses the upper bound of ∇ θ r θ in Assumption 1. Therefore, we have the following relation:

where ( c ) follows the fact that θ 0 = ¯ θ and ( d ) follows the fact that α t ≤ 1 -γ λ .

Recall that the loss function L i ( θ ; ( s E i , a E i )) = -γ i log π θ ( a E i | s E i ) + λγ i 2 || θ -¯ θ || 2 and thus ∇ L i ( θ ; ( s E i , a E i )) = -γ i [ ∇ θ Q soft θ ( s E i , a E i ) - ∇ θ V soft θ ( s E i )] + λγ i ( θ -¯ θ ) . From Lemma 3, we know that ∇ θ V soft θ ( s E i ) = E π θ S,A [ ∑ ∞ t =0 γ t ∇ θ r θ ( S t , A t ) | S 0 = s E i ] and ∇ θ Q soft θ ( s E i , a E i ) = E π θ S,A [ ∑ ∞ t =0 γ t ∇ θ r θ ( S t , A t ) | S 0 = s E i , A 0 = a E i ] . Then, ||∇ θ V soft θ ( s E i ) || ≤ ¯ C r 1 -γ and ||∇ θ Q soft θ ( s E i , a E i ) || ≤ ¯ C r 1 -γ .

Now we can see that

Therefore, we have that

Lemma 4. Suppose Assumptions 1-2 hold, the we have the following for any ( s, a ) ∈ S ×A and any θ 1 , θ 2 , t : ||∇ L t ( θ 1 , ( s, a )) -∇ L t ( θ 2 , ( s, a )) || ≤ C L || θ 1 -θ 2 || and | Q soft θ 1 ,π θ 1 ( s, a ) -Q soft θ 2 ,π θ 2 ( s, a ) | ≤ C Q || θ 1 -θ 2 || , where C L = 2 ˜ C r 1 -γ + 4 ¯ C 3 r (1 -γ ) 4 + λ and C Q = ¯ C r 1 -γ .

Proof. Note that Q soft θ,π θ = Q soft θ and ∇ θ Q soft θ ( s, a ) = E π θ S,A [ ∑ ∞ t =0 γ t ∇ θ r θ ( S t , A t ) | S 0 = s, A 0 = a ] (proof of Lemma 3). Therefore, we have that

Weknowfrom Lemma 1 that ∇ L t ( θ ; ( s E t , a E t )) = -γ t [ ∇ θ Q soft θ ( s E t , a E t ) -∇ θ V soft θ ( s E t )]+ λγ t ( θ -¯ θ ) . To find the smoothness constant of L t , we need to compute the Hessian of L t . First, we have that

Keep the expansion, we can get

Now we take a look at the second term in the above equality:

Therefore, we have that

Similarly, we can get ||∇ 2 θθ V soft θ ( s ) || ≤ ˜ C r 1 -γ + 2 ¯ C 3 r (1 -γ ) 4 . Therefore, we have that

B.4 Proof of Lemma 2

This proof is based on the proof in ML-IRL [5]. The differences are: (i) their proof only holds for finite state-action space while we extend to continuous state-action space; (ii) their analysis is for offline settings while we extend to online settings to quantify the local regret. We first introduce the following claims which serve as building blocks in this subsection.

Proof.

Similarly, we can get that

Claim 3. The soft Bellman operator T soft θ :

is a contraction map with constant γ .

Proof. It has been proved that T soft θ Q is a contraction map with constant γ (Appendix A.2 in [34]). Here we show that T soft θ V is a contraction map with constant γ . Define a norm of V as || V 1 -V 2 || = sup s ∈S | V 1 ( s ) -V 2 ( s ) | and suppose || V 1 -V 2 || = ϵ . Then we have that

Similarly, we can get T soft θ V 1 ( s ) ≥ T soft θ V 2 ( s ) -γϵ . Therefore, ||T soft θ V 1 - T soft θ V 2 || ≤ γϵ = γ || V 1 -V 2 || .

Claim 4. It holds that Q soft θ t ,π t +1 ( s, a ) ≥ T soft θ t ( Q soft θ t ,π t )( s, a ) and V soft θ t ,π t +1 ( s ) ≥ T soft θ t ( V soft θ t ,π t )( s ) for any ( s, a ) .

Proof.

where ( i ) follow equations (2)-(3) in [35] and ( ii ) follows policy improvement theorem (Theorem 4 in [34]). Similarly, we can get that

Claim 5. The following holds for any ( s, a ) ∈ S × A and θ 1 , θ 2 , t :

Proof. Note that ∇ θ V soft θ ( s ) = E π θ S,A [ ∑ ∞ t =0 γ t ∇ θ r θ ( S t , A t ) | S 0 = s ] (proof of Lemma 3), therefore

We first show the convergence of π t in Algorithm 1. For any ( s, a ) ∈ S × A , we know that

Now we take a look at the term | Q soft θ t ,π t ( s, a ) -Q soft θ t ( s, a ) | :

where ( a ) follows Lemma 2 and Claim 2, ( b ) follows the fact that π θ is the optimal solution, ( c ) follows Claim 4, ( d ) follows the fact that Q soft θ t -1 is a fixed point of T soft θ t -1 (Theorem 2 in [34]), and ( e ) follows Claim 3.

Similarly we can bound the term | V soft θ t ,π t ( s ) -V soft θ t ( s ) | :

where ( f ) follows Claim 2 and claim 5 and ( e ) follows Claim 4.

Now we take a look at the term || θ t -θ t -1 || :

where ( h ) follows claim 1. Recall that

| log π t +1 ( a | s ) -log π θ t ( a | s ) | ≤ | Q soft θ t ,π t ( s, a ) -Q soft θ t ( s, a ) | + | V soft θ t ,π t ( s ) -V soft θ t ( s ) | .

Summing from i = 0 to t , we get

where ( i ) follows (8), ¯ D 1 = 16 C Q ¯ C r (1 -γ ) 2 , and ¯ D 2 = 1 1 -γ ( | Q soft θ 0 ( s, a ) -Q soft θ 0 ,π 0 ( s, a ) | + | V soft θ 0 ,π 0 ( s ) -V soft θ 0 ( s ) | - | Q soft θ t ( s, a ) -Q soft θ t ,π t ( s, a ) | - | V soft θ t ,π t ( s ) -V soft θ t ( s ) | ) . Therefore, we can see that

Define the loss function ¯ L ( θ ) ≜ E ( S,A ) ∼ µ π E ( · , · ) [ -log π θ ( A | S ) + λ 2 || θ -¯ θ || 2 ] , then we can see that E ( S E i ,A E i ) ∼ µ π E ( · , · ) [ L i ( θ ; ( S E i , A E i ))] = γ i ¯ L ( θ ) . Moreover, we have that

where ( j ) follows Lemma 3 and ( k ) follows Claim 1. Therefore, we have that

where ( l ′ ) follows the fact that a bounded variable X ∈ [ -a, a ] has bounded variance at most a 2 . Now we take a look at the term g t -1 -γ t +1 1 -γ ∇ ¯ L ( θ t ) :

From equation (64) in [5], we know that

where C d is the product of the area of S and the area of A . Therefore, we can get that

From (7) and (8), we know that

and similarly we can see that

Telescoping from i = 0 to t -1 , we get

where ( l ) follows (12) and ( m ) follows (13). Similarly, we can see that

Now, we start to quantify the local regret:

where ( n ) follows (8), ( o ) follows (10), ( p ) follows (15). Therefore, we have that

where D 1 = 8 ¯ C 2 r (2 -γ ) 2 (1 -γ ) 4 . The ( q ) follows (11). Note that to achieve this local regret rate, we actually need to take an extra expectation over the dynamics P because we need to roll out π t to formulate g t . Here, we omit the expectation over the dynamics.

B.5 Proof of Theorem 1

We know that

where ( a ) follows (18) and (11).

Therefore,

where ( s ) follows (19) and Proposition 1. Note that to achieve this local regret rate, we actually need to take an extra expectation over the dynamics P because we need to roll out π t to formulate g t . Here, we omit the expectation over the dynamics.

B.6 Proof of Theorem 2

Suppose the expert reward function r E and the parameterized reward function r θ are both linear, i.e., r E = θ ⊤ E ϕ and r θ = θ ⊤ ϕ , where ϕ : S × A → R n + is an n -dimensional feature vector such that || ϕ ( s, a ) || ≤ ¯ C r for any ( s, a ) ∈ S × A . The proof follows the similar idea with that of Theorem 1 where follow two-step process: step (i) quantifies the regret under the stationary distribution µ π E ( · , · ) and step (ii) quantifies the difference between the correlated distribution P π E t and the stationary distribution µ π E . We start our proof with the following claim:

Claim 6. If the parameterized reward function r θ is linear, the function ¯ L ( θ ) is λ -strongly convex for any θ .

Proof. Recall that ¯ L ( θ ) = E ( ¯ S, ¯ A ) ∼ µ π E ( · , · ) [ -log π θ ( ¯ A | ¯ S )] + λ 2 || θ -¯ θ || 2 . Define ¯ L ( θ ; ( S, A )) ≜ -log π θ ( ¯ A | ¯ S ) + λ 2 || θ -¯ θ || 2 , from Lemma 3, we can see that ∇ ¯ L ( θ ; ( ¯ S, ¯ A )) = E π θ S,A [ ∞ ∑ i =0 γ i ϕ ( S i , A i ) ∣ ∣ ∣ ∣ S 0 = ¯ S ] -E π θ S,A [ ∞ ∑ i =0 γ i ϕ ( S i , A i ) ∣ ∣ ∣ ∣ S 0 = ¯ S, A 0 = ¯ A ] + λ ( θ -¯ θ ) . Therefore, we have that ∇ 2 θθ ¯ L ( θ ; ( ¯ S, ¯ A )) , = ∇ θ E π θ S,A [ ∞ ∑ i =0 γ i ϕ ( S i , A i ) ∣ ∣ ∣ ∣ S 0 = ¯ S ] -∇ θ E π θ S,A [ ∞ ∑ i =0 γ i ϕ ( S i , A i ) ∣ ∣ ∣ ∣ S 0 = ¯ S, A 0 = ¯ A ] + λ. Now, we take a look at ∇ θ E π θ S,A [ ∞ ∑ i =0 γ i ϕ ( S i , A i ) ∣ ∣ ∣ ∣ S 0 = ¯ S, A 0 = ¯ A ] = ∇ θ { ϕ ( ¯ S, ¯ A ) + ∫ s 1 ∈S P ( s 1 | ¯ S, ¯ A ) ∫ a t +1 ∈A π θ ( a 1 | s 1 ) E π θ S,A [ ∞ ∑ i =1 γ i ϕ ( S i , A i ) ∣ ∣ ∣ ∣ S 1 = s 1 , A 1 = a 1 ] da 1 ds 1 } , = ∫ s 1 ∈S P ( s 1 | ¯ S, ¯ A ) ∫ a 1 ∈A { ∇ θ π θ ( a 1 | s 1 ) · E π θ S,A [ ∞ ∑ i =1 γ i ϕ ( S i , A i ) ∣ ∣ ∣ ∣ S 1 = s 1 , A 1 = a 1 ] + π θ ( a 1 | s 1 ) · ∇ θ E π θ S,A [ ∞ ∑ i =1 γ i ϕ ( S i , A i ) ∣ ∣ ∣ ∣ S 1 = s 1 , A 1 = a 1 ]} da 1 ds 1 . Keep the expansion, we can see that ∞

Thus we have that

where ( a ) follows Lemma 3, X ≜ E π θ S,A [ ∑ ∞ i =0 γ i ϕ ( S i , A i ) | S 0 = ¯ S, A 0 = A ] is a random vector, EX ≜ E π θ S,A [ ∑ ∞ i =0 γ i ϕ ( S i , A i ) | S 0 = ¯ S ] is the expectation of X over the action distribution, and Cov ( X ) is the covariance matrix of the random vector X with itself, which is always positive definite because the policy π θ is always stochastic.

Therefore, we can see that where ( b ) follows (20).

Step (i) . We first quantify the regret under the stationary distribution µ π E ( · , · ) . Suppose θ ∗ ∈ arg min ¯ L ( θ ) = arg min 1 -γ T 1 -γ ¯ L ( θ ) = arg min E ( S E t ,A E t ) ∼ µ π E [ ∑ T -1 t =0 L t ( θ ; ( S E t , A E t ))] .

where ( c ) follows (8)

Since ¯ L ( θ ) is λ -strongly convex, we have that

where ( d ) follows (21), and ( e ) follows Claim 1 which shows that the trajectory of θ t is bounded and thus there is a positive constant ˆ C such that || θ t -θ ∗ || ≤ ˆ C .

Telescoping (22) from t = 0 to t = T -1 , we can see that

where ( f ) follows (16), and ¯ D 4 = 1 -γ -λα 0 2 α 0 E ( S E t ,A E t ) ∼ µ π E ( · , · ) [ || θ t -θ ∗ || 2 ] -1 -γ 2 α T -1 (1 -γ T ) E ( S E t ,A E t ) ∼ µ π E ( · , · ) [ || θ T -θ ∗ || 2 ] + 4 ¯ C r C d ˆ C (1 -γ ) 2 [ | Q soft θ 0 ,π 0 ( S, A ) -Q soft θ 0 ( S, A ) | -| Q soft θ T ,π T ( S, A ) -Q soft θ T ( S, A ) | ] .

Step (ii) . We now quantify the difference between the stationary distribution µ π E ( · , · ) and the correlated distribution P π E t ( · , · ) . Note that || θ t || is bounded (proved in Claim 1), the soft Bellman policy π θ is continuous in θ and ( s, a ) , and we assume that the state-action space is bounded, there is a positive constant C π such that || log π θ t ( a | s ) || ≤ C π for any ( s, a ) ∈ S × A . We start with the following relation:

Therefore, we have that

where ( f ) follows (23) and (24), D 4 = 2 C π C M 1 -γρ + ¯ D 4 , and D 5 = 32 C Q C d ˆ C ¯ C 2 r λ (1 -γ ) 3 .

C Meta-Regularization Algorithm and Convergence Guarantee

To solve problem (5), we use a double-loop method, i.e., we first solve the lower-level problem for K iterations to get an approximate ϕ j,K of ϕ j and then use the approximate ϕ j,K to solve the upper-level problem. We do not use a single-loop method to solve (5) because we need to get the task-specific adaptation ϕ j for each task T j . The single-loop method only partially solves the lower-level problem by one-step gradient descent and thus the obtained parameter can be far away from ϕ j .

We roll out the policy π ϕ from s tr t to get a trajectory s ϕ 0 , a ϕ 0 , · · · where s ϕ 0 = s tr t and approximate the lower-level gradient by g ϕ = ∑ ∞ t =0 γ t ∇ ϕ r ϕ ( s ϕ t , a ϕ t ) -∑ ∞ t =0 γ t ∇ ϕ r ϕ ( s tr t , a tr t ) + λ 1 -γ ( ϕ -¯ θ ) . We update ϕ j,k +1 = ϕ j,k -β k g ϕ j,k for K times to get ϕ j,K where β k is the step size and ϕ j,k is the parameter at time k , and use ϕ j,K to calculate the approximate of the hyper-gradient d d ¯ θ L ( ϕ j , D eval j ) .

Lemma 6. The hyper-gradient (i.e., gradient of the upper-level problem in (5) ) is d d ¯ θ L ( ϕ j , D eval j ) = [ I + 1 -γ λ ∇ 2 ϕϕ L ( ϕ j , D tr j ) ] -1 ∇ ϕ L ( ϕ j , D eval j ) , where

The |D eval j | is the number of trajectories in D eval j , ( s v t , a v t ) ∈ D eval j , and e is an extra term whose expression can be found in Appendix C.2.

To approximate the hyper-gradient, we first roll out π ϕ j,K to get two trajectories s ϕ j,K 0 , a ϕ j,K 0 , · · · and ¯ s ϕ j,K 0 , ¯ a ϕ j,K 0 , · · · where s ϕ j,K 0 is drawn from P 0 and ¯ s ϕ j,K 0 = s tr 0 . We estimate ∇ ϕ L ( ϕ j , D eval j ) via ¯ ∇ ϕ L ( ϕ j,K , D eval j ) = |D eval j | · ∑ ∞ t =0 γ t ∇ ϕ r ϕ j,K ( s ϕ j,K t , a ϕ j,K t ) -∑ |D eval j | v =1 ∑ ∞ t =0 γ t ∇ ϕ r ϕ j,K ( s v t , a v t ) and estimate the term ∇ 2 ϕϕ L ( ϕ j , D tr j ) via ¯ ∇ 2 ϕϕ L ( ϕ j,K , D tr j ) = ∑ ∞ t =0 γ t ∇ 2 ϕϕ r ϕ j,K (¯ s ϕ j,K t , ¯ a ϕ j,K t ) -∑ ∞ t =0 γ t ∇ 2 ϕϕ r ϕ j,K ( s tr t , a tr t ) . Therefore, we can approximate the hyper-gradient term d d ¯ θ L ( ϕ j , D eval j ) via h j = [ I + 1 -γ λ ¯ ∇ 2 ϕϕ L ( ϕ j,K , D tr j )] -1 ¯ ∇ ϕ L ( ϕ j,K , D eval j ) . We omit the extra term e in the approximate h j and its impact on the convergence can be bounded (proved in Appendix C.4).

To solve the upper-level problem in (5), at each iteration n , we sample a batch of B tasks, and compute the task-specific adaptation ϕ j,K and the hyper-gradient h j for each task. The update law to solve the upper-level problem is: ¯ θ n +1 = ¯ θ n -τ n B ∑ B j =1 h j where τ n is the step size. Note that the time index k is for the lower-level problem and n is for the upper-level problem.

Algorithm 2 Meta-regularization

Input

: Initialized meta-prior ¯ θ 0 and task-specific adaptation ϕ j, 0 : Learned prior ¯ θ N

Output

1: for n = 0 , 1 , · · · , N -1 do

2: Sample a batch of B tasks {T j } B j =1 ∼ P T

3: for each task T j do

4: for k = 0 , 1 , · · · , K -1 do

5: Compute the soft Bellman policy π ϕ j,k via soft Q-learning or soft actor-critic

6: Roll out the policy π ϕ j,k to get a trajectory s ϕ j,k 0 , a ϕ j,k 0 , · · ·

7: Compute the gradient g ϕ j,k = ∑ ∞ t =0 γ t ∇ ϕ r ϕ j,k ( s ϕ j,k t , a ϕ j,k t ) -∑ ∞ t =0 γ t ∇ ϕ r ϕ j,k ( s tr t , a tr t ) + λ 1 -γ ( ϕ j,k -¯ θ n )

8: Update ϕ j,k +1 = ϕ j,k -β k g ϕ j,k

9: end for

10: Compute the soft Bellman policy π ϕ j,K via soft Q-learning or soft actor-critic

11: Roll out the policy π ϕ j,K to get two trajectories s ϕ j,K 0 , a ϕ j,K 0 , · · · and ¯ s ϕ j,K 0 , ¯ a ϕ j,K 0 , · · ·

12: Compute the hyper-gradient h j = [ I + 1 -γ λ ¯ ∇ 2 ϕϕ L ( ϕ j,K , D tr j )] -1 ¯ ∇ ϕ L ( ϕ j,K , D eval j )

13: end for

14: Update ¯ θ n +1 = ¯ θ n -τ n B ∑ B j =1 h j

15: end for

Lemma 7 (Convergence of the lower-level problem) . Suppose Assumptions 1-2 hold and λ ≥ C L 2 + η where η ∈ (0 , C L 2 ) . Let β k = 1 -γ η ( k +1) , then we have

Assumption 3. The parameterized reward function r θ has bounded third-order gradient, i.e., ||∇ 3 θθθ r θ ( s, a ) || ≤ ˆ C r for any ( s, a ) where ˆ C r is a positive constant.

Theorem 3 (Convergence of the upper-level problem) . Suppose Assumption 3 and the condition in Lemma 7 hold. Let τ n = ( n +1) -1 / 2 and define F ( ¯ θ ) as E j ∼ P T [ L ( ϕ j , D eval j )] under the meta-prior ¯ θ . Then we have the following convergence:

and the expression of C 1 can be found in (27) .

C.1 Proof of Lemma 5

The proof is similar to that of Lemma 1. We first prove the case of deterministic dynamics and the corresponding result can be an unbiased estimate in cases of stochastic dynamics (proved in Subsection B.2).

where ( a ) follows the proof of Lemma 3.

C.2 Proof of Lemma 6

Since the lower-level problem of (5) is unconstrained and ϕ i is the optimal solution, we know that

We further take derivative of both sides in (25) with respect to ¯ θ and we get that

Therefore, we have that

Similar to Lemma 5, we can know that

where ( s v t , a v t ) ∈ ζ v and ζ v ∈ D eval j .

Since there is usually abundant data in D eval j and S 0 ∼ P 0 , we can reformulate (26) as the following:

Claim 7. The second-order information ∇ 2 ϕϕ Q soft ϕ ( s, a ) = ∆( s, a ) + E π ϕ S,A [ ∑ ∞ t =0 γ t Cov ( S t ) | S 0 = s, A 0 = a ] and ∇ 2 ϕϕ V soft ϕ ( s ) = ∆( s ) + E π ϕ S,A [ ∑ ∞ t =0 γ t Cov ( S t ) | S 0 = s ] where ∆( s, a ) = E π ϕ S,A [ ∑ ∞ t =0 γ t ∇ 2 ϕϕ r ϕ ( S t , A t ) | S 0 = s, A 0 = a ] , ∆( s ) = E π ϕ S,A [ ∑ ∞ t =0 γ t ∇ 2 ϕϕ r ϕ ( S t , A t ) | S 0 = s ] , and Cov ( s ) ≜ ∫ a ∈A π ϕ ( a | s )[∆( s, a ) -∆( s )]∆( s ) da is the covariance matrix of ∆( s, · ) at state s .

The proof of Claim 7 follows the proof of Lemma 4

Now we take a look at the term ∇ 2 ϕϕ L ( ϕ j , D tr j ) and consider the case of deterministic dynamics:

where ( a ) follows Claim 7. As proved in Subsection B.2, we can still use this expression in the case of stochastic dynamics. We define e ≜ E π ϕ j S,A [ ∑ ∞ t =0 γ t Cov ( S t ) | S 0 = s tr 0 ] . However, e is intractable to compute because it requires to compute COV ( S t ) at every visited state. To approximate the covariance matrix COV ( S t ) , we need to empirically roll out the policy π ϕ j from S t for enough times to get enough samples. We need to do these policy roll-outs at every state S t . For example, suppose we roll out the policy π ϕ j ten times to get ten samples at each state S t . Empirically, we need to do these roll-outs 10 × ¯ T where ¯ T is a very large integer that we regard as infinity because the trajectory horizon is infinite. This is intractable because we need to roll out the policy for too many times. Moreover, we cannot guarantee that we can approximate COV ( S t ) well given that we only use ten samples to approximate it.

C.3 Proof of Lemma 7

1 1 -γ C L where ( a ) follows (6). Therefore, the lower-level objective function in (8) is C L 1 -γ -smooth. Moreover, since λ ≥ C L 2 + η , then ||∇ 2 ϕϕ L ( ϕ, D tr j ) || ≤ C L -2 η 2(1 -γ ) . Therefore, the lower-level objective function in (8) is 2 η 1 -γ -strongly convex. Following the standard result for strongly-convex and smooth stochastic optimization, we can reach the result in Lemma 7.

C.4 Proof of Theorem 3

In this proof, we first bound the hyper-gradient approximation error (i.e., || d d ¯ θ L ( ϕ j , D eval j ) -h j || ) and then prove the convergence. Define ¯ h j ≜ [ I + 1 -γ λ ¯ ∇ 2 ϕϕ L ( ϕ j , D tr j )] -1 ∇ ϕ L ( ϕ j , D eval j ) where ¯ ∇ 2 ϕϕ L ( ϕ j , D tr j ) ≜ E π ϕ j S,A [ ∑ ∞ t =0 γ t ∇ 2 ϕϕ r ϕ j ( S t , A t ) | S 0 = s tr 0 ] -∑ ∞ t =0 γ t ∇ 2 ϕϕ r ϕ j ( s tr t , a tr t ) . Therefore, we have that

where ( a ) follows the fact that || I + 1 -γ λ ∇ 2 ϕϕ L ( ϕ j , D tr j ) || ≥ 1 - || 1 -γ λ ∇ 2 ϕϕ L ( ϕ j , D tr j ) || ≥ 1 -C L -2 η 2 λ given that ||∇ 2 ϕϕ L ( ϕ, D tr j ) || ≤ C L -2 η 2(1 -γ ) (proved in the proof of Lemma 7). Therefore || [ I + 1 -γ λ ∇ 2 ϕϕ L ( ϕ j , D tr j )] -1 || ≤ 2 λ 2 λ +2 η -C L . Similarly, we can bound || [ I + 1 -γ λ ¯ ∇ 2 ϕϕ L ( ϕ j , D tr j )] -1 || ≤ λ λ -2 ˜ C r given that || ¯ ∇ 2 ϕϕ L ( ϕ j , D tr j ) || ≤ 2 ˜ C r 1 -γ . Note that λ > C L 2 and C L > ˜ C r (proved in (6)), therefore C 1 is a positive constant.

Now, we bound the term || h j -¯ h j || . We define ∆ ϕ j = I + 1 -γ λ ¯ ∇ 2 ϕϕ L ( ϕ j , D tr j ) and ∆ ϕ j = I + 1 -γ λ ¯ ∇ 2 ϕϕ L ( ϕ j,K , D tr j ) . Thus we have || ∆ -1 ϕ j || ≤ λ λ -2 ˜ C r (follows (27)) and similarly || ∆ -1 ϕ j,K || ≤ λ λ -2 ˜ C r . Therefore,

where ||∇ 3 ϕϕϕ L ( ϕ j , D tr j ) || ≤ ˜ C L . The expression of ˜ C L can be derived following the proof of Lemma 4 by notifying Assumption 3. The ( b ) holds because (i) ||∇ 2 ϕϕ L ( ϕ, D eval j ) || = |D eval j | |D tr j | ||∇ 2 ϕϕ L ( ϕ, D tr j ) || ≤ |D eval j | · C l -2 η 2(1 -γ ) and (ii) ||∇ ϕ L ( ϕ j , D eval j ) || ≤ |D eval j | · 2 ¯ C r 1 -γ (see the expression of ∇ ϕ L ( ϕ j , D eval j ) in Lemma 5).

where ( c ) follows (27)-(28). Define F ( ¯ θ ) as E j ∼ P T [ L ( ϕ j , D eval j )] under the meta-prior ¯ θ . Note that || h j || ≤ || ∆ -1 ϕ j,K || · ||∇ ϕ L ( ϕ j,K , D eval j ) || ≤ λ λ -2 ˜ C r · |D eval j | · 2 ¯ C r 1 -γ . Therefore,

where |D eval | ≜ sup i ∼ P T {|D eval j |} and ( d ) follows (29).

D Experiment details

The code was running on a laptop whose processor is AMD Ryzen 7 4700U with Radeon Graphics, 2.00GHz, and the installed RAM is 20.0GB. The operating system is Ubuntu 18 . 04 . We use a neural network to parameterize the learned reward function. The neural network has two hidden layers where each hidden layer has 64 neurons. The activation functions are respectively ReLU and Tanh.

D.1 Baselines

Here we provide the update rule for each baseline. Given a learned reward function r θ , the policy updates of the four baselines are the same with that of MERIT-IRL, i.e., one-step policy iteration. The difference is the reward update.

IT-IRL : IT-IRL is MERIT-IRL without the meta-regularization term. Therefore, the reward update of IT-IRL is θ t +1 = θ t -α t g ′ t where g ′ t = ∑ ∞ i =0 γ i ∇ θ r θ t ( s ′ i , a ′ i ) -∑ ∞ i =0 γ i ∇ θ r θ t ( s ′′ i , a ′′ i ) . Recall from Subsection 4.1 that { ( s ′ i , a ′ i ) } i ≥ 0 is generated by the learned policy π θ t starting from the initial state s ′ 0 = s E 0 , and { ( s ′′ i , a ′′ i ) } i ≥ 0 is generated by the learned policy π θ t starting from ( s ′′ t , a ′′ t ) where ( s ′′ i , a ′′ i ) = ( s E i , a E i ) for 0 ≤ i ≤ t .

Naive MERIT-IRL : This method has the meta-regularization term, however, it uses the naive way (depicted in the middle of Figure 1) to update the reward function. In specific, it only compares the partial expert trajectory { s E i , a E i } t i =0 and partial learner trajectory { s ′ i , a ′ i } t i =0 . Therefore, the reward update of Naive MERIT-IRL is θ t +1 = θ t -α t g ′′ t where g ′′ t = ∑ t i =0 γ i ∇ θ r θ t ( s ′ i , a ′ i ) -∑ t i =0 γ i ∇ θ r θ t ( s E i , a E i ) + λ (1 -γ t +1 ) 1 -γ ( θ -¯ θ ) .

Naive IT-IRL : This method does not have the meta-regularization term and uses the naive way to update the reward function. Therefore, the reward update of Naive IT-IRL is θ t +1 = θ t -α t g ′′′ t where g ′′′ t = ∑ t i =0 γ i ∇ θ r θ t ( s ′ i , a ′ i ) -∑ t i =0 γ i ∇ θ r θ t ( s E i , a E i ) .

Hindsight : This method is a standard IRL method with the meta-regularization term where the complete expert trajectory { s E i , a E i } i ≥ 0 and the complete learner trajectory { s ′ i , a ′ i } i ≥ 0 are compared to update the reward function. Therefore, the reward update of Hindsight is θ t +1 = θ t -α t g ′′′′ t where g ′′′′ t = ∑ ∞ i =0 γ i ∇ θ r θ t ( s ′ i , a ′ i ) -∑ ∞ i =0 γ i ∇ θ r θ t ( s E i , a E i ) + λ (1 -γ t +1 ) 1 -γ ( θ -¯ θ ) .

D.2 MuJoCo

D.2.1 Walker

Figure 2b shows that MERIT can achieve similar performance with the expert after t = 600 while the other three in-trajectory learning baselines fail to imitate the expert before the ongoing trajectory terminates. Note that the naive methods (i.e., Naive MERIT-IRL and Naive IT-IRL) have much smaller improvement from t = 0 compared to MERIT-IRL and IT-IRL. The reason is that the naive reward update method is flawed. Intuitively, the reward update mechanism of these two baselines are myopic as explained in Subsection 4.1. Theoretically, the gradients g ′′ t of Naive MERIT-IRL and g ′′′ t

of Naive IT-IRL are biased estimate of (4) even if π t approaches π θ t since (4) includes the trajectory suffix ( i > t ) terms while g ′′ t and g ′′′ t only include the trajectory prefix ( i ≤ t ) terms.

MERIT-IRL performs much better than IT-IRL. The reason is that the meta-regularization term restricts the learned reward parameter within a certain neighborhood of the meta-prior ¯ θ (proved in Appendix B.3). Given that ¯ θ is trained over a family of relevant tasks, it is expected that the actual reward function parameter of our task shall be 'close" to ¯ θ [25, 44, 57], i.e., inside this neighborhood. Therefore, MERIT-IRL can efficiently learn the expert's reward function. On the contrary, IT-IRL does not have the meta-prior ¯ θ as a guidance and thus has to search over the whole parameter space, which is extremely difficult to learn the expert's reward function when the data is lacking. Note that MERIT-IRL and Naive MERIT-IRL have better initial performance than IT-IRL and Naive IT-IRL since MERIT-IRL and Naive MERIT-IRL starts at the meta-prior ¯ θ while IT-IRL and Naive IT-IRL initializes randomly.

D.2.2 Hopper

Figure 2c shows that MERIT can achieve similar performance with the expert after t = 500 while the other three in-trajectory learning baselines fail to imitate the expert before the ongoing trajectory terminates.

D.3 Stock Market

We use the real-world data of 30 constitute stocks in Dow Jones Industrial Average from 2021-01-01 to 2022-01-01. The 30 stocks are respectively: 'AXP', 'AMGN', 'AAPL', 'BA', 'CAT', 'CSCO', 'CVX', 'GS', 'HD', 'HON', 'IBM', 'INTC', 'JNJ', 'KO', 'JPM', 'MCD', 'MMM', 'MRK', 'MSFT', 'NKE', 'PG', 'TRV', 'UNH', 'CRM', 'VZ', 'V', 'WBA', 'WMT', 'DIS', 'DOW'.

The state of the stock market MDP is the perception of the stock market, including the open/close price of each stock, the current asset, and some technical indices [48]. The action has the same dimension as the number of stocks where each dimension represents the amount of buying/selling the corresponding stock. The detailed formulation of the MDP can be found in FinRL [48, 58].

The turbulence index is a technical index of stock market and is included as a dimension of the state [48, 58]. The function p 2 is defined as the amount of buying the stocks whose turbulence index is larger than the turbulence threshold. Therefore, the more the target investor buys the stocks whose turbulence index is larger than the turbulence threshold, the larger p 2 will be and thus the smaller reward the target investor will receive.

Discussion on the experiment results . In Figure 2d, MERIT-IRL can achieve the similar cumulative reward with the expert when only the first 60% of the trajectory is observed while IT-IRL can achieve performance close to the expert after t = 220 . This shows that the meta-regularization can help imitate the expert faster. In contrast, Naive MERIT-IRL and Naive IT-IRL barely improves because the naive reward update method is flawed. Intuitively, the reward update mechanism of these two baselines are myopic as explained in Subsection 4.1. Theoretically, the gradients g ′′ t of Naive MERITIRL and g ′′′ t of Naive IT-IRL are biased estimate of (4) even if π t approaches π θ t since (4) includes the trajectory suffix ( i > t ) terms while g ′′ t and g ′′′ t only include the trajectory prefix ( i ≤ t ) terms.

The last row in Table 1 shows the final results of the algorithms. We can see that MERIT-IRL achieves much better performance than the other in-trajectory learning baselines (i.e., IT-IRL, Naive MERIT-IRL, and Naive IT-IRL). MERIT-IRL achieves comparable performance with Hindsight and the expert. Note that it is not expected that MERIT-IRL outperforms Hindsight since Hindsight has the complete expert trajectory to learn.

E Potential negative societal impact

Since MERIT-IRL can infer the reward function of the expert, potential negative societal impact may occur when the learner is malicious. Take the stock market experiment as an example, private information like preferences or habits of the investors may be leaked by using MERIT-IRL. To avoid this situation, the investors needs to take additional strategies such as protecting its investment data from unsecure resources.

F Limitations

From the objective (2), we can see that the goal of MERIT-IRL is to align with the expert demonstration, i.e., finding a reward function such that its corresponding policy makes the expert trajectory most likely. An ideal case is that we can also directly quantify the reward learning performance and study the reward identifiability issue. Thus, a future work is to study the reward identifiability issue in the context of in-trajectory IRL.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract summarizes our contributions, and the introduction has a 'contribution statement" part which elaborates our contributions and mentions the consistency with theoretical and experiment results. The assumptions and limitations are included in 'theoretical analysis" (Subsection 4.2) and Appendix F.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation is mentioned in Appendix F. Under the assumptions, i.e., Assumptions 1 and 2, we have a justification of either the assumption is widely used in literature or the assumption can be satisfied by real scenarios.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We clearly state our assumptions in Assumptions 1 and 2. The theoretical statements are also clearly stated in Subsection 4.2 and the correct proof is included in Appendix B. All the assumptions and theoretical statements are numbered and cross-referenced.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: This paper provides pseudocode and elaborates each step of the algorithm in Subsection 4.1 and Appendix 4.3. We also include the experiment details in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is submitted in the supplementary materials and we include a document in the code folder to describe how to run the code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Appendix D includes the experiment details, and we also submit the code.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: In the experiment results (i.e., Table 1, Table ?? , Table ?? ), we include both the mean and standard deviation. In Figure 2, we also plot the mean and standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We include the type of operating system, CPU, and RAM used in the first paragraph in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper follows NeurIPS Code of Ethnics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer:[Yes]

Justification: We discuss the potential negative societal impact in Appendix E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks because we do not have data nor model to release.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use the SAC code from a Github repository, and we clearly state it at the top of that code script.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The submitted code can be considered as an asset, and we provide a file along with the code to document the code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
In this paper, a novel framework for performing Inverse Reinforcement Learning (IRL) from an ongoing trajectory, i.e., to learn a reward function that induces an optimal policy that best explains the expert's demonstrations sequentially, without waiting to having observed the expert's trajectory entirely. Authors propose an online learning algorithm for solving the problem, and this algorithm has the added feature of meta-regularization. Both a theoretical and empirical analysis of the algorithm are provided to validate the results.

STRENGTHS:
- The framework of learning from ongoing trajectories is interesting and finds some applications.

WEAKNESSES:
- There is no clear discussion and intuition of what kind of reward function is extracted by the algorithm and what it can be used for. For instance, various works about identifiability or the feasible set concern what we can do with the extracted rewards, because in most cases we cannot just extract a single reward explaining the demonstrations. However, this work does not analyse this fact.
- The presentation of the contents and contributions is poor.

QUESTIONS:
- What reward are you extracting among all the rewards that explain the observed demonstrations? What can it be used for?
- Since in RL what matters is the cumulative sum of rewards, how can you be sure that you extract a meaningful reward even though you have not observed the expert's policy for the entire horizon?

LIMITATIONS:
The authors have adequately addressed the limitations and potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-18

comments: SUMMARY:
The paper "Learn Incrementally from An Ongoing Trajectory: A Provable In-Trajectory Inverse Reinforcement Learning Framework" proposes an innovative approach to inverse reinforcement learning. The authors introduce an online learning algorithm to address the IRL problem with incomplete expert demonstrations. A thorough theoretical analysis guarantees convergence, and a comprehensive experimental section validates the approach.

STRENGTHS:
1.Very interesting problem and strong theoretical analysis.


2.The paper introduces a practical algorithm that appears easy to reproduce and yields good results.

WEAKNESSES:
1.In the experiments, I don’t understand why different algorithms have different initial points? When t=0, are you using a same random reward function or the prior-meta reward function to train your policy?

QUESTIONS:
1. Can the author explain why comparing the suffixes \(\{s_i', a_i'\}\) and \(\{s_i'', a_i''\}\) encourages the reward function to predict the future? Since you are completing the incomplete trajectory using the same policy, can reward function can learn something even if both are generated by the same policy.


2. Just wondering how you complete the incomplete trajectories in MuJoCo using saved states, since the state in MuJoCo only includes partial information about the simulator.

LIMITATIONS:
The authors address the limitations of this work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider a new problem setting, called in-trajectory IRL, where a reward function and a corresponding policy need to be learned from an ongoing trajectory. The authors propose a novel reward update mechanism specially designed for this scenario and incorporate a meta-regularization strategy to embed prior knowledge and avoid overfitting. Theoretical analysis is provided for the convergence of the algorithm towards stationary points (and global under linear reward parametrization). MuJoCo benchmark and a real-world stock market example are used for experiments.\n\nSTRENGTHS:\n1.\tThe topic is novel to IRL community. The new problem setting of learning the reward function and policy on the fly aligns with many real-world scenarios. In this new situation, standard IRL algorithms are not suitable anymore.\n2.\tThe special reward update design of predicting the future trajectory is reasonable and interesting for this ongoing trajectory setting.\n3.\tThe meta-regularization is novel to IRL, which helps to better find a reward from limited data.\n4.\tThe theoretical analysis is solid and tackles the issue of non i.i.d. input data. The theoretical distinction from literature is well clarified. The experiment includes real-world stock market data.\n\nWEAKNESSES:\n1.\tAssumption 1 assumes that the parameterized reward is smooth, which can be a strong assumption, especially for neural networks with non-smooth activation functions such as ReLU.\n2.\tThe meta-regularization part seems to be important to the performance. However, the meta-learning needs to pre-collect data for many related tasks, which can be difficult in real world.\n\nQUESTIONS:\nHow the authors collect data for the meta-regularization in the experiment?\n\nOther questions please refer to the Weaknesses.\n\nLIMITATIONS:\nThe authors have discussed the limitations in Appendix F.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}