Meta Inverse Constrained Reinforcement Learning: Convergence Guarantee and Generalization Analysis
Authors: Shicheng Liu
Minghui Zhu
Venue: ICLR 2024 poster (2024)
Keywords: inverse reinforcement learning
meta learning
Primary area: reinforcement learning

Abstract: This paper considers the problem of learning the reward function and constraints of an expert from few demonstrations. This problem can be considered as a meta-learning problem where we first learn meta-priors over reward functions and constraints from other distinct but related tasks and then adapt the learned meta-priors to new tasks from only few expert demonstrations. We formulate a bi-level optimization problem where the upper level aims to learn a meta-prior over reward functions and the lower level is to learn a meta-prior over constraints. We propose a novel algorithm to solve this problem and formally guarantee that the algorithm reaches the set of $\epsilon$-stationary points at the iteration complexity $O(\frac{1}{\epsilon^2})$. We also quantify the generalization error to an arbitrary new task. Experiments are used to validate that the learned meta-priors can adapt to new tasks with good performance from only few demonstrations.

Published as a conference paper at ICLR 2024
META INVERSE CONSTRAINED REINFORCEMENT LEARNING: CONVERGENCE GUARANTEE AND GEN- ERALIZATION ANALYSIS
Shicheng Liu & Minghui Zhu Department of Electrical Engineering Pennsylvania State University University Park, PA 16802, USA {s [sf15539, 9, muz1 1 6 6pepsu.eeee edu
ABSTRACT
This paper considers the problem of learning the reward function and constraints an expert from few demonstrations. This problem can be considered as a meta- learning problem where first learn meta-priors straints from other distinct but related tasks and then adapt the learned meta-priors new tasks from only few expert demonstrations. We formulate a bi-level op- timization problem where the upper level aims to learn a meta-prior over reward functions and the lower level is to learn meta-prior over constraints. We propose a novel algorithm to solve this problem and formally guarantee that the algorithm reaches the set of e-stationary points at the iteration complexity 0 (²))) We also quantify the generalization error to an arbitrary new task. Experiments are used to validate that the learned meta-priors can adapt to new tasks with good perfor- mance from only few demonstrations.
reward functions and con-
INTRODUCTION
Inverse reinforcement learning (IRL) has been receiving substantial research efforts due to its effec- tiveness to recover reward function from expert's demonstrations that can well explain the expert's s behavior. In practical applications, however, constraints are ubiquitous and a reward function com- bined with set of constraints can better explain complicated behaviors than single reward function (Malik et al. 2021). Therefore, inverse constrained reinforcement learning (ICRL) is proposed to learn constraints from expert's 's demonstrations. Current state-of-the-arts on IRL (Fu et al. 2018 Imani & Ghoreishi] 2021) and ICRL (Scobee & Sastry 2019) can either learn a reward function in unconstrained environments infer constraints with access to the ground truth reward but cannot infer both. To solve this challenge, distributed ICRL (Liu & Zhu 2022) is proposed to learn both the reward function and constraints of the expert. In this paper, follow the definition of ICRL in (Liu & Zhu 2022), which means learning both the reward function and constraints of the expert.
While the aforementioned literature can recover the reward function and constraints for single tasks, they typically need large amounts of expert demonstrations (Yu et al. 2019). When it comes to multiple related tasks that share common structural patterns, e.g., navigating different locations in common environment (Xu et al., 2019). it could be expensive and inefficient to collect enough demonstrations for each task and then learn the corresponding reward function and constraints sep- arately. Meta-learning (Rajeswaran et al. 2019) has a potential to learn the reward functions and constraints efficiently from few demonstrations. It can exploit the structural similarity of a group of related tasks by learning meta-prite meta-priors. The Iearned meta- meta-priors allow for rapid adaptation to new related tasks from only limited data. Therefore, it motivates us to leverage meta-learning to infer the reward functions and constraints of the experts in tasks from only few demonstrations.
Related works. IRL (Abbeel & Ng 2004: Ziebart et al. 2008 Ziebart 2010) and ICRL (Scobee & Sastry/ 2019, Malik et al. 2021 Liu & Zhu 2022) have shown great success in recovering the reward function and constraints from expert's 's demonstrations. However, when comes to multiple related tasks, they all require large amounts of demonstrations for each task. Meta-learning (Finn)

---

Published as a conference paper at ICLR 2024
et al. 2017 Rajeswaran et al., 2019 Xu & Zhu 2023b) provides way to learn from limited data by learning the common structural patterns (i.e., meta- meta--prors) of the related tasks and then optimizing for rapid adaptation to unseen tasks from only few data. It has achieved state-of-the of-the-art performance in few-shot regression, classification (Finn et al.) 2017). 7). and reinforcement learning (Fallah et al. 2021a Xu & Zhu 2022). Recently, several meta IRI. algorithms
proposed to recover reward functions from few demonstrations. In specific, (Yu et al. 2018 Xu et al. 2019) propose to learn a reward parameter initialization that can be adapted to new tasks via only one or few gradient descent step(s). YYu et al. 2019, Seyed Ghasemipour et al., 2019) propose to learn context-cone cantext-condttionll model that, given new task, can encode the task and output the corresponding reward parameters.
However, the existing works on meta IRL have two limitations. (i) They do not explicitly deal with constraints. Existing meta-learning algorithms can directly compute the gradient of the meta objective (i.e., syper-gradient) when only reward functions are learned (Xu et al.) (019), but cannot compute the hyper-gradien: when we also need to deal with constraints. (ii) They do not theoretically guarantee the proposed algorithms" convergence, and more importantly adaptation performance (i.e., generalization error) to new tasks. This paper proposes the first theoretical framework and thereby algorithm that can learn the reward function and constraints of new task from only few demonstrations by first learning meta-priors over reward functions and constraints. While there is no meta IRL theoretical work, there are several theoretical works on other meta-learning problems. We discuss our distinctions from other related meta-learning theoretical works in Appendix AA.14
Contribution statement. Our contributions threefold. First, we extend ICRL (Liu & Zhu 2022)
to meta-learning setting where we learn meta-priors over reward functions and constraints in order adapt to new related tasks from few demonstrations. We formulate a novel bi-level optimization problem to solve it. Second, we propose a novel "meta inverse constrained reinforcement learning" (M-ICRL) algorithm, that can efficiently compute the hyper-gradiems to solve the problem. Third, we provide the iteration complexity ( ) of the algorithm reaching the set of e-stationary points. More importantly, we quantify the generalization error to an arbitrary new task. It is shown that the generalization error can be sufficiently small if the new task is "close" to the training tasks.
2
PROBLEM FORMULATION
This section introduces the definition of single task and then formulates the meta-learning problem.
2.1 SINGLE TASK: ICRL
In our problem, a single task T; is ICRL problem (Liu & Zhu 2022) where a learner aims to learn the reward function and constraints of an expert from the expert's demonstrated trajectories. The expert's decision making is based on constrained Markov decision process (CMDP). The task Tt's CMDP (S, A, Po. P. rıs C; by) is defined via state set S, action set A, discount factor Y, and initial state distribution Po- The probability of state transition to s' from s by taking action is P(s'|s, a). The reward and cost functions of the expert are ri, G S x A - R. A trajectory of the CMDP is a state-action sequence Ç = 5o, 0o, $1; $1;01, ... and we use Px denote the trajectory distribution generated by an arbitrary policy π where the initial state is drawn from Po. Define (T) A ECMP. EE "ra(se, at)] as the expected cumulative reward under the policy and Jes (T) A Ec-P, D ('i(s,, at)] as the expected cumulative cost. The expert's policy wants to maximize (T) subject to Jc, (T) S by where b; is a pre-defined budget. The expert can roll out Ty to demonstrate a set of D, trajectories D. = } where Ç1 = sg. ag. s4 af.af.". af.....
A learner observes D; and aims to use parameterized models re and e, with parameters and w to leam the expert's reward function rú and cost function c; by solving the following ICRL problem:
min 8
L2(0, w" (0)), s.t. w" (0) = arg min G (w: 0).
(1)
The upper-level problem aims to learn a reward function re that can minimize the expected negative log-likelihood (0, -Ec-P, I2 7º log n (dzls:) where Tw. is the constrained soft Bellman policy (see the expression in Appendix A.2 (Liu & Zhu 2022; 2024) under the reward function r'g and cost function cu. The constrained soft Bellman policy
extension of soft Bellman policy (Ziebart et al., 2010 Zhou et al. 2017) to CMDPs. The soft Bellman policy is widely used in soft Q-laarning (Haamoja et 2017) and soft actor-critic (Haarnoja et al. 2018).
2

---

Published as a conference paper at ICLR 2024
The lower-level function (2(w; f) - max, (T) + Jr, - (x) + Je. (TT) can be regarded
as an RL problem which aims to find the policy that maximizes the mntrroy-regularing deward-minus-onn
cumulative (T) Jo__ (=)) where (T) & E -i= 7 log π [a)]s:)] is the causal entropy. Note that the likelihood L, is defined on the expert's trajectory distribution Ps. while (x)) is defined on the trajectory distribution of its current policy и. The last term Jz.. (T:) in
(i.e., (x) +
G2 is constant w.r.t It is proved (Liu & Zhu (022) that the constrained soft Bellman policy is the optimal policy of the RL problem in Gi(w;; 0), i.e., Tuxll arg max, (T)++ (T) - J,_ (T)) The lower-level problem min Gi(w; 0) uses adversarial learning to find a cost function cu that makes the best policy (i.e., kw:0) perform the worst and the last term Je.. (I,) penalizes cost functions where the expert has high cumulative cost. We discuss the formulation of (I) in detail in Appendix (
Since problem ① is defined in expectation but the learner only observes D., in practice, the learner solves an empirical problem defined on D.. Given trajectory Ç³ = sy ag. ..., we define fc(5²) & 2i=00" ag) as the empirical cumulative cost. Then the empirical problem the learner solves is ming L,(0, (O), Di), s.t. (0) = arg min,, G,(w; Di) where the (, w, D,) _ - D. E Ei 7a log Tw:a (a) and G: 0, D.) _ max; (TT) + (T) (T) + D. ΣSn Ja (c)).
2.2 MULTIPLE TASKS: M-ICRI
ICRL in (T) can successfully recover the reward and cost functions of the expert (Liu & Zhu 2022 2024). However, it typically needs large data set for each task when it comes to multiple related tasks. To learn the reward and cost functions from few demonstrations, we leverage meta-learning which optimizes for the ability to learn efficiently on new tasks. It is typically assumed in meta- learning that there is a set of m training tasks {T: which share the CMDP. The difference of tasks is that each task T; has its own reward function Ti, cost function c; and budget 6;. The goal of meta-learning is to optimize for meta-priors of reward and cost functions over the m training tasks (T.) such that the reward and cost functions adapted from the learned meta-priors have good performance on new tasks even if the new tasks only have limited data.
When comes meta-learning, two of the state-of- the-t-rstsrsssss are model agnostic meta-learning (MAML) (Finn et al.) 2017) and meta-learning with implicit gradients (iMAML) (Rajeswaran et al. 2019. MAML is simple and widely implemented in RL (Fallah et al., 2021a) and IRL (Yu et al, 2019 while iMAML shows better empirical performance (Rajeswaran et al. 2019) at the expense of heavier computation in the lower level since MAML only needs one gradient descent but iMAML needs to fully solve optimization problem in the lower level. In M-ICRL, aim propose problem formulation that utilizes the advantages of both methods.
The proposed problem formulation (2)(3) has bi-level structure (Ji et al. 2021 Xu & Zhu 2023a)
where we learn the reward meta-prios preaa-prior in the upper level and the cost meta-prior in the lower level.
1 m
min Bw
Σ
L. L, (p),
m
A
s.t.
ny (vP: w) arg G,( + 2 ll wll?
where yo: 0 a 00 L, m (0, w)) is the task-specific reward adaptation and v/ (px w) is the task-specific cost adaptation. Note that problem - reduces to the ICRL problem (T if we only consider one task and do not perform meta-learning on reward parameter 0 nor cost parameter w, and x= O. In this case, we do not have task-specific adaptations (p:,, nf). Problem - reduces to MAMI if we only do meta-learning on the reward parameter 0 and do O. this case, we only have the task-
i.e., m 1. a
not perform meta-learning on the cost parameter w. i.e,, specific reward adaptation yp, Problem (2 (3) reduces to iMAML (cxplained in Appendix (A.4) if we only do meta-learning on the cost parameter w and do not perform meta-learning on the reward parameter 0, i.e., a 0. In this case, only have the task-specific cost adaptation n
Problem 2 can reduce to the MAML that only learns and the iMAML that only learns W. utilizes iMAML but does not suffer from the extra computation burden usually caused by iMAML because ICRL in ( is already a bi-level formulation and we need to fully solve the lower-level problem anyway. We do not use iMAMI. for 0 because this will lead to "three-level" problem.
3

---

Published as a conference paper at ICLR 2024
3 THE PROPOSED ALGORITHM
This section proposes a novel algorithm that solves problem 2 Following (Fallhh et al. 2020). we partition the data set D; of each training task T: into three subsets D. Di and D. with sizes DJr, D and D respectively. The training set D.r with limited data is used to compute the task- specific adaptations (a and n7." the evaluation set D) with abundant data is used to compute the hyper-gradicna gradicnts (i.e., the gradients of the upper-level loss function in (2) with respect to 0 and w), and the set D. is used to compute the second-order terms in the apprr-grddients.
For an arbitrary data set 'D with size D, we solve the empirical version (i.e., arg min, Gi (p:: D) à 1117-w|1 of the lower-level problem 3) using (K - -ttep gradient descent i w. D, 1
i (PP w. D, k) - (i, (Pp.. w, D, k): sP,, D) ((fxx w, D, k) w]] where is the step size.
We then use (P, w, D, K) as an approximation of rjī (fps w. u, D) - arg min. G:(n: 9s. D)+ + 2
2 ||
wil². We provide the expressions of all the gradients, including V, V,,C, in Appendix A.3
Algorithm Meta inverse constrained reinforcement learning (M-ICRL)
Input: Initialized reward meta-prior (()) and cost meta-prior w()), task batch size B, step size a Output: Learned meta-prior (()) and cost meta-prior (n)
1: for = 0, 1.. do
2: Samples batch of training tasks (Ti)ee with size B
3: for all Ti do
4: Samples the demonstration set DJ to compute n ((((n) DJ, K) and (.) 0(n)
o (.(())), i (0(n), ((),, D. DF)
5: Samples the demonstration sets Deval and DD
(n),, D. Deval D.!)) 6: Va Va,i Hyper- gradient (0(n) (w)))))))(())))) ((n),
7: end for
8: 8(n + 1) = ((n)
a(n) B
B
Vo.
w(n = (m) (w)
a(n) Σ Vus
B
9:
end for
At each iteration n in Algorithm|] 1 the learner samples tasks from the set of training tasks (Ti)m For each sampled training task T; the learner first uses the training set D to compute f); and the task-specific reward adaptation @ (line 4). Then the learner uses the training set D. evaluation set Drval and D. to compute the hyper- gradients Ve., and Va, (line 6). Finally, the learners utilizes stochastic gradient descent to update the reward and cost meta-priors (line 8)
The computation of the hyper-gradients critical to Algorithm I the following context, we first identify the difficulties of computing the syper-gradients and then provide our solutions.
3.1 CHALLENGES OF COMPUTING THE HYPER-GRADIENT
The arper-gradients 0T.( (PWW)) and
(1 of problem 2 are hard to compute.
86
8w
Take
0Le(pp.n) (p(w))
84 as an example (the derivation of the hyper-gradients is in Appendix A.5):
aLi n (p)WW)
T
000 L. mt (0 (0.w))
8²
V mo (P))))
00
2 9in (p), V G. ( (): - ( (p.)))
(i) The second-order term a0 (0, n (0, w)) in the first bracket is intractable to compute since it requires to compute Vionz (0, w) which needs to calculate the gradient of an Sreeseeof--essiaae term [V2
8²
on G.(n) w):4:) A7 -1.
(ii) The inverse-of of Hessian 2 my (n (p),w): + A in the second bracket is expensive to compute, especially when use neural networks as parameterized models.
(iii) We cannot get 17,' but only its approximation since the optimization oracle is not guaranteed to find the exact optimal solution. This will cause errors when compute the hyper-gradients.
4

---

Published as a conference paper at ICLR 2024
3.2 MAIN IDEA TO SOLVE THE CHALLENGES
Solution to challenge (i) (Algorithm approximate the hyper-gradients:
The learner uses sampled data sets Dj, Deval, and Dj to
0²
90,,
I
062
f% (0, D.), D.)
D!) Ao,i.
(4)
I. -
w. (0, f, w.. D.), Do. + A.it Owd0
(5)
where Do,i and Aw.s are partial gradients of L,( ff (Gi, w, Deval), D.val) with respect to [
and w. While the second-order terms (i.e., agz
0² L, and 8² L.)) in the hyper-gradient (4 ()
are directly computed in many meta-learning works (Finn et al.) 2017; Xu et al.] 2019). m
our case, it is prohibitively hard to compute. To compute the second-order terms, we need to
calculate Vini (0,w, DF) which needs to calculate the gradient of an inverse-of-Hessian term
since Voff(0,w,
(0, w. D -V (fi (0, w. DF); 0, (V, Gr (f (0, Dy); 0, D + A1]-1 and
Wi (0, w. D = A|V?
my (ng (0, w, D 0, D + XT| (derived in Appendix A.5 To tackle this challenge, we use the first-order approximation to approximate the products:
82 002
0 L, 8D0. (0 + w D.))
(0, n (0, w. D.), Do.
20 00
(0 8D0.; M 8 0 w, D.T,, D.!)
(6)
08
8
1 θ
(0 i7 (0, w. D.), Do. ~
L, + 8D0.s, (0 + dd w. D!))
Owo0
26
Ow
ow 1. - 8. (0- 8... w, D. D?!
where 5 is perturbation magnitude. In Algorithm the learner first approximates the partial gradi- ents Ag.; and A (line in Algorithm 2 and then computes the first-order approximation (lines
2 4 in Algorithm 2). The output of Algorithm 2
is the approximation of the hyper-gradients 45)
Solution to challenge (ii) (Algorithm with respect to ip; and w are respectively:
The partial gradients of Le(ps,, fe (1ix w. Dival), D
Do.: V.. Lo i (Pi, w, ) Deval) - v pn Gi (Pi, w, De ); s0,, Drval).
[]] + 2 m G. (17 (6,, w. )i G.,, Di - 'v, , L (p: f. (p), w, Drmal,, ), Deval),
(8)
A A|V 2 (1 (p,, w. Deval) 3, D. - Le (p:, fi (13s w, Deval), D.
(9)
Note that the partial gradients (8) contain [V V2 ( (p,,w, Dival); ); @ú Devall ) V Li f (&e, w, D.val), D where the inverse-of Hessian term is expensive compute. Therefore, solve the following optimization problem instead:
min x.
AI+ my G. (,,, D. p.,, D.val)
x- V L, (P:, Dival), DT x. (10)
It is obvious that the optimal solution of problem 10) is [V] V V G2 (9es w. Deval); p., 1
V V,,L((p fz w. (Dvval,, Deval).
In Algorithm the learner solves the problem 10 for IK 1) step gradient descent to get an
approximation r r(K) of the optimal solution of problem 10 (line in Algorithm 3 and then use xx (K) to help approximate the partial gradients (9) (lines in Algorithm 3).
K). In practice, we use this approximation substitute for 7) (0, w, D in ()) Similarly, we use
Solution to challenge (iii). We cannot get 1j, (0, w. D but an approximation nn,0,,,, D
the approximation (fx; w, Deval, K) to substitute for 7i, (fi, w. Deval
To quantify the
5

---

Published as a conference paper at ICLR 2024
approximation error of the hyper-gradiend -4 caused by [[ '., K) o (-.,, ->l. exploit the Lipschitz continuity of the hyper-gradient, with respect to 17. In specific, we first prove the w.L.. 7) in Appendix AA.7 and then prove the Lipschitz continuity of the first-order approximation (6 (6 w.r.t. 1/ in Appendix A.8 Then, we can the Lipschitz continuity of the hyper-gradients
Lipschitz continuity of the partial gradients
w.r.t. 17.
Algorithm 2 Hyper-graadien((f
Pi; D.,,, D D!)
Input: Reward parameter 0, cost parameter w, task-specific reward adaptation Φi, training set D evaluation set D the data set Dj to compute the second-order terms, perturbation
Output: Approximate hyper-gradients Âg, - aV AW,i aV
1: Ag,i, Au,i paartial-grrddeeee 4;, Deval Deval)
2: Aptii Aw+,;
+ d DHr D!t)
Âø_,t Aw_, Partiaa-gradient 8D0 w., D. D.)
4: Vo,5 = (do+. - (2-,) Va,a = (Aw+.s ()/()))2))
Algorithm Partial-gradien((,.10 w, Di.Dy,
Input: Reward parameter 0, cost parameter w, data set D1, data set D2, step size 3 Output: Approximate partial gradients ás,i, Âw.i
1: Compute f (0, wi D1, , K) and initialize x0)
2: for k = 0.1,
K do
<<k + 1)
=x(k)
[AT V (,) (i,
4: end for
5: ÅØ,1 (OL P), D2) V D1, 6), (1){(K))
6: Aw.i Xx(K)
THEORETICAL ANALYSIS
This section has two parts: the first part provides the convergence guarantee of Algorithm and the second part quantifies the generalization error to an arbitrary new task.
4.1 CONVERGENCE GUARANTEE
Compared to the standard stochastic gradient descent, the main difficulty of guarantecing the vergence Algorithm [ lies in quantifying the approximation eиror of the hyper-gradients. The approximation error comes from three aspects which correspond to the three challenges in Subsec- tion 3.1 (i) We cannot obtain the exact optimal solution ri (:,::) of the lower-level problem but an approximation i K). (ii) We cannot compute the inverse-of Hessian term [A]+ + V
off-sssiaa but use an iterative method to approximate the product [X/ + V G.J -V,,,, in Algorithm This will result in the error between the approximate partial gradients oo,; Âw. (i.e., the output of Algo- rithm 3 and the real partial gradients A9,5, Aw, i in ()-
(9. (iii) We use the first-order approximation
(6)-(7 Algorithm to approximate the real syyper-gradient 44 (5).
In what follows, we first sequentially quantify the three approximation errors identified in the last paragraph and then analyze the convergence of Algorithm We start with the following assumption.
Assumption 1. (i) The parameterized reward function satisfies (ra a)) < Cr. I|V a)ll < C., and 1 ragra(s (s, a)ll < Cr for any (s, S x A and any 0 where Cr. Cr, and č, positive constants; (û) The parameterized cost function Cy has similar properties with positive constants C. C., and C.: (iii) The third and fourth order gradients of the reward and cost functions with respect their parameters are bounded for any (s, a.) and (0.w).

---

Published as a conference paper at ICLR 2024
Note that Assumptions (i) and (ii) standard in RL (Wan al. 2019: Kumar et al. 2019
Zhang et al. 2020; Zheng et al. 2023) and IRL (Guan et al. 2021). Assumption ( is needed to exploit the Lipschitz continuity of the nyper-gradients. Moreover, the bounded third order gradients of the parameterized model in Assumption (iii) are commonly assumed in meta RI (Fallah et al.)
2021a).
Approximation error (i). Proved in Appendix A.6 the function G; and its empirical approximation
G, using any data set are Cvp G-Smooth for any task T; where Cva n0 G is a positive constant whose
expression is in Appendix A.6 Therefore, the lower-level objective function in (3) becomes (A (vz,,c)-stroon convex and (A + Cv,,,c)-smon
if > Cv3, Choosing T the standard result for strongly-convex and smooth objective functions (Nesterov
and following
2003 Boyd &
Vandenberghe, 2004), have (| K) o7 ( x O((Cv,,. G/A)*).
Approximation error (ii). We next quantify the approximation error of the partial- gradients.
Lemma 1. Suppose Assumption 7 holds and let 3 X where À Ova G. then the outputs of
Algorithm 3 satisfy::
Ov G
G) +
Cez G x)
)
|| Ao <0
Cvz G C G
|| A..il s
A
A
Lemma [ shows that the approximation error of the partial gradients diminishes we increase the iteration numbers of solving the lower-level problem and in Algorithm 3
Approximation error (iii). With the approximation error of the partial gradients, we can quantify the approximation error of the hyper-gradients.
Lemma 2. Suppose the conditions in Lemmal 7 hold, then the oulputs of Algorithm 2 satisfy:
C G Cvz
G
|| Vo. g0.1|| 50
)
X +
R
+
66)
X
A| - aV - gwill S
guill
Cvz K
+
Cvz
G
R + +6)
Lemma 2 indicates that the approximation error of the hyper-gradients can be arbitrarily small if we solve the lower-level problem 33) for enough iterations, run Algorithm 3 for enough iterations, and choose sufficiently small 5.
To reason about the convergence of Algorithm [ we introduce e-approximate first order stationary point (e-FOSP) (Fallah al. 2020): the variable (0,w) is e-FOSP if En Ve mf (FP, w)]l
S E where Li 7/ (4P w)) - I à Li(Pe oi ())) (I DE (P),
30
Theorem 1 (Convergence of Algorithm
Suppose the conditions Lemma 2 hold.
[0, 11 and a<n) (n+1)p where ā E (0, 4- p (2.1), and Do and Cj are positive constants De
whose existence is proved in Appendices A.8 and A.9 respectively. Then Algorithm 7 reaches the set of e-FOPP i.e.,
E
1=1
se+o Cve G
Ov G
+ min { D Jm J
after at most N min(Cr,Coo) Bed
iterations. The expressions of the positive constants C₁ and C2 are
in Appendix A.9
Theorem shows that Algorithm I reaches the set of e-FOSP at the iteration complexity 0(
Moreover, reduce Fll Em VTal]] and reach the of e-FOSP within fewer iterations in
7

---

Published as a conference paper at ICLR 2024
Algorithm 1 have the following choices: (i) increase the iteration number K of solving the lower-level problem (3) and the iteration number K in Algorithm 3 (ii) choose smaller 8 in the first-order approximations 6 (7): (iii) sample larger size B of training tasks at each iteration n in Algorithm (iv) choose larger size Dÿ of training data of each training task Tí.
4.2
GENERALIZATION ANALYSIS
The goal of meta-learning is to learn good meta-prions such that the reward and cost functions adapted from the learned meta-prion can have good performance on new tasks. Theorem [ shows that Algorithm| can find meta-priors (0, ()) such that the average loss function of the m training tasks can reach the set of c-FOSP. However, it does not provide insights into how the task-specific reward and cost adaptations (fm+1. +1 (fm+11,0 (. D 1m)))), adapted from the learned meta-priors (0,w), perform on an arbitrary new task Tm+1 where Dm+1 is the small data set of the new task Tm+1- Given that the loss function in problem is the negative oog-likeihood function L, use (0, w))==p... 1 (fmt1 1,0 (m)+1) as the metric to reason about the performance of the task-specific adaptations (fm+1 1, fm+1 (9m+1, 1,, mm 1)) on arbitrary new task Tm
We start our analysis with the definition of stationary state-action distribution. For given policy m, the corresponding stationary state-action distribution is μ² (s, a) _ (1 - Ei=o "TP, where P. (s, a) is the probability of policy π visiting (s, a) at time t. We then define the distance between two tasks T; and T, as "u=",,,,")) A -sEs SaEA (8, n² (8, a) |ads.. Recall that n, is the expert's policy in task Tr.
Remark on the definition of the task distance. While it seems natural the reward functions and the distance between the cost functions to define the distance between dif- ferent tasks, this kind of definition can cause ambiguity because different reward and cost functions may result in the same task. For example, in an unconstrained environment, multiplying the reward function by constant does not change the task because this will lead to the same optimal policy.
the distance between
Proposition 1. For any new task Tm+1 1 and any parameters (f,w). the following relation holds: EE (L),O,)) -VLm+1(0,w) S (d E==1""', .)(===))-
Theorem 2. For an arbitrary new task Tm+1, the task-specific reward and cost adaptations (Pm+1, (0++1(9m44) u. Dm+1)) adapted from the learned meta meta-priors (0, ()))) have the property:
E V LL (0,w)lo p. : 1. 10 Don
<0 E d(Hc " mm+1) + dl
Σ utth (
Theorem shows that if the new task's 's stationary state-action distribution is sufficiently close the training tasks', the task-specifir adaptations (fm+1, f1 are near- cear-stationary.
Theorem 3. If the learned meta-priors (0, w) of Algorithm satisfy ElL Ex Lx(p,, nf (Fss x))] ming... 1 Dih Lilyi, ni (P),w)) S c where G. a 00 L,, (, nú (0, W)), then holds that m
E[Lm+1 (Pmm+111@mmm11mm 1m (6m+1, 1. D Dm+1))l min L (0,w),
0,00
se+ 0
1 Σ
d( p mm + d(
1
Σ
Theorem 3 shows that if the learned meta- piiioapppoosss/eoo (0, w) are e-optimal and the new task is close to the training tasks, the task-specific adaptations (Pm+1, im++) are near-optimal.
If the reward and cost functions are linear, we have the following stronger results:
Theorem 4. If the expert's reward and cost functions and the parameterized reward and cost
functions re, Cu are linear, we have that (i) El]Jrmzi (Tam=119041)) Jrmei (Tm+1)l] < O(e +
Em ((("",115m+1))
Em u=: "..t,pmmm (f]m+))); )); (ii) El|Jemts (T (1:m111mm+4)) Jemen (mm-1)))
O(e m Em d( (m==+)) + dl En 11= ,p=++)))))))
8

---

Published as a conference paper at ICLR 2024
Theorem 4 shows that (i) the cumulative reward difference and (ii) the cumulative cost difference between the adapted policy #6m+1166..1 and the expert's policy #m+1 on an arbitrary new task Tm+11 can be sufficiently small the new task is close to the training tasks.
EXPERIMENT
This section includes two classes experiments to validate the effectiveness of M-ICRL. The first experiment is conducted on physical drone and the second experiment is conducted in Mujoco. Due to space limit, the experiment details are included in Appendix B
5.1 DRONE NAVIGATION WITH OBSTACLES
We conduct navigation experiment on an AR. Drone 2.0 (Fig- ure [ where the drone (in the yellow box) needs navigate to the destination (in the green box) while avoiding collision with the obstacles (in the red box). We use an indoor motion capture system "Vicon" to record the trajectories of the drone. For dif- ferent tasks, we vary the locations the goal and the obstacles. Given that there is ground truth reward in this experiment, use two metrics "constraint violation rate" (CVR) and "success rate" (SR) where CVR is the percentage of the learned policy col- liding with any obstacle and SR is the percentage of the learned policy reaching the destination and avoiding obstacles. We use 50 training tasks and 10 test tasks where each test task has only one demonstration. We use three baselines for comparisons: ICRL (Liu Zhu 2022) which does have meta-priors and directly learns from one demonstration without meta- priors, (pRL(pre) which naively pre-trains -eeaa-priors by maximizing the likelihood across all the demonstrations of the training tasks, Meta-IRL (Xu al., (019) which only learns reward meta-prior using MAML. We include the experiment results The experiment details are included in Appendix
Figure 1: Drone navigation
the second row Table [
5.2 MUJOCO EXPERIMENT
We also conduct three experiments in Mujoco: Swimmer, HalfCheetah, and Walker. Given that Mujoco output the ground truth reward, we use cumulative reward (CR) replace the metric SR. Since there are no constraints the original Mujoco environments, we add several constraints the three Mujoco environments. The experiment details are in Appendix
Table 1: Experiment results.
Task
Metric
M-ICRL
ICRL
ICRL(pre)
Meta-IRL
Expert
SR
0.02
0.62 0.07
0.71 0.06
0.45 1 0.10
1.00
0.00
Drone
CVR
0.02 0.02
0.16 0.10
0.11 = 0.08
0.33 0.12
0.12
0.00 = 0.00
322.56 1:
76.44 = 18.26
199.03
1: 53.24
113.66 1 32.51
106000::0 51.51
Swimmer
CVR
0.04 0.00
0.22 0.13
0.16 0.06
0.35 0.18
228.78
54.23
60.74
32.63
156.85
1 50.47
108.05 1 36.89
264.00
165.56
HalfCheetah
CVR
0.13 0.01
0.28
0.19
0.20 0.111
0.31 0.10
0.00
0.00
CR
712.40 + 96.53
144.79 + 66.37
311.86
+ 56.99
165.86 + 70.08
752.40
84.71
Walker
CVR
0.00 + 0.00
0.26
0.18
0.22 0.09
0.42 + 0.266
0.00
0.00
From Table I observe that M-ICRL achieves the best performance in all the four experiments. Meta-IRL has much higher constraint violation rate than the other three algorithms. This shows the benefits of learning both the reward function and constraints. ICRL(pre), which simply learns meta- priors across all the demonstrations all the tasks, performs poorly. This illustrates the benefits of meta-learning design for M-ICRL, We discuss the experiment results in detail in Appendix B

---

Published as a conference paper at ICLR 2024
6 CONCLUSION AND FUTURE WORKS
We propose M-ICRL, the first theoretical framework that can learn reward and cost functions of the expert from few demonstrations by first learning meta-priors from other related tasks. It is shown both theoretically and empirically that M-ICRL is effective to adapt to new tasks from few demon- strations. Despite its benefits, one limitation is that M-ICRL assumes that the states and actions fully observable, however, this may not hold in some real-world problems due to practical issues such noise. future direction is extend M-ICRL to partially observable MDPs (POMDPs).
ACKOWLEDGEMENT
This work is partially supported by the National Science Foundation through grants ECCS 1846706 and ECCS 2140175. We would like to thank the reviewers for their insightful and constructive suggestions.
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, pp. 1-8, 2004.
Stephen Boyd and Lieven Vandenberghe. Convex Optimization. Cambridge university press, 2004.
Suiyao Chen, Jing Wu, Naira Hovakimyan, and Handong Yao. Recontab: Regularized contrastive representation learning for tabular data. In NeurIPS Second Table Representation Learning Work- shop, 2023.
Giulia Denevi, Carlo Ciliberto, Riccardo Grazzi, and Massimiliano Pontil. Learning-to-lest stochastic gradient descent with biased regularization. In International Conference Machine Learning, pp. 1566-1575, 2019.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence theory of gradient- based model-agnostic meta-learning algorithms. In International Conference on Artificial Intelli- gence and Statistics, pp. 1082-1092, 2020.
Alireza Fallah, Kristian Georgiev, Aryan Mokhtari, and Asuman Ozdaglar. On the convergence the- ory of debiased model-agnostic reinfrreiffor learning. In Advances in Neural Information Processing Systems, pp. 3096-3107, 2021a.
Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Generalization of model-agnostic meta- learning algorithms: Recurring and unseen tasks. In Advances in Neural Information Processing Systems, pp. 5469-5480, 2021b.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International Conference Machine Learning, pp. 1126-1135, 2017.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse rein- forcement learning. In International Conference on Learning Representations, 2018
Ziwei Guan, Tengyu Xu, and Yingbin Liang. When will generative adversarial imitation learning algorithms attain global convergence. In International Conference Artificial Intelligence and Statistics, pp. 1117-122, 2021.
Tuomas Haamoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference Machine Learning, pp. 1352-1361, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Confer- ence on Machine Learning, pp. 1861-1870, 2018.
Hongrong Huang and Juergen Sturm. org/tum simalator, 2014.
Tum simulator.
ROS package at http:/wiki, ros.
10

---

Published as a conference paper at ICLR 2024
Yin Huang, Letian Zhang, and Jie Xu. Adversarial group linear bandits and its application to col- laborative edge inference. In IEEE International Conference on Computer Communications, pp- 1-10, 2023.
Mahdi Imani and Seyede Fatemeh Ghoreishi. Scalable inverse reinforcement learning through mul- tifidelity Bayesian optimization. IEEE Transactions on Neural Networks and Learning Systems, pp. 1- 2021.
Kaiyi Ji, Junjie Yang, and Yingbin Liang. Bilevel optimization: Convergence analysis and enhanced design. In International Conference on Machine Learning, pp. 1882-4892 2021.
Vanshaj Khattar, Yuhao Ding, Bilgehan Sel, Javad Lavaci, and Ming Jin. andp-within-onil framework for meta-safe reinforcement learning. In International Conference on Learning Rep- resentations, 2023.
Harshat Kumar, Alec Koppel, and Alejandro Ribeiro. On the sample complexity of actor- critic method for reinforcement learning with function approximation. arXiv preprint 191.0:111. 2019.
Shicheng Liu and Minghui Zhu. Distributed inverse constrained reinforcement learning for multi- agent systems. In Advances in Neural Information Processing Systems, pp. 33444-33456, 2022.
Shicheng Liu and Minghui Zhu. Learning multi-agent behaviors from distributed and streaming demonstrations. In Advances in Neural Information Processing Systems, 2024.
Shehryar Malik, Usman Anwar, Alireza Aghasi, and Ali Ahmed. Inverse constrained reinforcement learning. In International Conference Machine Learning, pp. 7390-7399, 2021.
Yurii Nesterov. Introductory lectures convex optimization: A basic course, volume 87, Springer Science & Business Media, 2003.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with im- plicit gradients. In Advances in Neural Information Processing Systems, pp. 1 113-124, 2019.
Dexter RR Scobee and S Shankar Sastry. Maximum likelihood constraint inference for inverse reinforcement learning. In International Conference on Learning Representations, 2019.
Seyed Kamyar Seyed Ghasemipour, Shixiang Shane Gu, and Richard Zemel. Smile: Scalable meta inverse reinforcement learning through context-conditional policies. In Advances in Neural In- formation Processing Systems, pp. 7881-7891, 2019.
Chuanneng Sun, Songjun Huang, and Dario Pompili. Hmaac: Hicrarchical multi-agent actor- critic for aerial search with explicit coordination modeling. In IEEE International Conference on Robotics and Automation, pp. 7728- 7734, 2023.
Lingxiao Wang. Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural policy gradient methods: Global optimality and rates of convergence. In International Conference Learning Representations, 2019.
Ruiqi Wang and Guohong Cao. Edge-Assisted camera selection in vehicular networks. In IEEE International Conference Computer Communications, 2024.
Jing Wu, Suiyao Chen, Qi Zhao, Renat Sergazinov, Chen Li, Shengjie Liu, Chongchao Zhao, Tian- pei Xie, Hanqing Guo, Cheng Ji, et al. Switchtab: Switched autoencoders are effective tabular learners. arXiv preprint arXiv:2401.0200 002013, 2024.
Kelvin Xu, Ellis Ratner, Anca Dragan, Sergey Levine, and Chelsea Finn. Learning prior over intent via meta-inverse reinforcement learning. In International Conference on Machine Learning, pp- 6952-6962, 2019.
Siyuan Xu and Minghui Zhu. Meta value learning for fast policy-centrie optimal motion planning. In Robotics Science and Systems, 2022.
Siyuan Xu and Minghui Zhu. Efficient gradient approximation method for constrained bilevel opti- mization. In AAAI Conference Artificial Intelligence, pp. 22509-12517, 2023a.

---

Published as a conference paper at ICLR 2024
Siyuan Xu and Minghui Zhu. Online constrained meta-leaming: Provable guarantees for general- ization. In Advances in Neural Information Processing Systems, 2023b.
Tengyu Xu, Yingbin Liang, and Guanghui Lan. CRPO: A new approach for safe reinforcement learning with convergence guarantee. In International Conference on Machine Learning, pp- 11480-1 1491, 2021.
Lantao Yu, Tianhe Yu, Chelsea Finn, and Stefano Ermon. Meta-inverse reinforcement learning with probabilistic context variables. Advances in Neural Information Processing Systems, pp. 11772- 11783, 2019.
Tianhe Yu, Chelsea Finn, Annie Xie, Sudeep Dasari, Tianhao Zhang, Pieter Abbeel, and Sergey Levine. One-shot imitation from observing humans via domain-adaptive meta-learning. arXiv preprint arXiv:1802.01555 2018.
Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Basar. Global convergence policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization, 58(6): 3586-3612, 2020.
Zhong Zheng, Fengyu Gao, Lingzhou Xue, and Jing Yang. Federated Q-learning: Linear regret speedup with low communication cost. arXiv preprint arXiv:2312.1502. 2023.
Zhengyuan Zhou, Michael Bloem, and Nicholas Bambos. Infinite time horizon maximum causal entropy inverse reinforcement learning. IEEE Transactions on Automatic Control, 63(9):2787- 2802, 2017.
Brian D Ziebart. Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy. PhD thesis, Carnegie Mellon University, 2010.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. National Conference on Artificial intelligence, pp. 1433-1438, 2008.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In International Conference on Machine Learning, pp. 1255-1262. 2010.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
This section includes the proof and simulation details. At the beginning, we define the empirical cumulative reward. Given a trajectory S, the empirical cumulative reward defined as J,(s) & Ei=0 (8))
A.1 THE DERIVATION OF PROBLEM
The derivation of problem is first introduced in (Liu & Zhu 2022). For the sake of better understanding, we include here. The fundamental idea of the ICRL problem (T) is to learn reward function in the upper level and learn the corresponding policy and constraints in the lower level. the following, we first introduce the lower-level problem and then introduce the bi-level problem formulation.
The lower-level optimization problem. Given reward function ræ, the lower-level problem aims to learn the corresponding policy (and constraints). Therefore, we formulate the following con- strained RL problem
max H(T)) + Jro (TT) s.t. Pa, (TT) = Mc. (Ti))
(11)
where Me, (T) - Ec 21=0 de, (st. ax) ]] the expected cumulative cost feature of policy π and De, is the cost feature of task T. Problem 11 aims to find a policy that maximizes the entropy-
regularized cumulative reward H(=)++ + Jr, (Tr) subject to the constraint of cost expectation matching. The cost feature expectation matching follows the idea of "feature expectation matching" in (Abbeel
& Ng 2004 Ziebart al. 2008 2010).
We cannot directly solve problem (II because it is non-convex. Therefore, we use dual methods and solve its dual problem. The dual problem of problem (TT) is (Liu & Zhu 2022)
min Ge(w; 0) H (T=::) + Jra (Tw:0) + (Heg (m;) Me, (x)),
(12)
where is the dual variable and Tw:0 is the constrained soft Bellman policy. Since (Liu & Zhu 2022] studies linear cost functions, w He, (T) Jrw. (r)). Here, we extend the domain from linear cost functions to non-linear cost functions and directly use Jr, (T). We use the dual problem (1) as
the lower-level problem in (1)
The bi-level optimization problem. (Liu & Zhu 2022) has proved that the optimal solution of
the primal problem (1I) is (0);0 where w (0) arg min,, (w; 0). The upper-level problem
aims to learn a reward Function such that the corresponding policy (i.e., the optimal solution of
problem I1 can minimize the negative likeliillhood of the expert's trajectories:
min L, w (B)) A EC-P. 8
Σ
7 log Tw (a))
w (0) arg arg min (,(w;
Then we reach the problem formulation in
A.2 NOTIONS AND NOTATIONS
Define JT (s) Ec-P, E ('r [s) s] and (s, & EsnP, 1
S. Gg = a]. Similarly, we can define J (s) and Ju (s,a).
Define H² (s, A Equr, E 7 log m (arls:))so 8,000 and Hd H*(ss* $ - -E E2e
7 log (Tt (:) s).
The constrained soft Bellman policy (Liu & Zhu 2022 is
exp( Q (x, a))
(als)
exp( Vioo (s))
(s, a) = ro(s. (s, +
vES
13

---

Published as a conference paper at ICLR 2024
Viont (s) log (/ (xp
We can approximate the constrained soft Bellman policy using soft Q-learning or soft actor-critic by treating (rg Cw) as the reward function.
Lemma 3. We have that
Ve log (ala(als) = E
Vero(st, 5, = a)
Vera(s;, = 8),
V log (a:a( Ee Panio
V (st,
t=
We (x), = 8, =]]
t=
Proof.
/ P(s' Vo (s')d'',
V 8 Quan (8) Vore(s,a)+ +7
JOES
Voro(s,a)+ +y / P(s'ss,a)
Vo Sa exp(Quml (s', da)
ds',
exp(Vaul (s'))
WES
Vero(s,a) + I P(sl(s,a) f
exp(Q (s', a') V V o (s', a')
da' ds',
WES
JAEEA
exp(V (s"))
Varo(s, +7 J.s. P()l),,)) JUEA
(a V Vi d'''
where (a) follows the definition of Tw.e. Keep the expansion, we can see that
V T (s, a) = EsmePann
a1)) s, au = a),
VoV (s) EcoePann
( s).
Therefore, we can see that
Vo log Twz (als)
= V VoQu (8) Vo Vo1 (s),
F
or Voro(ss, = 8, 00 = a) E Powe
Voro(s, = s].
1=0
1=0
Similarly, get
Va log (als) F
EquPaun
Va az) (st, = s.
V
1=0
14

---

Published as a conference paper at ICLR 2024
Lemma 4. We have the following relations:
W. (s))))
- log Tw: (als) Pau
Vw log Twu (anlss) Tu: (sr, (s),ar) -1]
90
8, 00
V H.H (8)= PiguPown
Cv'V Vw log (an (st, - 1] =
Vodia (8,a) = Vwew(s,a)
I'l Va ar) + Ve log ( Jā (st, ae)) 80 = 8, ag =
V Wi (s) Prpi
> Wu ar) + V log Twn (arls)) Je (st,
(s), 0z)l
80
V Vurrs Twn (s, = Ee
Vw log ((rls))) Sie (s), so 8, 00
1=1
Vudra Ba: (8) EE-Po
V log ((r St (8c, az) 80 =
1=0
Proof.
Vull ()) = - log Twi Tw (als) +7 +7 f a() V Hm (d)dds
d'ES
= V log To:o (als) + I P(dlls,a). Vo f SWEN Twrold'ls' Hm (s', d' ds',
PES
= Vu log Twn (a|s) + P(61)0,0) /
JoEAl (utwso(a'ls") (s
NEES do'ds',
+ (a)ls') V H (5', (F a')
= Ve log Tog (als) + Joes P(d|(s,)) / Twso(a'ls"). Va log (ogall) (
+ Vwll af da'
Keep the expansion, we have that
V. (s,a)
-w log Nw:o(als) Ex-Po
7 V log Twya (arl((a)ss)] mw (st, 1]] 8o 8, 00
V. (s) (8)=
Σn
Vi log Tw: (asls=) H/uu (8t,at) 1]
Similarly, we can see that
Wud (s,a) Vucu +r I P(s'(s,a). (s
JES
Ve / Tunoldlss)) (s', da' ds',
= =Vocu(s,)
P(s')s,a) .V WEE
YES
P(s'(8,)) / (wso(allss)
= -Voro(s,a)+ +y
(8") J (s', ,a)
YES JoEA
15

---

Published as a conference paper at ICLR 2024
V Vinee V )d da'ds. Keep the expansion, we can get Vuneew (s) = (s)
+ Pr
Ww + V log (aro(())ss) Sec (st, ac)l s0 s, 00
Σ
Vucc(ss + V log (a)o(())))) Jew (oe, arl] 80
t=
V Tui (s - -
lss P(s'|s,a). Vw (d''d''
=> I
=n/e P(s'(s,a) Vw f (a (s', d' d')dd'd's ds',
jes
WEEA
=> =>/ee P(s'(s,a) / Vutwro(d'lss) J:o " (s',a') + mw:o(dlls)) VwJro (5', Vs ad) de dd'
Jees
W'EN
=> I P(s'(s,a)
/ Twro(a' ('))ss)V log Tww (a' (a'ls') J (8', d'' ds',
Js'ss
J'EAA
V log (ar Ji (st, at) 80 =.000 =
Therefore, we can get that
20
Jaan (s) = Egpe+
? V log Tw (a (t) "
az) su =
wil
1=0
A.3 EXPRESSIONS OF GRADIENTS
We provide the expressions of all the gradients below:
V=Gi(w;)) VwJas (T() EENPPon
1 D D
V.G.(Wu;,,))
Σ
V VoJcn (C5)
Eca
V (ancas
1=0
VoG,ww;)) PCMPPWD
(oro((r: VoJro (m²),
t=
1 D
VoG,(w;0,D) PeenPous ?" (1)
Σ
(C))
D
1=1
G(w;,) = V2 Van (,(w;,,)) = -EgnPrue
Σn (Cor_(s)) + V (en)
V20G((::0) V2 G(;;; EECPPIIN
2 Vd
VoLi(0,w) EC-Prand
VoJro (T:),
1=
VeL,(0,w, D) ESNPPND
?'
Voro az)l
1
Σ Vo)roo('))
D
1=
16

---

Published as a conference paper at ICLR 2024
Vol4(0,w) Vuda Pan V (st,
V.L,00,,D)- 1 D
x Vudca (c5) -EcuPtuss
Σ j=l
(u
V2 2.w,(0, V L0,0,,, (D), -EC-Prane
(ov + Von (s), (2
(2,,)(0,w)) Vuo w0,,w, PPe
7' Va [1
where the expression of Cov. is in 14
Proof. Derivation of V.G; and V.G.
Since Tu. arg max, (T) + Jro (x) - Jew (TT), the following holds for any (s, S A:
H(T) + Jro (T) Jew
(T) (TT)
7.= = 0,
(als))
where we change policy to be time-dependent but force it to be stationary i.e., x. Therefore, we have that
H(T)) Jro (mm m==w:
(al()
= (s)lor" (all) + 1) + TP,.. (s) Egw Pani
(s)((og
-y" log Two = 8,4 = a)
r=t+1
+ +P, (s)
- - + Egp.P Σ
7 (ro(st, - (ul = 8, at = a)
T=1+1
= = (8)
HT - + Joh (8.a) Je (s) 0.
(13)
Recall that Gs(w;:) H (Twie) + Jro (Tw:o) Jow (Tw::) Sew (TT).. Therefore, we have that
V.G,(w;0)
Po(so) -
HTii (5o) + V..tt.ri0
Twi (so) ViJer dso + (wJan
SOES
(a)
Po(so) EenPr
Vi log Tw:o
JgES
1=
Ja ai)) (e )
(Hawi (o (8t,
- 11 + J
(s)
dso + VuJees
h (Te) - EcoPrund
(st,
t=0
where (a) follows Lemma and (b) follows
13
With similar derivation, get
1 D
V. = D
Σ
V JaJa, (5') - EE
V (se,
==1
1=0
Derivation of 2 G, and 2 Gi-
17

---

Published as a conference paper at ICLR 2024
We know that Vu (s, m= VuEgmPrn.. 1E V' ailso s,a0 ao a| and
V V.r (5) V Eggup
IEt=0 " n' Cw at) su = s).
V ww
Gi(w; = V. BewP
(v.
-- --/ sOES
Vo Twno(a VuJir (50, a0)) daodso,
JapEA
Po(so)
V Varazo(aolso) Vuree (s0, ao)l==
JagEdl
+ Tu;o V VV(Waee (su. ao)) dandso:
Po(so) [ JagEAL V V
( (sp, ao)
+ Tu:o(aolsi) V (V ao) +
P(sı ss, Vo ( (s1) dsi)) dss) daodso.
Keep the expansion, get that
G)( JoES 1 Soc JaEd
1 (8)
Ve Gi(w: =-
Vu (all (s, =)
dads
Pe 3' Cw ar)l.
t=0
Define the covariance
(s)
Twid (alo(a)s)V.. log Twy (als(a))( (s, amm Ta, da,
JaEA
[
(al)(als)
Vora (8, m= VaJa (8) (+)
da, (14)
JaEA
where (c) follows Lemma Therefore, we have that
V www (u(w; -EquP
Σ nd (Cov((s)) + (st
and similarly we can get
20
V G, -EcmPa
Σ nd (ov + (a
Derivation of V G and 2
we
V GeoGe V E
- - Po
(aul())(so) V (so, dagdaa
LaoAA
/
Pa(so)
VoTw; Vani (so, ao)l
agEAL
Twy (ag) *T;o(aglso) (8)) V W au) + /f J. P(s (s))) dsi)) dandsu.
Keep the expansion, we can get
Vio 0) = - (TTTr( (8)
SES 1 1-7 JaSA Tw: (als) Vo log (a)s) (8, a) dads,
18

---

Published as a conference paper at ICLR 2024
=- Prwse mau
(alls (5)
and similarly we can get
G, D) P
Vo log (a))) (8, (8, (1)
t=
Derivation of VoL, and VoL.
(oL,(0,w) - Es
V log Tw; (ar(s)))
t=
-BgNpP
Σn 7
E E 2919
Ai) = Ao arl
1=0
1=0
0
= =.)]]
-EemPrun
7" Vero Ar] (S......)5)55
1=
Σ9
Vora(Se; PenP
Vora(S, A,)10 Sc, Ao aa
1=0
7' (orr(S);:A))/50 =
t=
-Ecppa
Vora(s:; ar) + +EEepPr
(Sr = 8141]
1=0
Verro(S Ar]|So = =s))]
1
=-
Voro(si, az) E
Voro(SS 4:)S0 = sol
1=0
Voro(s(, VoJro
Similarly, get
D
1
Vol,(0,,;D) Fic t=
(s))
V
VoJen (5)).
D
Ø=1
Derivation of VuL; and V.Llg.
The derivation follows the similar steps of the derivation of VoL; and VoLi, thus we omit it.
Derivation of 22 L; and 2 W Ly
We know that L. = G and thus V La
V2 V2n.G
A.4 THE REDUCTION OF (2)-3) TO IMAMI.
From A.3 we can see that (WG,(w; 0) = VaL; w) and V.,G((w;)) = (0,,,( w). Therefore,
know that (.(0ww) w) Ge(w; 0) where is constant, and thus G, also serve negative
19

---

Published as a conference paper at ICLR 2024
oggllikelhhod function.
When 0. problem 2 3 becomes:
1
min 0,w
(0, ()), s.t. nz (0,w) = arg min (T; + 2 wl(,
which is equivalent to
1 EE L. (B,W)), s.t. 77. (6, w) = arg arg min ()( + 11 wil? 8,00 m
min
2
"
we ignore the reward parameter 0, this is the standard formulation of iMAMI. Rajeswaran et al. for the cost parameter w.
2019
A.5
THE DERIVATION OF THE SYPPERGGRADEENT
From implicit function theorem,
get that
V Von (0,w) IV Gi(n Gn ()))) +; (, V V (0,w) >(V? mm G, (1) (0, w); -
- A/--
Therefore using the chain rule, we can derive the partial gradients:
0
= VoL
00
= (O.)) - (rg.:/0000)0..a.a..1.00000006661 - (0
( [10,0(0)0))))000444000000)00010) X(V> mm (n (0, + -V (0, 17 w).
8w
With the partial gradients, we can derive the eyper-gradients:
T o ()):W)
0e. θ
Li(s, n ()),
00
00 0e
8²
(1 n (0)
L.(p, n. (p),,)),
a
002
J²
(0, n (0, (w)
VoLo(1i 17 (P)
V Gily w); pi) 2V2 G. (1p,,, wp)p)) + A1I V, m (p),
OLi (ph,w)
OLi(ps vi (vi,
pppTT +
aLi m ()): y Oyg (yp; w)
Ow
O4
On
Ow
a dw00 (0, n/ (0,w)) TT, n ()),
Φ²
2 Gi (0, 0) + - (o, (0,w)
A.6
THE SMOOTHNESS OF G2 AND G
From Lemma
A.3 we know that
2 G.(; V2 Gi(;,, = - -ECN
Σ 76 (Cov((s)) + V2 (e
1=0
Covw(st) = I Two((ass Vunt x=
Vurt (s) (s)
(VwJ (== Ta. da.
20

---

Published as a conference paper at ICLR 2024
Assumption 1 (ii) assumes that ( & Ce and Cs & C. for any (s.
A. Therefore, we have that
Ce 5
C. 1-7
V eu (8, a=
Vi
(s,
1-7
Ce
B Pa
V Cw
0z)]l
1-7
t=0
Then, we know that
2C. 2Cc + 2cc & Cv.,o.
V waw Ge(w;0)| = 2 G, D)Il v 1-7 1 - 1-7
Similarly, we can prove that there exists constants Cv and Cvz such that I G, I[V 2o D)Il S Cv3.a and V Gw;; Vo Gi(w; 0, D)Il S Cva
From Appendix A.3 we can see that VwLi V.Ga and VoLi = VoGi. Therefore, the second- order terms of L, can also be bounded by these constants.
Lemma 5. There are positive constant
C Cr Go Cps and Cvspo such that Gi D)Il s
V pom (i 0)1 = || pm Gi(n 0, D)Il S Cvs G. II 1po
Ci Ci( = | Gn
Ovs Go and V 100 (m; 0)Il V 100 (r D)Il & Cvs for any (1) 0) and any task Te.
Proof. From Lemma
A.3 we know that
D
Vin Gi(w; V Van Gi(w;,,D)
EE Σ 76 (ovi(si) V (ouccs(t
Cov = SaEA (wwa(als)
Vules (s, 1) xT T=T Vude (8) (V,Je (8,00)===5 da.
Now we take look at the term (st) V Cw
(s)) + (se (()o
(z)1
$ We (s, a)) Vw (5)1 || -| (s, ر) | ca a.]l.
20. Co
+ +C.
x 1-7 1-7
(15)
Cov
V. /
(a)s(
(s, (s) (V.J (s, (== To, da, (s - (s) (V (s. (s) da
JaEA
$
(aTs)
JOEA
(16)
(als)
(V.Ja (8, Tda da Vi (s, Vura ())
SAEA
(17)
+/ aEA
(als)
VaJa
(s, VeJa (8) (s) Vel (s, da.
(18)
Now, we bound each term 16
First, we bound the term (6)
f AEEA
VwJ
(8, - -wJ (s) (VaJa, (8, a)m== ) da,
21

---

Published as a conference paper at ICLR 2024
- Twia(als) log (alo(als)-
nEE
Vu (8, -
(8) (t) (VaJe, (0, m))== da,
(0)
JAEA Tuna(als)) EEM
s
t=0
y (s, 90 = =, 00 = a)
t=
Vure (8, a) Vude (8)1==xuno) (Vude (0, a)) = daa Tda
where (a) follows Lemma
Therefore, we can see that
If VaTurolals).
Vurt (8, m) Var (s)) (Vw a (s, ()) T dall;
JEEd
2Cc 20.
Ce
$ 1-7 1-7 1-7
(19)
Second, we bound the term
Va 6o (8.0)
Eg Ps
V (acc(st, -V Gi
(20)
(b)
Therefore, |/ - (8, a) | Gi(w;
0]1 S C where (b) follows A.6 Similarly,
we can see that Jr (T) ml S C a.
Then we have
nn
I Twz (a)o Vunte (8, a) I= Vuve (8)1== Tun WEEA
(VwJr (8, (8))) Jdal]
s 20vp G C.O G.
(21)
Third, we bound the term (8)
/ (alo(als)
Vo (w (s.a)l== dall, Vusi (s, a)lm= Vor (5)l
dalll,
JOEA 2Ca
1-7 Cvz.a-
(22)
Therefore, we can see that
Cov (c) 20.
20. C.
20. + 20v G G.Ov G
Crz
(23)
1-7 y 1-7
1-7
where (c) follows 19 19--22
22
Define Cov (s) Een E 7 (Cov.((s)) + (s), s, ao
a] and
Cov (s) EENPaaee Deeo n²t (So + Van (5t, ar))ss0 = s). We can see that
1 7 (1 21- + +c.)
20
Cov
(24)
where (d) follows 15
Therefore,
/
2 C((w))==-) Ja Po(so)
JagEA Tw:e (ooe((a)so)Cv..... (3o, ao )dagdso
22

---

Published as a conference paper at ICLR 2024
--/ Po(so) I
V Vamwno((olso) Cov 00)
SOES
AgEAL
+ Tu;0(000000 ( Cov (80, ao) dandoo:
- --/ S0ES
Po(so) [ LaEAL
V Vutune(aolso)) Cov 0o) + Twia (anls)))
V. (ov (0) + V cwlo ao) + If
Cov
daodoo-
MES
Keep the expansion, can that
V ww (: = E
Σ
Ve log Twna (aels)) Cov (8t, at)
1=0
+ V wwww (w ax) )
+
(e)
20c
2C
+ +Ge
+ Eggesnn Co 01) V Vonn
1 1-7 ? 1
(1 7)2
5 1 20.
1
20
022
1 1-1 y 7
1-7
(1 - 7)2
20. 20.
C.
+ 20v2 G C G+ 1-7 Co2 + Ece Vinn (an
2Cc
1-7 1-7 1 7
where (e) follows Lemma 3 and 24 and (f) follows 23 Assumption (iii) assumes that
I| Cw, a)ll is bounded for any (s.a) S,A, therefore, there is positive constant
Ova inn G, such that ||V 9m G(v 0)1 S C Ray
for any (n, 0) and any task T, We can also get
IIV myy (r 0, D)Il S Cv.
Similarly, we can prove the existence Cra 979 a and Cpa
G-
Lemma 6. There are positive constants Cps inon G. Cas vnno G. Ces Gr Crs
200d Go and Ces
h0od such
that ||V Gnon,G((: 0)11 i T (r 0, D)Il S Cvs
nnon G. I[ umo Ge(m;
0)1 G
V inmo G1 0,
D)Il I Ce. D 1 (n 0)ll
V Tooo G (m. 0 D)1l S C yood G 0)
C
I|V 1000 Gr 0, D)ll S Cv4
1900 Go Go and 0000 (1;
|[ 1000 (n; 0, D)1l S Cr
We can derive the constants following the proof idea of Lemma 5 and thus omit the proof.
Proof.
A.7 PROOF OF LEMMA
Following the standard results for (A - Cv2 a) strongly convex and (A + Cv3 csmmooth objective functions (Nesterov) 2003 Boyd & Vandenberghe 2004), we know that
[k<(K) - AI + V (((h,((),, K); P. ) V, V,,Li( (f,, D.val K), Devall |||
C G
(25)
Define H(TI: A XI +
22
(1 Pi, Deval), therefore x Cvg < ||H p:)ll < x Cva (a)
mm
and (V,HHmp):)ll | mm G.(n: , D. < Co. where (a) follows Lemma Therefore,
[[H(m, [7(122+))] -11,
-I[H(()2::))) - H(m:p))))((mmmm)))
(7((1234p))
23

---

Published as a conference paper at ICLR 2024
(17(())p)))) -11 -10 H(m :x)l+
C
S (A Cv3 G) 1lm - 1/21-
(26)
(c) I C where (6) follows A.3 and (c)
We know that 11 1,,,L(00,, 17, D)I
| G. D)I
(d)
follows
A.6 and
|V (0, n7. D)1l
C_ where (d) follows A.3 Therefore we have the following
" m, D) - [M( [o:) (y 72, D)11:
[[[-((i::)] m11 D) [H(122 V V Li m1 D)1l
+
(p m. D) -
V L (pii, 12, D)1b
[[H[(]):))) [H(122 -111_ ,,, 7/1 D)
,1,,I,, 71 D) V 7, (1 12. D)I| (1 sp:)] - -1:
Cv. Cran cCo
Cv 1|m 72ll.
m 722l
(27)
1->)(1- Cva
X Cvz
where (e) follows 26p. Therefore, we have that
(x) - [H(A] (G w. Di ): -,) f (8i, w, Deval), Dival))l
< (k((K) [7 (gi, w. D.vall (.)] V (5is in (fi,, w. Divall K), , De
+ (1 w, De K): 6. ' (p:: M (1:: w, Drval K). D
- [H( (G0ss w, Devall ); p:)1 V L, M7 (6 w, Dival), D.val))
0
(1x00 Ove
an G yk C G yk(
(28)
where (ʃ) follows 25 and 27
From A.3 we know that
99 Giww; V CL
Eq
Vo log Tw (als) (s,
(g) 2cr Therefore, 2 Gr 0, DD]]
S 1-7 1-y
Ovp G where (g) follows Lemma 3 and 20
Then we have that
-
+ (,)
V2, (, (pr,w, D), D)Il.
(6) C,
< 1-7 - D)I1
+ (,),
(B,, D), D (,, () (9., D): " (7
+ (,), D) (B), D): (0: )) (Bi, w. D), D)
V2, (() ((() (Q.,w, D): (.) -V, 77 (p:, w. D), D)
C (i,(0,,, D, (0,w, D)11
(0, K);00, [|x(()) [H( (pi w. D):
24

---

Published as a conference paper at ICLR 2024
Va, (() D); D)11-
[P D) D)1,
@ C. (1 D, D)I 0 V jR +
CV jk
(. Vi, (0, w. 0): D)Il
(), (P:, w, D), D)1b
2 C2
Cvz G )R
CV C G
| w. D. K) (0, w. D)I + +0 (1-
C.
CPs G CV3nG -CC3, 1 7
[0, w. D. K) - 'i (0, D)11:
50
Cvan G 9R +
Cv3,, G ) )
where (h) follows
A.3 (i) follows A.6 and (8 and (j) follows Lemma 5
A AwiIl (l(=((K) [H( (0, w. D): - D),
(k) o CV G
k +
C kn
G
)8)
x
where (k)) follows 28
A.8 PROOF OF LEMMA 2
We first provide the approximation error of the first-order approximation and then provide the ap- proximation error of the -raadnggrddents.
Claim 1. There are positive constants De. Do and Dw such that || _h L, no (l,w))l d8 807 L, 1/ (O,w),DII S De. ll 32 88
30 L, nt (w,)))l = 11 L, oi (0, D)1l S Do, and Tweese L; 1T (0, ())l = Tweeeo (. m (0, w), D)I S D..-
9a Proof. We first show that || 1 L, (0, o/ (0, w)) ||||| is bounded and the boundedness of 363 (0, nz (0.
w))|| follows the similar idea.
0²
+ 002 L. L:(0,, (0, w)) (2 L0, (0,
+ (Vonn (0.w)) (0, nf w)) (onn (0. V (. m (0, w) V Von w).
(29)
8 To show that laoz L, (0, n/ (0, w)) is bounded, we need to show that each term in 29 is bounded.
From A.3] we know that V,, G = V L, and VoG.- = VoLi, and thus G3 = L; + C where C' is n Lil. and //2 [ Lill are bounded from Appendix
constant. Therefore, we know that V 00 L.l.
A.6
Co Moreover, [|V < I-
Then, we only need to show that | V (0,w)) and V2 (8. (0, w))l are bounded next. From we know that Von,
A.5
2 m G, (6. (): + AI] V2 oo G. Therefore,
(a)
Co2.a
Von (f. 2 X- where (a) follows A.6
Viont (0, w)
V3 Vin (O,) (())(())))
+ v3 Go (0, w):
- G(, 0) + ~
25

---

Published as a conference paper at ICLR 2024
We can that N (8, (2,2 is bounded because all its terms are bounded. In specific. A.6 shows that the second-order terms of G; are bounded and Lemma [ shows that the third-order terms of G bounded. Therefore, Dg exists.
From A.3 we know that V.G; = V.LLD and VoG, = VoLi, and thus G, = L; + where C' is a constant. Therefore, we know that II V mwn I. II my Lill, and || V Lill are bounded from Lemma 8a
5 Then we can get the boundedness of aga L nz (0. wi)ll once we prove the boundedness of
V 000n7 w)]-
V doon, (0, = - (L,
+ V T http://000000011000000000///11/0
+ V. wwce::00000||33..000000011111111 |1
1 (01
V
1FC00100000000010000000000000010 Gn (0, w); 0)]-
my V V nee [-
+
(0,,)),)) [,)
Even if the expression looks complicated, we can conclude that | 890 (0, w)]l is bounded because each term in the expression is bounded. In speicife, the fourth-order terms of G, are bounded (Lemma 6 Then Da exists and similarly D. exists.
Then, we have the following inequality:
0
20 10 + 8Dg 17 (0- + w))
0 (0 ni (0, + L, (0, + Do |110012,
Φ²
$
00
ag²
0
(0 80a, n1 (0- w))
00
0
8²
De 161el12
2 00
L, mi (0,
w))oaa
30
0
(. 8D0, (0 ) w))
[,
-Do n o)e,w))
00
0B
20t
L, (0, + Do1102o11?
002
0
(, + 8bg (0 w)
00
0
8²
De 161.112
2
L, (0,w)) +
Li (0, ni. (),w))
00
000
2
0
( 84g (0 w))
00
0
Do
<
(0,0,) (0,ww)))
L; nz (6, w))o +
1820112
00
002
2
0
0
(, + 860 (0 (0 + w)o
L. SA.. n, (le.w))
00
00
20a
00 L,, ni (0. 12110
Therefore, we can conclude that
(, + (0. + w))
0, 54g (f (0- aD wi))
25
a6
00
26

---

Published as a conference paper at ICLR 2024
8 002 L, 1, (0,
Dø
S 2 |
(30)
Similarly, we can find that the approximation error Swas (0, ny (o is upper bounded by 6 ||
| - aV gooiil
0²
|| - -0.1l
002 (0, (0, (()(,,,,)))))))
(4)
G y ( +
aDo6
2
-0
CvanG G ) +
C jk + t0).
A
where (a) follows Lemma [nnd 30
Similarly, we can get |س aV swall ( + Coan (K
+8)
A.9 PROOF OF THEOREM 1
We define function f((0, w) such that Vef((O,w) Âo,. Vo, and (w -wii-aa -V
Recall that
Áa. = V VpLi( A (Bo, w. Drval K), D - V
bqy (n) (6 w, D.val K); P.: Deval).
[AT + V G; (fr (9i,, w, Deval K); 9ú, Dival) 1 V. (, n ( w, Dr K) D.vall)
A A[V + V G. (n (Pi w, D ); pi, Di K)] V (B: ( Drvall K), Deral).
Therefore, we can see that |لب and ||ر are bounded because the second-order terms of G bounded (A.6) and the first-order terms of L, are also bounded (see the expressions in A.3 Since Aa,i and Aw,i are first-order approximations using partial gradients, || | and ||A are bounded. Therefore, there exists positive constants Cv.f and Cv., such that [V. S Cv.s and fi0, will < Cv.f-
There are m training tasks and we denote the distribution of the training tasks by Py. We define s(0,w) A Ei-py w)] and thus we have:
Ei-Pr [of(O,w) - Vol((0.w) -af(0.))Vee((...uu = ETPP [Ivof( - Σ
B
Goof
V vef((0,
B
(31)
Εi-PT Vwf(0,w)- vwf((0,w) - = 0, EiPrr [ - B V (1,,(0,)))1)
B
OF
5
B
(32)
The gradient Vef.(0,w) w) is Ce-Lipschitz continuous in (0, w) and (wf((0w)) is Cw- Lipschitz continuous in (0, for any task Te. Thus is Cy-smooth.
Claim 2.
Proof. To prove the existence of Cg. suffices to show that (1 and (1 w)]l are bounded. We know that (/V),./(0,,, w|ll < |1 A + all 00 Vo.ell- Since Vo. is the first-order approximation, // V < 11 2e.ll/8 Therefore, it suffices show that 1/ & is bounded.
a Ags: 0.::
00
(0,
27

---

Published as a conference paper at ICLR 2024
We know that the second order terms of L, and G2 (Appendix A.6 and the third-order terms of G; are bounded (Lemma 5) Therefore, suffices to show that | K and [[Var(K)] are bounded.
a(k + 1),
= =x(k) - 3 SIAL+ V Di);
D1, V, 1,
We know that I + V oy (,,, bounded. Therefore, || (K)) is bounded because K is finite number.
Di)lll and V,,L(0,,, w. D1, K), D2)1 are
We define (kk A [A + V mn Gi(in(0, D1, K);0, Da))a(k) -
(0,0, D1, K), D2).
Then, we know | (o)l and [[Vo((0)] are bounded. Suppose /|V and oo R)]l are bounded, then Vaakk + 1)ll S [[ + V[[Vag(A)) is bounded and [|Vag(k 1)ll S
IIV mm Ge(ff w, D1, K);0, Di)II [|=()| + +| V (f: D1, K); I
1 (0, w. DI, K), K2) is bounded because the second-order terms of Lg and G2 are bounded (Appendix (.6 and the third-order terms of G; are bounded (Lemma 5 By induction and given that K is a finite number, we know that || V []]l is bounded.
Therefore, Cs exists and similarly we can prove that C. exists. Thus, we can conclude the existence of Cy.
From Claim we can also see that is Cy-smooth. Therefore
[(O(n + w) 1)) - ((0(m) + ( 0(n)] 1| +1 (())l
[10((n 11) 1) -
+ [V..(0(mm
a(n) = ((((m),
B
B
a(n)
B
(O(m) (..
B
2=1
Cr(a(n))²
1 |
V [
Take expectation over training task distribution Py both sides, get that
Elf(A(n + 1), 1
([((())).w()))] F E -o
C)(a(n))
(110//00))..00)01111100000000000 + C)(a(n)22 [IVof((())
2
B
B
w(n)) 1
vo.100)),dc00 Σ vafkm))))
B
(2) (((())) + (C(((())² ((((a()))a(mm))) (m
C)(a(n))
(C :RGg):
2B
N-1
[(()) Cz(a(n))2
E
Cy(a(n))
< E[((0(0), (()) - [(N))
(C + CZ,r),
2B
0=
28

---

Published as a conference paper at ICLR 2024
N-1
a(n) Cr(a(n))²)
Σ
E (0))
2
Cy(a(n))²
& ())) (0000)0)))0)
(C + C
2B
NO [
aa(n) c
2 - Cy(a(n))² (C C)
2B
M=0
1
Cy(a(n))²
22 NN-I B [(0(0),=(0)) - ((((N)))(N))
+
(C
Gz,)).
22N
2B
n=0
and ā S 24c, Therefore, we have that where (a) follows 312-332 32
N
C1
C2
((I))
V Ni-p
BN
(33)
where Cj & E f(0(0)..(0) (((N),w(N)) w(N)) and C2 & a En= Crla(n))
(C +
). )) Note that 0 (a(n))² 2 is finite because (n)) (n+1)² and P 12
We have the convergence of f, the next step is to quantify [ lo Ls m (p;, and [V[ E l I [4 nf ()]l
Claim 3. There exist positive constants Do and Dw such that 11 Lolpi, (1p; I Do and I L. n (l), < Dw
Proof. From the proof of Claim we know that
2 L: (0, or (0, (])l is bounded. From Appendix 30
A.3 we know Voo = Ve and from Appendix A.6
we know Gell is bounded, therefore
8² 30 Li(yee, of (sp;, w))l S ||T - a aede (0, n7 (0, w))l
0p L, (, n (p)) bounded. Thus,
Do exists and similarly we can prove that D., exists.
E
Enppan Dppal Di (of(((n),w(n)) Expry L Lx((p(n)), ot (p:((),
00
<
E
E Eppin Dand D Emp,
(of( - 90.1|
+
Engpin Doat De Ea-ry
190. 08
(8)
_ 0
C G Cv3 G
8
E
Erpoin paol DD Fai [g0 190
I. (
00
(c)
(
+
O G
K
+6)
+
aD3
(34)
A
min; ( (DD k
29

---

Published as a conference paper at ICLR 2024
where (b) follows Lemma 2 and (c) follows Lemma 5.10 in (Fallah al. 2020 if a
[0, Do!
Similarly, we can get
E
E Exnis D. De Bi (V
(P.))) dw
Cvz G. )) + o9
Cvz G
nm )) + 6)
aD
< 0
+
(35)
min D
With 34
and
35
we have that
E
Exqus D. Da Bi (VF((()))
<0
Cp G m9
C
G
R
+0)
a(D D2)
$
+
m9
+
+
(36)
min; ( (DD.) ) 1=1
Therefore, we have that
1
E
VL.(pi(()
E E (10
< E
Epiii D De p. || [lV
+
F.
Pappin D Dy F V
(0(
- (
2
ow
(d)
$
E
EpppinnDe Dz [[(((n)
+0
C
G
K
CF2
G
+ K
0)
a(D + D2)
+
nm
A
A
min; { (VD. 1
N-
E
Erppan Demi D [[[[((())
N
99
G
K
+
Ovz nm
G
R +8
+
a(D3 + D3)
A
min:f (DD)
Ci
C2
a(Dz + D2)
Ov G
Cvs G
)
+8)
+
+
Ni-p
BN
minit (DDH
m
Ji=1
Cvz G K
C G R
S $
un
+
+
X
min;
where (d) follows
36p and (e) follows 33
To find the iteration number N, we have that
C1
C2
+
S c,,
N1-p BN
G
C2
C
Ca
min(Ci, C2)
82 N1-p
+
>
BN
BNI-P
BN
BN
N min(C1, C2)
B
A.10
PROOF OF PROPOSITION I
From Appendix A.3
we know that
V V)))(0ww) = EC- Prwe
Voro(St, -oohr (Ti))
t=0
30

---

Published as a conference paper at ICLR 2024
V=l,(0,w)) VuJre (ms)) P ao
7 (e
t=
Note that ( Eq Ei=u ' Vorolss, iyy and Vulee (T:) = PC 1=r=0 7 (st, arl] = I-q SoeS SeEA (8, ( Vull(s a dddds Therefore, we have that
Sos Soes ( (s) a)dads
1
(L,(0,)) VLm+1 wlll
m
m
1 m
(OL:(0,w) (6Lm+1(0,w.)) +
Σ
Val (() (0.m+1(0,w)ll
m
m
V VoJro (m.)
(Tm+1)ll (
1
Σ
VoJew (m:s) VoJes (Tm+1)ll:
m
1=1
1
1
n a) m1 a))
#ES MEA
i=1
1
/
m
+ 1
4 (s,a) - -m=mit (s. Vu cu (s. a)dads
BES JEE
1
(s, dads (C, + Co), n nr (s,a)
KES JEA
m
C Ce
5 1
1r (8,a) =m (s, dads,
SES JaEA
m
C,+ + Co 1
m
1 mm = (d(
Σ
1tt , mm)++)))
= 1-y
m
where (a) follows Assumption
(i) (ii).
A.11 PROOF OF THEOREM
2
From Theorem we know that we can find meta-priors (0, w) such that
m
ni (p را sc,
E
VL
m
where 6. = a a (0, 11 (0, W)).
From 4 we know that
02
á
00 L. 1/ (p), =
L (8, n (0, x))
L.
no (p,,
a
ag
ap
8²
a Li m x)) + ow Iu(pi, no (vx,, i)).
La 1/ (P;, = -t
11 (0, x)) (0
ow
awo0
0e
From Appendix A.6
we know that the second-order terms of L; are bounded. From the expression we can see that | bounded. Therefore, we know that
of VoL, Appendix A.3
L.
m (p., x))| O(I
(. m (p.
dip
00
0
à Bw
L. (G,, n (p,, a))1 O(I
L. (p,,
m (g, I)))
dw
Therefore, we can see that
i L, ( ni (p:, 2| +
a
(0, ()),:w))l==ocm (10) 11 S
L, L. nt ())
Dip
aw
31

---

Published as a conference paper at ICLR 2024
=0 a Lo (2.
0 dw (p: (
= 0L
[IV - O 7i (P..)))II)
Ell
VL, (0, n7 (0, ()) |1 - O
Σ VL (P: ())II) - 5(().
(37)
From Appendix A.6, we know that there exists positive constant CVel such that (| OVRL. Therefore, we have that
m
Ell m
L; -- (1 (L,(0, 10 2m-1. Mints [11,
Elll
(0, 7)) (0 VI, (=000:))l==mmt1m0e- (p (0m+
m73
E |1ps - pm+zll + m (p:, - nio 1 2111
m
1
m
d(p pmm Rm 12),,
(38)
772
where (a) follows Claim 4
Claim 4. 1 m
Ex || Sm+1ll I O d(pptepmmmt)) )m)+))) and 1 m E (p,,
1 (Pm+1.0)] s 5 1 Ei 1= d(w " m==m1)))
Proof. Follow the proof of Claim we can see that that there exists a positive constant Can such I Con. Moreover, we know that there exists constant C such that < C because the second-order terms of G2 are bounded (Appendix
that
0² L, (0, nz (0,w))I
2080
2
G. V mm G. + -
A.6
1 m
m
θ Lm 10, 10, 20
| pm+ill
171
m
00
i Lm- 10, (0,w)
(0.
m
00
00
0
+1
11
L m+ (0, ni (0,w))
(0, ni (0, (0,w))]l
00
00
aCon
| (0,w)) - (0,0ww) (0)l))l
a m
a Li (0, w))l.
Lm- (0, 17 (0, w)) 00
m
08
aCon
m
|mm+1 (0,w)) v/ (0,w)))
|V L. (0,
m
m
Vo (0, C (0, 1/ (0,w)) L0,
aCe
In i (0.w) nz (8, w||l + at Σ T=1
d mm++)
where (b) follows Proposition
Now, we bound the term
|1 (0,w) n/ (0,w)ll-
32

---

Published as a conference paper at ICLR 2024
We know that
V. Cmm1(1n+a (1m++(0;w): + 1m
wwl2 11 (0,
1 w 1, M (0,w)
Therefore, we can get that
1 (0, w))l = - 1 1h -, 0)1,,
= =0 Je (Tm+1) Vy (T:))l):
S c.0(dd(m)) , ,m* ))
Therefore, we that 1/ Σ x=
| pm+1l || 5 550 (
m Li=1 d(a u 1)) and similarly
can get 1 Σn || (Fi, m (m W| m
$ 2m , (==+t))).
Therefore, we have that
V (w.
< E
Lm 10,
V Li (L:(0,))l (0,:://=-=pp..... pm
m
[l:(0, (7:(0,0)l==p,-=00(00) w 1 1
m
V w- (=00:/0000000000000000000000
m
0
d(H km+1 + d(
nh mm+)) ) + c)2
where (c) follows
38
37
and Proposition 1
A.12 PROOF OF THEOREM 3
The proof similar to the proof in Appendix A.II The key step to find the relation similar to P is the following claim.
Claim 5. For given (0, w). holds that
1
L.(0,w) Lm+1 (0,) O(d(
Σ
th # (m++))) ))
1=1
Proof. Recall that
(,(0,w) Pas log Te:0 =- S
1r (8, (1,((s))))) log Tw: mw:o (als)da))
JaEd
Therefore, we have that
L((,w) Immz(0,w)]l
log (als) dads, ne (s. ((ssa)--H1)) (s.=a))s
SES JEE
im
Old( m E
13 1 mm+1))
where (a) follows that fact that (0, w) is fixed and thus there positive constant D such that log Tw:0 (a:o( < D for any (s, a) S A.
33

---

Published as a conference paper at ICLR 2024
Remark 1. Note that Claim does not hold for any (s, a) S A. it only holds for given (specific)
(9,w). However, this is enough because we only need to use Claim 5 at several specific points, e.g., (Ps, n (Pi, ()) and (fm+1 1 1m)
We can see that
ElLm+1 (fm+) 12 (pm+1, Dm+1))] - min Lm++ (0, w),
0,w
El (fm+1. +m+ (9m-1. [))]
min (1,+1(,,w)
S
E
Lm+1 Lm+1lfm+tn Min (pm+1,
L: (atmttlfmmmmm (Fm+1. x))
m
m
+
L Li(Pm+1, 1
La (p m (p:
m
m
112
1
E
L, n (3),
min L, ni (P)
m
0:00
min Li 77 (P) - min L Lm+ 1001(0,w) w). 0,w
7
0,00
The first and fourth terms are bounded by O(d( L Σi=1/ nr, (===))) (Claim 5 and the third term is bounded by 6. Now, we look at the second term. Since L. will and T= will are both bounded (see the expressions in Appendix A.3). there is a positive constant CV such that | LO, will S CVL
m
E
Le Tiut1 (
Li n (P,,
OVL
1| - g I1 (mm+1, - n (5 xل)]
m
0
d(u n +=)++)))
m
where (b) follows Claim Therefore, we have that
AlIm+11(pmm (fm+1, 10 mm (fm+1, ūo Dm+1)))]
min lmm+1 w)) 8,w
1 Σ d(u"),1m=+1) )+ + d( s O m
Σ n m=+))
m
A.13 PROOF OF THEOREM 4
If the reward functions and cost functions are linear, there are reward and cost feature vectors (,()))) and )e(.))) defined over state-action space. The expert's reward function of the new task Tm+11 are 7m+11 = T and Cm+1 = w of The parameterized reward and cost func- tions are respectively Te = 0T or and Cw = w (Øc). We define the expected cumulative re- ward feature as (T) A Ec-Pe I2 ('dr(s, at) and expected cumulative cost feature as (T(TT) & EC-P, [E (st,, ar)]. Therefore, from Appendix A.3 we can see that for any (0, w) and any task Te:
VoL:(0,w) = ur (mw:)) ur (T()),)) V.I.(0,w) = Mo(TT)) (Fwg0))
Therefore, we know that
Elll V L (6, (0,w)la (0) - (5m 1. P. 1) ||l-
F|[| V V (0.w))l=ppm w (pm+s D Pm
34

---

Published as a conference paper at ICLR 2024
1
$ O
+ m
Σ
d(p th
: (m++)) ) + dl
1
mm+1
m
Thus we have that
B BlJ (T Jrm 11 (Tm
2 F[1 (T Po (m++s)ll
Elll prr (Ti (Ti - Mer (Tm++))ll))
o Elll Lem+ (0, p0= 1athmmiww-fhho L. (pm D. (mmmmill
1
1
s O
+ m
d( v p f + d(
: m
m
Similarly, we can get
El|Jem-x (8 Jemts (Tm+1) 11)++]]] S Oce
Σ m
m
1p d Tm
1 m
m
A.14 OUR DISTINCTIONS FROM RELATED META-LEARNING THEORETICAL WORKS
Although there is no theoretical work on meta IRL, there are theoretical works on general meta- learning problems that study convergence and generalization. In specific, (Fallah et al. 2020) studies the convergence of MAML, (Fallah et al., 2021b) studies the generalization of MAML, and (Denevi al. 2019) studies the generalization of 1.MMLLL In this section, we discuss our significant distinc- tions from these works from three perspectives: problem, algorithm, and theoretical analysis.
Problem: Neither MAML nor iMAML can be directly applied to solve our problem even we reduce their general problem formulation to the context of IRL. Therefore, we propose the novel problem formulation
learn both the reward and cost meta-priors.
Algorithm: A key step of both MAMI and iMAMI is to compute the eyper-gradient In MAMI., the hyper-gradient is assumed be directly computed. iMAMI., they use conjugate gradient to help compute the hyper-gradien, In our case, we have an additional challenge to compute the hyper- gradient, i.e., Challenge (i) mentioned in Subsection 3.1 that does not occur in MAML or iMAML. To solve this new challenge, design additional algorithm (i.e., Algorithm 2 that uses the first-order approximation (6)-77 Moreover, instead of using conjugate gradient to solve Challenge algorithm (Algorithm
B to solve it. Compared to iMAML (Rajeswaran et al., 2019) that assumes to find a 8' "'-appoxxmmate. Algorithm enables us to guarantee the finite-time approximation error in Lemmal [ and Lemma]
(ii) mentioned in Subsection 3.
Theoretical analysis: We first talk about the convergence guarantee and then talk about the gen- eralization analysis. For the convergence guarantee, first, we have additional algorithms (i.e., Al- gorithms 3) to tackle the additional challenges and thus we need additional analysis (Subsections to justify Lemmas 2 Note that Lemmas T 2 and the corresponding proof are novel com-
A.7
A.8
pared to Fallah et al. 2020 2021b Denevi et al. 2019). Moreover, for the proof of Theorem [ in Subsection A.9, we use a novel technique prove the convergence, i.e., we first construct new function Si and provide the convergence of ft. Then the convergence of L, can be derived by using the error between J; and T.,. This technique can simplify the analysis and is totally different from
Fallah al. 2020).
For the generalization analysis, Proposition[]ae and its proof is novel compared to (Fallah et al. 202 lb Denevi et al. 2019) since we leverage unique property (i.e., distance between tasks and bounded gradient of the parameterized models re and cw) of our problem to prove this. Similarly, Theorems
3 require leverage the special property of our problem to prove. Theorem 4 quantifies the
cumulative reward and cost difference between the adapted policy and the expert policy, which is definitely novel.
35

---

Published as a conference paper at ICLR 2024
A.15 ET AL. 2023)
COMPARISON TO META CONSTRAINED REINFORCEMENT LEARNING (KHATTAR
First, (Khattar al. 2023) studies different problem. (Khattar et al. 2023) studies meta con- strained RL where the constraint (cost function) is given while we study meta ICRL where the cost function needs be learned.
For dealing with the constraint, (Khattar et al. 2023 has a constrained RL problem to solve for each task. It uses a constrained RL algorithm called CRPO etu al. 2021) solve the constrained RL problem and get corresponding task-speciff policy. We have a cost learning problem in the lower level and we use gradient descent to obtain corresponding task-specific cost adaptation. One similarity between ((Khattar et al. 2023) and our work is that we both do not require the exact task- specific adaptation and this makes the theoretical analysis of the meta learning performance more challenging.
For dealing with meta-learning, (Khattar et al., 2023) studies an online setting where at each online iteration, new task is input and a corresponding task-specific policy adaptation is computed. At each online iteration, it updates the policy meta-prior by minimizing the KL divergence between the policy meta-prior and the current task- policy adaptation via one multiple online gradient descent steps. In contrast, we utilize a bi-level optimization framework where we learn the meta- priors in the upper level such that the corresponding task-specific adaptations can maximize the likelihood of the demonstrations of each task. In order to optimize for the meta-priors, we need to compute the hyper-gradient which very challenging in our case. We propose several novel approximation methods and algorithm designs to approximate the hyper-gradient. In conclusion, the meta-prior in our case is learned such that the task-specific adaptations adapted from the meta- prior have good performance on each specific task while the meta-prio in (Khattar et al., (023) is learned such that the meta-prior is close to task-specific adaptations according to the metric of KL divergence.
B EXPERIMENT DETAILS
This section includes the experiment details. It has two subsections where the first subsection in- cludes the experiment details of the drone experiment and the second subsection includes the exper- iment details of the Mujoco experiment. We first explain the four baselines in detail.
The baseline ICRL does not have meta-priors and directly learn from one trajectory from scratch.
The baseline (pRe( naively learns meta-priors across all demonstrations of all the training tasks. In specific, ICRL(pre) first solves ming m
1 -m (, w0 (/0==0(0 (0), D:), s.t. (0) arg min 1 2 G,(w; 0, Dj) where D; A {D], ). The obtained results of this problem (i.e., and w0))) are the meta-priors of ICRL(pre). ICRL(pre) then uses these meta-priors ming Lm41 (0,,1(00 (0), Dm+1), wn (0) arg min Gm+i(w;0, 0, Dm+1) for arbitrary new task Tm+--
m
initializations to solve the problem
The bascline Meta-IRL is from (Xu et al. 2019) which is a combination of maximum entropy IRL (Ziebart et al.) 2008 and MAML (Finn et al. 2017).
B.1 DRONE NAVIGATION WITH ORSTACLES
RL has been applied to many applications, including wireless network Huang et al. (2023) and motion planning [iu & Zhu (2024). Here, we study a motion planning problem. For the drone experiment, we cannot directly train the algorithm on the physical drone because this may cause damage to the drone. In specific, given learned reward and cost parameters (0,w), we need to use soft Q learning soft actor-critic to find the corresponding constrained soft Bellman policy. This RL step requires the drone to interact with the environment and thus improves its policy. During the learning process, the drone may inevitably execute some dangerous behaviors, such colliding with obstacles or the wall. To avoid the damage of the drone, we build a simulator in Gazebo (Figure 2 that imitates the physical environment with the scale 1 1. We train the algorithm the simulated
36

---

Published as a conference paper at ICLR 2024
drone in the simulator and the empirical results, i.e., CVR and SR, counted in the simulator. Once we obtain a learned policy that has good performance in the simulator, we implement the policy the physical drone.
Discussion of the sim-to-real problem. In some cases, the models that have good performance in the simulator may not have good performance in the real world due to the reason that the simulator cannot 100% precisely imitate the physical world Sun al. (2023); Wang Cao (2024). However, in our case, the sim-to-real issue is not significant because of two reasons: (i) the simulated drone is built according to the dynamics of real Ar. Drone 2.0 (Huang & Sturm 2014); (ii) the states and actions are just the coordinates of location and heading direction of the drone instead of some low-level control such as the motor's s velocity, etc. Given that Vicon can output precise scale, If learned trajectory can succeed in the simulator, can succeed in the real world given that the low-level control both the simulated and physical drones given.
pose of the physical drone and the simulator is built on the
In this experiment, the state of the drone its 3-D coordinate (xr, y, 2) and the action of the drone is also 3-D coordinate (dz, dy. dz) which captures the heading direction of the drone. We fix the length of each step as 0.1 and thus the next state is dx
Figure 2: Simulator
(x+ 10 +dd
dy V+ (ddd 2+
In this experiment,
10 ((cyd=))=icc))=(dd33
we do not need the drone change its height so that we usually fix the value of and set d2 = 0. The goal an x square. Denote the coordinate of the center of the goal as (Iggua:: Mgeat), then for 11). The obstacle is 3 1 square. Denote the coordinate of the lower left end the obstacle as (Tobstacle: yobstacte), the for the different tasks,
all the different tasks, Fgoal (0.5, 6.5) and Vgoal (10,11).
Tobstacle E 4) and Yobstacle (),
Note that we do not need features to help learn the reward function. Even if features can be learned Wu et al. 2 Chen et al. (002),, the extra requirement of needing features be impractical in various scenarios. We use neural networks to parameterize the reward and cost functions. In specific, the neural networks have
layers where the activation functions are relu and each layer has 64 neurons. For each training task, the training set only has one demonstration and the evaluation set has 50 demonstrations. We D Deval The result table shows the mean and the standard deviation over the 10 training tasks.
B.2 MUJOCO EXPERIMENT
(a) Swimmer
(b) HalfCheetah
(c) Walker2D
Figure The three Mujoco environments.
For all the three Mujoco experiments, we consider the random velocity task (Seyed Ghasemipour (et al. 2019). In specific, the robots (i.e., swimmer, halfcheetah, and walker) need to reach and sustain at target velocity. For different tasks, the target velocity is different. The design of the reward function that the robots will receive reward +1 if they are at the target velocity and reward 0 otherwise. The neural networks of all the three experiments have two hidden layers, and each layer has 256 neurons. The activation function of the first hidden layer is relu and the activation function of the second hidden layer is tanh. For all the three Mujoco experiments, have 50
37

---

Published as a conference paper at ICLR 2024
training tasks and 10 test tasks. Each test task only has one demonstration. For the training tasks, the training set has one demonstration and the evaluation set has 64 demonstrations. Similar to the drone experiment, set D D.
The target velocity of the three experiments is designed in a similar way. In specific, the target velocity
[0.5, 3.5]. For given target velocity e, the reward is +1 if the velocity is within [v. u++ + 0.3] and O otherwise. For the constraint, the Swimmer experiment constrains all the states whose front tip angle is larger than do where ag [0.9, 1.2]. The HalfCheetah experiment constrains all the states whose front tip height is larger than ho where hn E [0.5, 1.0]. The Walker experiment constrains all the states whose top height is smaller than hı where h1 [0.6,0.35
Discussion of the experiment results in Table 1 From Table ! we can observe that M-ICRL achieves similar SR/RR and CVR with the expert. ICRL has much worse SR/CR performance because it only has one demonstration for each test task, and it does not have meta-priors, MeL. has the worst CVR performance because it only learns a reward function and it difficult for single reward function to capture the function of both the ground truth reward function and ground truth constraints. The bad CVR performance will also result bad performance of SR and CVR even Meta-IRL may estimate the ground truth reward function well, because the episode terminates if any constraint is violated. (CRL(pre) has the second-bee bentt performance because it has abundant training data and it learns both the reward function and constraints. However, (CRL(pre) has much worse performance than M-ICRL because ICRL only naively trains over all the data of all the training tasks.
38


Reviews:
[{"review_id": "Wh47hhszE6", "reviewer": "Reviewer_5qhh", "summary": "This paper extends the inverse constrained RL problem studied in (Liu & Zhu 2022) to the meta-learning setting where authors propose to first learn meta priors over reward/cost functions from similar tasks and then adapt the priors to new task via few-shot learning. This problem is then formulated using the bi-level optimization which is intractable in general. Authors propose novel approximate methods to solve the formulated bi-level optimization problem and quantify the approximation errors. Both physical and numerical experiments are conducted to demonstrate the effectiveness of the proposed method.", "questions": "1. In terms of novelty, it would be helpful for readers if authors can emphasize the unique novelty and challenges of solving meta inverse constrained RL beyond simply combining the techniques in the inverse constrained RL problem studied in (Liu & Zhu 2022) and the meta RL/ICRL studied in (Xu et al 2019; Rajeswaran et al 2019)? \n\n2. There is a related work \n\n\"A CMDP-within-online framework for Meta-Safe Reinforcement Learning, Vanshaj Khattar, et al, ICLR, 2023\" \n\nwhich studies the meta learning for the constrained RL. What is the similarity and differences between authors' methodology compared with their works in terms of how to deal with constraints and meta-learning? It is important to add such comparisons in the paper. \n\n3. Can authors further explain why the regularization term ||\\eta - \\omega||^2 in equation (3) is required for the M-ICRL problem?\n\n4. The formulated M-ICRL is a complicated formulation, and the proposed approximate methods further increases the complexity of the algorithm. Can authors summarize the tricks in the implementation level to achieve the reported good results? This is helpful for the future researchers who want to extend this paper.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The author proposed a new setting which is based on the inverse constrained RL problem (Liu & Zhu 2022) and the meta RL/IRL (Xu et al 2019). It is a creative combination of existing ideas.\n\n2. The paper is well-written with the main ideas, algorithms, theories and experiments well presented.\n\n3. The demonstration using the physical experiment on drone navigation makes the proposed method more convincing.", "weaknesses": "See questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aeFh67FgF9", "reviewer": "Reviewer_NXcx", "summary": "In this work, the authors optimize the ability of inverse constrained reinforcement learning (ICRL) to learn the reward and constraint(s) for a new task by meta-learning reward and constraint function priors over multiple similar tasks. The ICRL problem is formulated as a bi-level optimization, as proposed by [1], and then the authors provide a way to meta-learn the priors efficiently through empirical approximations and iterative techniques. ICRL is a growing field, and being able to perform ICRL more efficiently across new tasks is definitely relevant and useful. The proposed approach could be slightly mis-formulated (as explained later in weaknesses), but is theoretically well-analyzed and has promising experimental results.", "questions": "1. While Equation 12 (Appendix A.1) is the dual of Equation 11, if the domain is extended from linear constraint functions to non-linear constraint functions, the equation would no longer behave as the dual of the original problem as formulated in Equation 11, right? Does it make sense to use this as the lower level problem, in that case?\n2. (Suggestion) more ablations (eg. empirical effect of $\\alpha$, batch size $B$, number of gradient descent steps $K$, etc.) can be performed. Also, are these values specified somewhere in the paper?\n3. (Suggestion) Notation can be slightly confusing at some places, so I would suggest mentioning in the algorithms what the inputs and the outputs are, in words (eg. meta-priors, etc.). Implicit hyperparameters should also be mentioned in the beginning of the algorithm, eg. $\\alpha$, $B$, etc. what do these refer to, in the algorithm?\n\n**References**\n1.  Distributed inverse constrained reinforcement learning for multi-agent systems, Liu & Zhu (2022)\n\n**Updates**\nIncreased score from 5->6 (22 Nov)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Extensive theoretical analysis about the approximation errors and (approximate) convergence result\n2. The baselines clearly demonstrate the effectiveness of the approach since the proposed method can learn a good meta-prior over several tasks, and thus performs well when tested later\n3. Successful real world experiments", "weaknesses": "1. The lower level optimization (as explained in Equation 11, Appendix A.1) maximizes the reward and causal entropy objective while matching the constraint feature expectations. On the other hand, the authors of [1] formulate the optimization as a maximization of the causal entropy while matching reward feature expectations and expected cumulative costs. The slight difference is that [1] has an additional Lagrange multiplier term $\\lambda$ in the dual formulation (see [1], page 4, below remark 2, in the definition of $G$). In this work, since the constraint feature expectations are directly matched, the overall form of the lower level objective $G$ (Appendix, equation 12) is not exactly the same as [1]. This is also apparent in the constrained soft Bellman policy definition adapted from [1] (no $\\lambda$ in the constrained policy for this work, whereas there is a $\\lambda$ in the constrained policy as described in [1], page 2, Appendix). The outcome is that the constraint function is just treated as a negative reward term that can be just added to the original reward and thus, constrained RL just amounts to running RL with reward $r-c$ (Appendix A.2 of this work also says this). I am not sure this is representative of typical constrained RL problems, since typically it is not possible to rewrite a constrained RL problem as an unconstrained RL problem with a different reward. (maybe the authors can elaborate on this)\n2. What if several demonstrations are available for each task? If more demonstrations are available, ICRL could perform better and the gap between M-ICRL & ICRL could be lesser. M-ICRL should still perform better, since it has a better meta-prior, but overall it would be useful to understand the empirical improvement of M-ICRL over ICRL as the number of demonstrations vary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KzHWc57TGY", "reviewer": "Reviewer_byVV", "summary": "The paper studies the meta inverse reinforcement learning problems in the constrained MDP. The paper proposes an approach that first learn meta-priors over reward functions and constrains from related tasks and then adapt the learned prior to new tasks with few expert demonstration. The problem is formulated as a bi-level optimization problem, where the upper level learns the prior on the reward functions and the lower level learns the prior on the constraints. The paper shows that the algorithm reaches the $\\epsilon$-station points at $O(1\\epsilon^2)$ and quantify the generalization error to new tasks. The theoretical results are supported empirically.", "questions": "See weaknesses for details.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a theoretical framework for  meta inverse constrained reinforcement learning, which extends previous study on meta \n inverse reinforcement learning and inverse constrained reinforcement learning.\n- The theoretical study is solid. The paper shows the convergence guarantee of the proposed algorithm. Then, the paper studies the generalization error for a new arbitrary task.\n- The paper provides empirical study on the algorithm in two settings: navigation with obstacles in AR and mojoco experiments, showing that M-ICRL performs better than other methods.\n- The paper provides a clear presentation on its study, with a detailed discussion on the challenges and the approach.", "weaknesses": "- The paper is an extension of meta IRL and ICRL. Building upon these, the contribution of the study is not significant.\n- The paper only shows the convergence to $\\epsilon$-FOSP, whose optimality is not discussed. Convergence of gradient-based algorithms to stationary point is not a novel contribution.\n- Assumption 1 assumes that the reward function and the cost function are bounded and smooth up to the fourth order gradient, which is a strong assumption, especially for neural networks with ReLU activation and unbounded state-action space.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8awB1tTEc0", "reviewer": "Reviewer_Eje2", "summary": "This paper addresses the challenge of inferring both reward and cost functions from expert demonstrations within a meta-learning environment. Specifically, the environment comprises a set of tasks, each with its unique rewards and constraints. To aid the inverse learning process, every task is paired with a demonstration dataset, capturing the actions of expert agents. The objective is to derive meta-priors for both reward functions and constraints that can be readily adapted to new tasks.\n\nTo tackle the meta Inverse Constrained Reinforcement Learning (ICRL) problem, the paper extends the bi-level optimization framework proposed by Liu & Zhu (2022). In this model, the upper-level problem focuses on reward learning, while the lower-level problem tackles constraint learning. Following this framework, the paper introduces an algorithm that estimates gradients at multiple levels. Due to the computational challenges associated with calculating the inverse-of-Hessian term (or hyper-gradients) and solving the exact lower-level problems, a set of approximation methods is proposed. The primary techniques include 1) a first-order approximation to second-order gradients, 2) substituting the computation of the inverse-of-Hessian term with an optimization problem solution, and 3) approximating the solution to the lower-level problem with multiple iterations of gradient descent.\n\nTo validate the convergence of the proposed algorithm, this paper extends the results of the ϵ-approximate first order stationary point (ϵ-FOSP) from Fallah et al. (2020) to its context. Specifically, it demonstrates that the norm of the empirical loss estimation is bounded upon convergence (with optimal estimate). Furthermore, this paper derives an upper bound on the empirical loss estimation for new tasks, reinforcing the generalization performance of the proposed method.\n\nThe empirical study primarily focuses on the physical drone setting and the Mujoco setting. These settings are used to substantiate the empirical performance of this method.", "questions": "1. Line 4, Algorithm 1: It's unclear what the second $\\mathcal{D}_i^{tr}$ in the gradient is intended for. It appears that the final parenthesis doesn't correspond to any elements in the algorithm.\n\n2. Why does objective (1) incorporate a discounted log-likelihood? This suggests that the policy in later time steps has less impact on the likelihood objective. It raises the question: Would a Markov Decision Process (MDP) with a finite horizon be more consistent with the current objective?\n\n3. The $\\delta$ present in gradients (6) and (7) is not defined anywhere in the paper. It would be beneficial to provide an explicit definition or reference for this term.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-structured and written, with many details provided to aid readers in understanding the principal contributions and the main claims. The notations are carefully designed and well-defined. \n2. The algorithm proposed is well presented and generally straightforward to understand, which speaks to the clarity and precision of the authors' exposition.\n3. The theoretical results appear sound overall. The requisite assumptions underlying the main proof are well conveyed. References to key methods appear comprehensive, though it's not entirely clear whether the convergence and generalization outcomes are consistent with and comparable to the principal results under the meta Inverse Reinforcement Learning (IRL) or Reinforcement Learning (RL) frameworks. In RL, the analysis of regret or sample complexity is typically expected, yet the theoretical results in this paper predominantly cater to gradient-based methods.\n4. The empirical results underscore the superior performance of the M-ICRL in comparison to other benchmarks. This highlights the practical efficacy of the proposed approach in real-world scenarios.", "weaknesses": "1.  **Novelty**. Liu & Zhu (2022) has proposed a bi-level optimization objective for ICRL. This paper expands upon that proposal, adapting the objective to fit within a meta-learning context. The primary innovation is the integration of a meta-prior into the initial objective (see objective (2)). The algorithm presented outlines a direct, gradient-based optimization method, which, unfortunately, is computationally infeasible under most circumstances. The core focus of this paper is on *mitigating this computational difficulty* through the implementation of three levels of approximation. Additionally, it explores the impact of these approximations on both convergence and generalization. However, it should be noted that this study deviates somewhat from the original goal of Meta ICRL. Moreover, the advancements it provides over Liu & Zhu (2022) could be considered somewhat marginal.\n\n2. **Theoretical Contributions.** I have several concerns about the theoretical results. \n- Theorem 1 sets an upper bound on the expected gradient of the exact optimal solution. When compared to other more frequently used results, such as the upper bound of regret or the sample complexity in the PAC (Probably Approximately Correct) analysis, this result doesn't appear particularly robust.\n- It is surprising to note that the upper bound is raised to the power of $K$ (the number of gradient steps), rather than being linear or sublinear with respect to $K$. This implies that the approximation error could accumulate rapidly with each gradient step. Unfortunately, the paper does not provide a satisfactory explanation for this issue.\n- I question the assertion that this upper bound can be regarded as an ϵ-approximate first-order stationary point. While the construction of ϵ being proportional to $\\sqrt{1/N}$ is reasonable, the inclusion of an additional Big O term seems less comprehensible. It is my belief that the approximations exert a significant influence on convergence, which should not be overlooked.\n\n3. **Bi-level approximation.** In Liu & Zhu's (2022), the bi-level optimization framework of ICRL is structured such that the upper-level problem was focused on reward learning, while the lower-level problem addressed constraint learning. In this current paper, the upper-level problem has been expanded to include the learning of meta-priors for both rewards and constraints ($\\theta$ and $\\omega$). This shift deviates from the original bi-level optimization framework of ICRL, raising some questions about its consistency. Moreover, the paper proposes that the task-specific reward adaptation is carried out via a gradient update. However, it is unclear whether this is a well-defined adaptation for specific tasks. There is a concern that this adaptation may be inadequate. It would be beneficial to provide additional supporting evidence. This could include referencing relevant papers where similar strategies have been successfully applied.\n\n4. **The Audience of the current paper.** My final concern pertains to the intended audience of this paper. The manner in which the content is presented seems to diverge from the primary interests of the mainstream Reinforcement Learning (RL) and Machine Learning (ML) community. This paper primarily focuses on the approximation of computationally intractable gradients and the subsequent implications for convergence and generalization. While these advancements are valid, they may not be entirely consistent with the broader ML community. Furthermore, it's unclear how subsequent work can leverage and benefit from these methods. In my view, the style of this paper aligns more closely with the interests of the optimal control and robotics community. Subjectively speaking, ICLR may not be the most appropriate venue for this paper.\n\n5. **Empirical Results** There are several concerns about the empirical results.\n- The environmental parameters in the MuJoCo experiments raise some questions. The paper states (see Appendix B.2) that \"The design of the reward function is such that the robots will receive a reward of +1 if they maintain the target velocity and a reward of 0 otherwise.\" and \"For the constraint, the Swimmer experiment constrains all the states where the front tip angle exceeds a0, where a0 ∈ [0.9, 1.2].\" These constraints do not appear to be significant. In particular, it's unclear why the agent would need to violate the constraint to maximize cumulative rewards, and whether maintaining the correct speed is a suitable reward for a MuJoCo task. More justification for these settings is required.\n\n- The results for Meta-IRL are counterintuitive. One would expect that imitation learning methods like IRL, which do not model constraints, would yield higher reward and constraint violation rates. The presented results do not align with this understanding. Please provide an explanation.\n\n- There's a considerable gap between the performance of the baselines and the expert performance. The lead of M-ICRL is substantial, suggesting that all the baselines fail in the task. Without careful design, the validity of these baselines could be called into question. More details or improvements on the baseline design are suggested.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)