Performance Bounds for Model and Policy Transfer in Hidden-parameter MDPs
Authors: Haotian Fu
Jiayu Yao
Omer Gottesman
Finale Doshi-Velez
George Konidaris
Venue: ICLR 2023 poster (2023)
Keywords: Reinforcement learning
Meta learning
Transfer learning
Theory

Abstract: In the Hidden-Parameter MDP (HiP-MDP) framework, a family of reinforcement learning tasks is generated by varying hidden parameters specifying the dynamics and reward function for each individual task. HiP-MDP is a natural model for families of tasks in which meta- and lifelong-reinforcement learning approaches can succeed. Given a learned context encoder that infers the hidden parameters from previous experience, most existing algorithms fall into two categories: $\textit{model transfer}$ and $\textit{policy transfer}$, depending on which function the hidden parameters are used to parameterize. We characterize the robustness of model and policy transfer algorithms with respect to hidden parameter estimation error. We first show that the value function of HiP-MDPs is Lipschitz continuous under certain conditions. We then derive regret bounds for both settings through the lens of Lipschitz continuity. Finally, we empirically corroborate our theoretical analysis by experimentally varying the hyper-parameters governing the Lipschitz constants of two continuous control problems; the resulting performance is consistent with our predictions.

Published as a conference paper at ICLR 2023
PERFORMANCE BOUNDS FOR MODEL AND POLICY TRANSFER IN HIDDEN-PARAMETER MDPS
Haotian Fu', Jiayu Yao", Omer Gottesman', Finale Doshi-Velez & George Konidaris' Brown University, 'Harvard University
ABSTRACT
In the Hidden-Parametee MDP (HiP-MDP) framework, family of reinforcement learning tasks is generated by varying hidden parameters specifying the dynamics and reward function for each individual task. The HiP-MDP a natural model for families of tasks in which meta- and lifelong-reinforcemeen learning approaches can succeed. Given learned context encoder that infers the hidden parameters from previous experience, most existing algorithms fall into two categories: model transfer and policy transfer, depending on which function the hidden parameters used parameterize. We characterize the robustness of model and policy transfer algorithms with respect to hidden parameter estimation error. We first show that the value function of HiP-MDPs Lipschitz continuous under certain conditions. We then derive regret bounds for both settings through the lens of Lipschitz continuity. Finally, we empirically corroborate our theoretical analysis by varying the control problems; the resulting performance is consistent with our theoretical results.
governing the Lipschitz constants of two continuous
INTRODUCTION
Hidden- Hidden-parameter parameter Markov Decision Processes (HiP-MDPs) (Doshi Velez & Konidaris 2016) describe family of related tasks by modeling task variations with set of low-dimensional hidden parameters. HiP-MDPs are widely used in recent meta- Reinforcement Learning (RL) and lifelong RL works, in which an agent needs to quickly adapt to new tasks by transferring the knowledge from previous tasks. To solve HiP-MDPs, most existing algorithms first infer the hidden parameters from previous experience (e.g. by learning a context encoder that maps trajectories to hidden parameter estimate 0), then use the estimated hidden parameters to solve new tasks with either [
(8). / S2.... "")
Context encoder
((f)
Sample
&& R
model (Killian et al. 2017 Lee et al. 2020 Fu et al., 2022
policy transfer (aoo et al.) 2018 Rakelly et al. 2019
algorithms. As shown Figure 1 in model transfer, the
Planner
inferred hidden parameters are used build simulator of the environment, which can then be used for planning. In policy transfer, the inferred hidden parameters are used parameterize the policy of the new task directly. Previous works have observed mixed empirical evidence for when each approach performs better (ao et al. 2018 Lee et al. 2020 Fu et al. 2022). In this work, we take a step in the theoretical direction by studying the regret bounds of model and policy transfer algorithms, respectively, which helps characterize the robustness of these algorithms.
à
Policy Transfer
Model Transfer
Figure 1: Difference between model and policy transfer in HiP-MDPs.
Two main factors affect the performance of HiP-MDP algorithms: (1) the estimation accuracy of the hidden parameter used as an input to either the learned model or policy, and (2) the quality of the learned model or policy (i.e., given accurately estimated hidden parameter, whether the agent reach the optimal performance). In practice, the HiP-MDP tasks considered most recent meta- Rett--RL papers (e.g., changing the physical properties of MuloCo-simulated robots) are relatively easy to

---

Published as a conference paper at ICLR 2023
solve when the hidden parameters are given. We therefore assume the learned model and policy optimal and theoretically analyze how hidden parameter estimation error affects the final performance of model and policy transfer algorithms.
Our contributions are as follows. We first derive the conditions under which the value function of HiP-MDPs is Lipschitz continuous. We then derive upper bounds for regret in model and policy transfer algorithms, respectively, when the estimation error of the hidden parameters is bounded. We also give an upper bound for multi-step prediction error in model transfer. We further prove that the derived bounds are tight when the transition dynamics, reward function and policy are linear and deterministic. As far we are aware. these are the first theoretical results about policy and model transfer performance bounds in HiP-MDPs. Given the same hidden parameter estimation eπror bound, our results characterize when model or policy transfer can be more robust than the other: a slower increase in the Lipschitz constant with respect to the hidden parameter implies more robustness.
In addition to theoretical analysis, we empirically study the performance of model and policy transfer algorithms in two continuous control domains. For each domain, we control the Lipschitz constants of the HiP-MDPs by altering the with our theoretical understanding of how the regrets of model and policy transfer algorithms scale with respect to the estimation error of the hidden parameters: a slower increase in Lipschitz constant with respect to the hidden parameter implies a smaller performance decay.
of the environments. The results are consistent
2 BACKGROUND AND RELATED WORK
HiP-MDPs model the variations in the transition dynamics and reward functions by assigning each task hidden parameter 0, drawn from the distribution Pg. The agent neither observes 0 nor has access to the the distribution Pg that generates the task family. For a given task, parameterized by 0 E O, the stochastic dynamics are given by ['s' Is, a; 0) and the deterministic reward function by R(s, a; 0). We consider continuous state and action space in this work (s E S, a E A). Upon encountering a new task, the agent estimates the new dynamics, f, and the new reward function, R, by inferring a distribution (0) over the hidden parameter. our estimation of is accurate, then (0) should be close to the true distribution w()), i.e. peaking at the true 0 for this specific task. We assume the learned dependency of I' and It on 0 is accurate. Thus, if the estimation of 0 is also accurate, the agent can solve the task completely.
The HiP-MDPin is important setting widely used in recent meta/lifelong RL papers (see Appendix G). A number of approaches have been proposed to infer the hidden parameter of HiP-MDPs in the latent representation space, such using Bayesian models leverage prior knowledge (Killian et al. 2017; Yao et al.) 2018 Fu et al., 2022), or training a context encoder that maps trajectories to the latent parameter (Rakelly et al. 2019 Zintgraf al. 2020 Lee et 2020
Given the inferred hidden parameters, various RL algorithms have been used to adapt to the new tasks. The downstream RL algorithm given the inference of the hidden parameter includes training an off-policy actor-critic that takes in the estimated parameter an additional input (Rakelly et al. [akoor et al., (2020) Fu al., 2021). or leveraging recurrent neural networks (Zintgraf et al. 2020 Duan et al. 2016). or planning based on the learned transition model that also takes in the inferred parameter (Lee et al. 2020; Mendonca et al., 2020). Yet, in practice, there is no clear winner among these algorithms, leaving the question that what factors affect the performance of different algorithms, and how. Nair & Doshi beshi-Velez)studye a similar problem but in a different setting where the hidden parameters are given (contextual MDP) and derive sample complexity bounds for model-based learning. In this work, we quantify how errors and uncertainty in estimating 8 affect the performance of model and policy transfer methods through the lens of Lipschitz continuity.
Given a distance metric on the space M, dy. Lipschitz continuity quantifies the smoothness of a function follows.
Definition 2.1. A function f: Mi + M2 is uniformly Lipschitz continuous if
(f(x),
K KM Mr = sup 21
(1)
d (I1, 122)
2

---

Published as a conference paper at ICLR 2023
We use / in some places for simplicity of notation. Throughout the work, we use the Euclidean distance for d in the case of point vectors and the Wasserstein distance in the case of probability distributions. The main goal of this work can now be stated as deriving the relation between ((w(0), (0)) S c, and the performance of model transfer and policy transfer methods.
The Wasserstein Metric can be linked to Lipschitz continuity using duality (Villani 2008):
/ (U(T)w (x) - ((r))2()))d sup
W (2)
J:Ky My Ma <1
Given distribution (0) over the hidden parameters, we define the generalized transition, reward and value functions in HiP-MDPs as:
Ts a. = a, (W))dd, Rg( I. R(s, a; ()w(O)do,
(g(sww)) = -f V(s, (O))d (,(s,a,w) = @(ss,( a, ()w(0)
These generalised functions are an expectation over the uncertainty in the hidden parameter. All the Q and V functions we use below refer to the generalized function Qg and Vg.
We are now equipped to formally state all the smoothness assumptions we make about the HiP-MDPs: Definition 2.2. A HiP-MDP is (K Kÿ KĀ. K1. Ky. k.)) Lippshitz if, for all E Ω and S and a E A: ( (82, a, w))l S (81, 82). W (-1s,, T, a. S Kidg(81,89 82) ()) a2 w))l < KA 02), W (I) a2, (2, - KaddA(a1, 02) (R,(s, Rg (s, a, w2)l - ( W2). W (1s. a.)) - (w1.
To simplify notation, we will write that a HiP-MDP is Lipschitz continuous when all the constants can be implied from the context.
In the following theoretical analysis, we assume the HiP-MDPs are Lipschitz. Intuitively, this implies that when the hidden parameters of two MDPs are close, their transition dynamics and reward functions are similar, and for given task, transitions dynamics and rewards do not change much given small changes in the states and actions. Similar assumptions have commonly been made in previous works (Hinderer, 2005 Asadi et al.) [018 Gottesman et al., 2021] Gelada et al.) 2019 Ren ctt al. 2019) 9. Luo et al. 2019). We extend the smoothness assumption to the hidden parameters for better understanding about the role of hidden parameter estimation error in the final performance.
In addition to the smoothness assumptions about the HiP-MDPs, we make the following assumptions about the smoothness of policies.
Definition 2.3. policy T(a s, (0)) is (K K Lippshhtt continuous if:
W ( ((2)),
K = sup sup
ds(s1, 82)
81.82
W wi (0)), ((1ss (0))
K = sup sup wi
W W(w (6), wyy (f))
are finite.
We evaluate policies by calculating their expected discounted reward given a distribution of initial states, (s)):
J Vi (s)))))))(ssss
(3)
Lipschitz continuity in MDPs has been studied by many previous works (Hinderer 2005 Rachelson Lagoudakis 2010; Tang et al. 2020 Gottesman et al. 2021). Pirotta et al. (2015) propose to use Lipschitz continuity property of MDPs speed up policy gradient algorithms. Asadi et al. (2018) show how the magnitude of the Lipschitz constant of the transition dynamics affects the performance of model-basee RL algorithm's 's performance is influenced by the magnitude of the Lipschitz constant of the model. Gelada et al. (2019) leverages Lipschitz continuity assumptions in regular MDPs to learn state abstractions. However, none of them conduct theoretical analysis HiP-MDPs.
3

---

Published as a conference paper at ICLR 2023
3 LIPSCHITZ VALUE FUNCTION OF HIP-MDPS
Our first main result is showing under which conditions the value functions are Lipschitz continuous for HiP-MDPs. We show this through a combination of Theorem 3.1 and Theorem 4.3 & 5.2 in the following sections. Theorem 3.1 mainly studies the Lipschítz continuity with respect to S and A while the hidden parameter is fixed. This can be seen as generalization of theorems derived in prior papers (Rachelson & Lagoudakis 2010 Pirotta et al. 2015; Asadi et al. 2018 Gelada et al. 2019) that study Lipschitz value function for regular MDP.
policy n, if 7KK + - Rt: Kÿ, K
Theorem 3.1. For a (KS
f. KG Ki: K -Lipschitz HiP-MDP with (K) K2 -Lipsshhit
()) <
the value function Lipschitz continuous with respect to S and
A with constants bounded by:
Q(s, a) Q(s, a2,w)) w)1
K 7( - K
KỘ= == sup sup sup
da a2)
(Kg. + K5
01
Q(1 a.w)
(8,,
K 7 (K K K
K == sup sup sup
ds(s1, 82)
(Kg + (イ
w
$1
(S1,,W) V (32,WW)
ky- = sup sup
< KQ² + KQ- K
ds(s1,82 82)
$1,82
Proof. (sketch) We derive the value difference bound by decomposing it into the combination of the Q-value difference with respect S & A using our previous definitions and the Bellman update rule (9). Then we get the bounds for K2 and K&= respectively by applying the dual form of the Wasserstein Metric (Equation 2 and with the fixed-point iteration. See appendix Foro all proofs. D
The above theorem shows the conditions under which the value functions of HiP-MDPs are Lipschitz continuous with respect to states and actions. We can get the following corollary when the policy is optimal:
Corollary 3.2. If the policy ι is optimal and 7 yKG < 1, then:
K$. K + >K - KK
KS S
1
Note that in contrast to K&., K&- is affected by the smoothness of the transition dynamics and reward functions with respect to action space (KA and K
Theorem 3.1 and Corollary 3.2 also directly apply to a (KS, K.S KÅ, -<isshipc regular MDP with a K Lipschitz policy, as the Lipschitz continuity assumptions with respect to hidden parameters are not used in both the final results and proofs. Specifically, the upper bound for KS. in Corollary 3.2 recovers the results in Asadi et al.) (2018): Gelada et al. (2019). If we assume the Lipschitz constants for state and action are the same, we get Corollary E. which recovers the results in RRacelson & Lagoudakis (2100); Pirotta et al. (2015). For regular MDPs, our results are more general compared to prior work as (1) do not assume that the Lipschitz constants for state and action are the same because that is usually not the case in practice; (2) our theory applies to all Lipschitz policies rather than only the optimal policy.
One of our major goals in this work is to derive the conditions for Lipschitz value functions of HiP-MDPs. Theorem B.I shows that the value functions in HiP-MDPs are Lipschitz continuous with respect to S and A under certain conditions. To complete the theorem for Lipschitz value function in HiP-MDPs, we must also explore the conditions under which the value function is Lipschitz continuous with respect to the remaining input of Q-function in HiP- HiP-MDPs. w. As we shown in the next two sections, exploring this actually results in different bounds for policy transfer and model transfer. By leveraging the results of Lipschitz value functions, we then find how the estimation error d(w)()) (0) S 6 of the hidden parameter at test time will affect the performance of model and policy transfer algorithms respectively. We quantify the performance gap by deriving the upper bound of the regret as a function of the error in estimating the hidden parameter. Note that throughout the following analysis, we abstract out the discussion of how (0) is estimated, and assume it is given with an estimation error bounded by e.
We make this assumption in many of our following theorems. See appendix F for explanations.
4

---

Published as a conference paper at ICLR 2023
4 PERFORMANCE BOUNDS OF MODEL TRANSFER IN HIP-MDPP
We now focus on the regret of model transfer in HiP-MDPs induced by the hidden parameter estimation error, d(w(
S We assume that the agent has learnt accurate transition function T (s'| o, (()) and reward function Rg (s, a, ())). Upon encountering new task, the agent first estimates the hidden parameter and then feeds it into the learned transition function T and reward function Rg. The agent will use them to generate samples and train task-specific policy. The algorithm framework can be found in Algorithm [2 and Figure] Note that similar paradig am has been widely used (Lee al. 2020 Mendonca et al. 2020) in complex continuous control tasks.
We first derive a bound of the compounding error of dynamics prediction given the estimation error e in hidden parameter at test time, with the Lipschitz model class (Asadi et al. 2018 assumption (we include definition in appendix [E:
Theorem 4.1. Assume the estimation ervor for hidden parameter 0 is bounded by c, that is, W (w)(0),((0) S e. Further assume an accurately learned Tg induced by Lipschitz model class F with the Lipschitz constant KD and K a fixed sequence of actions oo," ,0n-1, and an initial state distribution p(s). Then Vn 1. the n-step prediction error E(n):
{(n) = W (lp,,), 1 (11) S K.
Σ [KK)
(4)
where (-(,,)) == (- (-1 To ( (-1e, au w)..., an 2, w), an-1, w) is a generalized n-step tran- sition function, and T (-1p,, R)) is defined similarly.
The result shows how the multi-step prediction error & of a generalized transition function HiP- MDP scales with respect to the hidden parameter estimation error e. Additionally, the smoothness of the transition function also affects the multi-step prediction error. As we show in the experiments, this multi-step prediction bound can have huge impact on the planning performance for model transfer.
Instead of directly planning, some methods choose further learn the policy using the learned model. We investigate the value estimation difference and regret of model transfer induced by the hidden parameter estimation error. Note that the remaining theoretical results in this paper do not involve the Lipschitz model class assumption.
Lemma 4.2. Given a HiP-MDP with learned (KO. K$) Liipshitt transition model T, reward function R, in model transfer, the generalized value function is Lipschitz continuous with respect to Ω with a constant bounded by:
K&-- I ft + KKO + +*KSK K..
We get the above bound for the Lipschitz value function when assuming the Lipschitz continuity of transition and reward function with respect to only the hidden parameter. The lemma shows the relationship between KP- and KQ- in model transfer case. Then, by further assuming the I Lipschitz continuity of dynamics & policy in Lemma we can get the following bound leveraging the dual form of Wasserstein Metric (Equation and computing the fixed point of recurrence.
Theorem 4.3. Given a HiP-MDP with learned (KS fc K KA K.A,
K K -ipssiichittz transition Ro
R0 dynamics T. reward function function R, model transfer, if 7 (KŞ K KĄ) function corresponding to the KS -Lipschitz policy (als)) is Lipschitz continuous with respect to Ω2 with constant bounded by:
< 1, the generalized value
KR + TKS
KO
(5)
Now if we further use the bounds derived in Theorem 3.1 to substitute KS. (for Theorem 5.2 we substitute KÅ-), we can get a bound of KP,- that does not depend on Kgr.
Compared to Lemma 4.2 the bound derived in Theorem 4.3 does not depend on K x by leveraging Bellman equation (the details be found in our proofs). In general, Theorem 4.3 characterizes
5

---

Published as a conference paper at ICLR 2023
how the value estimation difference in model transfer is affected by the reward/transition functions' robustness to the hidden parameter. Theorem 4.3 and 3.1 together show that the value function of a HiP-MDP for model transfer is Lipschitz continuous under certain conditions.
Then we can derive the regret of model transfer given hidden parameter estimation error e. Here we further assume the agent gets the optimal policy in the "simulated environment" created by the learned transition and reward function. We first introduce the following lemma:
Lemma 4.4. Given a (K KG K KF) -Lipschitzz HiP-MDP, for the optimal policy n². [V" (8, wi (0) - V* (2)) S max A IQ" (8, a, wi (0) Q" (s, (0))
Given the optimal policy, this lemma shows the relationship between value estimation difference and Q-value estimation difference. Combining the results in Corollary 3.2 we have:
Lemma 4.5. Given HiP-MDP with learned (KS K KR. KG) -ii-sicchitz transition model T,
&
reward function R. the optimal policy "" {als)) T and R, starting from state distribution μu ). k 0 the regret of model transfer given hidden parameter estimation ervor is bounded by:
T
KQ R
7K (1 - 71 (k(1-~kk)) C,
J
(6)
7
Besides the reward and transition functions' smoothness with respect to the hidden parameter 0, the derived bound also shows that the distance between the expected return of the learned policy A* and the environment's true optimal policy increases with K and K. given the hidden parameter estimation error e. In other words, the performance of model transfer algorithms decreases as the sensitivity of the dynamics changes in states increases, which aligns with the intuition that the model-based methods need to accurately predict state transitions and expected rewards given states.
Furthermore, for the theorems derived in this section, show that (proofs included in the appendix):
Claim 4.6. Given a linear and deterministic transition function, the bound derived in Theorem 4.1 is right. Given linear and deterministic transition futioonfuneee function and reward function, Lemma 4.5 is tight.
PERFORMANCE BOUNDS OF POLICY TRANSFER IN HIP-MDPS
We now focus on the regret bound induced by policy transfer algorithms in HiP-MDPs. We assume the error bound of the hidden parameters is the same (i.e., d(w((), (()) < <), and that the agent has learnt an accurate joint policy π [aas, w()) during training. Upon encountering new task, the agent feeds the estimated hidden parameter into the learned policy instead of the model. The agent will then directly use the joint policy to interact with the environment. The algorithm framework can be found in Algorichm| T and Figure T Similar policy transfer paradigm has also been widely used (Yao et al. 2018. Rakelly et al. 2019). We investigate how the estimation error for the hidden parameter will affect the performance of the joint policy on new task and compare with the performance bounds derived above for model transfer cases.
Lemma 5.1. Given HiP-MDP with learned K. -ipschhchtttz joint policy a[a)s, (0)), in policy transfer, the generalized value function is Lipschitz continuous with respect to Ω with a constant bounded by:
Ky K8 + KK K..
We obtain the above bound for the Lipschitz value function when assuming the Lipschitz continuity of the learned policy with respect to only the hidden parameter. The lemma shows the relationship between KG-- and K-2- in the policy transfer case. Then, similar to model transfer, if we further assume the Lipschitz continuity of the dynamics and the policy and compute the fixed point of the recurrence of the Bellman update, we can get the following bound:
Theorem 5.2. Given (K R KA K -Lipschitz HiP-MDP, in policy transfer, the generalized value function corresponding to the pretrained (K ) joint policy aaass, (()).. if (KK + K²KA) < 1, is Lipschitz continuous with respect to St with a constant bounded by
K KA
KO V <

---

Published as a conference paper at ICLR 2023
Compared to Lemma 5.1 the bound derived in Theorem 5.2 does not depend on KQ= by leveraging Bellman equation. Theorem 5.2 characterizes how the value estimation difference in policy transfer is affected by the policy's 's robustness to the hidden parameter. Theorem 5.2 and 3.1 together show the HiP-MDP for policy transfer is Lipschitz continuous under certain conditions.
value function Different from the bound in Theorem (.rrrmmm33jrt 4.33 model transfer, for policy transfer methods this value estimation error is also affected by the policy's robustness with respect to the hidden parameters, as well as the Q-value's 's robustness the actions (only with respect to states in Theorem 4.3), with no dependence on the reward and transition functions' smoothness with respect to the hidden parameters.
We can further derive the Lipschitz constant of the expected discounted reward (regret) starting from the state distribution p. Similar to what we did in model transfer case, if we also assume the learned joint policy over (0) optimal, by leveraging the results in Corollary 3.2 we have:
Lemma 5.3. Given (K KG. K --ipschhittzz HiP-MDP, pretrained - Lipschitt optimal joint policy TTals, (())), starting from state distribution u(s). y < 1, the regret of policy transfer induced by hidden parameter estimation error is bounded by:
J - J (K + + KA 7
(8)
(1-7) - 7 K
Comparing the derived performance bound with the one in Lemma 4.5 we find that the distance between the expected return of the learned joint policy and the environment's 's true optimal policy x² increases also with K and A besides K and K In other words, the performance of the policy transfer decreases as the sensitivity of the dynamics changes in both states and actions increases, while the model-base method is not quite sensitive to the dynamics change in actions. This result is reasonable policy- dbaiey method is influenced mostly by direct impact of actions. Regarding the tightness of the derived bound, we get similar results as for model transfer:
Claim 5.4. Given linear and deterministic transition function, reward function and policy, the bounds derived in Lemma 5.3 are tight.
Summary Besides characterizing the robustness of model and policy transfer methods with respect to hidden parameter estimation error separately, Lemma and [a3]a also imply that, policy transfer methods are expected to perform more robust when difference in effects of different actions is small, whereas model transfer methods are expected to perform better when neighboring states have relatively similar dynamics. Given these bounds, one direct implication to HiP-MDP algorithms in practice is that we can infer the performance trend of policy transfer and model transfer algorithms by either qualitatively or quantitatively estimating the dynamics and reward functions' sensitivity to states and actions. This can further help us determine which method is probably more advantageous for a specific HiP-MDP problem. Note that K and KP in general can be dependent on the model class (architecture) of the model (e.g. neural nets) used to parameterized the policy or model.
f
EMPIRICAL EVALUATION
We now corroborate our theoretical results in relatively large scenarios where neural networks needed to parameterize models and polices. We evaluate model and policy transfer methods on two continuous scenarios, ball-goal and ball-wind, where we can quantitatively estimate the influence of changing environment hyperparameters on different Lipschitz constants. Then by investigating whether the empirical performance changing is consistent with what we expect from theoretical results, we can corroborate whether our derived theorems match what happened in practice.
For both scenarios, the goal of the agent is to control ball to quickly reach a target position. In ball-goal, the angle of the goal direction is the hidden parameter. That is, the goal direction changes across tasks, and the agent will obtain maximum cumulative reward it figures out the right angle for the current task and keeps moving in that direction until reaching the goal. In ball-wind, the goal position is fixed across different tasks but there wind across the plane with different directions. The angle of the wind direction is the hidden parameter in this scenario. The reward function consists of a dense reward proportional to how closer the ball moves towards the goal compared to last step, a control cost, as well as larger final reward for reaching the goal. Thus for ball-goal, only the reward function is changing with respect to different hidden parameters, while in ball-wind, both reward and transition functions are changing. More details about the environments can be found in Appendix B
7

---

Published as a conference paper at ICLR 2023
For each HiP-MDP, manually calculate the value of different Lipschitz constants approximately given the actual reward and transition functions for both scenarios. We further calculate the bounds derived in Lemma 4. and Lemma 5.3 using the results. The Lipschitz constants are given in Table [ We mainly investigate the effect the step size v, goal distance g, and state accelerator m (ball-wind). The results imply how different Lipschitz constants are changing
change those environmental hyper-parametery if our derived bounds are relatively tight and close to what happened in practice.
Env
K3
K
KT
KR
KR
KR
R
K. (Model transfer)
K. (Policy transfor)
ball gool
1
0
2
2g
ball-wind
m
1
1
VBc
K
20+9)
(m)(11))
Table Lipschitz constants for ball- ball-goal and ball-wind
Ball goal (g)
Ball goal [v]
Ball wind (g)
Ball wind
7.5
22
Average Return
Average
5.0
Policy
Policy transfer Model transfer
5.0
Model transfer
34
Figure 2: First row: Average return on test tasks v.s. environmental properties (u and g). Second row: -Ky v.s. and g: how -K, j is expected to change based on our our theoretical computing results in Table Each plot in the first row is associated with two plots below with corresponding color. The performance of the algorithm is expected decrease K decreases.
6.1 EMPIRICAL RESULTS CORRESPONDING TO LEMMA 4.5 AND LEMMA 5.3
As both Lemma 4.5 and Lemma 5.3 can written in the form |Jx - Jx- < K, we investigate how K2 is changing we change different environmental hyperparameters both theoretically: manually computing the Lipschitz constants given the environment's dynamics (appendix B and the two lemmas (results shown in Table and empirically: investigating how the algorithm'' 's empirical performance is changing as we alter the environment's 's hyperparameters (K, is expected to inversely related the expected return), and see whether the theoretical and empirical results are consistent.
For empirical training, we use state-of-the-art meta-RI. algorithms PEARL (Rakelly et al.) 2019) the policy transfer method to learn the joint policy, well as CaDM (Lee et al.) 2020) to learn the dynamics model. The context encoder (probabilistic encoder) is updated together with loss from Q-value prediction like in PEARL, as well loss from dynamics model prediction like in CaDM. Given a new task, we let the agent collect transitions by interacting with the environment for only one episode with randomly initialized
pardmeerr distribution Then the agent will use the learned context encoder to infer a distribution over the hidden parameter of the current task. This distribution over the hidden parameter is fixed afterwards for the current task, i.e. e is fixed and the same for both methods. Then, for policy transfer, sample from the hidden parameter distribution and directly feed that inferred latent parameter into the learned joint policy. We evaluate the average return of this policy the current task. For model transfer, we feed that inferred latent parameter into the learned transition model and reward function. We use the models to generate samples and run SAC (Haarnoja al. 2018 learn the task-specific policy for this "simulated environment".
The empirical results are shown in Figure 2 Recall that K2 is expected to inversely related to the average return. i.e., increase in K, implies
We plot - KJ v.s. v/g using the results in Table
'The hidden parameters here and the inferred hidden parameters mentioned below are all referring to the mapping of the hidden parameter in the latent representation space through the context encoder. We assume the true distribution over the hidden parameters is also implicitly mapped into the latent space.
8

---

Published as a conference paper at ICLR 2023
decrease in the regret of the algorithm, thus the average return is expected to increase if our theoretical results match what happened in practice. Almost all of the results show that the change of the expected return is inversely related to the change of the approximated Lipschitz constant K-- The environment parameters directly affect K and the trend of the performance change in a way that is highly consistent with our theoretical analysis. In general, we find that a slower increase in Lipschitz constant with respect to the hidden parameter implies a smaller performance decay. The empirical results implies that for real-world HiP-MDP tasks, if we can quantitatively estimate the Lipschitz constant K for model and policy transfer methods given the environment parameters, we can estimate which one is expected have better final performance. For instance, in ball-wind, g is quite large, given the computed KJ, we will expect model transfer to have better performance as is less affected by the value of ! compared to policy transfer method. Note that our scenarios, the performance drop is mainly induced by the estimation error of the hidden parameters, and the learned dynamics and policies are near optimal. We show the empirical evidence in appendix C
6.2 EMPIRICAL RESULTS CORRESPONDING TO THEOREM 4.
We also show how the bound of the multi-step prediction error derived in Theorem 4.1 is re- lated to the performance of the other planning method used in model transfer - Model Pre- dictive Control (MPC) (Garcia et al.) 1989) MPC is widely used in recent deep model-based RL works (Chua et al.) 2018 Nagabandi al. 2019). Different from the previous approaches, MPC does not need to explicitly estimate the value function and its performance is directly affected by the multi-step prediction error. With the results in Table we can get the multi-step step prediction upper bound for ball-wind using The- orem 4.1 {(n) S v Ei m e. The bound implies that if the theorem matches what happens in practice, the n-step prediction error will increase as we increase v and m, and thus the performance is expected to drop as we increase and m. (The performance of MPC and the multi-step prediction error are inversely correlated.) Similar to the previous subsection, we investigate whether the empirical performance of MPC matches our predictions from theoretical results.
Ball wind
Ball wind (m)
20
33
-20
Figure 3: Average return on test tasks v.s. environ- mental properties (m & v) that affect the Lipschitz constants for MPC method on ball-wind.
During empirical evaluation, we first let the agent infer the hidden parameter and feed that into the learned transition function and reward function. Then at each time step, we let the agent randomly sample large number of actions, and use the learned model to predict N steps into the future for each of them and choose the action with the highest predicted cumulative reward. We investigate how and m affect the algorithm's 's final performance empirically. The results are shown in Figure 3 the average return of the algorithm on test tasks decreases as we increase the value of v and m, which is consistent with theoretical result. We also find that the variance of the performance (the width of error bar) is increasing as we use longer planning horizon for MPC.
CONCLUSION
Assuming the learned model/policy is optimal, we investgiated how the hidden parameter estimation error affects the robustness of model and policy transfer algorithms from a theoretical perspective. respectively. We show the conditions under which the value functions of HiP-MDPs are Lipschitz continuous. We further derive regret bounds for model and policy transfer, which are proved to be tight in linear and deterministic cases. Our empirical results are consistent with the theoretical results and indicate that a faster increase in Lipschitz constant with respect to the hidden parameter implies a larger performance decay. We note that in real-world HiP-MDP problems, especially when applying deep RL algorithms, the suboptimality of the learned model/policy can still play an important role in the potential performance drop new task, independently of the estimation error of the hidden parameter. And in many cases, policy transfer performs better than model transfer simply because the downstream model-free deep RL. algorithms are consistently better in regular MDPs.

---

Published as a conference paper at ICLR 2023
8 ACKNOWLEDGEMENT
The authors would like to thank Kavosh Asadi, Saket Tiwari, Michael Littman for discussions and helpful feedback, and the anonymous reviewers for valuable feedback that improved the paper substantially. This work was supported in part by an NSF Graduate Research Fellowship under grant #2040433, NSF grants #1717569 #1955361 #IIS-2007076 and CAREER award #1844960, DARPA grant W91 INNF1800268 and ONR contracts N00014-17-1-2699 and was conducted using computational resources and services at the Center for Computation and Visualization, Brown University. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The content is solely the responsibility of the authors and does not necessarily represent the official views of DARPA, the NSF, the ONR, the AFOSR.
REFERENCES
Kavosh Asadi, Dipendra Misra, and Michael L. Littman. Lipschitz continuity in model-based reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, volume 80 of Proceedings of Machine Learning Research, pp. 264-273, PMLR, 2018.
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning handful of trials using probabilistic dynamics models. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.). Advances Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, pp. 7759-4770, 2018.
Finale Doshi-Velez and George Dimitri Konidaris. Hidden parameter markov decision processes: A semiparametric regression approach for discovering latent task parametrizations. In Subbarao Kambhampati (ed.). Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016, pp. 1432-1440. LCCAVAAAI Press, 2016.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, llya Sutskever, and Picter Abbeel. RIS"2S:
Fast reinforcement learning via slow reinforcement learning. CoRR, abs/1611.02779, 2016.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander J. Smola. Meta-q-learning, In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning. ICML 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1126-1135. PMLR, 2017.
Haotian Fu, Hongyao Tang, Jianye Hao, Chen Chen, Xidong Feng. Dong Li, and Wulong Liu. Towards effective context for meta-reinforcement learning: approach based contrastive learning. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, 1, Virtual Event, February 2-9, 2021. pp. 7457-7465. AAAI Press, 2021.
Haotian Fu, Shangqun Yu, Michael L. Littman, and George Dimitri Konidaris. Model Model-based lifelong reinforcement learning with bayesian exploration. CoRR, abs/2210, 10. 10.17999., 2022.
Carlos E. Garcia, David M. Prett, and Manfred Morari. Model predictive control: Theory and practice A survey. Autom,, (5(3):335-348, 1989.
Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, and Mare G. Bellemare. Deepmdp: Learning continuous latent space models for representation learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference on Machine Learning. ICML 2019, volume 97 of Proceedings of Machine Learning Research, pp. 2170-2179, PMLR, 2019.
Omer Gottesman, Kavosh Asadi, Cameron Allen, Sam Lobel, George Konidaris, and Michael Littman. Coarse-grained smoothness for RL in metric spaces. CoRR, abs/2110. 12276...........11111111111111 2021.
10

---

Published as a conference paper at ICLR 2023
Tuomas Haarojja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforeement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, ICML 2018, volume 80 Proceedings of Machine Learning Research, pp. 1856-1866 PMLR, 2018.
Karl Hinderer. Lipschitz continuity of value functions in markovian decision processes. Math. Methods Oper. Res., 62(1):3-22, 2005.
Taylor W. Killian, Samuel Daulton, Finale Doshi-Velez, and George Dimitri Konidaris. Robust and efficient transfer learning with hidden parameter markov decision processes. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 6250-6261. 2017.
Kimin Lee. Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynamics model for generalization in model based reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 5757-5766. PMLR, 2020.
Yuping Luo, Huazhe Xu, Yuanzhi Li, Yuandong Tian, Trevor Darrell, and Tengyu Ma. Algorithmic framework for model-based deep reinforcement learning with theoretical guarantees. In 7th International Conference on Learning Representations, ICLR 2019. OpenReview.net, 2019.
Russell Mendonca, Xinyang Geng, Chelsea Finn, and Sergey Levine. Meta-reinforcemeen learn- ing robust to distributional shift via model identification and experience relabeling. CoRR, foss2006.07117.. 2020.
Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforremmntt learning. In 7th International Conference on Learning Representations, ICLR 2019. OpenRe- view.net, 2019.
Yash Nair and Finale Doshi-Velez Pac bounds for imitation and model-based batch learning of contextual markov decision processes. arXiv preprint arXiv:2006.063: 2020.
Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Policy gradient in lipschitz markov decision processes. Mach. Learn., (00(2-3):255-283 2015.
Emmanuel Rachelson and Michail G. Lagoudakis. On the locality of action domination in sequential decision making. In International Symposium on Artificial Intelligence and Mathematics, ISAIM 2010, 2010, 2010.
Roberta Raileanu, Max Goldstein, Arthur Szlam, and Rob Fergus. Fast adaptation to new environ- ments via policy-dynamics value functions. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1930- PMLR, 2020.
Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reinforcemmett learning via probabilistic context variables. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, volume 97 of Proceedings of Machine Learning Research, pp. 5331-5340. PMLR. 2019.
Zhizhou Ren, Kefan Dong, Yuan Zhou, Qiang Liu, and Jian Peng. Exploration via hindsight goal generation. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Gamett (eds.), Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, pp. 13464-13474, 2019.
Ziyang Tang, Yihao Feng, Na Zhang, Jian Peng, and Qiang Liu. Off-policy interval estimation with lipschitz value iteration. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems
11

---

Published as a conference paper at ICLR 2023
33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Cédric Villani. Optimal transport: Old and new. 2008.
Jiayu Yao, Taylor W. Killian, George Dimitri Konidaris, and Finale Doshi- Velez. Direct policy transfer via hidden parameter markov decision processes. 2018.
Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep RL via meta-lcarning. In 8th International Conference on Learning Representations, ICLR 2020. OpenReview.net, 2020.
12

---

Published as a conference paper at ICLR 2023
A MODEL TRANSFER AND POLICY TRANSFER
Algorithm 1:
Policy transfer
Input:
Test tasks hiiik with hidden parameters 0; Pa
for each task 7; do
Roll out policy л (aass (0)) with randomly generated Gaussian (0)) to collect data c {81, "1, 8n, On: Tn)
Infer the distribution (0)) over the hidden parameter of task with the learned encoder (ww(f)lc)
Roll out policy л (a]s,, (()) to interact with environment (evaluation)
end for
Algorithm 2: Model transfer
Input:
Test tasks (=i with hidden parameters 0; ~ Po
for each task 7; do
Roll out policy π [als,, (0)) with randomly generated Gaussian (0)) to collect data c= {s1, G1. ri. Sn, an Fh]
Infer the distribution (0) over the hidden parameter of task with the learned encoder q(w(O)l)
Planning with learned transition function (s"1s, a, (0)) and reward function (g a, (()) (evaluation)
end for
Empirically, in typical deep meta-RL settings (Finn et al., 2017 Rakelly et al., 2019), the true value of hidden parameters are not known to the agent both during training phase and evaluation phase. We follow this setting in out experiments and train context encoder like the previous approaches to infer the hidden parameter in latent representation space. Then the true distribution over hidden parameter is also assumed be implicitly mapped into the latent representation space. Our empirical experiments show that our theories can be extended to hidden parameter estimation in the latent space, and the results show consistent patterns with the theoretical results.
B EXPERIMENT SETTINGS
Ball-goal:
The state space consists of the ball's x, y coordinates. The action space consists of the ball's x, y velocity. v controls the step size in the direction of x axis in the transition function. The reward is proportional to the how far the agent has moved towards the goal's position (g cos Φg, sin Φg). We fix the value of v. and use dy as the hidden parameter create HiP-MDPs. Details are shown below:
State space: {8I, 88}
Action space: (azz a.)
Transition function:
S 8 + az V
Figure 4: The Ball environment used in our experiments.
8y = 8y + ag
Reward function:
R (s. - - (sy 9 sin o,)2 ( g cos oos + (s/, g sin )²
Ball-wind: The state space consists of the ball's z, coordinates. The action (one dimension) describes the direction of the agent's next move. In the transition function, m is describes the value of state accelerator, controls the step size of the wind, 0 describes the direction of the wind. The reward is
13

---

Published as a conference paper at ICLR 2023
proportional to the how far the agent has moved towards the goal's position cos Øg. g sin 0g), plus a goal reward when succeeds and control penalty (-0.5). We fix the value v, g. m, 0g and use as the hidden parameter to create HiP-MDPs. Details shown below:
State space: { }y}
Action space: {a}
Transition function:
8 ms, a cos
8y msy + sin sin
Reward function:
R ==v (8z - gcos 0g) + (sy gsin 0g)² v V(l cos cos@,)² + (s/, - g sin 0g)²
, 1f 4 g cos 0²)² + (s'y - g sin , - 0.1} 20 - 0.5
When computing the Lipschitz constants for Ball-wind, only consider the difference of distance towards goal part in the reward function for ease of calculation.
C ADDITIONAL EXPERIMENTS WHEN THE HIDDEN PARAMETER IS BETTER INFERRED
We further show that in these two scenarios, the performance drop as we change environment parameters is mainly induced by the estimation error of the hidden parameters, and the learned transition/reward functions and policies are near optimal. As shown Figure when we let the agent interact with the environments for more trajectories and keep using the collected data to infer the hidden parameter, changing of the environment's Lipschitz constant does not affect the performance much as before when the estimation of the hidden parameter is far less accurate.
Ball goal (v)
Ball wind (g)
ade sa
passente
Figure 5: Average return on test tasks v.s. environ- mental properties (v & g) for different estimation accuracy of the hidden parameter in ball- call-goal& & ball-wind.
D COMPARISON OF THEOREM
3.1 WITH PREVIOUS RESULTS
Assumption 1. The policy is optimal.
Assumption 2. S and A share the same Lipschitz constant.
Theorems
Value difference bound
Assamption
Assumption
Theorem 3.1
7 Kg)
R. LKAKS
X
Asadi et al. 2018 Gelada et al.
2019
x
v
x
Rachelson Lagoudakis 2010 Pirotta
2015
K (
x
2 ( K
Table 2: Comparison of Theorem 3.1 with previous results in regular MDPs
Theorem 3.1 and Corollary 3.2 also directly apply to a (K k K KÅ K -Lpsshic regular MDP with a K2 Lipschitz policy, as the Lipschitz continuity assumptions with respect to hidden parameters not used in both the final results and proofs. In this sense, we show comparison of Theorem 3.1
14

---

Published as a conference paper at ICLR 2023
with previous results in regular MDPs in Table
2 Note that in Theorem 3.1 have:
K
K (K7 + K)
KV. S Kg-
K
K
Specifically, the upper bound for KỘ- in Corollary 3.2 recovers the results in (Asadi et al. 2018 Gelada al. 2019). If we assume the Lipschitz constants for state and action are the same, we get Corollary E.1 which recovers the results in (Rachelson & Lagoudakis 2010: Pirotta et al. 2015). For regular MDPs, our results are more general compared to prior works as (1) we do not assume that the Lipschitz constants for state and action are the same because this is usually not the case in practice; (2) our theory applies to all Lipschitz policies rather than only the optimal policy.
E PROOFS AND MORE THEORY RESULTS
Using the generalized functions defined in Section 2 HiP-MDPs:
we can easily extend Bellman update rule to
a,, <- (s +
where Qn+1 converges to Q" a
Theorem 3.1 For a (KS h KSS K KA, Ki Lips-hiscchtzzz HiP-MDP with a (K K -Lipschtt policy n, if (Kg + K KA) 1, the value function is Lipschitz continuous with respect to S&A with constants bounded by:
Q(s a1 - Q(s, 02,
KĀ
- Y(K + K2
KA- = supsup sup
dx(a),
w 01
(11 ((32, [.
K (K K
Kg = sup sup
ds(s1, 82)
(Kg + K2
(s1, [(2,
K sup sup sup
K + K K
ds(s1,82 82)
$1.92
Proof. The proof is mainly based on the Bellman update rule (9). the dual form of Wasserstein Metric (Equation D and fixed point iteration.
Recall that:
(s, - (s,,,,, + (1)66
Now let:
Qut1 () Quti(s, (s, (2)
|R(s, w) - R(s, a2 a2)
K QR n+1 = sup sup
S sup sup sup a1.02 (2, VT (s', wdss'
da(a1,
01.02
([(T(5'ss, - -s,
sup sup an
(2)
[185'(s,02,00 w)) K (T(s''ss, (TT
(T' ds'
K +y supsup sup 01.03
Ki+
a2)
K$- (T('ss) - T(s' az w))ff(s ())s(s's
< -K +
sup sup sup sup
(2)
J-K
01.02
< K R
+ KA
+
(10)
The last inequality holds according Equation 2
15

---

Published as a conference paper at ICLR 2023
Similarly, we can get:
1n++1 (52: [,w]]
[R(s), a, - Ris2, a, w)]
K == sup sup sup
sup sup
sup sup
ds(s1, 82)
da(a1, 02)
w $1.52
81
a.) - (T(
w)ds'
? sup sup sup a $1
dg(81,82) 82)
K (1, (755 (o ds'
-K + y supsup sup
Kin
d5()),82)
61,02
KE (T(()))))) - T(s'|s2, a, w))
KR + sup sup sup sup f $1.82
82)
s K + >k KT
(11)
We need to further derive the upper bound of Ky:
(V),,w) - W\) = sup sup 82)
T(1),,) ( π (82),
Kyy == sup sup e $1.52
ds(s), $2)
e $1.82
1Q T (81), - Q(51. ())
((1 (82), (( (82),
I sup sup
sup sup 3 81.42
ds(31,82) 82)
dg(s1, 82)
#1.02
K K + K
(12)
Plugging Eqn 12 into Equ I0 Eqn
we get:
K k K + K KS Q. + +KQn K3)
(13)
K k S K + k S + kk)
(14)
By computing the fixed point of the recurrence, we get:
KA (K - K
K$ lim KA n-+00
Q
(15)
1 7(K + K
KS 7Kg K K
KS lim
KS
Ko.+1
S
(16)
7000
1- [Kg + K
Corollary E.1. we assume the Lipsehitz constants for state and action are the same K
K
K.S.A K = KS = S.SAA = =K.
(s1, (s, (1, w) - (22 (2) < S.S.A A.S.ds..((1))
a2))
((T,((11,, (- w)) < K
A...daa
do.a.
then:
K S.A
KQ = KZz = K2- <
R
1- 1
K. (1 + KS)
Proof. Let KĀ K
K K.A S.A
K4
K
K
K S.A in the bounds of Theorem 3.1
Corollary 3.2 If the policy n is optimal and 7 < ,the::
K$
K4 + KK - -K K
KSO <
K <
1-
1 7K3
16

---

Published as a conference paper at ICLR 2023
Proof. When the policy is optimal, have (Asadi al. 2018 Gelada al. 2019): I (s1,W) V (52,
Q" a,,, max Q (s2) aEA max !QT (s1, Q" ($2,0, atA
max
atA
Thus,
(Q(s1,a,w) - (22 a.
ky x Kg = sup sup sup
(17)
ds(s1,82 82)
$1.82
Plugging Eqn 17 into Eqn 10 and Eqn 11
we get:
K KQ Qm+11 S KR
y
K
K$
(18)
Q.
K Q K 7
Q.n K
(19)
First compute the fixed point of the recurrence of Eqn
19
we get:
K
lim K
Q.n-1
5
R
(20)
Now plugging Eqn 20 back into Eqn
18
KS 7
K + K
Kº-- lim K2
Q
KR
K4
(21)
1-
Definition E.2. Given a metric state space (S, ds), an action space A, and a metric hidden-paramett space (G, de), we define Fg as collection of functions: Fg = (f S x θ S) distributed
according to ((fla) where E A. We say that (F FF) is Lipschitz model class in HiP-MDPs if
K = sup KS
S.S
(22)
JEPS
and
K sup K K8.S
(23)
fEF
finite.
The transition function associated with a Lipschitz model class then can be defined by:
T(s |s, a, Σ 1(f(s,0) s')g(flo),
(24)
And the generalized transition function associated with a Lipschitz model class would be:
- ((f(s,)) s') sf)g(fla)w(())))))
T,(5',,,,,w)
(25)
We introduce the following two lemmas regarding Lipschitz model class (Aadi al. 2018 (We also include definition in appendix):
Lemma E.3. (Asadi et al. 2018) A generalized transition funition Tg induced by a Lipschitz model class Fg and fixed (8, a) Lipschitz with a constant:
(T,, 0), T- ())) W(H1 F2)
KT = A (T₂)
sup sup sup
KT
(26)
8e a P1.22
Lemma E.4. A generalized transition function Tg induced by Lipschitz model class Fg and fixed (s, a) is Lipschitz with a constant:
a, w1 (IT,(lls,,,,,), wz))
K$ w.w (Tg) == sup sup sup S aE win42
< K
(27)
W(w1, W2)
17

---

Published as a conference paper at ICLR 2023
Intuitively, the above Lemmas give bound the differences in distributions agent transitions to, as a function of how different the distributions of states it transitions from are, or how different the distributions of the hidden parameters are. It is theoretically pleasing that these bounds are given as the Lipschitz constants for the corresponding differences between point distributions of either states hidden parameters.
Proof for Lemma E.4
Proof.
sup hi a
sup ( h:Kaga(h)51 -
sup a=Kag,(M)) / 1 E ((((o) (w) (2((0))((s' ) da
Σ g(tla) 1// 1(t(5o, (w) we (O)) d'''dd
sup a
E g(tla)
/ (w1100 -wi(0)-w2(0))) (0) 0))d0
sup hiKag
$ Σ g(tla)
| (wi(0) (2())))((((s. 0))
sup x
(wi w2) 1 1((((50 0))
K2 Σ g((la)
sup l:Kag
K
K
Eg(((aa sup
(wi(0) wa ()))(((( 0)d0
Kē
Σ s(t)a)W(wn, = [ ((w),42)
(28) O
Using Lemma 33
we can derive the bound of the compounding error dynamics prediction given the estimation error in hidden parameter at test time (Theorem 4.1
Theorem 4.1
Assuming the estimation for hidden parameter 0 bounded by e, that is, (ww (() S c, an learned accurate f, induced by a Lipschitz model class Fg with the Lipschitz constant 7 and K a fixed sequence of actions ag, ... an-1, and a start state distribution μ. Then Vn 2 1:
((n) = W(1 (-1,,w), 17 (-1,,W)) S K... Σ (K
(29)
where (-),,))) (. -...f T. an, an-2,4 an-1,w), and Fy (-11,10) is defined
similarty.
Proof. We first derive the bound for one-step prediction given Lemma E.3
18

---

Published as a conference paper at ICLR 2023
5(1) = w(F (g((,,,)))
sup / (T(60)0, - (b'1s, ao: (s) (s)))) (p(s)ddds
$ / up (f((5lss,00)))) - (s'1s, an, ()) p(s)ds
(30)
/ W w), (-1s,, ao (s)dss
$ / Kffe(s)dss K.fe
Then, we can prove the bound for E(n):
{(n) = W(T (:1, w), Tg (),,
W (-1u, w), 17
- (-12 (-1,,),, + (1 (-1,, (),
(-),,, ((),,())
w(T ( (1),, w), pm (-17 (+1,,0), + ( (-1,,
(),, w) w) (g 100-1((l,,,,, ()),
K&W (
(-),,), 1 (.
a)) = K.fe
2-1
-KjE(n - 1) + KR S K.c
(K
(31)
We can get the same results if we replace Ty (-12g- (,,,,,,,,,,) with Fy (- (-l,,,,),,,, a)) in the triangle inequality.
Lemma 4.2 Given a HiP-MDP with learned (K KG) Lipschit transition model f. ft, in Model
R Transfer, the generalized value function is Lipschitz continuous with respect to Ω with a constant bounded by:
Ky. S K2 h +2k + KT
Proof. Recall that:
( = (,), +
(0))ds
Then we have:
V (s*(sww((0) (1):((()) (**(s
K.O. sup sup
(2(
S Widw2
< sup sup #ES w1 402
ww(
J& (1)
(0)) - Jag Fy (obls ( (s)) क (s). wz )dss
+7 sup sup
do (w(
SES w1 ,49
( ddsh Sog Tg (5 (s), (5/2 (T)), w2 )ds'
R + 7 At
sup sup m22
(1())
19

---

Published as a conference paper at ICLR 2023
Recall that we quantify the distance between W1 (6) and (y( using Wasserstein Metric W W(w1, w2). then
Is+; (s11s, (6), )dsí Sag (1)) (s))
K- Kp.
+ + sup sup
1t
W W( W2) 1 (s1s,, (s), w1)
eES w1:02
sup sup #ES w1
W W(11, W2)
)ds} - J2,
+y sup sup #ES w
w2)
1s To (0110, x7 (0), w)) (8), क wi) 27 π (8)
< KR R
sup sup eES w:0g
W (ww)ww))
Sa
Sa, (() (s), (62) T()), ((5), w2)ds/)
+y sup sup .ES wi wي
W W(w, w2)
1s T. (8 K3= W (2).
K
sup sup #ES w1:0
((w1ww))
Sx s M (0)
wo)ds - Jo (2)) (522 (T)), wa dssy
+ y sup sup S wi ,0
W W(w1,
1sg
w2 )dsi Sag (()) क (+)) wa )ds)
KR + f it
Q + y sup sup sES w1
W W(w1,
E of (s (5 π (s) (2) - E ( (5) W (w1,w2) KS. W (fg (- (T))((() w1) ), fg(- (s), w2))
= =K ft
+ k + sES wi
sup sup
SK' Rt
+ + +
sup sup .ES w7
W W(w1 w2)
< k2 -K8-- + k KG
Theorem 4.3 Given HiP-MDP with learned (K K. KA R:
At: K, K. KQ -.)--Lissshitt transition model T. R2 in Model Transfer, if 7(K + KŠK < 1, the generalized value function corre- sponding to the KşSLipschitz policy (als)) is Lipschitz continuous with respect to Z with constant bounded by:
K + KK K$
KB- <
(32)
1-?
Proof. By further assuming the Lipschitz continuity of dynamics & policy in Lemma 4.2, we can get the result bound leveraging the dual form of Wasserstein Metric (Equation 2 triangle inequality, and computing the fixed point of recurrence.
Recall that:
Qn+1(s,a,w(0) + (s,s,,, + / (00))
(5), (8',,, ((,)))) where (()
20

---

Published as a conference paper at ICLR 2023
It is well known that as n oc, Qu+1 converges to Q" now let:
(0)) (9,+1((,,,,10)) d (w1,ww)) W2)
a, wl
(s+
Kq = sup sup sup #ES aEA ww
(8, (8, a, w1 (0)) Rg a (2))
I sup sup sup #ES aEA wwg
d (w( W2)
1Ss Fg a, a. (1)) - Soy [5 a (22))) w2 w2 (0))
+y sup sup sup SES aE wi.wa
du(wi, w2)
Sss T, (s1 (51, a wi )ds) Say To a, (2
K& ft y sup sup aEA w1:42
(2)
(33)
Recall that we quantify the distance between w1 (0) and (2( using Wasserstein Metric ((w1,w2). then
Is,; (, a, )ds S,g T. (s'2 s, a, a2 )
K2 + sup sup K KQ
sES aEA w1.02
W(w1,w2)
(1 w1) - [, W W(w1, w2)
< KR
supsup sup sup sES aEA w1
Ss ( 1l wi W a we )ss - Saz (5)
+ 7 sup sup sup sES a w1:2
(w))
1s T. (s( s, a, wi (Q a, wi) 2% (s) a, (2,22)llss)
S K8 +7 sup sup aEA w1,02
R
W (2) 1s (, a, doss Saz ( a,,),, W W(w1, W2)
+7 sup sup sup SES aEA wio40
Js T (s s, wi) KQ,n W (W1 (hy
< Kg Ω 7 SES aEA w1,00
supsup sup sup
W W(w1,w2)
Usx (611,, ( a, w2 dss - Saz (, a,
a, (2)
+7 sup sup sup sES aEA w1,02
supsup
W(w1,
1,g (s, (,,1)ss a, ds' - Jog ( (5/2, a, w2)ds')
= =K
ft + yKgn +y sup sup sup sES aE w1,
W ((wI, w2)
EE of I To (s)ls,a..) Q7(11, a, w2) - Bag (7 a, w2))
= =k + +KKon +y sup sup sup sES aE w1
W W(w1,w2)
KS W (1s,,,,, (-1s, a, w2)))
w1)
< A + -KK +7 sup sup sup sES of A W1.22
W W(w),
< K + 7 KQ R K
(34)
Recall the bound for (KS K -ipsschtt MDP: R
[( (())
K$ 7( (KSK -
KS- sup sup sup A 81
R R
ds(s1, 82)
1 (K + K2KK))
Equivalently:
a+1
KQ < Σ
R R
+
Σ
o
KS
K + + Ko.0
(35)
1=0
1=1
21

---

Published as a conference paper at ICLR 2023
By computing the limit of both sides:
11
n+1
K8 lim Ko S
lim Σ
7 K
lim
Σ 7 Kg²
lim 7"h K000 n
70000
n 12000
1-000
R
7K
K2
R
+
T
+0
1-?
From Lemma 4.2 we have:
kv- S KR + +2k2² Qr + + Kg
Thus,
K + K 0 KT
KV- S
7
Lemma 4.4
Given a (K KT. KD Lipschitt HiP-MDP, for the optimal policy x²,
|V" (1) - V* (2)))())) S max IQt (s, (0)) Q" (s, a, (0))
Proof.
|V (8)) V** (8, max Q" (s,a, (0)) max Q" (s,0, aEA
aEA
max IQ" (s, Q" (s, aaA
Lemma 4.5 Given HiP-MDP with learned (K h. K.,, n. Lipp--ipshhitzz transition model T,, R.
and the optimal policy R* (als) for I and Rt starting from state distribution μ., the expected discounted reward difference induced by hidden parameter estimation error c bounded by:
7 SK5K K (1 (1- (1-
J J4-1
- 7k)) '6,
(36)
if 7 < 1.
Proof. Recall that:
J = / (s, (O))
We have:
Sa V (0)) V (s,w2(0))
K3 sup sup
(O), (0)
wi 222
11 (i (s,),
< sup sup a wis
w2)))
lim KQ n=
n+1
lim Σ
KK ft
+ lim
7 KS Q-
K + lim nd KQ
n=000
n
im0
7Kg K2
+0
From Corollary 3.2 we have:
KS
KSH - 1 K
22

---

Published as a conference paper at ICLR 2023
Thus,
KJ = 4 - -7k3) + K (1 (1- 71 7
Given linear and deterministic transition function and reward function, the bound derived in Theorem Lemma 4.5 are tight.
Claim 4.6
Proof. Assume a linear transition function T for new task with hidden parameter A defined as:
T(s,a) As Ba + C0,
(37)
And reward function defined as:
R(s,a) Ds + Ea + F0.
(38)
Assume there's estimation emror for 0, then the transition function and reward function becomes:
T(s, a) As Ba C(O e),
(39)
R(s a) Ds + Ea F(O c), as,a ((s,,a a)l Ce,
(40)
First observe that:
(41)
And that for n = 2:
vs (sTT(s,0)),00 - = IA(C(e + 0) - -CO) + Bay
Bai + C(e + 0) - C#
1
= |ACc+ + Ccl = Ce Σ
A'
1=0
(42)
More generally, for step compounding error of dynamics prediction T and 7", given fixed sequence of actions ag, 01, "r .n
n
vs (s, - (0, (s,an)) = Ce
Σ A 1=0
(43)
Thus, the bound in Theorem 4.1 is tight.
Now consider the state 0 and the action space only consists of one action = 0 (thus the policy is optimal), calculate the value of predicted with 7 and R:
(s 0) R(0) + [R(0+ +e)+ + +e) + C(0 C c()+ +(...))+++..))
F(0++) +6) Σ + DC + Σ A'
1=0
n=1
1=0
(44)
F(0+E)
1-7 + DC-(0+e)
(1
(F(1-17 + DC
(1 (1- (1 -A)
Thus:
(Vs= = 0) - (s==0) F(1 F(1-7A)+ DCT
(45)
(1-7)(1-3)
The result matches the bound derived Lemma 4.5
(In deterministic cases, we assume we directly
estimate the value of f, so KR & Kg becomes K & K in this case and the Wasserstein distance becomes the Il1 distance 0 0l.)
23

---

Published as a conference paper at ICLR 2023
Lemma 5.1 Given HiP-MDP with learned K-- Liischhht joint policy T(als, w))).. in Policy
Transfer, the generalized value function with respect to with constant bounded by:
kV S KQ- + K
Proof. Given test task, recall that:
Villww (s = -2
(2(l) (s. (sssww))))
where ó to denote the true distribution (0)) over hidden parameter for the current task (which should be a Dirac d-function in practice) of the environment. Then we have:
T( T-|w0)
(8, 6) Vis (=
(s, ()
sup sup
d(11, w2)
(s, ( wI).
Q7 ( (s',)(ss)2)))) w2)
(s,
sup sup sup
d(wi, W2)
& ww
Q (1( (s, ((. o()) Q TT. (00)
y sup sup sup
W()
w7 Q (12))
T (s. (0))
(1 (s, オ (8,, (z) (8))
d(wi, W2)
K2 du w2)
K d( (w](TT(W))))))))) ) =(w2))ds'
3 susdup sup @ win2
d(wrw w22
d(w إجم
K K kfd(wwlks W2)
d(w),w2)
< Kg- + k2
Theorem 5.2 Given a (K KA, K Lippschttt HiP-MDP, in Policy Transfer, the generalized value function corresponding the pretrained (Kş K Lipscchtt joint policy (als, (()), if (KK + K4) < 1, is Lipschitz continuous with respect to Ω with a constant bounded by
K Qr
KV - 1-9
(46)
Proof. Similar to model transfer, if we further assume the Lipschitz continuity of dynamics & policy Lemma 5.1, we can get the result bound leveraging the dual form of Wasserstein Metric (Equation triangle inequality, and computing the fixed point of recurrence.
Recall that:
((s,T(s,w), + (T(s,T(s,)),)) + 11(6)1s, () o)ds'
24

---

Published as a conference paper at ICLR 2023
Qatt (w)) (s, (1,,()) 2utlee) (8, (8, 6)
Kg = sup sup sup wh م
w(w),W2)
1 (8, (T, Φ) Qutz (8, (T, w2), 60)
suppup sup
d(w
1 (s, (s, 0) @=(( (s, (8, Ø))
sup sup o w
d(w1,w2)
Q (3, (8, ) Φ) Quti (w(l (s, (8, 6)
d(<(-|wi): n((|W22) d(wi,
supsup sup
d( (1) π7 (-(w2)
@ Quti (s, (T, w2) Φ) @u(l
(())
(s. (S, Φ)
supsup sup w1:42
W2)
@ (s, (T, W2), Ø) ( (2))
(s,
<K K sup sup w1a4
w2)
(47)
Now let:
1 (8, a, x) 2utluz)) (s, v)
K Q" a+1 sussupssn sup sup ww.wa
d(w1,w2) W2)
Sx ( (8'1s, (00)) (6', (s', ,w,), x) (1((1))) (5', ,w2) D)lds' d(wi, W2)
dnsdnndn dassssssssss 7 44+5
sup
Coeediia dhs aaednsdnns
W1:42
a Coaeedine
S Tg a, 2 (s', T(' w1 )) 3) - (1 (s), (1), dds' d(w1, w2)
wiim
s. (1
(s', (T (ss, - (1(() (s', a(s ((s,(w2), d)
d(w1,W2)
Sa T a, の)) K Wyd
dnsdnndnn ..ss.... S sup
d(wi, W2)
a wi
o Ty(s'ss.a の) KG-d( n (ds
((1),w)) w2)
Sa. (8'1 a, ()) K k n d(w (y) wds' + So Ty (s, a, ()) K.) Kfd(w1,, w2 Ids' d(w1,w))
4 sup sup
d(w),w2)
wi <>(Komm Kg. K2)
Equivalently:
n+1 +
Ke n+ S n KTn
K2 k
1=1
Computing the limit of both sides:
a+1
K2- lim K2r,++ n+1 S lim 7n _"Kr.p
+ lim Σ
o 2 K S K
no000
n=
no
YK K
-0+ 1 1-7
Plugging the results back in to Equation
47 we have:
KA KQ. K
K&.
K.
<
1 1-7
25

---

Published as a conference paper at ICLR 2023
Given a (K K2 Kÿ. -ipsschzt HiP-MDP, pretrained K -iisschhtt optimal joint policy x (als, (()), starting from state distribution μ. the expected discounted reward difference induced by hidden parameter estimation error is bounded by:
Lemma 5.3
J 1 2K (K + K - -KE R)
- 11 - -
6,
(48)
if y S 1.
Proof. When the policy is optimal, we have:
K m+1 supsup sup sup wi ( (s, a, 6)) @x(lws)) (s, 2))
W2)
canssa < 440> 7 saasdns
Jx T (())) (T'' ())) Qu((1 (5', ( w2): Ids
dnsddsdns
w2)
Wi 201
S a Q (1w)) (5', " (11,:')),,), w1),") - Qirllwa)) (s', ( wi), G)
sssiissnn sup
d(w,,
J (1(8''s,a,,))Q0 (wz)
(s', "(s ((ss,wi),)) - Q (1) (s' 7(s w2),, @)lds'
d(w),wz)
So (o a, Φ) ii ,d(,,,,,
y. sassasss dnsdnndds sup a winw2
d(w1,
Ie ( K Kg. (((((W)))) (w2)
d(w, w2)
Jor (5'1 a, KQ ,nl( wg Wy)ds'
+ S3 Tyl' a, K) ( wg)dd d(w1,w))
& dus.. sussssdds sup
w2)
wiog
Kg- <T(Kg
Equivalently:
a+1
K& 2"
S
7th
TmK 0 +
Σ
7
K2 K
m+1
1m1
Computing the limit of both sides:
a+1
K3-- lim KQ n11 S lim 7tt
lim Σ
K Qr
K2
Q-,0
n-000
n
n
7K K
=0
1
If the policy is optimal, we have: | (s, 6(0)) (s,
va(wz) (s,
< max (u((1) (s,a, max (M(wz) (s)),
< max (Q)((n) (s,a, - (s,
Then for the Lipschitz constant of the expected return function, have: (a( (s, ((0))l(ssdd Ja (v (s, o())
K& = supsup sup wi 20g
d(wi wz)))
I (Q (s (s, (0) - (w( ((0)) d(w) (0) wy (0))
< supsss sup wn
-2-
26

---

Published as a conference paper at ICLR 2023
Using the results from Corollary 3.2. get:
+ (
[K
+ - {)
KJ <
(1 11-7(1--73)
Given a linear and deterministic transition function, reward function and policy, the bounds derived in Lemma 5.3 are tight.
Claim 5.4
Proof. Assume a linear transition function T for new task defined as:
T(s, a) As Ba,
(49)
reward function defined as:
R(s, a) Ds + Ea,
(50)
And the optimal policy corresponding to hidden parameter & defined as:
(-)) = H8
(51)
Assume there's estimation error for 8, then the policy becomes:
π x-)) H( + e),
(52)
Now consider the state can calculate the value predicted with T, and n::
(ss 0) = R(0) + DBH EH (0) E + (DABH DBH +e)
c0))+ +...))
EH +c)
Σ
+DBH Σ>"> Σ A R(O)
j=1
DBHT
1 1-7
+
(1 (1- (1-7)(1-7/)) (0++) R(0)
(53)
Thus:
(5 =) 0) ((s 00] (7EH + (1 71(1 7A)
DBH
(54)
>H(E - 7AF + BBD)
(1-7)(1-74)
The result matches the bound derived in Lemma 5.3 (In deterministic cases, we assume we directly estimate the value of 6, so K becomes K in this case and the Wasserstein distance becomes the LI distance 0l.)
F FURTHER CLAIM ABOUT THE ASSUMPTION <K < 1
Similar assumptions have been made most of previous papers discussing Lipschitz continuity in
RL (Rachelson & Lagoudakis, 2010 Pirotta et al. 2015 Asadi al.) 2018 Gelada al. 2019). Here, we make the following claim:
Claim F.1. When HiP-MDP has a fixed horizon N, the assumption that rKK + K KK) and the assumption (when the policy is optimal) 7 7 1 are both unnecessary.
Proof.
V(si,w) V(sg,w)
IN ((1 R2)l
KSS sup sup
sup sup
88)
(5(81,82)
Firstly, if the agent is the last step N of episode:
Σ Di=p ( R2)l
KS
sup sup
S K
dg(81, 82)
81.82
27

---

Published as a conference paper at ICLR 2023
If the agent is at step N - 1:
Ei=o 7 R2)
K sup sup
(2)
3 $1.82
[R(s1, ((1), R(82, (82). + H ((8),, +()) - R(s
(5)) w)
sup sup $1,02
(2)
[R(s ())),, R(s2 π (s1), + [R($2, (81), ((52)
S sup sup w 61,82
88)
|R(s π ( - R(6l, v (
+7 sup sup 81.82
ds(s1, 82)
|R(55 (s w) - R(52, π ((5)) (s2),
< K + K + sup sup w $1.82
ds(s1, $2)
- R( n (8) () + [R(62 n (s1), R[s,, (+))
<Kj K sup sup $1.42
ds (s( 82)
[(51, (
(s2,, ①7 (82), w)
< <K + K + TK K sup sup
ds(81,82)
< k K + + ) + KAK
If the agent is at step N - 2, similarly, we have:
K < K + K + (
+ K + +2 K K4
By induction, if the agent is at step N we have:
K S K + )) (KG Kイ
Thus, the assumption that (K + K K 1 is unnecessary.
Similarly, when the policy is optimal, at step N 1: IE R2)l
K sup sup $1.92
(8)
a,(s), R[s2, w) (18((),,,,) R[s2 a, w)
S sup sup $1.82
ds(s1, 82)
RR(1 a, w) ((
< KS + sup sup $1.32
ds(s1, 82)
T(s a, w) - T(82, w)
< Kg + T sup sup
82)
81.82
< K + K
At step N n, we have:
KS S KS 7" (K$)"
KS
Thus, the assumption that 7KŠ < 1 is unnecessary.
Intuitively, for tasks with infinite horizons, the assumption requires that the future states generated by close states are not too divergent. The threshold of "divergent" depends on how farsighted the agent is.
G APPLICATIONS OF HIP-MDPS
HiP-MDP is an important setting widely used in recent meta RL papers. It provides natural testbed for meta/lifelong RL algorithms as the difference between tasks can be controlled by a low- dimensional latent vector. A commonly-used meta-RI. benchmark (almost in every recent meta-RI.
28

---

Published as a conference paper at ICLR 2023
papers) creates set of tasks by changing the environmental parameters (e.g. mass, damping) (eee et al., 2020; Raileanu et al. 2020 Fu et al. 2022) or reward functions (e.g. target position, target velocity) (Rakelly et al., 2019; Zintgraf al. 2020 of mujoco-simulated robots. Moreover, in real-world robotic applications, we want the robot to learn a policy that is robust to small dynamics changes like unexpected perturbations. Then one of the solutions is let the agent pretrain over set of HiP-MDPs considering the possible dynamics changes (Nagabandi et al. 2019).
29


Reviews:
[{"review_id": "hV8eLVOx6DM", "reviewer": "Reviewer_zeh4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studied the performance bound when transferring model or policy between HiP-MDPs. The authors introduced some assumptions on the model/policy Lipschitz continuty and established the Lipschitz continuty property of (optimal) value functions in HiP-MDPs. In Sec. 4 & 5, they derived the upper bound when transferring the optimal policy in learned Lipschitz model and pretrained Lipschitz optimal joint policy, repsectively. Finally, in Sec. 6, they conducted some evaluation experiments to verify the theoretical finding.", "summary_of_the_review": "This paper studied an interesting setting. However, as I discussed in ``Weakness'' section, I think this paper is lack of novelty and the contribution is not enough for ICLR.", "strength_and_weaknesses": "### Strength\n\nThe Lipschitz continuty assumptions about the model/policy are reasonable, from theoretical perspective. It's also good to see the theoretical results are verified by empirical evaluation in Sec. 6.\n\n### Weakness\n\nSince the experiment part is just some evaluation about the theoretical results, instead of proposing some novel and effective empirical algorithms inspired from theory, I would believe the novelty of theory part is what to be evaluated the most.\n\nHowever, it seems to me the theory part is not significant enough, and the results, although somewhat new, are kind of trivial and not surprising. Theoretical results about value difference bounds w.r.t. Lipschitz constants had been established in previous works (e.g. [1]), and the authors main contributions seems just analyzed the extended setting called HiP-MDPs, where the model/reward functions will take an additional ``hidden parameter'' $\\theta$ as input which capturing the variation across different models. I found the techniques are similar and there is not much novelty.\n\nBesides, the theory also has some restriction in practice. Although similar assumptions occurs in previous papers, the requirement that $\\gamma$ times (combination of) Lipschitz constants is less than 1 occurs in almost all the main theorems (seems because of the $1-\\gamma K$ occurs in the denominator). However, in practice, it is hard to verify whether such condition holds, or give any prediction about what will happen if such conditions do not hold. I think the authors should provide at least some examples about when such assumptions can be true in practice.\n\n\n[1] DeepMDP: Learning Continuous Latent Space Models for Representation Learning. Carles Gelada, Saurabh Kumar, Jacob Buckman, Ofir Nachum, Marc G. Bellemare", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "QKZ3y93ADO", "reviewer": "Reviewer_g5Zd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors studied the Hidden-Parameter MDP (HiP-MDP) framework, which is kind of non-stationary MDP. There is a set of parameters controlling the dynamics and reward, which may vary by tasks. This setting is very important since it can be applied in meta reinforcement learning and lifelong reinforcement learning. The authors characterized the robustness of both model transfer and policy transfer algorithms with respect to the hidden parameter estimation error, which I believe is a meaningful research. In this paper, the authors not only obtained the Lipschitz continuity, but also the regret bound under both settings. ", "summary_of_the_review": "To sum up, I would like to give this paper an \"accept\" mainly because of its theoretical significance and novelty.", "strength_and_weaknesses": "Strength:\n(1) Very clear paper writing and very high paper quality. The literature review is very complete. \n(2) Proofs in this paper are very clear and easy to follow. \n(3) The experiments are very complete. I believe they are reproducible. \n\nWeakness:\nThe context width is quite tricky in the supplement. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9SHJmScatM", "reviewer": "Reviewer_qyA5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the HiP-MDPs where a set of low-dimensional hidden parameters models the variations in tasks, this setting is naturally applicable to settings with transfers. The main contribution of this paper is the theoretical analysis on the regret upper bounds of model and policy transfer algorithms, as well as multi-step prediction error. These bounds are proven to be tight in linear deterministic settings. The theoretical analysis in this paper provides insights on the performance of model/policy transfer algorithms and its connection with Lipschitz constants.\n", "summary_of_the_review": "Overall, this paper is strong in theory and the experimental results support the main theoretical claim. The presentation can be improved at some point.", "strength_and_weaknesses": "\nThe strength of this paper is mainly theoretical, it rigorously characterizes the robustness of model/policy transfer methods by hidden parameter estimation error. Thus, it provides practical perspectives on choosing between model or policy transfer methods. For example, policy transfer is likely to outperform model transfer in the sense of robustness when the differences between actions are small. On the other hand, model transfer is likely to outperform policy transfer in terms of robustness when the different between neighboring states are small. Empirical results further validates the theoretical findings.\n\nThe main weaknesses of this work are listed as follows:\n- It is unclear how the analysis presented in this work is related with previous theories. In many places the authors have mentioned that their results share similarity with some of the previous results, for example Theorem 3.1 is a generalization of theorems in a bunch of papers, similar paradigm of the algorithm framework has been widely used in complex continuous control tasks. The authors emphasis the major significance of their work, but it seems that it fails to provide a more detailed comparison on how much it improves upon previous results on HiP-MDPs.\n- There is little intuition on how the results are derived, a brief outline of proof on at least one of the main theorems would help readers to better understand the technical novelty of this work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster