Learning a subspace of policies for online adaptation in Reinforcement Learning
Authors: Jean-Baptiste Gaya
Laure Soulier
Ludovic Denoyer
Venue: ICLR 2022 Poster (2022)
Keywords: Deep Reinforcement Learning
Online adaptation

Abstract: Deep Reinforcement Learning (RL) is mainly studied in a setting where the training and the testing environments are similar. But in many practical applications, these environments may differ. For instance, in control systems, the robot(s) on which a policy is learned might differ from the robot(s) on which a policy will run. It can be caused by different internal factors (e.g., calibration issues, system attrition, defective modules) or also by external changes (e.g., weather conditions). There is  a need to develop RL methods that generalize well to variations of the training conditions. In this article, we consider the simplest yet hard to tackle generalization setting where the test environment is unknown at train time, forcing the agent to adapt to the system's new dynamics. This online adaptation process can be computationally expensive (e.g., fine-tuning) and cannot rely on meta-RL techniques since there is just a single train environment. To do so, we propose an approach where we learn a subspace of policies within the parameter space. This subspace contains an infinite number of policies that are trained to solve the training environment while having different parameter values. As a consequence, two policies in that subspace process information differently and exhibit different behaviors when facing variations of the train environment. Our experiments carried out over a large variety of benchmarks compare our approach with baselines, including diversity-based methods. In comparison, our approach is simple to tune, does not need any extra component  (e.g., discriminator) and learns policies able to gather a high reward on unseen environments.

LEARNING A SUBSPACE OF POLICIES FOR ONLINE ADAPTATION IN REINFORCEMENT LEARNING
Jean-Baptiste Gaya*
Laure Soulier!
Ludovie Denoyer
ABSTRACT
Deep Reinforcement Learning (RL) is mainly studied in setting where the train- ing and the testing environments are similar. But in many practical applications, these environments may differ. For instance, in control systems, the robot(s) on which policy is learned might differ from the robot(s) on which a policy will run. It can be caused by different internal factors (e.g. calibration issues, system at- trition, defective modules) or also by external changes (e.g., weather conditions). There need to develop RL methods that generalize well to variations the training conditions. In this article, we consider the simplest yet hard to tackle generalization setting where the test environment unknown at train time, forcing the agent to adapt the system's 's new dynamics. This online adaptation process can be computationally expensive (e.g.. fine-tuning) and cannot rely on meta-RL techniques since there is just single train environment. To do so, propose an approach where we learn a subspace of policies within the parameter space. This subspace contains an infinite number of policies that are trained to solve the training environment while having different parameter values. As a consequence, two policies in that subspace process information differently and exhibit different behaviors when facing variations of the train environment. Our experiments car- ried out over a large variety of benchmarks compare our approach with baselines, including diversity-basee methods. In comparison, our approach is simple to tune, does not need any extra component (e.g., discriminator) and learns policies able to gather a high reward unseen environments.
INTRODUCTION
In recent years, Deep Reinforcement Learning (RL) has succeeded at solving complex tasks, from defeating humans in board games (Silver et al,, (017) to complex control problems (Peng et al. 2017 Schulman et al.] (007)). It relies on different learning algorithms (e.g.. A2C (Mnih et al., 2016 PPO (Schulman et al. 2017)). These methods aim at discovering a policy that maximizes the expected (discounted) cumulative reward received by agent given particular environment. If existing techniques work quite well in the classical setting, considering that the environment at train time and the environment at test time an example, when learning to drive a car, student learns to drive using a particular car, and under specific weather conditions. But test time, we expect the driver be able to generalize to any new car, new roads, and new weather conditions. It is critical to consider the generalization issue where learn policy that generalizes and adapts itself unseen environments.
similar is unrealistic in many practical applications. As
one of the challenges
Different techniques have been proposed in the literature (Section 6b to automatically adapt the learned policy to the test environment. In the very large majority of works, the model has access to multiple training environments (meta-RL setting). Therefore, the training algorithm can identify which variations (or invariants) may occur at test time and how to adapt quickly similar variations. But this setting may still be unrealistic for concrete applications: for instance, it supposes that the student will learn to drive on multiple ears before getting their driving license.
Facebook AI Research, correspondence to jlgaya@Bb.....m fb.com
CNRS-ISIR, Sorbonne University, Paris, France
Now at Ubisoft
Cooee
available at
salina |

---

(a) The figure represents the parameter space, The red (resp. blue) region is the space of good policies over the training (resp. lesting) environment. A single learned policy (red point) may be inefficient for the test envi- ronment and has be adapted (e.g., fine-tuning) to come good at test test-time (blue point). Instead of learn- ing a single policy, we learn convex sub-space (the pentagon) delimited by anchor policies (red stars) that aims capturing large of good policies. Then the adaptation is just made by sampling policies in this sub- space, keeping the best one (blue star).
(b) Qualitative example of k-shot adaptation on mod- ified Ant environment (20% of observations masked), 5 policies (i.e 5 values of z) are tested on one episode. In this case, for 2 = 0., the Ant is able to adapt to this new environment. More example of LoP trajectories in
google. Figures [ and 6 See wtw.ooppolieeeeeeeeeoeeeeeeeeeee /home) for
coom v ew/ subspace-of---
vidcos of the learned behaviors.
Figure 1: (Left) An illustration of the process of learning a subspace of policies. (Right) Comparison between PPO and our model in a test environment.
In this paper, we address a simpler yet harder to tackle generalization setting in which the learning algorithm is trained over one single environment and has to perform well test environments; preventing us from using meta-RL approaches. A natural way to attack this setting is to start by learning single policy using any RL algorithm, and fine-tune this training policy at test time, over the test environment (See red/blue points in Figure Lap, but this process may be costly in terms of environment interactions.
Very recently, the idea of learning set of diverse yet effective policies (Kumar et al. 2020b Osa al., 202T) has emerged as a way to deal with this adaptation setting. The intuition is that, if stead of learning one single policy, one learns set of 'diverse" policies, then there is a chance that least one of these policies will perform well over a new dynamics. The adaptation in that case just consists in selecting the best policy in that set by evaluating each policy over few episodes (K-shot adaptation). But the way this set of policies built and the notion of diversity proposed these methods have few drawbacks: these models increase diversity by using additional intrinsic reward which encourages the different policies to generate different distributions states. This ob- jective potentially favors the learning of policies that are sub-optimal train time. Moreover, these approaches make use of an additional component that may be difficult to tune, particularly considering that, at train time, we do not have access to any test environment and thus cannot rely on validation techniques to tune the extra architecture.
the policy architecture (e.g., discriminator)
Inspired by recent research on mode connectivity (Benton al. 2021 Kuditipudi al. 2019) and by (Wortsman et al, 202T) which aims to learn a subspace of models in the supervised learning setting, we propose learn a subspace of policies in the parameter space as a solution the online adaptation in the RL setting (see Figure Ta). Each particular point in this subspace corresponds to specific parameter values, and thus to a particular policy. This subspace learned by adapting a classical RI. algorithm (PPO and A2C in our case, see Section (.3) such that an infinite contin- uum policies is learned, each policy having different parameters. The policies thus capture and process information differently, and react differently to variations of the training environment (see Figure lb). We validate our approach (Section 5 over a large set of reinforcement learning envi- ronments and compare it with other existing approaches. These experiments show that method competitive, achieves good results and does not require the use of any additional component of hyper-parameters tuning contrarily to baselines.
2

---

2 SETTING
Reinforcement Learning: one has access to a training Markov Decision Process (MDP) denoted M defined by transition distribution P(s'|s.a) a) S A x
Let us define a state space S and an action space A. In the RL setting,
R+, initial state distribution (s) S - R+ and
reward function r(s, a) S x A R.
R+. where I denotes the parameters of the policy. A trajectory sampled by a policy To given a MDP M is denoted T ~ (M). The objective of an RL. algorithm is to find policy that maximizes the expected cumulative (discounted) reward:
A policy is defined as To (als) S x A
0r =
arg max (M)
(1)
where R(T) is the discounted cumulative reward over trajectory
Online adaptation: We consider the setting where the policy trained over M will be used over another MDP (denoted M) that shares the same state and action space M, but with different dynamics and/or initial state distribution and/or reward function[] Importantly M is unknown at train time, and cannot be used for model selection, making the tuning of hyper-parameters difficult.
Given a trained model, we consider the K-shot adaptation setting where the test phase is decom- posed into two stages: a first phase in which the model adapts itself to the new test environment over K episodes, and a second phase in which the adapted model is used to collect the reward. We thus expect the first phase to be as short as possible (few episodes), corresponding to a fast adapta- tion to the new environment. Let us consider that a model wo generates a sequence of trajectories T1, T2 ..... T+ee over M, the performance of such a model, is defined as:
T Σ R(TK++)
M, K) lim To T
which corresponds to the average performance of the policy To over M after K episodes used for adapting the policy. Note that we are interested in methods that adapt quickly new a test environment and we will consider small values of K in our experiments. In the following, for sake of simplicity, K will refer to the number of policies evaluated during adaptation since each policy may be evaluated over more than single episode when facing stochastic environments.
3 LEARNING SUBSPACES OF POLICIES
Motivation and Idea: To illustrate our idea, let us consider a toy example where the train envi- ronment contains states with correlated and redundant features, in such a way that multiple subsets of state features can be used to compute good actions to execute. Traditional RL algorithms will discover one policy T-- that is optimal w.r.t the environment. This policy will typically use the state features in a particular way to decide the optimal action at each step. If some features become noisy (at test time) while, unluckily, Tg. particularly relies on these noisy features, the performance of the policy will drastically drop. Now, let us consider that, instead of learning just one optimal policy, we also learn second optimal policy T.- but enforcing 6* to be different than 0*, This second policy may tend to make use of various features to compute actions. We thus obtain two policies instead of one, and we have more chances that at least one of these policies is efficient at test time. Identifying which of these two policies is the best for the test environment (i.c., adaptation) can simply be done by evaluating each policy over few episodes, keeping the best one. Our model is built on top of this intuition, extending this example to an infinite set of policies and to variable environment dynamics.
Inspired by Wortsman et al. (2021) proposing to learn subspace of models for supervised learning, we study the approach of learning a subspace of policies in the parameter space, and the use of such a model for online adaptation in reinforcement learning. Studying the structure of the param- eter space has seen a recent surge of interest through the mode connectivity concept (Benton et al.] 2 Kuditipudi et al. 2019 Wortsman et al./ (222]) and obtain good results in generalization, but it has never been inv in the RL setting. As motivated in the previous paragraph, we expect that, given a variation of the training environment, having access a subspace of policies that process in- formation differently instead of a single policy will facilitate the adaptation. As result, our method very simple, does not need any extra paramm-t tuning and achieves good performance.
In the experimental study, one training environment is associated to multiple test environments to analyze the ability to adapt to different variations.
3

---

3.1 SUBSPACES OF POLICIES
Given @ the space of all possible parameters, subspace of policies is subset C + that defines a corresponding policies IT {
Since our objective is to learn such subspace, we have to rely on parametric definition of such subspace and consider as a simplex in O. Let us define N anchor parameter values 0I,...ON E
O We define the Z-space as the set of possible weighted sum of the anchor parameters: 2
2 = (21,..EN)) E [0, 1] Σ 2 ===1 =13. The subspace we aim to learn defined by:
θ
2g 42 E 2}
In other words, aim to learn convex hull of N vertices in O. Note that policies in this subspace can be obtained by sampling 2 ~ p(z) uniformly over 2.
The advantages of this approach are: a) the number of parameters of the model can be controlled by choosing the number N of anchor parameters, b) since policies are sharing parameters (instead of learning set of independent policies),
expect that the learning will be sample efficient. Such subspace is illustrated in Figure Iaa through the "pentagon" (i.e., N = 5) in which angles correspond the anchor parameters and the surface corresponds to all the policies in the built subspace.
K-shot adaptation: Given a subspace of policies θ, different methods can be achieved find the best policy over the test environment. For instance, it could be done by optimizing the distribution p(z) test time. In this article, we use the same yet effective K-shot adaptation technique than Kumar et al. (2020b) and Osa et al. (202T): we sample K episodes using different policies defined by different values
that are uniformly spread over Z. our example, it means that we evaluate policies uniformly distributed within the pentagon to identify good test policy (blue star). Note that, when the environment is deterministic, only one episode per value of a needs to executed to find the best policy, which leads to very fast adaptation.
3.2 LEARNING ALGORITHM Learning subspace of policies can be done by considering the RL. learning problem as maximizing:
(
=
Ey [d(
(4)
Joe
Considering that Ô is convex hull defined in Equation 3 and using the uniform distribution p(2) over Z, the loss function Equation 4 can be rewritten as:
N
C(0 = Ez-p((= [Er [R(T)]] with
(5)
Maximizing such objective function over 01, By outputs (uniform) distribution of policies trained to maximize the reward, all these policies sharing common parameters.
Avoiding subspace collapse: One possible effect when optimizing L(01,...ON) is reach solution where all 0g values are similar. In that case, all the policies would have the same parameters value, and will thus all achieve the same performance at test-time. Since we want encourage the policies to process information differently, and following Wortsman et al. (222)). we encourage the anchor policies to have different parameters. This implemented through the of regularization term denoted C(OI.....0 that measures how much anchor policies are similar in the parameter space. This auxiliary loss is defined pairwise loss between pairs of anchor parameters:
C(O
Σ cosine? 20,, 0,)
(6)
#
The final optimization loss is then:
(x) = Ez-p(z) Eran [[[T]] 3 Σ cosinee(f,, (0, with 0
Σ 2
Jo
where is an (see Section 5 for a discussion abot the tuning of this term) that weights the auxiliary term.
4

---

Initialize: 01, 82. Φ (Critic), batch size
for k 0, 1, 2... do
Sample 21, ....... An u,11]
Define 2z + 00 + 2:)02
Sample trajectories {r:}1 " using {TO.,,
Update 0 and 02 to maximize: Ei=1 Erro (0..) - 3 cosine? (01.02)
Up. @ to m n. Ei=1 LASE (d. 2;)
end
Figure 2: The adaptation of the PPO Algorithm with the LoP model. The different with the standard PPO algorithm is that: a) trajectories are sampled using multiple policies 0s, b) The policy loss is augmented with the auxiliary loss, and e) The critic takes the values 2; as an input.
3.3 LINE OF POLICIES (LoP)
In the case of N - 2, the subspace of policies corresponds to a simple segment in the parameter space defined by 0, and 0g as extremities. 0, and 02 are combined through a scalar value E [0; 1]:
= 201 + 101-241 2)0²
(7)
Computationally, learning line of policie is similar to learning a single policy for which the number of parameters is doubled, making this particular case good trade-off between expressivity and training speed. It corresponds to the following objective function:
(2) [2-1[[11]]
+(L-)) [((())]] cosine? ( (2)
We provide in Algorithm 7 the adapted version of the clipped PPO algorithm (Schulman et al., 2017) for learning subspace policies. In comparison to the classical approach, the batch of trajectories is first acquired by multiple policies sampled following p(=) ( (line 2-3). Then the PPO objective is optimized taking into account the policies used when sampling trajectories (line 4). At last, the critic is updated (line 5), taking as input the value so that it make robust estimations of the expected reward for all the policies in the subspace. Adapting off-policy algorithms would be similar. Additional details are provided in appendix. Note that, for environments with discrete actions, we have made the same adaptation based on the A2C algorithm since A2C has less hyper- parameters than PPO and is easier tune, with similar results.
EXPERIMENTS
We perform experiments in 6 different environments. Implementations based on the SaLinA (De- noyer et al.) (021) library together with train and test environments will be released upon acceptance. For each environment, we consider one train environment on which we trained the different methods, and multiple variations of the training environment for evaluation resulting in 50 test environments in total. The details all the environment configurations and detailed performance are given in Ap- pendix B Note that the complete experiments correspond to hundred of trained policies, and dozens of thousands of policy evaluations. For simple control environments (i.e., CartPole, Pendulum and AcroBot), we introduce few variations of the physics constant test-time, for instance by varying the mass of the cart, the length of the pole. For complex control environments (i.e., HalfCheetah and Ant using the BRAX library (Freeman al. (02I), both use variations of the physics (e.g.. gravity), variations of the agent shape (c.g., changing the size of the leg, or of the foot) and sensor alterations. At last, in MiniGrid and ProcGen we perform experiments where the agent is trained in one particular levels, but is evaluated in other levels (single levels on MiniGrid, and set of 10 levels ProcGen). Note that ProcGen a pixel-based environment where the architecture of the policy much more complex than in control environments. Toy experiments in Appendix B.8 to show the nature of the policies learned by the different methods.
simple Maze 2d are given
We compare approach LoFd with different state-of-the-art methods: a) The Single approach is just a single policy learned on the train environment, and evaluated on the test ones. b) The DI-
Other ways to control the shape the subspace can be used and we investigate some of them Section 'We consider the LoP-A2C and the LoP-PPO models for environments with respectively discrete and con- tinuous actions. LoP-PPO could he also used the discrete case but requires more parae-parrmeter tuning.
5

---

AYN+R(reward) method is an extension of DIAYN EEsenbacch et al. 2018) where a set of discrete policies is learned using a weighted sum between the DIAYN reward and the task reward:
RDIAYN+R(S, a) r(s, a) + 3 p(zss)
Critically, this model requires to choose discriminator architecture to compute log p(z]() and mod- ifies the train reward by defining an intrinsic reward that may drastically change the behavior of the policies at train time. <) At last, we also compare with the model proposed in (Osa et al. (021) denoted Lc (Latent-condittonne) that works only for continuous actions. This model is also based on continuous variable sampled uniformly at train time, but only uses an auxiliary loss without changing the reward. This auxiliary loss is defined through the joint learning of a density estima- tion model log P(z|s, a) where back prockpropagation is made over the action As in DIAYN+R, this model needs to carefully define a good neural network architecture for density estimation. Since Lc cannot be used with environment that have discrete actions, DIAYN+R Cont.) using a continuous 2 variable (instead of a discrete one) and density estimation model log P(z(s) as in Osa et al. (021). Note that we do not compare to (Kumar et al. 2020a) for the exact same reason as the one identified in (Osa et al. 2021]: SMERL assumes that the re- ward is known over the complete trajectories which results in unnatural adaptation of on on-policy RL algorithms like PPO. Moreover, preliminary experiments with SMERL does not demonstrate any advantage against DIAYN+R correctly tuned. We also provide (see Table in Appendix B. results where K independent policies are learned, the best one being selected over each test environment. This approach obtains lower performance than the proposed baseline and needs K more training samples making it unrealistic in most of the environments.
have adapted DIAYN+R (called
As network architectures, we use multi-layer perceptrons (MLP) with ReLU units for both the policy and the critic (detailed neural network architectures are described in Appendix). For DIAYN+R log P( s..)) is also modeled by a MLP with a soft-max output. For Le and DIAYN+R Cont., log (( ...) is modeled by MLP that computes the mean of Gaussian distribution with fixed variance. For these baselines, z is concatenated with the environment observation as an input for the policy and the critic models.
To choose the Syper-parameetrs of the different methods, let us remind that test environments cannot be used train time for doing typer-parameten search and/or model selection which makes this setting particularly difficult. Therefore, we rely on the following procedure: a grid-search over hyper-parameters is made, learning single policy over the train environment. The best value of the
is then selected as the one that provides the best policy at train time. These hyper-parameters are then used for all the different baselines. Concerning the 3 value, for LoP, we
report test results for B = 1.0 while, for Le and DIAYN+R, we use the best value of / on test environments. This corresponds to an optimistic evaluation of the baseline performances; aiming at showing that our method is much more efficient since it does not need such beta-tuning (3 1.0 giving good performance in the majority of cases). Said otherwise, we compare our model in the less favorable case where baselines have been unrealistically tuned.
For the adaptation step, each policy is evaluated over 10 episodes for stochastic environments or single episode for deterministic environments. We repeat this procedure over 10 different training seeds, and report the reward over the different test environments together with standard deviation. All detailed results available in Appendix.
ANALYSIS
We report the test performance of the models on different environments in Table In all the environ- ments, the adaptive models perform better than learning a single policy over the train environment which is not surprising. In most of the cases, LoP is able achieve better performance than other methods. For instance, on HalfCheetah where we evaluate the different methods over 16 variations of the train environments, LoP achieves an average reward of 10589 while Le and DIAYN+R obtain respectively 9547 and 9680 (standard deviations are reported in Appendix B). Some examples of the discovered that behaviors in Ant and HalrChectalffoor the different methods, and for different values of z are illustrated in Figures 1b 5 and 6 This outlines that learning models that are optimal on the train task reward, but with different parameter values, allows us to discover policies react differently to variations of the training environment. It seems to be a better approach than encouraging poli- cies to have a different behaviors (i.e., generating different state distributions) at train time. Same
Videos available https::/ / sisessttee.. .oogl ccm wwwp://vwwwwwweeoeom
i ciiee/hoeeee

---

CarPole
Acmbol
Fendulum 3
Minignd 6
Brax HalrCheetah
Bax Ani
Proction
Nb. Test Env.
6
16
15
Type actions
Diser:
Discr.
Diser:
Diser:
Cont.
Cont.
Diser:
Diser.
Single Policy
143.4
-99.7
-52.7
0.169
-83.2
LoP
149.9
-93.2
22899
0.447
10589
4031
16.38
33.6
DIAYN+R
168.1
-97.0
-47.1
0.248
DIAYN+R Lg
156.1
93.6
44.0
0.443
Table I: Average cumulated reward of the different models over multiple testing environments aver- aged over 10 training seeds (higher is better). For DIAYN and Lc, we report the results and tested 10 policies for LoP, Ic and DIAYN+R using episodes per policy for stochastic environments, and
episode per policy on deterministic ones. Performance is evaluated using the deterministic policy.
Standard deviation is reported for each single test environment in Appendix B
SmallFest
TlayPriction
BigGroviit
Train Perf.
K=d
8283
10425
19464
Lop
8
=
1.
0.1
4035
3905
3991
4031
4174
4164
7659
7630
DIAYN+K
K-10
8704
8805
19662
10734
10807
10578
10
3998
4012
4145
7670
10003
9766
Le DIAYN
0.1
3558
38.33
3949
7739
8156
3531
7166
3451
3759
2878
5388
K-10
8186
9661
9488
10
3356
3400
3109
4430
K-20
8107
9661
9506
Bop
K-3
0773
7507
7375
0.1
3909
4150
7767
(N-3)
K-10
6660
7640
9026
3820
3947
4126
7650
K-20
6996
7963
5015
3870
7710
COP
K-3
8996
9468
928T
(Nw3)
K-10
9230
9823
9568
K-20
9155
9979
9695
Figure 3: (left) Performance of the models at train time that shows that for LoP 3 is not hurting train performance while it is DIAYN+R. Standard error deviation is reported in Table [8.2 for each environment. We also report the performace at train time that shows that a too high value of / hurts DIAYN+R performance while is less critical in LoP. (right) Ablation study on the number of policies K used at test time on 3 HalfCheetah environment variations (see Appendix B.I for further details and additional results) together with the performance of the BoP and CoP variants. Standard deviation is given in appendix, Table 4
conclusions can be drawn in most of the environments, including MiniGrid where LoP is able to ex- plore large mazes while being trained only on small ones. On the ProeGen environment, where the observation is an image processed through a complex ConvNet architecture (See Appendix B.7), en- forcing functional diversity (DIAYN+R) does not allow to learn good policies while the LoP model is able to better generalize to unseen levels. Note that the performance at train time is the same for all the different approaches reported (see the Table for instance) but quickly decreases in DIAYN for larger values of B3 while it stays stable for LoP where the best results are obtained for 3 = 1.0.
Interestingly, in CartPole, DIAYN+R performs quite well. Indeed, when analyzing the learned policies, it seems to be a specific case where it is possible to obtain optimal policies that are diverse w.r. the states they are sampling (by moving the cart more or less on the right/left while maintaining the pole vertical).
We have also performed experiments where test environments have the same dynamics the train- ing environment, but with defective sensors (i.c., some features at test time have null value see Appendix Table Table on the Ant environment). The fact that LoP behaves also well confirms the effectiveness of our approach to different types of variations, including noisy features on which baselines methods were not applied in previous publications.
Sensitivity to hyper-parameterrs. One important characteristic of LoP is that it can be used with = 1.0 and does not need to define any classifier architecture as opposed to DIAYN+R and Lc. Indeed, shown in Figure 3 (left), the training performance of DIAYN drastically depends on good tuning of B. Lc, which is less sensible, needs to use a correct classifier architecture as in DIAYN. LoP is simple to tune since the cosine term is usually easy satisfy and approach, at convergence, always reaches a 0 value on this term when 3 > 0.0. As illustrated in Appendix B. and B2, it is also interesting to note than, the BRAX environments, the number of environment interactions needed to train LoP is similar than the one needed to train single policy and LoP comes with very small overhead in comparison classical methods.
Online adaptation: One interesting property is the number of policies (and thus of episodes) to test over new environment to get a good performance. For LoP and Lc, given trained model,
7

---

Figure 4: (left) Trajectories generated by K = 10 policies (one rows) on an unseen maze (objective to go from left to right, see details Appendix B.8) for DIAYN+R (left column with best value) and LoP (right column with 3 1.0). It illustrates the diversity obtained with DIAYN+R and LoP. (right) Number of times (y-axis) each policy (x-axis) (over K tested policies) chosen over the 16 test environments of the HalfCheetah setting for each of the 10 seeds. Blue is LoP and orange is DR Different policies are used for different test environments showing the interest of learning subspace of policies. Note that in LoP, the anchor policies are rarely chosen. Results for = 10 and K = 20 in Appendix (Figure
one can evaluate as many policies (i.e., different values of 2) as desired. For DIAYN+R, testing more policies also means training provides the reward of the different methods when testing K policies on different HalfCheetah settings: as expected, the performance of DIAYN+R tends to decrease when K is large since the model has difficulties to learn too many diverse policies. For LoP and Lc, spending more episodes evaluate more policies naturally leads better performance: these two models provide better way to deal with the sxploration-exploiaaae
policies which is expensive and less flexible. Table (right)
trade-off at test time. Again, please consider that Le also needs define additional neural network architecture model log P(z|s, a) while LoP does not, making our approach simpler.
Beyond Line of Policies: While LoP based on the learning of N anchor parameters, is possible to combine more than two anchor parameters. We study two approaches combining N anchor parameters (that can be extended to 3): a) the first approach is convex combination of policies (CoP) where is sampled following Dirichlet distribution.
(b) The second approach is Bézier combination (BoP) as explained Appendix A.2 The results are presented in Table (right) multiple HalfCheetah environments. can be seen that these two strategies LoP is thus a good trade-off between the number of parameters to train and the performance (Note that BoP and CoP need more samples converge), at least given the particular neural network architectures we have used in this paper. We also performed an in-depth analysis of the evolution of the reward when K is increasing for LoP and CoP in Halfcheetah Annex). While we expected CoP to outperform LoP when K is high, the best reward becomes stable when K=20 for both methods, and most test environments, CoP is not able reach the same best reward as LoP.
efficient.
environment (Figure D in
Analysis of the learned policies: To better understand the nature of the policies discovered by the different approaches, we have made a qualitative study in which analyze i) the robustness of the methods to corrupted observations, ii) the functional diversity induced by the different models, and iii) the specificity of the different learned policies to particular test environments. First, LoP is more robust to input feature corruption (see Table B.2 for the results, and Table 3 for the setting in Appendix) and we conjecture that it is because the diversity in the parameter space allows this model to learn policies that does not take into account the same input features equally. also measure the functional diversity induced by the different models by training posteriori a classifier that
8

---

aims at recovering which policy (i.e which value of 2) has generated particular trajectories (Exact protocol in Figure in Appendix, with the training curves). On LoP with K = 5, such a classifier obtains 82% accuracy at validation time showing that the 5 policies are quite diverse, but less than the DIAYN+R policies where the classifier reaches a 100% accuracy which is logical knowing the auxiliary loss introduced by DIAYN which enforces this type of diversity. It is interesting note that with the trajectories generated in the test environments with LoP policies, the accuracy of the classifier is reaching 87 %: when LoP is facing new environments, tends to generate more diverse policies. We think that it is due to the fact that, since the policies have different parameter values, they react differently to states that have not been encountered at train time. At last, Figure (right) (and Figure 7 in appendix for k=)0, illustrates which upon K 5 policies is used for different test environments. It shows that both LoP and D use different policies over different test environments, showing that these methods are able to solve new environments by learning various policies and not a single but robust one. Examples of policies on a simple maze2d are given in Figure 4 (left) and Appendix which illustrate the diversity of the discovered policies.
6 RELATED WORK
Our contribution shares connections with different families of approaches. First of all, it focuses on the problem of online adaptation in Reinforcement Learning which has been studied under different
terminologies: Multi-task Reinforcement Learning (Wilson et al. 2007: Teh et al.. (017), Transfer
Learning Finn el al.) 2017 Hausman et al. [2018 Humplik et al., 2019). Many different methods have been proposed, but the best majority considers that the agent Is trained over multiple environments such that it can identify variations (or invariant) at train time. For instance, Duan et al. (2016) assume that the agent can sample multiple episodes over the same environments and methods like (Kamienny et al. 2020 Liu et al. 202I) consider that the agent has access to a task identifier at train time.
Learning (Taylor and Stone 2009; Lazaric 2012) and Meta-ReinJoreemen
More recently, diversity-based approaches have been adapted to focus on the setting where only one training environment is available. They share with our model the idea of learning multiple policies instead of single one. For instance, DIAYN (Eysenbach et al., 2018) learns discrete set of policies that can be reused and fine-tumed over new environments. It has been adapted to online adaptation in (Kumar et al., 2020a) where the authors propose to combine the intrinsic diversity reward together with the training task reward. This trade-off is obtained through a threshold-based method (instead of a simple weighted sum) with good results. But this method suffers from a major drawback identified in (Osa et al. 202T): it necessitates to sample complete episodes at each epoch which is painful and not adapted to all the RL learning algorithms. Osa et al. (202T) also proposed an alternative based on learning a continuous set of policies instead of a discrete one without using any intrinsic reward.
The method we propose is highly connected to recent researches mode connectivity with neural networks. Mode connectivity is a set of approaches and analyses that focus on the shape of the parameter space. It has been used as tool to study generalization in the supervised learning setting (aariovv et al. 2018). but also as a way to propose new algorithms in different settings (Mirzadch let al. (021). Obviously, the work that is the most connected to our approach is the model proposed in (Wortsman et al.) (02T) that provides a way to learn a subspace of models in the supervised learning setting. Our contribution adapts this approach to RL for learning policies in completely different setting which is online adaptation.
At last, our work is sharing similarities with robust RL which aims at discovering policies robust to variations of the training environment (Oikarinen et al.) 2020 Zhang et al.] 202I). The main difference is that robust RL techniques learn policies efficient "in the worst case" and are not focused on the online adaptation to test environments (the objective is usually to learn single policy efficient on different variations while we are learning multiple policies, just selecting one at test time).
7 CONCLUSION AND PERSPECTIVES
We investigate the idea of learning a subspace of policies in the reinforcement learning setting, and describe how this approach can be used for online adaptation. While simple, our method allows to obtain policies that are robust to variations of the training environments. Contrarily to other techniques, LoP does not need any particular tuning or definition of additional architectures to handle diversity, which is a critical aspect in the online adaptation setting where saper-parameters tuning is impossible at least very difficult. Future work includes the extension of this family of approaches in the continual reinforcement learning setting, the deeper understanding of the the built subspace and the investigation of different auxiliary losses to better control the shape of such a subspace.
9

---

8 REPRODUCIBILITY STATEMENT
We have made several efforts to ensure that the results provided in the paper are fully reproducible. In Appendix, we provide a full list of all hyperparameters and extra information needed to repro- duce our experiments. The source code is available on SaLinA repository such that everyone can reproduce the experiments
REFERENCES
G. W. Benton, W. Maddox, S. Lotfi, and A. G. Wilson. Loss surface simplexes for mode connect- ing volumes and fast ensembling. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021. 18-24 July 2021. Virtual Event, vol- ume 139 of Proceedings of Machine Learning Research, pages 769-779, PMLR, 2021. URL
http::/pprccediing mlr. ptress//13//eeettn222 1tm.mmmlllllllmm..
G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. Openai gym, 2016.
M. Chevalier-Boisvert,, L. Willems, and S. Pal. Minimalistic gridworld environment for openai gym. 2018.
https://9 thub. cetmmmmaiimmett
L. Denoyer, A. de la Fuente, Duong, J.-B. Gaya, P.-A. Kamienny, and D. H. Thompson. Salina: Sequential learning of agents. https://githni wwm http:////.cccccooeeeercccceeeeea 2021.
Y. Duan, J. Schulman, X. Chen, P. L. Bartlett, I. Sutskever, and P. Abbeel. RIS Fast re- inforcement learning via slow reinforcement learning. CoRR, 0bs/1611.027799 2016. URL http:://rxi:> org/
L. Espeholt, H. Soyer, R. Munos, Simonyan, V. Mnih, T. Ward, Y. Doron, Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures, 2018.
B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without reward function. CoRR, abs/1 802.06070, 2018. URL
06070
C. Finn, Abbeel, and S. Levine. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks. arXiv e-prints, art. 1tXiv:1703.03400 Mar 2017.
C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax differ- entiable physics engine for large scale rigid body simulation, 2021. URL http://github. com/ google/brax
T. Garipov, P. Izmailov, D. Podoprikhin. D. P. Vetrov, and A. G. Wilson. Loss sur- faces, mode connectivity, and fast ensembling of dnns. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianche and R. Gamett, editors, Advances in Neu- ral Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pages 8803 8812, 2018. URL -astsaacccbbetttt-aaatttttt be3087e
neuripss........................ wwwppauwppps..pppppepeeeee pc/ppaper/211//haas http:://coooccccaeeeeeeeeeeeaaas
Abstract html
K. Hausman, J. T. Springenberg. Z. Wang, N. Heess, and M. A. Riedmiller. Learning an embedding space for transferable robot skills. In ICLR (Poster). OeenReview.net, 2018.
Henderson, W.-D. Chang, Shkurti, Hansen, Meger, and G. Dudek. Benchmark environ- ments for multitask learning in continuous domains. ICML Lifelong Learning: A Reinforcement Learning Approach Workshop, 2017.
http:////cttiaarrroooeeeeeeeeeea wttpaaaiiiinnnnaaaaaeaaeaee _a___exapp'' examp'ees es/
r1/ 11/subspace_o__aallici.
10

---

J. Humplik, A. Galashov, L. Hasenclever, P. A. Ortega, Y. Whye Teh, and N. Heess. Meta reinforce- ment learning as task inference. arXiv e-prints, art. M.Kiv:1905..0422. May 2019.
P. Kamienny, M. Pirotta, A. Lazaric, T. Lavril, N. Usunier, and L. Denoyer. Learning adaptive exploration strategies in dynamic environments through informed policy regularization. CoRR, 0bs/2005.02934, 2020. URL https://arxiv org. abs 2005. 02934
R. Kuditipudi, X. Wang, H. Lee, Zhang. Z. Li, W. Hu, R. Ge, and S. Arora. Explain- ing landscape connectivity of low-cost solutions for multilayer nets. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alcéé-uuc, vances in Neural Information Processing Systems 32: Annual Conference Neural Informa- tion Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 14574 14583, 2019. hash/ 0aeaa4444551/1333344434444333333
B. Fox, and R. Garnett, editors, Ad-
http:tppsspprroeeeineeeooooooooe ngs www.rps......... wwtpurrisscccpppppeeeeee cctpaper//019////////pppp/
wtw.:/aaaddda--bstrreeaaae htmll
S. Kumar, A. Kumar, S. Levine, and C. Finn. One solution is not all you need: Few-shot ex- trapolation via structured maxent RL. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020a. URL httpp::/ haah/5d151
www.ssneurrp.p.. wttpeipp....ppppee../eeep ccppappe//22// Awettrtstttt........ html
9atphh/55511111199222222 2 335a 335 wtw.al333330111322113330
S. Kumar, A. Kumar, S. Levine, and C. Finn. One solution is not all you need: Fhw-shot extrapola- tion via structured maxent RL. CoRR, abs/2010. 2020b. URL (ttth:::://arxx org/ abs/2010. 14484
A. Lazaric. Transfer in reinforcement learning: a framework and a survey. In Reinforcement Learn ing, pages 143-173. Springer, 2012.
E. Z. Liu, A. Raghunathan, Liang, and C. Finn. Decoupling exploration and exploitation for meta-rett reetrrreiforrmmmnnt learning without sacrifices. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual PMLR,
Event, volume 139 of Proceedings of Machine Learning Research, pages 2021. URL httt:::ttpppceeeeeee
S. Mirzadeh, M. Farajtabar, D. Görür, R. Pascanu, and H. Ghasemzadeh. Linear mode connectivity in multitask and continual learning. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https:// openreview.nete
f
V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Harley, T. P. Lillicrap. D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33rd Interna- tional Conference on International Conference on Machine Learning Volume 48, ICML'16, page 1928-199 1937. JMrg 2016.
T. P. Oikarinen, T. Weng, and L. Daniel. Robust deep reinforcement learning through adversarial loss. CoRR, abs/2008.0 01976, 2020. URL URLlhttss://axxt http:/vv..rrabb/////baaa/ 2ttp://////////////aa/////////// 01976
T. Osa, V. Tangkaratt, and M. Sugiyama. Discovering diverse solutions in deep reinforcement learn- ing, 2021.
C. Packer, K. Gao, J. Kos, P. Krähenbühl, V. Koltun, and D. Song. Assessing generalization in deep reinforcement learning, 2018.
X. B. Peng, M. Andrychowicz. W. Zaremba, and P. Abbeel. Sim-to-Real Transfer of Robotic Control with Dynamics Randomization. arXiv e-prints, art. arXiv:1710.06537, Oct 2017.
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization
algorithms. CoRR, abs/1707.063477 2017. /Rtptttpp//aax....orrr/
J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation, 2018.
11

---

D. Silver, T. Hubert, J. Schrittwieser. I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Ku- maran, T. Graepel. T. P. Lillicrap. K. Simonyan, and D. Hassabis. Mastering chess and shogi by self-play with general reinforcement learning algorithm. CoRR, abs/1712.01815, 2017. URL htt.p://arxx
M. E. Taylor and P. Stone. Transfer leariigg for reinforcement learning domains: A survey, J. Mach. Learn. Res., 10:1633-1685, Dec. 2009. ISSN 1532-4435.
Y. W. Teh, V. Bapst, W. M. Czameckii Quan, J. Kirkpatrick, Hadsell, Heess, and R. Pascanu. Distral: Robust multitask reinforcement learning. CoRR, abs/1707.04177 2017. URL http:
//arxiv. wwwxivorr/aab/////000 2ttgg//bb/1700.1177 04175
N. Ward, A. Smofsky, and A. J. Bose. Improving exploration in soft-actor-critic with normaliz- ing flows policies. CoRR, abs/1996.0277/ 2019. URL http://aax....rg//bb/////
02771
A. Wilson, A. Fern, S. Ray, and P. Tadepalli. Multi-task reinforcement learning: A hierarchical bayesian approach. Proceedings of the 24th International Conference on Machine Learning, ICML '07, page 1015-1022, New York, NY. USA, 2007. Association for Computing Machin- ery. ISBN 978 595933933, doi: 10.1145/1273496.12733 1145/1273496 1273624
URL
M. Wortsman, M. Horton, C. Guestrin, A. Farhadi, and M. Rastegari. Learning neural network subspaces. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Con- ference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 11217-11227. PMLR, 2021. URL http: \/proceedings. www. htl..ppess/vl39. http:///w.../o.mmmmmmaaaaaaaaaaa html
H. Zhang. Chen, S. Boning, and C. Hsieh. Robust reinforcement learning on state observations with learned optimal adversary. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, 2021. URL https://
openreview net/forum 21=mmmmc1ccccccceeee
12

---

A IMPLEMENTATION DETAILS
In this section, we provide details about the implementations of the baselines and our models. Ap- pendix B provides details and additional results for each environment used.
A.1 LOP-PPO
First, we recall the clipped-PPO surrogate objective de- scribed in Schulman et al. (017)) For given trajectory {(st. oe, re) collected by the policy
We detail the losses used in algorithm 7
Toas and denoting P: (0) Tolae the goal to maximize:
Matta
1 Σ min (ec (0) A (st, clip (pi (0) 1 c, -) A (st, (10) EppO (0)
T
Where function Rota is computed thanks to value function V by using Generalized Advantage Estimation (Schulman et al.] (2018)). This function simultaneously updated by regression on mean-squared error over the rewards-to-go R. In our case, this function not only takes st as an input, but also the value
1 EMSE (6. (0:2)== == T
Σ 1=0
Vo (0s,, 2) Rs)
(11)
In HalfCheetah and Ant experiments, we sampled actions from a reparametrized Gaussian distri- bution using squashing function (Ward al. (2019)), but set the standard deviation fixed (so is an hyper-parameter encouraging exploration, called action sid in Tables and 6 the policy network only learns the mean of this distribution.
A.2 BoP AND CoP
The only change between LoP and these models resides in the way we combine the N anchor policies. For CoP. is just the generalization of LoP for N > (see B.I). BoP, makes use Bezier parametric curve that uses Bernstein polynomials (the anchor parameters being the control points). For N = 3, it defined by:
θ= (11 2)a 0, 21 2) 2), + 2 0g. Ve [0, 1]}
(12)
Concerning the policies evaluated at test time, BoP uses the same strategy LoP by testing values that are uniformly distributed in [0; 1]. For CoP, we opted for sampling K policies using Dirichlet distribution over [0, 113.
A.3 DIAYN+R AND LC
In order to find the best trade-off between maximizing environment rewards and intrinsic rewards in DIAYN+R algorithm, we add the
3:
RDIAYN4R(S,0) r(s, + B logp(z|s)
(13)
As alternative to DIAYN+R Osa et al. (202T) proposes algorithm where the discriminator takes not only observations as an input but also the policy output, updating both discriminator de
and policy ne when back propagating the gradient. In this case, it is not necessary to add intrinsic reward. While Osa et al. (2021) 1) illustrate their methods with TD3 and SAC, we adapted to PPO.
The surrogate loss is given by:
Lio = Eppo B 1ogog 4s (z 8, ne (-18,2)) 2))
(14)
13

---

B EXPERIMENTS DETAILS AND ADDITIONAL RESULTS
B.1 HALFCHEETAH
Task originally coming from OpenAl Gym (Brockman et al. 2016). Instead of using MuJoCo engine, we decided to use Brax (Freeman et al. 2021) as it enables the possibility to acquire episodes on GPU. We use the vanilla environment for training. The policy and the critic are encoded by two different multi-layer perceptrons with ReLU activations. The base learning algorithm is PPO.
Test environments: we operated modifications similar as the ones proposed in (Henderson et al, 2017). Morphological variations: we changed the radius and mass of specific body parts (torso, thig, shin, foot). Variations physics: we changed the gravity and friction coefficients. Table precisely indicates the nature of the changes for each environment.
Env name
Modifications
BigFeet
Feet mass and radius x1.25
BigFriction
Friction coefficient x x..25
BigGravity
Gravity coefficient 1.25
BigShins
Shins mass and radius x 1.25
BigThighs
Thighs mass and radius x x1.25
Torso
Torso mass and radius x1.25
SmallFeet
Feet mass and radius x0.75
SmallFriction
Friction coefficient x0.75
SmallGravity
Gravity coefficient x0.75
SmallShins
Shins mass and radius x0.75
SmallThighs
Thighs mass and radius x0.75
SmallTorso
Torso mass and radius x0.75
HugeFriction
Friction coefficient x1.5
HugeGravity
Gravity coefficient x1.5
FritFoiction
Friction coefficient x0.5
TinyGravity
Gravity coefficient x0.5
Table Modified HalfCheetah environments used for testing. Morphological modifications include variation on the mass and the radius of a specific part of the body (torso, thighs, shins, or feet). We also modified the dynamics (gravity and friction). Environment names are exhaustive: Big refers to increase of 25% of radius and mass, Small refers to decrease of 25%. For example, "BigFoot" refers to an HalfCheetah agent where feet have been increased in mass and radius by 25%. For gravity and friction, we also tried an increase/decrease by 50% (respectively tagged "Huge" and "Tiny").
14

---

Hyper-parameter
Value
Ir policy:
0.0003
Ir critic:
0.0003
parallel environments:
2048
n acquisition steps per epoch:
20
batch size:
512
num minibatches:
32
update epochs:
discount factor:
0.99
clip ration:
0.3
action std:
0.5
gae coefficient:
0.96
reward scaling:
1.
gradient clipping:
10.
layers (policy):
4
n neurons per layer (policy):
64
layers (critic):
5
n neurons per layer (critic):
256
LoP. BoP, CoP
3:
DIAYN
A:
0.1
Ir discriminator:
0.0001
n layers (discriminator):
2 64
neurons per layer (discriminator):
Lc
3:
dimensions of Ir discriminator: layers (discriminator):
1
0.001
2
neurons layer (discriminator):
64
Table 3:
for PPO over HalfCheetah
15

---

Single Policy (K=1)
Ensemble (5 policies)
LoP'
DIAYN +
Le
Bol
Col
K=5
BigFect
7433 + 1988
8115 344
8895 289
8454 = 655
8353 804
7993 702
8087 496
BigFriction
8579 +: 2224
10757 2 319
11635 355
9962 + 2113
10649 2085
8846 = 2279
10596 : 1738
BigGravity
7508 + 2086
9131 = 260
10464 798
8989 1723
9360 1531
7878 1593
9287 = 1200
BigShins
7274 898
7992 + 294
8879 98
8099 806
8206 770
7726 903
8226 1144
Thig
7963 +: 1466
9331 +: 310
10054 +: 724
8940 1558
9360 #: 1669
8105 # 1622
9525 2 832
Big Torso
7091 2221
8942 + 94
9834 + 310
8701 1472
9198 1342
7790 1713
8927 1128
SmallFeet
5973 + 2490
10012 + 1921
8283 + 648
7580 1411
8186 1512
6775 2308
8996 + 1150
SmallFriction
8652
7299
11391
9813 2074
1617
8652 2075
1387
SmallGiravity
9004 4: 1665
8004 +: 595
11840 406
10132 +: 1903
10434 a: 1951
9428 : 2180
10594 = 1438
SmallShin
7492 + 2999
10379 257
10840 196
9540 = 1592
9837 1343
8451 = 1889
9967 1079
SmaliThig
8914 1: 1837
11133 a: 384
11294 = 46
10078 1410
10603 1160
9340 1751
10603 1540
SmallTorss
8885 + 1.522
9669 + 213
11433 2 360
9850 #: 1761
10010 #: 1856
9182 1779
10092 # 1541
Fnigeoictton
6999 + 3441
10006 560
11537 613
9483 2228
10305 2104
8387 2515
10749 1591
HugeGravity
6133 + 2147
10452 +: 371
8425 + 554
7700 #: 1216
7621 = 1137
6532 1133
7632 903
TinyFriction
91.32 1862
2003
TinGrrvvty
7304 + 3484
9887 + 1822
11385 169
9363 1734
9620 a: 2041
9118 2360
9020 2028
Average
7697
10413
9465
Single Policy
LoP
DIAYN R
L.c
BoP
CoP
(K=1)
K-10
BigFeet
+
8553 1033
BigFriction
8579 + 2224
11982 +30 330
10781 1870
10705 + 2092
9093 + 2436
10850 + 1526
Big BigGravity
7508 + 2086
10578 648
9766 1424
9488 1616
8026 1809
9568 =
BigShins
7274 1 898
8854 : 153
8276 607
8340 1 986
7831 1 978
8753 866
Big Thig
7963 #: 1466
10335 #: 1001
9644 a 1323
9427 + 1532
8080 =: 1676
9591 : 884
BigTorso
7091 + 2221
10023 427
9271 1317
7928
SmallFeet
5973 + 2490
8805 :: 495
8255 504
8186 1458
6660 =: 1842
9210 1063
SmallFriction
+
1505
2007
10459
SmallGravity
9004 + 1665
11969 341
10568 1906
10444 + 1984
9569 2418
11334 1429
SmallShin
7492 + 2999
10764 + 147
9990 + 1107
9933 = 1315
8441 1801
9898 = 982
SmalfThig
8914 1: 1837
11524 298
10689 1137
10632 1170
9430 1881
10600 1473
SmalTTarso
8885 # 1522
11567 + 381
10328 1736
10273 + 1917
9228 = 1736
10541 # 1135
HugeFriction
6999 3441
116.9 307
10335 1955
10379 2302
8526 2710
10899 1541
GrgeGravity
6133 # 2147
8793 = 791
8124 # 1086
7811 1115
6573 = 1138
8071 = 352
TinyFictiion Friction
1226
9661 1497
7840 2019
9523 1537
TinyGravity
7304 + 3484
11578 460
9723 + 2167
9650 2228
9221 2224
5107 6910
Average
7697
10589
9680
9547
8331
9506
Single Policy (K=1)
DIAYN
K=20
BigFect
7433 + 1988
9096 + 257
8363 + 800
8431 + 803
8405 + 961
8754 + 104]
BigFriction
8579 2224
9655 2304
10779
9108 2437
11124 1397
BigGravity
7508 a: 2086
10807 592
8695 1550
9506 1609
8015 1834
9695 + 879
BigShins
7274
192
670
959
7994 1046
415
BigThig
7963 + 1466
10521 + 1016
8639 1315
9537 1633
8159 1602
9652 + 1012
Borso
7091 + 2221
10084 449
8494 1376
9383 1363
7873 1612
9462 299
SmallFeet
5973 + 2490
8794 + 614
7540 1155
8107 1506
6996 2483
9155 497
SmallFriction
8652 + 1717
11429 + 319
9603 2022
10271 = 1680
8844 +: 2162
10498 = 1331
SmaliGrravity
9934 2370
10521
9666 2280
:
SmallShin
7492 +: 2999
11041 = 420
91.35 1715
10057 = 1281
8709 2127
10025 803
SmallThig
1837
9970 1847
10682
9563 1820
10921
SmallTorso
8985 + 1522
11673 434
9620 2009
10315 = 1880
9301 1814
10602 + 1297
HugePriction
6999 +: 3441
11724 : 685
9438 2471
10505 = 2355
8553 2 2603
11047 = 1167
HugoGravity
6133 +: 2147
89330 714
7498 837
7897 1147
6772 1413
7925 + 740
TrnyFriction
7953 + 1843
10734 2: 381
9001 1908
9661 1519
7963 2234
9979 + 878
Traviravity
11604
9419 2317
9822 2200
9329
Average
7697
10691
8933
9617
8453
9990
Table 4: Mean and standard deviation of cumulative reward achieved on HalfCheetah test sets per model (see Table 2 for environment details). Results are averaged over 10 training seeds (i.e., 10 models are trained with the same hyper- hyper-parameters. parameters and evaluated on the 16 test environments). K is the number of policies tested at adaptation time, using episode per policy since this environment is deterministic. Ensembling with K = 5 models takes times more iterations to converge and testing values of K > 5 is very costly in terms of GPU consumption.
16

---

0.25
0.5
0.75
Figure 5: Qualitative example of LoP trajectories on HalfCheetah "BigShins" test environment (5-
shot setting). The best reward obtained for 0.75.
0.75
Figure 6: Extreme case: when torso radius and mass are increased by 50%. Only one policy is able adapt without falling down 0.5).
17

---

K=6
K=10
=
Lill
Mm ld
Figure 7: Number of times (y-axis) each policy (x-axis) is chosen by k-shot evaluation over the 16 test environments of the HalfCheetah for each of the 10 seeds (one table per seed). In blue, LoP, in orange, DR Please note that the 10 same LoP models are used for K=5, K=10, K=20 which the case for DIAYN+R.
18

---

10
09-
08
accuracy
06-
0.4
train ensironmnnt environment
02-
DIYN++
20000
40000
GOO00
00000
300000
Figure 8: We trained small discriminators over dataset (100,000 environment interactions) of tra- jectories obtained with the learned policies of LoP and DIAYN+R when K=5. For each environment, for each seed, we trained single discriminator and averaged the results. While the discriminators trained on DIAYN+R reach 100% accuracy rapidly on both train and test environments, they learn slower for LoP, with slight advantage for the environment, validating the fact that the diver- sity induced by the cosine similarity on the weights more visible variations of the environment rather than the environment on which the model has been trained. We evaluated the discriminator on a validation dataset (also 100,000 environment interactions) resulting in 100% accuracy for DIAYN in both train and test environments. For LoP, we obtained 82% accuracy on the training environment, and 87% on the test environments. The discriminator architecture consists in neural network of two hidden layers of size 16, taking the unprocessed states as an input and outputting the predicted policy used (like in DIAYN).
19

---

BigFeet
SmallFeet
Bigthigs
Smalltthigs
75.00
21000
8000
3000
reward
3500
5500
7000
8000
5000
5000
8002
7500
8500
8000
BigShins
Smalishins
Big Torco
Smalmorro
0000
11000
10000
reWard
9000
5500
305.0
0000
8000
5000
10000
8500
95000
5000
60
80
100
60
100
40
60
100
SmaliGravity
SiGGravity
SmallFrictio
12000
11000
reward
10000
30000
10000
11000
5000
9000
5500
20000
60
40
60
linyGravit
HkugeGravity
Fimyeriction
Hugelriction
12000
10000
05.00
10750
11000
reward
8000
8000
10000
10000
9750
2000
20:00
LoP (N=2)
65.00
CoP (N=3)
Figure 9: Evolution of the best reward obtained with respect to K for LoP (N=2) and CoP (N=3) for each Halfcheetah test environment. We ran the K-shot evaluation for each K from K=1 to K=100 using the method described in Appendix A.2] simply sample K random coefficients using the uniform distribution over [0, 1] for LoP and the Dirichlet distribution over [0, 1 for CoP. Results
averaged over 10 run for each K, and over the 10 models learned for each method.
20

---

B.2 ANT
Task originally coming from OpenAl Gym gine, decided to Brax (Freeman al. (202T) as it enables the possibility to acquire episodes on GPU. We use the vanilla environment for training. The policy and the critic are encoded by two different multi-layer perceptrons with ReLU activations. The base learning algorithm is PPO.
Brockman et al. (2016)) Instead of using MuJoCo en-
Test environments: As for HalfCheetah, we operated variations in physics (gravity and friction coefficients). We also designed environments with percentage of masked features to simulate defective sensors (They are sampled randomly and remain the same for each run). preciesyy indicates the nature of the changes for each environment
Env name
Modifications
BigFriction
Friction coefficient x1.25
BigGravity
Gravity coefficient x x1.25
SmallFriction
Friction coefficient x0.75
SmallGravity
Gravity coefficient x0.75
HugeFriction
Friction coefficient x1.5
HugeGravity
Gravity coefficient x1.5
TinyFriction
Friction coefficient x0.5
TinyGravity
Gravity coefficient x0.5
DefectiveSensor 5%
5% of env obs set to O
DefectiveSensor 10%
10% of env obs set to 0
DefectiveSensor 15%
15% of env obs set to 0
DefectiveSensor 20%
20% of env obs set to 0
DefectiveSenssrr 25%
25% of env obs set to 0
DefectiveSensor 30%
30% of env obs set to 0
DefectiveSensor 35%
35% of env obs set to 0
Table 5: Modified Ant environments used for testing.
21

---

Hlyperpaaammetr
Value
Ir policy:
0.0003
Ir critic:
0.0003
parallel environments:
2048
n acquisition steps per epoch:
20
batch size:
1024
num minibatches:
16
update epochs:
discount factor:
0.99
clip ration:
0.3
action std:
0.4
gae coefficient:
0.96
reward scaling:
1.
gradient clipping:
10.
layers (policy):
4
n neurons per layer (policy):
64
layers (critic):
5
n neurons per layer (critic):
256
LoP
3:
{0.1, 1, 10)
DIAYN
3:
{0.1, 1, 10) 0.001 2 64
Ir discriminator:
n layers (discriminator):
neurons per layer (discriminator):
Le
3:
(..1, 1, 10) 1 0.001 2
dimensions of z: Ir discriminator: layers (diseriminator):
neurons per layer (discriminator)::
64
Table Hyper-parameters for PPO over Ant
22

---

Single Policy
LoP
DIAYN +
Le
B
(K=1)
0.1
K=5
BigIiicttin
7454 + 166
7544= = 140
7470 + 96
7541 + 202
7256 1010
6403 + 726
6267 627
7666 103
7573 + 154
7538 # 1336
BigGravity
7038 211
6937
7027 = 182
5865 543
7123
7075 168
SmallFriction
6755 + 2073
7695 = 156
7652 1 126
7599 4 167
7046 4 1777
6491 1 807
6133 = 706
127 7846
7673 + 185
7734 4 211
SmallGiravity
7057
7738=
7748 169
6486 769
7876
7745 105
7772
HugeFtiction
7505 4: 252
7634 = 152
7616 2 68
7601 + 253
7220 4 1331
6357 752
6384 = 733
7799 103
7647 4: 128
7668 4 77
HugoGravity
380 +: 507
1111 538
1407
1494 934
556
2924 1992
2847 1598
934
915 465
1440
ThyFFicction
2747 + 1241
3716 = 996
4584 + 801
4070 1751
3540 + 948
2950 = 1113
2600 = 572
3550 843
4140 + 294
3487 508
TinyGrvvity
-520
-106 125
116 322
-479
4 202
224
790
-401 373
+29
DefectiveSensor 5%
4308 + 59
5243 322
56,30 + 124
5560 + 272
4958 + 126
4492 211
4360 338
5428 424
4988 + 123
5225 562
DefectiveSensur 10%
2770 171
3620 238
3660 + 328
3625 + 291
3440
3371 5:
3223 400
3519 76
3406 215
3491 207
DefectiveSensor 15%
1531 + 90
2583 247
2738 + 312
2740 + 167
1774 + 84
1984 383
2055 311
2408 340
2226 + 78
2349 + 271
DefectiveSensor 20%
1026 +
1750 249
1965 + 234
2008 + 199
1104 + 101
1490 387
1450 347
1658 245
1546 108
1714 = 2.38
DefectiveSensor 25%
736 9:
1595 = 317
1757 1: 154
1526 2 338
762 1 34
1107 4 396
1120 = 241
1344 262
1271 4: 205
1363 254
DefectiveSensor 30%
5:
695 112
793 166
674 120
577 34
859 + 190
766 + 212
673 115
575 +:
622 100
SefectiveSensor 35%
424 1 48
606 135
684 2 182
635 4 85
449 a. 68
6.50 2 245
565 4 151
618 4: 170
525 4: 92
602 A 175
Average
3338
3905
4035
3998
3558
3451
3356
3909
3820
3670
K=10
BigFriction
7454+ + 166
134
7487 + 101
7556
+
+ 748
7685 103
7580 163
BigGravity
6905 + 138
7107 + 153
7015 + 89
7009 166
7250 57
6282 + 476
5638 + 600
71.77 + 96
7083 + 184
7114 98
SmallFriction
6755 2073
7693 103
7671
7823 142
590
5980
7864 131
7743+ 161
SmallGravity
7057 1875
7788 + 135
7732 + 86
7764 + 185
7886 # 127
6947 +: 619
6354 + 818
7896 = 91
7757 = 117
7767 = 75
HugeFriction
7505 + 252
7640 199
72
7613
7860 +
6788 630
6160
7846 107
7695- 115
7668
HugeGravity
380 +: 507
1329 + 593
1711 74
1583 a: 442
921 = 302
4564 2033
3009 + 747
874 = 358
1237 = 662
1156 = 812
TriFriitioon
2747 1241
4015 1167
4918 723
1252
560
3001 1261
3676 1382
4545 731
4119 555
Gravity
-520 = 426
55 + 1844
11 +: 226
154 293
.56 + 256
6 236
236 a: 336
-149 + 800
-248 +: 362
227 + 357
DefectiveSensoe
4308
5323 + 173
5098
4498 191
3992.
5355 262
5016 109
DefectiveSense 10%
2770 171
3974 + 380
3978 + 117
3934 297
3486 # 161
3413 213
3061 +: 184
3960 106
3711 = 194
3779 196
DefectiveSensoe 15%
1531=90 90
2504 +
2423 154
2496
142
2:009 137
2374 120
DefectiveSensee 205
1026 77
1795 + 174
1629 + 111
1774 147
1205 63
1777 +: 466
1737 + 203
1834 #: 326
1987 =: 127
1787 + 180
DefectiveSensoe 25%
736 95
1486
1351- 220
1372 85
851
1422
1380
1342 121
1368 95
1390
DefectiveSensee 30%
472 50
1103 87
968 + 93
1133 + 158
635 + 99
967 297
866 157
1069 144
925 109
962 149
DefectiveSersne 35%
424
640
63
441
109
583
630 107
Average
3338
3991
40.11
4012
3833
3759
3400
4020
3947
3945
K-20
BigFriction
7454 1: 166
7592 147
7510
7569
4328 3086
5301 4:2
94
7589
BigGravity
6905 + 133
7121 # 162
7028 +: 50
7090 # 171
7196: # 139
4172 2949
4867 +: 370
7162 = 86
7135 147
7132 121
SmallFFiction
6755
7767 111
7712
7703 228
4360 3118
5021
7862 131
7726
SmallGravity
7057 +: 1875
7807 + 123
7698 +. 86
7773 203
7863 + 109
4512 3258
5343 4: 404
7906 : 78
7774 113
7806 = 95
HugeFriction
7505 252
7674 171
7648
7699
1
4331 3089
5061 334
7851 119
7714
7737
HugeGravity
380 4 507
1655 + 712
2111 1405
1587 343
2005 + 743
3596 2437
4231 + 121
1357 +: 197
2122 + 1006
1514 845
Tay Friction
2747 1241
4437 827
1145
2203 1432
3117 816
4707 490
3979 6.6
Gravity
-520 2 426
259 + 324
188 + 241
289 4: 316
357 + 525
291 620
348 = 152
-121 +: 812
-110 257
177 + 370
DefectiveSensoee
4308 39
5816
216
175
3647
6423 257
5780 241
DefectiveSeusor 10%
2770 171
3979 + 248
38.50 à. 199
4083 313
3500 + 208
3475 +: 242
2827 4: 261
4173 4: 325
4023 4: 176
4038 :: 391
Defectiveeensorr 15%g
2789 368
2526
2790
2495
2125 218
2596 306
3011
DefectiveSensor 20%
1026 77
1890 + 107
1794 4: 88
1750 2: 278
1353 86
1875 26
1714 2: 189
1746 4: 123
1789 151
1901 #: 185
DefectiveSensorr 25%
736
1582
150
992
1320 217
1480 128
1286
DefectiveSeusor 30%
472 50
1102 + 133
1069 à: 128
968 t
657 + 75
931 2 136
994 2: 70
909 2: 178
1035 4 99
992 + 51
DefectiveSensorr 35%
1
996
104
77
485
719
113
Average
3338
4164
4174
4145
3949
2878
3109
4150
4126
4108
Table 7: Mean and standard deviation of cumulative reward achieved on Ant test sets per model. Re- sults are averaged over 10 training seeds (i.e., 10 models are trained with the same hyper-parameters and evaluated on the 12 test sets). K is the number of policies tested at adaptation time, using episode per policy since this environment is deterministic. For this environment, we split the results per / value as it has been used for beta ablation study (see Figure 3)
23

---

Training
algorithm LoP
6000
3000
beta
0.1
10
1000
10.0
Figure 10: Evolution of the cumulative reward during training on the generic Ant environment for LoP, DIAYN+R and Lc for different values of beta. On can see that DIAYN+R struggles to perform well on the train set for 3 = 1 and 3 = 10. Results are averaged over 10 seeds.
24

---

B.3 CARTPOLE
We use the openAl gym implementation of CartPole as training environment. The test environ- are provided by Packer al. (2018) where three different factors may vary: the mass of the cart, the length of the pole and the force applied to the cart. The length of each episode is 200. The policy and the critic are encoded by two different multi-layer perceptrons with ReLU activations. The base learning algorithm is A2C.
Environment
Characteristies
(Train) CartPole
mass 0.1, length = 0.5, force 10.0
HeavyPole CartPole
mass 1.0
LightPole CartPole
moss 0.001
LongPole CartPole
length 1.0
ShortPole CartPole
length 0.05
StrongPush CartPole
force 20.0
WeakPush CartPole
force 1.0
Table 8: CartPole train and test environments
Hyper-parameter
Value
learning rate:
0.001
acquisition steps per epoch:
8
parallel environments:
32
critic coefficient:
1.0
entropy coefficient:
0.001
discount factor:
0.99
gae coefficient:
1.0
gradient clipping:
2.0
neurons per layer:
8
layers:
2
LoP
8:
1.0
DIAYN
8:
1.0
neurons per layer discriminator:
8
layers discriminator:
2
learning rate discriminator:
0.001
Table 9: Hyper-parameters for A2C over CartPole
Single
LoP
DIAYN+R
DIAYN+R L
HeavyPole CartPole
200.0 + 0.0
2.0
2.0
200.0 0.0
LightPole CartPole
2.0
2.0
2.0
200.0 0.0
LongPole CartPole
81.6
56.1
13.3
123.8 + 86.2
ShortPole CartPole
63.0
25.3
50.7
64.8 31.2
StrongPush CartPole
200.0 +
200.0 0.0
200.0 1000000.00
199.9 + 0.2
WeakPush CartPole
138.9 + 43.8
164.4 18.3
194.3 = 10.0
148. + 64.8
Average
143.4
149.9
168.1
156.1
Table 10: Results over CartPole, using 10 policies, and 10 episodes per policy at adaptation time.
25

---

B.4 ACROBOT
We use the openAl gym implementation of Acrobot as a training environment. The test environ- ments are provided by Packer al. (2018) where two different factor may vary: the intertia factor and the lengh of the system. We have used A2C as a learning algorithm.
Environment
Characteristics
(Train) Acrobot
mass 0.1, lenglh = 1.0, inertia = 1.0
Heavy Acrobot
mass 1.5
Highlnertia Acrobot
inertia 1.5
Light Acrobot
mass 0.5
Long Acrobot
length -.5
LowInertia Acrobot
inerlia 0.5
Short Acrobot
length = 0.5
Table 11: Acrobot train and test environments
Hyper-parameter
Value
learning rate:
0.001
acquisition steps per epoch:
8
parallel environments:
32
critic coefficient:
1.0
entropy coefficient:
0.001
discount factor:
0.99
gae coefficient:
0.7
gradient clipping:
2.0
neurons per layer:
16
layers:
2
LoP
8:
1.0
DIAYN
B:
1.0
neurons per layer discriminator:
16
layers discriminator:
learning rate discriminator: 0.001
Table 12:
for A2C over Acrobot
Single
LoP
DIAYN+R
DIAYN+R L2
Heavy Acrobot
+188.4
-105.1 1.0
-108.0
-108.2
HighInertia Acrobot
-108.7
-99,8
-106.0
-106.8
Light Acrobot
-120.7 71.3
48.82
-115.2
-93.1
Long Acrobot
-1.4.3
-1.
41.7.3
.11..5
Lowlnertia Acrobot
2.3
-70.7 0.7
-.7
-71.3
Short Acrobot
-65.1 2.8
-60.7
-64.2 + 2.6
-64.7 + 5.6
Average
-99.7
-93.2
-97.0
-93.6
Table 13: Results over Acrobot, using 10 policies, and 10 episodes per policy at adaptation time.
26

---

B.5 PENDULUM
We use the openAl gym implementation of Pendulum as a training environment. The 3 test environ- ments are provided by Packer al. (2018) where different factor may vary: the mass and the length of the pendulum. We have considered 5 discrete actions between -1 and +1. We have used A2C learning algorithm.
Environment
Characteristics
(Train) Pendulum
1.0, length 1.0
Light Pendulum
mass 0.5
Long Pendulum
length 1.5
Short Pendulum
length 0.5
Table 14: Pendulum train and test environments
Hyper-parameter
Value
learning rate:
0.001
acquisition steps per epoch:
8
parallel environments:
32
critic coefficient:
1.0
entropy coefficient:
0.001
discount factor:
0.99
gae coefficient:
0.7
gradient clipping:
2.0
neurons per layer:
16
layers:
2
LoP
8:
1.0
DIAYN
8:
1.0
neurons per layer discriminator:
16
layers discriminator:
2
learning rate discriminator:
0.001
Table 15: Hyper-parameters for A2C over Pendulum
Single
LoP
DIAYN+R
DIAYN+R T2
Light Pendulum
-36.5 1 58.5
-11.4
-39.3 + 10.9
-32.1 15.4
Long Pendulum
-82.1 20.4
-64.5 13.0
-70.6 15.2
-71.9 # 17.3
Short Pendulum
-39.6 66.9
-10.7 2.1
-31.3 11.7
-28.2 # 13.3
Average
-52.7
-28.9
47.1
-44.0
Table 16: Results over Pendulum, using 10 policies, and 10 episodes per policy at adaptation time.
27

---

B.6 MINIGRID
We have use Gym Minigrid to perform experiments on mazes Chevelier-Boisvert et al. (2018). We have used the MultiRoom-N2-S4 for training considering one single maze. At test time, have tested on three different MultiRoom-N2-S4 environments composed of two rooms, but also on three MultiRoom-N4-S composed of four rooms. This allows us evaluate the generalization power of the different methods to larger mazes. We have used A2C as learning algorithm.
Hyper-parane parameer
Value
learning rate:
0.001
acquisition steps per epoch:
8
parallel environments:
32
critic coefficient:
1.0
entropy coefficient:
0.001
discount factor:
0.99
gae coefficient:
0.7
gradient clipping:
2.0
neurons per layer:
16
layers:
2
LoP
8:
1.0
DIAYN
B:
1.0
neurons per layer discriminator:
16
layers discriminator:
2
learning rate discriminator:
0.001
Table 17: Hyper-parameters for A2C over Minigrid
Single
LoP 0.865
DIAYN+R 0,348 = 0.348
DIAYN+R L2
Two Rooms Maze
0.387 # 0.447
0.656 0.328
Two Rooms Maze 2
0.433 0,499
0.627 0.363
0.692 0.346
Two Rooms Maze 3
0.194 + 0.387
0.617 0.309
0.348 + 0.353
0.656 + 0.328
Four Rooms Maze
0.0
0.294 0.36
0.0
0.24 0.359
Four Rooms Maze 2
0.0
0.004 0.007
0.0 = 0.0
0.137 + 0.274
Four Rooms Maze
0.0
0.281 0.345
0.166 0.287
0.28 0.345
Average
0.169
0.447
0.248
0.443
Table 18: Results over Minigrid, using 10 policies, and episode per policy at adaptation time.
28

---

B.7 PROCGEN (FRUITBOT)
We performed experiments on pixel-based environment with ProcGen. We used the FruitBot game for training considering 10 levels sampled uniformly each episode. At test time, selected different environments, each of them composed of 10 uniformly sampled levels, sampled uniformly, and that were not seen train time. We used the CNN architecture described in the ProcGen paper (IMPALA architecture (Espeholt et al. (2018)). For LoP. the two first blocks are fixed and only the parameters of the last block depend on the value of For DIAYN, the value is provided as one-hot vector stacked to the observation. As for Brax environment, we used PPO algorithm.
Hyper-parameter
Value
learning rate:
5e 4
n acquisition steps per rollout:
128
n batches epoch:
8
n epochs per rollout:
5
n parallel environments:
64
critic coefficient:
1.0
entropy coefficient:
0.01
discount factor:
0.99
gae coefficient:
0.0
gradient clipping:
20.0
LoP
3:
0.01,0.1, 1.0
DIAYN+R
0.001, 0.01, 0.1, 1.0
Table 19: Hyper- parameters for PPO over ProcGen
Single
LoP
DIAYN+R
Levels 100 to 110
11.9 2.6
20.1 # 4.4
5.1 5.2
Levels 200 to 210
7.1 1.7
10.3 0.2
Levels 300 to 310
14.3
18.7 2.7
1..1
Results over Procgen, using K=10 at adaptation time, (averaged over 16 episodes per shot), averaged over 5 runs.
Table 20:
29

---

B.8 MAZE 2D WITH WALLS
To visualize the policies learned by the different methods, we just implemented a simple discrete maze actions up,down, left, right) where the objective is go from the top-middle tile the bottom-middle tile by moving through a corridor (size is 21 x 11). Reward is -1 at each step until goal is reached and the maximum number of steps 100. The optimal policy in the training environment achieves -16. At test time, we generate walls in the corridor such that the agent has to avoid these walls to reach the goal. The observation space a square around the agent. Policies are learned by using PPO. Illustrations of the trajectories over the train and the 4 test environments are illustrated in Figures
14
Hyper-parammter
Value
learning rate:
0.001
acquisition steps per rollout:
16
batches epoch:
epochs per rollout:
3
n parallel environments:
32
critic coefficient:
1.0
entropy coefficient:
0.01
discount factor:
0.99
gae coefficient:
0.0
gradient clipping:
20.0
LoP
3:
0.01, 0.1,
DIAYN+R
8:
0.01, 0.1, 1.0
Table 21: Hyper-parameters for PPO used for Maze 2d
Single Policy
LoP
DIAYN
8
0.01
0.1
1.
0.01
0.1
1.
Test Eav #1
0.0
1
-17.2
.17.6 2.6
-16 0
16
.16
Test Fav
-100 0.0
23.6 3.6
-39.4 25.8
-30.2 11.6
-56.2
42.6
+
Test Env
0.0
-37.8 18.9
-41.6 10.6
-33.6
-54.2
35.
-60.4
Test Env #4
1.0
-42.2 11.
-60.8 27.9
28.
28.8
-5344
-72.8
33.8
Env
Average
-332
-29.9
40.08
-33.44
-3.72
-2.2
-48.88
Table 22: Results over Maze2d, using K=10, averaged over runs.
13
Figure 11: Trajectories learned by Single Policy. First column is the training environment. Other columns are test environments. The lighter red the tiles are, the longer the agent stays on particular location.
30

---

Figure 12: Trajectories learned by LoP (3 1.0). Rows are environments, columns the K 10 policies test during online adaptation. For each test environment, at least one policy is able to reach the goal
31

---

Figure 13: Trajectories learned by DIAYN+R (8 0.01). Rows are environments, columns are the K 10 policies test during online adaptation. For many test environment, at least one policy is able to reach the goal
32

---

Figure 14: Trajectories learned by DIAYN+R (3 1). Rows are environments, columns are the
K = 10 policies test during online adaptation. With a too high value of 3, the training policy suboptimal, and does not achieve good performance at train and test times.
33


Reviews:
[{"review_id": "dZU_bf3WnD3", "reviewer": "Reviewer_6h1N", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "While I do find the observations interesting, I am not convinced that the method is working for the reasons that the authors have formulated. The training of uniform samples of convex combinations of anchor policies, while regularizing the anchors to be as different as possible, is effectively searching for regions in the policy space that are flat -- the policies parameterized in this way are trained to perform well in expectation. The robust performance should come from the fact that with this region of parameters, the behavior of the system can tolerate small variations better in the first place, regardless of whether it comes from the environment dynamics or the policy parameter changes. One limitation is of course this search is unlikely going to be successful for hard control problems, where not arbitrary perturbations on the policy parameters would work, and in those cases the training itself may in fact perform badly. These problems can be hard to see in the Mujoco environments. \n\nThere are a lot of issues of using wrong mathematical terms in the technical part. First of all, the approach is not learning a \"subspace\" of the parameter space -- in the current formulation it is clearly not closed under linear operations. Also, \"simplex\" is reserved for the convex hull of n+1 points in an n-dimensional space. So taking a few samples in the policy parameter space, which by itself has a large number of dimensions, the subset described in the paper can not be called \"simplex\". In section 3.3, it should not be called a \"line\" which needs to be unbounded in either directions, since it's just a segment. \n\nOverall I think the authors made interesting observations, but the mathematical analysis is weak and does not fully explain the observations. ", "summary_of_the_paper": "The work proposed a method for training robust policies in RL by optimizing convex combinations of different policies, and showed that it achieves good performance against parameter variations in various Mujoco environments. ", "summary_of_the_review": "Overall I think the authors made interesting observations, but the mathematical analysis is weak and does not fully explain the observations. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T1zD0geylCs", "reviewer": "Reviewer_qrCv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I enjoyed reading this paper, it studies an important problem and proposes new methods to tackle it. Thats said, I have some questions about the proposed new ideas in this paper and the connection to related work. I also have a few questions about the experiments. \n\nRegarding the line of policies. Let's assume for a moment that the anchor policies are deterministic, and consider the state occupancies of these policies. Using standard arguments (see [1] for example), we will get that any policy along the line (or the convex Hull) is a stochastic (or mixed) policy, and in addition, that for any reward signal, one of the deterministic policies will always be at least as good than any of the stochastic policies. That is, the anchor policies will be at least as good as any policy along the line. Now, there are a few differences between this construction and the approach taken in the paper, in particular, since the line is in the parameter space and not in the policy/state occupancy space. Nevertheless, I would like to see some evidence that the non-anchor policies are actually used over the anchor policies and to understand why that is. I am also very interested to see if the line in the parameter space extrapolates the convex Hull in the state occupancy space. This should be correlated with having policies along the line that outperform the anchor policies for some reward signals.  \n\nRegarding the experiments, the results are very impressive. One thing that I am missing is the data about performance during training time, and in particular, what is the average reward and diversity score (e.g. the cosine similarity or mutual information) achieved. This is important, since a potential scenario in Cheetah might be that a set that is more diverse but less good in terms of reward is leading to better fast adaptation. If this is the case (if one has to sacrifice reward to quickly adapt better) then I would like to learn that. \n\nOne baseline that I am missing, and follows the same line of thinking is a constrained MDP approach as was taken in [2,3]. The idea here is to set a constraint on how much a method will satisfy the extrinsic reward and then to maximize diversity under this constraint. Also note that there is a strong connection between the cosine similarity and the average reward in [3], but in [3] it is an inner product (not dividing by the norm) and being used in the successor features space instead of the parameter space. \n\n[1] Zahavy, T., Barreto, A., Mankowitz, D.J., Hou, S., O'Donoghue, B., Kemaev, I. and Singh, S., 2020, September. Discovering a set of policies for the worst case reward. In International Conference on Learning Representations. \n\n[2] Kumar, S., Kumar, A., Levine, S. and Finn, C., 2020. One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL. Advances in Neural Information Processing Systems, 33.\n\n[3] Zahavy, T., O'Donoghue, B., Barreto, A., Flennerhag, S., Mnih, V. and Singh, S., 2021, June. Discovering diverse nearly optimal policies with successor features. In ICML 2021 Workshop on Unsupervised Reinforcement Learning. ", "summary_of_the_paper": "The paper studies how to discover a subset of policies that will become useful for fast adaptation of future tasks. The subspace is defined as the convex hull of some anchor policies which are discovered, and are encouraged to perform well with respect to the extrinsic reward while being as different as possible from each other in the parameter space. The quality of the set is then evaluated by its ability to quickly adapt to new tasks. ", "summary_of_the_review": "Nice paper, some questions about the proposed idea needs answering and some of the experimental results need further clarification. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A96sy71zt8w", "reviewer": "Reviewer_UNwx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe problem is interesting, and the proposed method is simple and apparently easy to tune. However, I am concerned about whether or not this method will scale to more challenging environments than the Gym benchmark tasks because of various qualities of the method I describe below. While this is a promising direction of research, I think that this paper requires some iteration before acceptance.\n\nStrengths:\n\n- The proposed method is very simple, and seems to be relatively simple to optimize hyperparameters\n- The empirical results seem to convincingly indicate that this approach outperforms prior work\n- I appreciated Figure 1a: it neatly conveys the main approach of the paper\n\nWeaknesses:\n\n- While the authors seem to try to argue otherwise, it appears that enabling generalization to new test environments requires diversity in behavior / state distribution ($\\pi(a|s)$ or $d^\\pi(s)$) and not diversity in parameter space (since if parameters differ, but result in the same function, then there is no benefit conferred). However, unlike prior work (e.g. Kumar et al 2020b), this method does not optimize for functional diversity, and only parameter diversity, and in fact there is nothing in the loss that prevents the objective learning the same policy (e.g. in a \"flat\" part of the parameter space). No analysis is done as to how functionally diverse the learned policies actually are in the test environments as well (it would be interesting to see that)\n- It seems concerning that the optimal instantiation of this method is with $n=2$ anchor points, effectively only leading to a \"line\" of policies, and not the more general subspace of policies alluded to in the introduction / title. While there is a little discussion with n=3 in the experiments section, I think the paper would be stronger if it analyzed more deeply why this approach does not do well with more anchors. As tasks get more complicated, it seems that the \"line\" of policies with only one factor of variation will be unlikely to capture the variation needed to perform well in a shifted test environment.\n- An important missing comparison is just learning an ensemble of policies (either w/ or w/out standard diversity based metrics), and using the best ensemble member in the test environment. While this (hopefully) should not outperform the proposed method, this comparison would be useful to understand how much the continuous space of policies helps the algorithm, over simply just having a diverse set of policies.\n- In the experiments section, it says that the hyperparameters for the comparison methods are tuned based on performance in the training environment. It was unclear to me if training environment performance was used for tuning $\\beta$ for the DIAYN comparisons, since later it also says that the results provided are for the best value of $\\beta$ (which seems to imply test env performance was used). I would appreciate this if this were clarified.\n- Is there a way to tune the method depending on how different we expect the test-time environment to be? Abstractly speaking, if we know that $D(\\mathcal{M}, \\hat{\\mathcal{M}}) < \\epsilon$, how should we choose the hyperparameters for this algorithm? It seems reasonable to expect our method to be more conservative when the test environment is expected to be far from the current one, and less conservative when not. My original reaction was that $\\beta$ in the equation after Eq5 may do this, but it seems in your experiments that the constraint $C(\\cdot, \\dots, \\cdot)$ goes to zero regardless, so the behaviour would not change for other values of $\\beta$? Please let me know if I am misunderstanding. \n- The particular structure of the expected reward objective in Equation 4 looks exactly like running policy gradients on a particular stochastic latent-variable policy (where $\\pi(a|s; z, [\\bar{\\theta_k}]\\_{k=1}^n)$ = $f_{\\sum_{k}z_k\\bar{\\theta}_k}(a|s))$). With that perspective in mind, I'd be curious to hear the authors intepretation of how the regularization in Eq 5 compares to other works that consider regularized latent variable policies, e.g. (Kumar et al 2020b)?", "summary_of_the_paper": "This paper considers the problem of learning policies in a training environment that adapts fast to a different test environment. The proposed method learn a subspace of policy parameters that are optimal for the training environment, with the hope that one of these policy parameters will generalize better to the test environment than a single point estimate.", "summary_of_the_review": "(Copied from above) The problem is interesting, and the proposed method is simple and apparently easy to tune; however, I am concerned about whether or not this method will scale to more challenging environments than the Gym benchmark tasks because of various qualities of the method I describe below. While this is a promising direction of research, I think that this paper requires some iteration before acceptance.\n\n-- Post Rebuttal -- \n\nGiven the new experiments and analyses, I am increasing my score to a 6. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "4srpb9ZX5fI", "reviewer": "Reviewer_D4JT", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is generally well-written and easy to follow. \n\nThe proposed method is motivated by a robustness argument which is intuitive but could benefit from further exploration. While it is easy to believe that parameters that are robust to perturbations will perform well when the model is perturbed, this is not obviously true (in fact, one can easily construct examples where this does not hold). It is also unclear why this property would be preserved if the robustness is limited to a 1-dimensional line in parameter space. Additional explanations would be especially welcome as the baselines used in this paper are based on diversity and thus follow a different intuition. Furthermore, the authors should situate this work within the broader literature of robust reinforcement learning which has existed for decades. \n\nThe evaluation on multiple variations of control domains is promising and shows that the method can lead to improved adaptation with a relatively simple training procedure. The analysis on hyper-parameter sensitivity is welcome and it is good to see that the method does not require a lot of fine-tuning. Overall, the analysis focuses on relatively easy problems for adaptation and it would be interesting to see how this scales to harder tasks where a non-adapted policy fails completely and to analyze at which point the method stops being able to adapt. It would also be interesting to see a characterization of the learned sub-space, i.e. what kind of policies are being learned by the agent and how do they qualitatively compare to the kind of policies learned by diversity objectives. Finally, while increasing K leads to apparent improvements, it would also be interesting to see how the method compares to regular reinforcement learning when K=0, i.e. if we see a benefit from finding robust parameters, even if we do not optimize within the learned subspace. \n\nIn the paragraph on K-shot adaptation (Section 3.1), the authors claim that only one value of z needs to be executed in a deterministic environment. This is only true if the learned policy is also deterministic which does not seem to be the case in the proposed set-up. \n\nIn Section 3.2, the authors claim that the proposed method does not lead to learning sub-optimal policies at train-time. This statement is too strong as there is still a trade-off. In particular, the objective encourages the agent to find wide local optima and may forgo a better, but more narrow local optimum that it would otherwise choose.\n\nIn Section 2 on notation, r(s,a) should map to R, not R+.\n", "summary_of_the_paper": "The authors propose a method for rapid adaptation from a single training task to an unseen test-task in reinforcement learning. The method optimizes to find a convex subspace, specifically a line, of parameters that minimize the objective in expectation over a uniform distribution. For adaptation, the authors propose to find a convex combination that performs well on the test task.", "summary_of_the_review": "The paper is generally well-written and easy to follow. The problem formulation proposed by the authors is both relevant and important and the proposed method is simple to implement while showing promising results; however, the paper could benefit from additional analysis and more comprehensive literature review covering robust reinforcement learning.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper tackles the problem of generalizing to a new environment by learning a small set up anchor policies (even just 2 for the final approach) which span a sub-space that can be searched efficiently in a new environment.
The discussion and additional experiments managed to convince most reviewers that the method indeed works as the authors had hypothesized (especially regarding functional diversity). At the moment the analysis is mainly based on empirical observations, it would be good to also have a thorough theoretical analysis of the method.
Decision: Accept (Poster)