Title: Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policies
Venue: NeurIPS 2023
Accepted: yes
Keywords: Deep Reinforment Learning

Contextual Markov Decision Process

Neural Network Architecture
Abstract: While reinforcement learning has achieved remarkable successes in several domains, its real-world application is limited due to many methods failing to generalise to unfamiliar conditions. In this work, we consider the problem of generalising to new transition dynamics, corresponding to cases in which the environment's response to the agent's actions differs. For example, the gravitational force exerted on a robot depends on its mass and changes the robot's mobility. Consequently, in such cases, it is necessary to condition an agent's actions on extrinsic state information and pertinent contextual information reflecting how the environment responds. While the need for context-sensitive policies has been established, the manner in which context is incorporated architecturally has received less attention. Thus, in this work, we present an investigation into how context information should be incorporated into behaviour learning to improve generalisation.  To this end, we introduce a neural network architecture, the Decision Adapter, which generates the weights of an adapter module and conditions the behaviour of an agent on the context information. We show that the Decision Adapter is a useful generalisation of a previously proposed architecture and empirically demonstrate that it results in superior generalisation performance compared to previous approaches in several environments. Beyond this, the Decision Adapter is more robust to irrelevant distractor variables than several alternative methods.

Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policies

Abstract

While reinforcement learning has achieved remarkable successes in several domains, its real-world application is limited due to many methods failing to generalise to unfamiliar conditions. In this work, we consider the problem of generalising to new transition dynamics, corresponding to cases in which the environment's response to the agent's actions differs. For example, the gravitational force exerted on a robot depends on its mass and changes the robot's mobility. Consequently, in such cases, it is necessary to condition an agent's actions on extrinsic state information and pertinent contextual information reflecting how the environment responds. While the need for context-sensitive policies has been established, the manner in which context is incorporated architecturally has received less attention. Thus, in this work, we present an investigation into how context information should be incorporated into behaviour learning to improve generalisation. To this end, we introduce a neural network architecture, the Decision Adapter , which generates the weights of an adapter module and conditions the behaviour of an agent on the context information. We show that the Decision Adapter is a useful generalisation of a previously proposed architecture and empirically demonstrate that it results in superior generalisation performance compared to previous approaches in several environments. Beyond this, the Decision Adapter is more robust to irrelevant distractor variables than several alternative methods.

1 Introduction

Reinforcement learning (RL) is a powerful tool, and has displayed recent success in several settings [14]. However, despite its potential, RL faces a significant challenge: generalisation. Current agents and algorithms struggle to perform well beyond the narrow settings in which they were trained [5-7], which is a major limitation that hinders the practical application of RL. For RL algorithms to be effective in real-world scenarios, they must be capable of adapting to changes in the environment and performing well in settings that are different, but related, to those they trained on [7].

To illustrate this point, consider a legged robot trained to walk on a tiled floor. While the agent would perform well in these conditions, it will likely struggle in other reasonable settings, such as walking on asphalt or when carrying cargo. When walking on another surface, the friction characteristics would differ from what the robot trained on, which may cause it to slip. When carrying additional weight, the robot is heavier, with a potentially different center of mass. This may mean that the robot must exert more force to lift its legs or be more careful to prevent falling over. This limitation is far from ideal; we want our agents to excel in tasks with minor differences in dynamics (the effect of the agent's actions) without requiring additional training.

Several methods have been proposed to address this problem, including training a single agent in various settings in the hope that it learns a generalisable and robust behaviour [3, 8-11]. However, this approach is not without its drawbacks, as it fails when variations are too large for a single behaviour to perform well in all settings [12, 13]. For example, using the same behaviour when walking unencumbered and under load may lead to suboptimal performance in one or both cases.

This observation has led to other methods that instead use contextual information to choose actions, allowing the agent to adapt its behaviour to the setting it is in. For instance, if the agent knows when it is walking unencumbered vs. carrying something heavy, it can perform slightly differently in each case [14]. The context differs conceptually from the state information, which generally includes data that the robot observes using its sensors, such as its joint angles, acceleration, and camera views. The crucial distinction between context and state is that context changes at a longer timescale compared to state; for instance, the floor friction and additional mass remain the same for a long time, whereas the robot's joint angles change after every action [15]. Currently, the predominant approach is to use context as part of the state information, ignoring the conceptual difference between these two aspects [16-18]. However, this may lead to the agent confounding the context and state, and thus exhibit worse generalisation [15]. This problem can be exacerbated in real-world settings, where identifying which variables affect the dynamics may be challenging [18]. This may lead to several irrelevant variables being added to the context, further expanding the state space and making learning more difficult.

To address this problem, we introduce an approach to incorporating context into RL that leads to improved generalisation performance. Our method separates the context and state, allowing the agent to decide how to process the state information based on the context, thus adapting its behaviour to the setting it is in. Our experimental results demonstrate the effectiveness of our approach in improving generalisation compared to other methods. We show that our approach outperforms (1) not incorporating context information at all; (2) simply concatenating context and state; and (3) competitive baselines. We also demonstrate that our approach is more robust to irrelevant distractor variables than the concatenation-based approach across multiple domains. We further provide a theoretical characterisation of problems where incorporating context is necessary and empirically demonstrate that a context-unaware method performs poorly. 1

2 Background

Reinforcement learning problems are frequently modelled using a Markov Decision Process (MDP) [19, 20]. An MDP is defined by a tuple ⟨S , A , T , R , γ ⟩ , where S is the set of states, A is the set of actions. T : S × A × S → [0 , 1] is the transition function , where T ( s ′ | s, a ) specifies the probability of ending up in a certain state s ′ after starting in another state s and performing a specific action a . R : S × A × S → R is the reward function , where R ( s t , a t , s t +1 ) = R t +1 specifies the reward obtained from executing action a t in a state s t and γ ∈ [0 , 1] is the environment discount factor, specifying how short-term and long-term rewards should be weighted. The goal in reinforcement learning is to find a policy π : S → A that maximises the return G t = ∑ ∞ k =0 γ k R t + k +1 [21].

We consider the Contextual Markov Decision Process (CMDP) [15] formalism, which is defined by a tuple ⟨C , S , A , M ′ , γ ⟩ , where C is the context space , S and A are the state and action spaces respectively, and M ′ is a function that maps a context c ∈ C to an MDP M = ⟨S , A , T c , R c , γ ⟩ . A CMDP thus defines a family of MDPs, that all share an action and state space, but the transition ( T c ) and reward ( R c ) functions differ depending on the context. Our goal is to train on a particular set of contexts, such that we can generalise well to another set of evaluation contexts. We focus solely on generalising over changing dynamics and therefore fix the reward function, i.e., R c = R , ∀ c ∈ C .

3 Related Work

Generalisation in RL is a widely studied problem, with one broad class of approaches focusing on robustness . Here, a single policy is trained to be robust to changes that may occur during testing, i.e., to perform well without failing catastrophically [3, 6, 8, 22-29]. Many of these approaches are successful in this goal and can generalise well when faced with small perturbations. However, using one policy for multiple environments (or contexts) may lead to this policy being conservative, as it cannot exploit the specifics of the current environment [12]. Furthermore, these policies are limited

to performing the same action when faced with the same state, regardless of the setting. This may be suboptimal, especially when the gap between the different settings widens [13, 30].

This observation motivates the next class of methods, context-adaptive approaches. These techniques often use a version of the CMDP [15] formalism, and use the context to inform the agent's choice of action. This allows agents to exhibit different behaviours in different settings, which may be necessary to achieve optimality or generalisation when faced with large variations [13]. There are several ways to obtain this context, such as assuming the ground truth context [13], using supervised learning to approximate it during evaluation [14, 31], or learning some unsupervised representation of the problem and inferring a latent context from a sequence of environment observations [18, 32-34].

Despite the progress in inferring context, how best to incorporate context has received less attention. The most prevalent approach is to simply concatenate the context to the state, and use this directly as input to the policy [16-18, 35-38], which is trained using methods such as Soft Actor-Critic [39] or Proximal Policy Optimisation [40]. This ignores the conceptual difference between the state and context, and may lead to worse generalisation [15]. This approach may also be limited when there are many more context features compared to state features [41]. While concatenation is the most common, some other approaches have also been proposed. For instance, Biedenkapp et al. [41] learn separate representations for context and state, and then concatenate the representations near the end of the network. They also use another baseline that concatenates a static embedding of the context features to the state. Both of these techniques generalise better than the concatenation approach. Another method, FLAP [42], aims to improve generalisation by learning a shared representation across tasks, which is then processed by a task-specific adapter in the form of a linear layer. Given state s , the policy network outputs features ϕ ( s ) ∈ R d , which are shared across tasks. The final action is chosen according to W i ϕ ( s ) + b i , with a unique weight matrix W i and bias vector b i which are learned separately for each task. Concurrently, a supervised learning model is trained to map between transition tuples ( s t , a t , r t +1 , s t +1 ) and these learned weights. At test time, this supervised model generates the weights, while the policy network remains fixed. FLAP generalises better to out-of-distribution tasks compared to other meta-learning approaches [36, 43, 44]. However, this approach requires a separate head for each task, which may scale poorly if we have many tasks or see each task only a few times.

One promising approach that has been used in other fields is Feature-wise Linear Modulation (FiLM) [45, 46]. In this method, features in one modality (e.g., natural language text) linearly modulate the neural-network features obtained from another modality (e.g., visual images). An example of this would be processing a visual scene, modulated by different natural language questions which would lead to a different final answer. In RL, Benjamins et al. [13] introduce cGate , 2 which follows a similar procedure to FiLM; in particular, the neural network policy receives the state as input and outputs an action. Before the final layer, however, the context is first transformed by a separate network, and then used to modulate the state-based features. This approach showed promise, but restricted the context features' effect on the state-based features to be linear.

Another research area that has garnered more attention recently is learning general foundation models for control [48-50]. Inspired by recent work in fields such as natural language processing [51] and computer vision [52], these models aim to provide a general foundation that can be easily fine-tuned for particular tasks of interest. For instance, Gupta et al. [53] learns a general controller for several different robot morphologies. The robot's configuration is encoded and passed to a transformer [54], allowing one model to control a wide variety of morphologies. While these works directly train policies, Schubert et al. [55] instead train a dynamics model, and use a standard planning techniquemodel-predictive control [56-58]-to choose actions. They find that this approach achieves better zero-shot generalisation compared to directly learning a policy. On a different note, Sun et al. [59] pre-train a foundation model that predicts observations and actions. Then, during fine-tuning, they train a policy to perform a particular task using the pre-trained representations, which enables the use of either imitation learning or reinforcement learning in the downstream fine-tuning stage.

Finally, much work in meta-reinforcement learning [60] relates to the problem of generalisation. Meta-learning approaches generally aim to use multiple tasks during training to learn how to learn ; this knowledge can then be used during testing to rapidly adapt to a new task [43, 60-67]. In this problem setting, Beck et al. [68] use a recurrent model to encode the current task, and generate the weights of the agent policy based on the task encoding. Sarafian et al. [69] take a similar approach, but

instead generate the weights using the environment state, and processing the encoded context using the generated network. Another approach, MAML [44] aims to learn the weights of a neural network such that, for any new task, performing only a few gradient updates would lead to a high-performing task-specific network. While meta-learning can lead to highly adaptable agents, many methods require multiple episodes of experience in the target domain [36, 44], and may therefore be less well-suited to the zero-shot setting we consider [7].

4 Theoretical Intuitions

Now, while only a context-conditioned policy is guaranteed to be optimal on a general CMDP [47], in many cases simply training an unaware policy on a large number of diverse environments can lead to impressive empirical generalisation [5]. In this section, we characterise and unify these two observations. In particular, for a specific problem setting-where the context defines the target location an agent must travel to-we find that:

(i) For some context sets, an unaware policy will perform arbitrarily poorly on average, due to it being forced to take the same action in the same state, regardless of context.

(ii) However, for other context sets-where the different contexts are similar enough-an unaware policy can perform well on average, as it can simultaneously solve each task.

We defer the formal statement of this theorem and its proof to Appendix A, but here we briefly provide some intuition. Overall, our results rely on the fact that a context-unaware agent must perform the same action in the same state, regardless of context. Therefore, in a setting where we have different goals (indicated by the context), and making progress on one goal leads to making negative progress on another, a context-unaware agent cannot simultaneously perform well on both of these contexts. By contrast, if the goals are aligned in the sense that making progress on one leads to progress on another, a context-unaware agent can perform well on average. A context-aware agent, however, can always choose the correct action in either scenario.

c

1

c

2

c

3

c

4

(a) Non-overlapping

c

1

c

3

c

2

c

4

(b) Overlapping

Fig. 1 visually illustrates these two cases and summarises the implications. In Fig. 1a, if there is no overlap between goals, an unaware policy is forced to visit each one in sequence. This is because it must perform the same action in the same state, regardless of the context, meaning that it cannot always travel to the correct goal. This policy will therefore have a low value function compared to the optimal context-aware policy. By contrast, in Fig. 1b, a single unaware policy can directly go to the intersection of all the goals, leading to only slightly less return than the optimal context-aware

policy. While the formal result in Appendix A merely shows the existence of some problems with these properties, we believe the concept generalises to more complex settings. In particular, if the problem exhibits some structure where the same policy can simultaneously make progress on each context, an unaware model can perform well. However, if making progress on one goal leads to the agent moving away from another, a context-unaware policy can at best perform arbitrarily worse than a context-aware one, and incorporating context is crucial. We empirically substantiate these findings in Section 7 and formally consider this example in Appendix A. Finally, see Appendix B for more details about how these theoretical results connect back to our empirical observations.

5 The Decision Adapter

An adapter is a (usually small) neural network with the same input and output dimensions. This network can be inserted between the layers of an existing primary network to change its behaviour in some way. These modules have seen much use in natural language processing (NLP) [70-74] and computer vision, usually for parameter-efficient adaptation [75-81]. In these settings, one adapter is usually trained per task, and the base model remains constant across tasks. Our method is inspired by the use of adapters in these fields, with a few core differences.

First, since our aim is zero-shot generalisation, having a separate adapter module for each task (corresponding to context in our case) provides no clear means for generalisation. Relatedly, since tasks are generally discrete in NLP, but we allow for continuous contexts, this approach would be problematic. Furthermore, the number of learnable parameters increases with the number of training contexts, presenting a computational limitation to the scalability. To circumvent these problems, we use a hypernetwork [82] to generate the weights of the adapter module based on the context. This allows the agent to modify its behaviour for each context by having a shared feature-extractor network which is modulated by the context-aware adapter. Moreover, since the hypernetwork is shared, experience gained in one context could transfer to another [72]. Second, instead of having separate pre-training and adaptation phases as is common in NLP [76], we do not alter the standard RL training procedure at all. We change only the architecture and train normally-allowing for the easy integration of our approach into standard RL libraries. Finally, our method is agnostic to the exact RL algorithm and can be applied to most off-the-shelf algorithms without much additional effort.

̸

Algorithm 1 Decision Adapter-Changes to the stan- dard forward pass in blue.

Adapter

s

s

a

c

Hypernetwork

Generated Weights

Figure 2: An illustration of our network architecture, the Decision Adapter, alongside pseudocode for the forward pass. The bottom network is the hypernetwork, which generates the magenta weights for the top, primary, network. The light grey nodes in the top network are added by the adapter.

To formally describe our approach (illustrated in Fig. 2), suppose we have a state s ∈ S , context c ∈ C , and a standard n -layer fully-connected neural network architecture that predicts our action a . Here x i +1 = L i ( x i ) = σ i ( W i x i + b i ) , x 1 = s and x n +1 = a . We add an adapter module A i between layers L i -1 and L i . This adapter module is a multilayer neural network A i : R d i → R d i consisting of parameters θ i A ∈ R P i . 3 These parameters are generated by the corresponding context-conditioned

hypernetwork H i : C → R P i (line 5 in Algorithm 1) and reshaped to form multiple weight matrices. Then, before we process x i using layer L i , we pass x i through the adapter module A i (line 6) defined by the generated weights and biases to obtain x ′ i = A i ( x i | θ i A ) . Furthermore, akin to the application of adapters in other fields [75, 76], we add a skip connection, by setting x i = x i + x ′ i (line 7). This updated x i is then the input to layer L i (line 9). Additionally, we could have multiple adapter modules in the primary network, with at most one module between any two consecutive layers (lines 3 and 4).

Finally, we note that the Decision Adapter model can learn to implement the same function as cGate [13], but is theoretically more powerful. While cGate uses a linear elementwise product between the context and state features, our model uses a hypernetwork to generate the weights of a nonlinear adapter module. Our hypernetwork is general enough to be able to recover this elementwise product, but is not constrained to do so. See Appendix C for a more formal treatment of this point.

6 Experimental Setup

6.1 Metrics

Our evaluation strategy is as follows: We take the trained model, and compute the average of the total episode reward over n = 5 episodes on each evaluation context c to obtain a collection of contexts C and corresponding rewards R . We then calculate the Average Evaluation Reward (AER) : AER = 1 c max -c min ∫ c max c min R ( c ) dc, where c min and c max are the minimum and maximum context values in C respectively, and R ( c ) is the reward obtained on context c . This metric is high when the agent performs well across a wide range of contexts, and thus corresponds to generalisation performance. However, since we generally have C train ⊂ C evaluation , this metric also incorporates how well the agent performs on the training contexts. 4 Despite this, it is still a useful metric for generalisation, as our training context set typically contains only a handful of contexts, with most of the evaluation contexts being unseen.

6.2 Baselines

We use Soft-Actor-Critic [39] for all methods, to isolate and fairly compare the network architectures. We aim to have an equal number of learnable parameters for each method and adjust the number of hidden nodes to achieve this. We use the following baselines, with more details in Appendix D:

Unaware This model simply ignores the contextual information and just uses the state. This approach allows us to evaluate how well a method that does not incorporate context performs.

Concat This method forms a new, augmented state space S ′ = S × C , and then gives the model the concatenation [ s ; c ] of the state and context [16-18, 37]. This baseline allows us to compare our method to the current standard approach of using contextual information.

cGate cGate [13] learns separate state ( ϕ ( s ) ) and context ( g ( c ) ) encoders, and predicts the action a = f ( ϕ ( s ) ⊙ g ( c )) , where f is the learned policy and ⊙ is the elementwise product.

FLAP FLAP [42] learns a shared state representation across tasks, which is then processed by a task-specific linear layer that is generated by conditioning on the context.

Our Adapter configuration is more fully described in Appendix D.3, and Appendix E contains ablation experiments comparing the performance of different hyperparameter settings.

6.3 Environments

6.3.1 ODE

This environment is described by an ordinary differential equation (ODE), parametrised by n variables making up the context. The dynamics equation is x t +1 = x t +˙ x t dt , with ˙ x = c 0 a + c 1 a 2 + c 2 a 3 + . . . , truncated at some c n -1 . The episode terminates after 200 timesteps, with the reward function:

This incentivises the agent to give control a to keep the state close to x = 0 . The context in this environment is c = [ c 0 , . . . , c n -1 ] ∈ R n . The action space is two-dimensional, continuous and bounded, i.e., a 0 , a 1 ∈ [ -1 , 1] . The action is interpreted as a complex number, a = a 0 + a 1 i , to allow the dynamics equation to be solvable, even in normally unsolvable cases such as ˙ x = a 2 . Here, restricting a to R would result in ˙ x ≥ 0 and an unsolvable system for initial conditions x 0 > 0 . We keep only the real part of the updated state and clip it to always fall between -20 and 20 .

The ODE acts as a conceptually simple environment where we can arbitrarily scale both the number (i.e., n ) and magnitude (i.e., | c i | ) of the contexts and precisely measure the effects. Furthermore, since many dynamical systems can be modelled using differential equations [83-85], the ODE can be considered a distilled version of these. Finally, context is necessary to perform well in this environment, making it a good benchmark. See Appendix D for more details.

6.3.2 CartPole

CartPole [86] is a task where an agent must control the movement of a cart to balance a pole vertically placed upon it. The observation space is a 4 -dimensional real vector containing the cart's position x and velocity ˙ x , as well as the pole's angle θ and angular velocity ˙ θ . We use a continuous action space where a ∈ [ -1 , 1] corresponds to the force applied to the cart (where negative values push the cart to the left and positive values to the right). The reward function is +1 for each step that the pole is upright. An episode terminates when the pole is tilted too far off-center, the cart's position is outside the allowable bounds or the number of timesteps is greater than 500 . We follow prior work and consider the variables Gravity , Cart Mass , Pole Length , Pole Mass and Force Magnitude collectively as the context [13], even if only a subset of variables change. Here, we change only the pole length and evaluate how susceptible each model is to distractor variables. In particular, during training, we also add k additional dimensions to the context, each with a constant value of 1 . Then, during evaluation, we set these values to 0 . This may occur during a real-world example, where accurately identifying or inferring the pertinent context variables may be challenging [18, 87], and lead to several irrelevant variables. Overall, this means that the context is represented as a 5 + k -dimensional vector. This environment is useful as it (1) is a simple setting with models that can be trained quickly, and (2) still has desirable aspects, namely a way to perturb some underlying variables to change the dynamics of the environment. Furthermore, prior work has shown that even standard, unaware RL algorithms can generalise well in this environment [88] (in the absence of distractor context variables), which allows us to investigate the benefits of context and detriment of distractors.

6.3.3 Mujoco Ant

As a more complex and high-dimensional problem, we consider Ant from the Mujoco suite of environments [89, 90]. Here, the task is to control a 4-legged robot such that it walks. The observation space consists of the robot's joint angles and velocities, as well as contact forces and torques applied to each link, totalling 111 dimensions. The action space A = [ -1 , 1] 8 represents the torque applied to each of the 8 joints. Furthermore, each episode is terminated after 1000 timesteps, with a positive reward for not having fallen over at each step and moving forward. The reward function also penalises large control forces. Here, the mass is the context variable, which is 5 by default. We train on the set { 5 , 35 , 75 } and evaluate on 200 evenly spaced points between 0 . 5 and 100 .

7 Results

7.1 Generalisation Performance

In this section, we consider the ODE environment, with a training context set of {-5 , -1 , 1 , 5 } and 300 k training timesteps. To measure generalisation, we use an evaluation context range consisting of 201 equally spaced points between -10 and 10 (inclusive). In the left pane of Fig. 3, we plot the average performance across the entire evaluation range (as discussed in Section 6.1) at different points during training.

Firstly, we can see that the Unaware model fails to generalise well, since the optimal actions in the same state for two different contexts may be completely different. This result substantiates our theoretical analysis in Section 4, highlighting the importance of using context in this domain. Concat, cGate and FLAP perform reasonably well, but our Decision Adapter outperforms all methods and converges rapidly. In Appendix F, we plot separate subsets of the evaluation range: Train , Interpolation and Extrapolation , corresponding to the contexts in the training set, the contexts within the region [ -5 , 5] and the contexts outside the convex hull of the training contexts, containing c ∈ [ -10 , -5) ∪ (5 , 10] , respectively. There we find that most methods, with the exception of the Unaware model, perform well on the training contexts, and the Adapter outperforms all methods when extrapolating. These results also extend to the multidimensional context case, shown in Fig. 3 (right). We train on the context set given by { 1 , 0 , -1 } 2 ∪ { 5 , 0 , -5 } 2 , with the context (0 , 0) being omitted due to it being unsolvable. 5 The evaluation range consists of the cartesian product A 2 , where A contains 21 equally-spaced points between -10 and 10 . The Adapter outperforms all methods, with the Concat model coming second. FLAP and the Unaware models struggle in this case.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0

50

100

150

200

AER

1D Context

0.0

0.5

1.0

1.5

2.0

2.5

3.0

1e5

2D Context

Training Time (Steps)

Unaware

Concat

Adapter

cGate

FLAP

Optimal

7.2 Robustness to Distractors

For this and the next section, we consider the Concat and cGate models as baselines. The reasons for this are that (1) the Concat model performed comparably to or better than the other baselines in the previous sections, making it a representative example of the context-aware approaches; and (2) the Concat model is currently a very prevalent way of incorporating context [16-18, 35-38]; and (3) cGate is the closest approach to our work in the current literature.

The results for CartPole are shown in Fig. 4. When there are no distractor variables, the Adapter, Concat and cGate models perform comparably. In this case, each architecture achieves near the maximum possible reward for the domain. Further, we see little effect after adding just one confounding context variable. However, as we add additional distractor variables, the Concat and cGate models' performances drop significantly, whereas the Adapter's performance remains relatively stable. Strikingly, the Adapter architecture trained with 100 distractor context variables is still able to perform well on this domain and significantly outperforms the Concat model with significantly fewer (just 20 ) distractor variables. This demonstrates that, given only useful context, concatenating state and context is a reasonable approach in some environments. However, this is a strong assumption in many practical cases where we may be uncertain about which context variables are necessary. In such cases, the consequences of ignoring the conceptual differences between state and context are catastrophic and it is necessary to use the Adapter architecture. We find a similar result in the ODE, which we show in Appendix F.5.

As an extension to the previous experiment, here we consider using non-fixed distractor variables, i.e., sampling them from Gaussian distribution with different means. The intuition behind this experiment is similar to before, but allows for small variations within training and testing. In particular, we use σ = 0 . 2 and a mean of either 0 or 1 . In this experiment, a new distractor context is sampled at the start of every episode. During training this is sampled from N (1 , 0 . 2) and during evaluation we

0

2

4

6

8

100

200

300

400

500

AER

Concat

0

2

4

6

8

Adapter

0

2

4

6

8

1e5

cGate

Training Time (Steps)

Number of distractor dimensions

0

1

20

100

Optimal Reward

sample from N (0 , 0 . 2) . In CartPole, the results in Fig. 5 show that the Adapter is still more robust to changing distractor variables compared to either Concat or cGate . See Appendix F.6 for more details, results on the ODE domain, and additional experiments when the mean of the Gaussian does not change between training and testing.

0

1

2

3

100

200

300

400

500

AER

Concat

0

1

2

3

Adapter

0

1

2

3

1e5

cGate

Training Time (Steps)

Number of distractor dimensions

0

1

20

100

Optimal Reward

7.3 High-Dimensional Control

We next consider a more complex and challenging continuous-control environment, that of the Mujoco Ant. We run the same experiment as in CartPole, where we add distractor variables, with no effect on the dynamics, and different values between training and testing. These results are shown in Fig. 6. Overall, we observe a similar result to that of CartPole-the Concat and cGate models are significantly less robust to having irrelevant distractor variables. By contrast, the Adapter consistently performs well regardless of how many distractor dimensions are added.

8 Limitations

While our Adapter model performs well empirically, it does have some limitations, which we summarise here and expand on in Appendix G. First, we find that using an incorrect and noisy context (Appendix G.1) generally resulted in the Adapter model performing worse, particularly as we increased the level of noise. However, the Adapter model still performs comparably to the Concat model in this case. Second, we generally normalise the contexts before we pass them to the models, relative to the maximum context encountered during training. While our Adapter performs well when the context normalisation is incorrect by a factor of 2 or 3, its performance does suffer when the normalisation value is orders of magnitudes too small, leading to very large contexts being input into the model (see Appendix G.2 for further discussion). Third, when training on a context range that is too narrow (see Appendix G.3) or does not exhibit sufficient variation, our model is susceptible to overfitting, which

0.0

0.2

0.4

0.6

0.8

1.0

2000

0

2000

4000

AER

Concat

0.0

0.2

0.4

0.6

0.8

1.0

Adapter

0.0

0.2

0.4

0.6

0.8

1.0

1e6

cGate

Training Time (Steps)

Number of distractor dimensions

0

1

20

100

leads to worse generalisation. Even if our model does not overfit, an overly narrow context range would also lead to poor generalisation. Thus, ensuring that agents are trained in sufficiently diverse environments is still important. Fourth, while we assumed access to the ground-truth context, it may not always be available in practice [31]. Therefore, integrating the Decision Adapter into existing context inference methods [18, 33] is a promising avenue for future work. A final limitation of our Adapter is its longer wall-clock training time compared to other methods (given the same training timesteps and a similar number of parameters). This is likely caused by the hypernetwork that we use. However, we did not utilise strategies such as caching the generated weights for a particular context (which would be possible since context remains constant for an episode), or more effectively batching the forward passes through the Adapter which could ameliorate this issue in practice.

Beyond addressing these limitations, there are several logical directions for future work. First, we only consider state and context features to be numerical vectors; therefore, extending our approach to different modalities-for instance, image observations and natural language context-would be a natural extension. In principle, our Adapter model could be directly applied in this setting without any fundamental modifications. One option would be to explore adding the hypernetwork directly into the modality-specific model (e.g., a convolutional neural network in the case of images), and the alternative would be to integrate the adapter module in the final (fully-connected) layers of the RL policy. Second, while we focused on the problem of zero-shot generalisation, our approach may also be suitable for efficient few-shot learning. In particular, adapters in NLP are often added to a large pre-trained model and then fine-tuned while freezing the primary model [76]. Using the Decision Adapter in this paradigm is a promising direction for future work, especially in cases where the difference between training and testing is so large that we cannot expect effective zero-shot generalisation. Combining our work with fine-tuning-based [91-93] or meta-learning [36, 60, 68, 69, 94-96] approaches, which generally focus more on this case, could likewise be promising.

9 Conclusion

In this work, we consider the problem of generalising to new transition dynamics. We first illustrate that for some classes of problems, a context-unaware policy cannot perform well over the entire problem space-necessitating the use of context. We next turned our attention to how context should best be incorporated, introducing the Decision Adapter-a method that generates the weights of an adapter module based on the context. When context is necessary, our Decision Adapter outperforms all of our baselines, including the prevalent Concat model. Next, when there are some irrelevant context variables that change between training and testing, the Concat and cGate models fail to generalise, whereas our Decision Adapter performs well. This result holds in several domains, including the more complex Ant. Furthermore, our Adapter is theoretically more powerful and empirically more effective than cGate , showing that our hypernetwork-based perspective is useful when dealing with context. Ultimately, we find that the Decision Adapter is a natural, and robust , approach to reliably incorporate context into RL, in a factored way. This appears to be of particular benefit when there are several irrelevant context variables, for instance, in real-world settings [97].

References

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science , 362(6419):1140-1144, 2018.

Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 , 2019.

Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wojtek Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, et al. AlphaStar: Mastering the Real-Time Strategy Game StarCraft II. https://deepmind . com/blog/alphastar-mastering-real-time-strategygame-starcraft-ii/ , 2019.

AmyZhang, Nicolas Ballas, and Joelle Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. CoRR , abs/1806.07937, 2018. URL http://arxiv . org/abs/1806 . 07937 .

Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generalization in reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning , pages 1282-1289. PMLR, 2019. URL http://proceedings . mlr . press/v97/cobbe19a . html .

Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. J. Artif. Intell. Res. , 76:201-264, 2023. doi: 10 . 1613/jair . 1 . 14174. URL https://doi . org/10 . 1613/jair . 1 . 14174 .

Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In Proceedings of the 34th International Conference on Machine Learning , pages 2817-2826. PMLR, 06-11 Aug 2017. URL https://proceedings . mlr . press/v70/pinto17a . html .

Mohammed Amin Abdullah, Hang Ren, Haitham Bou-Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. CoRR , abs/1907.13196, 2019. URL http://arxiv . org/abs/1907 . 13196 .

Daniel J. Mankowitz, Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg, Yuanyuan Shi, Jackie Kay, Todd Hester, Timothy A. Mann, and Martin A. Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In 8th International Conference on Learning Representations . OpenReview.net, 2020. URL https://openreview . net/forum?id=HJgC60EtwB .

Gilbert Feng, Hongbo Zhang, Zhongyu Li, Xue Bin Peng, Bhuvan Basireddy, Linzhu Yue, Zhitao Song, Lizhi Yang, Yunhui Liu, Koushil Sreenath, and Sergey Levine. Genloco: Generalized locomotion controllers for quadrupedal robots. In Conference on Robot Learning , pages 1893-1903. PMLR, 2022.

Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment probing interaction policies. In 7th International Conference on Learning Representations . OpenReview.net, 2019. URL https: //openreview . net/forum?id=ryl8-3AcFX .

Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Contextualize me - the case for context in reinforcement learning. CoRR , abs/2202.04500, 2022. URL https://arxiv . org/abs/2202 . 04500v1 .

Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. In Robotics: Science and Systems XIII , 2017. doi: 10 . 15607/ RSS . 2017 . XIII . 048. URL http://www . roboticsproceedings . org/rss13/p48 . html .

Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. CoRR , abs/1502.02259, 2015. URL http://arxiv . org/abs/1502 . 02259 .

Philip J. Ball, Cong Lu, Jack Parker-Holder, and Stephen J. Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. In Proceedings of the 38th International Conference on Machine Learning , pages 619-629. PMLR, 2021. URL http://proceedings . mlr . press/v139/ball21a . html .

Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In Proceedings of the 38th International Conference on Machine Learning , volume 139, pages 9767-9779. PMLR, 2021. URL http://proceedings . mlr . press/v139/sodhani21a . html .

Shagun Sodhani, Franziska Meier, Joelle Pineau, and Amy Zhang. Block contextual MDPs for continual learning. In Learning for Dynamics and Control Conference, L4DC 2022, 23-24 June 2022, Stanford University, Stanford, CA, USA , pages 608-623. PMLR, 2022. URL https://proceedings . mlr . press/ v168/sodhani22a . html .

Richard Bellman. A markovian decision process. Indiana Univ. Math. J. , 6:679-684, 1957. ISSN 0022-2518.

Martin L Puterman and Eugene A Feinberg. Markov decision processes: discrete stochastic dynamic programming. SIAM review , 38(4):689, 1996.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. In International Conference on Intelligent Robots and Systems , pages 23-30. IEEE, 2017. doi: 10 . 1109/IROS . 2017 . 8202133. URL https://doi . org/10 . 1109/IROS . 2017 . 8202133 .

Fereshteh Sadeghi and Sergey Levine. CAD2RL: real single-image flight without a single real image. In Robotics: Science and Systems XIII , 2017. doi: 10 . 15607/RSS . 2017 . XIII . 034. URL http:// www . roboticsproceedings . org/rss13/p34 . html .

Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In International Conference on Robotics and Automation , pages 1-8. IEEE, 2018. doi: 10 . 1109/ICRA . 2018 . 8460528. URL https://doi . org/10 . 1109/ ICRA . 2018 . 8460528 .

Jan Matas, Stephen James, and Andrew J. Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Conference on Robot Learning , pages 734-743. PMLR, 2018. URL http: //proceedings . mlr . press/v87/matas18a . html .

Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojciech Zaremba. Learning dexterous in-hand manipulation. Int. J. Robotics Res. , 39(1), 2020. doi: 10 . 1177/0278364919887447. URL https://doi . org/10 . 1177/ 0278364919887447 .

Alejandro Escontrela, George Yu, Peng Xu, Atil Iscen, and Jie Tan. Zero-shot terrain generalization for visual locomotion policies. CoRR , abs/2011.05513, 2020. URL https://arxiv . org/abs/2011 . 05513 .

Zac Wellmer and James T. Kwok. Dropout's dream land: Generalization from learned simulators to reality. In Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part I , volume 12975 of Lecture Notes in Computer Science , pages 255-270. Springer, 2021. doi: 10 . 1007/978-3-030-86486-6 \ _16. URL https://doi . org/10 . 1007/978-3-030-86486-6_16 .

Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforcement learning for robotics: a survey. In Symposium Series on Computational Intelligence , pages 737-744. IEEE, 2020. doi: 10 . 1109/SSCI47803 . 2020 . 9308468. URL https://doi . org/10 . 1109/ SSCI47803 . 2020 . 9308468 .

Samuel Kessler, Jack Parker-Holder, Philip J. Ball, Stefan Zohren, and Stephen J. Roberts. Same state, different task: Continual reinforcement learning without interference. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 7143-7151. AAAI Press, 2022. URL https:// ojs . aaai . org/index . php/AAAI/article/view/20674 .

Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: rapid motor adaptation for legged robots. In Robotics: Science and Systems XVII , 2021. doi: 10 . 15607/RSS . 2021 . XVII . 011. URL https://doi . org/10 . 15607/RSS . 2021 . XVII . 011 .

Alvaro Sanchez-Gonzalez, Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin A. Riedmiller, Raia Hadsell, and Peter W. Battaglia. Graph networks as learnable physics engines for inference and control. In Proceedings of the 35th International Conference on Machine Learning , pages 4467-4476. PMLR, 2018. URL http://proceedings . mlr . press/v80/sanchez-gonzalez18a . html .

Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwoo Shin. Context-aware dynamics model for generalization in model-based reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning , pages 5757-5766. PMLR, 2020. URL http: //proceedings . mlr . press/v119/lee20g . html .

Younggyo Seo, Kimin Lee, Ignasi Clavera Gilaberte, Thanard Kurutach, Jinwoo Shin, and Pieter Abbeel. Trajectory-wise multiple choice learning for dynamics generalization in reinforcement learning. In Advances in Neural Information Processing Systems , 2020. URL https://proceedings . neurips . cc/ paper/2020/hash/9739efc4f01292e764c86caa59af353e-Abstract . html .

Seyed Kamyar Seyed Ghasemipour, Shixiang Gu, and Richard S. Zemel. Smile: Scalable meta inverse reinforcement learning through context-conditional policies. In Advances in Neural Information Processing Systems , pages 7879-7889, 2019. URL https://proceedings . neurips . cc/paper/2019/hash/ 2b8f621e9244cea5007bac8f5d50e476-Abstract . html .

Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy metareinforcement learning via probabilistic context variables. In Proceedings of the 36th International Conference on Machine Learning , pages 5331-5340. PMLR, 2019. URL http://proceedings . mlr . press/ v97/rakelly19a . html .

Hamid Eghbal-zadeh, Florian Henkel, and Gerhard Widmer. Context-adaptive reinforcement learning using unsupervised learning of context variables. In NeurIPS 2020 Workshop on Pre-registration in Machine Learning , pages 236-254. PMLR, 11 Dec 2021. URL https://proceedings . mlr . press/ v148/eghbal-zadeh21a . html .

Yao Mu, Yuzheng Zhuang, Fei Ni, Bin Wang, Jianyu Chen, Jianye Hao, and Ping Luo. Decomposed mutual information optimization for generalized context in meta-reinforcement learning. CoRR , abs/2210.04209, 2022. doi: 10 . 48550/arXiv . 2210 . 04209. URL https://doi . org/10 . 48550/ arXiv . 2210 . 04209 .

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning , pages 1856-1865. PMLR, 2018. URL http: //proceedings . mlr . press/v80/haarnoja18b . html .

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR , abs/1707.06347, 2017. URL http://arxiv . org/abs/1707 . 06347 .

André Biedenkapp, David Speck, Silvan Sievers, Frank Hutter, Marius Lindauer, and Jendrik Seipp. Learning domain-independent policies for open list selection. 2022.

Matt Peng, Banghua Zhu, and Jiantao Jiao. Linear representation meta-reinforcement learning for instant adaptation. CoRR , abs/2101.04750, 2021. URL https://arxiv . org/abs/2101 . 04750 .

Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL 2 : Fast reinforcement learning via slow reinforcement learning. CoRR , abs/1611.02779, 2016. URL http: //arxiv . org/abs/1611 . 02779 .

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning , pages 1126-1135. PMLR, 2017. URL http://proceedings . mlr . press/v70/finn17a . html .

Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3942-3951. AAAI Press, 2018. URL https://www . aaai . org/ocs/index . php/ AAAI/AAAI18/paper/view/16528 .

Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill , 2018. doi: 10 . 23915/distill . 00011. https://distill.pub/2018/feature-wise-transformations.

Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, Sebastian Döhler, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Contextualize me - the case for context in reinforcement learning. Transactions on Machine Learning Research , 2023. ISSN 2835-8856. URL https://openreview . net/forum?id=Y42xVBQusn .

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research , 2022. ISSN 2835-8856. URL https://openreview . net/forum?id=1ikK0kHjvj . Featured Certification, Outstanding Certification.

Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. CoRR , abs/2303.04129, 2023. doi: 10 . 48550/arXiv . 2303 . 04129. URL https://doi . org/10 . 48550/arXiv . 2303 . 04129 .

Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee, Maria Bauzá, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al. Robocat: A self-improving foundation agent for robotic manipulation. CoRR , abs/2306.11706, 2023. doi: 10 . 48550/arXiv . 2306 . 11706. URL https://doi . org/10 . 48550/arXiv . 2306 . 11706 .

Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. A comprehensive survey on pretrained foundation models: A history from BERT to chatgpt. CoRR , abs/2302.09419, 2023. doi: 10 . 48550/arXiv . 2302 . 09419. URL https://doi . org/10 . 48550/ arXiv . 2302 . 09419 .

Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In Conference on Computer Vision and Pattern Recognition , pages 15617-15629. IEEE, 2022. doi: 10 . 1109/ CVPR52688 . 2022 . 01519. URL https://doi . org/10 . 1109/CVPR52688 . 2022 . 01519 .

Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. In The Tenth International Conference on Learning Representations . OpenReview.net, 2022. URL https://openreview . net/forum?id=Opmqtk_GvYL .

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , pages 5998-6008, 2017. URL https://proceedings . neurips . cc/paper/2017/hash/ 3f5ee243547dee91fbd053c1c4a845aa-Abstract . html .

Ingmar Schubert, Jingwei Zhang, Jake Bruce, Sarah Bechtle, Emilio Parisotto, Martin A. Riedmiller, Jost Tobias Springenberg, Arunkumar Byravan, Leonard Hasenclever, and Nicolas Heess. A generalist dynamics model for control. CoRR , abs/2305.10912, 2023. doi: 10 . 48550/arXiv . 2305 . 10912. URL https://doi . org/10 . 48550/arXiv . 2305 . 10912 .

J. Richalet, A. Rault, J. L. Testud, and J. Papon. Model predictive heuristic control: Applications to industrial processes. Autom. , 14(5):413-428, 1978. doi: 10 . 1016/0005-1098(78)90001-8. URL https://doi . org/10 . 1016/0005-1098(78)90001-8 .

Carlos E. Garcia, David M. Prett, and Manfred Morari. Model predictive control: Theory and practice - A survey. Autom. , 25(3):335-348, 1989. doi: 10 . 1016/0005-1098(89)90002-2. URL https://doi . org/ 10 . 1016/0005-1098(89)90002-2 .

Max Schwenzer, Muzaffer Ay, Thomas Bergs, and Dirk Abel. Review on model predictive control: An engineering perspective. The International Journal of Advanced Manufacturing Technology , 117(5-6): 1327-1349, 2021.

Yanchao Sun, Shuang Ma, Ratnesh Madaan, Rogerio Bonatti, Furong Huang, and Ashish Kapoor. SMART: self-supervised multi-task pretraining with control transformers. In The Eleventh International Conference on Learning Representations . OpenReview.net, 2023. URL https://openreview . net/ pdf?id=9piH3Hg8QEf .

Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa M. Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforcement learning. CoRR , abs/2301.08028, 2023. doi: 10 . 48550/arXiv . 2301 . 08028. URL https://doi . org/10 . 48550/arXiv . 2301 . 08028 .

Learning to Learn . Springer, 1998. ISBN 978-1-4613-7527-2. doi: 10 . 1007/978-1-4615-5529-2. URL https://doi . org/10 . 1007/978-1-4615-5529-2 .

Jonathan Baxter. Theoretical models of learning to learn. In Learning to Learn , pages 71-94. Springer, 1998. doi: 10 . 1007/978-1-4615-5529-2 \ _4. URL https://doi . org/10 . 1007/978-1-4615-5529-2_4 .

Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive metalearner. In 6th International Conference on Learning Representations . OpenReview.net, 2018. URL https://openreview . net/forum?id=B1DmUzWAW .

Luisa M. Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In Proceedings of the 36th International Conference on Machine Learning , pages 7693-7702. PMLR, 2019. URL http://proceedings . mlr . press/v97/zintgraf19a . html .

Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt in dynamic, real-world environments through meta-reinforcement learning. In 7th International Conference on Learning Representations . OpenReview.net, 2019. URL https://openreview . net/forum?id=HyztsoC5Y7 .

Haozhe Wang, Jiale Zhou, and Xuming He. Learning context-aware task reasoning for efficient meta reinforcement learning. In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems , pages 1440-1448. International Foundation for Autonomous Agents and Multiagent Systems, 2020. URL https://dl . acm . org/doi/abs/10 . 5555/3398761 . 3398927 .

Richa Upadhyay, Ronald Phlypo, Rajkumar Saini, and Marcus Liwicki. Sharing to learn and learning to share - fitting together meta-learning, multi-task learning, and transfer learning: A meta review. CoRR , abs/2111.12146, 2021. URL https://arxiv . org/abs/2111 . 12146 .

Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in metareinforcement learning. In Conference on Robot Learning , pages 1478-1487. PMLR, 2022. URL https://proceedings . mlr . press/v205/beck23a . html .

Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. In Proceedings of the 38th International Conference on Machine Learning , pages 9301-9312. PMLR, 2021. URL http://proceedings . mlr . press/v139/sarafian21a . html .

Qinyuan Ye and Xiang Ren. Learning to generate task-specific adapters from task description. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics , pages 646653. Association for Computational Linguistics, 2021. doi: 10 . 18653/v1/2021 . acl-short . 82. URL https://doi . org/10 . 18653/v1/2021 . acl-short . 82 .

Jonathan Pilault, Amine El hattami, and Christopher Pal. Conditionally adaptive multi-task learning: Improving transfer learning in NLP using fewer parameters & less data. In International Conference on Learning Representations , 2021. URL https://openreview . net/forum?id=de11dbHzAMF .

Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-efficient multi-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics , pages 565-576. Association for Computational Linguistics, 2021. doi: 10 . 18653/v1/2021 . acl-long . 47. URL https://doi . org/10 . 18653/v1/ 2021 . acl-long . 47 .

Hamish Ivison and Matthew E. Peters. Hyperdecoders: Instance-specific decoders for multi-task NLP. In Findings of the Association for Computational Linguistics , pages 1715-1730. Association for Computational Linguistics, 2022. URL https://aclanthology . org/2022 . findings-emnlp . 124 .

Haoxiang Shi, Rongsheng Zhang, Jiaan Wang, Cen Wang, Yinhe Zheng, and Tetsuya Sakai. Layerconnect: Hypernetwork-assisted inter-layer connector to enhance parameter efficiency. In Proceedings of the 29th International Conference on Computational Linguistics , pages 3120-3126. International Committee on Computational Linguistics, 2022. URL https://aclanthology . org/2022 . coling-1 . 276 .

Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems , pages 506-516, 2017. URL https://proceedings . neurips . cc/paper/2017/hash/ e7b24b112a44fdd9ee93bdf998c6ca0e-Abstract . html .

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning , pages 2790-2799. PMLR, 09-15 Jun 2019. URL https://proceedings . mlr . press/v97/houlsby19a . html .

Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pages 46-54. Association for Computational Linguistics, 2020. doi: 10 . 18653/v1/2020 . emnlp-demos . 7. URL https://doi . org/10 . 18653/v1/2020 . emnlp-demos . 7 .

Jonas Pfeiffer, Ivan Vuli´ c, Iryna Gurevych, and Sebastian Ruder. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7654-7673, Online, November 2020. Association for Computational Linguistics. doi: 10 . 18653/v1/2020 . emnlp-main . 617. URL https://aclanthology . org/2020 . emnlp-main . 617 .

Hang Le, Juan Miguel Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. Lightweight adapter tuning for multilingual speech translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics , pages 817-824. Association for Computational Linguistics, 2021. doi: 10 . 18653/v1/2021 . acl-short . 103. URL https://doi . org/10 . 18653/v1/ 2021 . acl-short . 103 .

Chin-Lun Fu, Zih-Ching Chen, Yun-Ru Lee, and Hung-yi Lee. Adapterbias: Parameter-efficient tokendependent representation shift for adapters in NLP tasks. In Findings of the Association for Computational Linguistics: NAACL 2022 , pages 2608-2621. Association for Computational Linguistics, 2022. doi: 10 . 18653/v1/2022 . findings-naacl . 199. URL https://doi . org/10 . 18653/v1/2022 . findingsnaacl . 199 .

Bethan Thomas, Samuel Kessler, and Salah Karout. Efficient adapter transfer of self-supervised speech models for automatic speech recognition. In International Conference on Acoustics, Speech and Signal Processing , pages 7102-7106. IEEE, 2022. doi: 10 . 1109/ICASSP43922 . 2022 . 9746223. URL https: //doi . org/10 . 1109/ICASSP43922 . 2022 . 9746223 .

David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In 5th International Conference on Learning Representations . OpenReview.net, 2017. URL https://openreview . net/forum?id=rkpACe1lx .

Andrew William Moore. Efficient memory-based learning for robot control. Technical report, University of Cambridge, Computer Laboratory, 1990.

Richard S. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances in Neural Information Processing Systems , pages 1038-1044. MIT Press, 1995. URL http://papers . nips . cc/paper/1109-generalization-in-reinforcementlearning-successful-examples-using-sparse-coarse-coding .

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR , abs/1606.01540, 2016. URL http://arxiv . org/abs/ 1606 . 01540 .

Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE Trans. Syst. Man Cybern. , 13(5):834-846, 1983. doi: 10 . 1109/TSMC . 1983 . 6313077. URL https://doi . org/10 . 1109/TSMC . 1983 . 6313077 .

Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau. Learning robust state abstractions for hidden-parameter block mdps. In 9th International Conference on Learning Representations . OpenReview.net, 2021. URL https://openreview . net/forum?id=fmOOI2a3tQP .

Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. CoRR , abs/1810.12282, 2018. URL http://arxiv . org/ abs/1810 . 12282 .

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE, 2012. doi: 10 . 1109/IROS . 2012 . 6386109. URL https://doi . org/10 . 1109/IROS . 2012 . 6386109 .

John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. In 4th International Conference on Learning Representations , 2016. URL http://arxiv . org/abs/1506 . 02438 .

Ryan Julian, Benjamin Swanson, Gaurav S. Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. In Conference on Robot Learning , pages 2120-2136. PMLR, 2020. URL https://proceedings . mlr . press/v155/ julian21a . html .

Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang-Wei Edward Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems XVI , 2020. doi: 10 . 15607/RSS . 2020 . XVI . 064. URL https://doi . org/10 . 15607/RSS . 2020 . XVI . 064 .

Laura M. Smith, J. Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In International Conference on Robotics and Automation , pages 1593-1599. IEEE, 2022. doi: 10 . 1109/ICRA46639 . 2022 . 9812166. URL https://doi . org/10 . 1109/ICRA46639 . 2022 . 9812166 .

Chelsea Finn, Tianhe Yu, Justin Fu, Pieter Abbeel, and Sergey Levine. Generalizing skills with semisupervised reinforcement learning. In 5th International Conference on Learning Representations . OpenReview.net, 2017. URL https://openreview . net/forum?id=ryHlUtqge .

Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptive deep RL via meta-learning. In 8th International Conference on Learning Representations . OpenReview.net, 2020. URL https: //openreview . net/forum?id=Hkl9JlBYvr .

Luisa M. Zintgraf, Leo Feng, Cong Lu, Maximilian Igl, Kristian Hartikainen, Katja Hofmann, and Shimon Whiteson. Exploration in approximate hyper-state space for meta reinforcement learning. In Proceedings of the 38th International Conference on Machine Learning , pages 12991-13001. PMLR, 2021. URL http://proceedings . mlr . press/v139/zintgraf21a . html .

Tao Chen, Adithyavairavan Murali, and Abhinav Gupta. Hardware conditioned policies for multi-robot transfer learning. In Advances in Neural Information Processing Systems , pages 9355-9366, 2018. URL https://proceedings . neurips . cc/paper/2018/hash/ b8cfbf77a3d250a4523ba67a65a7d031-Abstract . html .

Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research , 23(274):1-18, 2022. URL http://jmlr . org/ papers/v23/21-1342 . html .

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems , pages 8024-8035, 2019. URL https://proceedings . neurips . cc/paper/2019/hash/ bdbca288fee7f92f2bfa9f7012727740-Abstract . html .

Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. CoRR , abs/2103.10427, 2021. URL https://arxiv . org/ abs/2103 . 10427 .

Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sebastian Tschiatschek, Cheng Zhang, Sam Devlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise injection and information bottleneck. In Advances in Neural Information Processing Systems , pages 13956-13968, 2019. URL https://proceedings . neurips . cc/paper/2019/hash/ e2ccf95a7f2e1878fcafc8376649b6e8-Abstract . html .

Xingyu Lu, Kimin Lee, Pieter Abbeel, and Stas Tiomkin. Dynamics generalization via information bottleneck in deep reinforcement learning. CoRR , abs/2008.00614, 2020. URL https://arxiv . org/ abs/2008 . 00614 .

Ben Eysenbach, Ruslan Salakhutdinov, and Sergey Levine. Robust predictable control. In Advances in Neural Information Processing Systems , pages 27813-27825, 2021. URL https://proceedings . neurips . cc/paper/2021/hash/e9f85782949743dcc42079e629332b5fAbstract . html .

Appendix

A Formalising the Theoretical Foundations

In this section, we more formally consider our theoretical results from Section 4. We first define our problem formulation in Assumption A.1, then state the theorem in Theorem A.5. Finally, we prove the theorem at the end of this section.

Assumption A.1. (Problem Formulation): Suppose we have a CMDP where S , A ⊆ R d and γ ∈ (0 , 1) . The context c ∈ R d defines a goal location, with a reward function R c ( s, a, s ′ ) = 1 if || s ′ -c || < ¯ τ and 0 otherwise. The episode terminates when the reward obtained is 1 . Suppose further that ¯ τ = τD , with D ∈ R + being the distance the agent travels in a single step and τ ∈ Z + being the number of steps it takes the agent to traverse a distance of ¯ τ . Here τ and D are kept fixed across contexts.

Definition A.2. (Context Averaged Value Function): Denote ¯ V π C ( s ) = E c ∼C [ V π c ( s )] , i.e., the expected value of the value function of policy π in state s , over all contexts in the set C . Similarly ¯ V ∗ C ( s ) = E c ∼C [ V ∗ c ( s )] is the average of the value functions of the optimal policies on each context.

Definition A.3. (Value Function Optimality Ratio): Let 0 ≤ α π C ( s ) = ¯ V π C ( s ) ¯ V ∗ C ( s ) ≤ 1 denote how close the average value function of policy π is to the optimal context-specific value function. We use the optimality ratio instead of the optimality gap [47] because the optimality gap depends heavily on the reward scale of the environment, whereas the optimality ratio depends only on the ratios.

̸

Definition A.4. (Minimum Inter-context Distance): Let d min = min c i = c j ∈C ⌈ || c i -c j || D ⌉ be the minimum number of steps required to travel between any two context centers.

Theorem A.5. Then, in this problem space, we have the following:

(i) For some set of contexts C far with cardinality N , and some state s , all deterministic unaware policies π will have:

i.e., as the contexts within C far move further away from each other the agent's performance scales inversely with the number of contexts.

(ii) For some set, C close and all states s , at least one deterministic unaware policy π g will have:

regardless of the number of contexts in C close .

Proof. (i) Consider a set of contexts C far such that for all c i , c j ∈ C far , i = j , || c i -c j || > 4¯ τ . Consider the state s = 1 N ∑ N i =1 c i , i.e., in the middle of all contexts. Now, suppose the optimal policy per context would require only β steps to reach the appropriate goal. An unaware policy that reaches every goal, by contrast, must travel to each context in sequence. If this policy visits c 1 , c 2 , . . . , c N , in order, then the value functions for each context would be:

V π c 1 ( s ) = γ β · 1 = V ∗ c 1 ( s )

V π c 2 ( s ) = γ β + ⌈ || c 1 -c 2 || D ⌉

Additionally:

̸

and so on. Let d min be the number of steps required to move between the two closest distinct contexts in C far . Thus, d min ≥ 4¯ τ -¯ τ -¯ τ D = 2 τ . 6 Then, V π c i ( s ) ≤ γ β +( i -1) d min . Now,

Hence, α = ¯ V π C far ( s ) ¯ V ∗ C far ( s ) ≤ 1 N 1 -γ Nd min 1 -γ d min . Finally, as d min →∞ , γ d min → 0 and γ Nd min → 0 . Hence, lim N →∞ α = 1 N .

(ii) Consider a set of contexts C close such that for all c i , c j ∈ C close , || c i -c j || < ¯ τ . Then, consider an unaware policy that always travels to the joint intersection point of these contexts. This intersection point is at most ¯ τ away from the closest context to the agent, therefore the agent will perform at most τ unnecessary steps compared to the optimal policy. Hence, its value function has a lower bound of V ∗ c i ( s ) γ τ , and we have

Therefore,

B Linking Theory and Practice

The formulation in Assumption A.1 has consistent dynamics and context-dependent rewards (as the context defines the goal location, and the transition dynamics do not depend on the context). This is in contrast to the problem we aim to study, which has consistent rewards, but context-dependent dynamics; however, we can show that for the environment considered in Appendix A, these two are equivalent.

We can obtain the context-dependent dynamics formulation under the assumptions in Section 2, i.e., that only the transition dynamics change, as follows. Suppose the context defines a rotation matrix C i ∈ R d × d , with dynamics equation T ( s, a ) = s ′ = s + C i a and initial state s 0 = 0 . Here the goal is a fixed location g ∈ R d . The episode would terminate with a reward of 1 if || s ′ -g || < ¯ τ . We can then perform a coordinate transform to obtain C -1 i s ′ = C -1 i s + a . The goal would then depend on the context g c i = C -1 i g and the effect of an action in this new space would be consistent across contexts.

In essence, we change the reference frame to be that of the agent, leading to separate goal locations for each context, but consistent dynamics-corresponding to the problem in Assumption A.1.

In the remainder of this section, we link our theoretical and empirical results, showing that the ODE (for some context distributions) is an example of a non-overlapping environment, while in CartPole, an unaware policy can perform well.

B.1 ODE

Here we show that, in the ODE domain, the Unaware model can either perform well or poorly, depending on the context set it is evaluated on. In particular, we train models on context sets C A train = { 1 , 5 } and C B train = {-5 , -1 , 1 , 5 } and evaluate on C A test = [0 , 10] and C B test = [ -10 , 10] . These results are shown in Fig. 7. When only considering the positive contexts (left), the Unaware model performs only slightly worse than the Concat model. This corresponds to case (ii) in Theorem A.5, where the contexts are similar enough for a single policy to perform well. However, when considering both positive and negative contexts (right), the Unaware model fails completely, and performs much worse than the context-aware ones. This is because, in the ODE domain, positive and negative contexts cannot be solved using the same action. This corresponds to case (i) in Theorem A.5, where making progress on one context results in negative progress on another.

0.0

0.2

0.4

0.6

0.8

1.0

0

50

100

150

200

AER

Positive

0.0

0.2

0.4

0.6

0.8

1.0

1e6

Positive and Negative

Training Time (Steps)

Unaware

Concat

Adapter

B.2 CartPole

We now investigate CartPole. In particular, we vary only the pole length, and consider the training contexts as { 1 , 4 , 6 } , with the evaluation contexts being 301 equally-spaced points between 0 . 1 and 10 .

These results are shown in Fig. 8. While the Unaware model can achieve comparable generalisation performance to the context-informed models, it takes significantly longer to do so. In particular, to obtain an average reward above 400 , the Unaware model must train for more than 600 k steps, whereas cGate , Concat and our Adapter reach this threshold before 100 k steps. This shows that, while context is not necessary in this domain, incorporating it may be beneficial and can lead to much more sampleefficient generalisation compared to the Unaware model. This is particularly relevant when training on multiple training contexts, which seems to cause interference for the Unaware model. The contextaware approaches, however, are not as susceptible to this problem, as they are able to distinguish experience from different contexts. Furthermore, most of the context-aware models perform similarly in this case. In particular, our Adapter, Concat and cGate converge rapidly, while FLAP converges slightly slower, but achieves comparable generalisation performance at the end of training.

0

2

4

6

8

Training Time (Steps)

1e5

100

200

300

400

500

AER

Unaware

Concat

Adapter

cGate

FLAP

0

20

40

60

80

100

Mass

0

50

100

150

200

250

Distance Travelled

Unaware

Concat

Adapter

B.3 Ant

Here we consider the Ant domain, and vary the mass, without any distractor variables. As seen in Fig. 9, the Unaware model performs worse than the context-aware ones. This performance gap is due to the Unaware model being unable to simultaneously perform well on light and heavy masses, as seen in Fig. 9. In contrast, by utilising context, the agent can perform well in both settings despite their significant differences.

C cGate as a Special Case of the Decision Adapter

As discussed in Section 5, here we show that our architecture is, in fact, a generalisation of another network architecture introduced by prior work. Benjamins et al. [13] introduce cGate , which obtains an action as follows. It first calculates ϕ ( s ) ∈ R d using the state encoder ϕ and g ( c ) ∈ R d using the

context encoder g . Then, it calculates features h = cGate ( s, c ) = ϕ ( s ) ⊙ g ( c ) , with ⊙ being the elementwise product. The action is obtained as a = f ( h ) , where f : R d →A is the learned policy. Here, both the state encoder ϕ and the context encoder g correspond to neural networks that each output a d -dimensional vector.

This approach is a specific instantiation of our general adapter architecture, where we have one adapter module A = A i . In particular, the state encoder ϕ corresponds to the partial neural network before our adapter module, i.e., layers L 1 , L 2 , . . . , L i -1 . The policy f corresponds to the layers after our adapter module, L i , L i +1 , . . . , L n . Thus, we have that ϕ ( s ) = L i -1 ( x i -1 ) and f ( h ) = L n ( x n ) . If we define our adapter network A ( ϕ ( s ) | c ) = ϕ ( s ) ⊙ g ( c ) , we can recover the same result as cGate . This would be possible if we let our adapter consist of one layer without a nonlinearity or a bias vector, with its weight matrix being set to W = H ( c ) = diag ( g ( c )) . This would result in

where we have used the fact that, for two vectors a, b ∈ R d , the matrix multiplication diag ( a ) b is the same as the elementwise product a ⊙ b . This output vector is then passed to the policy network f . Our approach, however, is strictly more powerful than a single elementwise product, as our adapter can be an entire nonlinear neural network.

D Experimental Details

D.1 Environments

This section contains more details about the experimental setup of our environments.

D.1.1 Context Normalisation

Before we pass the ground truth context to the models, we first normalise them to a range between 0 and 1 . In particular, for each context dimension i , we calculate the context to be given to the agents c norm,i as

where max i is the largest value for dimension i in the training context set. This results in the largest training contexts having values of 1 . 0 . During evaluation, we perform the same normalisation, so contexts outside the convex hull of the training set will potentially result in values larger than 1 . 0 .

D.1.2 ODE

During training, for each context, we cycle between starting states, with each episode taking a new starting state from the set { 1 . 0 , 0 . 5 , -0 . 5 , -1 . 0 } . Thus, the agent trains on the first context for 4 episodes, each with a different initial state. Then it goes on to the second context, and so on. When we reach the end of the context list, we simply start at the first context again. We do this to ensure training is comparable for each method, and that there is sufficient diversity in the initial state distribution. During evaluation, we fix the starting state at x 0 = 1 . 0 to isolate the effect of context.

D.1.3 CartPole

Table 1 illustrates the context variables we use in CartPole. While we consider all five variables as the context, we varied only the pole length for our experiments. The value for each state variable at the start of the episode is randomly sampled from the interval [ -0 . 05 , 0 . 05] .

D.2 Hyperparameters

Weuse ReLU for all activation functions. We also aim to have an equal number of learnable parameters for each method and adjust the number of hidden nodes to achieve this.

We use Soft-Actor-Critic [39, SAC] for all methods, to isolate and fairly compare the network architectures. We use the high-performing and standard implementation of SAC from the CleanRL library [98] with neural networks being written in PyTorch [99]. We use the default hyperparameters, which are listed in Table 2. SAC learns both an actor and a critic. The actor takes as input the state and outputs µ and log( σ ) , which are used to construct the probability distribution over actions N ( µ, σ ) which is sampled from to obtain an action. The critic receives the performed action a as well as the state s , and outputs a single number, representing the approximate action-value ˆ q ( s, a ) . Fig. 10 illustrates each baseline.

D.3 Adapter Configuration and Design Decisions

Now, our method provides a great degree of flexibility in terms of configuration, and we must make multiple design decisions to obtain a usable instantiation of the Decision Adapter. In this section, we briefly detail the default settings we use and justify why these are reasonable. We empirically investigate the merits of these choices and compare them against alternative options in Appendix E.

D.3.1 Bottleneck Architecture

We use a bottleneck architecture in our generated adapter modules, as described by Houlsby et al. [76]. In essence, we have a down-projection layer that transforms features from d i -dimensional to p -dimensional, with p < d i . Then, the second layer up-projects this back to a d i -dimensional vector.

The bottleneck architecture is beneficial for two reasons. Firstly, it reduces the number of required parameters in our adapter model. For instance, if we have two layers in this bottleneck architecture, the number of parameters would differ substantially compared to one layer that transforms the d i -dimensional features to d i dimensions. In the first case, the number of parameters would be on the order of O ( d i × p + p × d i ) , corresponding to a d i × p matrix for the down-projection and a p × d i matrix for the up-projection. 7 If we only have one layer, then we have a single d i × d i matrix. Thus, the bottleneck requires O ( pd i ) parameters, whereas the single layer requires O ( d 2 i ) . If we, for example, have d i = 256 and p = 32 , the bottleneck architecture has four times fewer parameters than the single layer. This, in turn, makes it easier for the adapter hypernetwork to generate useful weights.

Secondly, the bottleneck architecture imparts a low-rank inductive bias on the adapter, which may prevent it from overfitting [100]. This idea is related to prior work that has shown that, counterintuitively, reducing the capacity of the agent's neural network can improve generalisation and reduce

s

s

a

σ

(a) The Unaware Model

s

c

s

a

σ

(b) The Concatenate Model

s

s

c

⊙

σ

a

(c) The

cGate

Model, with

⊙

indicating an

elementwise product

s

s

c

·

a

σ

(d) The FLAP Model, with

·

indicat-

ing a dot product

overfitting [101-103]. Although our approach does not incorporate information-bottleneck losses like these methods, the observed benefits of reduced capacity provide additional intuition for utilising the bottleneck architecture. Finally, despite any possible information loss due to the bottleneck layer, the skip connection allows the network to retain any important information from the input.

D.4 Compute

For compute, we used an internal cluster consisting of nodes with NVIDIA RTX 3090 GPUs. For a single experiment and a single method, the runs took between 1 and 3 days to complete all seeds.

E Adapter Ablations

Here we experimentally justify some of the adapter-specific design decisions we made for our empirical results. The experiments in this section also give us insight into how robust the Decision Adapter is to changing its configuration options. We investigate the effects of the following factors, with our conclusions listed in bold .

The network architecture of the Adapter module. We find that Most adapter architectures perform well. This result is expanded upon in Appendix E.1.

Whether the hypernetwork uses chunking. Hypernetwork chunking outperforms nonchunking methods while using significantly fewer parameters (Appendix E.2) .

Using a skip connection. The Decision Adapter can perform well with or without a skip connection (Appendix E.3) .

The location of the adapter module in the main network. Most locations lead to high performance, except if the adapter is at the very start or very end of the network (Appendix E.4) .

Having an activation function before the adapter. Not having an activation function before the adapter in the actor model outperforms the alternatives (Appendix E.5) .

For this entire section, we consider the ODE environment and follow the same procedure as in Section 7.1. We use the ODE due to its fast training, and the fact that context is necessary, which enables us to determine if a particular design choice of the Adapter results in poor use of the context.

E.1 Adapter Architecture

Here we examine the effects of changing the Adapter module's network architecture. We are interested in how the architecture impacts performance for two reasons. First, we wish to empirically justify our bottleneck architecture. Second, we wish to show that the Adapter performs well with a wide variety of network architectures, and is not particularly sensitive to this choice.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training Time (Steps)

1e5

0

50

100

150

200

AER

Adapter Architecture

[]

[8]

[16]

[32]

[64]

[256]

[32, 32]

As discussed in Appendix D.3, we use a bottleneck architecture of size 32 . Thus, the 256 -dimensional features are transformed into 32 -dimensional features and then projected back to 256 dimensions. We denote this architecture as [32] , since the adapter has a single hidden layer of 32 neurons. Fig. 11 illustrates the performance for various other adapter architectures. Most methods perform comparably, but using 8 hidden nodes or not having any hidden layers performs slightly worse than the base architecture. A large hidden layer of 256 nodes also does not outperform our bottleneck architecture. Finally, there is no particular benefit to having multiple layers in the adapter module.

E.2 Hypernetwork Chunking

Next, we compare our default architecture choice, which uses hypernetwork chunking, against non-chunking settings. We do this because hypernetwork chunking allows us to have significantly fewer learnable parameters (resulting in faster training) compared to the alternative of predicting the entire weight vector in one forward pass. Thus, if our approach performs comparably or better than non-chunking settings, then the chunking architecture is justified.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training Time (Steps)

1e5

0

25

50

75

100

125

150

175

200

AER

Hypernetwork Architecture

Base

[]

[100,100]

[100,100,100]

Chunked [66, 66]

We consider a non-chunking hypernetwork architecture of no hidden layers, as well as 2 and 3 hidden layers of size 100 , respectively. Our final option is a chunked hypernetwork, with chunks of size 660, and two hidden layers of size 66 each (this is roughly twice as large as our default configuration option discussed in Appendix D.3). These results are shown in Fig. 12. We find that both chunking configurations perform the best. The non-chunking approaches perform slightly worse, particularly if the hypernetwork has no hidden layers.

These non-chunking configurations also have significantly more learnable parameters than the default. For instance, the base setting results in the actor's hypernetwork having around 16 k parameters, whereas the [100 , 100] , non-chunking setting corresponds to roughly 1 . 7 Mparameters. In summary, by using hypernetwork chunking, we can obtain high performance while using a very small number of learnable parameters.

E.3 Skip Connection

Here we investigate whether a skip connection is beneficial or not. As mentioned in Section 5, the skip connection sets the updated x as x = A ( x ) + x , whereas if we remove it, the function becomes just x = A ( x ) . In Fig. 13, we can see that using a skip connection performs similarly to the alternative. This shows that the Decision Adapter is robust to this parameter, and can perform well with or without a skip connection.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training Time (Steps)

1e5

0

25

50

75

100

125

150

175

200

AER

Skip Connection

No Skip Connection

a

s

s

A

B

C

D

E.4 Adapter Location

We next examine the effects of changing the location of the adapter module in our network, and what happens if we have multiple modules. This information would be useful when using our method in practice, and deciding where and how many modules to place. There are generally many options regarding the location, as an adapter module can be placed between any two layers of the primary network.

As described in Section 5, our actor network consists of a 'trunk', containing one hidden layer, which maps the state to a 256 -dimensional feature vector, and a one-layer action head, directly mapping the trunk features to an action. Here we consider the following locations for the adapter in both the actor and critic networks:

Base The standard adapter used in the rest of this chapter, which is placed before the action head's layer. In the critic, the adapter is placed before the last layer. This corresponds to location (C) in Fig. 14.

Base F Placing the adapter in the actor network before the final layer of the trunk, leaving the critic unchanged (location B).

Start The adapter here is at the start of both networks, i.e., the adapter module receives the state as input (location A).

End Here, we place the adapter at the very end. Thus, the adapter takes in the predicted action and outputs a modified one. The critic's adapter is also placed at the end. This corresponds to location (D) in Fig. 14.

All F Three adapter modules, one at the start of the trunk, one before and one after its final layer. The critic also has three adapters, one at the start, one before the last layer and one after the last layer. The actor has adapters in locations (A), (B) and (C).

In all cases, to ensure comparability against the base model, there is no activation function after the trunk in the actor model. These results are illustrated in Fig. 15. Overall, the base model performs the best. The adapter in the trunk performs slightly worse and the performance is quite poor if we put the adapter modules at the start of the networks. If we have three adapter modules, the performance is similar to only having one before the final layer. Finally, when adapting only the final action, the performance is also poor. Overall, the performance is low if we have the adapter only at the very

start or end of the network. Within a range of locations in the 'middle' of the network, however, the performance is high and only slightly worse than the default setting.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training Time (Steps)

1e5

0

25

50

75

100

125

150

175

200

AER

Base

BaseF

Start

End

All

F

E.5 Activation Function Before Adapter

Our final ablation considers the benefits of not having an activation function before the adapter module inside the actor network. We perform this experiment to determine if our choice of not having an activation function before the adapter in the actor is justified, and if it is necessary to do the same for the critic.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training Time (Steps)

1e5

0

25

50

75

100

125

150

175

200

AER

No Activation

No Activation (Critic)

Activation

Only log(

) Activation

Here we consider four options: (1) not having an activation function after the trunk; (2) having one; (3) having an activation function after the trunk, but only for the standard deviation head; and (4) not having an activation before the adapter in the critic. Option (1) corresponds to the default choice. These results are shown in Fig. 16. Not having an activation function outperforms having one, and having an activation for only the standard deviation head is similar to not having one. Thus, not having an activation function before the adapter inside the critic network performs as well as the base option, which has this activation. Overall, this result justifies our design choice by showing that it performs comparably to or better than the alternatives.

F Additional Results

This section contains some additional results that we refer to in the main text.

F.1 Single Dimension Interpolation, Train & Extrapolation

See Fig. 17 for the performance of each model on the ODE on the particular context sets, Interpolation, Training and Extrapolation. These plots therefore correspond to subsets of the evaluation range in the left pane of Fig. 3.

0

1

2

3

4

5

0

50

100

150

200

AER

Train

0

1

2

3

4

5

Training Time (Steps)

Interpolation

0

1

2

3

4

5

1e5

Extrapolation

Unaware

Concat

Adapter

cGate

FLAP

F.2 Multiple Dimension Heatmaps

Fig. 18 illustrates the granular performance of each method in the ODE domain, corresponding to the right pane of Fig. 3. Our Decision Adapter outperforms all baselines and, as can be seen in Figs. 18a and 18b, generalises further away from the training contexts than the Concat model. In particular, these heatmaps show that both models perform well on the training contexts (the blue squares). However, the Adapter generalises much better, especially to contexts far outside the training range. For example, when | c 1 | ∈ [8 , 10] (corresponding to the far left and far right of the heatmaps), the Adapter's rewards are higher than the Concat model's. The Adapter obtains between 50 and 150 more reward than the Concat model - which is a substantial difference, as the maximum reward in this environment is 200 . This region of context-space is far outside of the convex hull of the training range, which contains values of | c 1 | only up to 5 . 0 . Closer to the training range; for instance, around the boundary of the green square, the Adapter still outperforms the Concat model, but the difference is less pronounced.

FLAP and cGate perform poorly, even though they obtain near-perfect rewards on the training contexts. There may be several reasons for this. First, here we train on 16 relatively sparsely distributed contexts, with 300 k steps in total, resulting in fewer episodes in each context compared to Section 7.1. Second, with two dimensions, the interpolation range makes up a smaller proportion of the evaluation range compared to the one-dimensional case. This is because, in one dimension, the range [ -5 , 5] makes up half of the [ -10 , 10] evaluation range. In two dimensions, however, the interpolation range makes up only a quarter of the evaluation set. Thus, the overall average performance in Fig. 3 is more skewed towards extrapolation. This, coupled with the fact that both cGate and FLAP perform poorly on extrapolation (Fig. 17, right), may explain why they underperform in this case. Lastly, it is generally easier to generalise in the one-dimensional ODE, as the optimal action depends mainly on the signs of the context and state variables. By contrast, the multidimensional case is more difficult, as the dynamics equation includes a nonlinear action term. This is also why, for instance, our Decision Adapter performs worse in this case (around 160 overall evaluation reward) compared to the one-dimensional ODE (around 190 ).

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

1

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

0

22

29

53

92

104

89

66

40

45

149 144 116

28

41

51

42

76

74

37

15

7

63

18

26

61

101 116 106

84

64

54

143 178 104

31

21

55

77

98

90

35

10

8

70

50

39

69

118 140 119 117

78

59

157 187

56

22

26

70

102 131 102

42

14

11

81

53

38

69

142 179 174 129 103

74

142 198

28

15

46

98

130 167 128

57

20

36

96

42

61

79

153 192 198 157 110

83

143 198

28

26

81

114 189 185 133

82

62

38

112

35

66

78

146 179 199 187 113

85

131 199

18

45

85

174 199 177 132

93

49

32

115

52

76

69

134 165 196 195 138 123 114 189

47

79

111 192 190 144

85

62

47

42

117

35

48

85

141 162 175 169 139 158 133 190

77

134 150 177 166 128 111 100

70

56

124

29

52

76

127 149 146 148 137 166 141 190

98

140 149 168 172 169 155 116

87

39

126

32

46

79

151 184 194 162 137 142 199 191 199 115 147 178 195 177 148 108

60

23

137

18

43

70

138 190 199 183 152 188 199

10

199 140 133 179 199 180 145 117

64

59

134

33

38

74

154 177 197 192 136 133 199 191 199 165 160 184 199 155 155

98

48

44

140

17

52

75

153 180 185 166 139 172 147 196 162 179 158 179 180 149 121

91

57

44

133

10

27

58

144 166 179 181 115 119

75

197

76

166 160 166 182 125 100

95

65

52

117

22

33

64

144 180 198 187 111 102

83

198

30

157 144 196 190 149

99

61

49

35

116

14

12

44

162 196 199 186 109 111 113 199

29

109 156 192 199 166 129

78

54

39

119

21

27

59

176 197 199 142 100

38

132 199

16

68

128 172 199 185 113

88

54

26

111

8

17

81

178 192 161 124

90

33

120 192

19

33

106 143 184 180 105

71

50

30

101

8

38

73

165 163 119 102

78

41

123 198 102

50

93

141 155 183 110

87

47

32

100

53

25

95

135 137 102

87

34

65

136 153 121

42

86

123 127 164 136

88

56

20

95

30

36

78

118 115

77

75

20

67

137 150 131

29

79

103 119 149 118

83

49

42

86

29

39

71

142 165 162 145 102

98

139 178

88

84

108 145 162 154 119

80

49

34

109

Average = 109.0

(a) Concat

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

1

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

0

147 186 171 171 138 118 114

88

106 167 186 179

85

76

87

110 126 144 127 129 108 132

152 188 177 174 180 130 103

97

102 185 192 178

84

82

93

127 158 165 135 126 126 141

145 174 183 185 183 169 112

94

94

187 198 144

86

82

109 147 183 176 140 136 123 145

143 192 185 186 187 187 160

84

64

180 198 112

86

91

139 175 197 195 163 137 124 152

148 182 185 186 187 187 180 104

82

186 198

99

76

121 173 198 199 197 165 137 133 158

151 180 186 186 187 187 187 140

90

175 198

99

75

141 199 199 199 197 174 163 137 164

158 183 186 196 187 187 187 161 123 167 198 103 145 163 198 199 198 186 178 170 128 171

148 183 190 185 192 187 187 162 191 139 197 118 193 177 182 184 186 182 165 147 126 172

158 180 189 192 199 199 198 187 198 133 195 163 198 190 199 198 182 177 162 131 112 178

147 174 182 184 199 199 199 186 197 199 191 199 198 178 199 199 192 187 146 152 119 182

144 161 177 177 192 193 199 175 198 199

10

199 197 174 196 199 193 180 171 134 135 172

131 159 172 198 199 199 198 175 196 199 191 199 191 168 199 199 194 187 165 147 130 181

136 180 186 185 185 182 162 166 198 150 195 123 198 168 184 182 190 197 177 160 133 173

122 148 176 170 164 175 186 168 189 111 197

45

131 129 170 183 172 164 146 136 107 152

130 148 167 179 198 199 198 166 126 137 198

27

114 157 198 199 198 191 160 141 105 159

131 141 188 198 199 199 192 129 100 150 198

17

77

158 199 199 199 198 191 149 111 158

127 154 177 198 199 199 183 113 110 150 198

27

45

112 186 199 198 198 198 170 120 155

133 167 184 197 187 177 140

89

122 198 198

62

47

87

171 191 195 194 194 168 140 154

143 161 184 181 175 163 126

89

140 198 198 173

54

56

122 188 187 194 181 170 141 154

130 159 172 171 161 143 112

99

149 192 198 188

53

57

103 141 183 180 170 170 143 146

132 141 164 158 150 128

72

118 130 189 195 182

68

70

78

123 158 172 157 155 153 138

141 169 180 184 183 177 162 133 138 171 187 126 114 126 161 178 185 184 165 149 126 159

Average = 159.0

(b) Adapter

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

1

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

0

7

14

18

22

19

37

36

49

56

88

108

88

53

40

15

15

17

32

28

25

16

37

5

12

15

70

68

28

45

39

59

102 130 104

53

51

25

22

45

28

39

35

31

48

16

20

43

59

70

56

45

50

81

134 140 104

57

25

43

42

61

39

43

52

36

58

21

29

42

65

101

92

59

57

76

146 184 112

51

29

75

88

101

97

39

49

25

73

25

20

36

67

108 136

78

58

95

186 198

96

45

51

102 148 115

84

40

35

44

84

19

25

49

65

112 153

84

54

110 193 198

62

62

73

111 198

98

61

45

65

67

91

21

38

27

48

113 144

92

105 102 199 197

71

63

90

112 164

81

44

76

67

42

90

26

35

48

53

91

114

98

140 162 172 196

64

114 172 125 105

93

78

50

59

59

98

11

33

27

60

102 118 100 140 178 127 195

89

197 127 137 142

98

77

85

66

63

103

13

25

51

83

140 150

99

143 190 199 188 199 166 130 123 167 155

89

102

69

74

122

24

14

20

62

147 139 137 129 187 199

10

199 182 128 152 198 162 114 100

41

51

114

12

21

46

66

158 144 122 137 159 199 188 199 136 127 159 188 130 113 105

55

43

119

8

8

50

96

137 128 104 147 195 150 195 168 198 128 179 169 108 133

90

75

58

120

8

16

34

79

130 127 108 154 161 151 196

86

191 149 139 168 131 124

76

62

55

112

22

17

23

56

107 135 108 109

94

159 197

77

103 121 143 182 172 121

60

76

53

102

6

9

33

51

132 184

90

72

95

184 198

74

70

88

151 199 133

83

69

57

51

97

20

9

13

63

137 146

75

53

78

180 198

55

44

51

139 172 149 105

72

65

41

89

4

9

29

63

75

103

44

52

66

168 174

66

38

45

122 121 132 100

63

58

53

75

9

12

14

37

49

59

49

35

85

124 130

99

21

36

60

89

86

68

74

39

54

59

6

27

14

34

36

43

45

42

69

90

84

93

26

27

42

53

74

70

58

33

28

47

7

11

35

36

27

43

42

34

53

82

72

72

22

29

30

59

50

55

44

44

48

43

14

19

32

59

98

109

79

86

112 154 161 104

90

82

104 128 104

82

65

54

47

85

Average = 85.0

(c)

cGate

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

1

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

0

10

6

5

6

15

11

13

22

9

54

68

104

44

24

22

41

21

19

33

11

17

26

10

21

15

7

19

22

17

18

14

77

101 118

27

27

25

22

19

25

18

27

25

31

10

25

17

16

48

24

17

17

31

107 111 126

19

25

46

27

26

43

30

7

12

37

11

23

37

29

73

41

30

28

37

99

124 129

22

13

32

51

49

55

9

16

29

45

9

5

30

57

102 108

73

27

24

65

185 135

9

16

45

119

97

31

36

34

17

58

27

31

19

42

72

184

68

45

47

61

197 151

21

38

109 179 100

17

14

24

15

70

28

27

17

50

56

77

62

61

69

69

197 122

59

66

75

130

55

31

40

24

16

63

24

26

29

53

43

40

63

125

96

76

196

66

121

77

49

46

52

38

44

20

16

62

26

51

53

80

72

35

48

82

157

94

194 112 170

96

48

53

79

54

30

29

33

76

13

12

35

81

101 107

80

36

90

197 190 199 117

98

64

124

87

38

23

11

25

82

16

29

40

66

124 188 114

79

137 197

10

199 135

75

142 195

84

49

47

21

27

94

23

43

33

58

98

137

59

67

102 197 190 199 105

57

75

106

45

41

10

46

38

82

24

36

28

50

80

87

33

48

169 118 195 105 138

63

54

64

88

36

16

41

29

72

25

24

37

56

53

59

60

95

117

83

196 107 137

87

53

60

47

57

15

31

39

68

36

24

44

51

45

122 116

85

51

71

197 158

55

50

73

107

48

36

16

29

28

69

27

23

27

38

108 179

62

44

48

86

197 147

33

43

63

173

68

18

14

28

32

69

15

23

26

94

129 109

57

23

43

77

198 136

33

7

30

120 101

51

39

13

8

63

9

31

44

70

83

41

36

29

64

103 163 156

36

19

36

95

80

36

14

25

20

57

19

38

42

48

75

44

30

59

63

106 124 132

26

9

24

43

58

54

14

15

7

49

14

24

32

25

26

27

37

51

41

111 114 104

31

24

21

29

37

56

29

5

5

40

11

11

22

28

28

24

38

30

46

110

86

92

44

27

20

24

13

45

44

10

10

36

18

25

30

48

69

79

53

51

69

103 154 133

66

45

53

86

60

40

25

22

21

60

Average = 60.0

(d) FLAP

F.3 Additional Baselines

We consider two ablations as additional baselines. The first, called AdapterNoHnet , is based on our adapter module, but there is no hypernetwork involved. The adapter's architecture and location are the same, but it is now a single MLP that takes in the concatenation of the state-based features and the raw context (as the model has to be context-conditioned, and must also process the state-based features). The second baseline, termed cGateEveryLayer , uses the same elementwise operation as cGate , but this happens at every hidden layer except at only one. Fig. 19 illustrate these results on the 1D and 2D ODE domains respectively, with the Adapter, Concat and cGate models there for reference. Overall, cGateEveryLayer does not outperform cGate , and AdapterNoHnet performs much worse than our hypernetwork-based adapter.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0

50

100

150

200

1D Context

0.0

0.5

1.0

1.5

2.0

2.5

3.0

1e5

2D Context

AER

Training Time (Steps)

Concat

Adapter

cGate

cGateEveryLayer

AdapterNoHnet

Optimal

F.4 AER Metric

Fig. 20 illustrates the difference between using the AER metric as defined in Section 6.1 (corresponding to Fig. 20a) and only considering the testing contexts (corresponding to Fig. 20b). Overall, the performance of each model is very similar; this is because the training contexts make up a small proportion of the entire evaluation range.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0

50

100

150

200

AER

1D Context

0.0

0.5

1.0

1.5

2.0

2.5

3.0

1e5

2D Context

Training Time (Steps)

Unaware

Concat

Adapter

cGate

FLAP

Optimal

(a) Training and Testing Contexts

0.0

0.5

1.0

1.5

2.0

2.5

3.0

0

50

100

150

200

AER Only Test

1D Context

0.0

0.5

1.0

1.5

2.0

2.5

3.0

1e5

2D Context

Training Time (Steps)

Unaware

Concat

Adapter

cGate

FLAP

Optimal

(b) Only Testing Contexts.

F.5 Distractors

The ODE domain displays similar distractor results to our other environments (see Section 7.2). In particular, Fig. 21 shows that the Adapter is robust to adding more irrelevant distractor variables, whereas Concat and cGate are not.

0

1

2

3

0

50

100

150

200

AER

Concat

0

1

2

3

Adapter

0

1

2

3

1e5

cGate

Training Time (Steps)

Number of distractor dimensions

0

1

20

100

F.6 Gaussian Distractor Experiments

This section expands upon the results in Section 7.2 and contains additional results where the distractor variables are not fixed.

F.6.1 Difference between training and testing

Fig. 5 contains the results on CartPole, and the ODE results are shown in Fig. 22. For both domains, the conclusion is similar in that the Adapter is still more robust to changing distractor variables compared to either Concat or cGate .

0

1

2

3

0

50

100

150

200

AER

Concat

0

1

2

3

Adapter

0

1

2

3

1e5

cGate

Training Time (Steps)

Number of distractor dimensions

0

1

20

100

F.6.2 Keeping the training and testing distractor distributions the same

Next, we consider a similar case, but the means of the Gaussians are the same across training and testing. Overall, the results for the ODE and CartPole (with a mean of 0 during both training and testing) are shown in Fig. 23. For the ODE, while the Adapter is insensitive to adding more distractor dimensions, both cGate and the Concat perform worse as we add more irrelevant distractor dimensions to the context. However, the difference is much less pronounced than in the case where the means are different between training and testing. Surprisingly, in the ODE, cGate performs slightly better with 20 distractor variables than with 0. These variables may effectively act as an additional, noisy, bias term that the model can utilise. For CartPole, the Concat and cGate models perform much worse with 100 distractor dimensions compared to none. The Adapter displays some noise, indicating that some seeds performed poorly while others performed well. This is similar to the results we observed in Appendix G.1-too much noise leads to unstable training.

Finally, we consider using a mean of 1 during both training and testing in Fig. 24. For the ODE, each model performs similarly regardless of the number of distractor variables. In CartPole there is not a

0

100

200

ODE

Concat

Adapter

cGate

0

1

2

3

0

200

400

CartPole

0

1

2

3

0

1

2

3

1e5

Training Time (Steps)

AER

Number of distractor dimensions

0

1

20

100

Optimal Reward

0

100

200

ODE

Concat

Adapter

cGate

0

1

2

3

200

400

CartPole

0

1

2

3

0

1

2

3

1e5

Training Time (Steps)

AER

Number of distractor dimensions

0

1

20

100

Optimal Reward

large difference between the various models when the mean of the distractor variables is 1 during both training and testing.

F.6.3 Summary

Overall, The Adapter is significantly less susceptible to overfitting to irrelevant distractors compared to the Concat model and cGate . However, if these distractors remain the same during training and testing (which may not be particularly likely), it matters less whether the model overfits or not, and all models are less affected by adding more distractors.

G Limitations

This section expands upon the limitations of our approach, outlined in Section 8.

G.1 Noisy Contexts

Now, while we use the ground truth context to isolate the effect of the network architecture on generalisation performance, this is not always available [12, 18]. A common approach is to learn a context encoder that uses experience in the environment to infer an approximation to the context, which is then used to act [18, 87]. This encoded context, however, may not always be entirely accurate. Thus, in this experiment, we examine how sensitive each context-aware model is to receiving noisy contexts during evaluation. We consider two cases; the first is where the models received the uncorrupted context during training, and the second is where the models also encountered noise during training. The first case could reasonably occur in reality, since we may have access to the ground truth context during training, but not evaluation [14]. The second case, where we train with noise, could correspond to training the RL policy alongside the context encoder, leading to some noise during training. This case additionally allows us to examine whether training with noise increases the models' robustness to context noise during evaluation.

We take the models trained during the primary CartPole experiments, in Appendix B.2, where we varied the pole length. We use the model checkpoints that were trained for 600 k timesteps, as both the Concat and Adapter models did not improve much by training for longer than this (see Fig. 8). The noise we add here, corresponding to normally distributed noise with varying standard deviations, was added to each context dimension instead of just the one representing pole length. We experiment with a standard deviation σ ∈ { 0 , 0 . 05 , 0 . 1 , 0 . 2 , 0 . 5 , 1 } . The noise was added after normalising the contexts such that the largest training context had a value of 1 . 0 . Thus, a standard deviation of 1 . 0 represents a substantial amount of noise. In addition to the models trained without noise, we

additionally train models that encountered noise during training. In particular, we consider standard deviations σ ∈ { 0 . 1 , 0 . 5 } , and change no other aspect of training.

The results are illustrated in Fig. 25. The left pane shows the performance of the models trained without noise. Overall, these results show that the Concat model is more susceptible to noisy contexts than our Adapter, bolstering the argument that treating context as a state dimension is not ideal [13]. Both models, however, generalise worse as the context becomes more inaccurate. This is in contrast to the Unaware model, which performs equally well regardless of the level of context noise. The Adapter still outperforms the Unaware model until the noise level reaches around σ = 0 . 5 . This noise level, however, is very large, and σ = 0 . 5 corresponds to a pole length of 3 meters. The default pole length is 0 . 5 meters, and the largest training length is 6 meters.

When training with a small amount of noise (Fig. 25, middle) both models perform similarly to training without noise. However, when training with a large amount of noise, corresponding to the right pane of Fig. 25, the Concat model becomes more robust to noisy contexts, but the Adapter performs worse. In particular, the Adapter model exhibits a large amount of variation across seeds, indicating that some seeds performed badly. On average, the Adapter performs similarly to the Unaware model until the level of context noise increases beyond the training noise of σ = 0 . 5 .

0.0

0.5

1.0

200

300

400

500

AER

Train Without Noise

0.0

0.5

1.0

Level of Context Noise During Evaluation (

)

Train With

=0.1

0.0

0.5

1.0

Train With

=0.5

Unaware

Concat

Adapter

G.2 Suboptimal Context Normalisations

G.2.1 ODE

As discussed in the main text, we first normalise the context before passing it to the agents. This is generally done by using the largest absolute value encountered during training. For instance, for the one-dimensional ODE experiment, we used c norm = c -0 5 as the normalisation function. Here we determine how sensitive the Decision Adapter is to changes in this procedure. This is useful as we may often need to choose the normalisation factor without knowing exactly which variations we will encounter during evaluation. To do this, we run four additional experiments, with normalisation values of 0 . 1 , 1 . 0 , 2 . 0 and 15 . 0 , respectively. We train the agents for only 300 k steps, as after training for this number of steps, most models performed similarly to their final performance. The results are shown in Fig. 26, and we can see that the Concat and Adapter models are both robust to the normalisation value. Our Decision Adapter loses less performance than the Concat model as we change the normalisation value. Despite losing some performance as the context normalisation value increases, both context-aware models still outperform the Unaware model.

G.2.2 CartPole

We now consider a similar setup in CartPole. In particular, we train on a Pole Mass of 0.1. We choose this variable as some of our experiments showed that the Unaware model generalises equally well regardless of the training contexts. We consider two scenarios: In the first, we normalise with respect to the Small setting (i.e. c norm = c 0 . 01 ). Thus, during training, the normalised context had a value

0

2

4

6

8

10

12

14

Context Normalisation

40

60

80

100

120

140

160

180

200

AER

Unaware

Concat

Adapter

0.0

0.2

0.4

0.6

0.8

1.0

0

100

200

300

400

500

AER

Normalisation = 0.01

0.0

0.2

0.4

0.6

0.8

1.0

1e5

Normalisation = 1

Training Time (Steps)

Unaware

Concat

Adapter

Optimal

of 1 . 0 ; during evaluation, it was nearly always larger than 1 . 0 . In the second case, we normalise with respect to the X-Large setting (corresponding to c norm = c 1 ). We trained on the Small pole length for both settings. Fig. 27 shows the results for this experiment. When normalising with respect to the Small setting, the Concat model fails to generalise at all, and the Adapter performs worse than Unaware. When changing the normalisation, however, the context-aware models perform significantly better.

G.2.3 Summary

Thus, the Decision Adapter is relatively robust to small changes in context normalisations. However, when encountering evaluation contexts that are 100 times larger than during training, appropriate

normalisation is crucial and can make a drastic difference - even when training on the exact same contexts. As seen in the ODE, having a normalisation value that is slightly too large does not cause significant performance penalties. Thus, normalising the contexts with a larger value than expected during training is a good heuristic.

G.3 Narrow Context Range

We next consider the effect of changing the set of training contexts. Here we aim to briefly illustrate that the training context range can have a large effect on the performance of the agents. In particular, we consider the ODE domain and 8 separate sets of training contexts. In all of these cases, we keep the normalisation consistent at c norm = c 5 to ensure comparability. The results when training on each context set are shown in Fig. 28. Overall, when we have a single positive and negative training context, performance is poor when this context is very small (a), and increases as it becomes larger (d and f). When we have multiple contexts, but insufficient variation (b, c, e), then performance is also suboptimal. For instance, in (b), the training contexts cover only the small region [ -1 , 1] . In (c), the training set of contexts does not contain any negative contexts, leading to poor generalisation. Finally, if contexts are varied, but spread out too far (g), the performance also suffers.

(a): {±0.1}

0

50

100

150

200

(b): {±0.1, ± 1.0}

(c): {1, 5}

(d): {±5}

0

1

2

3

(e): {±4, ± 5}

0

50

100

150

200

0

1

2

3

(f): {±7.5}

0

1

2

3

(g): {±0.1, ± 7.5}

0

1

2

3

(h): {±1, ± 5}

1e5

Training Time (Steps)

AER

Concat

Adapter

G.3.1 Overfitting

Another potential issue we may encounter is overfitting, which can occur when the training range is not diverse enough. To illustrate this problem, we conduct an experiment in the multidimensional ODE setting using the following set of training contexts:

The results, presented in Fig. 29, demonstrate that the Decision Adapter performs well initially but its generalisation performance suffers as training progresses. Specifically, as shown in Fig. 29b and Fig. 29c, our model exhibits worse extrapolation performance due to overfitting on the narrow training contexts.

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Training Time (Steps)

1e5

0

20

40

60

80

100

120

140

AER

Unaware

Concat

Adapter

(a)

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

1

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

0

60

64

79

95

95

114 135 110 117 114 120 133 132 111 109

93

108 109 108 113 111 106

59

63

55

70

99

124 120 135 132 123 119 123 130 112 110 108 106 113 132 127 126 109

69

70

58

81

93

121 128 157 142 135 129 127 127 122 124 115 112 117 125 132 129 115

88

82

52

52

72

111 110 153 139 126 128 134 140 138 115 118 123 144 142 136 123 116

93

91

73

76

76

87

115 142 141 134 130 138 140 139 118 142 137 121 116 127 130 117

88

98

89

77

90

120 142 147 146 151 140 150 151 142 138 131 127 128 121 119 115 124

75

82

80

92

95

124 136 159 151 138 167 173 176 171 150 146 128 130 121 116 122 130

81

83

85

72

78

105 148 161 156 158 170 169 172 168 150 146 122 126 118 101

99

127

56

60

65

71

105

88

110 160 158 172 159 158 166 165 164 122 104 125 131 124 119 123

82

80

86

72

77

98

134 119 139 160 119 135 158 167 164 131 130 127 127 109

94

119

83

84

85

86

91

107 120 167 159 110

10

119 159 172 172 147 127 117

92

54

32

109

82

85

82

90

83

92

102

97

143 160 119 135 160 162 165 156 142 116 110

79

54

115

65

86

84

89

82

98

138 182 180 173 160 158 166 172 174 167 151 133 114 107

68

131

59

72

77

100 105 140 175 179 178 176 173 171 172 169 170 170 154 128

99

98

84

136

80

87

92

89

102 152 173 176 180 176 174 178 176 172 173 172 167 129 102

92

82

139

81

93

89

97

137 155 165 170 175 180 177 168 176 170 171 170 169 134 105 101

80

141

83

88

81

123 144 155 168 164 176 167 153 138 174 173 168 166 138 122 115

90

85

137

76

80

117 122 138 159 156 160 143 141 153 131 155 158 149 143 135 130 105

77

70

128

86

107 126 132 139 157 153 157 134 133 146 131 130 152 142 140 123 107 100

82

72

126

102 119 124 130 131 137 144 130 132 132 134 125 126 115 136 133 118 101

93

99

74

121

109 119 118 126 134 136 151 120 113 128 131 129 106

95

129 128 105

91

94

97

99

117

79

85

86

92

103 123 139 150 149 147 139 144 152 150 147 140 130 121 113 104

94

123

Average = 123.0

(b)

50

k steps

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

1

-10

-9

-8

-7

-6

-5

-4

-3

-2

-1

0

1

2

3

4

5

6

7

8

9

10

avg

c

0

55

39

46

35

37

53

49

51

52

63

65

75

77

98

67

54

54

51

51

53

56

56

54

53

34

33

36

44

46

51

48

65

66

61

80

96

69

54

57

52

54

56

56

55

47

54

51

44

36

41

44

62

56

69

67

62

86

97

72

70

54

54

57

77

61

60

41

43

55

47

35

38

51

36

65

72

70

90

106

99

91

82

56

56

43

60

57

62

42

43

42

49

48

39

40

64

73

79

76

93

111 107

83

80

58

43

83

81

64

67

34

35

40

37

48

46

39

63

87

85

112

92

113 115

96

77

54

62

92

75

75

70

33

30

33

35

40

55

55

45

100 119 137 118 124 124

87

84

104 111

92

71

72

79

23

26

24

24

30

22

47

56

107 150 141 144 137 115 106 110 111

76

71

77

72

79

21

19

24

20

24

32

16

56

117 173 160 157 147 119 111

92

73

74

74

75

72

79

20

15

15

20

24

20

20

38

128 160 120 136 160

99

90

89

72

69

68

71

37

70

8

8

8

9

16

10

14

49

121 120

10

120 160 128

94

94

70

60

61

53

47

60

9

8

8

10

10

31

37

73

113 160 120 136 160 124

97

80

81

69

65

57

56

72

10

10

10

10

29

38

46

69

153 153 160 151 166 131 115 101

81

71

71

60

50

80

10

10

10

30

36

45

65

98

108 115 130 138 136 117 104 101

79

73

74

72

71

77

10

31

31

31

45

57

80

103

83

75

110 109 112

99

96

86

85

73

73

73

72

73

31

31

31

56

59

58

100

89

57

56

75

70

96

75

70

75

73

55

35

34

29

60

36

46

54

58

58

59

52

60

59

55

76

47

75

100

48

44

40

58

57

55

50

57

48

59

61

60

43

53

59

59

61

75

70

50

76

60

51

43

38

58

58

53

51

56

60

61

49

62

33

57

52

59

62

75

70

47

70

76

43

44

37

47

62

53

53

56

46

47

62

49

36

52

52

59

59

69

70

31

70

46

60

47

42

41

52

56

54

52

41

54

31

36

52

52

51

57

76

70

71

46

48

58

77

42

40

36

41

53

48

51

32

34

34

36

37

43

48

62

85

98

94

94

110

99

82

74

65

61

64

63

57

65

Average = 65.0

(c)

300

k steps

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel way of learning how to adapt policies in contextual RL. The work proposes to use \"adapter\" modules whose parameters are determined by a hypernetwork that solely focuses on the context variable.\n\nSTRENGTHS:\nThe paper is generally easy to follow and well written. Illustrations are well placed and utilized to communicate the content of the paper and to emphasize the concepts that are being discussed.\nThe work also addresses an important issue for RL, namely generalization, by combining existing ideas in clever ways to tackle this issue. I particularly commend the authors for the discussion of the limitations and the experiments that are conducted to explore the limitations. Overall, the presented method \"adapter\" seems well suited to be used for training more general RL agents. As such I believe the work is of high significance for the RL community and should inspire future works.\n\nOverall I think this paper is a clear fit for the NeurIPS (RL) community\n\nWEAKNESSES:\nThe presentation could be improved a bit. For example:\n* The title does not seem to fully fit with the work, or rather, it is overly general and gives the impression that a novel cRL formulation will be presented.\n* Similarly, the abstract only spends ~1/3 of the space discussing the \"Decision Adapter\" method\n* The wording in line 45 makes it seem like context could not be observed by a sensor. However, the robot mass example would be a sensable context.\n* The restriction on only changing dynamics and not changing rewards seems not really needed.\n* In line 115, Procgen is cited for examples where context unaware policies can work well, but I'd argue that the context is embedded in the visual state representation\n* Theorem 4.5 can only be understood when reading the Appendix. This is the case due to $\\mathcal{C}\\_{far}$ and $\\mathcal{C}_{close}$ never getting defined in Section 4 but only in the appendix.\n* Similarly, its easy to overlook why the inequalities in Definition 4.3 make sense, since it depends on the reward definition. \n* The theoretical results from Section 4 seem a bit arbitrary since they heavily depend on the construction of the example. It might make sense to move this part to the appendix and move some more of the empirical evaluation to the main paper.\n* The AER metric is introduced but \"AER\" does never show up again in the paper. Is it maybe labeled as \"Mean Eval. Reward\" in the plots?\n* It is not clear why AER can not be defined over a context set that is distinct from the training set\n* The citation to Benjamins is by now a TMLR publication\n\n\n\nThere is one more niche paper using cRL for learning dynamic configuration of AI planning algorithms that should be of interest to you:\n* Biedenkapp et al. (2022) https://rlplab.com/papers/biedenkapp-et-al-icaps2022wsprl.pdf (follow up to Speck et al. 2021 https://ojs.aaai.org/index.php/ICAPS/article/view/16008)\n\nThis work discusses how to treat state and context information separate and also discusses different architectural choices that one can make in this regard. I don't think this would form a reasonable baseline to compare against, however it feels like an appropriate reference. In particular, it feels like a good reference since the work considers 300+ noisy context features and how to learn with them, compared to the relatively few context features considered in the present work.\n\n\nWhile I listed many bullet points for potential improvement, I am very happy with the state of the paper and I think it should be accepted. I am even willing to increase my score if code will be released.\n\nQUESTIONS:\n* Will the code be made available?\n* Why is the pre-training phase (as done in NLP) not used for adapter? I would imagine that the hypernetwork would benefit from that and only need to \"fine-tune\" the generated weights to work well with the observed reward signals.\n* With many references to the work of Benjamins et al. 2022, why are not more environments from the proposed CARL benchmark used?\n* Intuitively it makes more sense to me to describe the Theoretical foundations in Section 4, using the optimality gap as defined by Benjamins et al. (2022). What is the advantage of using an Optimality ratio instead?\n* Are the same context distributions used in the cartpole example as done by Benjamins et al. (2022)?\n\nLIMITATIONS:\nLimitations are discussed thoroughly and experiments are provided that support the discussion on limitations.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of learning policies that can generalize to multiple contexts. The problem is formulated as a contextual MDP (CMDP). The paper starts by analyzing analytically the performance of policies that are not aware of the context in two different scenarios of a particular instance of CMDP. After that, a network architecture (decision adapter), which incorporates contextual information through a hypernetwork, is introduced. The architecture is empirically evaluated in three environments and its performance is compared against 4 baselines.\n\nSTRENGTHS:\nThe paper tackles an important problem in RL. The idea of using a hypernetwork to incorporate contextual information is sound and the empirical results are promising. The paper is very clear and easy to follow.\n\nWEAKNESSES:\nI believe that the paper's novelty is somewhat limited. While I found the solution of using a hypernetwork quite neat, I was slightly disappointed to discover that a similar architecture called cGate had already been proposed to address the same problem. The cGate architecture is mentioned for the first time in Section 5 and is not introduced as a related work.\n\nThe authors argue that the adapter architecture is a more powerful generalization of cGate as it can also capture non-linear relationships between states and contexts. This is confirmed by the empirical results obtained in the ODE domain. However, the results of cGate on the other two environments (CartPole and Mujoco Ant) are not reported. Moreover, the ODE domain seems to be specially designed to make cGate fail, as the dynamics are given by an nth-degree polynomial (equation Line 229).\n\nFinally, I found the example provided in Section 4 to be very helpful for understanding. However, in my view, framing the analysis o as a theorem is not appropriate in this case, since the applicability of the results seems limited to this specific instance of the problem. Typically, a theorem should offer broader insights into the overarching problem at hand.\n\nQUESTIONS:\n1. Could the authors clarify why the results of cGate in CartPole and Mujoco Ant are not provided?\n\n2. The paper attributes cGate to [13]. However, I did not find any reference to cGate in [13]. Could the authors clarify this?\n\n3. Line 193 says *\"For this and the next section, we consider only the Concat and Unaware models as baslines\"*. However, the results for the unaware policy are not reported in Figures 4 and 5.\n\nLIMITATIONS:\nThe limitations of the approach are discussed extensively in Section 8 and Appendix G.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies how to better incorporate context (such as MDP-specific parameters) in context-conditioned robot policies. Traditionally, this is done simply by concatenating the context vector to the state vector and processing both inputs simultaneously with the policy network. This work proposes a context-conditioned hypernetwork to output the weights for an adapter module, which adds a residual value to the policy’s intermediate representation. Along with other proof-of-concept tasks, the paper evaluates on cartpole (varying pole length) and mujoco ant (varying mass) and finds that the proposed method is more robust to distractor dimensions but does not necessarily perform better than naive concatenation methods.\n\nSTRENGTHS:\nThe authors study a problem that is important in multitask learning/generalization such as with respect to language/goal-conditioned policies.\n\nThe paper is well-written and proves some interesting theoretical results. Empirically, experiments test generalization in a meaningful way. For instance, the evaluated mass ranges are not within the training ranges in the ant environment, which is a challenging problem in robotics.\n\nWEAKNESSES:\n(1) Section 4 talks about performance guarantees for a context-unware policy. Why is this important to the rest of the paper, which is about how to best incorporate context into a policy for dynamics generalization? Furthermore, I don’t see the link between Lines 154-163 (talking about how context-dependent rewards are equivalent to context-dependent dynamics in a context of matrix rotation example) to the rest of the section or the rest of the paper.\n\n(2) The abstract promises to “establish conditions for when it is necessary to incorporate context,” but when reading the text, I did not see any section of the paper that brought this out explicitly.\n\n(3) The work would really benefit from architectural ablations, especially when the abstract said the paper would “examine how to best leverage this external context information to improve generalization.” Hypernetworks are not a common architectural component in context-conditioning, so were alternatives not involving hypernetworks considered? For instance, it would be good to compare with the following architectural results:\n(i) The proposed approach, with no hypernetwork, where the adapter module (a simple MLP) only takes the context as input.\n(ii) Something like cGate, but with an elementwise product with the context embeddings at every layer, instead of just at one layer.\n\n(4) Figures 4 and 5, which contain the most difficult domains examined by the paper (ant and cartpole), show that in the 0 distractor dimension case, naive concatenation performs roughly the same as the proposed method (adapter). This means that in this case, naive concatenation is preferable due to its simplicity and training speed relative to the proposed adapter. It is only in the differential equations domain, a non-control task, where adapter clearly outperforms naive concatenation.\n\nQUESTIONS:\n(1) How might this framework be improved to incorporate multiple contexts at the same time? For instance, both a goal image and language instruction for a given task. If it is known that some context modality is more beneficial on average to inducing strong task performance than some other context modality, how might we handle that with adapters and hypernetworks?\n\n(2) I don’t understand how Theorem 4.5 relates to the rest of the paper, or even how it guided the experimentation toward the proposed adapter + hypernetwork architecture.\n\n(3) How would the proposed method perform in image-based domains where the main policy network is a CNN or transformer? It seems like the main results are working with an MLP operating on robot state. My main concerns are that this work may be useful primarily for simpler, older architectures and might not prove as helpful for transformer-based robotics policies that are moving toward cross-attentional context conditioning (such as VIMA [Jiang et al 2023]).\n\n(4) Perhaps include a citation of FiLM (Perez et al 2017), since it is a widely used context conditioning method and does a sort of context-conditioned “adapter” with a residual connection as well, which is similar to the proposed method but does not have a hypernetwork. A comparison would be helpful too.\n\n(5) Most of the crux of the experimental section hinges on the argument that the adapter is a better context conditioning architecture because it is more robust to distractor context variables. However, the distractor variables were set to some pretty contrived values--all to 1 during training and 0 during testing. How were these values chosen? Why not just gaussian noise during both training and testing, which seems more likely for a model to encounter, in which case concat will probably be as good as an adapter at filtering out the distractor dimensions?\n\nLIMITATIONS:\nAuthors were upfront of the limitations of the adapter to noisy contexts and its tendency to overfit when trained on narrow context ranges and had a very detailed limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an architecture that uses a hypernetwork to learn network parameters for a context-dependent policy adapter. The authors discuss an illustrative example that emphasizes in what cases a universal (context-dependent) policy is necessary, and in which cases a non-context-conditional (\"unaware\") policy is sufficient. In the experiments, they compare their adapter approach with a context-unaware policy, a policy that concatenates context information with state and processes them equally, and other approaches using a linear policy adapter (FLAP and cGate).\n\nSTRENGTHS:\nThe idea to hyper-learn a nonlinear context-dependent adapter as the final few layers of a context-general policy is original and makes sense. The paper is well written, and experiments are quite thorough. The relatively detailed ablation experiments, some of which are found in the appendix, are quite interesting and a strength of the paper, for example the study of nuisance context parameters in section 7.2. I also think that the distinction between extrapolating and interpolating generalization is important and rarely seen (presumably because this distinction is usually hard to make in more complex environments), and interesting enough to maybe move it to the main part. General policies are an important and active research field, and the paper makes a significant contribution.\n\nWEAKNESSES:\nI appreciate the illustrative example for unaware vs aware policies in section 4, but I feel like this is quite intuitive for many readers, and could potentially be shortened a bit in favor of more experimental results in section 5. Also, I was a little confused by the section title being \"Theoretical Foundations\".\n\nI think that the related work section should probably discuss recent advances in foundation-model-style generalist agents (e.g. https://arxiv.org/abs/2203.11931, https://arxiv.org/abs/2305.10912, https://arxiv.org/abs/2301.09816, https://arxiv.org/pdf/2306.11706.pdf) as well. The authors briefly touch on \"learning some unsupervised representation of the problem and inferring a latent context from a sequence of environment observations\", which is similar in spirit to this line of work, but then only cite one reference. Considering that this field has been quite active lately, it probably would make sense to discuss this briefly.\n\nGenerally, I think the paper would benefit from experiments in a larger and more diverse number of experiments. However, I think this is at least partly compensated by the fact that the authors report relatively detailed results for the two (or three) environments they consider.\n\nMinor comments:\n- The shaded areas in Fig 3 and elsewhere are very hard to read\n\nQUESTIONS:\n- In 6.1: Would it not be possible to calculate the average evaluation reward only on an entirely unseen set?\n\nLIMITATIONS:\nThe authors discuss limitations in quite some detail in section 8, and underpin this with additional experiments in the appendix. This is a strength of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}