Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policies
Authors: Michael Beukman
Devon Jarvis
Richard Klein
Steven James
Benjamin Rosman
Venue: NeurIPS 2023 poster (2023)
Keywords: Deep Reinforment Learning
Contextual Markov Decision Process
Neural Network Architecture
TL;DR: We introduce the decision adapter, an improved way to incorporate context into an agent's policy in contextual reinforcement learning.

Abstract: While reinforcement learning has achieved remarkable successes in several domains, its real-world application is limited due to many methods failing to generalise to unfamiliar conditions. In this work, we consider the problem of generalising to new transition dynamics, corresponding to cases in which the environment's response to the agent's actions differs. For example, the gravitational force exerted on a robot depends on its mass and changes the robot's mobility. Consequently, in such cases, it is necessary to condition an agent's actions on extrinsic state information and pertinent contextual information reflecting how the environment responds. While the need for context-sensitive policies has been established, the manner in which context is incorporated architecturally has received less attention. Thus, in this work, we present an investigation into how context information should be incorporated into behaviour learning to improve generalisation.  To this end, we introduce a neural network architecture, the Decision Adapter, which generates the weights of an adapter module and conditions the behaviour of an agent on the context information. We show that the Decision Adapter is a useful generalisation of a previously proposed architecture and empirically demonstrate that it results in superior generalisation performance compared to previous approaches in several environments. Beyond this, the Decision Adapter is more robust to irrelevant distractor variables than several alternative methods.

Dynamies Generalisation in Reinforcement Learning via Adaptive Context-Aware Policies
Michael Beukman 1.2+
Devon Jarvis 1.3
Richard Klein
Steven James'
Benjamin Rosman'
University of the Witwatersrand University of Oxford
University College London
Abstract
While reinforcement learning has achieved remarkable successes in several do- mains, its real-world application is limited due to many methods failing to gener- alise to unfamiliar conditions. In this work, we consider the problem of generalising to new transition dynamics, corresponding to cases in which the environment' s response to the agent's 's actions differs. For example, the gravitational force exerted on a robot depends on its mass and changes the robot's ' mobility. Consequently, in such cases, it is necessary to condition agent's 's actions extrinsic state information and pertinent contextual information reflecting how the environment responds. While the need for sonsexe-sensitive policies has been established, the manner in which context is incorporated architecturally has received less attention. Thus, in this work, we present an investigation into how context information should be incorporated into behaviour learning to improve generalisation. To this end, we introduce a neural network architecture, the Decision Adapter, which generates the weights of an adapter module and conditions the behaviour of an agent on the context information. We show that the Decision Adapter is useful generalisation of previously proposed architecture and empirically demonstrate that it results in superior generalisation performance compared to previous approaches several environments. Beyond this, the Decision Adapter is more robust to irrelevant distractor variables than several alternative methods.
Introduction
Reinforcement learning (RL) is powerful tool, and has displayed recent success in several settings [1- 4]. However, despite its potential, RI. faces significant challenge: generalisation. Current agents and algorithms struggle perform well beyond the narrow settings in which they trained [5-7], which is a major limitation that hinders the practical application of RL. For RL algorithms to be effective in real-world scenarios, they must be capable of adapting to changes the environment and performing well in settings that are different, but related, to those they trained on [7].
To illustrate this point, consider legged robot trained to walk perform well in these conditions, will likely struggle in other reasonable settings, such as walking asphalt when carrying cargo. When walking another surface, the friction characteristics would differ from what the robot trained on, which may cause it to slip. When carrying additional weight, the robot is heavier, with potentially different center of mass. This may mean that the robot must exert more force to lift its legs or be more careful to prevent falling over. This limitation is far from ideal; we want agents to excel in tasks with minor differences in dynamics (the effect of the agent's 's actions) without requiring additional training.
tiled floor. While the agent would
"Correspondence to mbeukman@robot ox. ac uk. Work done while the University of the Witwatersrand, now at the University of Oxford.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Several methods have been proposed to address this problem, including training single agent in various settings in the hope that learns generalisable and robust behaviour [3, 8-11]. However, this approach is not without its drawbacks, as it fails when variations are too large for a single behaviour to perform well in all settings [12, 13]. For example, using the same behaviour when walking unencumbered and under load may lead to suboptimal performance in one or both cases.
This observation has led to other methods that instead use contextual information to choose actions, allowing the agent to adapt its behaviour to the setting it is in. For instance, if the agent knows when it is walking unencumbered vs. carrying something heavy, it can perform slightly differently in each case [14]. The context differs conceptually from the state information, which generally includes data that the robot observes using its sensors, such as its joint angles, acceleration, and camera views. The crucial distinction between context and state is that context changes at a longer timescale compared to state; for instance, the floor friction and additional mass remain the same for a long time, whereas the robot's is joint angles change after every action [15]. Currently, the predominant approach is to use context as part of the state information, ignoring the conceptual difference between these two as- pects [16-18]. However, this may lead to the agent confounding the context and state, and thus exhibit worse generalisation [15]. This problem can be exacerbated real-world settings, where identifying which variables affect the dynamics may be challenging [18]. This may lead to several irrelevant vari- ables being added to the context, further expanding the state space and making learning more difficult.
To address this problem. we introduce an approach to incorporating context into RL that leads to improved generalisation performance. Our method separates the context and state, allowing the agent to decide how to process the state information based on the context, thus adapting its behaviour to the setting it is in. Our experimental results demonstrate the effectiveness of our approach in improving generalisation compared to other methods. We show that our approach outperforms (1)
not incorporating context information at all; (2) simply concatenating context and state; and (3) competitive baselines. We also demonstrate that our approach is more robust to irrelevant distractor variables than the concatenation-based approach across multiple domains. We further provide a theoretical characterisation of problems where incorporating context is necessary and empirically demonstrate that a context-unane unaware method performs poorly.
2 Background
Reinforcement learning problems are frequently modelled using a Markov Decision Process (MDP) [19, 20J. An MDP is defined by a tuple AS,A, R,,,) where S is the set of states, A is the set of actions. T : S x A S > [0, 1] is the transition function, where T (sl|s, a) specifies the probability of ending up in a certain state s' after starting in another state and performing a specific action a. R S x A x S + R is the reward function, where R(st, ae, st+1) Re+1 specifies the re- ward obtained from exccuting action a, in a state s, and y E [0, 1] is the environment discount factor, specifying how short-term and long-term rewards should be weighted. The goal in reinforcement learning is to find a policy π. : S + A that maximises the return G. = Ei=o?" RRR+&411 [21].
We consider the Contextual Markov Decision Process (CMDP) [15] formalism, which is defined by a tuple (C, A,
where C is the context space, S and A are the state and action spaces respectively, and M' is a function that maps a context E C to an MDP M = (S,A,T= Re, 7). A CMDP thus defines a family of MDPs, that all share an action and state space, but the transition (T") and reward (Ra) functions differ depending contexts, such that we can generalise well to another set of evaluation contexts. We focus solely on generalising over changing dynamics and therefore fix the reward function, i.c., R R, Vc e C.
the context. Our goal is to train a particular set of
3 Related Work
Generalisation in RL is widely studied problem, with one broad class of approaches focusing on robustness. Here, single policy is trained to be robust changes that may occur during testing, ie., to perform well without failing catastrophically [3, 6, 8, 22-29]. Many of these approaches are successful in this goal and can generalise well when faced with small perturbations. However, using one policy for multiple environments (or contexts) may lead to this policy being conservative, as it cannot exploit the specifics of the current environment [12]. Furthermore, these policies are limited
We publicly release code at https: http//ggttttbccoot http:/hhhee..euuaaa////eeaaaaaaa
2

---

to performing the same action when faced with the same state, regardless of the setting. This may be suboptimal, especially when the gap between the different settings widens [13. 30].
This observation motivates the next class of methods, context-adaptive approaches. These techniques often use a version of the CMDP [15] formalism, and use the context to inform the agent's choice of action. This allows agents to exhibit different behaviours in different settings, which may be necessary to achieve optimality or generalisation when faced with large variations [13]. There are several ways to obtain this context, such as assuming the ground truth context [13], using supervised learning to approximate it during evaluation [14, 31], or learning some unsupervised representation of the problem and inferring a latent context from a sequence of environment observations [18, 32-34].
Despite the progress in inferring context, how best to incorporate context has received less attention. The most prevalent approach is to simply concatenate the context to the state, and use this directly as input the policy [16-18, 35-38], which is trained using methods such as Soft Actor-Critic [39] or Proximal Policy Optimisation [40]. This ignores the conceptual difference between the state and context, and may lead to worse generalisation [15]. This approach may also be limited when there are many more context features compared to state features [41]. While concatenation is the most common, some other approaches have also been proposed. For instance, Biedenkapp et al. [41] learn separate representations for context and state, and then concatenate the representations near the end of the network. They also use another baseline that concatenates a static embedding of the context features to the state. Both of these techniques generalise better than the concatenation approach. Another method, FLAP [42], aims to improve generalisation by learning a shared representation across tasks, which is then processed by task-specific adapter in the form of a linear layer. Given state the policy network outputs features ((s) E Red, which are shared across tasks. The final action is chosen according to W;O(s) + b, with a unique weight matrix W; and bias vector b, which are learned separately for each task. Concurrently, a supervised learning model is trained to map between transition tuples (st, dz, 1,+1,, 11+1) and these learned weights. At test time, this supervised model generates the weights, while the policy network remains fixed. FLAP generalises better to out-of-distribution tasks compared to other meta-learning approaches [36, 43, 44). However, this approach requires separate head for each task, which may scale poorly if we have many tasks or see each task only a few times.
One promising approach that has been used in other fields is Feature-wise Linear Modulation (FiLM) [45, 46). In this method, features in one modality (e.g., natural language text) linearly modulate the neural-network features obtained from another modality (e.g., visual images). An example of this would be processing visual scene, modulated by different natural language questions which would lead to a different final answer. In RL, Benjamins et al. [13] introduce cGate,' which follows a similar procedure to FiLM; in particular, the neural network policy receives the state as input and outputs an action. Before the final layer, however, the context is first transformed by separate network, and then used to modulate the state-based features. This approach showed promise, but restricted the context features' effect on the state- baste features to be linear.
Another research area that has garnered more attention recently is learning general foundation models for control [48-50]. Inspired by recent work in fields such as natural language processing [51] and computer vision [52], these models aim to provide a general foundation that can be easily fine-tuned for particular tasks of interest. For instance, Gupta et al. [53] learns general controller for several different robot morphologies. The robot's configuration is encoded and passed to transformer [54], allowing one model to control wide variety of morphologies. While these works directly train policies, Schubert et al. [55] instead train a dynamics model, and use a standard planning technique- model-predictive control [56-58] choose actions. They find that this approach achieves better zero-shot generalisation compared to directly learning a policy. On a different note, Sun et al. [59] pre-train foundation model that predicts observations and actions. Then, during fine- tuning, they train a policy to perform a particular task using the pre-trained representations, which enables the use of either imitation learning or reinforcement learning in the downstream fine-tuning stage.
Finally, much work in renfareinforceemnt learning [60] relates to the problem of generalisation. Meta-learning approaches generally aim to use multiple tasks during training to learn how to learn; this knowledge can then be used during testing to rapidly adapt to a new task [43, 60-67]. J. In this problem setting, Beck et al. [68] use a recurrent model to encode the current task, and generate the weights of the agent policy based on the task encoding. Sarafian et al. [69] take similar approach, but
2 Beujamins et al. [13] introduced the cGate method, and then published a revised version of the paper not containing cGate-unde the same name. We therefore cite both versions [13, 47J.
3

---

instead generate the weights using the environment state, and processing the encoded context using the generated network. Another approach, MAML [44] aims learn the weights of a neural network such that, for any new task, performing only few gradient updates would lead to high-performing task-specific network. While meta-learning can lead to highly adaptable agents, many methods require multiple episodes of experience in the target domain [36, 44], and may therefore be less well-suited to the zero-shot setting we consider [7].
Theoretical Intuitions
Now, while only context- contixioco policy is guaranteed be optimal on a general CMDP [47], in many cases simply training an unaware policy on a large number of diverse environments can lead to impressive empirical generalisation [5]. this section, we characterise and unify these two observations. In particular, for a specific problem setting-wher where the context defines the target location an agent must travel to-we find that:
(i) For some context sets, an unaware policy will perform arbitrarily poorly on average, due to it being forced to take the same action in the same state, regardless of context.
(ii) However, for other context sets-where the different contexts are similar enough an unaware policy can perform well on average, as it can simultaneously solve each task.
We defer the formal statement of this theorem and its proof to Appendix A, but here we briefly provide some intuition. Overall, our results rely on the fact that a context-unaware agent must perform the same action in the same state, regardless of context. Therefore, in setting where we have different goals (indicated by the context), and making progress on one goal leads to making negative progress on another, context-unawars agent cannot simultaneously perform well on both of these contexts. By contrast, if the goals are aligned in the sense that making progress on one leads to progress on another, a context-unawan agent can perform well on average. A context-aware agent, however, can always choose the correct action in either scenario.
(a) Non-overlapping
(b) Overlapping
Figure 1: An illustration of the two cases discussed above, with the agent's 's start state being the origin. The agent will receive a reward if it enters the circle corresponding to the current context and zero reward otherwise. each goal in sequence, whereas a context-awar policy (green) can go directly to the correct goal. In (b), the goals overlap, so an unaware policy can go directly to the joint intersection point the goals, leading to only slightly suboptimal value function compared to the optimal context-aware policy.
(a) there no overlap between the goals, so an unaware policy (red) must visit
Fig. visually illustrates these two cases and summarises the implications. In Fig. 1a, if there is no overlap between goals, an unaware policy is forced to visit each one in sequence. This is because must perform the same action in the same state, regardless of the context, meaning that it cannot always travel to the correct goal. This policy will therefore have low value function compared to the optimal context-aware policy. By contrast, in Fig. 1b, single unaware policy can directly go to the intersection of all the goals, leading to only slightly less return than the optimal context-aware

---

policy. While the formal result in Appendix A merely shows the existence of some problems with these properties, we believe the concept generalises to more complex settings. In particular, the problem exhibits some structure where the same policy can simultaneously make progress on each context, an unaware model can perform well. However, if making progress on one goal leads the agent moving away from another, context-unaware policy can at best perform arbitrarily worse than a context-aware one, and incorporating context is crucial. We empirically substantiate these findings in Section 7 and formally consider this example in Appendix A. Finally, Appendix B for more details about how these theoretical results connect back to our empirical observations.
The Decision Adapter
An adapter is (usually small) neural network with the same input and output dimensions. This network can be inserted between the layers of an existing primary network to change its behaviour in some way. These modules have
much use in natural language processing (NLP) [70-74] and computer vision, usually for parameter-efficient adaptation [75-81]. In these settings, one adapter usually trained per task, and the base model remains constant across tasks. Our method is inspired by the use of adapters in these fields, with a few core differences.
First, since aim is zero-shot generalisation, having separate adapter module for each task (corresponding to context in our case) provides no clear means for generalisation. Relatedly, since tasks are generally discrete in NLP, but allow for continuous contexts, this approach would be problematic. Furthermore, the number of learnable parameters increases with the number of training contexts, presenting computational limitation to the scalability. To cireumvent these problems, we use a hypernetwork [82] to generate the weights of the adapter module based on the context. This allows the agent to modify its behaviour for each context by having a shared feature-extractor network which is modulated by the context-aware adapter. Moreover, since the hypernetwork is shared, experience gained in one context could transfer another [72]. Second, instead of having separate pre-training and adaptation phases as is common in NLP [76], we do not alter the standard RL training procedure at all. We change only the architecture and train normally -allowiig for the easy integration of our approach into standard RL libraries. Finally, our method is agnostic to the exact RL algorithm and can be applied to most aff-the-shelf algorithms without much additional effort.
Algorithm 1 Decision Adapter Changes to the stan- dard forward pass in blue.
Adapter
1: procedure ADAPTERFORWARDGG E C,CEC)
2: 261
3: for E (1,2,...,,u do
4: if A, F null then
5:
0 H.(c)
// Generate Weights
Hypemnetoork
6:
x. = (,(x;((A))
// Forward Pass
Generated Weights
7:
2; D4 x¹d
// Skip connection
8:
end if
9:
Fi+1 (x(x))
10:
end for
11:
return Fn+1
12:
end procedure
Figure 2: An illustration of our network architecture, the Decision Adapter, alongside pscudocode for the forward pass. The bottom network is the hypernetwork, which generates the magenta weights for the top, primary, network. The light grey nodes in the top network are added by the adapter.
To formally describe our approach (illustrated in Fig. 2), suppose we have state S, context e E C, and standard n-layer fully-connected neural network architecture that predicts our action a.
We add adapter module A, between layers Li-1 and L. This adapter module is a multilayer neural network A; : R4 + 4 consisting of parameters 0% E R.P.? These parameters are generated by the corresponding context-conditioned
Here ri+1 L,(x,) a; (W,r;;
and xx+
d, is the output dimension of layer Li-1 and Pt is the number of parameters in the adapter.
5

---

hypernetwork H, : C - R. (line 5 in Algorithm 1) and reshaped to form multiple weight matrices. Then, before we process 2; using layer L, we pass x; through the adapter module A, (line 6) defined by the generated weights and biases to obtain x = Ax, (0')). Furthermore, akin to the application of adapters in other fields [75, 76], we add skip connection, by setting æ; = 24 + 2 (line 7). This updated is then the input to layer I, (line 9). Additionally, we could have multiple adapter modules in the primary network, with at most one module between any two consecutive layers (lines and 4).
Finally, we note that the Decision Adapter model can learn to implement the same function as cGate [13], but theoretically more powerful. While cGate uses linear elementwise product between the context and state features, our model uses hypemeewwok to generate the weights of a nonlinear adapter module. Our hypernetwork is general enough be able to recover this elementwise product, but is not constrained to do so. See Appendix C for more formal treatment of this point.
6 Experimental Setup
6.1 Metrics
Our evaluation strategy is as follows: We take the trained model, and compute the av- erage of the total episode reward over 5 episodes each evaluation context to obtain a collection of contexts C' and corresponding rewards R. We then calculate the Average Evaluation Reward (AER): AER
Crees Grrin Sema= R(c)de, where Cmin and Cmoz the minimum and maximum context values in C respectively, and R(c) the reward obtained on context This metric is high when the agent performs well across a wide range of contexts, and thus corresponds to generalisation performance. However, since we generally have Ctrein Cevalvation this metric also incorporates how well the agent performs on the training contexts." Despite this, it still a useful metric for generalisation, as our training context set typically contains only a handful of contexts, with most of the evaluation contexts being unseen.
6.2 Baselines
We use Soft-Actor- Actt--cco--Criicc [39] for all methods, to isolate and fairly compare the network architectures. We aim to have equal number of learnable parameters for each method and adjust the number of hidden nodes to achieve this. We use the following baselines, with more details Appendix D:
Unaware
This model simply ignores the contextual information and just uses the state. This approach allows to evaluate how well method that does not incorporate context performs.
Concat
This method forms a new, augmented state space S' S C, and then gives the model the concatenation [s; c] of the state and context [1 [16-18, 37j. This baseline allows us to compare our method the current standard approach of using contextual information.
cGate
cGate [13] learns separate state (o(s) (6)s) and context (g(c)) encoders, and predicts the action
= f ()((s @ g(c)), where f is the learned policy and @ is the elementwise product.
FLAP
FLAP [42] learns a shared state representation across tasks, which is then processed by task -sec-fpec linear layer that is generated by conditioning on the context.
Our Adapter configuration more fully described in Appendix D.3, and Appendix contains ablation experiments comparing the performance of different hyperparameter settings.
6.3 Environments
6.3.1
ODE
This environment is described by an ordinary differential equation (ODE), parametrised by variables making up the context. The dynamics equation is xt+11 = x2, + di, with coa+tjat +22a +..., truncated at some cN The episode terminates after 200 timesteps, with the reward function:
*The AER can be computed over only the unseen evaluation contexts, but we include the training contexts to obtain a more general measure of performance. However, the difference between the two versions is minor, see Appendix F.4.

---

if [r| 0.05
1/
if |g|
< 0.1
33 if [-| < 0.2 0 otherwise
R₁
1 if [
< 0.5
20 if |
<
This incentivises the agent to give control to keep the state close to x O. The context in this environment is c = (ca....... Cn-1] IR. The action space is two-dimensional, continuous and bounded, i.e., (0, d1 1- The action is interpreted as complex number, = 40 + @,ú, to allow the dynamics equation to be solvable, even normally unsolvable cases such as 2 = a². Here, restricting to I would result in i 2 and an unsolvable system for initial conditions 2o 0. We keep only the real part of the updated state and clip it to always fall between -20 and 20.
The ODE acts as conceptually simple environment where we can arbitrarily scale both the number (i.e., n) and magnitude (i.e., (cs)) of the contexts and precisely measure the effects. Furthermore, since many dynamical systems can be modelled using differential equations [83-85], the ODE can be considered a distilled version of these. Finally, context necessary to perform well in this environment, making it a good benchmark. See Appendix D for more details.
6.3.2 CartPole
CartPole [86] is a task where an agent must control the movement of a cart to balance a pole vertically placed upon it. The observation space is 4-dimensional real vector containing the cart's position and velocity å as well as the pole's angle and angular velocity 9. We use a continuous action space where a E [-1, 1] corresponds to the force applied to the cart (where negative values push the cart to the left and positive values to the right). The reward function is +1 for each step that the pole upright. An episode terminates when the pole is tilted too far off-center, the cart's position is outside the allowable bounds or the number of timesteps is greater than 500. We follow prior work and consider the variables Gravity, Cart Mass, Pole Length, Pole Mass and Force Magnitude tude collectively as the context [13], even only a subset of variables change, Here, we change only the pole length and evaluate how susceptible each model is to distractor variables. In particular, during training, we also add & additional dimensions to the context, each with constant value of 1. Then, during evaluation, accurately identifying or inferring the pertinent context variables may be challenging [18, 87], and lead to several irrelevant variables. Overall, this means that the context is represented as 5 + k- dimensional vector. This environment is useful as it (1) is a simple setting with models that can be trained quickly, and (2) still has desirable aspects, namely way perturb some underlying variables to change the dynamics of the environment. Furthermore, prior work has shown that even standard, unaware RL algorithms can generalise well in this environment [88] (in the absence of distractor context variables), which allows us to investigate the benefits of context and detriment of distractors.
set these values to O. This may occur during real- world example, where
6.3.3 Mujoco Ant
As a more complex and high- sigh-dimensional problem, we consider Ant from the Mujoco suite of environments [89, 90]. Here, the task is to control 4-legged robot such that it walks. The observation space consists of the robot's joint angles and velocities, as well as contact forces and torques applied to each link, totalling 111 dimensions. The action space = [-1, 1] represents the torque applied to each of the joints. Furthermore, each episode is terminated after 1000 timesteps, with a positive reward for not having fallen over each step and moving forward. The reward function also penalises large control forces. Here, the mass is the context variable, which is 5 by default. We train on the set (5, 35, 75) and evaluate on 200 evenly spaced points between 0.5 and 100.
Results
7.1 Generalisation Performance
In this section, we consider the ODE environment, with training context set of (-5, 1,, and 300k training timesteps. measure generalisation, we use an evaluation context range consisting of 201 equally spaced points between - 10 and 10 (inclusive). In the left pane of Fig. 3, we plot the average performance across the entire evaluation range (as discussed in Section 6. at different points during training.
7

---

Firstly, we can see that the Unaware model fails to generalise well, since the optimal actions in the same state for different contexts may be completely different. This result substantiates our theoretical analysis in Section 4, highlighting the importance of using context in this domain. Concat, cGate and FLAP perform reasonably well, but our Decision Adapter outperforms all methods and converges rapidly. In Appendix F, we plot separate subsets of the evaluation range: Train, Interpolation and Extrapolation. corresponding to the contexts the training set, the contexts within the region [] and the contexts outside the convex hull of the training contexts, containing e E [-100 5) U (5, 10), respectively. There we find that most methods, with the exception of the Unaware model, perform well on the training contexts, and the Adapter outperforms all methods when extrapolating. These results also extend to the multidimennional context case, shown in Fig. (right). We train the context set given by {1,0, -1] 2 U {5, 0, -5] with the context (0,0) being omitted due to being unsolvable." The evaluation range consists of the cartesian product A2, where A contains 21 equalty-spaced points between 10 and 10. The Adapter outperforms all methods, with the Concat model coming second. FLAP and the Unaware models struggle in this case.
Context
Context
200
150
AER
50
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Training Time (Steps)
1e5
Unaware
Concat
Adapter
cGate
FLAP
Optimal
Figure 3: The average reward over the evaluation range various points during training for the ODE domain. Mean performance shown, with standard deviation (across seeds) shaded. The left pane shows the one-dimensional context results, while the right pane shows the two-dimensional results.
7.2 Robustness to Distractors
For this and the next section, we consider the Concat and cGate models baselines. The reasons for this are that (1) the Concat model performed comparably to or better than the other baselines in the previous sections, making it representative example of the context-awarg approaches; and (2) the Concat model is currently a very prevalent way of incorporating context [16-18, 35-38]; and (3) cGate is the closest approach to our work the current literature.
The results for CartPole are shown in Fig. 4. When there are no distractor variables, the Adapter, Concat and cGate models perform comparably. In this case, each architecture achieves near the maximum possible reward for the domain. Further, we see little effect after adding just one con- founding context variable. However, as we add additional distractor variables, the Concat and cGate models" performances drop significantly, whereas the Adapter's performance remains relatively stable. Strikingly. the Adapter architecture trained with 100 distractor context variables still able to perform well on this domain and significantly outperforms the Concat model with significantly fewer (just 20) distractor variables. This demonstrates that, given only useful context, concatenating state and context is reasonable approach in some environments. However, this is a strong assumption in many practical cases where we may be uncertain about which context variables are necessary. In such cases, the consequences of ignoring the conceptual differences between state and context are catastrophic and it is necessary to use the Adapter architecture. We find a similar result the ODE, which we show in Appendix F.5.
As an extension the previous experiment, here we consider using non-fixed distractor variables. i.e., sampling them from Gaussian distribution with different means. The intuition behind this experiment similar before, but allows for small variations within training and testing. particular, we use 0.2 and a mean of either 0 or 1. In this experiment, a new distractor context is sampled at the start of every episode. During training this sampled from (V11, 0.2) and during evaluation
Here, for S, S² S S ((a,b))a S} denotes the cartesian product of S with itself.

---

Concat
Adapter
cGate
500
400
300
200
100
4
2
8
Training Time (Steps)
Number of distractor dimensions
20
Optimal Reward
Figure 4: Showing the sensitivity of the (left) Concat, (middle) Adapter and (right) cGate models to distractor context variables in CartPole. The mean and standard deviation over 16 seeds are shown.
sample from (0, 0.2). In CartPole, the results in Fig. 5 show that the Adapter is still more robust changing distractor variables compared to either Concat or cGate. See Appendix F.6 for more details, results on the ODE domain, and additional experiments when the mean of the Gaussian does change between training and testing.
Concat
Adapter
cGate
500
400
200
100
2
1e5
Training Time (Steps)
Number of distractor dimensions
20
100
Optimal Reward
Figure 5: The sensitivity of Concat, Adapter and cGate to distractor context variables sampled
from Gaussian dnstribution-in CartPole. We show the and standard deviation over 16 seeds.
7.3 High-Dimensione Control
We next consider more complex and challenging continuous-control environment, that of the Mujoco Ant. We run the same experiment as CartPole, where we add distractor variables, with no effect on the dynamics, and different values between training and testing. These results are shown in Fig. 6. Overall, we observe similar result to that CartPole- the Concat and cGate models are significantly less robust to having irrelevant distractor variables. By contrast, the Adapter consistently performs well regardless of how many distractor dimensions are added.
Limitations
While our Adapter model performs well empirically, does have some limitations, which we sum- marise here and expand on in Appendix G. First, we find that using an incorrect and noisy context (Appendix G.1) generally resulted in the Adapter model performing worse, particularly as in- creased the level of noise. However, the Adapter model still performs comparably to the Concat model this case. Second, we generally normalise the contexts before we pass them the models, relative to the maximum context encountered during training. While our Adapter performs well when the con- text normalisation is incorrect by a factor of 2 or its performance does suffer when normalisation value is orders of magnitudes too small, leading to very large contexts being input into the model (see Appendix G.2 for further discussion). Third, when training on a context range that is too narrow (see Appendix G.3) or does not exhibit sufficient variation, model is susceptible to overfitting, which

---

Concat
Adapter
cGate
AER
2000
-2000
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
1e6
Training Time (Steps)
Number of distractor dimensions
20
100
Figure 6: Showing the sensitivity of the (left) Concat, (middle) Adapter and (right) cGate models to distractor context variables in Ant. The mean and standard deviation over 16 seeds are shown.
leads worse generalisation. Even if our model does not overfit, overly narrow context range would also lead to poor generalisation. Thus, ensuring that agents are trained in sufficiently diverse environments is still important. Fourth, while assumed access the ground-truth context, may not always be available in practice [31]] Therefore, integrating the Decision Adapter into existing context inference methods [18, 33] is promising avenue for future work. A final limitation of Adapter is its longer wall-clock training time compared to other methods (given the same training timesteps and a similar number of parameters). This is likely caused by the hypernetwork that use. However, we did not utilise strategies such as caching the generated weights for a particular context (which would be possible since context remains constant for episode), more effectively batching the forward passes through the Adapter which could ameliorate this issue in practice.
Beyond addressing these limitations, there are several logical directions for future work. First, only consider state and context features to be numerical vectors; therefore, extending our approach to different modalities-for instance, image observations and natural language context-would be a natural extension. In principle, our Adapter model could be directly applied in this setting without any fundamental modifications. One option would be explore adding the hypernetwork directly into the modality-specific model (e.g., convolutional neural network in the case of images), and the alternative would be to integrate the adapter module in the final (fully-connected) layers of the RL policy. Second, while we focused on the problem of zero-shot generalisation, our approach may also large be suitable for efficient few-shot learning. In particular, adapters in NLP are often added pre- pre-trained model and then fine-tuned while freezing the primary model [76]. Using the Decision Adapter in this paradigm is promising direction for future work, especially in cases where the difference between training and testing is so large that we cannot expect effective zero-shot generali- sation. Combining our work with fine-tuning-base [91-93] or meta-learning [36, 60, 68, 69, 94-96] approaches, which generally focus more on this case, could likewise be promising.
Conclusion
In this work, we consider the problem of generalising to new transition dynamics. We first illustrate that for classes of problems, context-unaware policy cannot perform well over the entire problem space-necessitating the use of context. We next turned our attention to how context should best be incorporated, introducing the Decision Adapter-a method that generates the weights of an adapter module based on the context. When context is necessary, our Decision Adapter outperforms all of baselines, including the prevalent Concat model. Next, when there are some irrelevant context variables that change between training and testing, the Concat and cGate models fail to generalise, whereas our Decision Adapter performs well. This result holds in several domains, including the more complex Ant. Furthermore, our Adapter is theoretically more powerful and empirically more effective than cGate, showing that our hypernetwork- based perspective useful when dealing with context. Ultimately, we find that the Decision Adapter is a natural, and robust, approach to reliably incorporate context into RL, factored way. This appears be of particular benefit when there are several irrelevant context variables, for instance, in real-world settings [97].
10

---

Acknowledgements
Computations were performed using High Performance Computing infrastructure provided by the Mathematical Sciences Support unit at the University of the Witwatersrand. M.B. is supported by the Rhodes Trust. D.J. is Google PhD Fellow and Commonwealth Scholar. B.R. is CIFAR Azrieli
Global Scholar in the Learning in Machines & Brains program.
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. Nature, 518(75400:529-33, 2015.
[2] David Silver, Thomas Hubert, Julian Schrittwieser, Joannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, ct al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419)) 140-1144, 2018.
[3] lige Akkaya, Marcin Andrychowicz, Macick Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, arXiv preprint arXiv: 19100011,, 2019.
Solving rubik's cube with a robot hand.
[4] Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu. Max Jaderberg, Wojtek Czarnecki, Andrew Dudzik, Aja Huang. Petko Georgiev, et al. AlphaStar: Mastering the Real-Time Strategy Game StarCraft I1 https: : /ttp:///eedee././//// www.edddd.oo////////e//eee/ee //ae///lg////a////e/e/leeee//a// walphastar-mmsteeretee
time- strategy-
game- samme-ttrcraftt-ii 11/, 2019.
[5] Amy Zhang, Nicolas Ballas, and Joelle Pineau. A dissection overfitting and generalization continuous reinforcement learning. CoRR, abs/ tbs//806.07937, 2018. URL http: http///axvvvv/////ee/ee wwwp://abs//.....//// 07937.
[6] Karl Cobbe, Oleg Klimov, Christopher Hesse, Taehoon Kim, and John Schulman. Quantifying generaliza- tion in reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, pages 1282-1289. PMIR, 2019. URL httpt:::pppoceeeeeeieiete
wwtps//eeeecoaaeeeeeeeeeeee 9a.html.
[7] Robert Kirk, Amy Zhang, Edward Grefenstette,
Rocktäschel. survey zero-shot generalisation in deep reinforcement lcarning. J. Artif. Intell. Res., 76:201-264 2023, doi: 10. 16. 1.1474, URL
[8] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In Proceedings of the 34th International Conference Machine Learning, pages 2817-2826. PMLR, 06-11 Aug 2017. URL https: http:////ooceedeeeeeeeeee mlr wwtpsppe///pppeieeeeeeep http//iitoooaaaatmmm
[9] Mohammed Amin Abdullah, Hang Ren, Haitham Bou-Ammar, Vladimir Milenkovic, Rui Luo, Mingtian Zhang, and Jun Wang. Wasserstein robust reinforcement learning. CoRR, abs/1997.13396 2019. URL http: : ht////arvvvorr///// abs/ /199/9991111 19196.
[10] Daniel J. Mankowitz. Nir Levine, Rae Jeong, Abbas Abdolmaleki, Jost Tobias Springenberg. Yuanyuan Shi. Jackie Kay. Todd Hester, Timothy A. Mann, and Martin A. Riedmiller. Robust reinforcement learning for continuous control with model misspecification. In 8th International Conference on Learning Representations. OpenReview.net, 2020. URL
[11] Gilbert Feng, Hongbo Zhang, Zhongyu Li, Xue Bin Peng. Bhuvan Basireddy, Linzhu Yue, Zhitao Song. Lizhi Yang. Yunhui Liu, Koushil Sreenath, and Sergey Levine. Genloco: Generalized locomotion controllers for quadrupedal robots. In Conference on Robot Learning, pages 1893-1903, PMLR, 2022.
[12] Wenxuan Zhou, Lerrel Pinto, and Abhinav Gupta. Environment prohing interaction policies. In
7th International Conference on Learning Representations. OpenReview.net, 2019. URL. https: 3AcFX.
[13] Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, André Biedenkapp. Bodo Rosen- hahn, Frank Hutter, and Marius Lindauer. Contextualize me the case for context reinforcement learning. CoRR, abs/2202.04500, 2022. URL hRtsttps://arxiv httaap:////axvv.oraaaabssss///
04500v1.
[14] Wenhao Yu, Jie Tan, C. Karen Liu, and Greg Turk. Preparing for the unknown: Learning a universal policy with online system identification. In Robotics: Science and Systems XIII, 2017. doi: 15.15/07 RSS. 2017 2017.0111.00000 X48. URL attp:://ww..robttceeproeeelle htwp:///wwn.nrrr//ii//e//.e
html. htm..

---

[15] Assaf Hallak. Dotan Di Castro, and Shie Mannor. Contextual markov decision processes. CoRR. 502/155,.02259, 2015. URL http:htaazii..orrggg.e//// hat///aa//aaaa /t/p//15222.222..
[16] Philip J. Ball, Cong Lu, Jack Parker-Holder, and Stephen J. Roberts. Augmented world mod- els facilitate zero-shot dynamics generalization from a single offline environment. Proceedings of the 38th International Conference on Machine Learning, pages (19-629. PMLR, 2021. URL http://procceddings htt.:/driiiimmm.rrrss//s/eseseee wttpp///////eepeeeteetpp watps///v////all//// http/baal12aaaaaaaa
[17] Shagun Sodhani, Amy Zhang, and Joelle Pincau. Multi-task reinforcement learning with context-basen representations. In Proceedings of the 38th International Conference Machine Learning, volume 139, mlr
pages 9767- 9779. PMLR, 2021. URL htttp::/prooeeeeeeeeeee wtw.tainnnw......ssss/eeeeemeeee www.spressssveee v139/ sodhani 21 a.ttml.
[18] Shagun Sodhani, Franziska Meier, Joelle Pineau, and Amy Zhang. Block contextual MDPs for continual learning. In Learning for Dynamics and Control Conference, LADC 2022, 23-24 June 2022, Stanford University, Stanford, CA, USA, pages 608-623. PMIR, 2022. wtK.hhtts://ppoceeeeee 2018//oohani22....
[19] Richard Bellman. A markovian decision process. Indiana Univ. Math. J., 6:679-684, 1957. ISSN 0022-2518.
[20] Martin I. Puterman and Eugene A Feinherg. Markov decision processes: discrete stochastic dynamic programming. SIAM review, 38(4))689, 1996.
[21] Richard S Sutton and Andrew Barto. Reinforcement learning: An introduction. MIT press, 2018.
[22] Josh Tohin, Rachel Fong, Alex Ray, Jonas Schneider, Wojeiech Zaremha, and Pieter Abbcel. Domain ran- domization for transferring deep neural networks from simulation to the real world. In International Con- ference on Intelligent Robots and Systems, pages 23-30. IEEE, 2017. doi: 10. 10.11009IROS,0 2017 2017.8233,0....822213.. 1302133. URL https: /ttp:://dooooooo///// wtw./////....//.....ee
[23] Fereshteh Sadeghi and Sergey Levine. CAD2RL: real single-image flight without single real image. Robotics: Science and Systems XIII, 2017. doi: 10.15607/RSS.200 201..XII..344 CRL http://
www. www.roboticeproceed..o wtwp://edddingg.oggrrrreeee wttps//////rs////////// 3/t3333333....m.....
[24] Xue Bin Peng. Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In International Conference on Robotics and Automa- tion, pages 1-8. IEEE, 2018, doi: 10.1 10.1099/ 1.018.840 8466028.. URL https:/ http://oooooor/o/o 1009/ ICRA.2018.8460528.
[25] Jan Matas, Stephen James, and Andrew J. Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pages 734-743. PMLR, 2018. URL http: / paroceedings.mlr.ppee www.pprraaavve.e //t///maaaaaammmmmmmmmm
[26] Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Józefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert. Glenn Powell, Alex Ray. Jonas Schneider, Szymon Sidor, Josh Tobin, Peter Welinder, Lilian Weng, and Wojeiech Zaremba. Learning dexterous in-hand manipulation. Int. J. Robotics Res., 39(1), 2020. doi: 10. 10.1177/02733644911877744 URL https:://oi.oo ottp//..////////// 0278364919887447.
[27] Alejandro Escontrela, George Yu. Peng Xu, Atil Iscen, and Jie Tan. Zero-shot terrain generalization for visual locomotion policies. CoRR, abs/2011.055 1,05513, 2020. URL https: httpes//aaxxiiooeee wrg. wrggabs///// 0abss//01.005111111
[28] Zac Wellmer and James T. Kwok. Dropout's 's dream land: Generalization from Iearned simulators to reality. In Machine Learning and Knowledge Discovery in Databases. Research Track European Conference, ECML PKDD 2021, Bilbao, Spain, September 13-17, 2021, Proceedings, Part volume 12975 of Lecture Notes Computer Science, pages 255-270. Springer, 2021. doi: 10. 00.1007/978-3-030-66666 86800066686-666_1_1 16. URL https: https:///oi.ooo
[29] Wenshuai 7hao, Jorge Peña Queralta, and Tomi Westerlund. Sim-to-real transfer in deep reinforce- ment learning for robotics: survey. In Symposium Series on Computational Intelligence, pages 737-744. IEEE, 2020. doi: 10. N0.110//SCI7880..22222 202099304.220098844... URL https:://dii.ooo http///iiiog/1..11101101 10.1109/ SSCI47803. 202. 9308468.
[30] Samuel Kessler, Jack Parker-Holde, Philip J. Ball, Stefan Zohren, and Stephen Roberts. Same state, different task: Continual reinforcement learning without interference. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7153-7151. AAAI Press, 2022. URL https:// iis.aaai.orgg. indei. http:p//ww....artttoe//////////p
12

---

[31] Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. RMA: rapid motor adaptation for legged robots. In Robotics: Science and Systems XVII, 2021. doi: 10.15607/RSS 2021 URL
https:///ab..org/111555////////
[32] Alvaro Sanchez-Coonzleez, Nicolas Heess, Jost Tobias Springenherg. Josh Merel, Martin A. Riedmiller, Raia Hadsell, and Peter W. Battaglia. Graph networks as learnable physics engines for inference and control. In Proceedings of the 35th International Conference on Machine Learning, pages 4467-4477 PMLR, 2018. URL http:/ / www.iinnggmm....pmmmmmm.. pwres/pees///ee.ee /aaas/vv000 h/////00////aa// sanchezz- wwweiizgonzazzzzz.n.
[33] Kimin Lee, Younggyo Seo, Seunghyun Lee, Honglak Lee, and Jinwao Shin. Context-aware dy- namies model for generalization in model-based reinforcement learning. Proceedings of the 37th International Conference on Machine Learning, pages 5757-5766. PMLR, 2020. URL http: // perroceddingsmmr..ppe
[34] Younggyo Seo, Kimin Lee, Ignasi Clavera Gilaherte, Thanard Kurutach, Jinwoo Shin, and Pieter Abbeel. Trajectory-wise multiple choice learning for dynamics generalization in reinforcement learning. In Advances in Neural Information Processing Systems, 2020. URL https: / http://///occe......omom www.iiiinnnnnuuuiippee neur ips.c// paper/2022 aaeee//220/hahshuu
Abt abstraact.hml.
[35] Seyed Kamyar Seyed Ghasemipour, Shixiang Gu, and Richard S. Zemel. Smile: Scalable meta inverse reinforcement learning through confext-conditional policies. In Advances in Neural Information Process- ing Systems, pages 7879-7889, 2019. URL httpp::/proceeeee 228ffe2.e2244ee5000rrreesooooo.0
nee. wttppppppppppppppppppppppppppppp / 2019/ hash/
Abstract.htm..
[36] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta- reinforcement learning via probabilistic context variables. In Proceedings the 36th International Confer- ence on Machine Learning. pages 5331 13300. PMLR, 2019. URL http::/pproceee//ieeee mlr press/ v97 *97/rakelly19a.htum htm...
[37] Hamid Eghbal-zadeh, Florian Henkel, and Gerhard Widmer. Context-adaptive reinforcement learning using unsupervised learning of context variables. In NeurlPS 2020 Workshop on Pre-registration in Machine Learning, pages 236 254. PMLR, 11 Dec 2021. URL https: //proceed////o mlr press// v1 v148/ 1448/eghbal-- www.zzzzd.eaaaaaaee
[38] Yao Mu, Yuzheng Zhuang, Fei Ni, Bin Wang, Jianyu Chen, Jianye Hao, and Ping Luo. De- composed mutual information optimization for generalized context in meta-reinforcement learning. CoRR, abs/2210.04209, 2022. doi: 10.48550/arXiv 11.5500ariiv2220..44422 2210.04209. URL https: orgp//.///000005500000 18.45500// arX1v.2210.04260
[39] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, pages 1856-1866 PMLR, 2018. URL http:
/ //ppoceeedngs.mlll www.diinns.mllr..... wwwpppresssv.. hat//haanojaaaa htmm.
[40] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, 0bs//707.006347 2017. URL http://arxiv.orr hargg//bs///ee aws//b/7..oee 006377.
[41] André Biedenkapp. David Speck, Silvan Sievers, Frank Hutter, Marius Lindauer, and Jendrik Seipp. Learning domain-independdet policies for open list selection. 2022.
[42] Matt Peng. Banghua Zhu, and Jiantao Jiao. Linear representation meta reiffreeinoreemeett learning for instant adaptation. CoRR, abs/2101 abs//1011044550, 2021. URL https: : http:///wwwwwwaa///////// http:///11110/1011 044750.
[43] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel. RL2: Fast reinforcement learning via slow reinforcement learning. CoRR, abs/1611.000 0tns/1611.027999 2016. URL http: / //arxiv.org//aaaaa /t/aa/b///111/ 1etp//619..079..
[44] Chelsea Finn, Pieter Abbeel, and Sergey Levine. agnostc meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1126-1135. PMLR, 2017. URL htttt::p/ppoceeddngee
[45] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 3942-3951. AAAI Press, 2018. URL https:/ //wp uuw aaai. www.ocs///ooooooooooooooooooocco ocs/ http///c//nneeesee index.ddxxpppppe phpp LAAI/AMA18//aaeez http//////ppppeeeeeeee/eee/e/e
[46] Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 2018. doi: 10.2391 15distill 1001 https://distilllpubb/1iBBiiieei wris-ttrasforrmaiions.
13

---

[47] Carolin Benjamins, Theresa Eimer, Frederik Schubert. Adilya Mohan, Sebastian Döhler, André Biedenkapp. Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Contextualize the for context reinforcement learning. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.
URL
[48] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-maron, Mai Giménez. Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Transactions on Machine Learning Research, 2022. Featured Certification,
ISSN 2835-8856. URL https: httts::/oopnnrviieee Outstanding Certification.
[49] Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Picter Abbeel, and Dale Schuurmans. Foundation models for decision making: Problems, methods, and opportunities. CoRR, abs/2303.04129 2023. doi: 10.48550/arXiv. 230.l..22333.00 2303.04129. URL https:/ http////ooo////////////oo///ooot
48550//rXiii 23.. 04129..
[50] Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee, Maria Bauzá, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, et al. Robecat: A self-improving foundation agent for robotic manipulation. CoRR, abs/2306.11700 2023, doi: 00.45500/rXiv.233330 1706. URL https:///oi.ooe org.oioorr////// 18650/arXii1 www.xivv..3...... 237..11766...
[51] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang. Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xic, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. A comprchensive survey pretrained foundation modcls: A history from BERT to chatgpt. CoRR, abs/2302.004111 2023. doi: 10.4 48.48550/arXiv... 2302.09419. URL https://doi.orrr /0/550 srXiv.2302.055
[52] Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach. and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In Conference Computer Vision and Puttern Recognition, pages 15617-15629, IEEE, 2022. doi: 10.1 CVPR522688.0222.0151 URL https:
[53] Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fci. Metamorph: Learning universal controllers with transformers. In The Tenth International Conference on Learning Representations. OpenReview.nel,
2022. URL https: / http:////peeeieeeeeeeeee www/teetenneeeeeee d=tpiitkpppmppaapmm GvYL.
[54] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in Neural Information Processing Systems, pages 5998-6008, 2017. URL https: http:///pocceennn....o nwurnngg....uuupp wwweurip..ccppppppppeee /217 2017 hashh/ $ffee2243547dee111t006633 htt.eeee911tt000311ccaa444ae wwseeeeaa-aaaaaaaatttaco. Abstract.htm.
[55] Ingmar Schubert, Jingwei Zhang. Jake Bruce, Sarah Bechtle, Emilio Parisotto, Martin A. Riedmiller, Jost Tobias Springenberg, Arunkumar Byravan, Leonard Hasenclever, and Nicolas Heess. A generalist dynamics model control. CoRR, abs/2305.109) 10912, 2023. doi: 10.448550 10912.. URL https://doi.oor 10. 48544/550////
[56] J. Richalet, A. Rault. J. L. Testud, and J. Papon. Model predictive heuristic control: Applications industrial processes. Autom.. 14(5):1334425 1978. doi: 10. 10.1016/0005- 1098779900000 URL
https://aoi.org// watp////.1.101/0000..00 1098 (78) 90001-8.
[57] Carlos E. Garcia, David M. Prett, and Manfred Morari. Model predictive control: Theory and practice survey. Autom,, 2333):335-348, 1989. doi: (0.1006/0000-00988899)0000000000 10. 10.1016/0005- 1098 900020
URL https://doi.orgg
[58] Max Schwenzer, Muzaffer Ay. Thomas Bergs, and Dirk Abel. Review model predictive control: An engineering perspective. The International Journal of Advanced Manufacturing Technology, 117(5-6): 1327-1349, 2021.
[59] Yanchao Sun, Shuang Ma, Ratnesh Madaan, Rogerio Bonatti, Furong Huang, and Ashish Kapoor. SMART: supervid multi-task pretraining with control transformers. In The Eleventh International Conference on Learning Representations. OpenReview.net, 2023. URL https://openrevieew.net pdf pdf7idd0pii3HH@QEf..
[60] Jacob Beck, Risto Vuorio, Evan Zheran Liu, Zheng Xiong, Luisa M. Zintgraf, Chelsea Finn, and Shimon Whiteson. A survey of meta-reinforremment learning. CoRR, abs/2301.0800 2023, doi: 10.48550/rrXiv
URL https: / /wtp///ooooooo http///ooooo/oor///////ooo org/ 48550/arz aroiv 230.028800.00..
14

---

[61] Learning to Learn. Springer, 1998. ISBN 97--1--1111--77 doi: 1--779978---4415-55522222 URL https://doi.org/10000
[62] Jonathan Baxter. Theoretical models of learning to learn. In Learning to Learn, pages 71-94. Springer, URL
1998. doi:
/970- 1 4615- 5529
2_4.
[63] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, Pieter Abbeel. simple neural attentive meta- learner. In 6th International Conference on Learning Representations. OpenReview.net, 2018. URL http://pweeeeeeeeeeetooorrreeeee
[64] Luisa M. Zintgraf, Kyriacos Shiarlis. Vitaly Kurin, Katja Hofmann. and Shimon Whiteson. Fast context adaptation via meta-learning. In Proceedingy of the 36th International Conference on Machine Learning, pages 7693-7702. PMLR, 2019. URL http: http:///poceeeen..c..o mmriiiinngmm........mmm wwt.:mwww.......eeemeemmmmmmeeee press/v97/ zintgraf 1a..tgrgaa199......1.. 19a.html.
[65] Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning to adapt dynamic, real-world environments through meta-reinforcemmnt learning. In 7th International Conference on Learning Representations. OpenReview.nel, 2019. URL https: http///ppeneeeeeeeeteeeeee net wtw.:e/uwweeeeeteeeeeeeeee /ttpe//wwww...eeeooo/ootttttttta idyyytt
[66] Haozhe Wang, Jiale Zhou, Xuming He. Learning context- aware task reasoning for efficient meta reinforcement learning. Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, pages 1440-1448. International Foundation for Autonomous Agents and Multiagent Systems, 2020. URL. https: htt.://a..mmm/////ee//// doi /abs/ /00 55555/3339666.33988833
[67] Richa Upadhyay, Ronald Phlypo, Rajkumar Saini, and Marcus Liwicki. Sharing to learn and learning to share fitting together meta-learning, multi-task learning, and transfer learning: A meta review. CoRR, abs/2111.12146, 2021. URL https: / http///axvvv/o////e/ee abs atbb////111 2111.111111 12146.
[68] Jacob Beck, Matthew Thomas Jackson, Risto Vuorio, and Shimon Whiteson. Hypernetworks in meta- reinforcement learning. Conference on Robot Learning, pages 1478-1487. PMLR, 2022. URL https://proceedii: http://cceeeedoaaeereeeeeeeee/ee
[69] Elad Sarafian, Shai Keynan, and Sarit Kraus. Recomposing the reinforcement learning building blocks with hypernetworks. Proceedings of the 38th International Conference on Machine Learning, pages 9301 9312. PMLR, 2021. URL htttp://proceedingss mll wwtpppeeesssseeeeeeeeeee innffiinn11i hmm.mml..
[70] Qinyuan Ye and Xiang Ren. Learning generate task-specific adapters from task description. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 646- 653. Association for Computational Linguistics, 2021. https: htttps:/ddi..org/// 1ttp///..155////222222
doi: 10.18653/v//2022 acl-shortt.8.. URL
[71] Jonathan Pilault, Amine El hattami, and Christopher Pal. Conditionally adaptive multi-task learning: Improving transfer learning in NLP using fewer parameters & less data. International Conference on Learning Representations, 2021. URL httpa: http://ppnnrrretetteet www.innnt/oooommmddeeeee diw.mm/ddddddddddddeaedea 1dbHzAME.
[72] Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-cfficient multi task fine-tuning transformers via shared hypernetworks. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 565-576. Association Computa- tional Linguistics, 2021. doi: 10. (0.18653/v1/2022 1221 URI https: :/tp:::///oiii .rgg 10.16533//// 2021.ac1- aoo.cclllonnce 1ong.47.
[73] Hamish Ivison and Matthew Peters. Hyperdecoders: Instance-specific decoders multi-task NLP, In Findings of the Asxociation for Computational Linguistics, pages 1715-1730. Association for Computa- tional Linguistics. 2022. URL https: //acl /tt///aaaaattolggy.oooooooo .rgg//oor////..
wwwp//22....22/.. 2222 inddnggseennii m.nlp.124.
[74] Haoxiang Shi, Rongsheng Zhang, Jiaan Wang, Cen Wang, Yinhe Zheng, and Tetsuya Sakai. Layerconnect: Hypernetwork-astee assisted inter-layer connector to enhance parameter efficiency, In Proceedings of the 29th International Conference on Computational Linguistics, pages 31 3120-3126. International Committee on Computational Linguistics, 2022. URL https: / h/wps///holaaaacgo/o/oa/ao////ae org.:
1.276.
[75] Sylvestre-Alvise Rebufli, Hakan Bilen,
Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Sys- tems. pages 506-516, 2017. URL https:///ree hwwpee/iwgnnuurps..ccp/pppeeeeee e7b24b1 1ilebb12aace 2at.ia122eaadddeeeesssoeoddd abstract.html.
15

---

[76] Neil Houlsby. Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe. Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter--fficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Leurning, pages 2790-2799, PMLR, 09-15 Jun 2019, URL https: http:::://rrcceennngs mmr.
[77] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth. Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhuh: framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 46-54, Association for Computational Linguisties, 2020. doi: 10. 18653//1/0020.cmmmpp-dee
URL
https://doi... www.o.oogg/...1655.10 (0..18653/v/1200 wwt.000000mmmmmmmemmmm canlp-demos.7)
[78] Jonas Pfeiffer, Ivan Vulić, Tryna Gurevych, and Sehastian Ruder. MAD-X: An Adapter- Based Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference Em- pirical Methods in Natural Language Processing (EMNLP). pages 7654-7673. Online. November 2020. Association for Computational Linguistics. doi: 1/116653/1/2220.ccm www.:peeeemlp---oee coannp-mmin.611 main.617. URL https://aclanthologg erg/ 202...mmm0mmpppp.mm mmmlp-main.617
[79] Hang Le, Juan Miguel Pino, Changhan Wang. Jiatao Gu, Didier Schwab, and Laurent Besacier. Lightweight adapter tuning for multilingual speech translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 817-824. Association for Computa- tional Linguisties, 2021, doi: 10. 10.18663/112221.aacc sch-r 103. URL htt.:httpp//ooooooo http:///o..oog/1.11/5.3333 10.1653/vv/ 2021.ac1-short.Il
[80] Chin-Lun Fu, Zih-Ching Chen, Yun-Ru Lee, and Hung-yi Lee. Adapterbias: Parameter-efficient token- dependent representation shift adapters NLP tasks. In Findings of the Association for Computa- tional Linguistics: NAACL 2022, pages 2608-2621. Association for Computational Linguistics, 2022.
doi: 10118865311/22222 finding.iinnnsssie naacl. 199. URL naac1.199.
indingss-
[81] Bethan Thomas, Samuel Kessler, and Salah Karout. Efficient adapter transfer of self-supervised speech models for automatic speech recognition. In International Conference Acoustics, Speech and Signal Processing, pages 7102 7106. IEEE, 2022. doi: 10.1 10111///CCSSS44992220029999449 / //doi.org/10.111 1tte:/.og/11111//////1 /1A.11/99///A199922..... 2ww.:99222222..92222222
URL https:
[82] David Ha, Andrew M. Dai, and Quoe V. Le. Hypernetorks. In 5th International Conference on Learning Representations. OpenRevicw.net, 2017. URL
[83] Andrew William Moore. Efficient memory-hased learning for rohot control. Technical report, University Cambridge, Computer Laboratory, 1990.
[84] Richard S. Sutton. Generalization in reinforcement learning: Successful examples using sparse coarse coding. In Advances Neural Information Processing Systems, pages 1038-1044. MIT Press, 1995. URL
httppprrriiip.cccppppe//00-----0 learning successful examples- using- sparse- coarse- coarse-coding.
[85] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. CoRR, abs/1606.011400 2016. URL attp://arxiv www.w/rrg///////////
[86] Andrew G. Barto, Richard S. Sutton, and Charles W. Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEF Trans. Syst. Man Cybern., 13(5):834-846, 1983. doi: 10. 1983://....98..33111111
URL httpp:://oiiorr///
[87] Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pincau. Learingg robust state abstrac- tions for hidden-pprammttr block mdps. In 9th International Conference Learning Representations. OpenReview.net, 2021. URL https: httlhhhtss///ppeeeeee /ttp://////pp//////eee//e/eeeeee http:////ppeeeeeeeeeeeeeerrrrria htt.t//oorormmiiiifo0000aaa333
[88] Charles Packer, Katelyn Gao, Jernej Kos. Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement leamning CoRR, abs/1810.12282. 2018. URL http://arxiv..rrz abs/1810.12282.
[89] Emanuel Todorov. Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In International Conference on Intelligent Robots and Systems, pages 5026-5033, IEEE, 2012. doi: wttp///OS.202..63363688 2012.6386109, URL https: http:::://doi.org///// wtt.:///////./////.///// IROS 228...6.661...
10. 109/1ROS
[90] John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. High- dinen-iinnsional continuous control using generalized advantage estimation. In 4th International Conference on Learning Representations, 2016. URL http:///rxivv org/ harg/////aa abs/bb/////// 11156002433
16

---

[91] Ryan Julian. Benjamin Swanson, Gaurav S. Sukhatme, Sergey Levine, Chelsea Finn, and Karol Hausman. Never stop learning: The effectiveness of fine-tuning in robotic reinforcement learning. In Conference on Robut Learning, pages 2120-2136. PMLR, 2020. URL SRLLLttps://prreeeddine http://eeddnng......rre/////ssoo julian21a.hee
[92] Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang- Wei Edward Lee, Jie Tan, and Sergey Levine. Learning agile robotic locomotion skills by imitating animals. In Robotics: Science and Systems XVI. 2020. doi: 10. 15607/RSS 2020. URL https: / /ttp///toooo//o////////ooo/oooot 11.11...11.77 20556677/SSS 2020.XVV XVI.064
[93] Laura M. Smith, J. Chase Kew, Xue Bin Peng. Schoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In International Conference Roboties and Automation, pages
IEEE, 2022. doi: 10.1009/CRR44663922229888811 URL. https: http:////iiiooo///.e///// 1111111999IIIAAE1 2022.9812166.
[94] Chelsea Finn, Tianhe Yu, Justin Fu, Pieter Abbeel, and Sergey Levine. Generalizing skills with semi- supervised reinforcement learning. In 5th International Conference on Learning Representations Open- Review.net, 2017. URL
[95] Luisa M. Zintgraf, Kyriacos Shiarlis, Maximilian Igl, Sebastian Schulze, Yarin Gal, Katja Hofmann, and Shimon Whiteson. Varibad: A very good method for bayes-adaptinn deep RL via meta-learning. In 8th International Conference on Learning Representations. OpenReview.net, 2020. URL https: / htt.rr//ieeeettttoom///ddddadaa
[96] Luisa M. Zintgraf, Leo Feng, Cong Lu, Maximilian Igl, Kristian Hartikainen, Katja Hofmann, and Shimon Whiteson. Exploration in approximate hyper-state space for meta reinforcement learning. In Proceedings the 38th International Conference on Machine Learning, pages 12991-13001. PMLR, 2021. URL
http: http://poceeeiis.ll..ppppeeee www.rrprees/vi3311 prreess/t19/iinttgrfallaaa. /iittggaf2aaaaaaa...m
[97] Tao Chen, Adithyavairavan Murali, and Abhinav Gupta. Hardware conditioned policies for multi-robot transfer learning.
Advances in Neural Information Processing Sys- tems. pages 9355 9366, 2018. URL attps://procest www.eeedd.ng..uu..s......... wwwpp/ps.c.pppppe//////e/ec pttpc//pappppe////aaaaaaaaaa 221///228///h/////
BBctbf77a33220445233ar (a5022333666666770000 a63a7 Abstract.
[98] Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms. Journal of Machine Learning Research, 23(274):1 18, 2022. URL http://jmlr.on papers/v23/21- 1342.html.
[99] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems, pages 8024-8035, 2019. URL https:/ http:///poeeee..eeeeeeee 5bbea288ffe777871115777777727222 wwt.:///tt--tttttthhhhh. www.eeediinn.eeuiss........ wwweerips.ccppppeeee wttpcccppppeee1eee. 99/hash/
[100] Minyoung Huh, Hossein Mohahi, Richard Zhang, Brian Chcung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity hias in deep networks. CoRR, ahs/2103. 10427, 2021. URL abs/2103.10427.
[101] Maximilian Igl, Kamil Ciosek, Yingzhen Li, Sehastian Tschiatschek, Cheng Zhang, Sam De- vlin, and Katja Hofmann. Generalization in reinforcement learning with selective noise in- jection and information bottleneck. In Advances in Neural Information Processing Sys- tems, pages 19956-13988 2019. URL http://pproeeenng..errrppppppppp pttp:ppp////////aa #2cc19ba7f2e1877E 666-6 Abstract html.
[102] Xingyu Lu, Kimin Lee, Pieter Abbeel, and Stas Tiomkin. Dynamics generalization via information bottleneck in deep reinforcement learning. CoRR, abs/2008.00614, 2020. URL https://arxiv..rrg abs/2008.0061..
[103] Ben Eysenbach. Ruslan Salakhutdinov, and Sergey Levine. Robust predictable control. In Advances in Neural Information Processing Systems, pages 27813-27825. 2021. URL
httpeddww...c.p....ppp/////////a Abstract.htm.
17

---

Appendix
A Formalising the Theoretical Foundations
In this section, we more formally consider our theoretical results from Section We first define our problem formulation in Assumption A. then state the theorem in Theorem A.5. Finally, we prove the theorem at the end of this section.
Suppose we have a CMDP where S,A R and Y E (0, 1). The context Rd defines goal location, with reward function (s a, 8, = ||s cll < F and ( otherwise. The episode terminates when the reward obtained is Suppose further that 7 == DD, with D R being the distance the agent travels in single step and E Z being the num- ber of steps it takes the agent to traverse a distance of F. Here T and D are kept fixed across contexts.
Assumption A.1. (Problem Formulation):
Definition A.2. (Context Averaged Value Function): Denote V Ee-c IV. expected value of the value function of policy æ in state over all contexts in the set C. Similarly (e(s) = Eewel VV (s) is the average of the value functions of the optimal policies on each context.
(s)]. i.e., the
Definition A.3. (Value Function Optimality Ratio): Let 0 S az(s)
denote how close the average value function policy т is to the optimal context-specific value function. We use the optimality ratio instead of the optimality gap [47] because the optimality gap depends heavily the reward scale of the environment, whereas the optimality ratio depends only on the ratios.
Vč (a)
0z (a)
Definition A.4. (Minimum Inter-context Distance):
Let dmin = menc, [[e--cyl|
be the
D
minimum number of steps required to travel between any two context centers.
Theorem A.S. Then, in this problem space, have the following:
(i) For some set of contexts Cfor with cardinality N, and some state s, all deterministic unaware policies will have;
Cjar (s)
1
Ndmin
1c (8)
5
Vo (8)
N
Additionally:
1 N
lim
(2)
drncn 700
i.e., as the contexts within Cfar move further away from each other the agent's 's performance scales inversely with the number of contexts.
(ii) For some set, Celose and all states 8, at least one deterministic unaware policy Tg will have:
T.a.y Colore
7-
Celose
regardless of the number of contexts in Celose-
Proof. (i) Consider set of contexts Cfar such that for all Cj E Cfar i + j. 1l|. - 1ccl > 4F.
Consider the state + Σ. Di=i C, i.e., in the middle of all contexts. Now, suppose the optimal policy per context would require only 3 steps to reach the appropriate goal. An unaware policy that reaches every goal, by contrast, must travel each context sequence. If this policy visits cı, . in order, then the value functions for each context would be:
(T(s) = 3 1 Vi (s)
(g((s) - االحصر
(s) - 1 |+1 1
18

---

and so on. Let dmn be the number of steps required to move between the two closest distinct contexts in Car Thus, dmin 1 2r. 6 Then, V (s) S -0+(i-1)dmm (i-1)mmin Now,
Ve (0) Be VV (s))
N
(s)
N
r 8+6 1)dmin
N
8 1
(6-1) 1)dmin
N
1
? Ndmas
N
Amnin
y Ndmsn
N Amiin
Hence, a
Ve Veyan N
Ndmin Pmin
Finally, as dini - 00, Aomin - 0 and y Ndmin + 0. Hence,
limn a1 N'
(ii) Consider set of contexts Celose such that for all Cy E Celoses ||q -cyl < F. Then, consider an unaware policy that always travels the joint intersection point of these contexts. This intersection point is at most away from the closest context to the agent, therefore the agent will perform at most T unnecessary steps compared the optimal policy. Hence, its value function has lower bound of V (s)y", and we have
(a) >
V
(8)7
Therefore,
Veennn
(s)
E BorCatone [s)
2 BerCr (8)
7 BewCatome (s)
7 Vicane (8)
And have: Valuxe (x) Ve (+)
Vy
7T
B Linking Theory and Practice
The formulation in Assumption A.1 has consistent dynamics and context-dependent rewards (as the context defines the goal location, and the transition dynamics do not depend on the context). This is in contrast to the problem aim to study, which has consistent rewards, but context-dependent dynamics; however, we can show that for the environment considered Appendix A, these two equivalent.
We can obtain the context- context-dependent dynamics formulation under the assumptions in Section 2, i.c. that only the transition dynamics change, as follows. Suppose the context defines rotation matrix C, E R with dynamies equation T(s, a) = s' + C,a and initial state Sp Here the goal is fixed location g E Rdd The episode would terminate with reward of if [|s' -g]l F. We can then perform coordinate transform obtain C 8' C 8 a. The goal would then depend on the context 9c, C g and the effect of an action in this new space would be consistent across contexts.
'This is because, while the centers of the circles and e, are further than 47 apart, their closest edges are only least 27 apart.
19

---

In essence, we change the reference frame to be that of the agent, leading to separate goal locations for each context, but consistent dynamics- corressoding the problem in Assumption A.I.
In the remainder of this section, we link our theoretical and empirical results, showing that the ODE (for some context distributions) is example of non-overlapping environment, while in CartPole, an unaware policy can perform well.
B.1 ODE
Here we show that, in the ODE domain, the Unaware model can either perform well or poorly, depending on the context it evaluated on. In particular, we train models on context sets C,4 train {1,5) and C Crrain { {-5, 1,,1,5 5} and evaluate C [0, 10) and C results are shown in Fig. 7. When only considering the positive contexts (left), the Unaware model performs only slightly worse than the Concat model. This corresponds to case (ii) in Theorem A.5, where the contexts are similar enough for a single policy to perform well. However, when considering both positive and negative contexts (right), the Unaware model fails completely, and performs much worse than the context-aware ones. This is because, in the ODE domain, positive and negative contexts cannot solved using the same action. This corresponds to case in Theorem A.5, where making progress on one context results in negative progress on another.
10, 10). These
Positive
Positive and Negative
200
150
100
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Training Time (Steps)
Unaware
Concat
Adapter
Figure 7: Here show results in the ODE domain. On the left, train {1, 5} and evaluate
[0, 10]. On the right, we consider both negative and positive contexts, training on (-5, 11, 1, 5) and evaluating on [-10, 10). The Unaware model performs much better when only considering positive contexts, even though the context-aware models still outperform it.
B.2 CartPole
We now investigate CartPole. In particular, we vary only the pole length, and consider the training con- texts as (1,4,6) 6), with the evaluation contexts being 301 equally-spaced points between 0.1 and
These results are shown in Fig. 8. While the Unaware model can achieve comparable generalisation performance to the context-informed models, it takes significantly longer do so. In particular, to obtain an average reward above 400, the Unaware model must train for more than 600k steps, whereas cGate, Concat and our Adapter reach this threshold before 100k steps. This shows that, while context is not necessary in this domain, incorporating may be beneficial and can lead to much more sample- efficient generalisation compared the Unaware model. This is particularly relevant when training on multiple training contexts, which seems to cause interference for the Unaware model. The context- aware approaches, however, are not as susceptible to this problem, as they are able to distinguish experience from different contexts. Furthermore, most of the context-aware models perform similarly in this case. In particular, our Adapter, Concat and cGate converge rapidly, while FLAP converges slightly slower, but achieves comparable generalisation performance the end of training.
20

---

500
400
AER
300
200
Unaware
Concat
Adapter
cGate
FLAP
Training Time (Steps)
Figure 8: Showing the average evaluation reward as we increase training time for CartPole. Mean performance over seeds is shown and standard deviation is shaded.
250
Distance Travelled
200
150
100
Unaware
Concat
Adapter
40
100
Mass
Figure 9: Mass vs. Distance Travelled (proportional to reward), at the end of training for the Ant domain. Red lines the training contexts. Mean performance is shown, with standard deviation over 16 seeds shaded.
B.3 Ant
Here we consider the Ant domain, and vary the mass, without any distractor variables. As seen in Fig. 9, the Unaware model performs worse than the context-aware ones. This performance gap is due to the Unaware model being unable to simultancously perform well on light and heavy masses, as seen in Fig. 9. In contrast, by utilising context, the agent perform well in both settings despite their significant differences.
C cGate as a Special Case of the Decision Adapter
As discussed in Section 5, here we show that our architecture is, in fact, generalisation of another network architecture introduced by prior work. Benjamins et [13] introduce cGate, which obtains an action follows. It first calculates d(s) Ed using the state encoder o and g(c) e IRd using the
21

---

context encoder Then, it calculates features h cGate(s,c) o(s) @ g(c), with @ being the elementwise product. The action is obtained as = [(h), where f RR + A is the learned policy. Here, both the state encoder ó and the context encoder g correspond neural networks that each output a d-dimensional vector.
This approach is specific instantiation of our general adapter architecture, where have one adapter module = A, In particular, the state encoder & corresponds to the partial neural network before adapter module, i.e., layers L1. L2, ................................ Li-1- The policy corresponds to the layers after our adapter module, L; Li+i...., L.. Thus, we have that ((s) [ and f(h) = L (..). If we define adapter network A()(s) c) ((8) @ g(c),
recover the result cGate. This would be possible if we let our adapter consist of one layer without nonlinearity or bias vector, with its weight matrix being
W H(c) diag(g(c)). This would result in
= g(c) @ 6(s) cGate(s, where we have used the fact that, for two vectors a, b Fd the matrix multiplication diag(a)b is the same as the elementwise product @ b. This output vector is then passed to the policy network f. Our approach, however, is strictly more powerful than a single elementwise product, our adapter can be an entire nonlinear neural network.
A(o(s)lc) Wo(s)
D Experimental Details
D.1 Environments
This section contains more details about the experimental setup of our environments.
D.1.1 Context Normalisation
Before pass the ground truth context the models, first normalise them to range between 0 and 1. In particular, for each context dimension i, we calculate the context to be given to the agents Crorm,
Cnorm, ma.;;
where mar; is the largest value for dimension i in the training context set. This results in the largest training contexts having values of 1.0. During evaluation. we perform the same normalisation, so contexts outside the convex hull of the training set will potentially result in values larger than 1.0.
D.1.2 ODE
During training, for each context, cycle between starting states, with each episode taking starting state from the set (1.0, 0.5, - 0.5, 1.0). Thus, the agent trains on the first context for episodes, each with different initial state. Then it goes to the second context, and on. When we reach the end of the context list, we simply start at the first context again. We do this to ensure training comparable for each method, and that there is sufficient diversity in the initial state distribution. During evaluation, we fix the starting state at 1.0 to isolate the effect of context.
D.1.3 CartPole
Table illustrates the context variables use in CartPole. While we consider all five variables as the context, we varied only the pole length for our experiments. The value for each state variable at the start of the episode randomly sampled from the interval [-0.05, 0.05).
Table The five context variables and their default values.
Name
Symbol
Default Value
Gravity
9.80
Cart Mass
me
1.00
Pole Mass
my
0.10
Pole Length
0.50
Force Magnitude
F
10.00
22

---

D.2 Hyperparameters
We use ReLU for all activation functions. We also aim to have an equal number of learnable parameters for each method and adjust the number of hidden nodes to achieve this.
We use Soft-Actor- architectures. We use the high-performing and standard implementation of SAC from the CleanRI. library [98] with neural networks being written in PyTorch [99]. We use the default hyperparameters. which are listed in Table 2. SAC learns both an actor and a critic. The actor takes as input the state and outputs μ and log(o)) which are used to construct the probability distribution over actions N (i,, σ) which is sampled from to obtain an action. The critic receives the performed action a as well as the state s, and outputs a single number, representing the approximate action action-value ((s, a). Fig. 10 illustrates each baseline.
[39, SAC] for all methods, to isolate and fairly compare the network
Table 2: The default hyperparameters we use in the CleanRL implementation.
Name
Value
Buffer Size
1 000 000
0.99
0.005
Batch Size
256
Exploration Noise
0.1
First Learning Timestep
5000
Policy Learning Rate
0.0003
Critic Learning Rate
0.001
Policy Update Frequency
2
Target Network Update Frequency
1
Noise Clip
0.5
Automatically Tune Entropy
Yes
D.3 Adapter Configuration and Design Decisions
Now, our method provides a great degree of flexibility in terms of configuration, and we must make multiple design decisions to obtain a usable instantiation of the Decision Adapter. In this section, we briefly detail the default settings we use and justify why these are reasonable. We empirically investigate the merits of these choices and compare them against alternative options in Appendix E.
D.3.1 Bottleneck Architecture
We use bottleneck architecture in generated adapter modules, described by Houlsby et al. [76]. In essence, we have a down-projection layer that transforms features from d, dy-dimensional to p-dimensional, with p d,. Then, the second layer up up-projects this back to dd-dimensional vector.
The bottleneck architecture is beneficial for two reasons. Firstly, it reduces the number of required parameters in our adapter model. For instance, if we have two layers in this bottleneck architecture, the number of parameters would differ substantially compared to one layer that transforms the d,-dimensional features to d, dimensions. In the first case, the number of parameters would be on the order of (d, x p + p x d,), corresponding to a d, X p matrix for the down-projection and a p x de matrix for the up-projection. If we only have one layer, then we have single d, d, matrix. Thus, the bottleneck requires (pp.)) parameters, whereas the single layer requires (dd)). we, for example, have d, 256 and p = 32, the bottleneck architecture has four times fewer parameters than the single layer. This, in turn, makes it casier for the adapter hypernetwork to generate useful weights.
Secondly, the bottleneck architecture imparts a low-rank inductive bias on the adapter, which may prevent it from overfitting [100]. This idea is related to prior work that has shown that, counterintu- itively, reducing the capacity of the agent's 's neural network can improve generalisation and reduce
'The bias vectors correspond to an additional + de parameters, but we omit this term as it is dominated by the number of parameters in the weight matrices.
23

---

The Unaware Model
(b) The Concatenate Model
(c) The cGate Model, with @ indicating an elementwise product
(d) The FLAP Model, with indicat- ing dot product
Figure 10: Illustrating the architectures of each of our baselines. Here we consider the actor's architecture, with separate output heads, one predicting the mean action a and the other predicting the log of the standard deviation (denoted as a in this figure). In each plot, the blue nodes represent the state inputs whereas the green represents the context. In general, for the bottom row, the network architectures for the context and state encoders are not necessarily the same.
overfitting [101-103]. Although approach does not incorporate information- -oottlenn--bttteekkkkkkk losses like these methods, the observed benefits of reduced capacity provide additional intuition for utilising the bottleneck architecture. Finally, despite any possible information loss due to the bottleneck layer, the skip connection allows the network to retain any important information from the input.
D.4 Compute
For compute, we used an internal cluster consisting of nodes with NVIDIA RTX 3090 GPUs. For single experiment and single method, the runs took between and days to complete all seeds.
E
Adapter Ablations
Here we experimentally justify some of the adapter-specific design decisions we made for our empirical results. The experiments in this section also give us insight into how robust the Decision Adapter is to changing its configuration options. We investigate the effects of the following factors, with our conclusions listed in bold.
The network architecture of the Adapter module. We find that Most adapter architectures perform well. This result is expanded upon Appendix E.
Whether the hypemetwork chunking methods while using significantly fewer parameters (Appendix E.2).
chunking. Hypernetwork chunking outperforms
Using a skip connection. The Decision Adapter can perform well with or without a skip connection (Appendix E.3).
24

---

The location of the adapter module in the main network. Most locations lead to high performance, except if the adapter at the very start or very end of the network (Appendix E.4).
Having an activation function before the adapter. Not having an activation function before the adapter in the actor model outperforms the alternatives (Appendix E.5).
For this entire section, we consider the ODE environment and follow the same procedure in Section 7.1. We use the ODE due to its fast training, and the fact that context is necessary, which enables us determine if particular design choice of the Adapter results in poor use of the context.
E.1 Adapter Architecture
Here we examine the effects of changing the Adapter module's network architecture. We are interested in how the architecture impacts performance for two reasons. First, we wish to empirically justify our bottleneck architecture. Second, we wish to show that the Adapter performs well with wide variety of network architectures, and is not particularly sensitive to this choice.
200
150
100
Adapter Architecture
[1.6]
(64)
Training Time (Steps)
165
Figure 11: Evaluating the effect of different adapter architectures. Mean performance is shown, with standard deviation shaded.
As discussed in Appendix D.3, we use a bottleneck architecture of size 32. Thus, the 336-dimensional features transformed into 32- 32-dimensional features and then projected back to 256 dimensions. We denote this architecture as [32), since the adapter has a single hidden layer of 32 neurons. Fig. 11 illustrates the performance for various other adapter architectures. Most methods perform comparably, but using hidden nodes or not having any hidden layers performs slightly worse than the base architecture. large hidden layer 256 nodes also does not outperform our bottleneck architecture. Finally, there is no particular benefit to having multiple layers in the adapter module.
E.2 Hypernetwork Chunking
Next, we compare our default architecture choice, which uses hypernetwork chunking, against non-chunking settings. do this because hypernetwork chunking allows us to have significantly fewer learnable parameters (resulting in faster training) compared to the alternative of predicting the entire weight vector in one forward pass. Thus, if our approach performs comparably better than non-chunking settings, then the chunking architecture justified.
25

---

200
175
150
125
100
50
25
Hypernetorr Architecture
0
[100.100]
[100,100,100]
Chunkeed [66, 66]
Training Time (Steps)
165
Figure 12: Comparing hypernetwork chunking (Base) compared to various non-chunking configura- tions as well as using larger chunks (Chunked (66, 66).. Mean performance is shown, with standard deviation shaded. Here, the values the brackets represent the number of hidden nodes per layer, with corresponding to hyperettork that has no hidden layers.
We consider non-chunking hypernetoork architecture of no hidden layers, as well as and 3 hidden layers of size 100, respectively. Our final option a chunked hypernetwork, with chunks of size 660, and two hidden layers of size 66 each (this is roughly twice as large our default configuration option discussed in Appendix D.3). These results are shown in Fig. 12. We find that both chunking configurations perform the best. The non-chunking approaches perform slightly worse, particularly if the hypernetwork has no hidden layers.
These non-chunking configurations also have significantly more learnable parameters than the default. For instance, the base setting results in the actor's hypernetwork having around 16k parameters, whereas the [100, 100), non-chunking setting corresponds to roughly 1.7M parameters. In summary, by using hypernetwork chunking, we can obtain high performance while using a very small number of learnable parameters.
E.3
Skip Connection
Here we investigate whether skip connection is beneficial or not. As mentioned in Section 5, the skip connection sets the updated x as x = (xx) + x. whereas if we remove it, the function becomes just 2: = A(x). In Fig. 13, we can see that using skip connection performs similarly to the alternative. This shows that the Decision Adapter robust to this parameter, and can perform well with or without skip connection.
200
175
150
125
100
50
25
0
Skip Connection
No Skip Connection
Training Time (Steps)
le5
Figure 13: Evaluating the effect of using skip connection. Mean performance is shown, with
standard deviation shaded.
26

---

B
C D
s
a
S
Showing the base network architecture, with letters indicating adapter positions. For instance, an adapter placed at location (D) receives the action as input and returns a modified one.
Figure 14:
E,4 Adapter Location
We next examine the effects of changing the location the adapter module in our network, and what happens if we have multiple modules. This information would be useful when using our method in practice, and deciding where and how many modules to place. There are generally many options regarding the location, as an adapter module can be placed between any two layers of the primary network.
As described in Section 5, our actor network consists of a "trunk", containing one hidden layer, which maps the state 256-dimensional feature vector, and one-layer action head, directly mapping the trunk features to an action. Here we consider the following locations for the adapter in both the actor and critic networks:
Base
The standard adapter used in the rest of this chapter, which is placed before the action head's layer. the critic, the adapter is placed before the last layer. This corresponds location (C) in Fig. 14.
Basey Placing the adapter in the actor network before the final layer the trunk, leaving the critic unchanged (location B).
Start The adapter here is at the start of both networks, i.e., the adapter module receives the state as input (location A).
End Here, place the adapter at the very end. Thus, the adapter takes in the predicted action and outputs a modified one. The critic's adapter is also placed at the end. This corresponds to location (D) in Fig. 14.
Alle Three adapter modules, one at the start of the trunk, one before and one after its final layer. The critic also has three adapters, one the start, one before the last layer and one after the last layer. The actor has adapters in locations (A), (B) and (C).
In all cases, ensure comparability against the base model, there is no activation function after the trunk in the actor model. These results are illustrated in Fig. 15. Overall, the base model performs the best. The adapter the trunk performs slightly worse and the performance is quite poor we put adapter modules at the start of the networks. If we have three adapter modules, the performance is similar to only having one before the final layer. Finally, when adapting only the final action, the performance is also poor. Overall, the performance is low if we have the adapter only at the very
27

---

start or end of the network. Within range of locations in the "middle" of the network, however, the performance high and only slightly worse than the default setting.
200
175
150
125
ABR
100
Pase
Dosty
Scart
End
Ail
Training (Steps)
165
Figure 15: Evaluating the effect of the adapter module's location. Mean performance is shown, with
standard deviation shaded.
E.5 Activation Function Before Adapter
Our final ablation considers the benefits of not having activation function before the adapter module inside the actor network. We perform this experiment to determine if our choice of not having an activation function before the adapter in the actor is justified, and if it is necessary to do the same for the critic.
200
175
150
125
No Activetion
No Activation (Critc)
Activation
Only logiot Activation
0.5
1.0
1.5
2.0
3.0
Training (Steps)
1e5
Figure 16: Comparing the effect of having activation function before the adapter not. The blue, green and red lines alter the actor's 's architecture, whereas orange changes the critic. Blue corresponds to the base choice outlined in Appendix D.3. Mean performance is shown, with standard deviation shaded.
Here we consider four options: (1) not having an activation function after the trunk; (2) having one; (3) having an activation function after the trunk, but only for the standard deviation head; and (4) having an activation before the adapter in the critic. Option (1) corresponds the default choice. These results are shown in Fig. 16. Not having activation function outperforms having one, and having an activation for only the standard deviation head is similar not having one. Thus, not having an activation function before the adapter inside the critic network performs as well as the base option, which has this activation. Overall, this result justifies our design choice by showing that performs comparably to better than the alternatives.
28

---

Additional Results
This section contains some additional results that we refer to in the main text.
F.1 Single Dimension Interpolation, Train & Extrapolation
See Fig. 17 for the performance of each model the ODE on the particular context sets, Interpolation, Training and Extrapolation. These plots therefore correspond to subsets of the evaluation range the left pane of Fig. 3.
Interpolation
Extrapolation
200
150
55
Traihing Time (Stepa)
1e5
Unaware
Concat
Adapter
FLAP
Figure 17: Showing the performance of each method in three different evaluation regimes, (left) the training contexts, (center) interpolation and (right) extrapolation. Mean performance is shown, with standard deviation shaded.
F.2 Multiple Dimension Heatmaps
Fig. 18 illustrates the granular performance of each method in the ODE domain, corresponding to the right pane of Fig. 3. Our Decision Adapter outperforms all baselines and, as can be seen in Figs. 18a and 18b, generalises further away from the training contexts than the Concat model. In particular, these heatmaps show that both models perform well on the training contexts (the blue squares). However, the Adapter generalises much better, especially contexts far outside the training range. For example, when |ca| E [8, 10] (corresponding to the far left and far right of the heatmaps), the Adapter's 's rewards are higher than the Concat model's. The Adapter obtains between 50 and 150 more reward than the Concat model which is a substantial difference, as the maximum reward in this environment is 200. This region of context-space far outside of the convex hull of the training range, which contains values of |c only up to 5.0. Closer to the training range; for instance, around the boundary the green square, the Adapter still outperforms the Concat model, but the difference is less pronounced.
FLAP and cGate perform poorly, even though they obtain near-perfect rewards the training contexts. There may be several reasons for this. First, here we train on 16 relatively sparsely distributed contexts, with 300k steps in total, resulting in fewer episodes in each context compared to Section 7.1. Second, with two dimensions, the interpolation range makes up smaller proportion of the evaluation range compared the one-dimensional case. This because, in one dimension, the range [-5,5, 5] makes up half of the [-10, 10] evaluation range. In two dimensions, however, the interpolation range makes up only a quarter of the evaluation set. Thus, the overall average performance in Fig. 3 is more skewed towards extrapolation. This, coupled with the fact that both cGate and FLAP perform poorly on extrapolation (Fig. 17, right), may explain why they underperform in this case. Lastly, it is generally easier to generalise in the one one-dimensional ODE, as optimal action depends mainly on the signs of the context and state variables. By contrast, multidimensional case is more difficult, as the dynamics equation includes nonlinear action term. This is also why, for instance, our Decision Adapter performs worse this case (around 160 overall evaluation reward) compared to the one-dimensional ODE (around 190).
29

---

22
390
36T
30
(a) Concat
(b) Adapter
Someer
(c) cGate
(d) FLAP
Figure 18: Multidimensional contexts in the ODE domain for (a) Coneat, (b) Adapter (c) cGate and (d) FLAP. Heatmap Performance for (a) cGate and (b) FLAP. Each cell represents the reward obtained when evaluated in the corresponding context, averaged over 16 seeds. The blue squares indicate the specific training contexts, whereas the green square represents the convex hull of training contexts. For the heatmaps, the last rows and columns, labelled avg, represent the average reward over the particular column or row, respectively.
F.3 Additional Baselines
We consider two ablations as additional basclines. The first, called AdapterNoHnet, is based on our adapter module, but there no hypernetwork involved. The adapter's architecture and location are the same, but it is now a single MLP that takes in the concatenation of the state-based features and the raw context (as the model has to be conditioen,,,,o,dd, and must also process the state- based features). The second baseline, termed cCateEverylayer uses the same elementwise operation as cGate, but this happens at every hidden layer except at only one. Fig. 19 illustrate these results on the ID and 2D ODE domains respectively, with the Adapter, Concat and cGate models there for reference. Overall, cGateEveryLayer does not outperform cGate, and AdapterNoHnet performs much worse than our syspernetwork-bssed adapter.
30

---

Context
200
150
AER
50
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
1.0
2.0
Training Time (Steps)
le5
Concat
Adapter
cGate
cGateEveryLayer
AdapterNoHnet
Optimal
Figure 19:
The performance on the ID and 2D ODE domains (i.e., similar to Fig. 3) of the additional baselines. The mean and standard deviation over 16 seeds are shown.
F.4 AER Metric
Fig. 20 illustrates the difference between using the AER metric as defined in Section 6.1 (correspond- ing Fig. 20a) and only considering the testing contexts (corresponding to Fig. 20b). Overall, the performance of each model is very similar; this is because the training contexts make up small proportion of the entire evaluation range.
203
1.5
Toa ring Time (Steoct
Baiinng Tima (Srep4))
(a) Training and Testing Contexts
(b) Only Testing Contexts.
Figure 20: Comparing (a) averaging performance over contexts and (b) restricting the AER metric to only unseen testing contexts.
F55 Distractors
The ODE domain displays similar distractor results to other environments (see Section 7.2). In particular, Fig. 21 shows that the Adapter is robust to adding more irrelevant distractor variables, whereas Concat and cGate
Concat
Adapter
cGate
200
100
50
3
2
1e5
Training Time (Steps)
Number of distractor dimensions
20
100
Figure 21: Showing the sensitivity of the (left) Concat, (middle) Adapter and (right) cGate models to fixed distractor context variables on the ODE, The mean and standard deviation over seeds are shown.
E.6 Gaussian Distractor Experiments
This section expands upon the results in Section and contains additional results where the distractor variables are not fixed.
31

---

F.6.1
Difference between training and testing
Fig. 5 contains the results on CartPole, and the ODE results are shown in Fig. 22. For both domains, the conclusion is similar in that the Adapter still more robust to changing distractor variables compared to either Concat or cGate.
Concat
Adapter
cGate
200
100
50
3
2
1e5
Training Time (Steps)
Number distractor dimensions
1
20
100
The Gaussian distractor variable results. This corresponds to using mean of 1 during training and mean of 0 during testing. We use a consistent o 0.2. These results are for the ODE domain, showing the mean and standard over 16 seeds.
Figure 22:
F.6.2 Keeping the training and testing distractor distributions the same
Next, we consider a similar case, but the means of the Gaussians are the same across training and testing. Overall, the results for the ODE and CartPole (with mean of 0 during both training and testing) are shown Fig. 23. For the ODE, while the Adapter insensitive to adding more distractor dimensions, both cGate and the Concat perform worse dimensions to the context. However, the difference is much less pronounced than in the case where the means are different between training and testing. Surprisingly, in the ODE, cGate performs slightly better with 20 distractor variables than with O. These variables may effectively act as an additional, noisy, bias term that the model utilise. For CartPole, the Concat and cGate models perform much worse with 100 distractor dimensions compared to none. The Adapter displays some noise, indicating that some seeds performed poorly while others performed well. This is similar to the results we observed in Appendix G. G-o much noise leads to unstable training.
add more irrelevant distractor
Finally, we consider using mean of during both training and testing in Fig. 24. For the ODE, each model performs similarly regardless of the number of distractor variables. CartPole there is not
Concat
Adapter
cGate
ODE
100
AER
CartPole
400
200
1e5
Training Time (Steps)
Number distractor dimensions
20
100
Optimal Reward
Figure 23: The Gaussian distractor variable results, using consistent mean of ( during training and testing, with o 0.2. These results are for the (top) ODE and (bottom) CartPole domains, showing the mean and standard over 16 seeds.
32

---

Concat
Adapter
cGate
200
ODE
100
AER
CartPole
400
200
1e5
Training Time (Steps)
Number of distractor dimensions
20
Optimal Reward
Figure 24: The Gaussian distractor variable results, using a consistent mean during training and testing, with 0.2. These results are for the (top) ODE and (bottom) CartPole domains, showing the mean and standard over 16 seeds.
large difference between the various models when the mean of the distractor variables is 1 during both training and testing.
F.6.3 Summary
Overall, The Adapter is significantly less susceptible to overfitting to irrelevant distractors compared the Concat model and cGate. However, these distractors remain the same during training and testing (which may not be particularly likely). it matters less whether the model overfits or not, and all models are less affected by adding more distractors.
G Limitations
This section expands upon the limitations of our approach, outlined in Section 8.
G.1 Noisy Contexts
Now, while we use the ground truth context to isolate the effect of the network architecture on generalisation performance, this is not always available [12, 18]. A common approach is to learn context encoder that uses experience in the environment infer an approximation the context, which is then used to act [18, 87J. This encoded context, however, may not always be entirely accurate. Thus, in this experiment, we examine how sensitive each context-aware model to receiving noisy contexts during evaluation. We consider two cases; the first is where the models received the uncorrupted context during training, and the second where the models also encountered noise during training. The first case could reasonably occur in reality, since we may have access to ground truth context during training, but not evaluation [14]. The second case, where we train with noise, could correspond to training the RI policy alongside the context encoder, leading to some noise during training. This case additionally allows us examine whether training with noise increases the models' robustness context noise during evaluation.
We take the models trained during the primary CartPole experiments, Appendix B.2, where varied the pole length. We use the model checkpoints that were trained for 600k timesteps. both the Concat and Adapter models did not improve much by training for longer than this (see Fig. 8). The noise we add here, corresponding to normally distributed noise with varying standard deviations, was added to each context dimension instead just the one representing pole length. We experiment with standard deviation E {0, 0.05, 0.1, 0.2, 0.5, 1}. The noise was added after normalising the contexts such that the largest training context had a value of 1.0. Thus, standard deviation of 1.0 represents substantial amount of noise. In addition to the models trained without noise,
33

---

additionally train models that encountered noise during training. In particular, we consider standard deviations {0.1, 0.5), and change no other aspect of training.
The results are illustrated in Fig. 25. The left pane shows the performance of the models trained without noise. Overall, these results show that the Concat model is susceptible noisy contexts than our Adapter, bolstering the argument that treating context as state dimension is not ideal [13]. Both models, however, generalise worse as the context becomes more inaccurate. This is in contrast to the Unaware model, which performs equally well regardless of the level of context noise. The Adapter still outperforms the Unaware model until the noise level reaches around o 0.5. This noise level. however, is very large, and o 0.5 corresponds to pole length of 3 meters. The default pole length is 0.5 meters, and the largest training length meters.
When training with a small amount of noise (Fig. 25, middle) both models perform similarly to training without noise. However, when training with large amount of noise, corresponding the right pane of Fig. 25, the Concat model becomes more robust to noisy contexts, but the Adapter performs worse. In particular, the Adapter model exhibits large amount of variation across seeds, indicating that some seeds performed badly. On average, the Adapter performs similarly to the Unaware model until the level of context noise increases beyond the training noise of 0.5.
Train Without Noise
Train With 0.1
Train With 0.5
500
400
AER
300
200
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Level of Context Noise During Evaluation (a)
Unaware
Concat
Adapter
Figure 25: Comparing the average evaluation reward (after 600k steps) function of the standard deviation a of the Gaussian noise added to the context during evaluation. In the left pane, the models trained without noise are shown. The middle pane corresponds to training with = 0.1 and the right pane shows the results when training with o 0.5. Mean performance shown, with standard deviation over 16 seeds shaded.
G.2 Suboptimal Context Normalisations
G.2.1 ODE
As discussed in the main text, we first normalise the context before passing it to the agents. This is generally done by using the largest absolute value encountered during training, For instance, for the meedimennioonll ODE experiment, we used Crorm we determine how sensitive the Decision Adapter is to changes in this procedure. This is useful as may often need to choose the normalisation factor without knowing exactly which variations will encounter during evaluation. To do this, we run four additional experiments, with normalisation values of 0.1, 1.0, 2.0 and 15.0, respectively. We train the agents for only 300k steps, as after training for this number of steps, most models performed similarly to their final performance. The results are shown in Fig. 26, and we can see that the Concat and Adapter models are both robust to the normalisation value. Our Decision Adapter loses less performance than the Concat model as we change the normalisation value. Despite losing some performance the context normalisation value increases, both context- aware models still outperform the Unaware model.
as the normalisation function. Here
G.2.2 CartPole
We now consider similar setup in CartPole. In particular, we train on Pole Mass of O.I. We choose this variable as some of our experiments showed that the Unaware model generalises equally well regardless of the training contexts. We consider two scenarios: In the first, we normalise with respect to the Small setting (i.c. Crorm
Thus, during training, the normalised context had value
0.07
34

---

200
180
160
140
AER
120
100
60
40
Unaware
Concat
Adapter
10
Context Normalisation
Figure 26: Showing the overall performance (y-axis) of each context-aware model after 300k steps change the context normalisation value (x-axis). Mean performance is shown, with standard deviation shaded.
Normalisationn 0.01
Normallsation
300
200
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
1e5
Training Time (Steps)
Unaware
Adapter
Optimal
Figure 27: Performance when evaluating on changing Pole Mass when normalising with respect to (left) Mass = 0.01 and (right) Mass = 1.0. In both plots, the models trained on Mass 0.01. Mean performance shown, with standard deviation shaded.
of 1.0; during evaluation, it was nearly always larger than 1.0. In the second case, we normalise with respect to the X-Large setting (corresponding to Cnorm = f). We trained on the Small pole length for both settings. Fig. 27 shows the results for this experiment. When normalising with respect to the Small setting, the Coneat model fails to generalise at all, and the Adapter performs worse than Unaware. When changing the normalisation, however, the context-awan models perform significantly better.
G.2.3 Summary
Thus, the Decision Adapter relatively robust to small changes in context normalisations. However, when encountering evaluation contexts that are 100 times larger than during training, appropriate
35

---

normalisation is crucial and can make drastic difference even when training on the exact same contexts. As seen in the ODE, having normalisation value that is slightly large does cause significant performance penalties. Thus, normalising the contexts with a larger value than expected during training a good heuristic.
G.3 Narrow Context Range
We next consider the effect of changing the set of training contexts. Here we aim to briefly illustrate that the training context range can have a large effect on the performance of the agents. In particular, we consider the ODE domain and separate sets of training contexts. In all of these cases, we keep the normalisation consistent Cnorm = ensure comparability. The results when training on each context set are shown in Fig. 28. Overall, when we have single positive and negative training context, performance is poor when this context very small (a), and increases as becomes larger (d and f). When we have multiple contexts, but insufficient variation (b, e), then performance is also suboptimal. For instance, in (b), the training contexts cover only the small region [-1, 1]. In (c), the training set of contexts does not contain any negative contexts, leading to poor generalisation. Finally, contexts are varied, but spread out too far (g), the performance also suffers.
200
150
0
(a): (=0.1)
(b): (10.1, x 1.00
(c): (1,5)
(d): (=5)
AER
150
100
1
2
0
2
0
2
3
1
3
(c): (14, 25)
(f): (27.5)
(g): (+0.1, + 7.5}
(h): (21, 2
Training Time (Steps)
Adapter
Figure 28: Here examine the performance when training different sets of contexts. The text beneath each plot indicates the training context set, with =c indicating that both c and -c are the context set. For instance, in (a), the models trained on context set of Ctrain (-0.1,0.1) Mean performance is shown, with standard deviation shaded.
G.3.1 Overfitting
Another potential issue we may encounter is overfitting, which can occur when the training range is not diverse enough. To illustrate this problem, conduct experiment in the multidimensional ODE setting using the following set of training contexts:
(11, (1, - (-10,, (-1, -11,,11,, (-1,0), (0,
The results, presented in Fig. 29, demonstrate that the Decision Adapter performs well initially but its generalisation performance suffers as training progresses. Specifically, as shown in Fig. 29b and Fig. 29c, our model exhibits worse extrapolation performance due to overfitting on the narrow training contexts.
36

---

140
100
AER
60
40
20
Unaware
Concat
Adapter
3.0 1e5
Training Time (Steps)
(a)
179
100
DA
129
32
90
(b) 50k steps
(c) 300k steps
Figure 29: Showing (a) the overall performance over time on the entire evaluation range. In (b) and (c) show the performance of the Adapter at two points in training. In (a), plot mean performance and shade standard deviation; for (b) and (c), we show only the mean over the seeds.
37


Reviews:
[{"review_id": "TqDcdnsDKy", "reviewer": "Reviewer_8dZQ", "summary": "The paper presents an architecture that uses a hypernetwork to learn network parameters for a context-dependent policy adapter. The authors discuss an illustrative example that emphasizes in what cases a universal (context-dependent) policy is necessary, and in which cases a non-context-conditional (\"unaware\") policy is sufficient. In the experiments, they compare their adapter approach with a context-unaware policy, a policy that concatenates context information with state and processes them equally, and other approaches using a linear policy adapter (FLAP and cGate).", "questions": "- In 6.1: Would it not be possible to calculate the average evaluation reward only on an entirely unseen set?", "limitations": "The authors discuss limitations in quite some detail in section 8, and underpin this with additional experiments in the appendix. This is a strength of the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea to hyper-learn a nonlinear context-dependent adapter as the final few layers of a context-general policy is original and makes sense. The paper is well written, and experiments are quite thorough. The relatively detailed ablation experiments, some of which are found in the appendix, are quite interesting and a strength of the paper, for example the study of nuisance context parameters in section 7.2. I also think that the distinction between extrapolating and interpolating generalization is important and rarely seen (presumably because this distinction is usually hard to make in more complex environments), and interesting enough to maybe move it to the main part. General policies are an important and active research field, and the paper makes a significant contribution.", "weaknesses": "I appreciate the illustrative example for unaware vs aware policies in section 4, but I feel like this is quite intuitive for many readers, and could potentially be shortened a bit in favor of more experimental results in section 5. Also, I was a little confused by the section title being \"Theoretical Foundations\".\n\nI think that the related work section should probably discuss recent advances in foundation-model-style generalist agents (e.g. https://arxiv.org/abs/2203.11931, https://arxiv.org/abs/2305.10912, https://arxiv.org/abs/2301.09816, https://arxiv.org/pdf/2306.11706.pdf) as well. The authors briefly touch on \"learning some unsupervised representation of the problem and inferring a latent context from a sequence of environment observations\", which is similar in spirit to this line of work, but then only cite one reference. Considering that this field has been quite active lately, it probably would make sense to discuss this briefly.\n\nGenerally, I think the paper would benefit from experiments in a larger and more diverse number of experiments. However, I think this is at least partly compensated by the fact that the authors report relatively detailed results for the two (or three) environments they consider.\n\nMinor comments:\n- The shaded areas in Fig 3 and elsewhere are very hard to read", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank the reviewer for their kind words. The endorsement of our work is greatly appreciated. We hope to address the reviewer's remaining concerns and increase the certainty of the reviewer in their endorsement.\n\nOn the point of shortening section 4: Thank you for noting the benefit of this section. We do believe section 4 motivates the need for our work - by illustrating a case where the unaware model fails. However, section 4 also notes that always being context aware is not entirely necessary. This is a necessary point in a work which aims to answer *when and how* to incorporate context. Moreover, we present a strong theoretical result which shows that, in a case where context is necessary, ignoring it becomes more costly as the number of different contexts increases. However, we will shorten this section while still keeping the illustrative and rigorous essence of what we have now.\n\nOn the point of the foundation-model-style generalist agents: We thank the reviewer for directing our attention to this important area of work. We will certainly add this to our related work and believe this grounds our work further in the literature.\n\nOn the point of having more experimental domains: We believe the three domains we considered represent a broad range of environments and constitute a representative selection of environments. In particular, the ODE domain is a particularly general environment, as many problems can be phrased as ODEs. Furthermore, CartPole is a relatively simple classic control environment that is widely used in the literature [1,2]. Finally, the Ant domain is a more complex and high-dimensional continuous control task and makes the applicability of our work to robotics explicit. We appreciate that the reviewer does acknowledge the detailed nature of our results in these domains, and also for their time in reading the Appendix to note the depth of our results. Indeed, there is a trade-off between the breadth of experimentation and depth of results, given a conference length page limit. We hope that we have navigated this trade-off appropriately by deeply exploring the representative and general domains. We will aim to bring more of our additional experiments into the final version of our paper - as the reviewer recommends - which we hope will further justify our experimental strategy. We thank the reviewer for their guidance on this point.\n\nOn the point of the shading in, among others, Fig. 3: We will rectify this in the revised paper. We thank the reviewer for pointing this out.\n\nFinally, on the reviewer's question regarding the definition of the AER: It would be possible, however the results do not differ significantly (see Figure 4 in the attached PDF). We opted for averaging over the full context range as it provides a better metric of the general performance of the agent, however we can include the figures for the revised version.\n\n\n*References*\n\n[1] Steindór Sæmundsson, Katja Hofmann, and Marc Peter Deisenroth. Meta reinforcement learning with latent variable gaussian processes. In Amir Globerson and Ricardo Silva, editors, Proceedings of the Thirty-Fourth Conference on Uncertainty in Artificial Intelligence, pages 642–652. AUAI Press, 2018.\n\n[2] Carolin Benjamins, Theresa Eimer, Frederik Schubert, Aditya Mohan, André Biedenkapp, Bodo Rosenhahn, Frank Hutter, and Marius Lindauer. Contextualize me - the case for context in reinforcement learning. CoRR, abs/2202.04500, 2022"}, {"review_id": "pu1MonBHJb", "reviewer": "Reviewer_sPQr", "summary": "The paper studies how to better incorporate context (such as MDP-specific parameters) in context-conditioned robot policies. Traditionally, this is done simply by concatenating the context vector to the state vector and processing both inputs simultaneously with the policy network. This work proposes a context-conditioned hypernetwork to output the weights for an adapter module, which adds a residual value to the policy’s intermediate representation. Along with other proof-of-concept tasks, the paper evaluates on cartpole (varying pole length) and mujoco ant (varying mass) and finds that the proposed method is more robust to distractor dimensions but does not necessarily perform better than naive concatenation methods.", "questions": "(1) How might this framework be improved to incorporate multiple contexts at the same time? For instance, both a goal image and language instruction for a given task. If it is known that some context modality is more beneficial on average to inducing strong task performance than some other context modality, how might we handle that with adapters and hypernetworks?\n\n(2) I don’t understand how Theorem 4.5 relates to the rest of the paper, or even how it guided the experimentation toward the proposed adapter + hypernetwork architecture.\n\n(3) How would the proposed method perform in image-based domains where the main policy network is a CNN or transformer? It seems like the main results are working with an MLP operating on robot state. My main concerns are that this work may be useful primarily for simpler, older architectures and might not prove as helpful for transformer-based robotics policies that are moving toward cross-attentional context conditioning (such as VIMA [Jiang et al 2023]).\n\n(4) Perhaps include a citation of FiLM (Perez et al 2017), since it is a widely used context conditioning method and does a sort of context-conditioned “adapter” with a residual connection as well, which is similar to the proposed method but does not have a hypernetwork. A comparison would be helpful too.\n\n(5) Most of the crux of the experimental section hinges on the argument that the adapter is a better context conditioning architecture because it is more robust to distractor context variables. However, the distractor variables were set to some pretty contrived values--all to 1 during training and 0 during testing. How were these values chosen? Why not just gaussian noise during both training and testing, which seems more likely for a model to encounter, in which case concat will probably be as good as an adapter at filtering out the distractor dimensions?", "limitations": "Authors were upfront of the limitations of the adapter to noisy contexts and its tendency to overfit when trained on narrow context ranges and had a very detailed limitations section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors study a problem that is important in multitask learning/generalization such as with respect to language/goal-conditioned policies.\n\nThe paper is well-written and proves some interesting theoretical results. Empirically, experiments test generalization in a meaningful way. For instance, the evaluated mass ranges are not within the training ranges in the ant environment, which is a challenging problem in robotics.", "weaknesses": "(1) Section 4 talks about performance guarantees for a context-unware policy. Why is this important to the rest of the paper, which is about how to best incorporate context into a policy for dynamics generalization? Furthermore, I don’t see the link between Lines 154-163 (talking about how context-dependent rewards are equivalent to context-dependent dynamics in a context of matrix rotation example) to the rest of the section or the rest of the paper.\n\n(2) The abstract promises to “establish conditions for when it is necessary to incorporate context,” but when reading the text, I did not see any section of the paper that brought this out explicitly.\n\n(3) The work would really benefit from architectural ablations, especially when the abstract said the paper would “examine how to best leverage this external context information to improve generalization.” Hypernetworks are not a common architectural component in context-conditioning, so were alternatives not involving hypernetworks considered? For instance, it would be good to compare with the following architectural results:\n(i) The proposed approach, with no hypernetwork, where the adapter module (a simple MLP) only takes the context as input.\n(ii) Something like cGate, but with an elementwise product with the context embeddings at every layer, instead of just at one layer.\n\n(4) Figures 4 and 5, which contain the most difficult domains examined by the paper (ant and cartpole), show that in the 0 distractor dimension case, naive concatenation performs roughly the same as the proposed method (adapter). This means that in this case, naive concatenation is preferable due to its simplicity and training speed relative to the proposed adapter. It is only in the differential equations domain, a non-control task, where adapter clearly outperforms naive concatenation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed review. We hope that we can address the reviewer's concerns and questions. In particular, we would like to direct the reviewer's attention to the additional PDF with the requested experiments of the additional baselines, which we elaborate upon below.\n\nRegarding the noted weaknesses:\n\n- 1. and 2. and Q2. Please see the general response for more elaboration. In essence, we believe Theorem 4.5 motivates the need for context-aware policies. However, the theorem, and its link to the rest of the work should be explained more clearly. We thank the reviewer for raising these points and helping us tie all results of this work together better.\n\n- 3. We agree that a more in-depth look into architectural ablations is important and we thank the reviewer for giving concrete suggestions. We performed these experiments on the ODE, with the results in the attached PDF (Figure 1). We considered:\n   - `AdapterNoHnet`: This is our adapter module without a hypernetwork. The adapter's architecture and location are the same, but it is now a single MLP that takes in the concatenation of the state-based features and the raw context (as the model must be context-conditioned, and must also process the state-based features).\n   - `cGateEveryLayer`: This is exactly what the reviewer suggested, the same elementwise operation as cGate, happening at every hidden layer instead of just one.\nBoth of these ablations perform worse than our adapter and cGateEveryLayer does not outperform the standard cGate.\nWe have also run several additional ablations investigating certain aspects of our adapter. We considered (1) the adapter module's architecture, (2) the hypernetwork's architecture and (3) the location of the adapter. For space reasons, ablation (2) is shown in Figure 3 in the attached PDF, and we will add the others to our paper. Overall, these experiments show that the adapter is generally robust to these hyperparameters, with most values performing comparably. We believe these additional results further strengthen our work.\n\n\n- 4. We believe the concat model performing comparably to the adapter when no distractor variables are present demonstrates the source of the concat method's vulnerability. Specifically, treating context as a state feature results in overfitting. The hypernetwork, however, is more robust and less prone to generate weights for the policy network that are overfit. Thus, in a setting where the context is completely reliable, and the notion of overfitting is not applicable, then concat and the hypernetwork are indeed comparable (in some domains). This is an important finding for a work which aims to identify *when and how* to incorporate context in RL. In the case of completely reliable context, then, it is unnecessary to use a more sophisticated model. However, in light of the other literature in context-sensitive RL, this is unlikely to be true in many cases - particularly those in which context is inferred or we are unsure about which context variables to incorporate. We thank the reviewer for drawing our attention to this point, and we see that this should be explained more fully in the paper as it is an important one. In sum, *when* your context may include distractors then *a hypernetwork is the appropriate model*\n\n\n\nWe next consider the reviewer's questions:\n\n- 1. and 3. One way to incorporate multiple context modalities would be to have one or more models in front of the hypernetwork that serve to transform the different modalities into embeddings (i.e., continuous vectors) which are subsequently concatenated. We would also argue that by using a hypernetwork it is possible to then learn, based on the task, which context information is more valuable. Indeed, if most of the useful contextual information can be found in one modality then this would make the other modality akin to a distractor context. Based on our results, this would suggest that the hypernetwork would appropriately handle this situation. Similarly, we believe there are no reasons that our approach would fail in image-based domains. One approach would be to have a CNN model process the image observation into a flattened vector, which is then passed to an MLP that outputs the action. We could add our hypernetwork to this MLP. We believe that we have taken a step towards robust context-sensitive RL agents in this work and are excited to see that the reviewer has immediate directions of future work in mind. Unfortunately, multi-modal learning (let alone multi-modal RL) is a challenging and important problem in its own right. Thus, it would seem to fall outside of the scope of this work.\n\n\n\n- 4. We apologise for the oversight of not citing FiLM, we will discuss it in the revised paper. We note that cGate is functionally very similar to FiLM, as both perform a single elementwise multiplication between the main (state) and external (context) features. Therefore, FiLM, like cGate, is still linear, in contrast to our model. Finally, we chose to focus on cGate as it was more recently introduced than FiLM, and was specifically introduced to address the problem of generalising to new contexts (as opposed to FiLM which is quite general).\n\n- 5. Regarding the distractor experiments: here, we aim to study the effect of having distractor variables in the context (that do not affect the dynamics), but that have different values between training and testing. To this end we chose $0$ and $1$, to indicate that during training, we had some context variables at particular (but fixed) values $X$; during testing it changed to other (still fixed, but different) values $Y$.\nThus, to the point of ``Why not just gaussian noise during both training and testing'' - there is no guarantee that the sampled context values will be meaningfully different to demonstrate a true test set of contexts. Thus we required a more explicit approach.  We will clarify and expand upon this explanation in the revised paper.\n"}, {"review_id": "y060FqYS9S", "reviewer": "Reviewer_DXBg", "summary": "This paper addresses the problem of learning policies that can generalize to multiple contexts. The problem is formulated as a contextual MDP (CMDP). The paper starts by analyzing analytically the performance of policies that are not aware of the context in two different scenarios of a particular instance of CMDP. After that, a network architecture (decision adapter), which incorporates contextual information through a hypernetwork, is introduced. The architecture is empirically evaluated in three environments and its performance is compared against 4 baselines.", "questions": "1. Could the authors clarify why the results of cGate in CartPole and Mujoco Ant are not provided?\n\n2. The paper attributes cGate to [13]. However, I did not find any reference to cGate in [13]. Could the authors clarify this?\n\n3. Line 193 says *\"For this and the next section, we consider only the Concat and Unaware models as baslines\"*. However, the results for the unaware policy are not reported in Figures 4 and 5.", "limitations": "The limitations of the approach are discussed extensively in Section 8 and Appendix G.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper tackles an important problem in RL. The idea of using a hypernetwork to incorporate contextual information is sound and the empirical results are promising. The paper is very clear and easy to follow.", "weaknesses": "I believe that the paper's novelty is somewhat limited. While I found the solution of using a hypernetwork quite neat, I was slightly disappointed to discover that a similar architecture called cGate had already been proposed to address the same problem. The cGate architecture is mentioned for the first time in Section 5 and is not introduced as a related work.\n\nThe authors argue that the adapter architecture is a more powerful generalization of cGate as it can also capture non-linear relationships between states and contexts. This is confirmed by the empirical results obtained in the ODE domain. However, the results of cGate on the other two environments (CartPole and Mujoco Ant) are not reported. Moreover, the ODE domain seems to be specially designed to make cGate fail, as the dynamics are given by an nth-degree polynomial (equation Line 229).\n\nFinally, I found the example provided in Section 4 to be very helpful for understanding. However, in my view, framing the analysis o as a theorem is not appropriate in this case, since the applicability of the results seems limited to this specific instance of the problem. Typically, a theorem should offer broader insights into the overarching problem at hand.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's acknowledgement of the importance of this problem in RL, and appreciate their saying that our paper is clear and easy to follow.\n\nRegarding the weaknesses raised by the reviewer:\n\nFirstly, we agree that cGate should have been discussed more thoroughly in the related work and we will rectify that in the revised paper. Secondly, cGate (which to date has only appeared in an Arxiv submission - however we are aware that NeurIPS policy is to consider this in prior work) is indeed a special case of our model. Although, it was not designed or introduced as a hypernetwork and conceptually our work is the first to take this perspective of cGate. Thus, we have drawn the comparison with cGate in an effort to accurately ground our work in the literature - but only with our change in perspective on cGate is it clear that an adapter layer with hypernetwork is an appropriate more general model. However, we emphasise that this requires an understanding of the mathematics driving cGate and was not a conceptually trivial point. Thirdly, we have run more experiments to ascertain the benefit of our model over cGate and indeed see that our model provides an empirical advancement whether or not the context contains distractor variables. As to the point that the ODE setting is adversarial towards cGate due to the use of an n-order polynomial. We would like to emphasise that the context provides the coefficient of this equation and no power is applied to these coefficients. In other words, the context features being input to cGate are appropriate for learning the task (unlike if some context $c_i$ features was used when $c_i^2$ is in the equation, in which case this would be needlessly adversarial). Indeed, cGate is able to learn on the trained contexts and the appropriate function is in the hypothesis space of the cGate model (see Fig. 3 in the main paper, cGate vastly outperforms unaware, but does perform worse than concat. In addition, Fig. 10 in the appendix shows that cGate is able to perform optimally on the training contexts, it suffers primarily in extrapolation).\nFinally, to say that ``cGate had already been proposed to address the same problem'' is not quite accurate. This is due to the second point - it was not introduced as a hypernetwork, and moreover, cGate was not used to adapt to distractor context variables. It could not have because, as our new experiments show, it offers no benefit there over the concat model. We thank the reviewer for pointing out the need for us to be clearer about the origins of cGate so that readers are better able to determine the contributions of our work. We hope that we have clarified the relationship between cGate and our hypernetwork, as the hypernetwork is more than just cGate with a nonlinearity.\n\nThird, we aim to have Theorem 4.5 as a particular example (and formal showcase) of certain cases where (a) an unaware model fails and context is necessary and (b) an unaware model succeeds and context is not necessary. However, based on this reviewer's and the other reviewers' comments, we will sightly shorten our theoretical section to make room for additional empirical results (see the general comment above). Regarding the scope of Theorem 4.5, we agree with the reviewer that it is not as broad as a theorem generally should be, and will instead refer to it as Proposition 4.5\n\n\nRegarding the reviewer's questions\n\n1. We only provided one other context-aware baseline, the concatenate model as a representative baseline of this class of methods. We believe this model is representative, as it is very widely used, and performed similarly to, or outperformed cGate in our prior experiments. However, in the attached figure PDF, we have the results for cGate on CartPole and Ant (Figure 2). We note that cGate fails in a similar fashion to the concatenation model in CartPole. In Ant, cGate also exhibits reduced performance when the number of distractor variables increases. In addition to this, cGate performs worse than the Adapter and Concat models when no distractor variables are present in Ant (at the end of training, cGate has a mean reward of around 2800, whereas the adapter/concat models have rewards of around 4000). We will add these results for CartPole and Ant to the revised paper as we think they further support the use of our model.\n\n\n2. Regarding reference [13], we noticed that the authors of [13] uploaded a new version of their paper to Arxiv, on the 2nd of June 2023 (after we submitted our paper). Their updated paper is significantly different to the one we cite (which is ``v1'' on ArXiv) and removes mention of cGate entirely. This is a slightly unusual case to be in as this would depict cGate as no-longer being prior work (at least based on the latest state of ArXiv). However, to be fair to the authors of [13] we will cite v1 and v2 of this paper separately. Hopefully then readers who would like to understand cGate can find the earlier version but for all other points where we cite [13] the  most up-to-date version will be used. As mentioned above we will still add [13] v1 to our prior works section, however, it seems prudent to avoid grounding our work too fully in an out-dated version of this paper.\n\n3. Thank you for pointing out that issue. We apologise for the oversight. The focus of these experiments was primarily on how the concatenate model differs from the adapter, in its robustness to distractor context variables. Therefore, the unaware model is not a valid comparison in this case and we will remove the mention of the unaware model in this line.\n"}, {"review_id": "jwPcpxfl8I", "reviewer": "Reviewer_rS79", "summary": "The paper proposes a novel way of learning how to adapt policies in contextual RL. The work proposes to use \"adapter\" modules whose parameters are determined by a hypernetwork that solely focuses on the context variable. ", "questions": "* Will the code be made available?\n* Why is the pre-training phase (as done in NLP) not used for adapter? I would imagine that the hypernetwork would benefit from that and only need to \"fine-tune\" the generated weights to work well with the observed reward signals.\n* With many references to the work of Benjamins et al. 2022, why are not more environments from the proposed CARL benchmark used?\n* Intuitively it makes more sense to me to describe the Theoretical foundations in Section 4, using the optimality gap as defined by Benjamins et al. (2022). What is the advantage of using an Optimality ratio instead?\n* Are the same context distributions used in the cartpole example as done by Benjamins et al. (2022)? ", "limitations": "Limitations are discussed thoroughly and experiments are provided that support the discussion on limitations.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper is generally easy to follow and well written. Illustrations are well placed and utilized to communicate the content of the paper and to emphasize the concepts that are being discussed.\nThe work also addresses an important issue for RL, namely generalization, by combining existing ideas in clever ways to tackle this issue. I particularly commend the authors for the discussion of the limitations and the experiments that are conducted to explore the limitations. Overall, the presented method \"adapter\" seems well suited to be used for training more general RL agents. As such I believe the work is of high significance for the RL community and should inspire future works.\n\nOverall I think this paper is a clear fit for the NeurIPS (RL) community", "weaknesses": "The presentation could be improved a bit. For example:\n* The title does not seem to fully fit with the work, or rather, it is overly general and gives the impression that a novel cRL formulation will be presented.\n* Similarly, the abstract only spends ~1/3 of the space discussing the \"Decision Adapter\" method\n* The wording in line 45 makes it seem like context could not be observed by a sensor. However, the robot mass example would be a sensable context.\n* The restriction on only changing dynamics and not changing rewards seems not really needed.\n* In line 115, Procgen is cited for examples where context unaware policies can work well, but I'd argue that the context is embedded in the visual state representation\n* Theorem 4.5 can only be understood when reading the Appendix. This is the case due to $\\mathcal{C}\\_{far}$ and $\\mathcal{C}_{close}$ never getting defined in Section 4 but only in the appendix.\n* Similarly, its easy to overlook why the inequalities in Definition 4.3 make sense, since it depends on the reward definition. \n* The theoretical results from Section 4 seem a bit arbitrary since they heavily depend on the construction of the example. It might make sense to move this part to the appendix and move some more of the empirical evaluation to the main paper.\n* The AER metric is introduced but \"AER\" does never show up again in the paper. Is it maybe labeled as \"Mean Eval. Reward\" in the plots?\n* It is not clear why AER can not be defined over a context set that is distinct from the training set\n* The citation to Benjamins is by now a TMLR publication\n\n\n\nThere is one more niche paper using cRL for learning dynamic configuration of AI planning algorithms that should be of interest to you:\n* Biedenkapp et al. (2022) https://rlplab.com/papers/biedenkapp-et-al-icaps2022wsprl.pdf (follow up to Speck et al. 2021 https://ojs.aaai.org/index.php/ICAPS/article/view/16008)\n\nThis work discusses how to treat state and context information separate and also discusses different architectural choices that one can make in this regard. I don't think this would form a reasonable baseline to compare against, however it feels like an appropriate reference. In particular, it feels like a good reference since the work considers 300+ noisy context features and how to learn with them, compared to the relatively few context features considered in the present work.\n\n\nWhile I listed many bullet points for potential improvement, I am very happy with the state of the paper and I think it should be accepted. I am even willing to increase my score if code will be released.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's kind words, their thorough review, concrete suggestions for improvement, and endorsement of our in-depth limitations section.\n\nWe thank the reviewer for their particular suggestions for improvement, and we generally agree with each point raised. We will rectify each of these points in the revised paper. \n\n> I am even willing to increase my score if code will be released.\n\nWe will publicly release our code once the anonymity period is over. In addition, we have provided a link for the AC to a preliminary (anonymised) version of the code.\n\nPlease see below for more thorough responses:\n\n- P1-2: We will revise the title and abstract to better reflect the paper's content.\n\n- P3: We did not intend for line 45 to convey this particular meaning and we will clarify it in the revised paper. Our intent was to highlight the qualitative difference in timescale with which state and context change.\n\n- P4: Regarding the restriction on only changing dynamics and not rewards, for clarity we chose to be explicit about defining the restricted problem. However, we do agree that the applicability of our model may be wider. We will mention this in our conclusion.\n\n- P5: We thank the reviewer for noticing the potentially incongruous citation to Procgen. We rather meant to focus on the idea that no explicit context variable was incorporated into the agent, and yet it learned a policy that could generalise well. However, we agree that the context in many Procgen games is embedded in the state. We will remove the mention of Procgen here and discuss it in a more appropriate location.\n\n- P6-7: We will provide the definitions in the main paper. We thank the reviewer for pointing this out.\n\n- P8: We believe formally illustrating example cases where (a) an unaware model fails and (b) where context is not necessary is important for our work. We have also aimed to make the construction of the setup as general as possible. Thus, we believe this represents a broader class of problems. However, we agree that section 4 could safely be shortened, with some of it being moved to the appendix and we will do so. Please see the general comment for more on this point.\n\n- P9: We apologise for the oversight of not being consistent with using the name AER in our plots. The reviewer is correct that \"Mean Eval. Reward\" is the same as AER. We will rectify this inconsistency in the revised paper.\n\n- P10: It would be possible to define AER in this way, however, the results are very similar (see Figure 4 in the attached PDF). We opted for averaging over the full context range as it provides a better metric of the general performance of the agent, however, we can include the figures for the revised version.\n\n- Regarding Benjamins et al.: While the paper was accepted to TLMR after we submitted our paper NeurIPS, their revised paper is substantially different from the paper originally uploaded to Arxiv. In particular, their revised version no longer introduces or discusses cGate. We will clarify this in our revised paper, and cite both the new TMLR paper (when discussing the work on benchmarks and context in general) and the old Arxiv one that introduced cGate (only when explicitly mentioning cGate).\n\n- Finally, thank you for listing the work of Biedenkapp et al. (2022). We believe the work is indeed relevant and will discuss it in our revised paper.\n\nWe next consider the reviewer's questions:\n\n\n- Q2: Our focus is on zero-shot rather than few-shot generalisation (where fine-tuning would be very relevant), so pre-training was not a primary consideration. However, if we were to explore pre-training and fine-tuning, some challenges arise that could be considered by future work: pre-training the entire model raises questions about context selection for pre-training vs fine-tuning, while pre-training only the main trunk limits us to a single context during pre-training.\n\n- Q3: We focus on three environments, ODE, CartPole and Ant. The latter two are in the CARL benchmark, although we used a different implementation. We believe these domains represent a broad range of environments and constitute a representative selection. In particular, the ODE is particularly general and CartPole is a relatively simple classic control environment that is widely used in the literature. Finally, Ant is a more complex and high-dimensional continuous control task and makes the applicability of our work to robotics explicit. We note that there is a trade-off between the breadth of experimentation and depth of results, given a conference length page limit. We hope that we have navigated this trade-off appropriately by deeply exploring the representative and general domains. We will aim to bring more of our additional experiments into the final version of our paper - as the reviewer recommends - which we hope will further justify our experimental strategy.\n\n- Q4: We chose to use the optimality ratio because the optimality gap depends heavily on the reward scale of the environment. For instance, a ratio of 0.1 signifies 10% of the optimal reward. In contrast, the optimality gap can yield varied results, such as 900 (e.g. 1000 - 100), 9, or 0.9, for the same scenario.\n\n- Q5: Our approach differs from Benjamins et al. (2022) [The TMLR version] as we use broader distributions. For example, regarding pole length, we trained on discrete values of 1, 4, and 6, while testing on a range of 0.1-10. In comparison, they tested a narrower range of 0.048-1.19. Our wider range emphasises a clearer distinction between interpolation and extrapolation performance, which we think is an important one to make (as reviewer 8dZQ also notes).\n\nIn closing, we again thank the reviewer for their thoughtful comments and overall endorsement of our work. We reaffirm our commitment to publicly releasing our code. We are also committed to actively engaging during the discussion period, and will gladly work to address any further concerns or questions the reviewer may have.\n"}]

Author rebuttal:
We thank all of the reviewers for their time and effort in reviewing our paper. We appreciate the reviewers' thoughtful comments, suggestions for improvement and generally kind words. In particular, we appreciate several of the reviewers praising the depth of our limitations section. We believe this is an important part of our work and are pleased that this was not overlooked.

There were some points raised by multiple reviewers, which we have addressed in the individual responses, but we will also add a global response here.

**Section 4/Theorem 4.5**
We originally added section 4 to motivate the need for context-aware policies, by illustrating a case (i.e., some conditions) where the unaware model fails. Section 4 also notes that always being context aware is not entirely necessary. We believe this is a necessary point in a work which aims to answer *when and how* to incorporate context. Furthermore, we added the explanation in lines 154-163 for completeness, to show how the theoretical problem we consider in section 4 is applicable to the problem of generalising to new dynamics that we consider in the rest of the paper. However, we realise that, due to the intuitive nature of our theoretical setup, this is already intuitive to most readers. Thus, we will move this to the appendix (for readers who still seek such rigour) but agree that the space could be used for other purposes.

However, based on all of the reviewers' comments, we will shorten this section (among others, moving lines 154-163 to the appendix, etc.) to make room for additional empirical results, without losing the essence of what we have now. Furthermore, we believe this section can be better contextualised and linked to our results. For instance, Figure 3 - where the Unaware model fails - on the ODE domain is an empirical illustration of Theorem 4.5(i). Figure 7 in the appendix illustrates that CartPole is an example of Theorem 4.5(ii), as the unaware model can learn a general policy that works on several contexts.

**AER Metric using only the testing contexts**
Regarding the primary metric we use to measure performance, the *Average Evaluation Reward (AER)*, we originally defined it over the entire testing context range, of which the training contexts were a subset of. We can readily change the definition of this to exclude the training contexts. However, since there are often only a handful of training contexts, the results would not look very different visually. Moreover, the conclusions will be identical. To demonstrate this fact we have added a new figure to the attached PDF containing the same result as Fig 3. in the main paper, just without the training contexts (Figure 4). We can also replace the figures in the current draft of our paper with these new ones, if so desired.


**New Rebuttal Figures**
In the attached one-page PDF, we have four new figures. We make it clear which reviewer requested each figure, however, we hope that they are of interest to all reviewers and further strengthen our submission. We would like to once again thank the reviewers for their thoughtful comments as we intend to include many of these new results in our revised draft. The figures are as follows:

1. (Figure 1) [Reviewer sPQr] Results on the 1D and 2D ODE domains of the suggested ablation methods.
2. (Figure 2) [Reviewer DXBg] The results of cGate on CartPole and Ant when adding distractor variables.
3. (Figure 3) [Reviewer sPQr] One of the adapter ablation plots that we will add to the revised version (in addition to several others, we show just this one for space reasons). This figure compares the effect of changing the Hypernetwork's architecture.
4. (Figure 4) [Reviewer 8dZQ, rS79] Showing the result of the AER metric using only the testing contexts. These results are very similar to the original results. Most importantly, our original conclusions all remain unchanged.


Meta-review: This paper studies the problem of learning policies that are performant under many different system dynamics (e.g., walking on different surfaces). The paper theoretically analyzes when being context-aware is useful and when it is unecessary. This analysis is done for the limited case of goal-conditioned learning. The paper then introduces an adapter module on a policy network with weights that are set based upon the context. An empirical study shows that this architecture for context-aware policy learning is effective at training generalizable policies.

Main Strengths: Reviewers found the approach novel and the use of hyper-networks within RL to be a significant contribution.

Main Concerns: The main weakness I note is that the theoretical analysis seems limited as it only focuses on the goal-conditioned setting whereas the paper is really focusing on more general contexts. One reviewer (DXBg) noted some similarities with an earlier work and recommends that these are discussed. The similarities did not limit novelty sufficiently for this to be a major concern.

Overall, the paper introduces a novel neural network architecture that may be of broader interest in the RL community and the paper shows this architecture to be useful for training more generalist policies. I recommend acceptance.
Decision: Accept (poster)