SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We can obtain the context-dependent dynamics formulation under the assumptions in Section 2, i.e., that only the transition dynamics change, as follows. Suppose the context defines a rotation matrix  $C_i \in \mathbb{R}^{d \times d}$ , with dynamics equation  $T(s, a) = s' = s + C_i a$  and initial state  $s_0 = 0$ . Here the goal is a fixed location  $g \in \mathbb{R}^d$ . The episode would terminate with a reward of 1 if  $||s' - g|| < \bar{\tau}$ . We can then perform a coordinate transform to obtain  $C_i^{-1}s' = C_i^{-1}s + a$ . The goal would then depend on the context  $g_{c_i} = C_i^{-1}g$  and the effect of an action in this new space would be consistent across contexts.
This is because, while the centers of the circles  $c_{i}$  and  $c_{j}$  are further than  $4\bar{\tau}$  apart, their closest edges are only at least  $2\bar{\tau}$  apart.
In essence, we change the reference frame to be that of the agent, leading to separate goal locations for each context, but consistent dynamics—corresponding to the problem in Assumption A.1.
In the remainder of this section, we link our theoretical and empirical results, showing that the ODE (for some context distributions) is an example of a non-overlapping environment, while in CartPole, an unaware policy can perform well.
B.1 ODE
Here we show that, in the ODE domain, the Unaware model can either perform well or poorly, depending on the context set it is evaluated on. In particular, we train models on context sets  $\mathcal{C}_{train}^{A} = \{1,5\}$  and  $\mathcal{C}_{train}^{B} = \{-5, -1, 1, 5\}$  and evaluate on  $\mathcal{C}_{test}^{A} = [0,10]$  and  $\mathcal{C}_{test}^{B} = [-10,10]$ . These results are shown in Fig. 7. When only considering the positive contexts (left), the Unaware model performs only slightly worse than the Concat model. This corresponds to case (ii) in Theorem A.5, where the contexts are similar enough for a single policy to perform well. However, when considering both positive and negative contexts (right), the Unaware model fails completely, and performs much worse than the context-aware ones. This is because, in the ODE domain, positive and negative contexts cannot be solved using the same action. This corresponds to case (i) in Theorem A.5, where making progress on one context results in negative progress on another.
Figure 7: Here we show results in the ODE domain. On the left, we train on  $\{1,5\}$  and evaluate on [0, 10]. On the right, we consider both negative and positive contexts, training on  $\{-5, -1,1,5\}$  and evaluating on  $[-10,10]$ . The Unaware model performs much better when only considering positive contexts, even though the context-aware models still outperform it.
B.2 CartPole
We now investigate CartPole. In particular, we vary only the pole length, and consider the training contexts as  $\{1,4,6\}$ , with the evaluation contexts being 301 equally-spaced points between 0.1 and 10.
These results are shown in Fig. 8. While the Unaware model can achieve comparable generalisation performance to the context-informed models, it takes significantly longer to do so. In particular, to obtain an average reward above 400, the Unaware model must train for more than 600k steps, whereas cGate, Concat and our Adapter reach this threshold before 100k steps. This shows that, while context is not necessary in this domain, incorporating it may be beneficial and can lead to much more sample-efficient generalisation compared to the Unaware model. This is particularly relevant when training on multiple training contexts, which seems to cause interference for the Unaware model. The context-aware approaches, however, are not as susceptible to this problem, as they are able to distinguish experience from different contexts. Furthermore, most of the context-aware models perform similarly in this case. In particular, our Adapter, Concat and cGate converge rapidly, while FLAP converges slightly slower, but achieves comparable generalisation performance at the end of training.
Figure 8: Showing the average evaluation reward as we increase training time for CartPole. Mean performance over seeds is shown and standard deviation is shaded.
Figure 9: Mass vs. Distance Travelled (proportional to reward), at the end of training for the Ant domain. Red lines are the training contexts. Mean performance is shown, with standard deviation over 16 seeds shaded.
B.3 Ant
Here we consider the Ant domain, and vary the mass, without any distractor variables. As seen in Fig. 9, the Unaware model performs worse than the context-aware ones. This performance gap is due to the Unaware model being unable to simultaneously perform well on light and heavy masses, as seen in Fig. 9. In contrast, by utilising context, the agent can perform well in both settings despite their significant differences.
C cGate as a Special Case of the Decision Adapter
As discussed in Section 5, here we show that our architecture is, in fact, a generalisation of another network architecture introduced by prior work. Benjamins et al. [13] introduce cGate, which obtains an action as follows. It first calculates  $\phi(s) \in \mathbb{R}^d$  using the state encoder  $\phi$  and  $g(c) \in \mathbb{R}^d$  using the
context encoder  $g$ . Then, it calculates features  $h = \mathsf{cGate}(s,c) = \phi (s)\odot g(c)$ , with  $\odot$  being the elementwise product. The action is obtained as  $a = f(h)$ , where  $f:\mathbb{R}^d\to \mathcal{A}$  is the learned policy. Here, both the state encoder  $\phi$  and the context encoder  $g$  correspond to neural networks that each output a  $d$ -dimensional vector.
This approach is a specific instantiation of our general adapter architecture, where we have one adapter module  $A = A_{i}$ . In particular, the state encoder  $\phi$  corresponds to the partial neural network before our adapter module, i.e., layers  $L_{1}, L_{2}, \ldots, L_{i-1}$ . The policy  $f$  corresponds to the layers after our adapter module,  $L_{i}, L_{i+1}, \ldots, L_{n}$ . Thus, we have that  $\phi(s) = L_{i-1}(x_{i-1})$  and  $f(h) = L_{n}(x_{n})$ . If we define our adapter network  $A(\phi(s)|c) = \phi(s) \odot g(c)$ , we can recover the same result as cGate. This would be possible if we let our adapter consist of one layer without a nonlinearity or a bias vector, with its weight matrix being set to  $W = H(c) = \text{diag}(g(c))$ . This would result in
$$
A (\phi (s) | c) = W \phi (s) = d i a g (g (c)) \phi (s) = g (c) \odot \phi (s) = \mathbf {c G a t e} (s, c),
$$
where we have used the fact that, for two vectors  $a, b \in \mathbb{R}^d$ , the matrix multiplication  $\text{diag}(a)b$  is the same as the elementwise product  $a \odot b$ . This output vector is then passed to the policy network  $f$ . Our approach, however, is strictly more powerful than a single elementwise product, as our adapter can be an entire nonlinear neural network.
D.1 Environments
This section contains more details about the experimental setup of our environments.
D.1.1 Context Normalisation
Before we pass the ground truth context to the models, we first normalise them to a range between 0 and 1. In particular, for each context dimension  $i$ , we calculate the context to be given to the agents  $c_{norm,i}$  as
$$
c _ {n o r m, i} = \frac {c _ {i}}{m a x _ {i}},
$$
where  $\max_{i}$  is the largest value for dimension  $i$  in the training context set. This results in the largest training contexts having values of 1.0. During evaluation, we perform the same normalisation, so contexts outside the convex hull of the training set will potentially result in values larger than 1.0.
D.1.2 ODE
During training, for each context, we cycle between starting states, with each episode taking a new starting state from the set  $\{1.0, 0.5, -0.5, -1.0\}$ . Thus, the agent trains on the first context for 4 episodes, each with a different initial state. Then it goes on to the second context, and so on. When we reach the end of the context list, we simply start at the first context again. We do this to ensure training is comparable for each method, and that there is sufficient diversity in the initial state distribution. During evaluation, we fix the starting state at  $x_0 = 1.0$  to isolate the effect of context.
D.1.3 CartPole
Table 1 illustrates the context variables we use in CartPole. While we consider all five variables as the context, we varied only the pole length for our experiments. The value for each state variable at the start of the episode is randomly sampled from the interval  $[-0.05, 0.05]$ .
Table 1: The five context variables and their default values.
Name	Symbol	Default Value
Gravity	g	9.80
Cart Mass	mc	1.00
Pole Mass	mp	0.10
Pole Length	l	0.50
Force Magnitude	F	10.00
We use ReLU for all activation functions. We also aim to have an equal number of learnable parameters for each method and adjust the number of hidden nodes to achieve this.
We use Soft-Actor-Critic [39, SAC] for all methods, to isolate and fairly compare the network architectures. We use the high-performing and standard implementation of SAC from the CleanRL library [98] with neural networks being written in PyTorch [99]. We use the default hyperparameters, which are listed in Table 2. SAC learns both an actor and a critic. The actor takes as input the state and outputs  $\mu$  and  $\log (\sigma)$ , which are used to construct the probability distribution over actions  $\mathcal{N}(\mu ,\sigma)$  which is sampled from to obtain an action. The critic receives the performed action  $a$  as well as the state  $s$ , and outputs a single number, representing the approximate action-value  $\hat{q} (s,a)$ . Fig. 10 illustrates each baseline.
Table 2: The default hyperparameters we use in the CleanRL implementation.
Name	Value
Buffer Size	1 000 000
γ	0.99
τ	0.005
Batch Size	256
Exploration Noise	0.1
First Learning Timesteps	5000
Policy Learning Rate	0.0003
Critic Learning Rate	0.001
Policy Update Frequency	2
Target Network Update Frequency	1
Noise Clip	0.5
Automatically Tune Entropy	Yes
D.3 Adapter Configuration and Design Decisions
Now, our method provides a great degree of flexibility in terms of configuration, and we must make multiple design decisions to obtain a usable instantiation of the Decision Adapter. In this section, we briefly detail the default settings we use and justify why these are reasonable. We empirically investigate the merits of these choices and compare them against alternative options in Appendix E.
D.3.1 Bottleneck Architecture
We use a bottleneck architecture in our generated adapter modules, as described by Houlsby et al. [76]. In essence, we have a down-projection layer that transforms features from  $d_{i}$ -dimensional to  $p$ -dimensional, with  $p < d_{i}$ . Then, the second layer up-projects this back to a  $d_{i}$ -dimensional vector.
The bottleneck architecture is beneficial for two reasons. Firstly, it reduces the number of required parameters in our adapter model. For instance, if we have two layers in this bottleneck architecture, the number of parameters would differ substantially compared to one layer that transforms the  $d_{i}$ -dimensional features to  $d_{i}$  dimensions. In the first case, the number of parameters would be on the order of  $O(d_{i} \times p + p \times d_{i})$ , corresponding to a  $d_{i} \times p$  matrix for the down-projection and a  $p \times d_{i}$  matrix for the up-projection. If we only have one layer, then we have a single  $d_{i} \times d_{i}$  matrix. Thus, the bottleneck requires  $O(p d_{i})$  parameters, whereas the single layer requires  $O(d_{i}^{2})$ . If we, for example, have  $d_{i} = 256$  and  $p = 32$ , the bottleneck architecture has four times fewer parameters than the single layer. This, in turn, makes it easier for the adapter hypernetwork to generate useful weights.
Secondly, the bottleneck architecture imparts a low-rank inductive bias on the adapter, which may prevent it from overfitting [100]. This idea is related to prior work that has shown that, counterintuitively, reducing the capacity of the agent's neural network can improve generalisation and reduce
The bias vectors correspond to an additional  $p + d_{i}$  parameters, but we omit this term as it is dominated by the number of parameters in the weight matrices.
(a) The Unaware Model
(c) The cGate Model, with  $\odot$  indicating an elementwise product
Figure 10: Illustrating the architectures of each of our baselines. Here we consider the actor's architecture, with two separate output heads, one predicting the mean action  $a$  and the other predicting the log of the standard deviation (denoted as  $\sigma$  in this figure). In each plot, the blue nodes represent the state inputs whereas the green one represents the context. In general, for the bottom row, the network architectures for the context and state encoders are not necessarily the same.
(b) The Concatenate Model
(d) The FLAP Model, with  $\cdot$  indicating a dot product
overfitting [101-103]. Although our approach does not incorporate information-bottleneck losses like these methods, the observed benefits of reduced capacity provide additional intuition for utilising the bottleneck architecture. Finally, despite any possible information loss due to the bottleneck layer, the skip connection allows the network to retain any important information from the input.
D.4 Compute
For compute, we used an internal cluster consisting of nodes with NVIDIA RTX 3090 GPUs. For a single experiment and a single method, the runs took between 1 and 3 days to complete all seeds.
E Adapter Ablations
Here we experimentally justify some of the adapter-specific design decisions we made for our empirical results. The experiments in this section also give us insight into how robust the Decision Adapter is to changing its configuration options. We investigate the effects of the following factors, with our conclusions listed in **bold**.
- The network architecture of the Adapter module. We find that Most adapter architectures perform well. This result is expanded upon in Appendix E.1.
- Whether the hypernetwork uses chunking. Hypernetwork chunking outperforms non-chunking methods while using significantly fewer parameters (Appendix E.2).
- Using a skip connection. The Decision Adapter can perform well with or without a skip connection (Appendix E.3).
- The location of the adapter module in the main network. Most locations lead to high performance, except if the adapter is at the very start or very end of the network (Appendix E.4).
- Having an activation function before the adapter. Not having an activation function before the adapter in the actor model outperforms the alternatives (Appendix E.5).
For this entire section, we consider the ODE environment and follow the same procedure as in Section 7.1. We use the ODE due to its fast training, and the fact that context is necessary, which enables us to determine if a particular design choice of the Adapter results in poor use of the context.
E.1 Adapter Architecture
Here we examine the effects of changing the Adapter module's network architecture. We are interested in how the architecture impacts performance for two reasons. First, we wish to empirically justify our bottleneck architecture. Second, we wish to show that the Adapter performs well with a wide variety of network architectures, and is not particularly sensitive to this choice.
Figure 11: Evaluating the effect of different adapter architectures. Mean performance is shown, with standard deviation shaded.
As discussed in Appendix D.3, we use a bottleneck architecture of size 32. Thus, the 256-dimensional features are transformed into 32-dimensional features and then projected back to 256 dimensions. We denote this architecture as [32], since the adapter has a single hidden layer of 32 neurons. Fig. 11 illustrates the performance for various other adapter architectures. Most methods perform comparably, but using 8 hidden nodes or not having any hidden layers performs slightly worse than the base architecture. A large hidden layer of 256 nodes also does not outperform our bottleneck architecture. Finally, there is no particular benefit to having multiple layers in the adapter module.
E.2 Hypernetwork Chunking
Next, we compare our default architecture choice, which uses hypernetwork chunking, against non-chunking settings. We do this because hypernetwork chunking allows us to have significantly fewer learnable parameters (resulting in faster training) compared to the alternative of predicting the entire weight vector in one forward pass. Thus, if our approach performs comparably or better than non-chunking settings, then the chunking architecture is justified.
Figure 12: Comparing hypernetwork chunking (Base) compared to various non-chunking configurations as well as using larger chunks (Chunked[66, 66]). Mean performance is shown, with standard deviation shaded. Here, the values in the brackets represent the number of hidden nodes per layer, with [] corresponding to a hypernetwork that has no hidden layers.
We consider a non-chunking hypernetwork architecture of no hidden layers, as well as 2 and 3 hidden layers of size 100, respectively. Our final option is a chunked hypernetwork, with chunks of size 660, and two hidden layers of size 66 each (this is roughly twice as large as our default configuration option discussed in Appendix D.3). These results are shown in Fig. 12. We find that both chunking configurations perform the best. The non-chunking approaches perform slightly worse, particularly if the hypernetwork has no hidden layers.
These non-chunking configurations also have significantly more learnable parameters than the default. For instance, the base setting results in the actor's hypernetwork having around 16k parameters, whereas the [100, 100], non-chunking setting corresponds to roughly 1.7M parameters. In summary, by using hypernetwork chunking, we can obtain high performance while using a very small number of learnable parameters.
E.3 Skip Connection
Here we investigate whether a skip connection is beneficial or not. As mentioned in Section 5, the skip connection sets the updated  $x$  as  $x = A(x) + x$ , whereas if we remove it, the function becomes just  $x = A(x)$ . In Fig. 13, we can see that using a skip connection performs similarly to the alternative. This shows that the Decision Adapter is robust to this parameter, and can perform well with or without a skip connection.
Figure 13: Evaluating the effect of using a skip connection. Mean performance is shown, with standard deviation shaded.
Figure 14: Showing the base network architecture, with letters indicating adapter positions. For instance, an adapter placed at location (D) receives the action as input and returns a modified one.
E.4 Adapter Location
We next examine the effects of changing the location of the adapter module in our network, and what happens if we have multiple modules. This information would be useful when using our method in practice, and deciding where and how many modules to place. There are generally many options regarding the location, as an adapter module can be placed between any two layers of the primary network.
As described in Section 5, our actor network consists of a "trunk", containing one hidden layer, which maps the state to a 256-dimensional feature vector, and a one-layer action head, directly mapping the trunk features to an action. Here we consider the following locations for the adapter in both the actor and critic networks:
Base The standard adapter used in the rest of this chapter, which is placed before the action head's layer. In the critic, the adapter is placed before the last layer. This corresponds to location (C) in Fig. 14.
$\mathbf{Base}_F$  Placing the adapter in the actor network before the final layer of the trunk, leaving the critic unchanged (location B).
Start The adapter here is at the start of both networks, i.e., the adapter module receives the state as input (location A).
End Here, we place the adapter at the very end. Thus, the adapter takes in the predicted action and outputs a modified one. The critic's adapter is also placed at the end. This corresponds to location (D) in Fig. 14.
$\mathbf{All}_F$  Three adapter modules, one at the start of the trunk, one before and one after its final layer. The critic also has three adapters, one at the start, one before the last layer and one after the last layer. The actor has adapters in locations (A), (B) and (C).
In all cases, to ensure comparability against the base model, there is no activation function after the trunk in the actor model. These results are illustrated in Fig. 15. Overall, the base model performs the best. The adapter in the trunk performs slightly worse and the performance is quite poor if we put the adapter modules at the start of the networks. If we have three adapter modules, the performance is similar to only having one before the final layer. Finally, when adapting only the final action, the performance is also poor. Overall, the performance is low if we have the adapter only at the very
start or end of the network. Within a range of locations in the "middle" of the network, however, the performance is high and only slightly worse than the default setting.
Figure 15: Evaluating the effect of the adapter module's location. Mean performance is shown, with standard deviation shaded.
E.5 Activation Function Before Adapter
Our final ablation considers the benefits of not having an activation function before the adapter module inside the actor network. We perform this experiment to determine if our choice of not having an activation function before the adapter in the actor is justified, and if it is necessary to do the same for the critic.
Figure 16: Comparing the effect of having an activation function before the adapter vs. not. The blue, green and red lines alter the actor's architecture, whereas orange changes the critic. Blue corresponds to the base choice outlined in Appendix D.3. Mean performance is shown, with standard deviation shaded.
Here we consider four options: (1) not having an activation function after the trunk; (2) having one; (3) having an activation function after the trunk, but only for the standard deviation head; and (4) not having an activation before the adapter in the critic. Option (1) corresponds to the default choice. These results are shown in Fig. 16. Not having an activation function outperforms having one, and having an activation for only the standard deviation head is similar to not having one. Thus, not having an activation function before the adapter inside the critic network performs as well as the base option, which has this activation. Overall, this result justifies our design choice by showing that it performs comparably to or better than the alternatives.
This section contains some additional results that we refer to in the main text.
F.1 Single Dimension Interpolation, Train & Extrapolation
See Fig. 17 for the performance of each model on the ODE on the particular context sets, Interpolation, Training and Extrapolation. These plots therefore correspond to subsets of the evaluation range in the left pane of Fig. 3.
Figure 17: Showing the performance of each method in three different evaluation regimes, (left) the training contexts, (center) interpolation and (right) extrapolation. Mean performance is shown, with standard deviation shaded.
F.2 Multiple Dimension Heatmaps