SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fig. 18 illustrates the granular performance of each method in the ODE domain, corresponding to the right pane of Fig. 3. Our Decision Adapter outperforms all baselines and, as can be seen in Figs. 18a and 18b, generalises further away from the training contexts than the Concat model. In particular, these heatmaps show that both models perform well on the training contexts (the blue squares). However, the Adapter generalises much better, especially to contexts far outside the training range. For example, when  $|c_1| \in [8, 10]$  (corresponding to the far left and far right of the heatmaps), the Adapter's rewards are higher than the Concat model's. The Adapter obtains between 50 and 150 more reward than the Concat model – which is a substantial difference, as the maximum reward in this environment is 200. This region of context-space is far outside of the convex hull of the training range, which contains values of  $|c_1|$  only up to 5.0. Closer to the training range; for instance, around the boundary of the green square, the Adapter still outperforms the Concat model, but the difference is less pronounced.
FLAP and cGate perform poorly, even though they obtain near-perfect rewards on the training contexts. There may be several reasons for this. First, here we train on 16 relatively sparsely distributed contexts, with 300k steps in total, resulting in fewer episodes in each context compared to Section 7.1. Second, with two dimensions, the interpolation range makes up a smaller proportion of the evaluation range compared to the one-dimensional case. This is because, in one dimension, the range  $[-5,5]$  makes up half of the  $[-10,10]$  evaluation range. In two dimensions, however, the interpolation range makes up only a quarter of the evaluation set. Thus, the overall average performance in Fig. 3 is more skewed towards extrapolation. This, coupled with the fact that both cGate and FLAP perform poorly on extrapolation (Fig. 17, right), may explain why they underperform in this case. Lastly, it is generally easier to generalise in the one-dimensional ODE, as the optimal action depends mainly on the signs of the context and state variables. By contrast, the multidimensional case is more difficult, as the dynamics equation includes a nonlinear action term. This is also why, for instance, our Decision Adapter performs worse in this case (around 160 overall evaluation reward) compared to the one-dimensional ODE (around 190).
(a) Concat
(b) Adapter
(c) cGate
(d) FLAP
Figure 18: Multidimensional contexts in the ODE domain for (a) Concat, (b) Adapter, (c) cGate and (d) FLAP. Heatmap Performance for (a) cGate and (b) FLAP. Each cell represents the reward obtained when evaluated in the corresponding context, averaged over 16 seeds. The blue squares indicate the specific training contexts, whereas the green square represents the convex hull of training contexts. For the heatmaps, the last rows and columns, labelled avg, represent the average reward over the particular column or row, respectively.
F.3 Additional Baselines
We consider two ablations as additional baselines. The first, called AdapterNoHnet, is based on our adapter module, but there is no hypernetwork involved. The adapter's architecture and location are the same, but it is now a single MLP that takes in the concatenation of the state-based features and the raw context (as the model has to be context-conditioned, and must also process the state-based features). The second baseline, termed cGateEveryLayer, uses the same elementwise operation as cGate, but this happens at every hidden layer except at only one. Fig. 19 illustrate these results on the 1D and 2D ODE domains respectively, with the Adapter, Concat and cGate models there for reference. Overall, cGateEveryLayer does not outperform cGate, and AdapterNoHnet performs much worse than our hypernetwork-based adapter.
Figure 19: The performance on the 1D and 2D ODE domains (i.e., similar to Fig. 3) of the additional baselines. The mean and standard deviation over 16 seeds are shown.
F.4 AER Metric
Fig. 20 illustrates the difference between using the AER metric as defined in Section 6.1 (corresponding to Fig. 20a) and only considering the testing contexts (corresponding to Fig. 20b). Overall, the performance of each model is very similar; this is because the training contexts make up a small proportion of the entire evaluation range.
(a) Training and Testing Contexts
(b) Only Testing Contexts.
Figure 20: Comparing (a) averaging performance over all contexts and (b) restricting the AER metric to only unseen testing contexts.
F.5 Distractors
The ODE domain displays similar distractor results to our other environments (see Section 7.2). In particular, Fig. 21 shows that the Adapter is robust to adding more irrelevant distractor variables, whereas Concat and cGate are not.
Figure 21: Showing the sensitivity of the (left) Concat, (middle) Adapter and (right) cGate models to fixed distractor context variables on the ODE. The mean and standard deviation over 16 seeds are shown.
F.6 Gaussian Distractor Experiments
This section expands upon the results in Section 7.2 and contains additional results where the distractor variables are not fixed.
F.6.1 Difference between training and testing
Fig. 5 contains the results on CartPole, and the ODE results are shown in Fig. 22. For both domains, the conclusion is similar in that the Adapter is still more robust to changing distractor variables compared to either Concat or cGate.
Figure 22: The Gaussian distractor variable results. This corresponds to using a mean of 1 during training and a mean of 0 during testing. We use a consistent  $\sigma = 0.2$ . These results are for the ODE domain, showing the mean and standard over 16 seeds.
F.6.2 Keeping the training and testing distractor distributions the same
Next, we consider a similar case, but the means of the Gaussians are the same across training and testing. Overall, the results for the ODE and CartPole (with a mean of 0 during both training and testing) are shown in Fig. 23. For the ODE, while the Adapter is insensitive to adding more distractor dimensions, both cGate and the Concat perform worse as we add more irrelevant distractor dimensions to the context. However, the difference is much less pronounced than in the case where the means are different between training and testing. Surprisingly, in the ODE, cGate performs slightly better with 20 distractor variables than with 0. These variables may effectively act as an additional, noisy, bias term that the model can utilise. For CartPole, the Concat and cGate models perform much worse with 100 distractor dimensions compared to none. The Adapter displays some noise, indicating that some seeds performed poorly while others performed well. This is similar to the results we observed in Appendix G.1—too much noise leads to unstable training.
Finally, we consider using a mean of 1 during both training and testing in Fig. 24. For the ODE, each model performs similarly regardless of the number of distractor variables. In CartPole there is not a
Figure 23: The Gaussian distractor variable results, using a consistent mean of 0 during training and testing, with  $\sigma = 0.2$ . These results are for the (top) ODE and (bottom) CartPole domains, showing the mean and standard over 16 seeds.
Figure 24: The Gaussian distractor variable results, using a consistent mean of 1 during training and testing, with  $\sigma = 0.2$ . These results are for the (top) ODE and (bottom) CartPole domains, showing the mean and standard over 16 seeds.
large difference between the various models when the mean of the distractor variables is 1 during both training and testing.
F.6.3 Summary
Overall, The Adapter is significantly less susceptible to overfitting to irrelevant distractors compared to the Concat model and cGate. However, if these distractors remain the same during training and testing (which may not be particularly likely), it matters less whether the model overfits or not, and all models are less affected by adding more distractors.
This section expands upon the limitations of our approach, outlined in Section 8.
G.1 Noisy Contexts
Now, while we use the ground truth context to isolate the effect of the network architecture on generalisation performance, this is not always available [12, 18]. A common approach is to learn a context encoder that uses experience in the environment to infer an approximation to the context, which is then used to act [18, 87]. This encoded context, however, may not always be entirely accurate. Thus, in this experiment, we examine how sensitive each context-aware model is to receiving noisy contexts during evaluation. We consider two cases; the first is where the models received the uncorrupted context during training, and the second is where the models also encountered noise during training. The first case could reasonably occur in reality, since we may have access to the ground truth context during training, but not evaluation [14]. The second case, where we train with noise, could correspond to training the RL policy alongside the context encoder, leading to some noise during training. This case additionally allows us to examine whether training with noise increases the models' robustness to context noise during evaluation.
We take the models trained during the primary CartPole experiments, in Appendix B.2, where we varied the pole length. We use the model checkpoints that were trained for 600k timesteps, as both the Concat and Adapter models did not improve much by training for longer than this (see Fig. 8). The noise we add here, corresponding to normally distributed noise with varying standard deviations, was added to each context dimension instead of just the one representing pole length. We experiment with a standard deviation  $\sigma \in \{0,0.05,0.1,0.2,0.5,1\}$ . The noise was added after normalising the contexts such that the largest training context had a value of 1.0. Thus, a standard deviation of 1.0 represents a substantial amount of noise. In addition to the models trained without noise, we
additionally train models that encountered noise during training. In particular, we consider standard deviations  $\sigma \in \{0.1, 0.5\}$ , and change no other aspect of training.
The results are illustrated in Fig. 25. The left pane shows the performance of the models trained without noise. Overall, these results show that the Concat model is more susceptible to noisy contexts than our Adapter, bolstering the argument that treating context as a state dimension is not ideal [13]. Both models, however, generalise worse as the context becomes more inaccurate. This is in contrast to the Unaware model, which performs equally well regardless of the level of context noise. The Adapter still outperforms the Unaware model until the noise level reaches around  $\sigma = 0.5$ . This noise level, however, is very large, and  $\sigma = 0.5$  corresponds to a pole length of 3 meters. The default pole length is 0.5 meters, and the largest training length is 6 meters.
When training with a small amount of noise (Fig. 25, middle) both models perform similarly to training without noise. However, when training with a large amount of noise, corresponding to the right pane of Fig. 25, the Concat model becomes more robust to noisy contexts, but the Adapter performs worse. In particular, the Adapter model exhibits a large amount of variation across seeds, indicating that some seeds performed badly. On average, the Adapter performs similarly to the Unaware model until the level of context noise increases beyond the training noise of  $\sigma = 0.5$ .
Figure 25: Comparing the average evaluation reward (after 600k steps) as a function of the standard deviation  $\sigma$  of the Gaussian noise added to the context during evaluation. In the left pane, the models trained without noise are shown. The middle pane corresponds to training with  $\sigma = 0.1$  and the right pane shows the results when training with  $\sigma = 0.5$ . Mean performance is shown, with standard deviation over 16 seeds shaded.
G.2 Suboptimal Context Normalisations
G.2.1 ODE
As discussed in the main text, we first normalise the context before passing it to the agents. This is generally done by using the largest absolute value encountered during training. For instance, for the one-dimensional ODE experiment, we used  $c_{\text{norm}} = \frac{c - 0}{5}$  as the normalisation function. Here we determine how sensitive the Decision Adapter is to changes in this procedure. This is useful as we may often need to choose the normalisation factor without knowing exactly which variations we will encounter during evaluation. To do this, we run four additional experiments, with normalisation values of 0.1, 1.0, 2.0 and 15.0, respectively. We train the agents for only 300k steps, as after training for this number of steps, most models performed similarly to their final performance. The results are shown in Fig. 26, and we can see that the Concat and Adapter models are both robust to the normalisation value. Our Decision Adapter loses less performance than the Concat model as we change the normalisation value. Despite losing some performance as the context normalisation value increases, both context-aware models still outperform the Unaware model.
G.2.2 CartPole
We now consider a similar setup in CartPole. In particular, we train on a Pole Mass of 0.1. We choose this variable as some of our experiments showed that the Unaware model generalises equally well regardless of the training contexts. We consider two scenarios: In the first, we normalise with respect to the Small setting (i.e.  $c_{norm} = \frac{c}{0.01}$ ). Thus, during training, the normalised context had a value
Figure 26: Showing the overall performance (y-axis) of each context-aware model after 300k steps as we change the context normalisation value (x-axis). Mean performance is shown, with standard deviation shaded.
Figure 27: Performance when evaluating on changing Pole Mass when normalising with respect to (left)  $\mathrm{Mass} = 0.01$  and (right)  $\mathrm{Mass} = 1.0$ . In both plots, the models trained on  $\mathrm{Mass} = 0.01$ . Mean performance is shown, with standard deviation shaded.
of 1.0; during evaluation, it was nearly always larger than 1.0. In the second case, we normalise with respect to the X-Large setting (corresponding to  $c_{norm} = \frac{c}{1}$ ). We trained on the Small pole length for both settings. Fig. 27 shows the results for this experiment. When normalising with respect to the Small setting, the Concat model fails to generalise at all, and the Adapter performs worse than Unaware. When changing the normalisation, however, the context-aware models perform significantly better.
G.2.3 Summary
Thus, the Decision Adapter is relatively robust to small changes in context normalisations. However, when encountering evaluation contexts that are 100 times larger than during training, appropriate
normalisation is crucial and can make a drastic difference – even when training on the exact same contexts. As seen in the ODE, having a normalisation value that is slightly too large does not cause significant performance penalties. Thus, normalising the contexts with a larger value than expected during training is a good heuristic.
G.3 Narrow Context Range
We next consider the effect of changing the set of training contexts. Here we aim to briefly illustrate that the training context range can have a large effect on the performance of the agents. In particular, we consider the ODE domain and 8 separate sets of training contexts. In all of these cases, we keep the normalisation consistent at  $c_{norm} = \frac{c}{5}$  to ensure comparability. The results when training on each context set are shown in Fig. 28. Overall, when we have a single positive and negative training context, performance is poor when this context is very small (a), and increases as it becomes larger (d and f). When we have multiple contexts, but insufficient variation (b, c, e), then performance is also suboptimal. For instance, in (b), the training contexts cover only the small region  $[-1, 1]$ . In (c), the training set of contexts does not contain any negative contexts, leading to poor generalisation. Finally, if contexts are varied, but spread out too far (g), the performance also suffers.
Figure 28: Here we examine the performance when training on different sets of contexts. The text beneath each plot indicates the training context set, with  $\pm c$  indicating that both  $c$  and  $-c$  are in the context set. For instance, in (a), the models were trained on a context set of  $\mathcal{C}_{train} = \{-0.1, 0.1\}$ . Mean performance is shown, with standard deviation shaded.
G.3.1 Overfitting
Another potential issue we may encounter is overfitting, which can occur when the training range is not diverse enough. To illustrate this problem, we conduct an experiment in the multidimensional ODE setting using the following set of training contexts:
$$
\{(1, 1), (1, - 1), (1, 0), (- 1, 1), (- 1, - 1), (- 1, 0), (0, 1), (0, - 1) \}
$$
The results, presented in Fig. 29, demonstrate that the Decision Adapter performs well initially but its generalisation performance suffers as training progresses. Specifically, as shown in Fig. 29b and Fig. 29c, our model exhibits worse extrapolation performance due to overfitting on the narrow training contexts.
(b) 50k steps
(c) 300k steps
Figure 29: Showing (a) the overall performance over time on the entire evaluation range. In (b) and (c) we show the performance of the Adapter at two points in training. In (a), we plot mean performance and shade the standard deviation; for (b) and (c), we show only the mean over the seeds.