SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We present the results of our experiments in Figure 19. We find that modulating the position-wise feedforward layer tends to be more important the modulating the self-attention mechanism. Overall, the best performance is obtained when modulating all considered targets. However, this comes at the cost of more parameters. Notably, these results narrow the performance gap to L2M-oracle.
(b) DMC6
Figure 18: Performance vs. fraction of parameters trained for different values for the rank in LoRA on (a) CW10 and (b) DMC6.
Figure 19: Modulation target ablation for L2M on CW10.
H Continual Fine-Tuning
Ultimately, our goal is to adapt the pre-trained model to multiple novel tasks, while alleviating forgetting of tasks learned during pre-training. In this regard, we adapt the pre-trained model to all fine-tuning tasks in a sequential manner, as proposed by Wolczyk et al. (2021) for CW10. Moreover, we evaluate forgetting by measuring the performance on tasks acquired during pre-training after fine-tuning. We compare 10 different methods in this setting, as listed in Section 3.3.
We train each method for 100K steps per task in CW10, with a batch size of 256. After every 100K update steps, we switch to the next task in the sequence. After every 50K update steps, we evaluate the current model on all tasks in the task-sequence. For L2M, we use a pool size of 30. For all L2P-based approaches, we use a prompt size of 25 and a prompt pool of 100 by default. Following Wang et al. (2022c), we use the selection count  $n(\mathbf{k}_i)$  up to the previous task and  $\lambda = 0.5$  for regularization of L2M and all L2P-based approaches. For EWC, we tune the value of the penalty (see Section H.5). For L2M, we use learning rate of  $5e^{-5}$  on CW10 and  $1e^{-4}$  on DMC6. For all L2P variants, we use a learning rate of  $1e^{-3}$ .
We show the performance and forgetting scores for all considered methods in Table 7. All metrics are reported at the end of training. The forgetting scores are computed as defined by Wolczyk et al. (2021). In Table 7, we also include the multi-task performance scores on all CW10 tasks (same as in Appendix G). Multi-task FT represents the upper bound in terms of performance. Overall, L2M achieves the highest success rates/normalized scores and outperforms all other methods in both domains.
Table 7: Continual RL experiments on (a) CW10 and (b) DMC6.
(a) CW10
Method	Success Rate	Mean Reward	Forgetting
FT	0.09 ± 0.01	234.57 ± 414.86	0.75 ± 0.32
FT-last+head	0.12 ± 0.03	266.59 ± 406.42	0.65 ± 0.31
FT-head	0.07 ± 0.03	95.34 ± 127.2	0.08 ± 0.27
L2M-oracle	0.77 ± 0.03	1294.72 ± 385.32	0.05 ± 0.16
L2M	0.65 ± 0.04	1136.17 ± 53.82	0.07 ± 0.08
L2P-Pv2	0.4 ± 0.02	707.84 ± 35.91	0.03 ± 0.06
L2P-PreT	0.34 ± 0.05	606.33 ± 61.97	0.01 ± 0.06
L2P-PT	0.23 ± 0.05	390.28 ± 64.87	0.11 ± 0.01
EWC	0.17 ± 0.01	257.48 ± 18.18	0.78 ± 0.0
L2	0.1 ± 0.0	237.07 ± 522.98	0.0 ± 0.0
FT-MT-scratch	0.85 ± 0.07	1476.61 ± 395.13	-
FT-MT-pre-trained	0.92 ± 0.02	1491.76 ± 332.0	-
(b) DMC6
Method	Normalized Score	Mean Reward	Forgetting
FT	0.27 ± 0.01	225.81 ± 10.2	0.83 ± 0.02
FT-last+head	0.25 ± 0.02	204.49 ± 22.07	0.51 ± 0.06
FT-head	0.06 ± 0.0	58.84 ± 0.64	-0.02 ± 0.0
L2M-oracle	0.7 ± 0.11	549.29 ± 254.69	-0.02 ± 0.27
L2M	0.56 ± 0.18	401.65 ± 122.75	0.11 ± 0.14
L2P-Pv2	0.32 ± 0.09	271.39 ± 72.49	0.02 ± 0.08
L2P-PreT	0.2 ± 0.04	182.33 ± 23.28	-0.03 ± 0.01
L2P-PT	0.18 ± 0.03	158.09 ± 22.88	-0.06 ± 0.02
EWC	0.3 ± 0.06	244.7 ± 54.12	0.49 ± 0.14
L2	0.2 ± 0.03	192.8 ± 23.86	-0.01 ± 0.01
FT-MT-scratch	0.92 ± 0.03	648.49 ± 354.7	-
FT-MT-pre-trained	1.04 ± 0.17	764.38 ± 230.28	-
H.1 Query Representation Ablations
For all L2M results, we use embedded state-tokens aggregated over the context as query to the modulation pool. It is possible to represent the query differently by using other tokens (see Figure 8), intermediate layers, or history lengths.
Layer Representation. Instead of using the representation of the embedding layer for the query, we can alternatively use the representations produced by intermediate Transformer layers. To this end, we conduct an ablation study in which we compare the representations extracted from the embedding layer against the representations from the first, middle (3rd block) and last Transformer block. We show the results of this experiment in Figure 20. Using the representations extracted from the middle or last Transformer blocks yields worse performance compared to the embedding layer. In contrast, the representation from the first Transformer block attains the highest scores overall, and narrows the gap to L2M-oracle. However, it is important to note that this performance gain comes at the cost of
Figure 20: Layer representation ablation for L2M on CW10.
Figure 21: History length ablation for L2M on CW10.
additional complexity since we need to propagate farther through the model than merely through the embedding layer.
History Length. To investigate the dependence of the query representation on the history length while keeping the context window as is, we conduct an ablation study. In Figure 21, we report the results of this experiment. By default, we used the five last timesteps in the sequence to construct the query. However, we find that using only the most recent timestep performs similarly.
H.2 Alternative Modulators
In L2M, we use LoRA by default. However, L2M can also be combined using other PEFT techniques. Therefore, we conduct an ablation study in which we compare L2M against L2M in combination with  $(\mathrm{IA})^3$  on CW10. Instead of the low-rank modulation matrices,  $(\mathrm{IA})^3$  employs element-wise multiplication with learnable modulation vectors. We present the results in Figure 22. While performance decreases with L2M-(IA) $^3$ , it compares favourably in terms of parameter-efficiency. Depending on the task at hand, this may be preferred.
Figure 22: Alternative modulators ablation on CW10. We compare L2M to L2M-(IA) $^3$ .
H.3 Single-domain DT on Meta-World
We pre-train a DT with 13M parameters on
MT40 and compare to the same methods as in Sections 3.2 and 3.3. In this setting, we additionally add two competitors, namely PromptDT (Xu et al., 2022b) and VIMA (Jiang et al., 2022). We experimented with larger models, but found the selected model size to perform well. Our MDDT used a unified state space, a discretised action space and autoregressive action prediction (at inference time) to handle varying state and action spaces. In contrast, the single-domain DT does not require these mechanisms. Instead, the single-domain DT is trained to predict the continuous actions via the MSE loss. The results for the single-domain experiment are available in Figure 23. The pre-trained model attains a success rate of  $81\%$  on MT40. Overall, we find that the specialised single-domain model obtains considerably higher performance scores than the MDDT (see Sections 3.2 and 3.3). However, these performance gains come at the cost of the loss of generality, as the specialised model can only handle the particular state/action space it was trained on.
H.4 Cross-domain FT
We conduct another experiment, in which we pre-train a DT on Meta-World only (MT40) and then fine-tune it on DMControl (DMC6). The pre-trained model relies on the unified state space and action discretisation. It has the same amount of parameters as the MDDT (40M) and attains an
(a) Single-task FT on CW10
(b) ContinuaI FT on CW10
Figure 23: IQM and  $95\%$  CIs on CW10 for (a) single-task FT and (b) continual FT using a single-domain non-discretized DT pre-trained on MT40 with 13M parameters.
average success rate of  $76\%$  on MT40 after pre-training. Prior to fine-tuning, the model has never seen trajectories from DMControl.
We show the results for different fine-tuning techniques in Figure 24. We observe that the fine-tuning performance on DMC6 is considerably worse than for the MDDT (79%). In addition, we also fine-tune the pre-trained single-domain model on the same domain (CW10). Interestingly, the final performance on CW10 is also lower compared to the MDDT model (75%) that was pre-trained on both domains. This experiment indicates that multi-domain pre-training can, indeed, have a positive effect on the fine-tuning performance.
(a) Single-task FT on CW10
Figure 24: IQM and  $95\%$  CIs for single-task FT on (a) CW10 and (b) DMC6 using a single-domain discretized DT pre-trained on MT40 with 40M parameters.
(b) Single-task FT on DMC6
H.5 EWC Hyper-Parameter Search
We observed that EWC (Kirkpatrick et al., 2017) performs worse than L2M in our experiments. By default, we used a regularization coefficient of  $\lambda = 10000$  for EWC, which performed best in preliminary experiments. Additionally, we compare different values for  $\lambda = \{10,400,1e^3,1e^4,1e^5\}$  as used in EWC. We find that the optimal choice for  $\lambda$  varies heavily between the two domains, but higher performance scores can be achieved by tuning  $\lambda$ .
I Hardware & Training Times
Pre-training. We run all our pre-training experiments on 4 NVIDIA A100 GPUs. Training times depend on the model size. Training the smallest model (40M parameters) takes roughly 1.5 days, while training the largest model (200M) takes roughly 3.5 days. To parallelize the computation across multiple GPUs/nodes, we leverage the distributed data parallel (DDP) feature provided by PyTorch. Throughout all our experiments, we use mixed precision training (Micikevicius et al., 2017) as supported in PyTorch to speed up training time.
Fine-tuning. For all our fine-tuning experiments, we use single GPU training on NVIDIA A100 or NVIDIA Titan V GPUs. Training times vary between the considered fine-tuning settings (single-tasks experiments, continual-learning experiments), domains (Meta-World, DMControl), and methods. For example, running a single seed for L2M on CW10 in the continual FT setting takes 1.25 days on one NVIDIA A100. Similarly, running a single seed for L2M on DMC6 takes roughly one day on one NVIDIA Titan V.
J Potential Societal Impact
Our work is purely academic. However, the presented ideas aim for more capable embodied agents. As such, there is potential for misuse, e.g., when used in real-world scenarios without proper verification or safeguards. While we do not expect to see the deployment of such agents for potentially malicious purposes in the foreseeable future, it is essential to ensure responsible use of this technology.