SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MHR- $z$  facilitates extreme parameter efficiency Fig. 2 (right) reveals intriguing findings regarding MHR- $z$ . When we restrict training to only the routing parameters  $Z$  in the original Poly, the results are unfortunately not up to par with its version where both routing and adapters are updated. However, when we apply the same constraint to MHR, the performance is significantly closer to the optimum achieved in this setting. In fact, MHR- $z$  surpasses prior baselines while simultaneously necessitating fewer parameters for effective adaptation to new tasks. Moreover, by controlling the number of layers which share the same  $Z$  allocation (see sec. 3), MHR- $z$  is able to trade-off performance for parameter efficiency, even surpassing Poly- $z$  in settings with only 3K trainable parameters per test task (see also § A.2.1 for a more in-depth analysis). This trend is similarly observed in the SuperNI benchmark (Tab. 2), where updates restricted to the routing parameters yield performance on par with standard fine-tuning. We therefore conclude that the MHR- $z$  represents a robust approach for achieving extreme parameter efficiency in adaptation.
Additional routing heads is more beneficial than extra modules In the original Poly approach, a tradeoff between capacity and parameter efficiency can be achieved by adding extra modules for each adapter layer. However, this results in a linear increase in the number of multi-task parameters, which can become impractical. To explore a more effective tradeoff, we investigate the option of adding additional routing heads instead of extra modules. Fig 3 (right) presents the comparison between the two approaches. It demonstrates that increasing the number of routing heads leads to better perfor
SuperNI Dataset	Rouge-L
LoRA	67.60.8
LoRA-big	67.20.7
Poly-z	64.60.3
Poly	67.80.8
MHR-z	68.00.2
MHR	68.50.3
mance compared to adding more modules. Importantly, the benefit of multi-head routing is twofold: it provides increased expressivity for the model, while also maintaining parameter efficiency. This finding highlights the advantage of multi-head routing as a more effective approach for balancing expressivity and parameter count in few-shot adaptation scenarios.
Routing-based methods also excel at the 11B scale We proceed to evaluate if Poly and MHR surpass established PEFT approaches when trained over a larger model backbone. To accomplish this, we employ the 11B version of T0. As depicted in Tab. 3, routing-based methods once again outshine standard adapter training, surpassing our reproduction of the previous state-of-the-art in Liu et al. [2022] by over  $2\%$ . We observe that Poly and MHR show similar performance in standard fine-tuning, but MHR  $z$ -tuning remains more performant in routing-only fine-tuning. Indeed, MHR- $z$  (221K params) outperforms Poly- $z$  (3.5K params) by  $2.9\%$  while still remaining more parameter efficient than Liu et al. [2022] (1.1M params).
Table 2: Results on SuperNI dataset. Subscripts are standard deviation.
T0 Dataset	Avg. Test
Backbone T0-11B
T-Few Liu et al. [2022]	72.50.9
LoRA	72.31.0
Poly-z	70.00.6
Poly	74.90.6
MHR-z	72.90.8
MHR	74.70.6
Table 3: Few-shot results over 11B parameter backbones.
5.2 Why do routing-based PEFT methods yield superior performance?
While our proposed methods have demonstrated promising results across a broad spectrum of datasets and varying adaptation parameter budgets, the question of why routing-based PEFT exhibits superior
Figure 3: Left: Gradient alignment between tasks during multi-task pretraining. Right: Increasing the number of heads offer better scaling properties than increasing the number of modules.
performance remains unanswered. In this section, we aim to uncover the key components that drive MHR's superior performance.
Learning the Routing Function is essential Given that Poly and MHR have access to more parameters than standard adapters during multi-task pretraining, we investigate whether this, and not the routing mechanism, is responsible for their superior performance. To do so, we compare them to a baseline approach. Instead of learning the routing function, we randomly assign a binary module allocation to each data point in a minibatch, disregarding task information. This random routing approach, akin to Wang et al. [2022b], allows us to directly assess the influence of additional parameters during multi-task training. At test time, the learned modules are averaged into a single one before fine-tuning; we therefore refer to this baseline as Random-  $\mu$ . On the T0 benchmark with the T5-XL backbone, Random-  $\mu$  performs similarly to a standard LoRA adapter (66.0%), while Poly and MHR outperform it by 2% and 3.1% respectively. Therefore, we conclude that learning a routing function is crucial, and merely increasing capacity during training does not directly lead to improvements.
MHR fosters transfer and mitigates interference across pretraining tasks Recognizing the pivotal role of the multi-task pretraining step in bolstering Poly's performance, we explore the extent of transfer and interference across training tasks. By monitoring the average gradient alignment for each task pair (in terms of cosine similarity) throughout the training process, we are able to gauge the level of positive transfer. As Fig. 3 (left) shows, MHR displays a greater degree of gradient cosine similarity across tasks compared to other PEFT alternatives, including Poly. This finding suggests that the enhanced flexibility offered by multi-head routing may serve to mitigate interference across tasks to a larger extent than standard routing while simultaneously promoting positive transfer.
5.3 Is routing important for task generalization?
We assessed the importance of routing during pretraining. We now proceed to verify whether it is important to learn routing during few-shot adaptation, too. Poly- $\mu$  and MHR- $\mu$  consistently outperform LoRA, and match the performance of Poly/MHR (Tab. 4). This demonstrates that, for few-shot adaptation, the average of the pre-trained modules provides a better initialization than learning an adapter shared across all the tasks during pre-training. The consistently superior performance of Poly- $\mu$  with respect to Random- $\mu$  and AdapterSoup stresses the importance of routing during multi-task pre-training (but not during adaptation), which provides an effective adapter initialization for few-shot learning. This
T0 Dataset	Test Acc.
LoRA	66.01.6
AdapterSoup	62.11.0
Poly	68.08
Poly-μ	67.80.6
MHR	69.11.1
MHR-μ	69.10.9
SuperNI	Rouge-L
LoRA	67.60.8
Poly	67.80.8
Poly-μ	68.30.5
MHR	68.50.6
MHR-μ	68.50.8
Table 4: Evaluating the impact of modular adaptation at test time.
Method	Zero-Shot Test on 11 Held-out Tasks
k=0	k=1000	k=5000	k=10000
Backbone T5-XL
LoRA	56.5	56.0	56.1	55.7
Poly-μ	46.0	53.0	56.8	56.3
MHR-μ	48.0	58.0	57.1	56.3
Backbone T0-11B [Sanh et al., 2022]	61.0
LoRA	61.2	61.6	61.5	61.5
Poly-μ	62.1	63.6	63.9	64.4
MHR-μ	63.5	64.5	64.5	64.4
Table 5: Zero-shot performance for MHR and the baselines, reported as the average over the 11 evaluation datasets from Sanh et al. [2022]. To obtain these zero-shot results, we average the learnt Poly/MHR adapters before performing  $k$  additional fine-tuning steps on the multi-task pretraining data. This effectively enables zero-shot transfer to downstream tasks using the same amount of parameters/flops as the baseline LoRA. MHR outperform baseline LoRA by up to  $3\%$  absolute accuracy points on T0-11B.
finding could potentially inspire future work for improving meta-learning and weight-averaging approaches [Izmailov et al., 2018].
MHR-  $\mu$  excels at zero-shot learning For many downstream tasks of interest, additional labelled data may not be available. In such settings, it is unclear how to leverage MHR-  $\mu$  and Poly-  $\mu$  methods. To address this, we fine-tune the average of the multi-task trained adapters on the multi-task pretraining data (instead of using the downstream few-shot data), for an additional  $k$  steps. The results are presented in Table 5. We find that without any additional fine-tuning  $(k = 0)$ , averaging the adapters does not yield good results. This is due to a potential mismatch between adapters learned via task-specific routing, and the uniform routing strategy. We can observe that when fine-tuning the average of the adapters on the multi-task pre-training data for an additional  $k$  steps, MHR-  $\mu$  show strong performance when evaluated in a zero-shot manner. For a fair comparison, we also additionally fine-tune LoRA for the same number of additional steps. Our best model achieves a zero-shot performance of 64.5 on top of T0-11B, achieving an absolute gain of  $3.5\%$  accuracy points.
Multi-task learning is effective for low-resource tasks [Wei et al., 2022, Aribandi et al., 2022, Sanh et al., 2022], as knowledge can be borrowed from similar tasks by sharing the model parameters. Multi-task learning has also been applied across languages and modalities [Ponti et al., 2019, Bugliarello et al., 2022]. In the context of NLP, several families of methods enable learning new tasks from a limited set of labelled examples. Few-shot in-context learning [ICL; Brown et al., 2020], where examples of a new task are concatenated into an input prompt, enables models to generalize to unseen tasks without any gradient-based training. Such approaches are however sensitive to the prompt format and example ordering [Zhao et al., 2021]. More importantly, ICL methods incur a significant compute overhead, as for every prediction, the full set of examples must be processed by the model [Liu et al., 2022]. To remedy this, many parameter-efficient fine-tuning (PEFT) methods have been proposed as an alternative to ICL, where a small number of new parameters are added over the frozen pretrained network. To name a few, LoRA [Hu et al., 2022] injects learnable low-rank matrices into each Transformer layer. Alternatively, the learnable matrix can be sparse, selecting nonzero shifts via the Lottery-Ticket hypothesis [Ansell et al., 2021] or via their approximate Fisher information [Sung et al., 2021]. Finally, prefix-tuning methods [Li and Liang, 2021] prepend learnable embeddings to the input or intermediate representations to specialize the model towards a downstream task.
Modular networks partition their parameters into several expert modules, each of them specialized to handle specific sub-tasks [Jacobs et al., 1991, Kirsch et al., 2018]. Modular networks are an appealing solution to the problem of adapting to unseen tasks [Corona et al., 2021], as the model
can leverage its existing modules and recombine them in a novel way, thus achieving systematic generalization [Bahdanau et al., 2019]. They have also been tested in learning scenarios with data presented sequentially [Ostapenko et al., 2021], and with changing environments Goyal et al. [2021]. In NLP, mixture-of-experts (MoE) models [Shazeer et al., 2017, Fedus et al., 2022], where a learned gating mechanism routes token representations to appropriate experts (Feed-Forward layers), have shown success in scaling the number of parameters while retaining time efficiency. This results in higher performance when compared to their dense counterparts using a similar compute budget.
In this paper, we tackle the challenge of generalizing to new tasks based on a few examples after multi-task pre-training. Specifically, we focus on Polytropon [Ponti et al., 2023], a model where each task is associated with a subset of adapters by a routing function. We investigate how varying the level of control afforded by the routing function impacts performance on two comprehensive benchmarks for multi-task learning, T0 and Super-Natural Instructions. First, a newly proposed variant of the routing function, where multiple heads are responsible for different subsets of input dimensions, improves consistently over all other baselines, including LoRA and (IA) $^3$  adapters. Second, we identify the cause of the success of routing in its ability to prevent interference between tasks, as it yields a better alignment between their gradients. Third, we find that simple averaging of all multi-task pre-trained adapters before few-shot adaptation to new tasks provides comparable performance, thus offering state-of-the-art performance for single-adapter few-shot learning. Multi-head routing demonstrates the importance of fine-grained adapter selection for sample-efficient generalization and holds promise to improve other modular methods, such as Mixtures of Experts [MoEs; Fedus et al., 2022] in future research.
A. Ansell, E. M. Ponti, J. Pfeiffer, S. Ruder, G. Glavaš, I. Vulić, and A. Korhonen. MAD-G: Multilingual adapter generation for efficient cross-lingual transfer. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4762–4781, Nov. 2021. doi: 10.18653/v1/2021-findings-emnlp.410. URL https://aclanthology.org/2021-findings-emnlp.410.
A. Ansell, E. M. Ponti, A. Korhonen, and I. Vulić. Composable sparse fine-tuning for cross-lingual transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022. URL https://arxiv.org/pdf/2110.07560.pdf.
V. Aribandi, Y. Tay, T. Schuster, J. Rao, H. S. Zheng, S. V. Mehta, H. Zhuang, V. Q. Tran, D. Bahri, J. Ni, J. Gupta, K. Hui, S. Ruder, and D. Metzler. Ext5: Towards extreme multi-task scaling for transfer learning. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=Vzh1BFUCiIX.
A. Asai, M. Salehi, M. E. Peters, and H. Hajishirzi. Attempt: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6655-6672, 2022.
D. Bahdanau, S. Murty, M. Noukhovitch, T. H. Nguyen, H. de Vries, and A. Courville. Systematic generalization: What is required and can it be learned? In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=HkezXnA9YX.
S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
E. Bugliarello, F. Liu, J. Pfeiffer, S. Reddy, D. Elliott, E. M. Ponti, and I. Vulić. IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 2370-2392.
Y. Wang, S. Mishra, P. Alipoormolabashi, Y. Kordi, A. Mirzaei, A. Arunkumar, A. Ashok, A. S. Dhanasekaran, A. Naik, D. Stap, E. Pathak, G. Karamanolakis, H. G. Lai, I. Purohit, I. Mondal, J. Anderson, K. Kuznia, K. Doshi, M. Patel, K. K. Pal, M. Moradshahi, M. Parmar, M. Purohit, N. Varshney, P. R. Kaza, P. Verma, R. S. Puri, R. Karia, S. K. Sampat, S. Doshi, S. Mishra, S. Reddy, S. Patro, T. Dixit, X. Shen, C. Baral, Y. Choi, N. A. Smith, H. Hajishirzi, and D. Khashabi. Benchmarking generalization via in-context instructions on 1,600+ language tasks, 2022a. URL https://arxiv.org/abs/2204.07705.
Y. Wang, S. Mukherjee, X. Liu, J. Gao, A. H. Awadallah, and J. Gao. Adamix: Mixture-of-adapter for parameter-efficient tuning of large language models. arXiv preprint arXiv:2205.12410, 2022b.
Z. Wang, Y. Tsvetkov, O. First, and Y. Cao. Gradient vaccine: Investigating and improving multitask optimization in massively multilingual models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=F1vEjWK-lH_.
J. Wei, M. Bosma, V. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=gEZrGcozdqR.
Q. Ye, B. Y. Lin, and X. Ren. CrossFit: A few-shot learning challenge for cross-task generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7163-7189, Nov. 2021. doi: 10.18653/v1/2021.emnlp-main.572. URL https://aclanthology.org/2021.emnlp-main.572.
Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh. Calibrate before use: Improving few-shot performance of language models. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12697-12706. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/zhao21c.html.
A.1 Additional Results
More detailed numbers on the T0 Sanh et al. [2022] and SuperNI Wang et al. [2022a] datasets using different backbones, and different adapter layouts over the base model are found in Table 6. Multi-Task params is the number of additional parameters that must be conserved after
Model	Multi-Task Params	Adaptation Params	Avg. Test
T0 Dataset
Backbone T5-XL-LM
Multi-Task Full Finetuning + LoRA (IA)3	2.8B	2.2M	68.9x.x
540K	540K	62.40.4
AdapterSoup	84M	2.2M	62.11.0
LoRA	2.2M	2.2M	66.01.6
LoRA-big	35M	35M	65.40.9
Poly-z	17M	3.5K	66.40.3
Poly	17M	2.2M	68.01.0
MHR-z (64 h)	17M	220K	68.30.8
MHR (64 h)	17M	2.2M	69.11.0
Backbone T0-3B
T-Few Liu et al. [2022]	540K	540K	66.20.5
AdapterSoup	84M	2.2M	66.10.6
LoRA	2.2M	2.2M	67.40.8
LoRA-big	35M	35M	68.00.8
Poly-z	17M	3.5K	65.31.0
Poly	17M	2.2M	69.00.8
MHRz (64 h)	17M	220K	68.41.2
MHR (8 h)	17M	2.2M	69.31.2
Backbone T0-3B light version : (k, v, ff layers only)
l-LoRA (rank 1)	934K	934K	66.20.9
l-LoRA (rank 16)	15M	15M	67.61.1
AdapterSoup (l-LoRA)	35M	934K	64.91.0
l-Poly-z	7.5M	2.1K	62.91.2
l-Poly	7.5M	934K	68.00.5
l-MHRz (32 h)	7.5M	74K	66.81.1
l-MHR (8 h)	7.5M	934K	68.50.7
SuperNI Dataset	Rouge-L
Backbone T5-XL-LM light version : (k, v, ff layers only)
l-LoRA	934K	934K	67.60.8
l-LoRA-big	18M	18M	67.20.7
l-Poly-z	7.5M	2.1K	64.60.3
l-Poly	7.5M	934K	67.80.8
l-MHRz (64 h)	7.5M	147K	68.00.2
l-MHR (8 h)	7.5M	934K	68.50.3
Table 6: (top) Results on T0 dataset Sanh et al. [2022], we report the mean of the best validation accuracy for each test task, when evaluated every 50 train epochs. T-Few is our reproduction of the results in Liu et al. [2022]. LoRA-big means a LoRA adapter with a larger rank. (bottom) Results on SuperNatural Instructions dataset.
multi-task pretraining to enable transfer to a downstream task. Adaptation Params refer to the number of parameters required to learn a new downstream task. For e.g. Poly and MHR, the multi-task parameters include the learned modules, but not the routing over the training tasks, as these are not required for transfer on a new task. Moreover, variants which average the learned modules prior to fine-tuning (MHR-  $\mu$  and Poly-  $\mu$ ) will have both multi-task and adaptation parameters equal to that of a single shared adapter, since after multi-task pretraining one can average the modules.
A.2 Navigating the parameter efficiency / performance trade-off of tuning only the routing
Here we provide additional results on how different routing based methods can be more expressive when only learning a new routing function (over frozen modules) to adapt to a new task.
Figure 4: Different ways to control the expressivity of routing based methods. Left : In Polytropon, one can only add additional modules, resulting in a linear parameter increase. Right : In MHR, additional heads only introduce routing matrices  $Z$ , resulting in a negligible parameter increase.
In Fig. 4 (left), we see that in order to build more expressive routing functions  $Z$ , in Poly one can only do so by increasing the number of skills at each layer. However, this has a significant impact on the number of multi-task parameters which much be kept in order to perform few-shot transfer. MHR on the other hand, can increase routing capacity in a much more parameter efficient way.
A.2.1 On the granularity of routing tensor in MHR
Here we provide additional results when modifying the granularity of  $\mathbf{Z}$  for MHR. We see that one can easily trade-off more parameters for better performance.
Figure 5: Routing-Only Fine-Tuning (MHR- $z$ )
In our work, we focus on advancing parameter-efficient fine-tuning methods for cross-task generalization. While our research primarily addresses technical challenges and performance improvements, when applying such methods, it is crucial to consider the potential negative societal impacts. Specifically, we believe that prior to applying our proposed adaptation method, critically examining the potential biases and ethical implications of the underlying large language model, and the data itself must be properly addressed. This includes issues related to fairness, privacy, and the spread of misinformation.