Modular Deep Learning
Authors: Jonas Pfeiffer
Sebastian Ruder
Ivan Vulić
Edoardo Ponti
Venue: Accepted by TMLR ()

Abstract: Transfer learning has recently become the dominant paradigm of machine learning. Pre-trained models fine-tuned for downstream tasks achieve better performance with fewer labelled examples. Nonetheless, it remains unclear how to develop models that specialise towards multiple tasks without incurring negative interference and that generalise systematically to non-identically distributed tasks. Modular deep learning has emerged as a promising solution to these challenges. In this framework, units of computation are often implemented as autonomous parameter-efficient modules. Information is conditionally routed to a subset of modules and subsequently aggregated.  These properties enable positive transfer and systematic generalisation by separating computation from routing and updating modules locally. We offer a survey of modular architectures, providing a unified view over several threads of research that evolved independently in the scientific literature. Moreover, we explore various additional purposes of modularity, including scaling language models, causal inference and discovery, programme simulation, and hierarchical reinforcement learning. Finally, we report various concrete applications where modularity has been successfully deployed such as cross-lingual and cross-modal knowledge transfer.

Published in Transactions on Machine Learning Research (11/2023)
Modular Deep Learning
Jonas Pfeiffer"
jonasp/eiffer@@goole..om
Google Mind
Sebastian Ruder*
ruder@google.com
Google DeepAiidd
Ivan Vulié University of Cambridge
in2500cum.acc ac...maacccc.a.. uk
Edoardo M. Ponti"
eponti@ed.ac. uk
University of Edinburgh University of Cambridge
Reviewed on OpenReview:
Abstract
Transfer learning has recently become the dominant, paradigm of machine learning. Pre- trained models fine-tuned for downstream tasks achieve better performance with fewer labelled examples. Nonetheless, it remains unclear how to develop models that specialise towards multiple tasks without incurring negative interference and that generalise systematically to non-identically distributed tasks. Modular deep learning has emerged as a promising solution to these challenges. In this framework, units of computation are often implemented as aulonomous paramneerr-fffcient modules. Information is conditionally routed to a subset of modules and subsequently aggregated. These properties enable positive transfer and systematic generalisation by separating computation from routing and updating modules locally. We offer survey of modular architectures, providing unified view over several threads of research that evolved independently in the scientific literature. Moreover, we explore various additional purposes of modularity, including scaling language models, causal inference and discovery, programme simulation, and hicrarchical reinforcement learning. Finally, we report various concrete applications where modularity has been successfully deployed such cross-lingual and pross-modke knowledge transfer.
Table of Contents
Introduction and Motivation
3
Modular Deep Learning
5
2.1 Taxonomy
2.2
Notation
Computation Function
8
3.1
Parameter Composition
3.2
Input Composition
12
3.3
Function Compusition
12
3.4
Hypernetworks
15
Authors contributed equally.

---

Published in Transactions on Machine Learning Research (11/2023)
3.5
Unitying Parameter, Input, and Function Composition
Routing Function
16
4.1
Fixed Routing
4.2
Learned Routing
4.2.1
Challenges of Learned Routing
4.2.2
Hard Learned Routing
4.2.3
Soft Learned Routing
21
4.2.4
Hypernetworks
23
4.3
Level of Routing
23
A Agregation Function
5.1
Parameter Aggregation
Representation Aggregation
Input Aggregation
Function Aggregation
Training Setting
Joint Multitask Learning
................................
6.2
Continual Learming
6.3
'aramcter-ofliiccnt Transfer Learning
A applications in Transfer Learning
30
7.1
Parameter- Efficient Fine-tuning
30
7.1.
Machine Translation
30
7.1.2
Cross-Lingus Lingual Transler
31
7.1.3
Domain Adaptation
33
7.1.4
Knowledge Injection
33
7.1.5
Speech Processing
34
7.1.6
Computer Vision and Cross Cross-Moda Learning
34
7.1.7
Comparison and Design Principles
35
7.2
Task Generalisation
36
Other Purposes of Modularity
Hierarchical Reinforcement Learning
Programme Simulation
8.3
Causal Discovery and Interence
Conclusions
Future Work
Modular Instruction Tuning
2

---

Published in Transactions on Machine Learning Research (11/2023)
1 Introduction and Motivation
Transfer learning has recently become pervasive in machine learning technology, such as in natural language processing (Ruder et al.] 2019b; Brown et al. 2020), computer vision (Dosovitskiy et al. 2021). and reinforcement learning (Reed et al. 2022), among other areas. In its most successful incarnation, transfer learning consists of pro-training a model on vast amounts of raw data in a self-supervised fashion and subsequently fine-tuning it for new tasks based on a small number of labelled examples. Despite its success, this paradigm for transfer learning suffers from a series of limitations in various settings. Firstly, in multi-task fine-tunin, the learning signals from different tasks may negatively interfere with each other (McCloskey & Cohen 1989). Similarly, in continuous learning, adapting to new examples can result in catastrophic forgetting of knowledge acquired from previous examples [Sutton 1986 French 1999 Secondly, in settings where the training and evaluation distributions are not identical, these models fail in generalising systematically Lake & Baroni 2018 Hupkes et al. 2020). This makes models brittle and inaccurate and hampers their deployment in real-world applications, where distribution shifts are common.
In contrast, many biological and artificial systems do not suffer from these weaknesses by virtuc of their modularity (Fodor 1983 Ballard 1986), defined as the correspondence between strongly interconnected components of a system (i.e., modules) and the functions they perform (Baldwin & Clark 2000: CIrich 1995). In other words, each module is specialised for a unique purpose, for which it is reused consistently. In animal brains, this favours evolvability, the ability to adapt quickly to new environments, and resilience to environment perturbations (Wagner et al. 2005 because it makes rewiring connections easier than in
monolithic, entangled networks (Kashtan & Alon 2005). Artificial systems, such as programming languages and computer hardware, are similarly designed in a modular fashion (Booch et al. 2008 Baldwin & Clark 2000 because this modular design favours consistency, ease of adaptation, and interpretability.
To what extent, then, do 'vanilla' neural networks display the desirable property of being modular? In principle, given their fully connected nature, they could develop such a structure as by-product of optimising loss for a downstream task. Recent structural analyses based on hierarchical clustering of neurons revealed that vanilla neural networks can indeed learn such modular pattern (Watanabe) 2019 Casper et al. 2022
Foroutan et al. 2022). Favourable conditions for the emergence of modularity include multi-task learning DDbs et al. 2022] and regularisation through dropout (Lange et al.] 2022). In particular, from a structural perspective, populations of neurons may activate jointly in response to specific features of the input or the output classee resulting in similar changes in model performance when ablated (Meyes et al., 2020). From functional perspective, multi-task learning may lead to segregated, specialised sub-networks. (Yang et al., 2019 Dobs et al. 2022). On the other hand, Csordás et al. (2021) revealed that given sub-network, does not tend to be re-used for similar sub-tasks nor to be combined with others to express more complex functions. In fact, in many cases, the performance of a model on simple tasks requiring a certain skill and composite tasks requiring a combination thereof is entirely uncorrelated (Li et al. 2022a).
For this reason, previous work explored the idea of designing neural networks that are explicilly modular (Jacobs et al.] [191a] [Rosenbamm et al.) 2018 Pontil 2021 Mittal et al., 2022). This has the goal of achieving not only functional specialisation (Zhang et al. 2022b), but also re-usability and composability. In particular, these methods involve identifying 1) modules in a neural network that can be updated locally and asynchronously, without affecting the rest. of the parameters; 2) rouling function that chooses a subset of modules for each example or task; and 3) an aggregation function that aggregates the outputs of the active modules. Each of these three ingredients can be manually specified or learned. We provide several case studies of different configurations of these components in Figure 1
The main advantages of modular neural architectures are positive tronsfer, compositionality, and parameter efficiency. Firstly, modularity encourages positive transfer by encoding similar functions with the same module. At the same time, it prevents interference and forgetting by allocating distinct functions to dilferent dedicated modules (Jacobs et al. 1991b). For instance, massively multilingual Transformer-bseed models in NLP are
These phenomena have also been referred to as spatial and temporal 'crosstalk' (Jacobs ct al. 1991b LLange et al. 2202 found that clusters identified through dowustream (output) information do not match with the clusters Identified through upstream (input) information. They attribute this phenomenon their different roles, namely disentanglement of the input structure and composition of the output structure, respectively.
3

---

Published in Transactions on Machine Learning Research (11/2023)
(a) MAD-X
(b) Polytropon
(c) Mixture-of-Exper Transformer
Figure 1: Case studies of modular deep learning; best viewed in colour. Green components illustrate different
routing functions (see $ 4); shade-of-purple components illustrate modular computation functions (see $3
lab MAD-X (Pfeiffer et al. 2020b) uses Adapter layers with fixed routing for zero-shot cross-lingual transfer.
Ib Polytropon (Ponti
2022 uses low-rank adapters (ToRA; Hu et al. 2022 with hard learned routing
for few-shot task adaptation. Ic MoE Transformers (Fedus et al.) 2021 Clark et al., 2022 inter alia)
Multi-Layer Perceptrons with top-k soft routing, in order to scale to larger model sizes. The three representative models illustrated here only fraction of possible configurations from the 'configuratinn manifold' that can be created by varying the components surveyed in $3
known to suffer from 'curse of multilingualityy (Conneau et al. 2020 due to the conflicting information that the gradient from each language-ppecific loss carries (Wang al. 2021b). A possible solution is augmenting these entangled, fully shared models with specialised modules responsible for individual languages (Pfeiffer et al.) 2020b 2022b). More generally, as the range of tasks modelled jointly by single model becomes increasingly diverse, modularity may be instrumental the advent of general-purposo, multi-modal agents that encompass vision, language, and action (Reed el al.] 2022].
Secondly, modules representing different skills (at the task level) features (at the example level) can be composed together and updated locally, without affecting the rest of the network. These two properties are crucial in two main settings, which correspond to different aspects of systemalic generalisation: one is the ability to re- re-compose, i.e. zero-shot transfer to tasks consisting of new combinations of learned skills, or examples consisting of new combinations of observed features (Hupkes et al. 2020). For instance, while modules for the Guaraní language and for dependency parsing can only be trained separately due to the lack of annotated data for dependency parsing Guaraní, they can be composed to perform inference on this unobserved task language combination (Pfeilfer al.) 2020b). Similarly, in hierarchical reinforcement learning, agent follow different sequences of modular policies known options in tasks requiring the completion of similar sub-goals in different orders (Sutton et al. 1999 Precupl 2000 The other aspect of systematic gencralisation is robustness. In fact, if modules are taken to correspond to independent and reusable physical mechanisms (Schölkopf et. al.) 2012), local shifts in their distributions require updating only the parameters accounting for the affected skills or features (Goyal et al. 2021 Schölkopf et al. 2021). while the rest of the model remains invariant to the change. In practice, the ability to perform local updates facilitates sample efficiency, as fewer examples are necessary to adapt models new tasks (Bengio al. Ponti et al. 2022
2020
4

---

Published in Transactions on Machine Learning Research (11/2023)
Thirdly, an additional advantage of modular neural architectures is parameter and time efficiency. In this framework, fine-tuning model on a specific task only requires storing a modular adapter rather than a separate copy of the entire (typically large) model. What is more, modules can be added or removed on-the-fly in an incremental manner, adjusting the model capacity according to the task complexity. This ability is known as conditional compulation (Bengio el al. 2015). Finally, modularity enables language models to scale to larger numbers of parameters while retaining the same time complexity, by selecting only a small set of experts per example (Shazeer et al., 2017 Fedus al., 2021
As the main contribution of this survey, we offer unified view of modular deep learning, illustrating how many families of methods can be defined along four key dimensions: 1) how they implement modules, which constitute the minimun unit of computation; 2) how they select active modules through a routing function; 3) how module outputs are aggregated; and 4) how the modules are trained with the rest of the model.
For module implementation, we discuss sparse subnetworks (Hu et al. 2022 Ansell et al. 2022), adapter layers (Rebulli et al., 2018: Pleiller et al. 2020b), and prefix tuning (Li Liang (021), among others. These methods have been proven
effective way to adapt large pre-trained models, achieving better performance and sample efficiency than alternative strategies such as in-context learning @Liu et al. (002b)), which may be brittle (Lu et al.) 2022). In fact, modules can also take the form of human-cngineere engineered prompts, where the model is provided with out examples (Brown et al., 2020) or task instructions (Wei et al. 2022a).
While many module implementations share the same underlying functional form (He et al. 2021], 1, they offer different trade-offs between efficiency and performance.
We then discuss how routing functions control the flow of information to the modules: in fixed routing, module allocation is mamually defined when expert knowledge is available (aaaashirr & Waibel 1992 Rajendran et al. 2017 inter alia). In learned routing, & parameterised routing function is inferred during training. This, however, poses a series of challenges, such as training instability, module collapse, and overfitting (Rosenbaum ct al. 2019). Orthogonally, we also distinguish between hard and soft routing. In hard routing, only a subset of modules is activated (Rosenbaum et al. 2018 Ponti et al. 2022 Fernando et al., 2017 inter alia). In soft routing, all modules are aggregated according to innuus scores (Jacobs et al. 1991b Jordan & Jacobs 1994). While soft routing is amenable to vanilla gradient descent, it is highly inefficient. On the other hand, hard routing requires approximate inference but facilitates conditional computation and module specialisation. When iult modules are selected, several aggregation stralegies are possible. For instance, these can be based on interpolating the parameters of active modules (Ansell et al. 2022) or an attention mechanism over the module outputs (Pfeiffer et al. 2021a).. Alternative methods include input prompt concatenation (Vu ct al.
2022b and function composition (Andreas et al. 2016b).
Finally, modules can be trained jointly with the rest of the basc model in multi-task learning (Caruana 1997:
Ruder 2017), added sequentially in classic continual learning [Rusu al.] 20166), or integrated post-hoc into already pre-trained and frozen model (Rebuffi et al.) 2017 Houlsby et al. 2019). The last scenario is most common with current state-of-the-art models, which are trained as dense, fully shared models and may be 'modularised' after pre-training.
Crucially, this taxonomy reveals unexpected connections between several independent threads of research, including aggregation functions and mode connectivity (Frankle et al. 2020), routing and hypernetworks (Ha) ct al. 2017), among others. We further illustrate scries of applications of modular networks in transfer learning across different areas such natural language processing, computer vision, and speech processing. In addition, we show how modularity plays an important role in causal inference and discovery, programme simulation, and hicrarchical reinforcement learning. We hope that our overview will spark future research on modular deep learning in areas that may benefit from it such as community-driven efforts to develop and maintain machine learning technology.
2 Modular Deep Learning
This survey focuses on modular deep learning: namely, on models composed of modules. These are autonomous computation functions that, depending on their architecture and purpose, are variously referred to as adapters
(Rebufii et al.] 2017 Pfeiffer et al. 2020a), options (Sutton et al. 1999 Precupl 2000), or experts (Jacobs
5

---

P Published in Transactions on Machine Learning Research (11/2023)
Notation
Definition
X
Input data
y E
Output data
h HEH
Hidden representation
tET
Task index A computation function
f TUNUH-D0
0
Shared paramelers
M - O
Set of module parameters Vector of routing scores Routing function Routing parameters Aggregation function Aggregation parameters
OEA
r: X -UTT-A
g
7
Table 1: Notation and definition of important variables, functions, and operators.
et al. 1991a Jordan Jacobs 1994). Crucially, these modules are distinguished from routing function, which controls the information flow to the modules. Finally, aggregation function aggregates their outputs. Modules can be optionally combined with fully shared (thus, non-modular) parameters part of the same neural architecture. In order to provide a unified view of the landscape of modular deep learning, we create a taxonomy of four dimensions of variation: computation, routing, aggregation, and training, These dimensions are mutually independent; hence, many methods can be interpreted as diffcrent combinations of these dimensions, listed [ 2.1 Concurrently, we provide unified, consistent. notation in illuminate the relationship among such methods.
S 2.2 which helps
2.1 Taxonomy
1) Computation function: How is each module implemented? ($ A module may consist of any component of neural architecture, such as multiple copies of model (Jacobs et al. 1991a) or one of its layers (Fedus et al. 2021). Alternatively, as it is common in transfer learning, modules can be combined with function paraneterised by fully sharod pro-trained weights. In this case, we distinguish between modification of parameters (parameter composition). concatenation with input features (input composition), and function composition by stacking neural modules.
2) Routing function:
How are active modules selected? 4 Under fized routing, categorise approaches where the routing function is fixed. This assumes that the specialisation of each module, well as the combination of modules required for each task, is known priorí. In learned routing, the parameters of the routing mechanism are learned during training. In this case, routing is soft if all modules are ranked through a continuous score, or hard if each module given a binary score (active or inactive).
3) Aggregation function: How are the outputs of the active modules aggregated? ($ 5 We differentiate between methods that compose the outputs of the active modules deterministically (e.g., based weighted average) from those where the aggregation function is implemented learnable neural network that depends on the output of all modules.
4) Training setting: How are the modules trained? ($ 6 Some methods, such as MoEs, train the modules (and possibly the routing function) jointly with the shared weights of randomly initialised model. As an alternative, transfer learning approaches introduce modules post-hoc after pro-training weights and adapt them during fine-tuning. In continuous learning settings, instead, new modules may be introduced iteratively for every new task in sequence.

---

Published in Transactions on Machine Learning Research (11/2023)
2.2 Notation
More formally, let a neural network fe X' - y be decomposed into a graph of sub-functions. In the simplest case, this graph is a linear chain fa, o fo, o... o fa, where o stands for function composition. These sub-functions refer to the model's | layers, each with unique indexed parameters 0,, i = I...., I In turn, these can be further decomposed recursively into a graph of their constituent sub-functions: for instance, Transformer layer (Vaswani al., 2017) includes linear mappings for the query, key, value, and output, well non-linear werd-f network, and residual connections. We further denote the values of the parameters at initialisation as 0¹ and the parameters after training are denoted 0*.
Any i-th sub-function with input æ can be modified by a module with parameters & from the inventory M = f in the following different ways:
1. parameter composition: (x) - (x)), where @ stands for an operation that composes the original
parameters with the module parameters, such as element-wis. wise addition. An example is low-rank (Hu
et al. 2022) or sparse (Ansell et al. 2022 adapters.
2. önput composition: f((x) = fo. ((Ø, æl), where [, stands for concatenation. An example is prefix tuning Li & Liang 2021
3. function composition: f((x) fo fe. (x). where the outputs of the first function is fed into the second function. An example are adapter layers (Rebuffi et al. 2017
For each i-th sub-function multiple modules from the inventory M, can be selected through routing function r(-), which returns score aj for each module fø, conditioned on the data itself, such language token visual region or the full input æ, or metadata such as the task identity i E T. Note that a can be fixed a priori through expert knowledge or learned through an appropriate parameterisation rp(+), where p refers to (learnable) parameters of the routing function. Often, the routing function takes special forms:
1. In hard routing, a E {0, 1M is a discrete binary vector. If these parameters are learned, inference usually relies on score function estimators, stochastic ro-parameterisation, or evolutionary algorithms.
2. In soft routing, a E [0, M is continuous probability distribution, such that Σ, 0g =
3. Finally, Rladl can be unnormalised score vector. This is the case in lincar hypernetworks (Ha el al. 2017], where a is usually interpreted as lask embedding and the row-wise stacked module parameters + = [oI..... Φ M]] act as a parameter generator.
Finally, the output of each module is combined through aggregation function g()) The aggregation function usually takes two possible forms. One consists of a deterministic operation based on the routing scores (e.g., weighted averaging of module parameters or outputs). The other consists of learnable neural network, such as an attention mechanism between the modules' inputs and outputs (Pfeiffer et al., 2021a). When we put the computation function, routing function, and aggregation function together, we obtain the general recipe for a modular function, rllustrated in Algorithm
We abuse notation by treating indexing functions, fú identical indexing the parameters of fuuction, fa,- In this survey, both used interchangeably. "To avoid clutter in terminology, throughout this work we the term composition to refer the merger of computation functions ($ 3. and the term aggregation refer to different approaches of combining the outputs different. modules ($ 5
7

---

Published in Transactions on Machine Learning Research (11/2023)
Algorithm 1: Forward pass of modular function
Inputs: example æ, task f
2 a s r2(2, t) // Routing
3 H + 0
for Φj E M; do
hj f(x; 0, Φ5) // Computation
H t Huh
y a H) // Aggregation
Given shared parameters 0; for the i-th sub-function and a corresponding inventory of modules M,, we first sample a task t, and an imput æ. The routing scores a are obtained from the routing function r(-). We now compute the hidden representation h2 of each module Φj and aggregate them with the function g(-) into the output y. We elaborate on the settings for training these different components in overview of representative computation, routing, and aggregation functions in Table 2
S 6 We provide an
3 Computation Function
The computation function determines the design of a module. Various module architectures have been proposed such as MLP layers (Rosenbaum et al. 2018 Kirsch et al. 2018
Chang et al. 2019), independent RNNs (Goyal et al. ), independent CNNs Parascandolo et al. 2018). or special-purpose architectures Andreas al. 2016b). However, in transfer learning, modules are most often integrated into base architecture whose parameters fully shared. We identify three methods to merge single module with the corresponding sub-function: parameter composition, input composition, and function composition. While all three methods instantiate modules differently, 3.5) We provide example illustrations of the three computation functions (in addition to hypernetwork) as part of a Transformer architecture in Figure 2 and provide a high-level overview of their trade-offs in Table which we further discuss in the respective sections
demonstrate how they can be seen in unified view in
3.1 Parameter Composition
Parameter composition methods augment the function fw of base model with weights W E oox with module parameters & E Rox where i is the input dimensionality, and o is the output dimensionality. In particular, the module inventory consists of a set of sparse or low-rank weights to ensure that the modules are parameter-efficient Therefore, the resulting function is parameterised as feed. where @ stands for element-wise addition.
Sparse Subnetworks Sparsity is common inductive bias based the assumptions (i) that only small number of parameters of an overpparameterised model are relevant for a particular task, and that (ii) similar tasks share similar sub-networks. This is the case, for instance, for language subnetworks in multilingual language models (Stanczak et al. 2022 Foroutan et al. 2022).
The most widespread method to induce sparsity is pruming. This can be interpreted as the application of binary mask (0, 1)101 that selectively keeps or removes each connection in model with trained parameters 0+: I' ferob where @ is element-wwse multiplication. The merger of 0 and b results in sparse subnetwork, but the corresponding model parameters usually remain dense for hardware and software T After training, the trained weights are sorted based on a criterion and a fraction (bottom-k) of the weights are set to zero. Examples of criteria include magnitude after convergence (Han et al. 2017 and change of magnitude between initialisation and convergence [Frankle & Carbin 2019
reasons
TThe comparison is mainly meant high-level guideline. Individual methods may have different trade-offs and mitigate certain weaknesses indicated in the table. 6In fact, sparse linear algebra operations graphic processing units remain highly inefficient, available all. Examples include the sparse tensor classes in Pytorch: https::/ppttorc.orr///essss wwt.salleesppseee.eeee sparse.html

---

Published in Transactions on Machine Learning Research (11/2023)
Method
Reference
Function
Sparse subnetwork
Frankle & Carbin
2019
fa
Supermasks
Wortsman ot 2020
f0,ob
Sparse fino-tuning
Ansell at al. 202
8 fotb00
Intrinsic dimension
Li et al.
201
f fo 1
Low-rank adaptation
Hu ct
2222
Prompting
Brown et al.
2020
4 = fa, ([0,æl) where Φ = Emb(p)
Retrieval augmentation
Guu et
2020
fo, ([d. x))
where Emb( (pp, c))
Computation function
Prompt tuning
Lester et
J = fo, ($b..)
Multi-layer prompt tuning
LI & Liang
2021
K fe. ([ob.
Parameter sharing
Ruder
2017
8. fā. Vi E
Convolutional adapter
Rebulti et al.
2017
F fa. (Ja, (z)) where fo. (a) F
Transformer adapter
Houlsby al.
2019
fa. (Jo. (x)) where fo. (x) ((
Sa. (z) W (oW"))))),
Compacter
Mahabadi et 12021a
fa. (So, (x)) where
W = Σ) A; 0 B
Parallel adapter
Rebulli al. 2018
F. fo, (xx) + (a)
Rescaling
Bilen & Vedaldi [2017
R. fo, (x) C
Hypernetwork
Platantos et 2018
(a W J
Fixed routing
Hampphhre & Waibel
1992
TK. Σ, f(dj)1;(K)
Top-1 learned routing
Rosenbaum et al. 2018
J f(x; O, ,,) where argmax
Routing function
Top-k learned routing
Goval al. 1222
ja topsaa) f(x: 0,, Φ3)
Variable-size (threshold)
Rahaman al. 2021
P. catjeMst a f(a:; O. 4,)
Variable-size (soft partition)
Ponti et al.
2022
f = E a EjeM ay f {xx 0, Ø3)
Mixture experts
Jacobs et a..
1991b
R. JEM a, fax: 0., 2,)
Weighted top-k routing
Shazeer et al. 2017
vop Σ f(x; 0, b,)
Aggregation
Sparse weight addition
Ansell et al. (2022
fententh
fmnetion
Representation averaging
Ma et al. 2018
f Σ ashi
Input concatenation
Vu ct al.
20226
f. fol(or, Di, 21,z))
Attention-base based aggregation
Pfeifer et al.
2021a
J. = Attn(h Qu H.K, H.V.)
Sequential aggregation
Pleifeer al.
2020b
(So (x)))
Table 2: An overview of representative computation, routing, and aggregation functions. Each method is paired with a represcntative reference. In computation functions, skip connoctions are omitted for simplicity. Definitions: the model J, a model's sub-function ft, model parameters 0, module parameters &, parameters at initialisation 0 parameters after training 0* binary mask (0, 1)19, random matrix M, group G, input x, a model's embedding layer Emb(-), text prompt p, retrieved context e, filter bank F, routing scores or task embedding a, routing function r, subset of modules K, module inventory M.

---

Published in Transactions on Machine Learning Research (11/2023)
(a) Parameter Composition
(b) Input Composition
(c) Function Composition
(d) Hypernetwork
Figure 2: Different modular designs for Transformer architectures; best viewed colour. specfip modular components are illustrated in magenta and purple, respectively. (a) Parameter Composition ($ 3.1 A
sparse sub-network in the linear layer part of multi- heti--head--ttettion (b) Input Composition ($ 3.2
Prefix-tuning (Li & Liang] 2021 extends the imput by prepending embeddings to the key and value matrices
in the Transformer layer. (c) Function Composition ($ 3.3 Task-specific bottleneck layers transforming
the hidden representations are inserted in each layer (Houlsby et al.) 2019). (d) Hypernetwork ($ 3.40:
A small separate neural network generates modular parameters conditioned on metadata. We show its application to function composition but it is compatible with all computation functions.
As pruning generally leads loss in performance due to the change in network connections, the non-pruned weights are typically re-wound to their initialisation value and re-trained. In practice, rather than pruning all weights in a single run, iterative pruning is carried out over multiple stages (Han et al. 2015 Frankle & Carbin 2019). The models pruned in this fashion often retain if not surpass - the performance of the original dense model. The existence of a subnetwork with this property in any given randomly initialised model known as the Lottery Ticket Hypothesis (LTH; Frankle & Carbin 2019 Chen et al.) 2020). These 'winning lickets' have also been shown to exist in RL and NLP (Yu al., 2020), as well as in computer vision
Fraankle et al. 2020). Subnetworks achieve above-random performance even when kept fixed at their random
initialisation Zhouu et al. 2019 Wortsman et al. 2020 Zhao et al. 2020 so f' f0, In this case, they
are known as supermasks.
Winning tickets also occur in pre-trained models, such as language models (Chen et al. 2020
Prasanna et al. 2020). These often outperform tickets from randomly initialised models (Prasanna et al. 2020 and are less sensitive to specific typer-parameter choices (Sun al. 2020a).. Magnitude pruning, which relies on zeroth-order information (the absolute value of weight), is sub-optimal in this setting as fine-tuned weights typically stay close to their pre-trained values. Thus, magnitude pruning selects similar set of weights for pruning regardless of the downstream lask. Pruning based on first-order (gradient-based) information better captures the task-specific relevance of each weight (Molchanov et al. (017). For instance, movement pruning (Sanh et al. 2020) learns the mask b jointly with the parameters 0. As the mask is a discrete binary variable, they rely on straight-throug estimators (Bengio et al. 2013). Alternatively, b can be first learned as real-valued mask and then binarised via a thresholding function (Mallya al., 2018).
In addition to pruning, sparsification techniques can be employed for adoptation. In particular, sparse module & can be merged with pre-trained parameters O. For instance, in Sparse Fine-Tuning (SFT; Ansell et al. 2022 the LTH is re-purposed such that, instead of zeroing out weights with the lowest change in magnitude, they are simply frozen. Thus, only a subset of weights is fine-tuned] The difference between these and the original pre-trained model results in a sparse module & where di 0 if b, = 0, which can be
This is typically implemented by masking the gradient based on the binary mask b VoL(Je. D) where loss function and D is dataset (nsell al. 2022).
10

---

Published in Transactions on Machine Learning Research (11/2023)
Parameter efficiency
Training efficiency
Inference efficiency
Performance
Compositionality
Parameter
+
+
composition
Input
++
+
composition
Function
composition
Table 3: Comparison of computation functions along different dimensions. See the end of 3.1 parameter composition), 3.2 (input composition), and 3.3 (function composition) for further explanation. Composi- tionality is discussed in
plugged in and out of the model as fo fees- Diff pruning (Guo al. 2021 instead obtains sparse adapter by fine-tuning dense difference vector & regularised to be sparse with differentiable approximation to the Lo-norm penalty. Sung et al. (2021) induce fixed sparse mask by selecting the top-k weights ranked according to (a diagonal approximation of) their Fisher information. This second-order information reveals the impact of the change of parameter on the model predictions. Thus,
11 jîj E top-k E Ey- (wlx.) (Va log So- (y x;))²
by { otherwise
Beyond the sparsification individual weights, sparse model adaptation can also be structured. In this case, only group of model sub-functions fine-tuned, while the rest of the parameters remain frozen. The most common setting is for such group to correspond to subset of layers, e-g. the last (Donahue al. 2014). Groups can also relate to more fine-grained parts of the model. For instance, group consisting of model's i bias parameters is practical choice as this removes the need to store the model's intermediate activations (Cai et al., 2020; Ben Zaken et al. 2022). At the level of parameter tensors, some methods prune filters in CNNs (Anwar et al. 2017 Newell ot al. 2019h whereas others prunc attention heads in pro-trained structured dilf pruning, members of group are
Transformers (Voita el al. 2019 Michel el al.j 2019).
encouraged to share the same mask value (Guo et al. 2021).
Low-Rank Modules Similar to sparsity, another efficient solution for the module parameters Φ; to lie in a low-dimensional subspace. Li et al. (2018) show that models can be optimised in low-dimensional, randomly oriented subspace rather than the full parameter space. In this setting, the module parameters & E RR are low-dimensional compared to the model parameters 0 E RD and d & D. A random matrix M E Rdx be used to project from d to D: fo ffe+MM- An efficient way to compute M is via the Fastfood transform [Le et al. 2014), which factorises M as random linear matrices. Specifically, M = HGTHB consists of a Hadamard matrix H, random diagonal matrix with independent standard normal entries G, a random diagonal matrix with equal probability #1 entries B, and a random permutation matrix II. Li et al. (2018) refer to the minimum that achieves within 90% of the full-parameter model performance as the intrinsic dimensionality of a given task. Aghaaanyyn et al. (2021) investigate the intrinsic dimensionality of various NLP tasks with different pre-trained models. They observe that it decreases during pre-training and that larger models have lower values.
However, storing the random matrices results in substantial memory overhead and is slow to train MMahabadi et al. 2021ah. If the weight matrix W E Rxx is small enough, we can directly compose it into low-rank the output
matrices W ABA where A E x and B E Book where is the input dimensionality, dimensionality, is the rank of the matrix, and À scaling syper-parameter. To save space, the factorisation may be only applied to certain groups of parameters G. In LoRA (Hu et al., (022)) this group corresponds the linear projections in the self-attention mechanisms of each Transformer layer: J/ =ee + B, A, VI E G.
11

---

Published in Transactions on Machine Learning Research (11/2023)
Overall, parameter composition methods (both sparse and low-rank) are very parameter-cfficient and often require updating less than 0.5% of model's parameters (Guo et al. 2021 At inference time, they keep the model size constant or even reduce it, if the resulting model is sparse. This is compelling as it enables a plug-in replacement of the original with the modular model without any changes to the underlying architecture. Sparse modules, however, increase the time complexity of optimisation as they lypically require multiple iterations of re-training. Finally, state-of-the-art parameter composition methods, e-g-, LoRA (Hu al., 2022 and SFT (Ansell al. 2022 achieve strong performance in zero-shot and few-shot transfer.
3.2 Input Composition
Input composition methods augment function's input æ by concatenating with a parameter vector Φ;: [(x=) ([o;,.)))) æ²). The most common strategy is to augment the input fed to the model's first. layer fi-
Prompting In a prompting setup with auto-regressive language models (Brown et al. 2020 or encoders (Schick & Schütze b), the input prompt consists of (optional) instructions and (optional) in-context examples that have been converted to natural language. From a different perspective, the task-specific text, prompt, when encoded using the model's 's embedding layer Emb(-), corresponds to modular parameters & that elicit the desired behaviour (Gao et al.) 20211 Liu et al., 2023): Emb(p) = Φ More claborate prompts p such rationales have led to improved few-shot reasoning performance Weei et al. 2022c Kojima et al.,
2022 Shi et al., 2023). However, models ostensibly sensitive to the formulation of the prompt as well
to the set and order of the (lew-shot) examples (Zhao el al. 2021 Lu el al. 2022 Webson & Paallck 2022
Continuous Prompts Instead, a continuous prompt vector & can be learned directly (Lester et al. 2021 Liu et al. 2021b Zhong et al.] 2021 Hambardzumyan et al.) 2021). However, if Φ is only concatenated with the first layer's input, the model has limited capacity to adapt to specific task. As result, such continuous (also called soft) prompts perform poorly smaller model sizes and on some harder tasks MMahaaadi et al. 2021a Liu et al.)
2022c To mitigate this, initialisation via multi-task learning has been
proposed Vu et al.) 2022c). As an alternative, module vectors Φ, can be learned for each layer of the model (Figure 2b Li & Liang, 2021 Liu et al. 2022c). While this increases the number of parameters, it increases the modules' capacity to adapt to a given task. In practice, module parameters in the form of prefix vectors Φ = P R(xd are prepended to the keys and values of every multi-head attention layer. Attention is defined as f(x))= = Attn(x: W CW,, ka CW;) where Wy. We, W, E Rdddd, are the projections that produce the queries, keys, and values, and C' E Buxd is sequence of context vectors. Multi-layer prompt tuning thus takes the following form:
f((x) Attn( [P CW.), CW:)).
Retrieval Augmentation Beyond individual prompts, the inpul can be augmented with additional context. e from a retrieval model. Retrieved documents are appended to the input and are used for conditioning the language model (Guu et al. 2020 Lewis et al. 2020a Emb([p,e) Φ.
In summary, input composition is exceptionally parameter-efficient as it only adds a very small number of parameters. However, these parameters extend model's is context window, which makes them less efficient during training and inference. Prompt tuning mothods also require large models to achiove decent performance.
3.3 Function Composition
While parameter composition deals with individual weights and input composition methods act only on a function's input, function composition methods augment the model with new task-specific sub- functions (see Figure 2ch: ((x)) = fo: Se. (m) = So. (fo, (x:),, where o stands for function composition.
Parameter Sharing Models in multi-task learning traditionally consist of shared layers So stacked under task-specific modules f [Ruder 2017). Conversely, given models for tasks t and expressed composition of functions f2, .... fú, and 13. fě,, respectively, multi-task architecture can also be obtained by tying sets of parameters between the models: S/. = fg. Vi E G where the group G contains the set of shared
12

---

Published in Transactions on Machine Learning Research (11/2023)
O
(a) Sequential Bottleneck Adapter
(b) Parallel Bottleneck Adapter
(c) (IA)3
Figure 3: Different approaches of function composition. (a) Sequential Bottleneck Adapter: The first adapter architecture proposed for transformers which consists of two bottleneck layers placed after the multi-head attention (MHA) and food-forward (FF) layers (Houlsby et al., 2019). (b) Parallel Bottleneck
Adapter: Bottleneck layers processed in parallel to the MHA and FF layers of the pre-trained transformer
components (Rebuffi et al. 2018 Stickland & Murray
2019 He et al. 2022a (c) (IA)3; Rescaling operations
performed within the MHA and FF layers (Liu et al. 2022b).
layer indices Many multi-task neural architectures be characterised in terms of their definition of G, which determines which modules task-specific and which ones shared. This is the case, for instance, of 'shared trunk' approaches in computer vision (Zhang et al.] 2014 Ma et al. 2018 and approaches with
supervision at different layers in NLP (Søgaard & Goldberg 2016 Sanh et al. 2019 Liu et al. 2019).
Some approaches learn finer-grained interactions between pairs of modules. Misra et al. cross-stilch unit, which linearly combines the inputs al every layer (xt x:")
2016 propose the a x where
W
ij
at
at
xtt
at
a
x
and a E R. Sluice networks (Ruder et al. additionally employ soft selection of the skip connections from all layers at the output layer l:
2019a) extend cross-stitch units to multiple modules per layer and
B9
x
of
and 3 E R. On the other hand, Gao et al. (2019) fuse features from multiple tasks through a 1xl convolution. Bragman et al. 2019h employ variational inference to assign filters in CNN to task-specific shared roles.
Rather than learning which modules should be shared among which tasks, which is combinatorially large problem, Lu et al. (2017) and Vandennende et al. (2020) start with fully shared model and then dynamically widen it during training, by cloning function fe, into now modules f
shared among a smaller subset of tasks, top-down order across layers. More information on parameter shariin strategies multi-task learning can be found in relevant surveys (Ruder) 2017 Crawshaw 2020
8nn this view, there clear differentiation between model parameters 0 and module parameters d. We omit the layer index simplify the presentation.
13

---

Published in Transactions on Machine Learning Research (11/2023)
Adapter Layers As an alternative to parameter sharing, new task-specific learnable function fø: can be composed with an (often frozen) shared function fa,- As the main purpose of such modules is adapting a pre-trained model to new tasks, they are also simply known as 'adapter layers' We provide examples of different adapter layers in Figure 8
The adapter's design and composition with the pre-trained model are often modality-speeiffc. In computer vision, the adapter typically consists of 1 Xx] convolution, i.e. fo, (a)) F*x where I' is bank of 1 x filters and is the convolution operation (Rebuffi et al. 2017). The module is then inserted between the convolutional blocks of pre-trained model, such as ResNet (He et al.) 2016). In NLP, a bottleneck architecture has become popular which consists of a down- and up-projection, coupled with an intermediate activation function a: (x) (d (W"")) where W E Rdaxk and W E R de is the dimensionality of the input (typically the hidden dimension), and is the bottleneck dimension. a is commonly a non- non-linearity such as a ReLU unit (Figure 3a} Houlsby et al. 2019; Pfeiffer et al.) 2020b). In a Transformer model, adapters are placed both after the multi-head attention and the food-forward layer (Houlsby et al., 2019). just after the multi-head attention (Bapna & Firal, 2019). ; or just after the leed-forward layer (Peiifeer et al. 2020b).
Other variants for such as the identity function, standard multi-head attention, and multi-head attention with shared projection matrices have also been explored (Stickland & Murray 2019). Mahabadi et al. (2021a) propose Compacter, a hyper-complex, low-rank adapter that reparameterises W in the adapter
88: W- -i=1 A. & B; where A; E R=*= is shared across layers (n is a hyper-parameter), B; E R x is parameterised as low-rank matrix B; = s;t and @ is the Kronecker product.
Adapters can be routed sequentially or in parallel. Sequential adapters, are inserted between existing functions: [[()) - fa, (Jo, Rebulli et al. 2017 Houlsby et al. 2019). Parallel adapters are applied in parallel to a
pretrained function: [x) = x fo. x fo. (x) (Figure 3b Rebuffi et al., 2018 Stickland & Murray 2019
He et al. 2022a). Moreover, adapters involve two residual connections: between the output of fe, and the output of fø.. which is further added to and normalised. Adapters have been shown to lead to increased sample efficiency, flatter minima, and more robustness to hyper-parameler choices compared to standard
model fine-tuning [Mahabadi et al. 2021b He et al. 2021 Han et al. 2021
Function Augmentation Adapters and more complex module designs can also be used to augment a base model with information and behaviour that it otherwise would not be able to access. This can be through adapter layers pre-trained on specific domains (Wang et al. 2021a) or other modalities (Alayrac et al. 2022).
Modules can also be designed to attend over explicit key-value memory reprosentations of entitics and facts (Verga et al. 2021)) and general domain knowledge (Cheng et al.
to enable a model to perform certain Andor et al.) 2019). More broadly, function composition enables the use of arbitrarily complex auxiliary modules. We highlight how function
2023
types of operations such as arithmetic reasoning (Trask et al.
2018
77.14 For overview of how modules can be used to allow language models to use tools and to act, we direct the reader to Mialon et al. (2023)
composition has been used to inject knowledge into models in
Rescaling The output representations can also be directly transformed via element- multiplication with a vector of learned parameters: [((=) = fo, (a)) @ Φ5 Crucially, this is equivalent to stacking the original function fo, with a linear transformation W = I Such task-specific rescaling is typically applied to batch normalisation parameters in computer vision Bilen & Vedaldi 2017) and to layer normalisation parameters
in NLP Houllby et al. 2019
The adapter (IA)3 (Figure 3c Liu al. 2022b) multiplics learned vectors with the keys and values in self-attention blocks and the intermediate activations in position-wise feedforward networks in the Transformer. Rescaling activations favours dimensions that are important for given task. Multiplication with binary mask is a special case of rescaling that incorporates sparsity: Sttezoskii et al. (2019) multiply a task-specific random binary mask b with a function's input æ at every layer.
Overall, standard function composition methods such as adapter layers typically require more parameters as the new function depends on a model's input size and hidden size. While they do not require storing the gradients of the frozen parameters, they increase the number of operations at training and inference time. State-of-the-art function composition methods match outperform standard fine-tuning.
14

---

Published in Transactions on Machine Learning Research (11/2023)
3.4 Hypernetworks
In the above-mentioned adapters, different modules $1...., .IM correspond to disjoint sets of parameters. However, the modules may benefit from sharing information. Rather than learning Φ; directly, a (small) neural network W. known as a hypernetwork, can generate the module parameters instead, conditioned on an embedding a (Ha et al. 2017 Platanios et al. 2018). Thus, & = Wa. As a result, the modules are "entangled', which violates the strong delinition of modularity that postulates that modules are autonomous (Goyal et al. 2021). In fact, in hypernetworks, computation and routing are inseparably intertwined. In fact, foreshadowing our discussion in 1.2.4 the embedding a can also be interpreted as unnormalised, learned routing scores for each task. In turn, the parameter generator weight would correspond to a set of modules stacked column-wise: W = [ M|]-
Hypernetworks can also be conditioned on inputs æ (Figure 2dh. For instance, in conditional batch normalisa- tion (de Vries et al. 2017), rescaling parameters are gencrated based on a representation of the model input obtained via an LST Feature-wise linear modulation (FiLM: Perez al., 2018 generates an element-wise affine transformation that is applied to image features, conditioned on the linguistic input of the model, for text-and tasks. In self-modulation for Generative Adversarial Networks (Chen et al. 2019), the affine transformation is applied to hidden representations of the generator conditioned on the noise sample. Bertinetto et al. (2016) conditions the parameter generator on individual examples, in order to perform one-shot learning.
Hypernetworks have been used to generate a diverse set of module parameters, including classifier heads Ponti et al. 2021), continuous prompts (He et al. 2022c), and adapter layers (Üstūn et al. 2020 Ansell et al., 2021 Mahabadi et al. 2021b). most commonly conditioned on task (Mahabadi et al. 2021b) or language embeddings (Ust et al. 2020 Baziolis et al. 2022). Such task or language embeddings a can themselves be learned directly from random initialisations or fixed as the typological features of a language (Üstūn et al.) 2020 Ansell ct al. 2021). This is a stratogy to integrate side (or metadata) information about the relationship among languages. Other examples of side information, such as the example label y, can be integrated into the hypernetwork input embedding via bi-linear interaction (Chen et al. 2019).
Nevertheless, even the smallest possible module generator network is a linear projection W Rd To make the hypernetwork more parameter-efficient it can be shared across layers by conditioning it on the module position in the neural architecture, in addition to the task index (Mahabadi et al. 202Tb). In general, the hypernet can be conditioned on multiple (concatenated) embeddings; e.g., one corresponding to the task index and another to the language index. This allows the hypernetwork to generalise systematically to new task language combinations at inference time, In particular, the hypernet can either generate a single module from all the emboddings (Ponti et al. 2021 or separate modules (Ansell et al.] 2021 Üstün et al. 2022). In turn, the embedding combination chosen for any example is a form of hard routing (cf. 4.2.2
3.5 Unifying Parameter, Input, and Function Composition
While the above methods may seem different, they all covertly share a similar functional form. He et al. (2022a) cast LoRA (Hu et al. 2022), prefix tuning (Li & Liang, 2021), , and bottleneck adapters (Houlsby let al., 2019), representative methods of the three composition functions, into the same framework. We extend their framework to cover parameter composition, input composition, and function composition in general. Specifically, all modular computation functions can be reduced to function composition: the output of the function fo, of a model is added to a new term that depends on a learned function fø: f((x) fo(x)) + fa. (2)).
For function composition methods, this form is the most natural. In the case of parallel adapters, for instance, f((x) = fe. (x) + fo,(x) (x) where fo. (v) may be a multi-head attention module fo, (x) = MHA(C, = hhead,..... head, W., with head, = Attn(æWW CW? CW?), and fq, (x) ('(a((">))) In this setting, 0, and Φ are independent and must only agree regarding the dimensionality of their inputs and outputs.
For parameter composition mothods, which modify the parameters directly, the dimensionality of the module parameters & should match exactly the original parameters 0,. For instance, if we apply the module to linear projection, then they should consist of weight matrices 0: = W, E Rd=x and Φi V; Rd
15

---

Published in Transactions on Machine Learning Research (11/2023)
respectively. Because of linearity:
f((z) = (x) = fw+v(x)) = (W V) Wa Væ fo. ) fo. (x)
For instance, in the case of LoRA (Hu et al. 2022 V = AB:A, the case sparse adapters Ansell et al. 2022 V sparse matrix.
For input composition methods, with the form f] fo. ((oi, æ:), the equivalence is derived follows. Prefix tuning (L & Liang 2021) generalises other continuous prompt methods by concatenating prefix vectors Φ P P. E Ri to the keys and values of self-attention. He et al. (2022a) show that prefix tuning can be expressed in the following way:
f((x) = Attn(x [Pg CW Cw:)) (1 fa (a) ((x) softmax(# Pp
where A(x) is a scalar that represents the sum of normalised attention weights on the prefixes and So, (2r) is the attention module in a Transformer. we set, fe. (z) = softmax( W P Pv. then we obtain a function composition (1- fo= (a) + f(. (a) that incorporates a weighted addition. For function and parameter composition, in contrast, the sum is unweighted.
Overall, despite their conceptual differences, most modular approaches are similar in their functional form and can be expressed as function composition. In practice, the way different methods are realised, however, leads to different trade-offs, which we illustrate in Table 3 Recent empirical studies (Mahabadi et al. 2021a He et al., 2022a Liu et al. 2022b) provide further evidence for the strengths and weaknesses of different methods. For instance, prompt tuning (Vu et al.
2022a) underperforms other methods due to limited capacity
while intrinsic dimensionality (Aghajanyan al. 2021 uses very small number of parameters but leads large memory footprint and poor performance. Fine-tuning only biases (Ben Zaken et al. 2022 has a small memory footprint but achieves lower performance. Finally, function composition methods such adapter layers (Pfeifler el al. 2021a) and compacter layers (Mahabadi el al. 2021a), achieve the best performance, but add more parameters. (IA)3 (Liu et al. 2022bb mitigates this by composing lightweight linear diagonal weight. Modular deep lcarning architecturos, however, have many other differences beyond their choice of compulation function. In the following sections, we discuss the routing, aggregation, and training settings for the modules presented so far.
Computation functions may consist of any neural module. Modules may modify the original paramcters, be concatenated to the input, or composed with the original function.
Parameter composition methods utilise sparsity or low-rank constraints. They are very earfieter-efficient and efficient at inference time and show strong performance.
Input composition methods concatenate a function's input with a parameter vector via prompting, continuous prompts, and retrieval augmentation. They are extremely parameter- efficient but inefficient during training and inference and require large models.
Function composition methods augment model with arbitrary functions via parameter sharing, adapters, or rescaling. They require more parancters but often achieve the best performance.
Rather than learning module parameters directly, hypernetworks can be used to generate module parameters, which enables sharing of information and conditioning on auxiliary information.
4 Routing Function
In the previous section, we described how to compose a sub-function fa with shared weights 0 with a single module function with weights Φ. However, in a modular neural architecture, multippe modules are available from an inventory M = før..... So A decision-making process required to determine which modules are active, conditioned on the model input or auxiliary metadata. This process is implemented through a routing function r(-) that assigns score 0; to each module from the inventory M. These scores determine
16

---

Published in Transactions on Machine Learning Research (11/2023)
which subset of modules is active, i.. contributes to the computation. We provide overview of different routing methods in Figure 4
When metadata such as expert knowledge about sub-tasks (or skills) involved in a task is available, r(-) can be designed as a fized function, that is, each routing decision can be made a priori (Figure Aap. For instance, when using a language model to generate dialogue in Swahili, a task module for dialogue generation and a language module for Swahili can be selected. When no such prior information is available for instance when modelling heterogencous unlabelled mata-routing of given example needs to be learned (Figures 4ch. In
this case, the routing function can be conditioned on the current example To
Unfortunately, learned routing is crucially under-constrained, as multiple possible ways of decomposing tasks into sub-tasks are reasonable (Jacobs et al. 1991a). In addition, it presents a series of unique challenges (see $ [1.2.1]. In an empirical study on synthetic data, Mittal et al. (2022) found that learned routing is sub-optimal compared to fixed routing, as it tends to under-utilisc modules and to specialise them a lessor degree. This behaviour exacerbated as the number of lasks in the data grows. In real-world applications, Muqeeth et al. [2022) report similar results; however, Ponti et al. (2022) find that learned routing may surpass expert module selection even in settings where tasks are procedurally constructed to require certain skills, such as instruction following in simulated environments.
Learning-to-route can roughly be split into hard routing and soft routing [Rosenbaum et al. 2019). Hard routing methods learn a binary selection of modules, similarly to the fixed routing scheme, where only a subset of modules is selected for each reccisionmmaknng step (Figure 4b). Inference for hard routing systems typically builds on score function estimators (Williams) 1988 1992] stochastic re-parameterrsation (Jang et al.
2017). On the other hand, soft routing methods learn probability distribution over modules (Figure 4c Jacobs et al. 1991a). While soft selection is more easily amenable to end-to-end learning via gradient descent, hard selection may lead to a sparse architectural design, owing to the fact that inactive modules are not part of the forward and backward computation graph. This reduces time complexity while augmenting the model capacity (Bengio et al. 2013).
While not the central focus of this paper, routing algorithms have recently garnered significant attention, due to their efficioncy implications. There exists an intricate interplay between routing techniques and sparsity within modular models: When modular architecture exhibits sparsity, signifying that only select few modules are active during inference, notable reduction in computational complexity during inference can be achieved (Fedus et al. 2022 II Finally, Shen et al. (203aa); Jang et al. (2023) showcase how sparse models, when combined with instruction tuning techniques, can yield substantial gains in performance and efficiency over dense models, potentially reshaping the landscape of large language model design.
4.1 Fixed Routing
Making the routing decision a priori i.e. when we utilise metadata (e.g. task identity t) to make the discrete routing decisions before training is referred to as fixed routing (Figure Ma). Here the routing function r(-) simplifies to selection of a subset of modules K C M for the examples with certain metadata:
{
ifi K
r().)
(3)
otherwise
This function defines binary matrix A E (0, ( where the number of rows corresponds to possible tasks and the number of columns corresponds to the size of the module inventory.
One simple example of fixed routing in multi- learning is when all parameters, except the final classification layer, are shared among all tasks (Ruder) 2017). Independently from the task identity, the examples are passed through the same network until after the penultimate layer. The penultimate layer's representations
Alternative non-parametric routing strategics include random routing Zuo al. 2022 Wang al. 2022 or routing based hash functions (Roller et al. 2021
1n $4.2.3 Teeeen-Level Routing we provide a hrief overview over recent works that focus on the efficiency aspect. However, it important note, that the focus of this paper is centred around modularity and not efficiency.
17

---

Published in Transactions on Machine Learning Research (11/2023)
(a) Fixed Routing
(b) Learned Routing (Hard)
(c) Learned Routing (Soft)
Figure 4: Different routing methods. (a) Fixed Routing: Examples passed to modules based
pre-defined logic, known a priori. (b) Hard Learned Routing: Learned hard selection modules. (c) Soft
Learned Routing: Soft selection and weighting of modules.
are then routed to their respective final classification layer according to the task identity. This boils down to setting |K] = 1, with the additional constraint that tasks cannot share modules, which results in the allocation matrix being an identily matrix, A=
While not immediately apparent, methods that adapt pre-trained models towards individual tasks (Rebuffi ct al.
2018 Houlsby al. 2019 Bapna & Firat 2019 Li Liang 2021 Liu ct al. 2022b Hu al. 2017
Ansell et al., 2022; Ben Zaken al. 2022, inter alia) discussed in S deterministically route representations through the newly introduced module fØ. Given that the pre-trained weights are frozen and modules trained different tasks can be added or removed, the components become modular even if they are developed asynchronously and independently of each other (Pfeiffer et al. 2021a). In a sense, community-based hubs of pre-trained adapters such as AdapterHub (Pfeiffer et al. 2020a can be considered as ever-evolving multi-task models, the development of whose components has been distributed throughout the community Moreover, since newly introduced weights are encapsulated between frozen (shared) weights, adapted representations of intermediate layers are implicitly aligned as they are passed as input to the same frozen components.
2022
Hampshire & Waibel 1992) were possibly among the first to train independent experts for series of sub-tasks known priori. In this case, the (fixed-size) subset of experts K associated with each task t is assumed as given, resulting in the rows of A being k-way vectors. In cross-lingual transfer, any problem can decomposed into a lask and language variety. Fixed routing can select separate language and lask components, and facilitate generalisation new, unobserved combinations of tasks and languages at inference time (Peeiffer et al. 2020b Ponti et 2021 Üstün et al., (202)). In this case, |K] 2. Similarly, in reinforcement learning, Heess et al. (2016) and Devin et al. (2017) design modular policy that is composed of robot-specific module and task-specific module, which are instantiated separate neural networks. Composing these modules enables generalisation to unseen robot task combinations.
Beyond task identity, routing can be performed based on other metadata such as language, domain, or modality information. Pfeiffer et al. (2022b) add adapters for each language to a multilingual language model during pro-training on unlabelled text. Fan al. (2021) route deterministically for multilingual machine translation according to the language family: a consequence, all languages in family share the expert. In similar vein, Gururangan et al. (2022) add domain-specific adapters to language models, deterministically routing based on the text source domain. This concept was further extended by [i et al. (2022b), who proposed the branch train merge method: copies of the same model are trained on different. domains and then averaged. Finally, modality can also inform fixed routing, such as in vision-and-language models (Peiffeer et al. 2022a). This allows for adapting the encoders of different modality streams.
12 Alternativvly, combining entire modcls stored in model repositorics via distillation Khanuja ct al. 2021 averaging Matena Raffel 2021) can also help avoid negative interference and subject limitations such as those discussed later m
2022 however, this usually less efficient.
18

---

Published in Transactions on Machine Learning Research (11/2023)
4.2 Learned Routing
When the routing function r(.) is not known in advance, it can be implemented as learnable neural network with parameters p. In input, it receives the example representation æ or metadata such as the task t. In output, it returns routing scores a. Usually, pр is a linear projection or a Multi-Layer Perceptron. While the former represents a less expressive family of functions, the latter may collapse into ignoring the imput features. Note that learning the routing function also implies that the specialisation of each module is unknown. Thus, modules are not trained on different sets of examples; rather, they are all trained jointly with the routing function.
4.2.1 Challenges of Learned Routing
Learned routing introduces a number of challenges, including training instability, module collapse (Kirsch et al. 2018), and overfilling. These were first systematically described by Rosenbaum et al. (2019), and we follow a similar taxonomy. In general, they identify two root causes for all these challenges: first, the need to balance between exploration and exploitation (Sutton, 1986). More specifically, routing must find the optimal trade-off between allocating information to the most suitable modules versus under-explored modules. Second, routing must share modules across examples or tasks in such a way to reap the benefits of positive transfer while avoiding negative interference. We elaborate the individual challenges below.
Training Instability emerges especially in the early phases of training; at this point, modules are randomly initialised and have no clear functional specialisation. Thus, the router cannot make any principled decision in selecting modules. On the other hand, modules do not start specialising until they are consistently routed to different subsets of tasks or examples.
Curriculum learning can mitigate this challenge to some extent Chang et al. 2019 as simpler tasks require simpler sets of skills. However, this assumes that information about task complexity is available and that the data can be ordered accordingly. As an alternative, the router parameters can be trained with a different learning rate than the module parameters, either lower (Rosenbaum et al. 2018) or higher (Ponti et al. 2022). These create two different dynamics: either the necessary skills for a task are determined after specialisation, or the relationship among lasks figured out first and modules are updated accordingly.
Module Collapse describes scenarios where only a small number of modules (in the extreme case, one) from the available inventory are selected. This leaves the remaining modules untrained and negatively impacts their overall diversity. Ofien, this results from excessively favouring exploilation over exploration, which leads to sub-optimal results. To amend this, Ahn et al. (2019) use e-greedy routing for initial exploration of all modules and afterwards switch to learned routing. Other strategies to avoid module collapse include auxiliary losses for load balancing (Shazeer et al. 2017: Fedus et al. 2021 and intrinsic rewards that encourage
diversity in module selection (Cases et al. 2019). The choice of information that conditions the router also plays an important role: metadata, e.g. text genre (Cases et al. 2019) or task identity (Kudugunta al.) 2021), make routing more robust than individual examples. The diversity of training tasks also facilitates diversity in routing selections (Chang et al. 2019 Caccia et al. 2022) Dua et al. (2022) warms up the sampling temperature over training, in order to over-sample domains with fewer examples in unbalanced distributions.
Overfitting to noise is a risk faced by deep modular networks due to their ability to model subsets of examples independently (Rosenbaum et al. (201)). For instance, routing at the token level was shown lead to performance drops in out-of-domain generalisation for MoEs (Artetxe et al., (022). For similar reason, gains in pre-training do not always translate into gains in fine-tuning for MoEs (Fedus et al. 2021 p.
Increased robustness can be achieved by routing conditioned on metadata if available (Chang et al. 2019
Cases et al. 2019 Kudugunta et al. 2021). In addition, strategies that favour the combinatorial behaviour of modules yield superior generalisation (Chang et al. 2019: Ponti et al. 2022
4.2.2 Hard Learned Routing
A model may learn how to select modules through hard routing. This implies that the choice of whether module is active or excluded from the computation graph is binary. Discrete decisions are not amenable
19

---

Published in Transactions on Machine Learning Research (11/2023)
to be learned through vanilla gradient descent: since small perturbations of parameters do not affect the selection of modules, the gradient of the loss with respect to the routing parameters is wero. Thus, various methods, including reinforcement learning, evolutionary algorithms, and stochastic re-parameterisation, have been proposed for inference. These are discussed separately below.
On the other hand, hard routing is more efficient than soft routing in terms of time and space complexity. Tn addition, binary selection implies that parameter updates are localised a subset of modules. This reflects the intuition that the shifts in distribution of the variables in environment et al. 2018; Goyal et al, 2021). Since the inactive module parameters are not affected, they remain invariant with respect to the distribution shift. On top of this, this type of routing may result in variable-size sets of active modules. This allocates model capacity according to task complexity, which follows the principle of conditional computation (Bengio et al. 2015). In fact, it is fair to assume that the skills required for complex tasks are a superset of those of simpler tasks. For instance, dialogue modelling requires (among others) intent. detection, slot filling, and conditional response generation.
similarly local (Parascandolo
Reinforcement Learning In Routing Networks (Rosenbaum al. 2018 9, Modular Networks (Kirsch et al. 2018), and the Compositional Recursive Learner (CRL; Chang et al. through reinforcement learning. Specifically, Routing Networks rely on multi-agent RL (MARI), Modular Networks rely on the score function estimator (REINFORCE), whereas the CRL relies on Proximal Policy Optimisation (PPO). Commonly, this family of methods alternate between a score function estimator for the routing parameters P and SGD for module parameters (.,...,,,1))) For vanilla score function estimator, where routing is conditioned on the input example and m E M, the update takes the form:
2019 a router work is trained
1 n
Vp Ply x, O, Φ1 ................................ M p) ~
[pls o,, 0, 0,, Φm) V log p(m xx)]
(4)
Under this lens, routing becomes a policy x(m| | x). If applied layer-wise, each hidden representation at a given layer 1 12 t I constitutes state h, H. The routing policy determines the action, i.c. the selection of module index m. In particular, this assumes that the inventory M is shared across layers Tn turn, applying the transformation of the corresponding module on the input is equivalent to a transition function a : H. + H. which returns the next layer's 's hidden state h,+1- The loss function at the top layer corresponds to a (delayed) negative reward, i.e. C()) = R [ Crucially, in this setting the transition functions are non-stationary, as the module parameters are amenable to change. Because modules are applied sequentially based on the policy, the number of steps of computation in the model can vary when a special halting action is available.
Evolutionary Algorithms Alternatively, routing can be learned via a genetic algorithm. In PathNet (ernanndo et al. 2017), the loss function indicates the fitness of configuration of active modules K C M. For each task, two configurations are selected at random and trained until stopping criterion is met. The one incurring the lower loss on a validation set overwrites the other. This copy, in turn, receives a random mutation, and then the procedure is repeated. In μNel (Gesmundo & Dean 2022a b), mutations involve cloning, insertion, and removal of layers. The fitness criteria include not only performance but also parameter efficiency. This approach has been extended to a multi-task setting where multiple agents update different modules asynchronously (Gesmundo) 2022). However, as is common for evolutionary algorithmss, this search is brute-force and thus highly inefficient.
Stochastic Re- Re-parametrisation Hard routing can also be performed via continuous relaxation of the discrete latent variable a determining the module allocation. Several stochastic re- pe-aaammttrristionns such as Gumbel-Softmax (Jang et al., (017)) or the Concrete distribution (Maddison et al.) (017) have been proposed for this purpose. Compared to the score function estimator, stochastic re-parameterisation are biased but have lower variance. Moreover, they are differentiable, which makes a hard router trainable in an end-to-end
This cucourages module ro-usage different layers. 14 Intrinsie rewards can be added, for instance favouring diversity in the module selection across time steps Rosenbaum et al.
2018
20

---

Published in Transactions on Machine Learning Research (11/2023)
fashion. For instance, AdaShare (Sun et al. 2020b) uses Gumbel-Sigmoid to learn a binary vector for each task that indicates whether model layer should be included in the forward pass skipped entirely. This may be interpreted as choosing between a parameterised module and an identity function at each layer.
Stochastic re- parameterisation also allows for selecting module subsets of varying sizes for each layer. In Neural Interpreters (Rahaman et al,, 2201], ; this is based on threshold. Each module associated with a "signature vector?. The dot product between this vector and the output of an unnormalised routing function ('type inference") conditioned on token determines score. If this surpasses certain threshold, then the module is allowed to access the given token. As an alternative, variable-size module routing be achieved by learning soft clustering (a.k.a. soft partition) of modules (Ponti et al. 2022 Caccia et al. 2022). Thus, each entry Ozj, which represents the routing of the j-th module to the i-th task, is constructed as follows:
(a..)
1/r
Oi sigmoid log
~~ Uniform(0..)
(1 sig
where daj represents the unnormalised routing score. This latent. variable also admits priors such the Indian Buffet Process (Griffiths & Ghahramani 2011 to encourage both diversification and sharing of module subsets across tasks (Ponti et al. 2022). Caccia et al. (2022) extend this framework to multi-head routing, where different modules can be allocated to contiguous subsets of dimensions of the layer's 's input and output. While this just requires as many copies of a as the number of subsets of dimension, provides higher expressivity to the routing function.
Top-k Selection Finally, hard selection can rely on top-k selection from (possibly unnormalised) scores modules. In the case of Independent Causal Mechanisms (Parascandolo et al. 2018), is given by discriminator that scores the outputs of a generator, and k 1. In the case of Recurrent Independent Mechanisms (Goyal et al.] 2021), the scores are derived from attention between modules and the inpul, and k >> 1. These methods are grounded on the assumption that the competition among modules to be activated facilitates their specialisation (sco 8.3 for more details).
4.2.3 Soft Learned Routing
Mixture of Experts To sidestep discrete selections of modules, several works propose soft routing methods, where all modules are selected and aggregated according to a weighted combination, i.e. mixture of experts (MoE; Jacobs et al. 1991b Jordan & Jacobs 1994] 15 Here, the router learns a probability distribution over the available modules, i.e. p(M) = r.()). Hence, routing and aggregation take place as:
f(x)= = Σ f(x))) ΦJEM
(6)
In contrast to the discrete selection of hard routing methods, this setup is easily trained end-to-end via gradient descent. A number of works (Eigen et al. 2014 Meyerson & Miikkulainen 2018 Wortsman et al.) 2020. inter alia) train a continuous weighting (i.e. a mixture) of all modules; however, this limits the degree of modularity as parameter updates are not local; instead, they always affect all modules. Additionally, activating all modules for each example significantly increascs the computational cost for cach forward and backward pass through the network. To circumvent this, Shazeer et al. (2017) and Lepikhin et al. (2021) only route to the top-k of |M] modules, where 1 < k [M]. The output representations of the k active modules are averaged according to the respective routing weights, whose sum is re-normalised to 1. Thus, top-k MoEs stand between hard routing, as only a subset of modules is active, and soft routing, as their average weighted by the routing scores. In practice, a layer performs the following computation:
r()))
f(())
Σ
f(x: O,;
6FE (r Σ r())
18tn the following sections we the term "expert" and "module" interchangeahly to reflect common practice the body of research on MoFs.
21

---

Published in Transactions on Machine Learning Research (11/2023)
Fedus al. (2021) and Clark et al. (2022) demonstrate that even top-1 routing can achieve competitive
results for language modelling.
Teken-Level Routing MoEs have recently undergone a revival as part of the efforts to scale Transformers. In particular, MoE Transformers route to a subset of Feed-Forward Network (FFN) modules per layer instead of a single FFN. The focus of these works is on computationally efficient training of very large models. This is achieved by splitting the input tokens across different (hardware) accelerators. The MoE routing algorithm is therefore required to (ideally) uniformly distribute the lokens of all the examples in an input batch across all accelerators, i.e. to load balance computation across "experts". The dominating routing strategy is for each token to choose the top-k eaperts (Shazeer al. 2017 Lepikhin et al. 2021 Fedus et al. 2021 Clark et al.
2022
Yang et al., 2021 Dua et al.
2022 Hazimeh et al.) 2021 Rajbhanndri et al. 2022; Riquelme et al.
2021
Du et al.
2022 Zoph et al.
2022).
Alternative approaches let each expert choose the top-k lokens
You
et al.
2022 Zhou et al. 2022b or globally determine the best routing path (Lewis et al. 2021 16
However, since routing conditioned on the token level, and the load balancing restriction limits the system from routing an entire example to single module, the system potentially has to relearn similar concepts in multiple modules. Hence, load balancing hinders the router from selecting the single best module for longer (e.g., repetitive) sequences. This is investigated further by Lewis et al. (202)), who find that sparse models route syntactically and semantically similar words (in contrast to sentences or phrases) to the same modules. This sheds light on the limited expressiveness of modules which are Iearned on the token-level. Since scaling is the main focus of these works, their goals are orthogonal to modular approaches centred on parameter efficiency, consser-interferente trade-offs, and combinatorial generalisation.
Levell Routing Nevertheless, one could imagine obtaining the best of both worlds by hybridising sparse MoE Transformers models with deterministic or learned routing strategies from [ 4.1 and [ 4.2.2 Instead of routing each individual token separately, all tokens of single example can be routed to the same experts. Kudugunta et al. (2021) experiment with two versions of example-level routing for machine translation: In sentence--evel routing, they average pool over the token embeddings, and condition the router on the resulting representation. In task-level routing, a task embedding is trained, based on which the router learns the distribution over modules. In a similar vein, Gupta et al. (2022) and Xi et al. (2022) implement task-level routing across modular experts to improve the amount of knowledge sharing during multi-task learning in NLP and computer vision, respectively.
Since task identity (or other metadata) is not always given, especially in continual learning, it can be inferred learning'. For instance, the current task can be identified based on the lowest predictive uncertainty or an auxiliary task classifier (von Oswald et al. 2020 In these cases, routing can depend on the predicted task identity.
through an auxiliary model. Van de Ven & Tolias (2019) refer to this scenario as
Mitigating Module Collapse To address the challenge of module collapse, which was previously discussed in $1.2.1 several strategies have been introduced to enhance the effectiveness of models in utilizing the available experts' capacity. One such approach, presented by [Shen et al. (2023b), introduces novel loss function centered on Mutual Information. This loss aims to maxinize the mutual information between the input and the target module, effectively mitigating module collapse issues. Another innovative solution, put forward by Chi et al. (2022), involves the modification of the routing algorithm. This modification incorporates techniques like dimension reduction, L2 normalization, and adjustment of gating temperature, all designed to address the challenges associated with module collapse. [uugcerver et al. (2023) employ a fully differentiable soft assignment mechanism by applying weighted combinations of representations module, allowing for enhanced model capacity without significantly increasing inference costs. Mugeethh et al. (2023) tackle module collapse by employing weighted average-based merging approach of the module's parameters.
each
16 For more details load balancing medh we refer to Fedus et al.
2022 Chapter 4.
22

---

Published in Transactions on Machine Learning Research (11/2023)
(a) Layer-wise routing
(b) Global routing (naive)
(c) Global routing
Figure 5: Different routing levels. (a) Layer-wise Routing: The indices are chosen based on the input to the current layer. (b) Naive Global Routing: The same indices of modules are chosen for all the layers of the model. (c) Global Routing: The configuration (possibly different for each layer) is chosen globally.
4.2.4 Hypernetworks
In addition to hard and soft routing, hypernetworks (Ha et al. 2017), as introduced in g 3.4 can be considered a third kind of routing, with unnormalised routing scores. More formally, the parameters 0, E Rd for a task can be generated by a línear function tag The task embedding a E IR M] can be interpreted as the output of a lask-level routing function with unnormalised scores over |M] modules. In turn, the generator @ E [dx]M|] can be considered matrix of module parameters stacked column-wei wise, where each module has d parameters. Thus, the generated parameters O, is linear combination of the columns of the linear generator. This is also reminiscent of tensor factorisation models where paramcters are factorised into shared tensors and lask-specific tensors (Yang & Hospedales) 2017), which in hypernetworks correspond to the generator and the task embedding, respectively. However, hypernetworks learn both sets of parameters jointly rather than obtaining them from a factorisation of task-spocific networks a posteriori.
4.3 Level of Routing
Another aspect of designing a routing function is its level of granularity. Routing can select modules globally for the entire network, make different allocation decisions per layer, or even hierarchically select sub-routers. This last method is also referred to as 'dispatched routing' by Rosenbaum et al. (0088). A naive version of global routing (Figure 5b) assumes that a single routing configuration shared across layers. Allowing for different decisions per layer (Figure 5a) is more challenging as the space of potential architectures grows exponcntially |MV' where I is the number of layers sub-functions of the network. In fact, to compute the posterior over parameters, one would need to marginalise over every possible configuration of A = [a1...., au). Kirsch et al. (2018) resort to Expectation Maximisation to make it tractable. Instead, per-layer routing (Figure 5cb assumes conditional independence among decisions, thus facilitating scaling. Crucially, routing scores are sometimes employed not only to select a subset of modules but also to aggregate their outputs. This second purpose addressed in more depth in S
Most methods assume that routing decisions occur in a sequence, whose length is bounded or unbounded. This is the case where the output of every layer is fed into the next. However, routing may also involve defining both the selection of modules and their order of composition (i.e., the model architecture). For instance, in Neural Module Networks (NMNs; Andreas et al. 2016b; 2017), the routing function consists of a parser that takes in a query and produces a dependency tree. This is post-processed and transformed into a tree graph where nodes are modules and directed edges control the flow the information, i.e. route the output(s) of a subset of modules as input another module. In Modular Meta Learning, Alet al. (2018) alternate between sampling compositional graphs using simulated annealing (Kirkpatrick et al. 1983) and performing step of gradient descent on the network parameters for set of meta-training tasks.
23

---

Published in Transactions on Machine Learning Research (11/2023)
The Routing Function is critical component in modular neural networks, responsible for determining how information flows through modules.
Routing can be categorized into Fixed Routing, Learned Routing, and Hypernetworks.
Fixed Routing uses predetermined rules direct information flow.
Learned Routing employs neural networks to dynamically allocate modules based on input data or task information.
Challenges Learned Routing include Training Instability, Module Collapse, and Overfitting, which require specialized strategies for mitigation.
Hard Learned Routing involves discrete module selection, often requiring reinforce- ment learning or stochastic re-parameterization for training.
Soft Learned Routing often use weighted combinations of modules, where predomi- nantly top-k routing strategies employed for computational efficiency.
Hypernetworks offer a flexible approach by generating task-specific parameters with unnormalized routing scores.
Routing decisions can occur different levels of gramularity, including global, per-layer, and hierarchical routing.
5 Aggregation Function
While in the previous section on routing we have covered the topic of how to select different modules during training, we will now focus on how we can aggregale these functions in order to combine the respective information. It is important emphasise that, for the majority of current approaches, routing and aggregation are inseparable; that is, the selection and aggregation of modules are performed simultancously On the other hand, the strategies for aggregaling functions in this section are reminiscent of the laxonomy previously discussed for compulation functions (see $3 while in the latter we looked into the composition of shared components with modules, in this section we provide insights into the composition of multiple modules. This is often required when modules are recombined for zero-shot transfer or task-level generalisation (for more details on these applications, see S
In particular, for a subset of active modules S M, the aggregation of modular components can (similarly) be realised the parameter level (x (x) (z), input level fz(x) fe, ([@1. h æ)), as well as function level (x) = Jor o... S (r)). In addition, we cover oulpul representation level aggregation f((x) fo, (x) @ fe, (x) @ ... @ f (z). Crucially, this differs from parameter aggregation if f non-lincar. We discuss these different strategies in the following sections.
5.1 Parameter Aggregation
Mode Connectivity A natural strategy to aggregate information from multiple modules is interpolating their weights. However, given that neural architectures differ, and that hidden representations might not necessarily be equivalent (e.g- under invariance to invertible linear transformations) even if the model architectures are the same (Kornblihh et al. 2019), naively aggregating module weights may have catastrophic consequences. However, recent work on linear mode connectivity (Frankle et al. 2020) suggests that under certain conditions, it is in fact possible to interpolate between multiple models, which has positive ramifications for modular aggregation mothods. To undersland these conditions, we first. provide a brief introduction to the constraints under which parameter aggregation is permissible.
The phenomenon where the minima found by two networks connected by path of non-increasing error, has been the subject of research for many years (Freeman & Bruna 2017 Draxler et. al.] 2018 Garipoy et al. 2018 Nagarajan & Kolter 2019). However, most works demonstrate that mode paths are in fact
Combining modules has the potential significantly improve inference speed.
24

---

Published in Transactions on Machine Learning Research (11/2023)
not linear. While Nagarajan & Kolter (2019) find linear paths between networks, their experimental setup requires initialising models with the same set of weights. Frankle et al. (2020) and Neyshabur et al. (2020) demonstrate that this lineor mode connectivity phenomenon is closely linked to the Lottery Ticket Hypothesis Frankle& & Carbin 2019 which suggests that only a small subset of randomly initialised weights are the main drivers for the final performance of model the so-called winning lickels (see S 3.1p. When interpolating between models trained on different tasks but initialised with the same set of weights, the models tend to stay in the same loss basin, indicated by the lack of sudden increase in loss when interpolating the weights. Consequently, it appears that the flatness of the basin of the loss landscape translates to better generalisation capabilities of model. Gueta et al. (023) find that fine-tuned models reside in distinct regions in weight space, and models within the same region exhibit high performance. On the other hand, Ainsworth et al. (2022) argue that the success of such interpolation is strongly connected to the inherent bias of the optimiser being used, and not the neural network architecture itself.
Weight Interpolation Building on the findings of interpolating the weights of models, Ansell et. al. (002 propose Lottery Ticket Sparse Fine-Tuning (LT-SFT), described in ( 3.1 In particular, they identify language, and task-specific sub-networks Φi and Φ.. These can be aggregated by simply adding them to the base model parameters, i.e. 0' 0o + Φ1 + Φb. Instead of identifying task adaptations on subsets of model parameters, IIharco al. 2022) propose to edit entire models with further arithmetic operations. For example, for toxic language generation and language modelling tasks, by performing the arithmetic negation operation 0' = 0 + (Øgeneral - ptoxic), their new model fa)( generates less taxic text. This idea was influenced by
the word analogy task (i.e., 'word arithmetics") (Mikolov et al. 2013 18
Rather than interpolating sparse adapters, Asai et al. (2022) aggregate parameters soft prompts learned via prefix tuning / 3.2p. In order to generalise to new tasks, (frozen) modules from past tasks and a learnable module created for the new task are interpolated according to the weights of attention mechanism between the modules and the input.
Model Merging Mode connectivity has enabled the fusion of entire models without extensive retraining, yielding performance improvements across a range of applications (Choshen et al. 2022 Gupta et al. 2020
Yadav et al. 2023 Jin et al.] 2023). These developments have made frameworks, such as Git-Theta (Kandpal et al. 2023 which facilitate collaborative model development through version control, reasonable. Soft Merging of Experts with Adaptive Routing (SMEAR) (Mugeeth et al.) 2023 introduccs gradicnt-based training for sparsely activated models, offering specialization benefits.
5.2 Representation Aggregation
Closely related to parameter aggregation, representation aggregation consists of interpolating the outputs of individual modules. Crucially, both operations are equivalent if the functions are lincar: (a,0, + ay a, a, a, + a, D,. However, this does not hold true for non- non-linear functions, e.g. if the module is an adapter layer (Houlsy et al. 2019 or a feed-forward component of a Transformer layer (Fedus et al. 2021
Weighted Representation Averaging At the i-th sub-function of the model, where multiple modules & e Mj cxist, the reprosentations are passod through the (active) modules, outputting [Ki] (latent) representations hi...., h,,,)- One way of performing aggregation is to learn the weights a to interpolate over the hidden representations:
K.
f((x) = Σ aayk
(8)
with aj being a module-specific scalar weighting.
This aggregation is equivalent to Equation ( when interpreting each weight ay E [0, 1] as the output of soft router, i.e. ag = r( r(h,). Consequently, all soft-learned routing approaches (e.g. MoE) that do not perform top-1 routing (sce 4.2.3 also determine how to aggregate the representations of different modules.
18 Dee (eeee))ngg)- vea('Man') + wee( me('"oman") z veel mee('Quenn)) with vec(-) denoting word embeddings of the respective words.
25

---

Published in Transactions on Machine Learning Research (11/2023)
As an extension to the traditional MoE cegregatiionrrottiinn function, Ma et al. (2018) propose to learn aggregation function per task t in multi-task setup. Gururangan et al. (2022) pre-train modular components for different textual domains E D. When utilissing the pre-trained modules on unseen data, they weight the output representations ha of the respective domain modules Φu according to the posterior distribution over the input examples, iee. a = ((D x):
f1(x)) = Epld p(d x) (x)
(10)=Ep(ee dED
(9)
This posterior is inferred through the Bayes rule. This does not require any auxiliary model, and only relies on the original d-conditioned language model. In fixed routing, module representations are often averaged without weighting (Zhang et al. 2022a] Chronopoulou et al.) 2022a). Similarly, in hard routing methods, the representations of all active modules are averaged, such as Polytropon (Ponti al.) 2022). or summed,
in PathNet (Fernando et al. 2017 19
One disadvantage of simply learning galing parameters is that the weights do not depend on the hidden representations. Thus, they do not take into account their information content. This issue is tackled by attention-based aggregation functions.
Attention-Based Representation Aggregation Instead of inferring the weighting before a module has performed its transformation on the latent representation, the aggregation decision can take place afterwards. This allows for identifying whether not the information added by the respective module is ancillary to the target task. In AdapterFusion, Pfeiffer et al. (2021a) propose attention mechanism (Bahdanau et al. 2015) between the stacked hidden representations H, produced by the modules and their input
f(()) = Att H.V,)
(10)
Rdxh are the projections that produce the queries, keys, and values, and is the input representation to each of the modules (i.e., the output representation of the previous layer). H, R R|Mi is matrix consisting of row-wise stacking of the output representations h1,..., hyjr, of each module. In other words, the input of each module is interpreted as the query and the output of each module is interpreted as the value and key. The attention mechanism thus learns to attend over the module representations and weigh them according to their relevance for the current task.
where
Instead of aggregating module outputs into a single representation, Recurrent Independent Mechanisms (Goyal et al. 2021) concatenate the outputs of the top-k active modules. However, in between the application of recurrent computation functions, they exploit an attention mechanism over hidden representations to enable sparse communication among modules.
One major disadvantage of both weighted and atlention-hased representation averaging, is that-when used in combination with soft routing they require a full forward pass through all modules, even if they contribute only marginally to the final aggregation. Thus, they incur significant increases time and space complexity. While this can be mitigated by pruming (i.e., dropping) some modules during inference (Rücklé et. al.) 2021 latency still remains issue for scalability. Thus, top-k hard routing offers more efficient solution for both weighted averaging (Shazeer et al. 2017 Lepikhin et al. 2021 and attention-based aggregation (Goyal et al.
2021
5.3
Input Aggregation
Input aggregation lends itself naturally to adapters such prompts prefix luning (Brwnn el al. 2020 Lester
et al.
Li & Liang, 2021 see 3.2). In prompting, we have a set of instructions or few-shot examples .I...., Φ|K|- Given that the na paturr of prompting is to prepend the prompts to the input, aggregating the respective modules boils down to concatenating all prompts. That is, providing the model with multiple
2021
Note that the latter strategy leads high variance the norms of hidden representations if the can select variable-size subsets modules.
26

---

Published in Transactions on Machine Learning Research (11/2023)
instructions, with multiple examples (i.e. few-shot in-context learning) is a version of module imput aggregation [x(()) = So, (Iø1..... & (),))))) 2)). This concept also extends to prefix-tuning, where we can simply concalenate all prefixes at every layer: f()))) = So,
(!!!! az)
In the context of prompting, [Schick et al. [202]] leverage input aggregation by concatenating multiple textual descriptions of undesired behaviours of language model to generate toxic text for model debiasing. In the context of prompt tuning, Vu et al. (2022b) learn separate task and language soft prompts that are
recombined for zcro-shot cross-lingual transfer in summarization, Nayak ct al. 2022 compose soft prompts of attributes and objects in visual tasks to generalise to new classes. Soft prompts can also be aggregated with methods different from concatenation such as attention- parameter interpolation (Asai et al. 2022
Furthermore, input aggregation methods have found significant utility in retrieval augmented generation Lewis et al. 2020a), a technique where retrieval models are employed to retrieve external knowledge for addressing www.lleeeiintensiiie NLP tasks 20 In RAG methods, retrieved documents are appended to the input, essentially aggregating external information with the model's input. These methods facilitate knowledge injection and editing (Verga et all, 2021; Cheng et al., 2023). allowing models to access and incorporate information ion from external sources, which can be crucial for tasks demanding domsin-specific knowledge or real-time data updates. This aligns with the broader theme of knowledge enhancement within modular neural architectures, extending their capabilitics to handle complex and dynamic information needs 21
Hypernetworks Similarly to soft prompts, hypernctworks may aggregate information from different embeddings by combining them in the input to the parameter generator. For instance, in (Ponti et al., 2021) task and language embeddingg are concatenated in the imput when training a multilingual multi-task architecture where the encoder is fully shared and the hypernetwork generates the classifier head. By recombining embeddings appropriately, this method allows for inferring the parameters of unseen task- language combinations. Combinations of embeddings have been used to generate adapters in multilingual (Üstün et al. 2020) and multi-task settings (Mahabadi et al. 2021b Pilault et al. 2021
5.4 Function Aggregation
Finally, aggregation can be achieved on the function level; M(x)) = for o faz( Different aggregation methods infer either a sequence or a (tree) structure that determines the order of the aggregation.
Sequential Aggregation By performing a forward pass through multiple modules, where the input to the next module is the output of the previous one, the respective hidden representations are sequentially transformed: f(xx)= = for (... (f)))(())) (x)))).
This form of information aggregation is often chosen in conjunction with fixed routing, as discussed in given that the routing order is determined by the role of each module (e.g. language and task adapters). Pfeiffer et al. (2020b) 2021b) propose two-stage setup where language-specific components are disentangled from task-specific components, in order to perform zero-shot cross-lingual transfer. First, language (adapter) modules fa,, and før, are trained on monolingual unlabelled data for the source language s and the target language t, respoctively. Then, in the second stage, the language component før, is inserted but frozen, and a new (adapter) module is added for a lask fø, and trained on annotated data for the source language: (for, (z)). Since this effectively disentangles language from task information, this also enables zero-shot inference the target language t without annotated data. In particular, fe., is substituted with fei, thereby hierarchically aggregating the information from the respective modular components: So. (Sor, (a)). Similarly, Stickland et al. (2021) perform function composition of a language module fo. and a domain module for. for multilingual multi-domain machine translation. For more examples, see $ 7.1
Hicrarchical Aggregation Alternatively, when global routers jointly determine the selection of modules and the model architecture, the order of function composition follows the structure of tree. For instance,
Notabl,,, RAG is also discussed in $ 3.2 due its dual capability of both input composition and aggregation, for instance when multiple documents are used in the retrieval process.
For further applications of RAG see 7.1.4
27

---

Published in Transactions on Machine Learning Research (11/2023)
Neural Module Networks [Andreas et al. 2016b) leverage semantic parse to infer graphical structure for module aggregation. While all leaf nodes find objects by identifying regions of an image through attention, intermediate nodes either transform or combine these representations (depending on the arity of the node). The root then predicts the label by describing or mcasuring the attended objects.
gggegation functions play crucial role in combining information from multiple modules in modular neural architectures.
Parameter aggregation strategies interpolate weights of multiple modules and are influenced by concepts like linear mode connectivity.
Weighted representation averaging and attention-based aggregation are methods for combining the outpuls of modules, with attention mechanisms allowing for dynamic weighting based on relevance.
Input aggregation methods, such as prompting and prefix tuning, involve concatenating instructions or examples to the input, enabling modular control over tasks and domains.
Hypernetworks can also perform input aggregation by combining different embeddings.
Function aggregation occurs on the function level and can be sequential or hierarchical, with
different methods determining the order of aggregation.
Sequential aggregation is often used with fixed routing, while hierarchical aggregation is employed when global routers jointly determine module selection and model architecture.
6 Training Setting
Finally, explore the training settings for modular architectures. We can identify three main strategies in the literature: 1) all modules are jointly trained for multi-task learning: 2) modules are introduced at different stages during continual learning; and 3) in transfer learning, modules are added post-hoc after pre-training, often as a way to fine-tune the model in a parameter-efficien fashion. Importantly, these strategies are not necessarily mutually exclusive and can be realised combination.
6.1 Joint Multitask Learning
In joint multi-task learning, there are two main settings. Firstly, task-specific parameterised components can be integrated into shared neural network architectures means to mitigate catastrophic forgetting negative interference (MeCloskey & Cohen 1989 French 1999 and way scale the model size efficiently (Kudugunaa et al.] 2021 In these scenarios, modules are often optimised on individual tasks via fixed routing and specialise accordingly (Hampshire & Waibel 1992 Rajendran et al. 2017 inter alia; see 44.1 for more details). As an alternative, the architecture can be fully modular, sharing only the parameters
for learned routing (Jacobs et al. 1991bla Rosenbaum et al. 2018 Kirsch et al. 2018 Chang et al., 2019 inter alia; see 4.2.3 for more details).
Joint training can also be performed before post-hoc training: shared base model can be pre-trained on way to provide useful initialisation for modular parancters (Vu et al.) 2022c). Dua et al. (0022) convert a dense language model pre-trained on text data into an MoE by decomposing the learned feed-forward layers. Pfeiffer et al. (202bb add language-specffic layers during multilingual pre-training of a language model. This prepares the model to be extended more languages post-hoc; when new languages become available, a new (randomly initialised) learnable layer can be added to the inventory of modules, whereas the shared parameters remain untouched.
multiple tasks as warm-up before creating task-specific sparse subnetworks (Sun et al. 2020a.)
28

---

Published in Transactions on Machine Learning Research (11/2023)
6.2 Continual Learning
In a similar vein to countering negative interference in multi-task learning, continual learning that is, continuously integrating new data into the model oflen aims at mitigating catastrophic forgetting (i.e., the knowledge learned at early stages training should not get overwritten by updates to the model later on).
Similar the multi-task learning approaches discussed in $ 6.1 new layers can be continuously introducod within the network which only updated on the new data, keeping the others untouched. In methods like Progressive Networks (Rusu et al., 2016), PathNet (Fernando et al.) 2017), and PackNet (Mallya & Lazebnik 2018) when the model is trained on a new task, the parameters of the previous tasks are frozen; however, for new lasks, new modules may be learned, which connect to the existing set of modules. Often, the decision of inserting new modules at a given stage is made dynamically based on outlier detection (Ostapenko et al.) 2021). Progressive Networks (Rusu et al. 2016), on the other hand, scale the model capacity linearly with the number of tasks. Aljundi al. (2017) train separate experts for every task and route new examples based the distribution of the reconstruction errors of Lask-specific auto-encoders.
Instead of adding new parameters to the model, other works in the continual learning landscape identify subnetworks for different tasks. For instance, some works identify subnetworks of the model, which have not been used by previous tasks. Consequently, updating these parts of the model will have little effect on the previously Iearned knowledge (Javaloy & Valera 2022). Similarly, 'supermasks' ($3.1) Wortsman et al., 2020), which learn a binary mask over a randomly initialised model, enable the extension to a potentially number tasks during continual learning. Supermasks of previous tasks can be also linearly combined way generalise to new tasks.
6.3 Parameter-efficien Transfer Learning
Recently, transfer learning has become the dominant, strategy for state-of-the-art results on most tasks. Auxiliary self-supervised objectives are utilised pre-train models on a large amount of data. Subsequently, the model's weights are fine-tuned the target (asks (Howard & Ruder 2018 Devlin al. 2019). Updating small set of parameters of these large models has been demonstrated to perform equally well full model fine-tuning, Icading to the emergence of parameter-cffiicnt fine-tuning strategics.
Most methods discussed in $ that are applied to large pre-trained models can be considered as post-hoc
adaptation. Modularity can be achieved through parameter composition ($ 3.11 using sparse subnetworks
[Mchta] 2019 Chen et al. 2020 Donahue al. 2014
Cai al. 2020 Ben Zaken et al. 2022 Guo al.)
2021), or low-rank adaplers (Li el al. 2018
Hu et. al.
2022), input composition 3.2 by augmenting
the function's input (Brown al.] 2020:
Li Liang 2021 ). and function composition ($ 3.3) through
adapter layers (Rebuffi et al. 2017
Houlsby et al. 2019) and rescaling (Liu et al. 2022b). Additionally,
hypernetworks can be used to generate the parameters of any of the above-mentioned types of modules ($ 3.4
Essentially, all of these methods are tightly connected as they share the same functional form (S 3.5
There are three main training strategies: (1) Joint Multitask Learning, (2) Continual Learning, and (3) Pfffieter-officien Transfer Learning.
In Joint Multitask Learning. task-specific components are integrated into shared neural architectures, allowing modules to specialize via fixed or learned routing.
Continual Learning methods aim to integrate new data while mitigating catastrophic forgetting, with options to introduce new modules dynamically or identify subnetworks for different tasks.
Parameter-efffiient Transfer Learning involves pre-training models on large datasets and fine- tuning on target tasks. Modular strategics can be applied post-hoc through various composition methods, including parameter composition, input composition, and function composition.
These training strategics not mutually exclusive and can be combined to achieve specific goals in modular neural archilectures.
29

---

Published in Transactions on Machine Learning Research (11/2023)
7 Applications in Transfer Learning
Most applications of modular deep learning revolve around transfer learning. In particular, the two main purposes are: 1) parameler-sficieel line-tuning ($ 7.1,, which achieves superior efficiency, prevents negative interference, and enables zero-shot transfer; and 2) zero /fetooowshhtt generalisation to new tasks ($ 7.2p. In what follows, we provide a quick overview of transfer learning applications of modular deep learning. For the in-depth discussions and illustrations of the key concepls, we will first focus on applications in NLP, and then draw direct analogies with other deep learning areas such as speech processing, computer vision, and multi-modal (representation) learning. In $ we will explore additional purposes of modular deep learning, including hierarchical reinforcement learning, programme simulation, and causal inference.
7.1 -arameter-Efficient Fine-tuning
Regardless of the application area, one of the principal uses of modules has been to boost parameter efficiency and decrease model storage requirements of fine-tuning, eschewing so-called full model fine-tuning which requires storing separate copy of the full model per task (Howard & Ruder 2018 Devlin al., 2019). see $6.3 In the simplest formulation, all task-specific updates are pushed to the parameters of the lightweight modules, while the parameters of the large base model are kept frozen throughout task fine-tuning The modules then store task-specific knowledge that can be composed with the 'gencral-purpose knowledge of the base model to adapt. it to the task al hand. In NLP, this led to a number of research papers that introduced diverse modular architectures, as surveyed in IS 3 and $ 6 A typical evaluation protocol is finettning a type of module on the popular GLUE and SuperGLUE benchmarks (Wang et al. 2019), comparing against full model fine-tuning or alternative modular architectures. The results usually corroborate either of two main goals: (i) improving performance with the same parameter budget versus (ii) maintaining performance with smaller parameter budget (Mahabadi et al. 2021a Zhou et al., 2023). In addition, modular adaptation has further benefits: first, it prevents negative interference between tasks (Bapna & Firat 2019). Second, it allows for combining adapters to enable zero-shot transfer (Pfeiffer et al.) 2020b). In light of the enormous size of state-of-the-art large language models (LLMs), parameten-offiient fine-tuning has emerged as the main way to update the pretrained models
(Hu et al. 2022
7.1.1 Machine Translation
In the seminal work of |Bapna& & Firat 2019 : belingual (i.c., language-pair) adapters (sec $3.3 were used to adapt massively multilingual NMT model (spanning 103 languages) to particular source larget translation direction. One benefit of such bilingual adapters is their ability to 'skew' the multilingual model to the language pair at hand without losing the benefits of massively multilingual training for low-resource languages. Another positive effect of bilingual adapters concerns recovering the MT performance also for high-rcsource languages. High-resource lauguages might lypically suffer from performance deterioration due to the particular interference phenomenon known as the 'curse of multilinguality" (Conneau et al. 2020)] Wann et al., 2020): when (too) many languages compete for the fixed parameter budget of the model, the model's expressivencss and representation power deteriorates for all languages. The use of modules extends the parameter budget to recover the detrimental effects of multilingual inference through dedicated (i.e,, modular) bilingual adaptation. Their work also demonstrates the superior performance of a multilingual model specialised towards a particular language pair over merely training a bilingual NMT model for the same pair from scratch.
However, fine-tuning bilingual adapters (or more generally, modules) for each translation direction assumes parallel data for all language pairs and requires n(n-1) 1) modules to cater for all possible language pairs (one dedicated module in the encoder and another module in the decoder). Therefore, follow-up work (Philip) et al. 2020 Üstün et al. 2 aimed to learn monolingual (i.e., language-specific) adapters. Again assuming standard encoder-decoder architectures for MT such as mBART (Liu et al. 2020), this design requires only 2n modules in Lotal. Besides improving parameter efficiency, this also by bypasses the critical dependency on parallel data for all language pairs and allows for learning from monolingual data. Crucially, this design also enables translation to or from languages without parallel data, in a fully unsupervised way, and even fo/from languages unseen by the base pre-trained encoder-decoder model. Put simply, when translating from language l, to 4,, only the encoder adapters for 4, plus the decoder adapters for 4 are activated: the model is
30

---

Published in Transactions on Machine Learning Research (11/2023)
able to translate from l. to 4 without seeing single parallel lg to l sentence. This application in the field of NMT exemplifies the power of modular design: available components, which were previously learned locally and asynchronously, can be recombined in novel ways to generalise systematically to unseen applications (i.c., in this particular case, to unseen translation directions). This is one of the main goals of modular deep learning ($ T.
The separation into dedicated language-specific modules mitigates interference and catastrophic forgetting; however, it also hinders any positive transfer between modules of similar languages. The positive transfer can be achieved through the use of hypernetworks (see $3.4): Baziotis et al. (2022) learn to generate monolingual language-specific adapters for NMT. In fact, sharing the paramcter generator takes advantage of language similarities (Platanios et al. 2018). As discussed in more detail later in 77.1.2 similar ideas of combining the modular design with hypernetworks have also been applied earlier and beyond NMT, e.g., for task fine-tuning with adapters in monolingual multi-task setups (Mahabadi et al. 2021b) and for cross-lingual transfer in
single-task (Ansell ct al. 2021) as well as in multi-task sctups (Ponti et al. 2021 Üstün et al. 2022).
The curse of multilinguullity and catastrophic interference in multilingual MT models have also been tackled through sparse sub-networks (see (..1). Lin et al. (2021) extract sparse sub-networks for specific language pairs from a trained multilingual MT model via pruning. Subnetworks are then trained separately in order to specialise towards the particular translation direction. In fact, there exist dedicated small sub-networks (which can be obtained via standard masking) that store language pair-specific knowledge within the large network, where such knowledge should not interfere with other language pair-specific sub-networks (Dua) et al. 2022). The same high-level idea has also been applied to domain adoptation of bilingual MT systems: e-g-: Liang et al. (2021) | show that it is possible to learn domain-specific sub-networks when fine-tuning the MT system on new domains, where a single large network (i.e., the full neural MT system) comprises multiple disjoint domain-specifie sub-networks specialised to particular domains.
Another approach that leverages modularity for an increased language-specific capacity in MT is mixture- of-experts. Each expert is typically dedicated to a particular language or translation direction (Kudugunta) et al. 2021 Costa-jussà et al. 2022). To maintain feasible decoding time, the procedure works as follows: (i) during training, mix the inputs from different translation directions in the same batch, in order learn the routing network and encourage positive transfer among related tasks; (fi) at inference time, different translation directions are decoded separately, and only the corresponding subset for elevant experts is loaded.
7.1.2 Cross Lingual Transfer
NMT focuses on translation single task and modularity was exploited mainly to carve language-specific and/or domain-specific modules that can support multilingual and multi-domain systems, respectively. In
more general cross-lingual transfer sctups, the aim is to transfer large models (Devlin et al., 2019 Conncau et al. 2020) fine-tuned for any lask (e.g., sequence labelling lasks such as NER, text classilical ion lasks such as NLI, sentiment analysis or intent detection for dialogue systems) on one or more source languages (where such task annotations exist) to one or more target languages (Hu et al., |220 Ruder et al.) 2021). Ideally, the transfer should be achieved without fine-tuning the full model [Hu et al.) 2020), which results in catastrophic forgetting and negative interference, or requires the creation of separate model copies for each task.
The idea of training language modules thus largely follows what already outlined for MT in 77.1. with the addition of another set of dedicated modules that aim to capture task-related knowledge: task modules. Such language modules and task modules can then be combined to 1) favour zero-shot cross-lingual transfer for particular source-target directions (Pfciffer et al., 2020b; Ansell et al. 2021 2022 Parović ct al.) 2022); 2) provide extra capacity to low-resource languages under-representee (or even not covered) in the large multilingual models such as mBERT or XLM-R (Pfeiffer et al.) 2021b 2022b Ponti et al. 2020 Faisal &
Anastasopoulos )022), independently from task knowledge; and 3) enable handling unseen language-task or even language domain task configurations (Ponti et al.) 2021 Stickland et al. 2021
As an example of zero-shot cross- lingull transfer, the original MAD-X framework (Peifferr et al. 2020b
Figure lab relics on bottleneck adapters to implement language and task modules: In particular: 1) Language modules are inserted into each layer of the original neural model and are fine-tuned on (unsupervised) data of the particular language (e.g., via Masked Language Modelling) while the weights of the original model are kept.
31

---

Published in Transactions on Machine Learning Research (11/2023)
fixed. 2) After obtaining language modules, task modules are stacked on top of the source language module(s) and are fine-tuned relying on the task objective and task-annotated data in the source language(s), while both the original model and language modules are kept fixed. 3) At inference, source language module(s) are replaced with the desired target language module while retaining the task module: this enables zcro-shot task inference the target language.
Recent work has introduced spectrum of variations and enhance- ments to this core idea. For instance, inspired by the bilingual
'translation direction' adapters for NMT systems ( 7.7.1.1 Parović
ct al.
2022 Iearn bilingual adapters instead of single language
adapters to boost transfer for & particular language pair. Faisal
& Anastasopoulos (2022) and Chronopoulou et al. (202bb) learn
language family adapters reduce data sparsity for low-resource languages and capitalise on language similarity and cross-language sharing. Stticlland et. al. (2021) decouple language and domain knowledge into dedicated modules (see also 77.1.3 later). Fur- ther, Ansell et al. (2022) implement dedicated modules as sparse sub-networks, the so-called language and task masks, which can be composed with the base model via parameter composition. Following the analogy between language-specific and bilingual adapters, instead of learning separate language and task sub- networks, Foroutan al. (2022) learn dedicated task-lan sub-networks, demonstrating the variance in the extracted sub- networks across different task language combinations. The use of such language sub-networks as language modules, even without dedicated task modules, improves cross-lingual transfer for depen- dency parsing when used within a meta-learning setup (Choenni ett al. p. . Litschko et al. (2022) compare sparse sub-networks and bottleneck adapters for transferring ranking functions for infor- mation retrieval tasks across languages and find them both superior to full model fine-tuning.
lang
layor
Figure 6: Hyper-X (Üstün et al. 2022 an example application of contextual module generalion where a hypernetwork takes the concatenation of task, language and layer embeddings as input and gen- erates flat parameter vector. This is further reshaped into an adapter module within each Transformer layer. Learn- ing independent layer embeddings and sharing a single hypernetwork across all layers (Ansell et al. 2021 (i) enables information sharing across layers, and (ii) reduces trainable parameters of the hyper-netword by a factor corresponding to the number of layers.
Finally, a body of work again focuses on 'contextually generating" the modules via hypernetworks, aiming to increase efficiency and benefit. from connections between different languages and tasks. A representative example is the Hyper-X framework (Üstūn et al., 2022) provided in Figure where the module parameter generation is conditioned on the (disentangled) task and language, and addi- tionally on the index of the Transformer layer where the generated module is inserted. Each task and language are paramcterised via separate embeddings, which enables adaptation to any task- language combination, where these embeddings are low-dimensional vectors which are lcarned together with the parameters of the hypernetwork (sce Figure G again). The framework thus leverages supervision and positive transfer from both multiple tasks and languages. Hyper-X can be seen a more general variant. of series of precursors backed by the idea of contextual generation: Ponti et al. 2021) condition the hypernetwork on both task and language embeddings but generates only the model's classifier head. Other methods generate modules but condition the hypernetwork only on tasks in a monolingual setup (Mahabadi et al. 2021b) or only on languages in a cross-lingual transfer setup (Ustän
et al. 2020 Ansell et al. 20211.
32

---

Published in Transactions on Machine Learning Research (11/2023)
7.1.3 Domain Adaptation
As already hinted at in $7.1.1 and 7...2 dealing with different domains adds another Lier to the modular
design: domain-specific knowlodge might be captured within dedicated domain modules 27 This can again be accomplished through similar modular architectures with language and task adapters. For instance, it is possible to inject domain-specife knowledge into (bottleneck) adapters (Zhang et al., 2021 Chronopoulou 2022a) or to extract sparse domain-specific or task-specific sub-networks (Thompson et al. 2018; Ke let al. 2021b) for multi-domain and multi-task learning. Mixturr--ofexxprts also enable multi-domain joint
iet al.
learning as well as domain adaptation (Guo et al. 2018 Zhong et al.) 2022). Similar strategies have also been used in multi-domain and cross-domain speech processing and computer vision applications (see 771.5
and 7.1. later).
In domain adaptation, it is common to combine both shared parameters and domain modules that are learned jointly (Bousmalis et al.) 2016). Beyond this standard setting, many approaches employ additional regularisation terms. The most common are 1) a domain-advorsarial loss on the shared paramotors in order to encourage them to be invaiininvariant (Ganin et al.) 2016 Chen & Cardie 2018); 2) an orthogonality constraint on the domain modules to ensure that they capture different information (Baktashmotlaghh et al.]
2013 Kim et al.] 2017);; and 3) similarity constraints that bring representations of similar modules close together (Bousmalis et al. 2016
7.1.4 Knowledge Injection
Naturally, dedicated modules can also be assigned to inject and store external knowledge (e.g., from manually curated external knowledge bases), which can then interact with language, domain, or task knowledge. This idca has boen explored with diverse external knowledge sources. For instance, Lauscher et al. (2020) aimod at complementing the distributional knowledge of large language models with conceptual and commonsense knowledge from Concept.Net. (Speer et al.] 2017). The external knowledge was captured within dedicated bottleneck adapters: they were fine-tuned via language modelling on synthetically created sentences from random walks over the ConceptNet graph structures. Majewska et al. (2021) stored verb-related knowledge from VerbNet (Schuler 2005), a human-created verb classification repository, into bottleneck adapters, and demonstrated its usefulness in a range of tasks that require understanding of verb semantics. Along similar lines, Wang et al. (2021a) offered a gemeralisation of these approaches where different knowledge sources (e-g., Wikipedia, WikiData) are mapped to different dedicated adapters, which can be aggregated according to the task at hand. The same idca has been explored by Lu et al. (202T) the biomedical domain, where the main knowledge sources were the UMLS Metathesaurus graph (Bodenreider) (004) and biomedical Wikipedia articles. Lu et al.] (2021) also introduce another component, the so-called knowledge controller, which can be seen as standard attention-based function aggregator from $5.4 As an example of another relevant application, Lauscher et al. (2021) learned bottleneck adapters without manually curated external data, with the locus on model debiasing: the debiasing adapters were fine-tuned via standard language modelling on a counterfactually augmented corpus.
Finally, the idca of modular knowledge injection is also directly linked the retrieval-angmented language models in text-only settings (Lewis et al. 2020b) as well in multi-modal settings (Yasunaga et al., 2023 where the standalone retrieval module, detached from the 'main' language model, is responsible to fetch knowledge from some external memory or a knowledge base, and that knowledge is then used to condition the language model. In this design, the retrieval step and capability is made explicit and decoupled from the language model generation capability: as such, one can work directly on a retrieval module without the need to change the other components of the entire model (Yu et al. 2023). The ability of standard language models to use external lools is also sparked by the modular design: different external tools specialised for performing particular functions (e.g., conducting Web search, performing mathematical operations) are stored scparate modules accessed from the main model via external API calls. For a comprchensive overview of
such angmenled languoge models, refer the reader to the recent survey Mialon el al. (202)).
22 instance, disentangling domain and language information yields benefits for NMT and cross-lingual transfer applications
Vilar 2018 Cooper Stickland et al. 2021 Pham et. 2021 Saunders 2022).
33

---

Published in Transactions on Machine Learning Research (11/2023)
7.1.5 Speech Processing
The use of modular deep learning for speech processing applications closely matches the ideas already exposed for NLP tasks. The landscape of the possible modular designs exactly the same, where the only crucial differences are (i) the choice of the underlying large model, and (ii) the corresponding objective functions used inject the specialised knowledge into the modules. For instance, the typical choice of the base model for automatic speech recognition (ASR) applications is one from the wav2vec family (Baevski et al.) 2020: Babu et al. 2022), while the ASR-oriented objective function is the standard Connectionist Temporal Classification (CTC) loss (Graves et al.) 2006 The high-level modular structure remains the same, as illustrated in Figure with an example from Thomas et al. 2022 which utilises standard bottleneck adapters.
CTC(-)
While in theory & large variely of possible modular configurations from 3
can be applied to diverse speech processing tasks, the majority of current work in the area has indeed focused on the use of bottleneck (sequentially placed) adapters for ASR in monolingual and multilingual contexts. Before that, the concept of modularity can be traced to the work of [Swietojanski et al. (2016), where the model re-weights hidden units using small amounts of unsupervised data to better adapt to a particular speaker or an environment. More recently, bottleneck adapters have been used lo perform ASR adaptation to alypical and accented speech (Tomanek et al.] 2021). unseen speakers with limited adaptation data [Wang & Van hamme 2022 Eeckt & Van hamme 2022 Chen et al. 2023), new domains and manners of speaking (e.g., children's speech) (Fan & Alwan 2022 Zhu et al. 2022). or to perform further model customisation to specific speakers (Biadsy et al.
31
ft
2022 Sathyendra et al.
2022) and for multilingual learning (Kannan et al.
2019 Hou et al. 2022). A notable exception, not resorting to adapter layers, is the method of (Winata
F Figure 7: The structure of the wav2vec 2.0 model with task- specific bottleneck adapters for parameter-efficienn ASR fine-tuning from Thomas et al. (2022); f(-) denotes volutional encoder followed by 12 standard Transformer encoder blocks. For down- stream ASR a linear classifier, CTC()), applied to the final encoder output.
ct al.) 2020) which aims to learn low-rank modules ($ 3.1. akin to the idea of LoRA (Hu et al. 2022), for end-to-end ASR.
Multi-task (where the term 'task' in this context can e.g. refer to different languages, domains, speakers, or accents) ASR sctups have also witnessed the usage of mixture-offexperts,, closely following the basic ideas already discussed for NMT ($7.1.1) where different languages are assigned their dedicated modules through fixed routing. For instance, in speech processing, MoE's have been applied to multilingual ASR and cross-lingual ASR transfer (Bai et al. propose MoE for ASR with learned routing.
2022 Gaur et. al. 2021 Kumatani et. al. 2021), while You el al. 20022
Beyond ASR, bottleneck adapters have also been used for speech translation [e et al. 2021 Most recently, modular adapter- approaches have been
applicd to text-to-speoch methods (TTS) (Hsich al. 2022 Morioka al. 2202), aiming extend standard large multi-speaker TTS models such FastPitch (Lancucki] 2021) to new speakers without compromising the TTS quality for the seen speakers. From high-level perspective,
direct analogy of this goal to the objectives in the MT literature of extending multilingual MT systems to unseen languagos without.
compromising seen languages (sce 7.1.1 againn.
7.1.6 Computer Vision and Cross-Modal Learning
computer vision, similar to NLP and speech processing (....)) dedicated modules are again used to enable parametor-cfficient fino-tuning across multiple tasks and domains (Rusu al. 2016 Rebuffi al., Berriel al. 2019 He al.] 2022b) among others). The core difference, again, is the choice of the actual neural architecture for the underlying model well as for the modules: egg., residual adapters (Rebuffi
2018
34

---

Published in Transactions on Machine Learning Research (11/2023)
et al.
2017 consisted of simple 1 x convolutions combined with the base ResNet neural model (He et al. 2016 while other work learned task-specific convolutional filters (Newell et al. 2019 Bragman et al. 2019
More recent work aims to exploit modular architectures from NLP (e.g., sequential or parallel adapters,
LoRA, prefix tuning) with pretrained Vision Transformer (ViT) architectures (Dosovitskiy et al. 2021]: c.g.,
He el al. (2022)) run a comparative empirical analysis of various modular architectures for vision lasks, while
Chen et al. (2021) rely on sparse sub-nctworks.
Modular design lends itself naturally to cross-modal and multi-modal applications, where different modalities may be captured by modality-specifff parameters and routing can also be modality con.liio--oodiiiioeeed. For instance, in multilingual Iision-and-language (V&L) settings, it is possible to conduct inference in languages that lack labelled task examples. In fact, language knowledge is again disentangled from the task and modality knowledge, and the knowledge for different input modality streams can be captured in dedicated modules. This idea has been heavily explored in recent work in multi-modal multi-task scenarios, both in monolingual (Sugg et al. 2022) and multilingual contexts (Bugliarello et al., 2022 Pfeiferr et al., 2022a), for tasks such as image captioning (Zhou et al., 2022a
Gao el al.] 2021a), text- io-iaae generation (Maharana et al.] 2022), Sung et al. 2022), visual reasoning (Liu et al. 2021a),, etc.
visual question answering @Liu et al.
2022a
uses frozen pretrained vision and language models, and only trains adapter layers to handle sequences of arbitrarily interleaved visual and textual data. It is trained with a sequence modelling objective on Web-scale data (Li et al.; 2021 and displays impressive zero-shot and few-shot capabilities. [Pfeiffer et al. (2022a) use adapter modules to equip multilingual text-only models with the ability to also process the visual modality, as well as to equip monolingual multi-moda models to deal with input from multiple languages. Papalampidi & Lapata (2022) rely on hierarchical adapters (akin to hicrarchical reprosentation aggregation discussed in [ 5) for the task of summarising long vidcos into textual descriptions. Pan et al. (2022) demonstrate that modular design also helps in inage-to-video transfer tasks: they use adapter modules to equip large image-based model without temporal knowledge with the ability to reason about dynamic video content.
For instance, Flamingo (Alayrac et al.
2022
We note that in this survey, we aim to list some exemplary applications and draw parallels between different yet similar application areas such as NLP, speech processing, and computer vision. While we acknowledge that there exists wealth of other work in these arcas, we have no pretence of exhaustiveness.
7.1.7 Comparison and Design Principles
While a full-fledged comprehensive empirical study of the plethora of modular architect across various application tasks and areas is still lacking, there exist initiatives such as the publicly available AdapterHub platform (Pfeiffer et al. 2020 it provides (re)implementations of reprosentative modular NLP architectures, within a unilied framework tied to HuggingFace Transformers (Wolf et al.) 2020 Among others, AdapterHub includes representatives of each computation method in S 3 LoRA (Hu et al. 2022 (i.c., low-rank parameter composition), prefix tuning of Li & Liang (2021) (input composition) and a number of bottleneck adapter configurations (function composition). The existence of AdapterHub delineates another crucial advantage of modularity: reusability of existing, already fine-tuned modules which can be (re)combined with the large neural models. In short, any practitioner can share or reuse a module specialised for a particular purpose (e.g., capturing specific task or language knowledge) with the community, facilitating community-wide sharing and thus avoiding time- and energy-costly repetitions of the same fine-tuning procedure 23 As discussed in & 4 one can observe initiatives such as AdapterHub as continuously updating communtty-distribuued distributed multi-task models.
The discussion in this section also points to a more general principle: different end-goals even within the same end-application (c.g., NMT, cross-lingual transfer, domain adaptation) require rethinking the actual modular design, and the desired level and nature of modularity. For instance, if the goal in NMT (or cross-lingual transfer) is to boost performance for a particular translation or transfer direction, it might be useful to trade off some modularity for better final performance by replacing language-specific monolingual modules with bilingual modules (Bapna & Firat 2019 Parović et al.) 2022) On the other hand, if the goal is to enable zero-shot or few-shot. Translation or transfer, the design with monolingual modules might be a better
23e (concept of) reusability enabled by the modular design also positively impacts energy consumption Strubell al. 2019). making an important leap towards Green(er) AT (Schwart al. 2020).
35

---

Published in Transactions on Machine Learning Research (11/2023)
choice. In another example, if the focus is on MT or transfer for particular low-resource language, the model designer should enable positive transfer to that language by "opening' the flow of information from a module storing knowledge on high-resource languages similar to the target language if such languages exist (e.g., from Spanish to Galician) [Ustnn et al.) 2021), or by learning modules for familics or groups of similar languages (Chronopoulou et al. 2022b; Faisal & Anastasopoulos 2022). Analogousl, related domains can also be grouped and hicrarchically organised to enable positive transfer for domain adaptation (Chronopoulon
et al. 2022a).
Other practical desiderata may also influence the selection of the actual modular design. If the final task performance is paramount, larger modules might be preferred, c.g., in order to offer enough extra capacity to store the wealth of language-specific information (Ansell et al. 2022). However, if model compactness is paramount, the criterion for choosing a specific design is instead the trade-off between efficiency (in terms of parameters and/or train and test time) and task performance; the optimisation of this trade-off has been the focus of recent rescarch (Rückléé et al.) 2021 Mahabadi et al. 2021a b Sun et al., 2022). In another example, if time efficiency during inference is crucial requirement. (e-g., real-time ASR in dialogue systems, low latency for information search systems) parameter composition methods such sparse subnetworks low-rank composition methods may be preferred over function composition methods as the latter increase the number of computations required during the forward pass, (see Table 3 In yet another example, if storage requirements are a critical constraint, one cannot resort to huge mixture-of-expert models where billions of parameters must be stored (Lepikhin et al. 2021).
7.2 Task Generalisation
The diverse applications of modular deep learning covered so far almost exclusively focus on learning modules associated with (arguably) well-formed and interpretable 'units of knowledge' such as languages, tasks, domains, dialects, accents, and speakers. However, modularity might also be achieved when such units are unknown. This relies on jointly learning arbitrarily sized inventories of so-called latent skills and a learned routing function ($ 4.2). Since such skills are learned end-to-end on a mixture of data from multiple tasks, they are often not straightforwardly interpretable. On the other hand, since arbitrary subsets of skills can be combined and each skill can be updated locally, these modular neural architectures are ideal for systematic
generalisation to new tasks (Zhang al. 2022a. Ponti et al. 2022).
In fact, another fundamental application in transfer learning is achieving zero-shot or few-shot generalisation to new tasks, where test examples are not i.i.d. with respect to training examples. The general experimental sptup involves disjoint. sets of training and test tasks. A model is pre-trained through multi-task learning on training tasks and then adapted to each new test task based on zero or few data points. Common examples of evaluation benchmarks for this setting include CrossFit (Ye et al.] 2021). the TO task suite (Sanh al. 2022), ; or Natural Instructions (Mishra et al. 2022). While a common strategy to tackle this problem is instruction tuning (Sanh et al.) 2022 Wei et al. 2022a where models are fine-tuned prepending the instructions for Kudugunta et al.
each task, modular deep learning has emerged as strong contender (Alet et al., 2018
2021
Ponti et al. 2022
8 Other Purposes of Modularity
In addition to scaling large models (for instance, through MoEs, as discussed in $ 4.2.3) and facilitating transfer learning, which we covered in SS 7, modularity serves multiple additional purposes. In particular, we devote this section to a cursory view of modularity for i) hicrarchical reinforcement learning ($ 8.1 ii)
neural programme simulation b ili) neural causal inference (8 (.3). While most of these applications predate the advent of neural networks, (modular) deep learning expands the scope and potential of these lines of rescarch for a series of reasons. First, it holds promise to induce the relevant latent structures (such as options, programmes, or causal graphs, respectively) in an end-to-end fashion. Second, it marries these traditional problems with the ability to jointly model low-level perception, such vision and language.
36

---

Published in Transactions on Machine Learning Research (11/2023)
Ts: make planks
n
T2: make sticks
Π2
bt get wood
K₁
TI1
b get wood
Ka
IT1
b2: use workbench
TT2
bs: use toolshed
TT3
Figure 8: An example of Hierarchical Reinforcement Learning ($ 8.1) Policy Sketches (Andreas et al., 2017). Two high-level policies corresponding task instructions T are illustrated. Each itcratively selects low-level policies π (options) corresponding to sub-tasks b from a shared inventory. These determine the choice of action given observations. In this case, options are implemented as predicateargummet pairs.
8.1 Hierarchical Reinforcement Learning
The goal of reinforcement learning is to learn policy, which predicts the next action of an agent based past obscrvation(s) from the environment, that maximises the return, i.c. the sum of future discounted rewards. However, many lasks span extremely dilated temporal horizons or provide only highly sparse or delayed rewards. In these cases, becomes helpful to model intermodiate abstractions between high-level goal specifications and low-level actions and observations (Sutton et al. 1999 Precup) 2000). This facilitates the planming abilities of the agent as well as their sample efficiency. In fact, the above-mentionee intermediate abstractions, known options skills, consist of sub-policies that are transferable across tasks.
More formally, each reinforcement learning task is a Markov Decision Process (MDP) consisting of states S and actions A, a transition function S x A x S + [0, 1] and a reward function r S x A + R. We aim to learn a policy π S x A + [0, 1]. We also define a value function as the expected (discounted) return from a given state s as (s) = E, 1
Sy = =s], as well as a Q function from a state s and an action a as Qx a) E. En 7 8p = 8,00 = a). Following (utton et al. (1999) and Precup (2000), each option w is defined as tuple (Iw, Rw, Sw), where I_ S S&i the initiation set, Tw S + [0 1] the option-specific policy, and B_ S J0, 1] is the termination function. For simplicity, many works assume that Vs E S, Vi Q Io: in other words, all options are available at every state. Angmenting task with options transforms it into Semi-MDP, with corresponding functions Vn(w) and @n(s,w).
Learning options involves a series of challenges (Jiang et al. 2019 Firstly, it is not trivial to specialise sub-policies towards distinct behaviours. This shortcoming is common to many modular architectures with learned routing (Mittal al.) 2022 see 84.2 Not only this, the problem of hard learned routing has often been cast in a reinforcement learning framework (S 4.2.2). Secondly, one must define the space where the actions of the high-level policy, which are latent variables, lic. In practice, one could treat them as discrete, unordered set. In this case, module from inventory is chosen for certain amount of time steps. However, alternative methods operate in structured spaces such as language, which is more transferable and scalable due to its combinatorial nature. Thirdly, training multiple options dilates the training time and requires collecting an appropriate amount of experiences for each of them. Fourthly, if trained jointly, options change simultancously with the master policy, which is source of non-stationarity. As consequence, previous experiences for the master policy become invalid the options have been updated in the meantime. Again, this is reminiscent of the challenges of learned routing exposed in 4.2
37

---

Published in Transactions on Machine Learning Research (11/2023)
The simplest solution to circumvent end-to-end joint learning of the master policy and options is to provide seporate supervision to both (Sutton et al., 1999: Dayan & Hinton 1992p. However, this may require extensive annotation, which is often not available. Thus, an alternative method is defining sub-goals, i.e. states an agent should reach as a stepping stone towards the high-level goal Dictterich (2000). Nevertheless, this similarly fails to scale due to the exponentially growing number of combinations of sub-goals some tasks may entail. Morcover, this does not eschew the need to train individual sub-policies for each sub-goal. A partial remody is offered by hindsight learning, where an off-policy correction is introduced (Nachum et al. 2018). Specifically, the original target sub-goal of the current option is substituted with the one maximising the probability of the past sequence of low-level actions. Similarly, the master policy can be trained in hindsight through the currently predicted sequence of high-level sub-goals. Overall, relabelling past experiences significantly improves the model's sample efficiency.
A more radical solution to the challenge of scalability is jointly training both the master policy and options in an end-to-end fashion. To this end, Bacon et al. (2017) put forth a new architecture, the option- critic, that discovers options from data, without supervision for the intermediate abstractions. This architecture is trained based on policy gradient theorems Bacon et al. (0177) derive for options. Moreover, they augment the set of actions A available to cach policy Tw with special end-of-policy action EOP instead of explicitly modelling S2 Intuitively, formulating the execution as call-and-return, a master policy To determines the active option w, whose policy Tu is followed until the ROP action is chosen. At this point, control returns to the master policy to choose the next option, and so on until termination. A downside of this method is that it is unstable and often diverges to degenerate solutions (Jiang et al., 2019). Thus, several inductive biases have been proposed to correct it. A popular method is leveraging óntrinsie rewards: an auxiliary loss diversifics options by maximising the mutual information between cach option and the next state conditioned on the current state (Florensa et al. 2017 Kulkarni et al., 2016).
An orthogonal question revolves around the ideal space for the option variables. In fact, compared to a discrote, unordered inventory of (possibly hard-coded) options, language affords more fiexibility (Andreas) et al. 2017 Jiang et al., 2019) as it solves many of the above-mentioned challenges. In fact, all sub-policies can be implemented through single model conditioned on the linguistic label of the current option. This not only allows options to borrow statistical strength from each other but also makes options reusable in new tasks. Moreover, the nature of language (through its infinite use of finite means) is suitable to capture the extremely complex combination of sub-goals of many reinforcement learning tasks. Note that linguistic options can be interpreted as a generalisation of sub-goals, as every instruction implicitly corresponds to a subset of states (Jiang et al. 2019
In practice, to learn linguistic options, Andreas et al. 2017 assumes that 'sketches' of options are provided for supervision (see Figure B) To induce them, subsequent methods rely instead on synthetic experiences through relabelling (Jiang et al. 2019), restricted vocabularies and syntax such as predicate-argument pairs (Da et al. 2018). Recently, the master policy has been frequently implemented as large language model. Since these are pre-trained on text, they already contain world knowledge that can serve as powerful
inductive bias for grounded learning. For instance, |uanng et al. (2022) use frozen language models to generate options through prompting in a zero-shot fashion.
8.2 Programme Simulation
Another distinct purpose of modular architectures is model programmes, as a means to induce them from data to simulate symbolic algorithms. The simplest (and least expressive) family of programmes are Finite State Automata (FSA). These receive neural implementation in the Compositional Recursive Learner (CRL: Chang ct al. 2019), similarly to Routing Networks (Rosenbaum et al. 2018) and Modular Networks KKrssch et al.] 2018). In these neural architectures, a loose equivalence can be drawn as follows: transformations induced by modules are transition functions (ares in the graph), input and output representations are the states (nodes in the graph), and the input is the starting state. memoryless routing function selects the transition based on the current state. Thus, the programme graph is constructed dynomically. The final states are defined as those reached after the router selects special end-of-computattio action.
38

---

Published in Transactions on Machine Learning Research (11/2023)
Memory usage
Controlter
b Reed and write heads
Merorry
Sad lemporal links
Output
Vite vectine
Write
8
Read
Road
Input
Differentiable Neural Computer (Graves et al, 2016). A recurrent neural controller iteratively receives an input from the environment, writes to reads from memory, and produces an output. Read and write operations are based on attention between generated keys and memory entries. A special mechanism keeps track of memory usage and temporal links between entries.
Figure 9: An example of Programme Simulation ($ 8.2)
On the other hand, programme graph can be constructed globally based on the task description before processing the data. In particular, Neural Module Networks (NMNs; Andreas et al. 2016b rely off-the-shelf semantic parser (and custom rules) to map a query in natural language into tree graph. The nodes of this graph are learnable modules characterised by: 1) their types of input (raw image features and/or attention scores) and output (attention scores or labels); and 2) the particular instances of type, indicated as an argument the form of natural language string. For instance, the module find[cat] takes an image and returns attention scores over the regions that contain cats. Compositionality is achieved by sharing weights across modules with the same type instance. NMNs have been further extended to be amenable to end-to-end training without the aid of an external parser (Hu et al. 2017). In this case, the mapping from queries to programme graphs is learned by imitating expert demonstrations while the module parameters are learned based on the downstream loss of visual question answering.
In addition to the routing function and computation functions, a model can be exlended with an external memory. In fact, these three mirror the fundamental components of computer architecture: elementary operations, logical flow control, and random-access memory that can be read and written to (von Neumann 1945 Graves et al.] 2014). While (appropriately wired) recurrent neural networks have been shown to be Turing-complete (Seegelmann & Sontag, 1995), separating the three functions into distinct components provides an inductive bias to simulate the workflow or computer programme. Neural Turing Machines (NTMs; Graves et al.) 2014) introduced a fully differentiable read write memory matrix that interfaces with the main recurrent network through an attentional mechanism. In particular, this memory can be addressed entries and the current input) and based on location, order store and retrieve temporally ordered information in contiguous entries. NTMs were further extended into the Differentiable Neural Computer (DNCs; Graves al. 2016 Figure which amended some of the limitations of NMIs, such avoiding interference in the memory, freeing up previously written locations, and storing temporally ordered sequences in non-contiguous chunks. Another family of memory-angmented methods include the Neural Programmer Interpreter (NPI; Reed & de Freitas 2016). This model is trained with full supervision from execution traces or through reinforcement learning (Pierrot et al. 2019). particular, a core recurrent network roceives information from a programme module, as well as representations from the environment module. In its output, it produces the index for the next sub-programme and its arguments (as well a special termination symbol).
both based on content (i.e., the match between
39

---

Published in Transactions on Machine Learning Research (11/2023)
Transformed example
Experts
E1
E2
E3
E
Canonical MNIST
7
DISCRIMINATOR
0.0
0.1
0.2
ARGMAX
Causal Independent Mechanisms Parascandooo et al. 2018). A transformed example is routed to an expert which maps it to the original distribution. An adversarial discriminator attempts to distinguish between reconstructed and original examples.
Figure 10:
An example of Causal Inference ($
8.3
Finally, a recent thread of research focused on simulating the behaviour of symbolic algorithms with vanilla (non-modular) neural networks. An example neural algorithmic reasoning (Veličković & Blundell 2021 First, a processor network is trained to emulate the output of symbolic programme (e.g., Dijkstra's 's algorithm for shortest paths) that operates abstract representations (e.g., weighted graphs). Second, encoder and decoder networks can be trained to operate sensory real-world data while matching the input-output types expected by the processor network.
Among the main applications for programme simulations are settings where sub-problems are shared, such as multi-task or curriculum learning. By distilling the most common functionalities into modules, these can reused generalise compositionally to new soquences of sub-tasks. Another application is compositional reasoning, such as (visual) question answering (Andreas et al. 2016bla). In general, external memory is useful for reasoning over complex data structures, such graphs (Graves et al. 2014 2016 Reed & de Freitas 2016). Finally, neural models can emulate symbolic algorithms to extend their capabilities to operate on sensory real-world data.
8.3 Causal Discovery and Inference
Modularity in the design of a model may be assumed to reflect the modularity in the (physical) mechanisms of the world. In fact, a crucial assumption in causal inference (Schōlkopf al. 2012] that such mechanisms underlying data generation are independent, as they do not influence each other, and reusable, as they may play role in multiple distributions. Consequently, one of the mechanisms,, which defines conditional distribution in the model graph, changes possibly because of an intervention the other modules remain invariant. If machine learning model mirrors this modular structure, it is better suited to generalise in a sample-efficient way to new tasks: in fact, local distribution shifts require updating only the corresponding module parameters, which in turn results in faster adaptation (Bengio et. al. 2020 Mittal et al., 2022
The key challenge for this problem how to specialisc each module towards specific mechanism based uniquely on observational data, especially when the number and nature of the mechanisms are unknown. Competition among the modules through top-k routing (see 4.2.2 common feature of many proposed
40

---

Published in Transactions on Machine Learning Research (11/2023)
solutions 24 Parascandolo et al. (2018) show how to invert causal independent mechanisms through modular neural architecture, given data from the original distribution and an unlabelled mixture of their transformations (see Figure 10h. Their model consists of a mixture of experts and an adversarial discriminator, which enforces that the inverted transformation lics in the support of the original distribution. Another architecture relying on module compelition and capable modelling sequential data Recurrent Independent. Mechanisms (RIMs; Goyal et al., 2021). Here, the modules are recurrent networks with separate parameters, each representing a different transition dynamics. However, their states are not entirely independent, active modules are allowed to communicate through attention. This reflects second assumption, namely that the dependencies among variables are highly sparse (Mittal et al. 2022). Attention can also serve to direct the flow of bottom-up and top-down information (Mittal et al. 2020
Another challenge of neural causal discovery is jointly inducing abstract latent variables (such as objects or entities) from low-level perception (e.g., pixels of an image) while simultancously learning the causal graph underlying such variables, which determines how they interact (Ke et al.) 2021a). The lacklustre abilitics of vanilla neural models to understand the compositional properties of symbolic building blocks, i.e. their 'binding problem', arguably explains their current shortfalls in systematic generalisation (Greff et al. 2020). Object-centric learning holds promisc to mitigate these limitations. For instance, it can be facilitated by slot attention, which is a fully differentiable and iterative attention mechanism that interfaces between perceptual representations and slots, a set of unordered placeholder variables (Locatello et al. 2020) (Didolkar et al., 2021) propose Neural Production Systems, where rule templates can be bound to specific entities present in the working memory, in order to update their representations, In particular, rules are MLP modules and the matching with entities (triggering updates) is parameterised by attention.
Crucially, observational data alone is ofter insufficient to learn structural causal models as they may not be identifiable {Pearl 2009). Hence the necessity to augment observation with interventions and counterfactuals. These allow for answering questions about cause- cause-effect relationships rather than mere correlations. In real-world sconarios, however, the nature and number of interventions are unknown Ke et al. (2021a). In this setting, there is no formal guarantee that causal discovery succeeds. Yet, Ke et al. (2019) finds that DAG discovery on interventional data based on continuous optimisation recovers causal graphs reliably. In particular, modular architectures surpass both vanilla models and graph neural networks (Ke et al. 2021a). Recently, Geffner et al. 2022) perform causal inference in a deep non-linear additive noise structural equation model, based on autoregressive flows. Variational inference is used to learn posterior over causal graphs. The learned functions can be further used to estimate conditional average treatment effects based on simulations.
The main purpose of these deep modular methods is causal inference and discovery, which has applications in several branches of medicine and economics (Geffner al. 2022 In addition, these methods are particularly relevant in grounded settings, where the distribution of the observations from the environment changes the agent learns better policies (Goyal et al. 2021). Moreover, causal discovery can be combined with model-based RL methods to learn a self-supervised model of the environment, iee. its variables and their cansal dependencios, from trajectories of observations, actions, and rewards. This allows for simulating the potential outcomes of a policy before execution and thus estimating better value functions, which dramatically improves sample efficiency in agents (Ke et al. 2021a). Another common application this family modular neural architectures is antoo--distribution generalisation: for instance, zero-shot transfer to images different sizes or sequences of different lengths (Goyal ct al. 2021).
24In addition causal inference, this stratogy is also inspired by the global workspace theory Baars postulates specialised modules compete update shared workspace, and the resulting communication bottleneck creates crucial inductive blas in human cognition. Unleess specific assumptions are made about the data generating process, such as linear but non- oon-Gasssian data.
2005 This theory
41

---

Published in Transactions on Machine Learning Research (11/2023)
9 Conclusions
Modularity is defined the functional specialisation of the components of system.
Specialised sub-networks may emerge in vanilla neural modules (from multitask training regularisation), but they are seldom reused and recombined.
Deep modular architectures rest on the separation between computation functions on the one hand and routing and aggregation functions on the other.
Computation functions may consist of any neural module. Modules may modify the original parameters, be concatenated to the input, composed with the original function.
All composition strategies are equivalent to summing the original output with term depending on the new module. In practice, however, they offer different trade-offs between efficiency (in time and space, during training and inference) and performance.
Routing controls the flow information, i... module selection. In fixed routing, it determined a priori based on expert knowlodge. When this is not available, routing parameters are learned.
Learned routing is challenging because of training instability, module collapse, and overfit- ting. Thus, learned routing often underperforms fixed routing.
Routing can be conditioned on (parts of) the input metadata such as task identity. Routing can take place at different levels, such as globally for the whole model or layer-wise.
Soft routing assigns every module a continuous score and performs a weighted combination of their outputs. It amenable being learned via gradient descent but highly inefficient.
Hard routing activates only subset of modules via top-1, top-k, or variable-size selection. is learned via reinforcement learning, evolutionary algorithms, or stochastic re-parameterisation. It corresponds to the principles of conditional computation and information bottleneck in cognition.
Hypernetworks can be interpreted as combining unnormalised routing (task embedding) with modules (generator). They can turn generate parameters of other modules.
routing selects multiple modules, these must. be aggregated via a function.
Module parameters outputs can be interpolated for aggregation, according scores from the routing function, attention mechanism, via simple averaging.
Alternatively, aggregation may involve composing the module functions, either sequentially based on tree graph obtained from global routing.
The applications include parameter-efficient fine-tuning NLP, computer vision, and speech processing. These rely on the same types of modules and fixed routing. In addition to increased efficiency, this prevents negative interference and enables zero-shot transfer.
Modularity also serves the purpose of generalising to new tasks systematically, by recombining modules and locally updating them.
Modular deep learning transcends the confines of private research: it enables community-driven sharing, expanding, reusing, and updating of the modules.
In hierarchical reinforcement learning, modular options serve as abstractions wetw task goals and low-level actions and observations. They facilitate planning in long-horizon and sparsc-reward tasks and increase sample efficiency due transfcrability.
In programme induction, the components of deep models mirror computer architecture: modules are elementary operations and routing is logical flow control. These are often augmented by an external read-write memory. Modules can also simulate symbolic algorithms.
In causal discovery and inference, modules may be taken to correspond to physical mechanisms that are independent and reusable.
Modular deep learning empowers these traditional applications by learning abstractions (options,
programmes, causal graphs) end-to-end from perceptual stimuli.
42

---

Published in Transactions on Machine Learning Research (11/2023)
9.1 Future Work
While recently modularity has attracted increasing attention in research, there remain many interesting open research questions along the axes of modularity introduced in this survey. We provide an overview of some of these directions for future work.
Combination of Different Types of Computation Functions Existing computation functions (see $ 3 are mostly associated with a single catogory: parameter composition, input composition, or function composition. There are a few exceptions such as compacter (Mahabadi et al.) (021a) low-rank adaplers which combine multiple types. In general, techniques from paramcter composition that incorporate sparsity, low-rank structural constraint are agnostic of the form of the module. In practice, this should enable more efficient learning and aggregation.
Learned Module Structure Most modules used in current works share the same architecture, which is reused across different settings. Depending on the skill or knowledge that should be learned, a module may need to be structured differently and might require access to another component or other type of data. In the extreme, a model may require special-purpose architecture to be able to perform a specific capability (Andreas et al., 2016b). As modules are more widely used, they may benefit from being learned in a more flexible manner, perhaps incorporating ideas from neural architecture search (Negrinho et al. 2019 in a module-specific space of architecture primitives.
Standardising Modularity Evaluation Depending on the dimension studied, modular approaches may be evaluated based on variety of factors including downstream performance, memory footprint, number of parameters, latency, and compositional generalisation. In order to make progress modular models in general, evaluation should be standardised. Current evaluation is additionally mainly based on existing datasets that are re-purposed to enable modular evaluation such as by framing them in a zero-shot transfer setting. Future work on modularity evaluation should design forward-looking evaluation benchmarks that are designed to test the capabilities the next generation modular models such as assessing the composition of skills and acquisition of new types of reasoning abilitics.
Nature of Modular Representations While modular representations have been aggregated and com- posed, it remains mostly unclear how the inductive bias of a computation function influences the modular representation that is learned. In addition, it remains unclear how computation functions differ on represen- tation level. Beyond the computation function, it is also unclear how the other dimensions of our taxonomy, i.c., the routing function, the aggregation function, and the training setting influence the nature of the modular representations.
Hierarchical Modularity Current approaches mostly do not different iate between high-level and low-level skills and how they relate to each other. might also be possible to designate particular parts of the model or dedicated modules to capture a set of specialised skills or options, and clearly distinguish between other (sets of) skills. At fino-tuning, even more specialised sub-modules could be learned focused only on the previously designated modules. One example might. be learning fine-grained specialised subnetworks over larger subnetworks of the original model, offering gradual module specialisation.
Learned Routing for Pre-training Fixed routing (see S 4 is the most common strategy to disentangle knowledge into modular parts of the model. However, fixed routing limits the usability of the proposed methods as they cannot be used on data, which lacks the metadata needed for fixed routing: for instance, when training on heterogeneous data, metadata such as domain information often does not exist. While learned routing methods do not require this metadata to perform routing a priori, they suffer from training difficulties (as discussed in $ 4.2). This opens up rescarch directions that enable modular pre-training with learned routing, which would make modular models applicable a broader set of data.
Modular Instruction Tuning The main way in which current LLMs are specialised to particular downstream settings is via instruction tuning Wei et al.. 2022b), i.e., fine-tuning on collection of tasks
43

---

Published in Transactions on Machine Learning Research (11/2023)
described via instructions. These tasks are increasingly defined based set of skills and capabilities that model should learn, which opens the room to developing modular instruction tuning methods that enable the acquisition, updating, and composition of specialised knowledge.
Benchmarking Routing Methods Existing studies mainly evaluate routing methods based on perfor- mance but do not take into account how different routing strategies influence modular representations. In order to make progress on better routing methods, benchmarks and metrics are necessary that compare routing mechanisms from a modularity perspective across different settings.
Structured and Sparse Aggregation Current aggregation methods (see 5) combine the information from multiple modular components by applying arithmetic operations such as addition and subtraction across all paramcters, which likely inchudes paramcters that should not be modified. Structured or sparse aggregation methods could focus on aggregating information within salient subnetworks or parameter groups, which might make aggregation more efficient and improve oustr--distribution generalisation.
Learned Aggregation Methods Most aggregation methods based on arithmetic operations. Depend- ing the nature of the modular information, it may be useful to (non-)linearl) transform the representations. More complex domain-specific aggregation methods can be learned in conjunction with the modular represen- tations to enable better generalisation to new settings.
Merging Modular Models In recent. work, merging models trained with different settings has led to improved performance (Wortsman et al. 2022] inter alia). Rather than requiring separate training runs of a model, a multi-task model can alternatively be trained with modular components that are designed to be merged at a later stage. This polentially allows for an architecture, which can be computationally efficiently trained while covering many modalities.
Extensible Multi-task Models Most approaches multi-task learning have focused on training dense models, with a key limitation being that models cannot casily be extended to new scttings. Focusing on training multi-task models with modular components ensures that the baseline models are much easier to adapt and extend to new settings. Given the trend of pre-training larger and larger models from scratch, modularising parts of such models and developing modular methods that can be shared across different. architectures and model sizes may lead to more sustainable model development.
Acknowledgements
Ivan Vulié is supported by a personal Royal Society University Research Fellowship (no 221137; 2022-).
We are grateful to Colin Raffel for his comments and suggestions, which have greatly improved the manuscript. We thank Andrea Gesmunndo for feedback on draft of this paper. We are thankful to Kyunghyun Cho and Alessandro Sordoni for stimulating discussions. We thank the anonymous reviewers for helpful suggestions and foodback.
References
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the effectiveness of language model fine-tuning In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7319-7328, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/
URL pattps://aclntthen 2rrgeeatoooooooooor222211111 wttp://///21...llloooo.. longg.6gg66 568
Chanho Ahn, Eunwoo Kim, and Songhwai Oh. Deep elastic networks with model selection for multi-task learning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Scoul, Korea (South), October 27 November 2, 2019, pp. https://doi.org//i wttp:///.......1111. 2019.00665
IEEE, 2019. doi: 10.1109/ICCV 2019.0066: URL
44

---

Published in Transactions on Machine Learning Research (11/2023)
Samuel K. Ainsworth, Jonathan Hayase, and Siddhartha S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries. CoRR, abs/2209.04836, 2022. doi: (w.48550//Xiv.2200000 URL https://doi.org//1 48850//55 2209.04836
lean-Baptise Alayrac Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. CoRR, abs/ htbs/2224.11111 2022. doi: ww.45550/arxiv.... 2tt.:i...22.41111 14198. URL https:///oo org/ 10. 48550/arXiv.... 2204 14198
Ferran Alet, Tomás Lozano-Pérez, and Leslie Pack Kaelbling, Modular meta-learning. In 2nd Annual Conference on Robot Learning, CoRI. 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pp. 856 868. PMLR, 2018. URL http:://proceedings..u mmr press/v87/alet1Ban html
Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate; Lifelong learning with a network of experts. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolula, HI, USA, July 21-26, 2017, pp. 7120 7129. TEEE Computer Society, 2017. doi: 10.1109/CV 2017.20 URL https://doi. org/ 10. /09/CVPR 201.C//V.....1...1. 753
Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. Giving BERT a calculator: Finding operations and arguments with reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing Hong Kong, China, November 2019. Association for Computational http://ccannttooloooooor////////
(EMNLP-IJCNLP), pp. Linguistics. doi: 10. 10.18653/vl D/199-1609 URL
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. In Proceedings of the 2016 Conference of the North American Chapter of the Associalion for Compulational Linguistics: Hummn Language Techmologies, pp. 1545 1554, San Diego, California, June 2016a. Association for Computational Linguistics. doi: 10.18653/v1/NIL http://caaanoooooooorgg//lileeee
URL
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 39 39-48.9991 IEEE Computer Sociely, 2016b. doi: 10.1109/CVPR.201..22 URL https: 7/doi.org/10 1109/CVRR. 2016. 12
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning, 1CML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 166-175. PMLR, 2017. URL http:://proceedingg.. mlr ppess//70//andeess html
Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen. MAD-G: Multilingual adapter generation for efficient cross-lingual transfer. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 4762 4781, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 111138/1122222 madings-cmnll...
URL https::/acllanthoooo...oor wwwp:ew...g.....oee htt.r/oooogg..or//2222....nnnnno findings-ennnnn.. 410)
Alan Ausell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. Composable sparse fine-tuning for cross- transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1778 1796, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 11118653/11/2022.acc
125. URL httpos://aclanthooogg.o///////// htw.:/aattnoooooooro222/2222.... long. 125
Sajid Anwar, Kyuyeon Hwang, and Wonyong Sung. Structured pruning of deep convolutional neural networks. ACM Journal on Emerging Technologies in Computing Systems (JETC), 13:3):32:1-32:18, 2017. doi: 10.1145/3005345 http:/ttps//aao..or///./1///////
45

---

Published in Transactions on Machine Learning Research (11/2023)
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Tyer, Ramakanth Pasumuru, Giridharan Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeffrey Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, and Veselin Stoyanov. Efficiont large scale language modeling with mixtures of experts. In Proceedings of the 2022 Conference on Empirioul Methods Natural Language Processing, pp. 11699-11732, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https://aalanttoology org/ wrg./022...o// 2022. emnlp-main. 804
Akari Asai, Mohammadreza Salehi, Matthew Peters, and Hannaneh Hajishirzi. ATTEMPT: Parameter-effiiient multi-task tuning via. attentional mixtures of soft. prompts. In Proceedings of the 2022 Conference on Empir- ical Methods in Natural Language Processing, pp. 6655 6672, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL http::::///..aattooooooor/////// eenie-mmmllppaaa. manpmmii..444 446
Bernard J. Baars. Global workspace theory of consciousness: toward a cognitive neuroscience of human experience. Progress in Brain Research, 150:45-53, 2005. doi:
URL
https://10.. 0ttps////00101160000730000 000116/6500009-11110555000006
Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, Alexei Baevski, Alexis Comneau, and Michael Auli XLS-R: self-supervised cross-lingual speech representation learning at scale. In Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pp. 2278 2282. ISCA, 2022. doi: 10.21437/Inte wwtt////eeeereeee....m Interspeech. 2022-1433
httpsspeecc...22-1ee. 2022- 143..143. URL https: http://dooo.rg//..2222// 21437/
Pierre-Luc Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In Satinder Singh and Shaul Markovitch (eds.), Proceedings of the Thirty- First AAAI Conference on Artificial Intelligence, February 4-9, 2017, Son Francisco, California, USA, pp. 1726-1734. AAAT Press, 2017. URL http:
//aaai. httgg///////ne/.///////////o//ee https///.dd..pp//////A/////// pttp://wA//AAAA//aaa/r////////// vi/8///58v 14858
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 2020, virtual, 2020. URL https://procceddingg wetpisssnurippp.... www.nis..ccppppeerrrrreees (oddeeeeicccfffffaa2222766666670
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL /ttp:///rxiv... org/ abs 1409. 1473.047
Ye Bai, Jie Li, Wenjing Han, Hao Ni, Kaituo Xu, Zhuo Zhang, Cheng Yi, and Xiaorui Wang. Parameter- efficient conformers via sharing sparsely-gated experts for end-to-end speech recognition. In Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pp. 1676 1680. ISCA, 2022. doi:
2022-709. URL https:
/idoi.oog//0..1 21437/Interspeech 209
Mahsa Baktashmotlagh, Mehrtash Tafazzoli Harandi, Brian C. Lovell, and Mathieu Salzmann. Unsupervised domain adaptation by domain invariant projection. In IEEE International Conference on Computer Vision, ICCV 2013, Sydney, Australia, December 1-8, 2013, pp. 769-776. IEEE Computer Socicty, 2013. doi: (..1009/CCC.2201.100 URL httptst/ttot.ooooo
1109/ICCV 2013. 100
Carliss Young Baldwin and Kim B. Clark. Design rules: The power of modularity. MIT Press, 2000. URL https::/66l..rr//1... http:/mmtppsss/2//// 221.336666000000 0001
Dana Ballard. Cortical connections and parallel processing: Structure and function. Behavioral and Brain Sciences, 9(1):67-90) 90, 1986. URL. https:///ssccee..a...oo////////e o05.:///.1....111....4445555111.
46

---

Published in Transactions on Machine Learning Research (11/2023)
Ankur Bapna and Orhan Firat. Simple, scalable adaptation for neural machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language. Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1538 1548, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: (0.18658/v1/D191110 URL https: : (oglanthology..rg/D1g---1
Christos Baziotis, Mikel Artetxe, James Cross, and Shruti Bhosale. Multilingual machine translation with hyper-adapters. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1170-1185, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL htttp:://cclanthol/// org 2022. wttg//o22.eenmmmmaiin... emnlp-main.77
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple paramcter-cfficien finc-tuning for transformer- masked language-models. In Proceedings of the 60th Anruaal Meeling of the Association for Computational Linguistics (Volume 2: Short Popers), pp. 1-9, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi:
URL
acl 1.
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. CoRR, abs/1511.066 1.669977 2015. URL
Yoshua Bengio, Nicholas Léonard, and Aaron C. Courville. Estimating propagating gradients through stochastic neurons for conditional computation. CoRR, abs/1308.3432, 2013. URL http://arxiv.or org/ abs/1308.3433
Yoshua Bengio, Tristan Deleu, Nasim Rahaman, Nan Rosemary Ke, Sébasticn Lachapelle, Olexa Bilaniuk, Anirudh Goyal, and Christopher J. Pal. A mela-transfer objective for learning to disentangle causal mechanisms. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nee net, 2020. URL
Rodrigo Ferreira Berriel, Stéphane Lathuilière, Moin Nabi, Tassilo Klein, Thiago Oliveira-Santos, Nicu Sebe, and Elisa Ricci. Budget-aware adapters for multi-domain learning. In 2019 TEEE/CVT International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 382 391, 2019. doi: 10. 10.1109/ICCV.200 20.220099.00000 URL https:://ooi org/10..001100. 101111111100 1109/ICCV 0.019.00007
Luca Bertinetto, João F. Henriques, Jack Valmadre, Philip H. S. Torr, and Andrea Vedaldi. Learning feed-forward one-shot learners. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 523 531, 2016. URL https://proceedigg.neese neuriss..e wwteurpp..ccppppereeeeeeeee cctppppr//01////aaaeeeeeee/ html
Fadi Biadsy, Youzheng Chen, Xia Zhang, Oleg Rybakov, Andrew Rosenberg, and Pedro J. Moreno. A scalable model specialization framework for training and inference using submodels and its application to speech model personalization. In Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pp. 5125 5129. ISCA, 2022. doi: 10.21437 /01143//Innesspecch.. 2022-10613 URL http::: aedooo.orre. .rg. 10. 21432 2e1.e/1Innnnnnneeeeeeee 2022-10613
Hakan Bilen and Andrea Vedaldi. Universal representations: The missing link between faces, text, planktons, 2017. URL.
and cat breeds. CoRR,
07275
Olivier Bodenreider. The unified medical language system (UMLS): intograting biomodical terminology. Nucleic Acids Research, 32(uupp 1):D267 -2700, 2004. URL httpps://ww..rt www.:/.nciinln... nih, tttpgpnc//atticlss///////
PMC308795/
Grady Booch, Robert A. Maksimchuk, Michael W. Engle, Bobbi Young, Jim Conallen, and Kelli A. Houston. Object-oriented analysis and design with applications, third edition. ACM SIGSOFT Software Engineering Noles, 33(5), 2008. doi: 00.1145/1402211.
http:://bbii.orr/0011114//121113
47

---

Published in Transactions on Machine Learning Research (11/2023)
Konstantinos Bousmalis, George Trigeorgis, Nathan Silberman, Dilip Krishnan, and Dumitru Erhan. Domain separation networks. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Tsabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Infor- mation Processing Systems 2016, December 5-10, 2016, Barcelons, Spain, pp. 343-351, 2016. URL https: :
approceedings.neurrie http://www.....aaaoa////ooaaaaaa
Felix J. S. Bragman, Ryutaro Tamno, Sébastien Ourselin, Daniel C. Alexander, and Manuel Jorge Cardoso. Stochastic filter groups for multi-task cnns: Learning specialist and generalist convolution kernels. Tn 2019 IEEE/CVE International Conference on Computer Vision, 1CCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 1385-1394. IEEE, 2019. doi: J0.100//CCVV.201.00014 2019.00147. URL
https://dol: org. 100. 10t1//00/....22 WWW.IICCVV........ 00147
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Nee- lakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Newral Information Processing Systems 33: Annual Conference on Neu- rul Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https: : neurips.
//proocedings.eurriss.ccc. http:c/capppe//2////aaee/// http://aaaae44400fffc44444774440
42f64 Abwtract.tttrtmmmmmmmaa
Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulić. IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 2370 2392. PMLR, 2022. URL https://proceedings.oo mlr. httpagss.......ss///ee wttpppe/eeee/ppglllllppp wal.aarello22ahatt..
Lucas Caccia, Edoardo Ponti, Lucas Liu, Matheus Pereira, Nicolas Le Roux, and Alessandro Sordoni. 03831
Multi-head adapter routing for data-efficient fine-tuning, 2022. URL
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. In Advances in Newral Information Processing Systems 33: Annual Conference on Newral Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https:
Rich Caruana. Multitask learning. Machine Learning, 28(1):41-75, 1997. doi:
URL httts::/doi.orr/// 100.112/2//11177777677733333
Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, Sandhini Agarwal, Joshua D. Greene, Dan Jurafsky, Christopher Potts, and Lauri Karttunen. Recursive routing networks: Learning to compose modules for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 3631 -6144, Minneapolis, Minnesola, June 2019. Association for Computational Linguisties. doi: 10. 10.18653/v1/N19 1365. URL httpss////cclaatoloooy org/N19-1365
Stephen Casper, Shlomi Hod, Daniel Filan, Cody Wild, Andrew Critch, and Stuart Russell. Graphical clustcrability and local specialization in deep neural networks. In ICLR 2022 Workshop on PAIR Struct, 2022. URL httpps://openrewe www.eeeneeeeepaiiiiieeeeeeeeeeee
Michael Chang, Abhishek Gupta, Sergey Levine, and Thomas L. Griffiths. Automatically composing representation transformations as a means for generalization. Tn 7th International Conference on Leurning Representattons, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin. The lottery ticket hypothesis for pre-trained BERT networks. In Advances in Neural Information Processing Systems 39: Annual Conference on Neural Information Processing Systems 2020, NeurIPS
48

---

Published in Transactions on Machine Learning Research (11/2023)
2020, December 6-12, 2020, virtual, 2020. URL httpps//pooeeeenng...errrssss//p html
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. In Advances in Newral Information Processing Systems 34: Annual Conference Neural Information Processing Systems 2021, NewrIPS 2021, December 6- 14, 2021, virlual, pp. 19974 19988, 2021. URL
html
Ting Chen, Mario Lučié, Neil Houlsby, and Sylvain Gelly. On self modulation for generative adversarial networks. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net,, 2019. URL
Xilun Chen and Claire Cardic. Multinomial adversarial networks for multi-domain text classification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Långuisties: Human Language Technologies, Volume (Long Papers), pp. 1226-1240, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-
URL
https://aclanthologg http:///ocaaattoooooooorr///1111
Zih-Ching Chen, Chin-Lun Fu, Chih- Chih-Ying Liu, Shang- Wen (Danicl) Li, and Hung-yi Lec. Exploring efficient- tuning methods in self-supervieed speech models. In IEEE Spoken Language Technology Workshop, SLT 2022, Doha, Qatar, January 9- 12, 2023, pp. 1120 1120-117..000 IEEE, 2023. doi: 10.11090.1111 (0.1009/SLT54489222233 1002003377 URL https:://oi..on org.io.rg111.0 wtt.g//.....//.T110000 119919155888 2023. 10023274]
Xin Cheng, Yankai Lin, Xiuying Chen, Dongyan Zhao, and Rui Yan. Decouple knowledge from paramters for plug-and-play language modeling. In Findings of the Association for Compulational Linguislies: ACL 2023, pp. 19288-14308, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: URL https///aclanttologg.oorr
10.18653/
sindings-acl.c.
Zewen Chi, Li Dong, Shaohan Huang, Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. On the representation collapse of sparse mixture of experts, 2022.
Rochelle Choenni, Dan Garrette, and Ekaterina Shutova. Data-officient cross-lingual transfer with language- specific subnetworks CoRR, abs/2211 2220022000000,,, 2022. doi: 10.48550/arXiv.222 20111/2110...22.000000 URL htttps://do 48550/0.45555/rxxivvvo
00106
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoar Katz. Fusing finetuned models for better pretraining, 2022.
Alexandra Chronopoulon, Matthew Peters, and Jesse Dodge. Efficient hierarchical domain adaptation for pretrained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1336-1383 1351, Seattle, United States, July 2022a. Association for Computational Linguistics. doi: 10.18653/v1 /220222aacclmmaali https:://alaanthol.oo wrw.llooo.oog/////
URL
http://owwo....oe////aaaaaaaaaaa waw..mmmnaaccmmaiinnnmm 96
Alexandra Chronopoulou, Dario Stojanovski, and Alexander Fraser. Language-family adapters for multilingual neural machine translation. CoRR, abs/ abbs/2209.1ns 22222.2292, 2022b. doi: 10.48550/arXiv 2ww.........9.3....... URL
https://doi..org/11 htt.://////....55//aarrr 18550/arXiv 2209. 15236
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake A. Hechtman, Trevor Cai, Sebastian Borgeaud, George den Driessche, Eliza Rutherford, Tom Hennigan, Matthew J. Johnson, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Mare' Aurelio Ranzato, Jack W. Rae, Erich Elsen, Koray Kavukeuoglu, and Karen Simonyan. Unified scaling laws for routed language models. In International Conference on Machine Learning, ICML 2022 17-23 July 2022 Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 4057-4086. PMLR, 2022. URL html
49

---

Published in Transactions on Machine Learning Research (11/2023)
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross- lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 9440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18653/v / URL https://aclaattoology htwp:/ttoooooooooo//2e2e/// wrg.oogg..2....o acl-mmnnnn 747
Asa Cooper Stickland, Alexandre Berard, and Vassilina Nikoulina. Multilingual domain adaptation for NMT: Decoupling language and domain information with adapters. In Proceedings of the Sirth Conference on Machine Translation, pp. 578 598, Online, November 2021. Association for Computational Linguistics. URL https://aclanthoolog hwwpooolooyo.oog/////e org/2021 wmt-1.64
Marta R. Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, A1 Youngblood, Bapi Akula, Toïc Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language left behind: Scaling human-centered machinc translation. CoRR, abs/2201 044772 www..iiiw...........
2022. doi:
URL https://doi. ortp//....r/00.0 48550/
Michael Crawshaw. Multi-task learning with deep neural networks: A survey. CoRR, abs/2009.09723 2020.
URL fttps:///arxiv.org/bbbssst
09796
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.ned net, 2021. URL
Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural modular control for embodied question answering. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Swilzerland,, 29-31 Oclober 2018, Proceedings, volume 87 of Proceedings of Machine Learning Research, pp. 53 62. PMLR, 2018. URL
wwrpiiininmmmmm....mmmmm.. wwwppprwssssspeeeeep ///////////// 8a. html http:/gss.lllllpe///////////////
Peter Dayan and Geoffrey E. Hinton. Feudal reinforcement learning. In Advances in Neural In- formation Processing Systems 5, [NIPS Conference, Denver, Colorado, USA, November 30 De- cember 3, 1992], pp. 271-278. Morgan Kaufmann, 1992. URL http::/ nips.cc/paper.
714-feudal-reinforr cementt cement-
Harm de Vries, Florian Strub, Jérémie Mary, Hugo Larochelle, Olivier Pietquin, and Aaron C. Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Sys- tems 30: Annual Conference on Newral Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 6594-6604, 2017. URL httppss/pproceedins.eeeet wewpeedddinnnnuuuiiii ips.
(FFbeaaaa222eeeeeaaeedddddddeee Abbtteeecdddd--Atttttt...ee html
Coline Devin, Abhishek Gupla, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. Tn 2017 IEFF International Conference on Robotícs and Automation, ICRA 2017, Singapore, Singapore, May 29- June 3, 2017, pp. 2169 2176. IEEE, URL https:://oi..rrg 100g//10.110 1119//199IRA... 2017 7989250
2017. doi: 10. 10. RAA.0//ICRAA.01.778P22288
Jacob Devlin, Ming- Wei Chang, Kenton Loe, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North Americon Chapter of the Associalion for Computational Länguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171 4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.1863//v1/111
URL httpps://aclantholoogy http://cwwaatttooooooooo//////// org/N199-1113
Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Charles Blundell, Philippe Beaudoin, Nicolas Heess, Michael Mozer, and Yoshua Bengio. Neural production systems. In Advances in Neural Information Processing Systems 34: Aveuual Conference on Neural Information Processing Systems 2021, NewrIPS 2021,
50

---

Published in Transactions on Machine Learning Research (11/2023)
2021. URL http:://ooceddngs.eerrrsssccooo December 6-14, 2021, virtual, pp. Abstrscct..ttt.tttt....cc html.
Thomas G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition.
Journal of Artificial Intelligence Research, 13:227-303, 2000. doi: 10.1613/jairr.33. URL https://doi
org/10. 1613/jar.. 639
Katharina Dobs, Julio Martinez, Alexander JE Kell, and Nancy Kanwisher. Brain-like functional specialization
emerges spontancously in doep neural networks. Science Advances, (111)cabb88113, 2022. URL https:
//pubmed.ncbi. n1m. nww. goh.goo/35224241/
Shachar Don-Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning, arXiv prerint, 2022. doi: 10.48550/ arXiv.2212.01378. URL https::/ddii.org// 48550/arXiv.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014, volume 32 of JMLR Workshop and Conference Proceedings, pp. 647-655. JMLR.org, 2014. URL http: //proceedings. mlr www. wwt.:rprrsss/ddnnnene/s
4.mml
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghami, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenRoview.nec, 2021.
URL htt:://pperrrevieeeeeee/frrornne
Felix Draxler, Kambis Voschgini, Manfred Salmhofer, and Frod A. Hamprecht. Essentially no barriers in neural network energy landscape. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceediigg of Machine mlr Learning Research, pp. 1308 1317. PMLR, 2018. URL http::/proceediii// wttp://nww......e/////////////// ppwpps//v//drrallle..p html
Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts In International Conference
Machine Learning, wttp://eeeeeeee....eesee//e////e
ICML, 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 5547 5569. PMLR, 2022. URL https::/prooceddnnss mlr www.pprrssssv///eee http://222ddd2222ec 2w22/.du2cccccc html
Dheeru Dua, Shruti Bhosale, Vedamuj Goswami, James Cross, Mike Lewis, and Angela Fan. Tricks for training sparse translation models. In Proceedings of the 2022 Conference of the North American Chopter of the Associalion for Compulational Linguislies: Human Language Technologies, pp. 3340-3344 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. ww..11653v112000 /aac:///22222allll---a 244. URL https://aclanthology/ wowphhooooooooggggoggooooo .or.ooggg//// wwwgg/222..nnnm www.:22n.aacmmmmmmmm www..-maaal.ann..mmmmmmmmm 244
Steven Vander Eockt and Hugo Van hamme. Using adapters to overcome catastrophic forgetting in end-to- end automatic speech recognition. CoRR, abs/203.16082, 2022. doi: 10.48550/ arXiv.2203.16000 URL
https:///axxivoorr/bb/////
16882
David Eigen, Mare' Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in a deep mixture of experts. In 2nd International Conference Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014. URL
Fahim Faisal and Antonios Anastasopoulos. Phylogeny adaptation of multilingual models to new languages. In Proceedings of the 2nd Conference of the Asia- Paci-Paciii Chapter of the Association for Compulational Länguistics and the 12th International Joint Conference on Natural Language Processing,
51

---

Published in Transactions on Machine Learning Research (11/2023)
pp. 434-452, Online only, November 2022. Association for Computational Linguistics. //aclanntollog.orr/////// org/2022. aacl-mainn 34
URL
https
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaune Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Michael Auli, and Armand Joulin. Beyond english-centric multilingull machine translation. Journal of Machine Learning Research, 22:107:1-107:4: 2021. URL http://jmll wwwpil..rrg//ppapee wttpppaper//////22222 13227 html
Ruchao Fan and Abeer Alwan DRAFT: A novel framework to reduce domain shifting in sclf- learning and its application to children's ASR. In Hanseok Ko and John H. L. Hansen (eds.), Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18-22 September 2022, pp. 4900 4904. ISCA, 2022. doi: 10.21497/Interspeechhoo 2022 11128. URL https: //doi.org/10 2ttp///.21147//nteeeppeeeee
2022-11128
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. CoRR, abs/2101.03961, 2021. URL httts:://axxiv.orr/bbs////
03961
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning. CoRR, (bs/2209.016677 2022. doi: (48550//rXi..2200000
URL https://dei.org/// /tap:///rrrr/11...5///////// 48550/arXiv.
2209
01667
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent. super neural networks. CoRR, abs/1701.08734, 2017. URL http:///arxxv.orr//b//111....0
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical reinforcement learning. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.nee 2017. URL http:ttsss/ppeeeeviieeeee//ooooo
id-B1oK8aoxe
Jerry A. Fodor. The modularity of Mind. MIT Press, 1983. URL http:://mmttrrrss..tttta//////22 the-modularrtty-of-min
Negar Foroutan, Mohammadreza Banaei, Rémi Lebret, Antoine Bosselut, and Karl Aberer. Discovering language-neutral sub-networks in multilingual language models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 7560-7575, Abu Dhabi, United Arab Emi- rates, December 2022. Association for Computational Linguistics. URL https://aclaanthologyy. 22tp://nnnnnoooooooo/222222 2r2/ 1aallpmmmnn55
Jonathan Frankle and Michael Carbin.
The lottery ticket hypothesis: Finding sparse, trainable neural networks. In 7th International Conference on Learning Representations, 1CLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 3259 3269. PMLR, 2020. URL httpp://proceedings... mlr gress/v1199/frankeeee.
http:///weeeeeeennn...eeeeeaaaaa html.
C. Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization. Tn 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview net. 2017. URL http::://oenneeveee.ttefffooorr BkOFWVcgx
Robert M. French. Catastrophic forgetting in connectionist networks. Trends in Cognâtive Sciences, 3(4): 128-135, 1999. doi: https://doi.org 10. 1000100000/111111111100000011110 URL https://www.scceencee
52

---

Published in Transactions on Machine Learning Research (11/2023)
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversaril training of neural networks. Journal of Machine Learning Research, 17:59:1 59:35, 2016. URL
html
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. CLIP CLIP-Adapter: Better vision-language models with feature adapters. CoRR, abs/2110.04544, 2021a. URL
fttps://arxxi.oorg/ab///2/ 04544
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pro-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3816-3830, Online, August 2021b. Association for Computational Linguistics. doi: 10.18653/v1 11/86530/1/2022.acc--lll//
URL
wttg//22..////oo aalaaclloolo mocl-long... 295
Yuan Gao, Jiayi Ma, Mingbo Zhao, Wei Liu, and Alan L. Yuille. NDDR- CNN: layerwise fea- ture fusing in multi-task cnns by neural discriminative dimensionality reduction. In TEFF Con- ference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 3205-3214. Computer Vision Foundation / IEEE, 2019. doi: 10.1109 2019.00332. URL http://opeeaccess: Chature_Fusing.Tne Multi aaaaaassssssbbaaa
http://f..coocccoottttteeet 2011////221//m//aGaaaaa DDDDR CNNRR-CNNLLLN Layerrwise lttp:iiee_eeeP__11________P__ 2019 _app_ppppeer___
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Newral Information Processing Systems 31: Annual Conference on Newral Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 8803 8812, 2018. URL http:pss/pprcceedinns.eeee hctp:srrcppppssccpp/////// hwweeeee8874ee990eeeeeee hash/be3087el 4e91 1eeee144cccc22220eeeee46-6-6666- wat.essssts---btaatt.......
Neeraj Gaur, Brian Farris, Parisa Haghani, Isabel Leal, Pedro J. Moreno, Manasa Prasad, Bhuvana Ram- abhadran, and Yun Zhu. Mixture of informed experts for multilingual speech recognition. In IEEE International Conference on Acouslics, Speech and Signal Processing, SCASSP 2021, Toronto, ON, Canada, June 6-11, 2021, pp. 6234 6238. IEEE, 2021. URL bettps://doi.org///
1119//1AASS999970 2021. 9414379
Tomas Geffner, Javier Antoran, Adam Foster, Wenbo Gong, Chao Ma, Fmre Kiciman, Amit Sharma, Angus Lamb, Martin Kukla, Nick Pawlowski, Miltiadis Allamanis, and Cheng Zhang. Deep end-to- end causal inference. In NewrIPS 2022 Workshop on Causality for Real-world Impacl, 2022. URL
Andrea Gesmundo. A multi-agent framework for the asynchronous and collaborative extension of multitask ML. systems. CoRR, abs/2209.1444 14745, 2022. doi: 10.48550/arxiv.2222 wtw.Xivv2299.144444 URL.
48550/arXiv. 2209. 14745
Andrea Gesmundo and Jeff Dean. An evolutionary approach to dynamic introduction of tasks in large- scale multitask learning systems. CoRR, abs/2205. 12205.22255, 2022a. doi: 00.48500/arXi..2255... URL
https://6oi.org// 48550..8585//iii. 2205. 12755
Andrea Gesmundo and Jeff Dean. munet: Evolving pretrained deep neural networks into scalable auto- tuning multitask systems. CoRR, abs/2205.1n 2022b. doi: 1.45550/arXiv... 10937. URL https:
4ttp///0.555///xxvvv..2 2205. 10937
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.nee net, 2021. URL https:// nett/ForumPidmmmcmmd.Ee..
Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsogmented sequence data with recurrent neural networks. In William W. Cohen -hind and Andrew W. Moore (eds.), Machine Learning, Proceedings of the Twenty- Third International Conference (IGML 2006), Pillsburgh, Pennsyylaana, USA, June 25-29, 2006, volume 148 of ACM International
53

---

Published in Transactions on Machine Learning Research (11/2023)
Conference Proceeding Series, pp. 369-376. ACM, 2006. doi: 10.114/114884414488111 URL
https:
htt..oor//10.115//14584444 1143891
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. CoRR, abs/1410.5401, 2014. URL
http:///arxi..org/bb///1111 5401
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwinkke Sergio Gomez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John P. Agapion, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Sum- merfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471 476, 2016. doi: 10.1035/nature200 n0t11 URL https://6oi.org//11 n038/nature22101
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. CoRR, abs/2012.05200 2020. URL
Thomas L Griffiths and Zoubin Ghahramani. The indian buffet process: An introduction and review, Journal of Machine Learning Research, 12(4), 2011. URL http:://www..i..rrrrrrroopeeeeps griffithslla.ppi
httggffffttss
Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Knowledge is a region in weight space for fine-tuned language models. CoRR, abs/2302.04866 2023. doi: 10.48550/ar arvvi. 2302.04863. URL htttp:://dii.rr / 48550/arXivy 2ww.2330220099
Demi Guo, Alexander Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joinl. Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4884-4899 Online, August 2021. Association for Computational Linguistics. doi:
acl.log.3... URL
https://aclanthologyy.oo org/2021 2ttp//2211..aaloggg.. 1o1-long..78 378
Jiang Guo, Darsh Shah, and Regina. Barzilay. Multi-source domain adaptation with mixture of experts. In Proceedings of the 2018 Conference on Empirical Melhods in Natural Language Processing, pp. 4694 4403, Brussels, Belgium, October-November 2018. Association for Computational Linguisties. doi: 10.18653/v1/ D18- 1498. URL https::/acclatthooogyy.o///////
Shashank Gupta, Subhabrata Mukherjoe, Krishan Subudhi, Eduardo Gonzalez, Damien Jose, Ahmed Hassan Awadallah, and Jianfeng Gao. Sparsely activated mixture-of-experts are robust multi-task learners. CoRR, abs/ abbs/2244.0768) 2022. doi: (18550/arXiv 220....200..06689
URL https:://oi.org/// 485.44/808ii..222 wwtp://............/////....... 2204
07689
Vipul Gupta, Santiago Akle Serrano, and Dennis DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. CoRR, abs/20011 /bs/2200.,231,, 2020. URL
02312
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. DEMix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5557-5576, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/ 2022. www.:aaaalmmi...... 407. URL
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmentedd language model pre-training. In International conference on machine learning, pp. 3929 3938. PMLR, 2020.
David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In 5th International Conference on Learning Representations, ICLR 2017, Touln, France, April 24-26, 2017, Conference Track Proceedings. OpenRe- view.nel, 2017. URL /tttp:///ppenreview.. watteerrvvieeneet//oooeee meee/forrmmiddrkpAceel
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adversarial ReProgram- ming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the
54

---

Published in Transactions on Machine Learning Research (11/2023)
11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4921- 4933, Online, August 2021. Association for Computational Linguistios. doi: 10.18653/v1 acl.oo.... URL https://acllanhhooogy.orr ottp://o.oooooo..///....ddoooooa acl-long.38 381
John B. Hampshire and Alex Waibel. The meta-pi network: Building distributed knowledge representations for robust multisource pattern recognition. JEEE Transactions on Pattern Analysis and Machine Intelligence, URL httpp:///oi.orgg
14(7)7751-769, 1992. doi: 10.1109/34.14299
1109/34. 142911
Song Han, Jeff Pool, John Tran, and William J. Dally, Learning both weights and connections for efficient neural network. In Advances in Newral Information Processing Systems 28: An- nual Conference Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pp. 1135-1143, 2015. URL https://procceedings.. http://sseeirips..ppprr/////ee/e httpppappr222///aaaaap
Song Han, Jell Pool, Sharan Narang, Huizi Mao, Enhao Gong, Shijian Tang, Erich Elsen, Peter Vajda, Manohar Paluri, John Tran, Bryan Catanzaro, and William J. Dally. DSD: dense-sparse-dense training for deep neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview. net, 2017. URL https: //openreview. net.//wwwweeeeeeeeeeeeeeeeee http//oorumiidddyyee
Wenjuan Han, Bo Pang, and Ying Nian Wu. Robust transfer learning with pretrained language models through adapters. In Proceedings of the 59th Annual Meeling of the Associalion for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 854- 861, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ 20/533/1/222/111 acl-shorr..100 aclsoo2acl-ssrorr URL
hwwp:/llogy..ggg//g/// otgo acl- 108
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed H. Chi. Dselect-k: Differentiable selection in the mixture of ex- perts with applications to multi-task task learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Newral Information Processing Systems 2021, NewrIPS 2021, December 6- 14, 2021, virtual, pp. 29335 29347, 2021. URL Abstsllatbsstttt.aaaeeeaaaaaaa.e html
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Kirk-aiirp,,tick, and Graham Neubig, Towards a unified view of crrameter-efficeee transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Ecent, April 25-29, 2022. OpenReview.net, 2022a. URL https !
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In Computer Vision ECCV 2016 14th European Conference, Amsterdam, The Netherlands, October 1 11-14, 2016, Proceedings, Part IV, volume 9908 of Lecture Notes in Computer Science, pp. 630 645. Springer, 2016. doi: 10.1007/978-33 0117888-319--45- 319-46499----
0'008,0038...00000 URL haLLhttps://ddo wtt.://///11.0/0//1.00 3-91------------ 46493-04440000 0_38
Ruidan Hc, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawci Low, Lidong Bing, and Luo Si. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 2208-2222, 2222, Online, August 2021. Association for Compulational Linguistics. doi: (0.18653/v1/2021..e acl-long... 172. URL
https:///allanthology org/ wttgg//21.aalloooooo.o acl-long. 172
Xuchai Hc, Chumyuan Li, Pengchuan Zhang, Jianwei Yang, and Xin Eric Wang. Parameter-officient fine- tuning for vision transformers. CoRR, abs/ /2///222.1122) 2022b. doi: 10.48550/arXi/ 2w....22223333330 URL
https:///ol.org/// 10tgg///0... 0011//00..88500 48550/ 11x.vv.222. 2203. 16329
Yun He, Huaixiu Steven Zheng, Yi Tay, Jai Prakash Gupta, Yu Du, Vansi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, Heng-Tze Cheng, and Ed H. Chi. Hyperprompt: Prompt-based task- conditioning of transformers. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 8678-8690. PMLR, 2022c. URL http::::/ https://ppocceeding... mlr. pttppp/eee////e22f////
hft.eeeddnnn.....psssss//////ena htmll
55

---

Published in Transactions on Machine Learning Research (11/2023)
Nicolas Heess, Gregory Wayne, Yuval Tassa, Timothy P. Lillicrap, Martin A. Riedmiller, and David Silver. Learning and transfer of modulated locomotor controllers. CoRR, abs/1610.05182, 2016. URL http:
Taaxxiv.rrg/bbs/6ii
Wenxin Hou, Han Zhu, Yidong Wang, Jindong Wang, Tao Qin, Renjun Xu, and Takahiro Shinozaki. Exploiting adapters for cross-lingual low-resource speech recognition. IEEE ACM Transactions on Audio Speech Language Processing, 30:317 329, 2022. doi: 10.1109/TASLP 31386777 URL httpp:://oi.on org/
16. 11 109//TTLL...... 2021. 3138674
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effiiint transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2799. PMLR, 2019.
URL http://procceddngs..llr mlr http://sm.....ree////////oll//// 19t. html
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 328 339, Melbourne, Australia, 2018, Association for Computational Linguistics. doi:
(0.18653/v1/B5 1031. URL /tteas//aaclanthoolloy.rr/// 1031
Cheng-Ping Hsieh, Subhankar Ghosh, and Boris Ginsburg. Adapter-basee extension of multi-speaker text- lo-speech model for new speakers. CoRR, abs/2211.00585, 2022. doi: 10.48550/ar (15500/aXii.... 2211.00585. URL
https://doi.org http:///ooo.rr//0.... 48550/ wttp0//rXii.... 00585
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference Learning Representations,, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
Junjie Hu, Scbastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, pp. 4411-4421, 2020. URL http://prooceedings. mlr wwtpp//e1e///ue////// http://cceennn.......eeeee////// html
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pp. 804-813. IEEE Computer Society, 2017. doi: 10.1109/1CCV.20111 2017.93. URL https://doi..o /00. 1009/ICV 2017.93
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9118 9147. PMLR, 17-23 Jul 2022. URL httpps://rroceedings .ml..mmr...... http:/dddnnnsnm.....rrss/////aaa wtwpprss/ii1//uuaggaaa.aaa
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research, 67:757 795, 2020. doi: 10.1613/jair.. 1.11677. URL https::/ 1613/jair... 11167
Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Fditing models with task arithmetic. CoRR, htt//222..04000 2022. doi: 10.48550/ aa15500///Xi..... 20t.iv2.2222.000... URL http::://doi.orgg////
48550/arXiv 0r8.oo0/////.....////rrr.ii. 2212. 04089
Robert A. Jacobs, Michael Jordan, and Andrew G. Barto. Task decomposition through competition in a
modular connectionist architocture: The what and where vision tasks. Cognitive Science, 15(2):219-250,
1991a. doi: 10.1207/s1551661 10.1207/115511170ccoog11111 2. URL https::/ddoi.org/// http:////...10111151117709990990
Robert A. Jacobs, Michael I. Jordan, Steven Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Computation, 3(1):79-87, 1991b. doi: 00.1662/ncoo.1991.3.11 URL https:
aibbi.orrg/e.116610 19.79..1.99
56

---

Published in Transactions on Machine Learning Research (11/2023)
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In 5th Interna- tional Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL
Joel Jang, Scungone Kim, Sconghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontac Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning, 2023.
Adrián Javaloy and Isabel Valera. Rotograd: Gradient homogenization in multitask learning. In The Tenth International Conference on Learning Represenlations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hicrarchi- cal deep reinforcement learning. In Advances in Neural Information Processing Systems 32: Annual Conference on Newral Information Processing Systems 2019, NeurIPS 2019, December 8- 8-14, 2019, Van- couver, BC, Canada, pp. 9414 9426, 2019. URL http:://proceddngg.enrri..pooo wttp:ipppccppppp////eeeca /01h/hahh/ 7887777945772199222fff33323555 hatp:/6555--Asstatt.....o Abstract.html
Xisen Jin, Xiang Ren, Daniel Preotiuc- Pietoo, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models, 2023.
Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the EM algorithm. Newral Compulation, 6(2):181-214 1994. doi: wtt.////cco.1994....... URL https:///di..org/1...11 1162//nec 1994.6.2.
Nikhil Kandpal, Brian Lester, Mohammed Mugoeth, Anisha Mascarenhas, Monty Evans, Vishal Baskaran, Tenghao Huang, Haokun Liu, and Colin Raffel. Gil-theta: A git extension for collaborative development of machine learning models, 2023.
Anjuli Kannan, Arindrima Datta, Tara N. Sainath, Eugene Weinstein, Bhuvana Ramabhadran, Yonghui Wu,, Ankur Bapna, Zhifeng Chen, and Soungji Lec. Large-scale multilingual speech recognition with a streaming end-to-end model. In Proceedings of Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pp. 2130-211 2134, 2019. doi: 2019-2858
211444///terspeecchh2000 2019-2858..1. URL https://doi. 0tt./1...214410 21437 Int.rpeeettt.p.chhc.cem ht4.g////1111443///eeeeeeeet
Nadav Kashtan and Uri Alon. Spontaneous evolution of modularity and network motifs. Proceedings of the
National Academy of Sciences (PNAS), 102(39):13773 13778, 2005, doi: https:/ http:://oooorg///// 10. 10.1073/1111/ 10013/7/paas pnas. 0503610102. URL httpps://6o..rrg//
Nan Rosemary Ke, Olexa Bilaniuk, Anirudh Goyal, Stefan Bauer, Hugo Larochelle, Chris Pal, and Yoshua Bengio. Learning neural causal models from unknown interventions. CoRR, abs/ 10101990000075, 2019. URL
http://arxiv. hwt.://ogg////////eeeee 1ttp///////1110075555
Nan Rosemary Ke, Aniket Didolkar, Sarthak Mittal, Anirudh Goyal, Guillaume Lajoie, Stefan Bauer, Danilo Jimenez Rezende, Michael Mozer, Yoshua Bengio, and Chris Pal. System- atic evaluation of causal discovery in visual model based reinforcement learning. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Newral Information Processing Sys- tems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual, 2021a.
Zixuan Ke, Hu Xu, and Bing Liu. Adapting BERT for continual learning of a sequence of aspect sen- timent classification tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4746-4755, Online, June 2021b. Association for Computational Linguistics. doi: 10.18653/v1/2022 maacl mann--mmn.n3 URL https:://acllanthloooy..o/r/ wtt.:orrr/22...aaammaaaa....aa naacl-main.
57

---

Published in Transactions on Machine Learning Research (11/2023)
Simran Khamuja, Melvin Johnson, and Partha Talukdar. MergeDistill: Merging language models using pre-trained distillation. In Findings of the Association for Compulational Linguisties: ACI-IJCNLP 2021, pp. 2874 2887, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022 findings-acl.254 URL https://aaclatthooooooor////
Young-Bun Bum Kim, Karl Stratos, and Dongchan Kim. Adversarial adaptation of syuthetic or stale data. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1297-1307, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P173
URL httts:///cclantnooooy..r../////p
Scott Kirkpatrick, C. Daniel Gelatt Jr, and Mario P. Vecchi. Optimization by simulated annealing. Science, 220 680, 1983. doi: https://doi.org/ 10. 10..1.1//sciiiiii 126/science.220) 229..99..777
URL. https: http::/ooi..oo... org/10
1126/science.i 220 1108.4598.677
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning to decompose neu- ral computation. In Advances in Newral Information Processing Systems 31: Annual Confer- ence Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Mon- tréal, Canada, pp. 2414-2423, 2018. URL http:://rrocedings.eeun ctw.reeeddnnn...euuii......//./e 310ce61 BBce6llco0ffa6ee3e0e2222/bb/bbba -ataas577bcceee--bbaattttt..... html
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in nearal information processing systems, 35:22199 22213, 2022.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey E. Hinton. Similarity of neural network representations revisited. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 3519 3529. PMLR, 2019. URL http:: http://proceedings..te pttppp///////o//lliieee.eee http://poceeddins...lgssssss///d 9a. html,
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, and Orhan Firat. Beyond distillation: Task-level offture-off-xperts for efficient inference. In Findings of the Association for Computational Linguisties: EMNLP 2021, pp. 3577-3599, 2021. URL https: //aclanthology.orgee www.eyy..gg/......// http://w.o...o.....iooooeeeeeeee
Tejas D. Kulkarni, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep rein- forcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in Newral Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Borcelona, Spain, pp. 3675 3683, 2016. URL https::/proceeeiings.. neurips................. html
Ken'ichi Kumatani, Robert Gmyr, Felipe Cruz Salinas, Linquan Liu, Wei Zuo, Devang Patel, Eric Sun, and Yu Shi. Building great multi-lingual teacher with sparsely-gated mixture of experts for speech recognition. CoRR, abs/2111.0558 2021. URL
05820
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-ssquuen recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 15, 2018, volume S0 of Proceedings of Machine Learning Research, pp. 2879 2888. PMLR, 2018. URL http::/pproceeding mlr. walr.press/vee
1ake18a, html
Adrian Lancucki. Fastpitch: Parallel text-to-speech with pitch prediction. In IEEE International Conference Acoustics, Speech and Signal Processing, ICASSP 2021, Toronlo, ON, Canada, June 6-11, 2021, pp. 6588 6592. IEEE, 2021. doi: 10.1109/ICAASSP332222202 21t.:/99977722222111113333339 URL https::/ddi.org//0
ICASSP39728. 2021 9991899
Richard D. Lange, David S. Rolnick, and Konrad P. Kording. Clustering units neural networks: upstream downstream information. CoRR, abs/22000 220111185,, 2022. doi: 10.48550 aaa8500/axXi..... Xiv 1281.1 URL https:
wttp:///...85500xxx...22222 11815
58

---

Published in Transactions on Machine Learning Research (11/2023)
Anne Lauscher, Olga Majewska, Leonardo F. R. Ribeiro, Iryna Gurevych, Nikolai Rozanov, and Goran Glavaš. Common sense or world knowledge? investigating adapter-based knowledge injection into pretrained transformers. In Proceedings of Deep Learning Inside Out (DeeL10): The First Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 43-49, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v 120220 1.5. URL https://aclanthollogg. wrgp//....rr////// deelio-1.5
Anne Lauscher, Tobias Lueken, and Goran Glavaš. Sustainable modular debiasing of language models, In Findings of the Association for Computational Lònguisties: EMNLP 2021, pp. 4782 4797, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/v1/200 findings-emnle een.plgg--enn...1111 411
findings-emmlp.411 URL
Hang Le, Juan Pino, Changhan Wang, Jiatao Gu, Didier Schwab, and Laurent Besacier. Lightweight adapter tuning for multilingual speech translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 817-824, Online, August 2021. Association for Computational Linguisties. 103
doi: 10.18653/v1, aac-shoo-ooo URL
Quoc Viet Le, Tamás Sarlós, and Alexander Johannes Smola. Fastfood: Approximate kernel expansions in loglinear time. CoRR, abs/1400 141111000000 2014. URL
3060
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL
7XHTmbb
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parametar-cfficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045 3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistios. doi: 10. 10.18653/v1, /22522 www.....mmpmmmii..... 243. URL https:://claatthll..g
orgg//21/ emnlp-main. 243
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. BASE layers: Simplifying training of large, sparse models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp-
6265 6274 PMLR, 2021. URL http:://proceedings.ooo ml.....m.l..p....... prwss/v1 wttss//119//wwws222222 http//ewws1aaaa html
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Netrieval-auggeent generation for knowledge- intensive nlp tasks. Advances in Neural Information Processing Systems, 33:9459-9477 2020a.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yib, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrievall-nggmenee generation for knowledge-intenen intensive NLP tasks. In Advances in Neural Information Processing Systems 33: Annwal Conference Neural Information Processing Systems 2020, NewrIPS 2020, December 6- 12, 2020, virtual, 2020b. URL https::/pproceedingg.. newrips...pppp..... http:///eeeeeeeeeeeeeesccc//c/// 010222002255788001b225553555 waw.ss/rrr..tS-tttaaat.....
Belinda Li, Jane Yu, Madian Khabsa, Luke Zeitlemoyer, Alon Halevy, and Jacob Andreas. Quantilying adaptability in pre-trained language models with 500 tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4696 4715, Seautle, United States, July 2022a. Association for Computational Linguistics. doi: 10. 10.18633/v1 mac.:e2..aacllmai334444 URL
wttp///22..aaammmaaaann. 346.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In 6th International Conference on Learning Representations, ICLR 2018, Vancoueer, BC, Canada, April 30 May 3, 2018 Conference Track Proceedings. OpenReview.nes net, 2018. URL
59

---

Published in Transactions on Machine Learning Research (11/2023)
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Gotmare, Shafqq R. Joty, Caiming Xiong, and Steven Chu- Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. In Advances in Neural Information Processing Systems 34: Annual Conference on Newral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 9694-9705, 2021. URL https:
puproceedings.nneuriie httpsssseerippp..pppppe/r//////e wttpcccpppppppee////eee 0ttpe///000hh1111010364444433370 hat.4444444400110200000005-55..0
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. Branch-trainn-mrrge: Embarrassingl parallel training expert language models. CoRR, abs/ 0t///2208.03000 2022b. doi: 10.48550/arXiv.2220 wwt.:/x....0..0000300 URL https://doi. http://ooooorr//////// www.:o../////////a..... 48550/ar2ii... 2208 03306
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Mecting of the Association for Computational Linguistics and the 11th International Joint. Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-45991 Online, August 2021. Association for Computational Linguisties. doi: 10.18653/v h/tp/////2222//// acll-long.3355 URL
https://aclanthollen wwwphooloooooogggg//// org/2021. acl long. 353
Jianze Liang, Chengqi Zhao, Mingxuan Wang, Xipeng Qiu, and Lei Li. Finding sparse structures for domain specific neural machine translation. In Thirty-Fifth AAAT Conference on Artificial Intelligence, AAAI 2021, Thirty- Third Conference on Innovatice Applications of Artificial Intelligence, LAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 13333 13342. AAAI Press, 2021. URL bittps://ojt aaai org/ wwwooorgggi/eeeeeeeeeeeeeeeeeeee htt/:/iddxx..pppe inndxx.ph//AAA//arriilllvvi 17574
Zehui Lin, Liwei Wu, Mingxuan Wang, and Lei Li. Learning language specific sub-network for multilingual machine translation. In Proceedings of the 59th Annuol Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 293-305, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/vi 2021.acl- acl.-lng.g.2. URL
www.///...//ac///. acl-loncl-oong 25
Robert Litschko, Ivan Vulié, and Goran Glavaš. Parameter-effiiiett neural reranking for cross-lingual and multilingual retrieval. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 1071 1082, Gyeongju, Republic Korea, October 2022. International Committee on Computational Linguistics. URL https: https::/accaantholle..oorr wwwp://loogyoooo/2...ooo
Chen Liu, Jonas Pfeiffer, Anna Korhonen, Ivan Vulié, and Tryna Gurevych. Delving deeper into cross-lingual visual question answering. CoRR, abs/2200..07630, 2022a. URL https::/arxiv..ortt wwtp://wwwwrg//////....oo
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva. Reddy, Nigel Collier, and Desmond Elliott. Visually grounded reasoning across languages and cultures. In Proceedings of the 2021 Conference on Empérical Methods in Natural Language Processing, pp. 10467-10485, Online and Punta Cana, Dominican Republie, November 2021a. Association for Computational Linguistics. doi: (0.18633/v1/200 .emmpppaenll 818. URL https:://cclantholog.. htwpe/oooooooog/ggeooee httg:/2221.e.mmmppmm www.l-mmin. 818
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. CoRR, hte//205.056633 2022b. doi: 10.48550/arXiv http://aaa.i.....055666 220.02230566... URL https://doi. org/ 4050///0..555////i1i 2205 05638
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhongbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pro-train, prompt, and predict: systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9))195:11199))) 2023. doi:
URL https://66i.org/1e..r
3560815
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujic Qian, Zhilin Yang, and Jie Tang. GPT understands, 10385
too. CoRR, abs/2103.1036, 2021b. URL htttps://arxxv.org///
60

---

Published in Transactions on Machine Learning Research (11/2023)
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61 68, Dublin, Ireland, May 2022c. Association for Computational Linguistics. doi: 2..18633/v1/022..cg achosttt.... URL https://aclantholoogy htwp:/llggg..gg//2222e/// wwrg/2222.aclsshorr
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Länguisties, pp. 1487-4496, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1 /1/653/11/P1----- 1441. URL
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726 742, 2020. doi: 10.1162/tacl a 00343. URL httpss:/acclatthlloe wwwp:orgg//000 tac1-1.47
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Newral Informalion Processing Systems 33: Annual Conference on Newral Information Processing Systems 2020, NearIPS 2020, December 6-12, 2020, virtual, 2020. URL 2020/hash/851 www.dof02200000eea222555560
3bc@f-Abstract. Aww.--btttttatttmmmmmaa
Qiuhao Lu, Dejing Dou, and Thien Huu Nguyen. Parameler-efficient domain knowledge integration from multiple sources for biomedical pre-trained language models. In Findings of the Association for Com- putational Lènguistics: EMNLP 2021, pp. 3855-3865, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2022 111.653/112221...ninneee/m///// findings-emmlpp.25.. URL https://aclanthology..oo httg:/222...ndings--- Sindinnsseenlp..... 325.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shol prompt order sensitivily. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: /22///////2222acclll/l https://aclanthhn wwwpelogg...ggg//21 wwt.oogg...o....ooooo acl www-llon....... 556
URL
Yongxi Lu, Abhishek Kumar, Shuangfei Zhaii Yu Cheng, Tara Javidi, and Rogério Schmidt Feris. Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification. In 2017 TEEE Conference on Computer Vision and Pallern Recognilion, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 1131 1140. IEEE Computer Society, 2017. doi: 10.1109 /C////CVR..2.112.0 URL https:://do..org//// 11t.//...11199/00 1109/CVPR. 2017. 126
Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Fd H. Chi. Modeling task relationships in multi- task learning with multi-gate mixture-of In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018 pp. URL https:://doi.org/1..rr 11tp/////...1//1111111 3220007
1930 1939. ACM, 2018. doi: 011144//219899.3220000
Chris J. Maddison, Andriy Mnih, and Yee whye Tch. The concrete distribution: A continnous relaxation of discrete random variables. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.neg 2017. URL. https:
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-
rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems 34:
Annoua Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-
14, 2021, virtual, pp. 1022-1035, 2021a. URL httpss///poceeinng.eerrrsssssssp DBBbeF9FdFFFFF33c888443356666663 -
61

---

Published in Transactions on Machine Learning Research (11/2023)
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-cfficient multi-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 565-576, Online, August 2021b. Association for Computational Linguistics. doi: 10. 10.18653/v1 /tt0///1.2221//ooog acl-long.47, URL httpss//aclanttollgg.oooom 2222 2021
acl-long.47 47.
Adyasha Maharana, Darryl Hannan, and Mohit Bansal. Storydall-e: Adapting pretrained text- imaage transformers for story continuation. In Computer Vision - ECCV 2022 17th European Conference, Tel Aviu, Israel, October 23-27, 2022, Proceedings, Part XXXVII, volume 13697 of Lecture Notes in Computer Science, pp. 70 70-87. Springer, 2022. doi: (0.1007/978--00011198888 978 00----0111886666.
5. URL https://doi. wwaeddii.rggg/// org/10. 1007/
Olga Majewska, Tvan Vulié, Goran Glavaš, Edoardo Maria Ponti, and Anna Korhonen. Verb knowledge injection for multilingual event processing. In Proceedings of the 59th Annual Meeting of the Association for Compulational Linguistics and the 11th International Joinl Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6952 6969, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1 11//55/////221...// (2ll..ac-ooonn5541 541. URL httpss///aclantnooogoooor/////// acl-ooo.....ooonn.. .41
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Sall Lake City, UT, USA, June 18-22, 2018, pp. 7765-7773 Computer Vision Foundation IEEE Computer Society, 2018. doi: 10.1109 D0..109/CVPR..C /C//CCCPRR.11 2018.00000. 00810. URL http: http:///openaccess.. thecvf. www.//ooteencccppp...ooo cttpaee__c_______hheee httppm///////mlalyyya/ PackNet_Adding Mul tiple_ www.e_ccpR_____________________e 111888_appp____e html
Arun Mallya, Dillon Davis, and Svetlana Lazebník. Piggyback: Adapting a single network to multiple tasks by learning to mask weights. In Computer Vision - ECCV 2018 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part IV, volume 11208 of Lecture Notes in Computer Science, pp. 72-88. Springer, 2018. doi: 11110077997833-0000-022-----1 978 3- 300-300012225-00..
5. URL https://doi http:///66i..rr/1.1000000
Michael Matena and Colin Raffel. Morging models with fisher-weighted averaging. CoRR, abs/ abs/2111 2111.09822 2021. URL http:////axxxv.oo/bbbbsoo/
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of Learning and Motivation, volume 24, pp. 109 165. Elsevier, 1989. doi:
URL
60536-8
Rahul Mehta. Sparse transfer learning via winning lottery tickets. CoRR, abs/1905.07755 2019. URL
http://axxii.orggab//////// 07785
Elliot Meyerson and Risto Miikkulainen. Beyond Shared Hicrarchies: Deep Multitask Learning through Soft
Layer Ordering. In Proceedings of ICLR 2018, 2018.
Richard Meyes, Constantin Waubert de Puiseau, Andres Posada- Moreno, and Tobias Meisen. Under the hood of neural networks: Characterizing learned representations by functional neuron populations and network ablations. CoRR, abs/2004.01254 2020. URL
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Railcamu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: A survey. CoRR, abs/ abs/2302.07882,, 2023. doi: 10.48550/ Xiv. 230.0.82222. URL https:://doi.org//1 http://.......g/....////////aaia 2302. 07842
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancowver, BC, Canada, pp. 14014-14024, 2019. URL https:
62

---

Published in Transactions on Machine Learning Research (11/2023)
Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. Tn Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pp. 3111-3119, 2013. URL httpps://proceedinng html
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 3470-3487, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: wttp:/3/v1/2222cllbbbe 244. URL https:
//aclanthology..o org/ /ttp://..aa... acl-long.. 244
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 3994-4003. TEEE Computer Society, 2016. doi: 10.1109 0..100//CVP..001..443 URL https://doi orggrrgg///// 10. 109/CVPR, 20T..2216..4 433.
Sarthak Mittal, Alex Lamb, Anirudh Goyal, Vikram Voleti, Murray Shanahan, Guillaume Lajoie, Michael Mozer, and Yoshua Bengio. Learning to combine top-down and bottom-up signals in recurrent neural networks with allention over modules. In Hal Dauné III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6972 6986. PMLR, 13 18 Jul 2020. URL htttps://proceeete wtwp:/iiiinnmmm..sseeeeeeee ptwp.prres//111//ttttll/lal html
Sarthak Mittal, Yoshua Bengio, and Guillaume Lajoie. Is modular architecture enough? CoRR, abs/2206.0277 02713, 2022. doi: 10.48550/arXiv 2tt.kivv.2..0022.... URL
02713
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource elficient inference. In 5th International Conference on Learning Represenlations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.nee .et. 2017. URL
Nobuyuki Morioka, Heiga Zen, Nanxin Chen, Yu Zhang, and Yifan Ding. Residual adapters for few-shot 485v0/
text-to-speech speaker adaptation. CoRR, abs/2210. 2210.15868, 2022. URL httpss://di..oon 1rt.../////..445/// 2210.15868
Mohammed Muqeoth, Haokun Liu, and Colin Raffel. Models with Conditional Computation Learn Suboptimal Solutions. arXiv preprint, 2022. URL https://collinrafeet htt.://wwww.o.ooo.aaaa/.oooo//na
pdf
Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing, 2023.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical rein- forcement learning. In Advances in Neural Information Processing Systems 31: Annual Con- ference on Neural Information Processing Systems 2018, NewrIPS 2018, December 3-8, 2018, Montréal, Canada, pp. 3307 3317, 2018. URL http:://prroceedngs..eees http:/sssrpipppcpppppeeeeeeee//e cc/ppapp///01////// 8/hash/ hawpoebbbee---bbbatt....tt.
Vaishnavh Nagarajan and J. Zico Kolter. Uniform convergence may be unable to explain generaliza- tion in deep learning. In Advances in Newral Information Processing Systems 32: Anrual Confer- ence on Neural Information Processing Systems 2019, NewrIPS 2019, December 8-14, 2019, Vancou- ver, BC, Canada, pp. 11611 11622, 2019. URL http:://pooeeddnnsgeerrrrisscc/s Abstract..tmmm
Nihal V Nayak, Peilin Yu, and Stephen H Bach. Learning to compose soft prompts for compositional zero-shot learning. Xiv preprint arXiv:2204.0357, 2022. CRL https://arxiv.. http://arkv...rr//d//22.4..00
Renato Negrinho, Matthew Gormley, Gcoffrey Gordon, Darshan Patil, Nghia Lc, and Daniel Ferreira. Towards modular and programmable architecture search. Advances in neural information processing systems, 32, 2019.
63

---

Published in Transactions on Machine Learning Research (11/2023)
Alejandro Newell, Lu Jiang, Chong Wang, Li-Jia Li, and Jia. Deng. Feature partitioning for efficient multi-task architectures. CoRR, has/1908.04339) 2019. URL
04339
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning?
Tn Advonces in Neural Information Processing Systems 39: Annual Conference on Newral Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https://prooceedingg
heurips.cc/pane http///pe00222222266666666666661 -70
Olcksiy Ostapenko, Pau Rodriguez. Massimo Caccia, and Laurent Charlin. Continual learning via local module composition. In Advances in Neural Information Processing Systems 34: An- nual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 30298 30312, 2021. URL
Aato0-bbbsttttttttec html
Junting Pan. Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. ST-Adapter: Parameter-effictne image-to-video transfer learning for action recognition. CoRR, abs/2206. 135559, 2022. URL https:
//6b0..rrg/10.44556//rX/ 2206. 13559
Pinelopi Papalampidi and Mirella Lapata. Hierarchical3d adapters for long video-to-text summarization. CoRR, abs/ 2220.0020,,,,, 2022. doi: (0.48550/arXx1..210.000 URL
2210. 04829
Giambattista Parascandolo, Niki Kilbertus, Mateo Rojas-Carulla, and Bernhard Schölkopf. Learning indepen- dent causal mechanisms. In Proceedings of the 35th International Conference on Machine Learning, 1CML 2018, Stockholmsmäsann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learn- ing Research, pp. 4033 4041. PMLR, 2018. URL http://prooceddings mlr. prress//80/praasaadddoo html
Marinela Parović, Goran Glavaš, Ivan Vulić, and Anna Korhonen. BAD-X: Bilingual adapters improve
zero-shot cross-lingual transfer. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1791 1799, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1 /2022 waa.:/aaaalllaaaa 130. URL
https://aclanthologg.oorrr wwgp:/22.....2/222../// mrap.r0...22....mmmmmiiin 130
Judea Pearl. Causality. Cambridge University Press, 2009. URL /0000//0///00444444000027/000000
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th Innovalive Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. index. Swwpee.hppAAAAIAAAA http://AAA/AAAAAT///ppa/e/////// httpa///////pppeeeee//////////// 16528
AAAI Pross, 2018. URL https://www www. org/oc//
Jonas Pfeiffer, Andreas Rücklé Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. AdapterHub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 46 54, Online, October 2020a. Association for Computational Linguistics. doi:
URL
Jonas Pfeiffer, Ivan Vulié, Iryna Gurevych, and Scbastian Ruder. MAD-X: An Adapter- Baste Framework for Multi-Task Cross-Lingual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7654-7673, Online, November 2020b. Association for Computational Linguisties. doi:
URL httpps://cclantholggy/ http://cccnnntooooyyyo.rr////22 orgg emmlp-main
617.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion Fusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Conference of the
64

---

Published in Transactions on Machine Learning Research (11/2023)
European Chapter of the Association for Computational Linguistics: Main Volume, pp. 487-503, Online, April 2021a. Association for Computational Linguistics. doi: 10.18653/ /2021 2e20002eelll2aaaii macl-main.39. URL
wrgp://g./////ee///o -mainn 39
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Scbastian Ruder. UNKs everywhere: Adapting multilingua language models to new scripts. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10186 10203,, Online and Punta Cana, Dominican Republic, November 2021b. Association for Computational Linguisties. doi: 10.18653/v1/2000 wewnle-manniiiie main.800. URL https: // 800.
Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin Steitz, Stefan Roth, Tvan Vulić, and Tryna Gurevych. xGQA: Croes-lingual visual question answering. In Pindings of the Association for Computational Linguisties: ACL 2022, pp. 2497-2511, Dublin, Ireland, May 2022a. Association for Computational URL
Linguistics. doi: 10. 22/5333/1/222.mmm// 196
org.2/22 wttp://g///.......n.io.... findings-acl
Jonas Pfeiffer, Naman Goyal, Xi Lin, Xian Li, James Cross, Sebastian Riedel, and Mikel Artetxe. Lifting the curse of multilinguallity by pre-training modular transformers. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3479 3495, Seattle, United States, July 2022b. Association for Computational Linguistics. doi: 10. /1/663//1//222..../ naacl-main.2555555 255. URL https://aclanthologgy wwtp:/p../.....oooo naacl naw.--mainnnnnnnnn 255
Minhiuuang Pham, Josep Maria Crego, and François Yvon. Revisiting multi-domain machine translation. Transactions of the Association for Computational Linguistics, 9:17-35, 2021. doi: 10.1162/tacl 00351.
URL https://aclantthloogs org/ tac1-1.2
Jerin Philip, Alexandre Berard, Matthias Gallé, and Laurent Besacier. Monolingual adapters for zero-shot neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4465 4470, Online, November 2020. Association for Computational Linguistics. URL https:://cclanthoooggg
doi: 10. /ttp://1112220..mmmmmaaaaa
wwwgg//22...om we20..emllpmmmmmeemmmm.mm emnlp-main. 361
Thomas Pierrot, Guillaume Ligner, Scott E. Reed, Olivier Sigaud, Nicolas Perrin, Alexandre Laterre, David Kas, Karim Beguir, and Nando de Freitas. Learning compositional neural programs with re- cursive tree search and planning. In Advances in Newral Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Van- couver, BC, Canada, pp. 14646- 14656, 2019. URL httpps:/pproceeeingg..no neurips ctteriips.cpppppprreeeee cc/ppaper/209/hhah// 000232e6611633322910053233224433
Jonathan Pilault, Amine Elhattami, and Christopher J. Pal. Conditionally adaptive multi-task learning: Improving transfer learning in NLP using fewer parameters & less data. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
URL
Emmanouil Antonios Platanios, Mrinmaya Sachan, Graham Neubig, and Tom Mitchell. Contextual parameter generation for universal neural machine translation. In Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Processing, pp. 425-435, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-10 URL. https: //aclanthology. www.hhhoooooyooo//-----18 org/
Edoardo M. Ponti. Inductive Bias and Modular Design for Sample-Efficient Neural Language Learning. PhD thesis, University of Cambridge, 2021. CRL httpss:/doi. org/10. 11/.....116633 17863/CAM. 66424
Edoardo M. Ponti, Ivan Vulié, Ryan Cotterell, Marinela Parovic, Roi Reichart, and Anna Korhonen. Parameter space factorization for zero-shot learning across tasks and languages. Trensactions of the Association for Computational Linguisties, 9:410-428, 2021. doi: 10.1162/taci a 00374. URL https:
wtw.tthhooogy.org/222220 1.t..///21....ll1..555.. tac1-1.25.
65

---

Published in Transactions on Machine Learning Research (11/2023)
Edoardo M. Ponti, Alessandro Sordoni, and Siva Reddy. Combining modular skills in multitask learning. CoRR, abs/2202.1399 2022. URL https://arxiv, h 13914
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulié, and Anna Korhonen. XCOPA: A multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2362-237H Online, November URL https:
2020. Association for Computational Linguistics. doi: 0tt0:633//120200mmmml/e //aclanthology.org/// org/2020. emnlp-main, 185
Sai Prasanna, Anna Rogers, and Anna Rumshisky. When BERT Plays the Lottery, All Tickets Are Winning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3208 3229, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1 wtw1022...mmmm////1 ewnle-mainnn55555555-- 259. URL httpss//aalllntnooooy.orr/. www.ey...rrr/02002 ott.:/aaaaooooggooooooo/omm//o/a emnlp-main. 259
Doina Precup. Temporal Abstraction in Reinforcement Learning. PhD thesis, University of Massachusetts Amherst, 2000. URL
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Neil Houlsby. From sparse to soft mixtures of experts, 2023.
Nasim Rahaman, Muhammad Waleed Gondal, Shruti Joshi, Peter V. Gehler, Yoshua Bengio, Francesco Locatello, and Bernhard Schölkopf. Dynamic inference with neural interpreters. In Advances in Newral Information Processing Systems 94: Annual Conference on Neural Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 109988 2021. URL https://procedingsseeeue neurips
Samyam Rajbhandari,, Conglong Li, Zhewei Yao, Minjia Zhang, Reza Yazdani Aminabadi, Ammar Ahmad Awan, Jeff Rasley, and Yuxiong Hc. Deopspoed-moe: Advancing mixture-of-cxperts inforence and training to power next- sent-geeeratinn AI scale. In International Conference on Machine Learning, 1CML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp- 18332 18346. PMLR, 2022. URL intpps://proceedings. mlr. https///w12//raaaaddrriiaa/
Janarthanan Rajendran, Aravind S. Lakshminarayanan, Mitesh M. Khapra, P. Prasanna, and Balaraman Ravindran. Attend, adapt and transfer: Attentive deep architecture for adaptive transfer from multiple sources in the same domain. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview net, 2017. URL https: ///oeereviee.ne//oormmltassssss
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Informa- tion Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 506 516, 2017. URL https //proccedings.neuriip wttp:cccaapee//1///aaaaae 1atpppprr/2217/hhshh////2244 http:///e.....eaeeeeeeea2222aaa0 batpaa/rgeccccccabbabaaat....... Abstract..ttttrraaaaaaaa
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi-domain deep neural networks. In 2018 IEEE Conference on Computer Vision and Pattern Recogrition, CVPR 2018, Solt Lake City, UT, USA, June 18-22, 2018 pp. 8119 8127. Computer Vision Foundation / IEEE Computer Society, 2018. doi: 10. 10.1109/CVPR.22 2018 00847. URL http:///peenaceess. thecvf .cmm/conteentt___pp_2011
CVPR 2018 20PP__0111__pper.hhtt
Scott E. Reed and Nando de Freitas. Neural programmerrinterppeeess... In 4th International Conference on Learning Representations, ICLR 2016, San Juan Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL
Scott E. Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Hoess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. CoRR, abs/ /2205.06175, 2022. doi: 10.48550/arXiv wtw.iv.225000000 006175. URL https:////o....... org/10. 48550/arXix 220.202222111
66

---

Published in Transactions on Machine Learning Research (11/2023)
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. In Advances in Newral Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 8583-8595, 2021. URL httpps:/pproceednngs... neuri ips cc/paper/ /221 http://6200aaa/22222772222222222 2acaaaaaa222222264622033aaarrr..
Stephen Roller, Sainbayar Sukhbaatar, Arthur Szlam, and Jason Weston. Hash layers for large sparse models. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information
Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 17555-17566, 2021. URL https: //proceedings.neut iss.
2021 httpa////2//aaaaf444400002666660
Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection functions for multi-task learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 8, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL
selection of non-linear
Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. CoRR, abs/1904.12774, 2019. URL http://arxxv.org// 1904. 12774
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Tryna Gurevych. AdapterDrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7930-7946, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: (1118553v112022 emmnl- main. 626. URL.
eem.l-mman. 626
Sebastian Ruder, An overview of multi-task learning in deep neural networks. CoRR, abs/ 1101600000.99 2017. URL http://arxiv.orrt wrwpiivworggbbbb. hbtgg//bb/776... 05098
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Sogaard. Latent multi-task architecture learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 4822 4829, 2019a. URL
Scbastian Rudor, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapler of the Association for Computational Linguisties: Tutorials, pp. 15 18, Minneapolis, Minnesota, June 2019b. Association for Computational Linguistics. doi: 10. (0.18653/v1 500440044 URL https://aclaanthology/ wrwp:/oyy.oo////---500
Scbastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjic Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10215-10245, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1 www.000eenlmmma.......mm 802. URL https:
//aclanthologg.orr///0// org/2221. mnw.m//.......mmmmma............ 802
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukeuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. CoRR, abs/ 1606.04671,
2016. URL http://aaxiv.org/ata/ 0ates//1666004661
Victor Sanh, Thomas Wolf, and Sebastian Ruder. hierarchical multi-task approach for learning embeddings from semantic tasks. In The Thirty- Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Shcrrt--Firse Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium Educational Advances in Artificial Intelligence, FAAT 2019, Honolulu, Hawaii, USA, January 27- February 1, 2019, pp. 6949 6956. AAAI Press, 2019. doi: 10.1609/ (3009//aai.v33001..33. http:://oio.org////// 1009//aai..3330 333101333111131199999
URL
Victor Sanh, Thomas Wolf, and Alexander M. Rush. Movement pruning: Adaptive sparsity by finetnnnig. In Advonces in Newral Information Processing Systems 39: Annuaal Conference on Neural Information
67

---

Published in Transactions on Machine Learning Research (11/2023)
Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedding
html
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit. Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. Multitask prompted training enables wero-shot task generalization. In The http://arxxxvooo//////111
Tenth International Conference on Learning Representations, 2022. URL 08207.pdt
Kanthashree Mysore Sathyendra, Thejaswi Muniyappa, Feng-Ju Chang, Jing Liu, Jinru Su, Grant P. Strimel, Athanasios Mouchtaris, and Siegfried Kunzmann. Contextual adapters for personalized speech recognition in neural transducers. In IEFE International Conference Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pp. 8537 8541, 2022. URL https:
//doi.org/T0. 010.0001100//AASP45522222 2022. 9746126 26
Danielle Saunders. Domain adaptation and multi-domain adaptation for neural machine translation: A survey. Journal of Artificial Intelligence Research, 75:351 424, 2022. doi: 10.166//jarr..133555 URL
https://6ol.org/100 1613/jair...... 1. 13566
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguisties: Main Volume, pp. 255-269, Online, April 2021a. Association for Computational Linguistics. doi: 10. 10.18653/v1, htt./222..aallllnn
20. URL
org. eacl-main. 20
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North Americon Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2339 2352, Online, June 2021b. Association for Computational Linguistics. doi: 10. 10.18653/v1, /011///221.aa/// maacl-mmin..8800 URL httpss://aclanthology org/
2021 w.n.aallmaaaa 185)
Timo Schick, Sahana Udupa, and Hinrich Schütze. Self diagnosis and self-debiasing: A proposal for reducing corpus-based bias in NLP. Trans. Assoc Comput. Linguistics, 9:1408-1424, 2021. doi: 10.1162/tacl,, aa) 00434. URL httpp:://di..orgg org/10.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris M. Mooij. On causal and anticausal learning. In Proceedings of the 29th International Conference on Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26- -July 1, 2012. icmlec / Omnipress, 2012. URL
Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 10965):612-663 2021. doi: N0.1100/JRRO..2001333555
URL https: //doi. http::///ooooor////// 01011/0..111... 1109/ 2021. 3058954
Karin Kipper Schuler. VerbNet: A broud-coverage, comprehensive verb lericon. PhD thesis, University of Pennsylvania, 2005. URL
Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green AI. Communications of the ACM, 63 (12):54-63 63, 2020. URL https:://oi.org/// 1145/338183
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outragoously large neural networks: The sparsely-gated mixturo-of-experts layer. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.nor net, 2017. http://ttss//peeeeeevveeeerreeon
68

---

Published in Transactions on Machine Learning Research (11/2023)
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, Tu Vu, Yuexin Wu, Wuyang Chen, Albert Webson, Yunxuan Li, Vincent. Zhao, Hongkun Yu, Kurt Keutzer, Trevor Darrell, and Denny Zhou. ofxture-of-experts meets instruction tuning; winning combination for large language models, 2023a.
Yikang Shen, Zheyu Zhang, Tianyou Cao, Shawn Tan, Zhenfang Chen, and Chuang Gan. Moduleformer: Modularity emerges from mixture--o-expeets,, 2023b.
Froda Shi, Mirac Suzgun, Markus Freitag, Xuczhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language Models are Multilingual Chain-of-Thoughe Reasoners. In Proceedings of TCL. 2023, 2023. URL http://///// hwtpsivv..rg///////////bes
wwt./rr////////e/ 22///20/s/22/..////////// 03057
Hava T. Siegelmann and Eduardo D. Sontag. On the computational power of neural nets. Journal of Computer and System Sciences, 50(1):132-150, 1995. doi: 10.1006/jess.1995. 1995.101... URL https://doi..nn 10. 10.1006/jcss. 1995. 1013
Anders Søgaard and Yoav Goldberg. Deep multi-task learning with low level tasks supervised at lower layers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 231-235, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 10.18653/v1 P16-2038. URL. https://accanntnlooo...rgg//
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 1444-445) 2017. URL httphhttttaaiirrrrrrerrr wtt.:aaaaa....oo/iiiiaaaaeeeeaae httg///c//nndx.../// wwwpieep...pp.A.....mp /AAI/ AAAI17/paper /view/ 14972
An open multilingual graph of general
Karolina Stanczak, Edoardo Ponti, Lucas Torroba Hemnigen, Ryan Cotterell, and Isabelle Augenstein. Same neurons, different languages: Probing morphosyntax multilingual pre-trained models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1589 1598, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v2 / 22222 maaaaclmaai.1144. URL https::/accannthooog..orrr
naacl
114
Asa Cooper Stickland and Tain Murray. BERT and PALs: Projected attention layers for efficient adaptation in multi-task learning, In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 5986-5995. PMLR, 2019. URL httt:: http://proceedings..tt .mr.imss............ pttpsss/v7/sttikaaaanaeee
http:///ccccciisss..pprrs//////s 1aa.dd99ahttmm.
Asa Cooper Stickland, Alexandre Berard, and Vassilina Nikoulina. Multilingual domain adaptation for NMT: decoupling language and domain information with adapters. In Loic Barrault, Ondrej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkicwiez, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno-Yepes, Philipp Koehn, Tom Koemi, André Martins, Makoto Morishita, and Christof Monz (eds.), Proceedings of the Sizth Conference on Machine Translation, WMT@EMNLP 2021, Online Event, Nowember 10-11, 2021, pp. 578-598. Association for Computational Linguisties, 2021. URL https://aclantholoog org/ 20tgg2221..22 wmt 1.64.
Gjorgji Strezoski, Nanne van Noord, and Marcel Worring. Many task learning with task routing. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 1375-1384. IEEE, 2019. doi: 10.1109/ICCV.2010.0 2019.00014. URL https: ottps:://6i..or//e..11111 1109///. 2019. 00019.00145 00146.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Avnual Meeting of the Association for Computational Linguistics, pp. 3645- 3650, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 1.18653/v1/P111 1355..
URL
69

---

Published in Transactions on Machine Learning Research (11/2023)
Tianxiang Sun, Yunfan Shao, Xiaonan Li, Pengfei Liu, Hang Yan, Xipeng Qiu, and Xuanjing Huang. Learning sparse sharing architectures for multiple tasks. In The
AAAT Conference on Artificial Intelligence, AAAI 2020, The Thirty-Secome Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAT 2020, New York NY, USA, February 7-12, 2020, pp. 8936 8943. AAAI Press, 2020a. URL https://ojs. www.ssaaaai.... arg. index. php/ /wt.///ttttee//iie//////
Tianxiang Sun, Zhengfu He, Hong Qian, Xuanjing Huang, and Xipeng Qiu. BBTv2: pure black- optimization can be comparable to gradicnt descent for few-shot learning. CoRR, abs/2205.1120) 2022.
URL https::/doi.org:: 48550/ 2205. 11200
Ximeng Sun, Rameswar Panda, Rogério Feris, and Kate Saenko. AdaShare: Learning what to share for efficient deep multi-task learning. In Advances in Neural Information Processing Sys- tems 39: Annual Conference on Newral Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020b. URL https::/proceedingseeut http://eeeeenss..eerrssccc//a//p 334841868314644440077777 2000444000772550073333333 307//t5-bbbbttttttttt Abstract.html
Yi-Lin Sung, Varun Nair, and Colin Raffel. Training neural networks with fixed sparse masks. In Advances in Newral Information Processing Systems 34: Annual Conference on Newral Information Processing Systems
2021, NewrIPS 2021, December 6- 64,4, 2021, virtual, pp. 24193 -42205, 2021. URL https://proceeddings sourips.cc/appee http/aaper/22//has/t// //t////haaacccc223/55///// /ttaaa///b0022344444888888888611 (009999888b5166788888 hat.:/8551666.8.055---arrra.... Aastsstttttrratthhhat html
Yi-Lin Sung, Jacmin Cho, and Mohit Bansal. VL-ADAPTER: parameter-effiiiet transfer learning for vision- and-language tasks. In IEEE/CVE CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 5217-5211 IEFE, 2022. doi: 10.1109/CVVPR3:0888222222. 201120000880222001011
URL https://doi org/10. 1099///55266888.... 2022. 00516
Richard S. Sutton. Two problems with back propagation and other steepest descent learning procedures for networks. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, 1986, pp. 823 802, 1986. URL https://cir... mii. ac. pptp/////////2/99999555535333331
Richard S. Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1-2):181-211 211, 1999. doi: URL https:///oi..rr//// /ttpg//0011111/00000330000000000
Pawel Swietojanski, Jinyu Li, and Steve Renals. Learning hidden unit contributions for unsupervised acoustic model adaptation. JEEE ACM Transactions on Audio, Speech, and Language Processing, 24(8):1450-1499 2016. doi: 10.1109/TASIA, 201.........22555.... URL http:::://oiiorrg////// 110.00111999ATAP.P. 2016. 2560534
Bethan Thomas, Samuel Kessler, and Salah Karout. Efficient adapter transfer of self-supervised speech models for automatic speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pp. 7102-7106. TEFE, 2022.
URL
9742222
Brian Thompson, Huda Khayrallah, Autonios Anastasopoulos, Arya D. McCarthy, Kevin Duh, Rebecca Marvin, Paul McNamee, Jeremy Gwinmup, Tim Anderson, and Philipp Koehn. Freezing subnetworks analyze domain adaptation in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 124-132, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10. 286533/11/1886630 URL
Katrin Tomanck, Vicky Zayats, Dirk Padfield, Kara Vaillancourt, and Fadi Biadsy. Residual adapters for parameter-offcient ASR adaptation to atypical and accented speech. In Proceedings of the 2021 Conference on Empérical Methods in Natural Language Processing, pp. 6751-6760, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. URL https:
//aclanthologgyorg/// wtwp://er////.....m..m.m...
541
Andrew Trask, Felix Hill, Scott E Reed, Jack Rae, Chris Dyer, and Phil Blunsom. Neural arithmetic logic units. Advances in newral information processing systems, 31, 2018.
70

---

Published in Transactions on Machine Learning Research (11/2023)
Karl Ulrich. The role of product architecture in the manufacturing firm. Rescorch Policy, 24(3) 4190:449-440, 1995. doi: (tt.//o.orr/111.101/0000000 (0000/0000000733334000777755555 URT. https://doi...... org/10. 1016/ 00555/00488111 (0388030-3333444))))) 00775-3
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, and Gertjan van Noord. UDapter:
Language adaptation for truly Universal Dependency parsing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2302-2315, Online, November 2020. Association for Computational 180. URL https::/aclanthhllo0 wwwpaannthooooooooooo/eeooeee/ org/2020. emmlp-main
Linguistics. doi:
180
Ahmet Üstün, Alexandre Berard, Laurent Besacier, and Matthias Gallé. Multilingual unsupervised neural machine translation with denoising adapters. In Proceedings of the 2021 Conference on Empérical Methods in Natural Language Processing, pp. 6650 6662, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. htt/://1112211mm//// //aclanthology.or/// org//2g1..22222 www.2...mmpmmmmi..533333
URL https
Ahmet Üstün, Arianna Bisazza, Gosse Bouma, Gertjan van Noord, and Sebastian Ruder. Hyper-X: A unified hypernetwork for multi-task multilingual transfer. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 7934-7949, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL
orgph/oooog...og2222222mmemm/aaa 541
Gido M. van de Ven and Andreas S. Tolias. Three scenarios for contimual learning. CoRR, abs/ htss/994.07734, 2019. URT. http:://rxxxi wttp/aaaiiii..gga////b////./ 1970//3194000734
Simon Vandenhende, Stamatios Gcorgoulis, Luc Van Gool, and Bert De Brabandere. Branched multi-task networks: Deciding what layers to share. In 31st Brilish Machine Vision Conference 2020, BMVC 2020, Virtual Event, UK, September 7-10, 2020. BMVA Press, 2020. URL https: /ttp://bmmc2220--offreennnn
commassetss/ppers/00113 pdff
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Tn Advances Neurul Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998 6008, 2017. URL #f5e222554/dee9 aatteeerrracsscaaaaaaaaasaaaaaaa
Petar Veličkovié and Charles Blundell. Neural algorithmic reasoning. Patterns, 2(7):100273, 2021.
Pat Verga, Haitian Sun, Livio Baldini Soares, and William Cohen, Adaptable and interpretable neural MemoryOver symbolic knowledge. In Proceediiggs of the 2021 Conference of the North American Chapler of the Association for Computational Linguistics: Human Language Technologies, pp. 3678 3678-3691, Online, June 2021. Association for Computational Linguistics. doi: 1118653/v112221.naaccl httpss///cclanthoogy..ooo
2888 URL
naacl-main. 288
David Vilar. Learning hidden unit contribution for adapting neural machine translation models. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 500 505, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: (.118833/v//NT 2080. URL https://acllantholegg oorg
N18- N18-2082]
Elena Voita, David Talbot, Fodor Moiscev, Rico Sennrich, and Ivan Titov. Analyzing multi-hoad self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 0797-5808, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/vl 11/119-1500.. URL httpp:://cllanthology
John von Neumann. First draft of a report on the EDVAC. Technical report, University of Pennsylvania, 1945. URL httph:://oi..orrggtpe 1109/85. 238389
71

---

http//2222..nnnss-asee Published in Transactions on Machine Learning Research (11/2023)
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. In 8th International Conference on Learning Representations, TCLR 2020, Addis Abala, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit lyyer, and Noah Constant. Overcoming calastrophic forgetting in zero-shot cross-lingual generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9279-9300, Abu Dhabi, United Arab Emirates, December 2022a.
Association for Computational Linguistics. URL https:://clanthooogg..o wrg. wtt.:///...e...mmmmai..... 630
Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit lyyer, and Noah Constant. Overcoming catastrophic forgetting in zero-shot cross-lingual generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp.
Abu Dhabi, United Arab Emiirates, December 2022b. Association for Computational Linguistics. URL https://aclanthoologyooo org
emnlp-maing mamllp-main..mm 630
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Votume 1: Long Papers), pp. 5039 5059, Dublin, Ireland, May 2022c. Association for Computational Linguistics. doi: 10.18653/vv (118633/1120222... //22//12.22222llllll/ moll-loo.3.6. URL
acl long. 346
Günter P. Wagner, Jason Mezey, and Raffaele Calabretta. Natural selection and the origin of modules. In Werner Callebaut and Diego Rasskin-Gutman (eds.), Modularily: Underslanding the Development and Evolution of Compler Natural Systems, pp. 33. MIT Press, 2005. doi: https://doi.org: 10.755111//1 /mitpress/ 4734.001.0001. URL h/ttps://doi...o /ttp:///oioorg/1.... 7551/mitppress44777 0t/1///434........... 0001
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language un- derstanding systems. In Advonces in Neural Information Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Van- cowver, BC, Canada, pp. 3261-3275, 2019. URL https://proceedngs.eerrr wtwp://gg...urrs..cccceeoeoose pwtpcppppe///////aassiii html
Pu Wang and Hugo Van hamme. Bottleneck low-rank transformers for low-resource spoken language understanding. In Proceedings of Interspeech 2022, 23rd Annual Conference of the International Speech Communication Association, Incheon, Korea, 18- 18-22 September 2022, pp. 1252, 2022. doi: 10.21437/ / Interspeech.2022 2022 10801. URL hattps://doi.org// 000. 21137//nterrspeeeeee 10821
Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou, K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters. In Findings of the Association for Computational Linguistics: ACI-IJCNLP 2021, pp. 1405-1418, Online, August 2021a. Association for Computational Linguistics. doi: 10. https::/aclanthhtn htw.:/oow.orrr222...nnnnnnnn---e findings--cl.1121
URL
Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. AdaMix: Mixture-of-adaplatioon for parameter-efficent model luning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5744-5760, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: //aclanthology. orgoor///// /02.. www..2emmmmmm....mmmmm
Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. On negative interference in multilingual models: Findings and meta-learning treatment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4438 4450, Online, November 2020. Association for Computational Linguistics. doi: 10118655/112002..o. 359.
main.359. URL h hwwpaaantooooooorgg/////eeeee org/2020. emnlp-main
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In 9th International Conference on Learning Representattons, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021b. URL
72

---

Published in Transactions on Machine Learning Research (11/2023)
Chihiro Watanabe. Interpreting layered neural networks via hicrarchical modular representation. In Neural Information Processing- 26th International Conference, TCONIP 2019, Sydney, NSW, Australia, December 12-15, 2019, Proceedings, Part V, volume 1143 of Communications in Computer and Information Science, pp. 376-388. Springer, 2019. doi: (00-000797883-00--300 103----00-3680-2
40. URL attps://ddos /ttps://dbi.orr/1e.. 1007/
Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their prompts? In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2300 2344, Seattle, United States, July 2022. Association .rrg/
for Computational Linguistics. doi: 10.18653/v1 /2///222.2aaa.. 167. URL
2022. 3022.naac1-maa -aann. 167
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a. URL
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. Finetuned language models are zero-shot learners. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25- 25, 2022. OpenReview. net, 2022b. URL http:://pppeeeeeeeenee///rrrrrrp
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022c.
Ronald J. Williams. Toward a theory of reinforcement-lerning connectionist. systems. Technical Report NU-CCS-88 88-3, Northeustern Universily, 1988.
Ronald J. Williams. Simple statistical gradient-folloingg algorithms for connectionist reinforcement learning. Machine Learning, 8(3):229-255) 1992. doi: N0.1007/BF00992680 URL https://doi.on http://oii..or//..0000000 BF00992696]
Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin, Zihan Liu, and Pascale Fung. Lightweight and efficient end-to-end speech recognition using low-rank transformer. In 2020 IEFE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, pp. 6144 6148, 2020. URL https::/dol: 0rg/10.1110 T109/CASSP48776... 2020. 9053878.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Te Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Nalural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020, Association for Computational Linguistics. doi: 10.18653/v1, vv/65///1/2200000 1et.200emmllp-deeee 6. URL https://aclanthollee htwp:/onnnnoooooooo//2222 www.:0eemmpppppemmm emn1p-demos.e 6
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi, Mohammad Rastegari, Ja- son Yosinski, and Ali Farhadi. Supermasks in superposition. In Advances in Neural Information Processing Systems 33: Annual Conference Newral Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL (at.r//22cccccceecabbbbbbbaaaaaa
html
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongscok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (ods.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 23965 23998, PMLR, 2022. URL mlr http://pooedddnn.....rrrrrrrs///
html
73

---

Published in Transactions on Machine Learning Research (11/2023)
Teng Xi, Yifan Sun, Deli Yu, Bi Li, Nan Peng, Gang Zhang, Xinyu Zhang, Zhigang Wang, Jinwen Chen, Jian Wang, Lufei Liu, Haocheng Feng, Junyu Han, Jingtuo Liu, Errui Ding, and Jingdong Wang. UFO: unified feature optimization. In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner (eds.), Computer Vision ECCV 2022 17th European Conference, Tel Avie, Isruel, October 23-27, 2022, Proceedings, Purl XXVI, volume 13686 of Leclure Noles in Computer Science, pp. 472-482844888 Springer, 2022. doi: 00.1007/976-3-031-19544-4 27. URL https://doi. http://oooorrg////// 978-3-931-19809-002
Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving interference when merging models, 2023.
An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, Di Zhang, Wei Lin, Lin Qu, Jingren Zhou, and Hongxia Yang. Exploring sparse expert models and beyond. CoRR, abs abs/2105.15082, 2021. URL WRLLhttps:// https://arxxi..rr/bbbbtt./
15082
Guangyu Robert Yang, Madhura R. Jogickar, H. Francis Song, William T. Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature Neuroscience, 22(2):297-3066 306, 2019. $41593-018-0310-2
doi: https://doia https:/doi..rg///// 100. 10088//41508-008-0001000 URL https://66i.orr//1//
Yongxin Yang and Timothy M. Hospedales. Deep multi-task representation learning: A tensor factorisation approach. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL. https:://openeevii www.wewwrettrrrm id=Skhu2fc11
Michihiro Yasunaga, Armen Aghajanyan, Weijia Shi, Richard James, Jure Leskovec, Percy Liang, Mike Lewis, Luke Zettlemoyer, and Wen-Tar Yih. Retrieval-augmented multimodal language modeling. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedinggs of Machine Learning Research, pp. 39755 39769. PMLR, 2023. URL html
https://prooceednng..ltt mlr wttp://nnnssm....ss/e/ee.//o/// vrress//v022/aassnaaa
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. CrossFit: A few-shot learning challenge for cross-tast generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7163 7189, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: N..18653/v112022 htt.//222eeemmlmmmmeeee www.lpmman....... 572. URL https:://clantholll org/
2021 wemnl--mainn..7 572
Zhao You, Shulin Feng, Dan Su, and Dong Yu. Speechmoe2: Mixture-ofeexperss model with improved routing. IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singupore, 23-27 May 2022, pp. 7217-7221. IEEE, 2022. doi: 10.110//CASSPP4322222222 2022 URL
https://doi.org http:///ooo.oo////// 10.1101...11 10011119//CCA 00009//CASS449222 2022. 9747065
Haonan Yu, Sergey Edunov, Yuandong Tian, and Ari S. Morcos. Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.nug net, 2020. URL https::/openre// net/forum?id-S1xmne XXXXVFFF
Zichun Yu, Chenyan Xiong, Shi Yu, and Zhiyuan Liu. Augmentation-adagy retriever improves gen- cralization of language models generic plug-in. In Proceedings of the 61st Annual Meeting of the Association for Compulational Linguisties (Volume 1: Long Pupers), pp. 2421 2436, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ v1 /120://////223..looo....... https::/aclanthollee http://cccanttooooooor////2222. ac1023.ac1--0 acl- long. 136.
URL
Fan Zhang, Duyu Tang, Yong Dai, Cong Zhou, Shuangzhi Wu,, and Shuming Shi. SkilINet-NLU: A Sparsoly Activated Model for General-Purpoot Natural Language Understanding. CoRR, abw.2223.00333333 033112, 2022a. doi: 10.48550/aam (18550/arxxv.222 www.rr..22303331111
URL httpps//aariiv.orr/b////2...1.
74

---

Published in Transactions on Machine Learning Research (11/2023)
Rongsheng Zhang, Yinhe Zheng, Xiaoxi Mao, and Minlie Huang. Unsupervised domain adaptation with adapter. CoRR, abs/2111 222110000667 2021. URL http:///aaxxii.ooaaa////////////
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task lcarning. In Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI, volume 8694 of Lecture Notes in Computer Science, pp. 94-108.. Springer, 2014. doi:
URL https://doi .rg./.r../.. 1007/978 3-0008---311----99----1
Zhengyan Zhang, Zhiyuan Zeng, Yankai Lin, Chaojun Xiao, Xu Han, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Emergent modularity in pre-trained (ransformers, 2022b. URL http:://openrevieewett//eee id=XHuQacT6sa6
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. Masking as an efficient alternative to finetuning for pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLPP), pp. 2226 2241, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1120000emmii eanlppmmmmllaaaaa main. 174. URL https://aclanthoology / 2020. 1020.emnlp-mmm main. 174
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp.
12697-12706 PMLR, 2021. URL http:///pooceedings. mlr
33t pteess/v13//hha2211 html
Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang, Yuanhao Yu, and Jin Tang. Meta-dmoe: Adapting to domain shift by meta-distillation from mixturc-of-cxperts. CoRR, abs/2210.03885,, 2022. URL https
//doi.org/10 00851/..50/1..//////// 48550/ 2210.03885
Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 5017-5033, Online, June 2021. Association for Computational Linguistics. doi: 10. 118653/v1202211 1 mainc-mmin.39... URL https://aclanthologgy org/ 2021 1021.naacl-maing 398
Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. AutoPEFT: automatic configuration search for parameter-efficien fine-tuming. CoRR, abss2300.112 2023. doi: 40w50///r.i..001....... URL https://doi.org: wtw.://///...../////// 20.54885//aa00
aar/0.//riv.... arXiv.2201... 12132
Hattic Zhou, Janice Lan, Rosanne Liu, and Jason Yosinski. Deconstructing lottery tickets: Zeros, signs, and the supermask. In Advances in Neural Information Processing Systems 32: Anneual Con- ference Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Van- couver, BC, Canado, pp. 3592 3602, 2019. URL Abstrrctt..bttttttt...... html
Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pp. 16795 16804, 2022a. URL CVPR52688 2022.01621
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Y. Zhao, Andrew M. Dai, Zhifeng Chon, Quoc Lc, and James Laudon. Mixture-of-cxperts with expert choice routing. CoRR, Abss/222.0092 2022b. URL httpss///arxi.. httpss//aaxiv.orr//aa/e wtgpobr///ab..s..//se 09368.
Yunzheng Zhu, Ruchao Fan, and Abeer Alwan. Towards better for kindergarten-agedd speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2022, Virtual and Singapore, 23-27 May 2022, pp. 1582-8588 2022. URL
with task augmentation
75

---

Published in Transactions on Machine Learning Research (11/2023)
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. ST-MoE: designing stable and transferable sparse expert models. CoRR, 2bs/2202.08906,, 2022. doi:
10.48550/ 1008550/ARXI.22000 2202.08906. URL
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Jianfeng Gao, and Tuo Zhao. Taming sparsely activated transformer with stochastic experts. In The Tenth International Conference on Learning Represenlations, ICLR 2022, Virlual Evend, April 25-29, 2022. OpenReview.nel, 2022. URL
76


Reviews:
[{"review_id": "txuiU9WWKq", "reviewer": "Reviewer_oaBm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The taxonomy of modular networks is well-defined, with each part explained thoroughly and insightful connections made between them. The authors also shed light on the relationship between hyper networks and these four parts, enhancing understanding.\n- It provides a wide-ranging perspective on modular applications like Hierarchical Reinforcement Learning, Programme Simulation, and Causal Discovery and Inference, going beyond common tasks such as Transfer Learning.\n- The use of simple yet intuitive examples is commendable. For example, in 7.1.1, the Machine Translation, the authors demonstrate the power of modularity through multilingual NMT adapters.\n- The authors also offer suitable suggestions for utilizing and implementing modular networks. They emphasize that while modularity is a good principle, the level and scope of modularity should be considered accordingly (as highlighted in sections 4.3 and 7.1.1).\n\n## Weaknesses\n\n- It would be helpful to include a catalog at the beginning, allowing readers to easily refer to specific parts and form a hierarchical understanding of the work.\n- In the Function Composition section (3.3), the reviewer suggests discussing semi-parametric modules [1] [2], which extend the base model with additional components like knowledge bases. Moreover, semi-parametric modules like retrievers also relate to Knowledge Injection in section 7.1.4.\n- The Soft Learned Routing section (4.2.3) could benefit from more comprehensive coverage. While different levels of routing and auxiliary information are addressed, it would be valuable to discuss loss [3] and architecture designs [4] that facilitate learning to route while encouraging module specialization (as mentioned briefly in 4.2.1).\n\n[1] Verga, P., Sun, H., Soares, L. B., & Cohen, W. (2021, June). Adaptable and interpretable neural memory over symbolic knowledge. In _Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: human language technologies_ (pp. 3678-3691).\n\n[2] Cheng, X., Lin, Y., Chen, X., Zhao, D., & Yan, R. (2023). Decouple knowledge from paramters for plug-and-play language modeling. _arXiv preprint arXiv:2305.11564_.\n\n[3] Shen, Y., Zhang, Z., Cao, T., Tan, S., Chen, Z., & Gan, C. (2023). ModuleFormer: Learning Modular Large Language Models From Uncurated Data. _arXiv preprint arXiv:2306.04640_.\n\n[4] Chi, Z., Dong, L., Huang, S., Dai, D., Ma, S., Patra, B., ... & Wei, F. (2022). On the representation collapse of sparse mixture of experts. _Advances in Neural Information Processing Systems_, _35_, 34600-34613.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This comprehensive study delves into modular designs in deep neural networks, providing a systematic taxonomy of different components and offering a broad view of their applications. The authors present detailed comparisons between various methods, suggestions for modular designs, and insightful future directions.\n\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "e2PdZZmTv7", "reviewer": "Reviewer_eFvt", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* I think this paper will be really helpful for the wider community to understand a burgeoning research direction.\n* The paper is very well written\n* The unification of different techniques under the same functional form is quite insightful!\n* The covered content is vast in scope, and quite a comprehensive survey.\n\nWeaknesses:\nAll the following weaknesses will only make the paper better and are not critical for securing my recommendation for acceptance.\n\n* The authors could mention BASE layers (Lewis et al 2021) as a specific technique to prevent expert collapse in Section 4.2.1\n* The authors could mention Shen et. al 2023 as another mechnanism to improve MoE finetuning approaches in Section 4.2.1\n* There is an interplay between aggregation techniques and sparsity which is not mentioned; if the modular model is sparse, ie only a few experts are active at inference time, one can get away with fewer active modules with ensembling at the same inference costs as other techniques like merging.\n* The authors could compare the inference costs of various aggregation techniques, ie that representation aggregation is necessarily more expensive than parameter aggregation (the latter of which reduces the cost of inference to a single model), but that representation aggregation techniques tend to outperform parameter averaging. \n* I think the paper sections would benefit from a similar summary box that you have at the end of the paper, since the content is so detailed it can be hard to keep track of the general takeaways from each section.\n* At each section, I would appreciate tables that the presented presented techniques across the main axes explored (performance inference costs, etc) \n* Compositionality is only briefly mentioned, but I think a key direction going forward. Can we compose different skills from specialized models towards new tasks? Joel et al 2023 suggests that it might be possible.\n* Additional merging techniques that could be mentioned:\n- dataless knowledge fusion: https://arxiv.org/abs/2212.09849\n- TIES merging - https://arxiv.org/abs/2306.01708\n- stochastic weight averaging - https://arxiv.org/abs/2001.02312\n- model fusing - https://arxiv.org/abs/2204.03044\n\nAdditional references:\nLewis et al 2021 - ​​https://arxiv.org/abs/2103.16716\nShen et al 2023 - https://arxiv.org/abs/2305.14705\nJang et al 2023 - https://arxiv.org/abs/2302.03202\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper\n* Presents a survey of modular deep learning techniques in the literature \n* Unifies a variety of techniques under a single function\n* Compares different modular techniques in terms of axes like parameter efficiency, training efficiency and inference efficiency\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See above.", "broader_impact_concerns": "I don't think there are any ethical concerns with this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UOD2C9gGSV", "reviewer": "Reviewer_5FSf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nOne of the major strengths of the paper is the way the authors unify the many methods of modular deep learning under a single framework. The fact that a single, relatively simple algorithm (Algorithm 1) can describe all these methods is surprising, but it also provides a refreshing and clarifying view of these methods. \n\nThe paper is very well written. The authors have done a wonderful job of clearly and carefully describing each method proposed in recent years. Sections 1 - 6 are excellently structured. Figures are well illustrated. Table 2 is quite useful. The choice of notation is appropriate. The boxed conclusion section is helpful.\n\nOverall, I believe this paper could be a strong new reference point for researchers who want an overview of the field of modular deep learning.\n\n**Weaknesses**\n\nAs a review paper, the paper doesn't present any technically new content, but this is ok since it doesn't claim to.\n\nIn my view, one of the drawbacks of this paper is its length. While the paper is quite comprehensive, it may be more helpful for the reader to read shorter sections that illustrate the key principles rather than a longer section covering details more finely. The authors may wish to consider succinctly describing only 1-2 representative examples of each of the possible methods for each aspect of modularity (e.g. parameter composition for computation function) rather than providing a more comprehensive overview.\n\nI am also uncertain about the value of sections 7 and 8. The previous sections describe how modular deep learning methods are designed, and the structure of the sections appropriately reflects the different aspects of these methods. Sections 7 and 8 describe the usefulness of modular deep learning methods. Unfortunately, there seems to be relatively little underlying structure in how these methods are used, and the structure of these sections reflects this; at certain points, the sections seem to simply list different use cases. I would suggest significantly condensing these sections to briefly describe only a few key applications.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a review of modular deep learning. It presents a unified view of different modular deep learning methods, characterizing them by computation function, routing function, aggregation function, and training setting. For each of these, the authors provide descriptions of the current approaches in the field. Next, the authors discuss applications in transfer learning and other settings such as hierarchical reinforcement learning. Finally, the authors conclude and discuss next steps.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical**\n\nPage 21 typo:  \"in Modular Meta Learning,...\"\n\n\n**Would strengthen**\n\nPlease consider making the paper overall more concise (please see \"Strengths and Weaknesses\" for more details)\nPlease consider bold font for the + and - in Table 3", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers found the paper to be well-written and a valuable contribution. The authors have incorporated suggestions including additional discussion, a table of contents and summary boxes to further strengthen the paper.
Decision: Accept (poster)