Soft Merging of Experts with Adaptive Routing
Authors: Mohammed Muqeeth
Haokun Liu
Colin Raffel
Venue: Accepted by TMLR ()

Abstract: Neural networks that learn to route their inputs through different "expert" subnetworks provide a form of modularity that standard dense models lack. Despite their possible benefits, modular models with learned routing often underperform their parameter-matched dense counterparts as well as models that use non-learned heuristic routing strategies. In this paper, we hypothesize that these shortcomings stem from the gradient estimation techniques used to train modular models that use non-differentiable discrete routing decisions. To address this issue, we introduce $\textbf{S}$oft $\textbf{M}$erging of $\textbf{E}$xperts with $\textbf{A}$daptive $\textbf{R}$outing (SMEAR), which avoids discrete routing by using a single "merged" expert constructed via a weighted average of all of the experts' parameters. By routing activations through a single merged expert, SMEAR does not incur a significant increase in computational costs and enables standard gradient-based training. We empirically validate that models using SMEAR outperform models that route based on metadata or learn routing through gradient estimation. Furthermore, we provide qualitative analysis demonstrating that the experts learned via SMEAR exhibit a significant amount of specialization.

Published in Transactions on Machine Learning Research (05/2024)
Soft Merging of Experts with Adaptive Routing
Mohammed Muqeeth
meqeethh11 @gmaail com
University of North Carolina at Chopel Hill
Hoakun Liu
haokun/ku,112@gmaaii com
University of Toronto Vector Institute
Colin Raffel
craffel@gmail. com
University of Toronto Vector Institute
Reviewed on OpenReview: https: // openreview. forum? id- 7411991c5544
Abstract
Neural networks that learn to route their inputs through different "expert" subnetworks provide a form of modularity that standard dense models lack. Despite their possible benefits, modular models with learned routing often underperform their marameter-mateoee counterparts as well as models that use non-learned heuristic routing strategies. In this paper, we hypothesize that these shortcomings stem from the gradient estimation techniques used to train modular models that use non-differentiable discrete routing decisions. To address this issue, we introduce Soft Merging of Experts with Adaptive Routing (SMEAR), which avoids discrete routing by using single "merged" expert constructed via weighted average of all of the experts' parameters. By routing activations through single merged expert, SMEAR does not incur a significant increase in computational costs and enables standard gradient-based training. We empirically validate that models using SMEAR outperform models that route based on metadata or learn routing through gradient estimation. Furthermore, we provide qualitative analysis demonstrating that the experts learned via SMEAR exhibit significant amount of specialization. All of the code used in experiments is publicly available.'
dense
1 Introduction
Neural networks typically use all of their parameters to process a given input. As such, the capabilities of a model are distributed across the parameters of a model in a self-organizing way (Zeiler & Fergus, 2014; De Cao al., 2021: Csordás et al., 2021; Bau al., 2020: Wang et al., 2022a). Explicitly specializing different parts of model to different capabilities can provide various benefits, including reduced interference across downstream tasks (Sanh et al., 2021; Wei al., 2021; Zamir et al., 2018; Bao et al., 2021) languages (Pires et al., 2019; Liu et al., 2020b; Xue et al., 2020). Furthermore, dedicating specific parameters to specific capabilities enables a form of modularity where capability can be added, removed, or modified by adding, removing, or modifying the corresponding paramcters (Pfeiffer et al., 2023). Activating only a subset of the model's parameter for a given imput also decouples the computational cost a model from the number of parameters has (Shazeer al., 2017; Fedus et. al., 2021), though we do not focus on this benefit. in this paper.
Conditional computation techniques provide way to build models that adaptively choose subset of their parameters to apply to given input. A common way to use conditional computation in this setting is to introduce specialized subnetworks called experts that are controlled by rowters that decide which experts should active. When the model is trained on diverse data, this form of conditional computation can enable
htttps//githhu... com/r- three/smees

---

Published in Transactions on Machine Learning Research (05/2024)
Discrete Routing
SMEAR
Gradient estimation
Merged Expert
Expert
Router
Expert
Expert
Expert
Expert 2
Expert 3
Expert 4
Router
Expert 4
The discrete routing decisions commonly used in models that route activations among experts require the use of gradient estimation (left). We propose SMEAR (right), which uses a given router's distribution to average the parameters of the corresponding experts and then routes the input through a single merged expert. SMEAR achieves better performance than models with discrete routing, can be trained with standard backpropagation, and does not incur significant additional computational costs.
Figure 1:
modular learning by allowing experts to specialize to different types of inputs and flexibly share knowledge (Ma et al., 2019). However, because routing involves making a discrete decision as to which expert to use, the loss on the model's prediction cannot back propagae though the routing decision to update the router. Consequently, models with conditional computation often require gradient estimation techniques for training (Clark et al., 2022; Fedus et al., 2021; Bengio et al., 2013). In practice, past work has shown that models with conditional computation do not. always learn effective routing strategies. For example, Mittal et al. (2022) investigate models with continuous router in a controlled setting and find the models do not route examples from the same group to the same experts and perform poorly compared to models with oracle routing. However, models with lask- or domain-specific subnetworks (Gururangan et. al., 2021; Kudugunta et al., 2021) provide evidence that it is possible to train performant models with specialized experts. As an extreme example, Roller et al. (2021) achieves results comparable to learned routing with a fixed random routing. Relatedly, Fedus et al. (2021) find the gain from scaling up parameters by 30x with a sparsely activated model is smaller than scaling up both parameters and FLOPs by 3x in dense model. As possible explanation, Clark et al. (2022) study how models with conditional computation improve with scale and find detrimental term that scales with the product of the log number of experts and active parameters.
In this work, we hypothesize that issues with conditional computation stem from issues with gradient estimation. Specifically, we focus on experimental settings where we can compare learned routing to a performant hand-designed heuristic routing scheme. We find that the gradient estimation techniques we consider often produce models that underperform heuristic routing, despite the fact that they could in principle learn better routing strategy. To address this shortcoming, with Adaptive Routing (SMEAR), a method for training modular models with specialized experts and learned routing. Given an input, which may be a sequence of tokens or an image, SMEAR works by using the router's distribution of the input over experts to compute a weighted average of the parameters of the individual experts. The imput activations are then sent through the merged expert, which results in a similar computational cost to discrete routing with a single expert. However, the fact that all components of SMEAR are fully diffcrentiable enables standard gradient-based training. Empirically, we show that SMEAR significantly attains a favorable performance/ estimation, 2) heuristic routing schemes, and 3) toite--f-thheart baselines for learning modular models. We also qualitatively validate that the experts learned by SMEAR specialize to different types of inputs and share parameters across related tasks. Put together, our results show that SMEAR provides an effective alternative for modular models that use adaptive routing among expert subnetworks.
introduce Soft Merging of Exports
tradeoff to 1) discrete routing solutions found via gradient
After providing background on conditional computation models and gradient estimators in the following section, we define SMEAR in section We then describe our experimental findings in section 4, discuss related works in section 5, and conclude in section 6.
2

---

Published in Transactions on Machine Learning Research (05/2024)
2 Background
To provide the necessary background for our work, we first explain how sparsely activated neural networks use conditional computation, then discuss gradient estimators that enable learning discrete routing strategies. In addition, we discuss different ways to hand-design "heuristic" routing strategies as well as preexisting techniques for learning modular models that we use as baselines.
2.1 Routing Among Experts
In models that use discrete routing among experts (i.c. subnctworks), experts are organized into blocks that are incorporated as an intermediale layer in a neural network. An expert routing block B comprises a set of N experts {fr. f.. and a router R. Experts in the same block accept inputs and produce outputs of the same dimensionality. Given hidden-state representation u, the output of the i-th expert with parameters 0g is fi(u,0))
2.2 Gradient Estimators
In sparsely activated models that involve discrete adaptive routing, it is not possible to train the router's parameters with standard gradient-based learning. Fortunately, gradient estimators can provide approximate gradients to the router parameters. There are a few common designs shared by models that use gradient estimators to train routers. Their router R often applics lightweight network to some intermediate hidden states e in the model. The output of the light routing net work R(v) parameterizes a discrete probability distribution over the N experts. Different gradient estimators vary in how they make the routing decision from R(v) and how they construct the output from the chosen expert. Additionally, some estimators may introduce additional loss terms.
REINFORCE Gradients can be estimated through discrete operations using reinforcement learning techniques (Schulman et al., 2015; Bengio et al., 2013). In reinforcement learning, policy loss is used to train an agent to learn optimal actions in an environment. In this paper, we experiment with the REINFORCE algorithm which computes the policy loss as log(t)r where r denotes the received reward for taking an action whose assigned probability is л. When applied to models that use discrete routing among experts, the goal is to train the model to choose the optimal expert to process a given input. Here, the router R acts as an agent that samples an expert to use according to the routing probabilities. In order to train such a router, the router's assigned probability to the sampled expert is used æ and the negative of the model's loss is used as the reward The router is therefore trained to pick experts that maximize the reward which, in turn, minimizes the loss. The REINFORCE estimator often suffers from high variance because of the sampling operation. This motivates the use of baselines, which reduce variance without changing the optimal solution. In our work, we follow Clark et al. (2022) and use a baseline that is generated by a small neural network with a single hidden layer that takes as input e and is trained with the Huber loss. The overall loss function is then
L - Re log ((v) b) BR(v) log R(v) + LL
where a, 3, and y are hyperparameters that correspond to policy gradient weight, policy entropy weight, and value loss weight. In practice, we approximate the expectation with a single sample. During inference, the output of the block B is just f((u, 0:) where i= = arg max R(v).
Straight Through Gumbel-Softmax (ST-Gumbel) The Gumbel-Softmax trick (Jang et al., 2016; Maddison et al., 2016) provides a continuous differentiable approximation to sampling from a categorical distribution like the one parameterized by a router. Specifically, Gumbel noise is added to the logits of the distribution and temperature scale is applied in the softmax operation. Denoting 9 Gumbel(O, 1) and 7 the temperature, the Gumbel-Softmax trick produces the following modified distribution:
R(v); exp((log R R(v).) + 9))/)) exp((log R R(v))) + 9))
3

---

Published in Transactions on Machine Learning Research (05/2024)
The expert fa with the highest assigned probability is chosen by applying arg max operation. In order to approximate gradients through the arg max operation, we use the Strought estimator which replaces fi(u,0, 02) with (1 (1- [g[[]] + (u, 0.) where stands for the stop-gradient operator. During forward pass, the multiplier for fi(u, 8,) becomes and the multiplier reccives gradients for the term A(v); in the backward pass. In practice, the temperature T is gradually annealed from a high to low value so that the approximated samples are more and more similar to discrete samples. During inference, we choose an expert. according to arg max R(v).
Top-k Shazeer ct al. (2017) propose a gradient estimation scheme where the router sends the input through the & experts that are assigned the highest probability. Fedus et al. (2021) later found that this router could be used effectively when & = 1. Specifically, the estimator selects the subnetwork with the highest probability and scales its output using its corresponding routing probability. The output of the block is therefore ((v) where i = arg max R(v).
DSelect-k Hazimeh et al. (2021) proposed a differentiable approximation for the discrete Top-k operation. They parameterize each selection among N experts using m binary variables 21, 22.... 2m, produced using a learnable weight W as = -W),, where m = (Ng) The selector function r takes these variables and computes
r(z);
II (z)) JEB(i-1) JEL...m\(((-I))
II
2g-
where i E I,...,N and B()) returns the non-zero indices in the binary representation of the integer 1. A differentiable step function S based a cubic polynomial is used for the variables. Finally, each selector operation happens k times to get Top-k selection and an additional learnable parameter G provides a probability distribution over these i selections using the softmax fuction. addition, entropy regularization is applied to the output of the selector function r, cnsuring that it results in one-hot selection during inference. In our work, we consider
to maintain a computational cost similar to other baselines.
2.3 Heuristic Routing
As point of comparison for techniques that learn adaptive routing, we experiment with three baseline routing strategies that do not require a trained router.
Tag Routing If we have prior knowledge about the data that model will be applied to, can hand-design heuristic routing strategy for choosing which expert to use for given example based on data properties. Tag routing lakes advantage of "lags" associated with a given example (such as its domain or task) and associates each expert in an expert routing block with a particular tag. In this work, we assume each example has a single tag and route each example to its tag's expert.
Hash Routing Roller et al. (2021) propose hash routing, which uses a fixed hashing function to determine which expert to use for a given example. Specifically, each example assigned random expert choice in each expert routing block which used consistently over the course of training and inference. This approach disregards any shared characteristics across examples.
As an additional baseline, we consider models where all inputs are routed a single expert in each routing block. To provide fair comparison to models with N experts per block the basis of both computational cost or parameter count, number compute-matchedd referred to as "1x x compute") or Nx (parameter-matched, referred to as "1 x parameters") as many parameters as single expert.
consider models with a single expert with either the same
2.4 Methods for Learning Modular Models
Beyond the simple baselines discussed above, we consider three recently proposed methods that aim learn modular models.
4

---

Published in Transactions on Machine Learning Research (05/2024)
Adamix Adamix (Wang al., 2022b) uses random routing for each example during training and adds consistency loss to encourage experts to share information and discourage divergence. During inference, the parameters of all experts are averaged together to form a single expert and no adaptive routing is used.
Latent Skills Latent Skills (Ponti et al., 2022) assumess that the task for each example is known and trains a task-skill matrix that specifies which experts are active for given task. The binary task-skill matrix is fixed and learned via the Gumbel-Sigmoid trick (Maddison al., 2016). During inference, a merged expert is formed for each task by averaging the paramcters of the skill experts weighted according to the task-skill matrix.
Soft MoE Puigcerver et al. (2023) recently proposed Soft MoF, which assigns "slots" to each expert and passes a weighted average of input tokens into each slot. All operations in Soft MoE' method are differentiable, avoiding the need for the gradient ostimation. We consider Soft MoE with a single slot per expert to ensure fair comparison by having computational cost equivalent to other discrete routing baselines.
3 Soft Merging of Experts with Adaptive Routing
As we will later show in section 4, the gradient estimation techniques used to train models with discrete routing often fail to produce performant routing strategies. Our goal in this work is therefore to explore whether it is possible to train models with adaptive routing among experts without resorting to gradient estimation. Specifically, we aim to achieve better performance by designing an expert and router architecture that facilitates standard end-to-end gradient-based training but does not increase computational costs.
Ensemble Routing One simple idea would be to pass the imput of a given expert routing block through every expert, and then compute an average of the experts' outputs weighted according the router's distribution, iee exactly computing fe 0.). We refer to this approach as an ensemble routing strategy since it corresponds to using the ensemble prediction of the experts. Since the operations involved in computing the average are all differentiable, using an ensemble routing strategy would allow for exact computation of gradients and end-to-end-learning. Unfortunately, such an approach would incur significant, increase in computational costs because it requires computing the output of every expert rather than single expert.
Merging Experts To explore an alternative fully-differentiable expert routing block, we take inspiration from recent work on merying models (Matena & Raffel, 2021; Wortsman et al., 2022b;c; Choshen et al., 2022b; Don-Yehiya al., 2022; McMahan et al., 2017). These works have shown that averaging the parameters of models that share a common architecture can often produce an aggregate model that shares the capabilities of the individual models. Notably, Wortsman et al. (2022b); Matena & Raffel (2021) found that averaging the weights of multiple fine-tuned models produced single model that performs comparably to an ensemble of the models. In addition, both Adamix (Wang et al., 2022b) and Latent Skills (Ponti et al., 2022) include steps that involve averaging expert parameters, though neither of these methods learn an adaptive per-example routing strategy. Motivated by these findings, we propose Soft Merging of Experts with Adaptive Routing (SMEAR), which constructs a single merged expert whose parameters are computed as the weighted average of the experts within a routing block. Each expert's weight is set according to the corresponding routing probability generated by the router. In SMEAR, the inpul. to the routing block is fed into the merged expert. and the merged expert's output is used the output of the block. By averaging parameters, SMEAR implicitly assumes that all experts in the routing block share an identical architecture (thereby inducing a natural one-to-one mapping between parameters in each expert). To the best of our knowledge, all past works focused on routing among experts use experts with a common architecture, so we do not see this assumption as a major limitation.
More explicitly, we define SMEAR as computing the output of an expert routing block using merged expert computed as
f(u. Σ R(v),o))
5

---

Published in Transactions on Machine Learning Research (05/2024)
The merged expert shares the same architecture with the individual experts ft. Notably, the input of the routing block is only ever processed by f; activations are never fed to any of the individual experts. To break symmetry, all experts are randomly initialized with different parameter values. Importantly, all operations in SMEAR are fully differentiable, enabling standard gradient-basod end-to-end learning. In addition, SMEAR relains the ability to learn an adaplive routing stralegy that can route different examples different experts without relying on hand-specified tags (as in Latent Skills and tag-based routing). We will later show qualitatively that this leads to meaningful specialization of different experts in real-world experiments (section 4.3).
Computational Costs Importantly SMEAR only ever computes the outpul of single expert, suggesting that SMEAR's computational cost could be comparable single-expert discrete routing and significantly lower than ensemble routing. However, the averaging operation in SMEAR incurs a nontrivial computational cost. To quantify this cost, focus on the common expert architecture comprising dense layer that projects from d-dimensional activations to an m-dimensional vector followed by a nonlinearity and an additional dense layer projecting from m dimensions back to d. For simplicity, we ignore the (relatively minor) cost of the nonlincarity. We assume the imput is a length-L sequence of activations with size L x d. In this case, computing the output of the merged experts incurs a computational cost of approximately I, x 4 x x m FLOPs and ensemble routing with N experts would require N x L x d m FLOPs. SMEAR additionally must average together the parameters of N experts, which costs an additional N
d x m FLOPs.
Some past work on models with discrete routing has the router choose different expert for each entry in the input sequence of activations (e.g. Fedus et al., 2021; Lewis et al., 2021; Roller et al., 2021). This would require computing the expert average L limes, which would make the cost of SMEAR similar to that of ensemble routing. We therefore focus on settings where models make a single routing choice for an entire input example (e.g. Gururangan et al., 2021; Kudugunta et al., 2021; Ye et al., 2022). This results in a m for SMEAR. Consequently, as long as L 4 N x 2, SMEAR and discrete routing have roughly the same computational costs. Given that L. is on the order of hundreds or thousands of tokens for text-based tasks and on the order of thousands for vision tasks, L 4 will be much larger than N x 2 as long as there is modest number of experts. In our experiments within the T5-GLUE selting, where I, = 128 and N == 8, this results in a minimal runtime difference. Furthermore, Nxf times cheaper than ensemble routing. More concretely,
total cost. of approximately xL 4 N x 2)
we would expect SMEAR to be approximately N+T. we will later experimentally validate that the wall-clock time required to process an example with SMEAR in real-world experiments is roughly the same as using discrete routing and significant faster than ensemble routing.
4 Experiments
In order to thoroughly evaluate the effectiveness of SMEAR, we perform experiments in two real-world settings that differ in model architecture and modality. We are particularly interested in whether a given approach for learning routing outperforms the heuristic routing strategies described in section 2. As such, we focus on experimental settings where a performant "tag routing" baseline can be designed, i.c. where we have oracle access to metadata that can be used to appropriately route examples. Specifically, we experiment with tine-tuning T5.1.1 Base (Raffel et al., 2020) on datasets from GLUE (Wang al., 2018) (referred lo as T5-GLUE) and (ResNet-Donie DomainNet). In these settings, we add experts to existing pre-trained backbone in the same way that Adapters are used for parameter-efficient fine-tuning (Houlsby et al., 2019). While past work has also considered using discrete routing among experts to train large-scale models from scratch (Fedus et al., 2021; Shazeer et al., 2017), we focus on modular finettnnd models in this work and leave large-scale experiments for future work.
ResNet18 (He el al., 2016) on DomainNet (Peng al., 2019)
4.1 Settings
T5-GLUE In this setting, we focus on training T5 model (Raffel et al., 2020) on the GLUE meta- benchmark (Wang et al., 2018) for natural language understanding. We provide background the GLUE
6

---

Published in Transactions on Machine Learning Research (05/2024)
dataset and the example format we use in appendix B.1. We follow the approach of Mahabadi et al. (2021) for splitting each GLUE dataset into train, eval, and test splits. Past work has demonstrated improved performance on RTE by co-training with MNLI (Phang et al., 2018; Devlin et al., 2018; Pruksachatkun et al., 2020; Vu et al., 2020; Choshen et al., 2022a), and we congruously found that sharing an expert between RTE and MNLI produced a stronger tag routing strategy. In the interest of making our baselines as strong as possible, we use this improved tag routing scheme in all experiments. We use the pretrained T5.1.1 Base model as the backbonn and adapt the model in a way similar to adding adapters (Houlsby al., 2019) for single task, i.e. we keep all pretrained parameters frozen except for layer normalization parameters and insert expert routing blocks after self-attention, ford-forward and cross-attention modules. The T5 1.1 Base model has 12 Transformer layers in both the encoder and decoder, resulting in a total of 12 x 2 = 24 blocks in the encoder and 12 x 3 = 36 blocks in the decoder, or 60 expert routing blocks in lotal. In each block, we introduce eight experts (one for each dataset in GLUE). The router architecture is simply a linear classifier, i.c. a linear projection layer consisting of weight matrix of shape x N, where d is the model dimension and N is the number of experts in the MoE layer, followed by a soflmax nonlinearity. To help avoid sat urating the softmax nonlinearity, the linear layer. In the encoder, the router takes input the preceding hidden states, which are averaged across the sequence and fed into the router. In the decoder, the routers receive the average of the encoder's 's final hidden states instead of the decoder hidden states to prevent information leakage from later target tokens. We also include expert dropout (Liu et al., 2022b) where each expert is dropped with a probability of 0.1 wherever it was found to be beneficial (a detailed ablation can be found in table 1). In GLUE, dataset sizes vary by three orders of magnitude, and we therefore found that load-balancing losses (as used e.g. in (Shazeer et al., 2017; Fedus et al., 2021; Lepikhin et al., 2020) to encourage uniform usage across experts) tended to hurt performance, so we did not include them.
apply layer normalization both to the input of the router well as the rows of
ResNNet-DomainNG In this setting, we focus on adapting an ImageNet pre-trained ResNet model (He et al., 2016) to datasets within DomainNet (Peng et al., 2019). DomainNet is a collection of object recognition datasets that cover six distinct domains and all share the same label space corresponding to 345 object categories. We treat the domain of each example as its tag. As in the T5-GLUE setting, we freeze the pretrained model and insert eight expert routing blocks after each of the eight residual layer groups in the model. Each block includes six experts corresponding to the number of domains. We use the same architecture for routers as in T5-GLUE and feed average- compute the routing probability. Experts in this setting use batch normalization on their input instead of layer normalization in the output, following Rebuffi et al. (2017). As in T5-GLUE, we omit load- had-balanning losses due to dramatically different sizes across domains in DomainNet.
hidden states into the router to
We only provide the average performance across natasets/domaaine in the main paper due to space limitations. Full results are provided in appendix F.
4.2 Results
To assess the overall effectiveness of routing strategies learned with SMEAR, we compare to learned routing using the gradient estimators, heuristic routing strategies, and modular baselines from section 2. A summary of our results is shown in fig. 2. First, we find that models using routing strategies learned through gradient estimation often underperform heuristic routing strategies while the best-performing estimator (REINFORCE) in T5-GLUE outperforms tag routing, all estimators perform worse than tag routing in Resaiet-DmmannNet. On the other hand, observed some cases where gradient ostimation-based routing outperforms hash single-expert routing, which suggests that the learned routing strategies were nontrivial. Pertinently, in all experimental settings, SMEAR matches or outperforms every other routing strategy, including both routing learned by gradient estimators and all heuristic routing strategies. In particular, SMEAR achieves 2.7% improvement over tag routing in T5-GLUE and 0.6% improvement over tag routing in ResNeet-DmmanNeet, suggesting effective specialization and sharing of exports. SMEAR additionally outperforms the single-cxpert parameter-matched baseline (1x parameters) by 1.4% in T5-GLUE and 1.2% in ResNeetDDmmiiNNl, further highlighting the importance of modularity.
7

---

Published in Transactions on Machine Learning Research (05/2024)
T5-GLUE
ResNet-DomminNNet
SMEAR
1x Parameters
Average accuracy
1x Compute
81
62
Adamix
Tag
Latent Skills
80
Top-k ST-Gumbel
60
79
REINFORCE
DSelect-k
78
58 G
Soft MoE
250
300
350
1400
1600 1800
2000
Ensemble
Examples/second
Examples/second
Figure 2: Average accuracy and inference speed (in examples processed per second) for models using different settings. Routing approaches are grouped by
routing approaches on our T5-GLUE and color; groups are (in order of the legend) our method (SMEAR), methods that do not use adaptive routing (1x compute, 1x parameters, Adamix, and Hash), methods that make use of metadata (Tag and Latent Skills), methods that learn adaptive routing (Top-ki; ST-Gumbel, REINFORCE, DSelect-k; and Soft MoE), and methods that ensemble expert outputs (Ensemble). We omit Hash routing from the plots because its poor hampers readability. Exact numerical
performance (66.9% on T5-GLUE and 52.4% on results for all methods and standard deviation across five runs are provided in appendix F.
As an upper bound on performance, we also compare SMEAR to expert ensembling (*Ensemble") which averages the outputs of all experts and incurs significantly higher computational cost. SMEAR matches the performance of ensemble routing in T5-GLUE and modestly underperforms in ResNet-Domainll despite being significantly computationally cheaper. Compared to Adamix, which similarly averages experts but does not learn routing strategy, SMEAR achieves 3.2% higher performance in T5-GLUE and 4% higher in ResNet-DomainnNel Since the Soft MoE method averages input tokens, it's inapplicable for the encoder-decoder model in T5-GLUE, where future tokens are not available for averaging in the decoder during inference. Hence, we include Soft MoE. for Nes where SMEAR outperforms it by 1.5%. Additionally, SMEAR exceeds the DSelect-k method by 2.7% in ResNet-DomainNet. However, despite extensive hyperparameter luning we encountered training instabilites with the DSelect-k method in T5- GLUE and therefore omit those results. Moreover, while the performance improvement of SMEAR over Latent Skills is relatively small (0.6% in T5-GLUE and 0.1% in ResNet-DomainNNe)) a major advantage of SMEAR over Latent Skills is that it does not assume access lo oracle lags (which are not always available in real-world settings) and instead learns adaptive routing strategy. Finally, we highlight the consistency of improvements achieved by SMEAR across a diverse range of datasets and architectures, confirming its generality and robustness.
We additionally plot the inference speed (in terms of number of examples processed per second) of cach method in fig. 2. The single-expert, Adamix, Hash, and Tag routing methods are the fastest since they do not use any routing networks. Despite the slight overhead of averaging the weights in SMEAR, we observe that its inference speed is almost identical to that of discrete adaptive routing (as learned via gradient estimation techniques). This confirms that the performance gains attained by SMEAR do not incur significant additional costs. Ensembling expert outputs is the slowest, with a 1.2x slowdown in T5-GLUE and 1.3x slowdown in compared to SMEAR.
Scaling Thus far, we have always set the number of experts equal to the number of lasks (in T5-GLUE) or domains (in DomainNet).. However, with learned routing there is no reason to force this constraint, so we therefore tested the scalability of SMEAR by evaluating its performance with twice as many experts (16 for T5-GLUE and 12 for ResNeetDomainNet). We found significant improvement (0.8%) when doubling the number of experts on RosNet-DomainNet, but no significant change on T5-GLUE (81.3 1.1 81.6 + 1.1).
8

---

Published in Transactions on Machine Learning Research (05/2024)
Router for Encoder FFN 3
Router tor Decoder FFN 5
RTE-0.05 0.05 0.01 0.07 0.01 0.51 0.3 0.04 0.02
0.14
0.0
0.0
0.82
0.0
0.0
0.0
SST-2
0.000..1.4..0000
0.0 0.0 0.1 0.0 0.84
1.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
MRPC-0.330.09
0.01
0.02
0.0
0.07
0.0
0.11
0.05
0.0
0.0
0.8
0.04
0.0
0.0
STS-B
0.0
0.0
0.0
0.14
0.0
0.0
0.0
0.77
QQP
0.0
0.0
0.0
1.0
0.0
0.0
0.0
0.0
0.0
0.52
0.0
0.0
0.4
0.01
0.0
0.06
MNLI-O.2
0.04
0.
0.16
0.03
0.0
0.15
0.33
0.0
0.0
0.13
0.2
0.22
0.0
0.05
0.39
QNLI
0.0 0.0
0.0 0.0 0.90
0.01
0.0 0.0
0.0 0.32 0.0 0.0 0.67
0.0
0.0 0.0
CoLA
0.01 0.0 0.0 0.83
0.0 0.02
0.23 0.0 0.0
0.0
0.11
Expert
Expert 2
Expert 3
Expert 4
Expert
Expert
Expert Expert 8
Expert
Expert 2
Expert 3
Expert 4
Expert
Expert 6
Expert 7
Expert 8
(a)
T6-GLUE model
Router for Residual Layer
Router for Residual Layer
Clipart-
0.19
0.17
0.16
0.11
0.14
0.23
0.14
0.1
0.18
0.11
0.19
0.29
Infograph-
0.23
0.14
0.13
0.12
0.13
0.26
0.13
0.09
0.43
0.11
0.07
0.16
Painting
0.18
0.12
0.12
0.15
0.14
0.3
0.31
0.11
0.17
0.15
0.13
0.13
Quickdraw-
0.1
0.35
0.28
0.08
0.1
0.07
0.29
0.08
0.32
0.13
0.11
Real-
0.16
0.12
0.13
0.14
0.16
0.29
0.34
0.11
0.15
0.13
0.11
0.16
Sketch
0.19
0.2
0.14
0.19
0.12
0.15
0.14
0.12
0.15
0.14
0.12
0.33
Expert
Expert 2
Expert
Expert 4
Expert 5
Expert
Expert
Expert 2
Expert 3
Expert
Expert 5
Expert 6
(b) ResNet- NeNo----- model
Figure 3: Average routing distributions produced by SMEAR for two routers from the T5-GLUE model and two from the examples from a given dataset.
model. For given router, we average all routing distributions across all
This suggests there benefit to increasing capacity in the T5-GLUE setting. The complete results for doubling the number of experts presented in appendix F (labeled "SMEAR 2x").
4.3 Qualitative Analysis
In this section, we provide qualitative analysis of the routing learned by SMEAR by visualizing the average router distribution across all examples given dataset for every router each model. Figure shows four select visualizations (two from SMEAR-based model trained in T5-GLUE and two from ResNNet-DomaiNNe)) Across the two T5-GLUE router distributions shown in fig. we observe significantly different behavior mainly follows a tag routing-style strategy whereas the other routes most datasets to the same expert. However, we note that the tag-style router utilizes shared experts for RTE, MRPC, and MNLI; notably, these tasks are somewhat similar in that they all involve determining similarity among pairs of sentences. In the single-expert-sty/e router, STS-B (the only regression task) and SST-2 (which has a distinct output vocabulary) are given dodicated experts, and MNLI (a large and relatively challenging datasct) is routed through many different experts. More broadly, we highlight that there generally a great deal of sparsity the learned routing distributions, suggesting significant. amount of expert specialization. In

---

Published in Transactions on Machine Learning Research (05/2024)
see that examples from the Quickdraw domain are routed to two specific experts in both cases. Additionally, we observe that the router distribution of the Painting and Real domains are highly correlated. Other domains such as Clipart and Sketch seem to evenly use experts. Interestingly, there is less expert specialization in the ResNet-DomainNet model, suggesting that there may be more similarities between the individual domains in DomainNet compared to the lasks in GLUE
In general, other approaches for learning routing did not exhibit as intuitive or meaningful specialization and sharing. In Resseet-DomainNe Top-k demonstrates uniform routing in the initial layers but chooses a single expert in the last layer. REINFORCE, ST-Gumbel, and DSelect-k tend to exhibit mostly degenerate single-cxpert routing. Interestingly, all these gradient ostimators learn to assign a distinct expert for the Quickdraw dataset. However, this degree of specialization is insufficient for achieving superior performance scores. In T5-GLUE, these estimators display degenerate routing in some layers, while showing a tendency to share a few experts (approximately 3 out of 8) in other layers across tasks. Methods such as Latent Skills and Ensemble utilize most experts in the MoE layer (similar to SMEAR). Routing distribution visualizations for all methods and all layers can be found in appendix G.
5 Related Work
Models with Conditional Computation Given the difficulties of training models with discrete routing strategies, there is a large body of literature investigating effective methods for training such models. Deecke et al. (2020); Hazimeh et al. (2021); Dua et al. (2021) start training with most of the experts activated and gradually introduce sparsity. Kudugunta et al. (2021); Ponti et al. (2022); Ma et al. (2019); Gupta et al. (2022) group examples from the same task together and introduce task-specific parameters in the router. Zuo et al. (2021); Wang ct al. (2022b) explore random routing with consistency regularization as a form of model ensembling. Other works avoid learned routing by hand-crafting heuristic routing strategies. Gururangan et al. (2021) built sparsely activated language models where different domains use separate experts and then weights the experts for new domains. Tang et al. (2022); Pfeiffer et al. (2022; 2020) assign experts based on task-related human knowledge. Our focus on settings where performant routing schemes can be hand-designed takes inspiration from this line of work.
Because sparsely activated models disentangle computation and parameter count, significant effort has gone into leveraging conditional computation to create massive pre-trained models with a feasible computation cost (Fedus et al., 2022; Shazeer et al., 2017; Fedus et al., 2021; Du et al., 2022; Zoph et al., 2022; Yu et al., 2022). Many works explore different routing methods in this setting, with major focus on balancing the load across experts (Lewis et al., 2021; Zhou et al., 2022; Kool et al., 2021; Roller et al., 2021). Li et al. (2022a) demonstrate that models structured as sparse mixture-of-expertee generalize effectively to novel domains, as compared to other domain generalization algorithms in vision transformers. Another line of work aims to introduce ways to convert trained dense models into similar-sized sparse models with a lower computational footprint (Lee-Thorp & Ainslie, 2022; Zhang et al., 2022; Komatsuzaki et al., 2022). More generally, there are other forms of conditional computation beyond the formulation we describe in this work (Han al., 2021). Modular networks (Kirsch et al., 2018; Jiang & Bansal, 2019; Hu et al., 2017) use a router to assemble heterogeneous subnetworks works into a network layout specific to a given input example. Early exiting (Graves, 2016; Xin et al., 2020; Liu et al., 2020a; Bengio et al., 2015) saves computation by terminating inference in carlier layers or recurrent stops.
Gradient Estimation Techniques Many gradient estimators have been proposed to produce approximate gradients for backpropagating through discrete steps. Clark al. (2022) uses a learned baseline to reduce the variance of the REINFORCE estimator. The REBAR estimator (Tucker et al., 2017) adds reparameterizable term to REINFORCE as baseline that results in more effective unbiased estimator. This additional term incorporates relaxed sample similar to Gumbel-Softmax (Jang et al., 2016). RELAX (Grathwohl et al., 2017) is similar to REBAR but uses a learnable neural network for the reparameterizable term. Kool et al. (2019) uses additional samples as a built-in baseline for REINFORCE. Yin & Zhou (2018) and Dong et al. (2020) use the idca of coupling between multiple samples to reduce the variance of the gradient estimator for improved training of models with binary latent variables. Dong et al. (2021) improve upon Yin & Zhou (2018) and Dong et al. (2020) by extending the estimator to categorical variables. These studies have
10

---

Published in Transactions on Machine Learning Research (05/2024)
theoretically analyzed gradient estimators, focusing on the bias and variance of these gradients and suggesting enhancements through improved relaxation techniques and variance reduction techniques. The fundamental theoretical advantage of our method lies in its ability to enable exact gradient computation through standard backpropagation. In preliminary experiments, we did not find any major gains from using more sophisticated gradient estimation Lechniques, but designing gradient estimators with discrete routing in mind could yield more performant routing strategies.
Issues with Conditional Computation A great deal of past work has highlighted issues with models that use conditional computation. Clark et al. (2022) study the scaling laws of sparse language models and discovered a computational cutoff above which no additional benefits are observed. Relatedly, Du et al. (2022) observe worse results when further scaling up the number of experts. Chi et al. (2022) highlight that using the model's activations as input to the router can cause the representations to "collapse". Dai et al. (2022) demonstrate that learned routing decisions can fluctuate significantly over training. Mittal et al. (2022) create set of simple and compositional data distributions and show that systems with modular architecture can not find the most performant solution when trained end-to-end. Ye et al. (2022) experiment with various designs for multi-task learning with task-level routing and find that the performance never surpasses simple multi-task baselines. We show a possibility to avoid these issues with a fully differentiable routing strategy that does not increase compatational costs.
Weight Averaging Methods Many prior works utilize parameter averaging for ensembling. Wortsman et al. (2022c); Ilharco et al. (2022) average the weights of a pre-trained and a fine-tuned model to improve performance on target tasks as well as robustness to distribution shift. Choshen et al. (2022b) similarly show that merging multiple models fine-tuned on different datasets can provide a better initialization than using the original pre-trained model for further fine-tunin finetuning on new unseen datasets. The phenomenon of linear mode connectivity, as observed by Frankle et al. (2020); Qin et al. (2022) and Entezari et al. (2021), refers to the tendency of models to converge to single low-loss basin, which makes it possible to average the weights of models without increasing loss. Juneja et al. (2022) found that tasks with different generalization strategies have high barriers when linearly interpolated, whereas lasks with similar generalization are linearly mode connected and can be averaged. They defined several generalization strategies and found that sets of tasks form clusters with those strategies. In our case, the router plays the role of identifying experts with similar generalization properties when conditioned on the input and performs the averaging.
Yang et al. (2019); Zhang et al. (2021) compute convolution kernels by averaging weights of individual kernels. Maziarz et al. (2019) learn task-specific binary allocation matrix, where each allocation corresponds to a convolution kernel. Since the convolution operation is linear, weight averaging and ensembling are mathematically equivalent in the aforementioned works. However, SMEAR performs averaging on non-linear and parameter efficient experts that, when trained alone, can match the performance of the fully fine-tuned model (Houlsby et al., 2019). T-Tuning (Wu et al., 2023) employs a set of existing lask specific experts, retrieving the top k experts for a downstream task and learns to interpolate among these experts for the downstream task. While r-Tuming enables transfer learning to a new downstream task by learning to interpolate, our focus is on developing a routing algorithm that learns how to share specialize experts without using any metadata. Wortsman et al. (2022b) average multiple models fine-tuned from large pre- trained models with different hyperparameters, using techniques such as uniform averaging and greedy averaging by picking those models that increase the average performance using a validation set. Gururangan et al. (2021); Li et al. (2022b) use data in novel domains to estimate weights to average each of their domain-specific subnetworks. Izmailov et al. (2018) average model parameters along a training trajectory and show that the resultant model generalizes better than the solution found by SGD at the end of training. Our work instead learns router that implements adaptive routing strategy over the course of the training process.
Concurrent work by Zadouri et al. (2023) used SMEAR for training a mixxuue-ofeexeerssttlll model with LoRA (Hu et al., 2021)- and I (Liu et al., 2022a)-basee experts. They show that the resultant model outperforms full model finetuning in generalizing to unseen tasks. Model averaging is also a common step in distributed optimization, where it is widely used in foderated learning (MeMahan et al., 2017) and has recently been used for distributed fine-tuning (Wortsman et al., 2022a), multi-domain training (Li et al.,
11

---

Published in Transactions on Machine Learning Research (05/2024)
2022b), and multitask training (Don-Ychiya et al., 2022). There are also works that utilize different styles of merging instead of weight averaging of parameters, such as reweighting parameters in accordance with their approximate Fisher information (Matena & Raffel, 2021), aligning features by fitting a linear projection (Jin et al., 2022), and permuting columns to account for permutation symmetrics (Ainsworth et al., 2022). We are interested in applying these more sophisticated merging methods to SMEAR in future work.
6 Conclusion
In this work, sought to address shortcomings of models with discrete routing among experts that lead them to underperforı heuristic non-learned routing. We hypothesized that these issues stem from the gradient estimation techniques required to propagate gradients through discrete routing decisions and therefore focused on designing an expert routing architecture that allows exact calculation of gradients. Our approach, called SMEAR, works by computing a weighted average of expert parameters where the weighting is set according to the output of a learned router. We compared the performance of models using SMEAR to discrete routing models that were trained via various gradient estimation techniques. In experimental settings covering different modalities and model architectures, we found that SMEAR outperformed all models with discrete routing well as performant heursitic routing strategies. Notably, this performance boost comes with no increase in computational costs. SMEAR also matched or outperformed existing statc-of-the-art methods for learning modular models through expert averaging while removing the requirement for oracle task labels. Through qualitative analysis, we further confirmed that the experts learned a model using SMEAR specialize to different types of inputs and that the router learns a nontrivial strategy that exploits commonalities across different examples. In future work, we are interested in exploring different expert architectures (Liu et al., 2022a; Hu et al., 2021) and improved merging methods (Matena & Raffel, 2021; Ainsworth et al., 2022; Jin et al., 2022). Given access to larger amount of compute, we would also be excited to try out SMEAR in the large-scale settings where discrete routing has been used (Fedus et al., 2021; Zoph et al., 2022; Du et al., 2022) to see whether it helps fix the poor scaling properties of models with discrete routing (Clark et al., 2022).
References
Samuel K Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetrics. arXiv preprint arXiv:2209.0488 2022.
Stephen H Bach, Victor Sanh, Zheng-Xie Xin Yong, Albert Webson, Colin Raffel, Nihal V Nayak, Abheesht. Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. Promptsource: An integrated development environment and repository for natural language prompts. arXiv preprint arXiv:2202. 01279, 2022.
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. ar preprial arXiv:2106.082.0 2021.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolci Zhou, and Antonio Torralba. Under- standing the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 317744):30071130000 2020.
Emmanuel Bengio, Pierre-Lauc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:1511.06297, 2015.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXio:1808.3432, 2013.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Daniel Cer, Mona Diab, Encko Agirre, Inigo textual similarity-miliinngaal and cross-lingual focused evaluation. Xrxiv preprint ar Xix: 700.000055 2017.
and Lucia Specia. Semeval-2017 task 1: Semantic
12

---

Published in Transactions on Machine Learning Research (05/2024)
Zewen Chi, Li Dong, Shaohan Huang. Damai Dai, Shuming Ma, Barun Patra, Saksham Singhal, Payal Bajaj, Xia Song, and Furu Wei. On the representation collapse of sparse mixture of experts. arv preprinl arXiv:2204.09119 2022.
Leshem Choshen, Elad Venezian, Shachar Don- Yehia, Noam Slonim, and Yoav Katz. Where to start? analyzing the potential value of intermediate models. arXiv preprint wrXie2221.001007 2022a.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining. arXiv preprint XXX::2244.00044 2022b.
Aidan Clark, Diego de Las Casas, Aurelia Guy, Arthur Mensch, Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman, Trevor Cai, Sebastian Borgeaud, et al. Unified scaling laws for routed language models. In International Conference on Machine Learning, pp. 4057-4086. PMLR, 2022.
Róbert Csordás, Sjoerd van Steenkiste, and Jürgen Schmidhuber. Are neural nets modular? inspecting functional modularity through differentiable weight masks. In International Conference on Learning Representations, 2021.
Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, and Furu Wei. Stablemoe: Stable routing strategy for mixture of experts. arXiv preprint arXiv:2304.08300 2022.
Nicola De Cao, Wilker Aziz, and Ivan Titov. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021.
Lucas Deecke, Timothy Hospedales, and Hakan Bilen. Latent domain learning with dynamic residual adapters. arXiv preprint arxiv:2006. 00996, 2020.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800 2018.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third International Workshop on Paruphrasing (IWP2005), 2005.
Shachar Don- Yehiya,, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning. arXiv preprint erXim:2212.01376 2022.
Zhe Dong, Andriy Mnih, and George Tucker. Disarm: An antithetic gradient estimator for binary latent variables. Advances in neural information processing systems, 33:18637-18647, 2020.
Zhe Dong, Andriy Mnih, and George Tucker. Coupled gradient estimators for discrcte latent variables. Advances in Neural Information Processing Systems, 34:24498 24508, 2021.
Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning, pp. 5547 5569. PMLR, 2022.
Dheeru Dua, Shruti Bhosale, Vedanuj Goswami, James Cross, Mike Lewis, and Angela Fan. Tricks for training sparse translation models. arXiv preprint xrXim22100.08240. 2021.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in lincar mode connectivity of neural networks. arXiv preprint arXie:2110.06296, 2021.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021.
William Fedus, Jeff Dean, and Barret Zoph. A review of sparse expert models in deep learning, arXie preprint ar arXim:2209.0166 2022.
Jonathan Frankle, Gintare Karolina. Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning, pp. 3259-3269 PMLR, 2020.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, and David Duvenaud. Backpropagation through the void: Optimizing control variates for black-box gradient estimation. arXiv preprinl or Xiv: k::111 1711.001102 2017.
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint orXic:1603.08900 2016.
Shashank Gupta, Subhabrata Mukherjee, Krishan Subudhi, Eduardo Gonzalez, Damien Jose, Ahmed Hassan Awadallah, and Jianfeng Gao. Sparsely activated mixtare-of-experts are robust multi-task learners. ArXin, abs/2200 abs/2224.07669, 2022.
Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A Smith, and Luke Zettlemoyer. Demix layers: Disentangling domains for modular language modeling. or Xrv preprint arXiv:2108. 210.0000,,, 2021.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Hussein Hazimeh, Zhe Zhao, Aakanksha Chowdhery, Maheswaran Sathiamoorthy, Yihua Chen, Rahul Mazumder, Lichan Hong, and Ed Chi. Dselect-k: Differentiable selection in the mixture of experts with applications to multi-task learning. Advances in Neural Information Processing Systems, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-effciint transfer learning for nlp. In International Conference Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXie preprinl arXiv:2106 09685, 2021.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering, In Proceedings of the IEEE international conference on computer vision, pp. 804-813, 2017.
Gabriel Ilaarco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. arXiv preprint arXiv:2208.0540 2022.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407 2018.
Fric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXio preprind arXiv:1611.01144, 2016.
Yichen Jiang and Mohit Bansal. Self-assembling modular networks for interpretable multi-hop reasoning. arXiv preprint arXiv:1909 1rX.o.1909005033, 2019.
Xisen Jin, Xiang Ren, Daniel Treotiuc-Pietre and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXio:2212. 09849, 2022.
Jeevesh Juneja, Rachit Bansal, Kyunghyun Cho, João Sedoc, and Naomi Saphra. Lincar connectivity reveals generalization strategies. arXiv prepriat or wxXiv:2205.12411 2022.
Louis Kirsch, Julius Kunze, and David Barber. Modular networks: Learning to decompose neural computation. A Adancee in neural information processing systems, 31, 2018.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Aran Komatsuzaki, Joan Puigcerver, James Leo-Thorp, Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie, Yi Tay, Mostafa Dehghani, and Neil Houlsby. Sparse upcycling: Training mixture-ofexxeers from dense checkpoints. arXiv preprint arXim:2212. 05055, 2022.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! 2019.
Wouter Kool, Chris Maddison, and Audriy Mnih. Unbiased gradient estimation with balanced assignments for mixtures of experts. arXiv preprint arXiv:2109.1188 2021.
Sneha Kudugunta, Yanping Huang, Ankur Bapna, Maxim Krikun, Dmitry Lepikhin, Minh-Thang Luong, and Orhan Firat. Beyond distillation: Task-level mixture-offexperts for efficient inference. arv preprint or arXiv:2110.03742. 2021.
James Lee-Thorp and Joshua Ainslie. Sparse mixers: Combining moe and mixing to build a more efficient. bert. arXiv preprint 2rXiv:2205. 12399, 2022.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint (rXii:2006.100 16668, 2020.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. Base layers: Simplifying training of large, sparse models. In International Conference on Machine Learning, pp. 6265-6274. PMLR, 2021.
Bo Li, Jingkang Yang, Jiawei Ren, Yezhen Wang, and Ziwei Liu. Sparse fusion mixtuure-o-exxertts are domain generalizable learners. arXie preprint arXiv:2206. 040666 2022a.
Margaret Li, Suchin Gururangan, Tim Dettmers, Mike Lewis, Tim Althoff, Noah A. Smith, and Luke Zettlemoyer. franch-train-merge Embarrassingly parallel training of expert language models. ArXie, abss/200800338 03306, 2022b.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficien fine-tuning better and cheaper than in-context learning. arXiv preprint ar Sff::2255005633 2022a.
Rui Liu, Young Jin Kim, Alexandre Muzio, and Hany Hassan. Gating dropout: Communication-ellcienn regularization for sparsely activated transformers. In International Conference on Machine Learning, pp. 13782-13792. PMLR, 2022b.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Haotang Deng, and Qi Ju. Fastbert: a self-distilling bert with adaptive inference time. arXiv preprint arXiv:2004.021 2020a.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. Transoctions of the Association for Computational Linguistics, 8:726-742, 2020b.
Jiaqi Ma, Zhe Zhao, Jilin Chen, Ang Li, Lichan Hong, and Ed H Chi. Sur: Sub-network routing for flexible parameter sharing in multi-task learning. In Proceedlings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 216 223, 2019.
Chris J Maddison, Andriy Mnih, and Yoe Whye Tch. The concrete distribution: A continnous relaxation of discrete random variables. arXiv preprint ar 1011:0161..0011,, 2016.
Rabech Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter-cfficien multi-task fine-tuning for transformers via shared hypernetworks. ar Xio preprint orXin: 06. 04489, 2021.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Michael Matena and Colin Raffel. Merging models with fisher-weighted averaging. arXiv:2111.09832, 2021.
arXiv preprint
Krzysztof Maziarz, Efi Kokiopoulou, Andrea Gesmundo, Luciano Sbaiz, Gabor Bartok, and Jesse Berent. Flexible multi-task networks by learning parameter allocation. arXiv preprint arXiv:1910.0490 2019.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- efficient learning of deep networks from decentralized data. In Artificial Intelligence and Sstatistics, 2017.
Sarthak Mittal, Yoshua Bengio, and Guillaume Lajoie. Is a modular architecture enough? orXiv preprint arXiv:2206.00 02713, 2022.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the LEEE/CVF international conference on computer vision, pp. 1406 1415, 2019.
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. Mad-x: An adapter-based framework for multi-task cross-lingual transfer. In EMNLP, 2020.
Jonas Pfciffer, Naman Goyal, Xi Victoria Lin, Xian Li, James Cross, Sebastian Ricdel, and Mikel Artetxe. Lifting the curse of multiinguuality by pre-training modular transformers. In NAACL, 2022.
Jonas Pfeiffer, Scbastian Ruder, Ivan Vulié, and Edoardo Maria Ponti. Modular deep learning. arXiv preprint ar Xiv: 0ttt2302.115222 11529, 2023.
Jason Phang, Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088 2018.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual multilingual bert? arXiv preprint wrXiv.1906.01500 2019.
Edoardo M Ponti, Alessandro Sordoni, and Siva Reddy. Combining modular skills in multitask learning. arXiv preprint Xrx:22200 19914, 2022.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R Bowman. Intermediate- task transfer learning with pretrained models for natural language understanding: When and why does it work? arXiv preprint Kww:2055.0062 2020.
Joan Puigcerver, Carlos Riquelme, Basil Mustafa, and Ncil Houlsby. From sparse to soft mixtures of experts. Xiviv preprinl Xiv:22300 00951, 2023.
Yujia Qin, Cheng Qian, Jing Yi, Weize Chen, Yankai Lin, Xu Han, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Exploring mode connectivity for pre-trained language models. arXio preprint arXiv:2210.1 14 102, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Malena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(14)) 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 0025000000 2016.
Sylvestro-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. Advances in newral information processing systems, 30, 2017.
Stephen Roller, Sainbayar Sukhbaatar, Jason Weston, et al. Hash layers for large sparse models. Advances in Neural Information Processing Systems, 34:17555-17566, 2021.
Victor Sanh, Albert Webson, Colin RAffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. Xix preprint Xiv 10. 10.08207, 2021.
16

---

Published in Transactions on Machine Learning Research (05/2024)
John Schulman, Nicolas Heess, Theophane Weber, and Picter Abbeel. Gradient estimation using stochastic computation graphs. Advances in Neural Information. Processing Systems, 28, 2015.
Tyer Shankar, Dandekar Nikhil, and Csernai Kornel. First quora datasel release: question pairs (2017). 2017.
https://www quora. https://www.uuura.cco////
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expert layer. arXiv preprint arXiv:1701.06338 2017.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642 2013.
Duyu Tang, Fan Zhang, Yong Dai, Cong Zhou, Shuangzhi Wu, and Shuming Shi. One model, multiple tasks: Pathways for natural language understanding. ArXiv, hts/2203.033122 2022.
George Tucker, Andriy Mnih, Chris Maddison, John Lawson and Jascha Sohl-Dickstein. Rebar: Low- variance, unbiased gradient estimates for discrete latent variable models. Advances Nearal Information Processing Systems, 30, 2017.
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke Subhransu Maji, and Mohit lyyer. Exploring and predicting transferability across nlp tasks. arviv preprint XxXx:2005.007700 2020.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman, Glue: A multi- task benchmark and analysis platform for natural language understanding. arXie preprint Xrvi 1xXx:11844.... 804.074641 2018.
Xiaozhi Wang, Kaiyue Wen, Zhengyan Zhang, Lei Hou, Zhiyuan Liu, and Juanzi Li. Finding skill neurons pre-trained transformer-based language models. arXiv preprint orXiv:2211.073.0 2022a.
Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: fixture-of-adaaper for parameter-effiient tuning of large language models. arXie preprint Stii:2205.124100 2022b.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Trans- actions of the Association for Computational Linguistics, 24625-64,, 2019.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0160 016622 2021.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704 05426, 2017.
Mitchell Wortsman, Suchin Gururangan, Shen Li, Ali Farhadi, Ludwig Schmidt, Michael Rabbat, and Ari S Morcos. lo-fi: distributed fine-tuning without communication. arXiv preprint arXiv:2210.112 11948, 2022a.
Mitchell Wortsman, Gabriel Ihaarco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongscok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning, pp. 23965 23998. PMLR, 2022b.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongscok Namkoong, et al. Robust fine- tuning of zero-shot models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7959 7971, 2022c.
17

---

Published in Transactions on Machine Learning Research (05/2024)
Chengyue Wu, Teng Wang, Yixiao Ge, Zeyu Lu, Ruisong Zhou, Ying Shan, and Ping Luo. pi-tuning: Transferring multimodal foundation models with optimal multi-task interpolation. In International Conference on Machine Learning, pp. 37713 37727. PMLR, 2023.
Ji Xin, Raphael Tang, Jaejun Lee, Yaoliang Yu, and Jimmy Lin. Deebert: Dynamic early exiting for accelerating bert inference. arXie preprint orXi::200,.12993 2020.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mt5: massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.1 1.0000.111990 11934, 2020.
Brandon Yang, Gabriel Bender, Quoe V Le, and Jiquan Ngiam. Condconv: Conditionally parameterized convolutions for efficient inference. Advances in Neural Information Processing Systems, 32, 2019.
Qinyuan Ye, Juan Zha, and Xiang Ren. Eliciting transferability in multi-task learning with task-level mixture-of-experts. arXiv preprint arXim:2205.1200) 2022.
Mingzhang Yin and Mingyuan Zhou, Arm: regemenntreinforr-mmergg gradient for stochastic binary networks. 2018.
arXiv preprint Xtv:::8877.11
Ping Yu, Mikel Artetxe, Myle Ott, Sam Shleifer, Hongyu Gong, Ves Stoyanov, and Xian Li. Efficient language modeling with sparse all-mlp. arviv preprint arXi::2203. 06850, 2022.
Ted Zadouri, Ahmet Üstün, Arash Ahmadian, Beyza Ermis, Acyr Locatelli, and Sara Hooker. Pushing mixture of experts to the limit: Extremely parameter elficient moe for instruction luning, arXiv preprinl arXiv:2309.0544 2023.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 3712 3722, 2018.
Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In 13th European
Conference Computer Vision, 2014.
Mingda Zhang, Chun-Te Chu, Andrey Zhmoginov, Andrew Howard, Brendan Jou, Yukun Zhu, Li Zhang, Rebecca Hwa, and Adriana Kovashka. Basisnet: Two-stage model synthesis for efficient inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3081-3090, 2021.
Zhengyan Zhang, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. Moefication: Transformer food-forward layers are mixtures of experts. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 877 890, 2022.
Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew Dai, Zhifeng Chen, Quoe Le, and James Landon. ofxture-of-expers with expert choice routing. arXiv preprint ar arXi::220000933 093688 2022.
Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. Designing effective sparse expert models. orXiv preprint arXi::2202.0880 08906, 2022.
Simiao Zuo, Xiaodong Liu, Jian Jiao, Young Jin Kim, Hany Hassan, Ruofei Zhang, Tuo Zhao, and Jianfeng Gao. Taming sparsely activated transformer with stochastic experts. ar Xiv preprint ar 2rx::22 10. 04260, 2021.
18

---

Published in Transactions on Machine Learning Research (05/2024)
A Compute resources used
Here we provide details on the compute resources used in experiments. All models were trained on 48GB A6000s, except for the Ensemble method, which was trained on 80GB A100s. The training time for each T5-GLUE experiment was approximately 108 hours while each ResNet-DomminNet experiment required approximately 11 hours training.
B Experiment Details
In this section, we provide details on the experimental setup and hyperparameter choices for the T5-GLUE and ReaNet--DomiinNet experiments described in the main text. We implemented Adamix (Wang et al., 2022b) and ran with the hyperparameters listed in the below subsections. In Latent Skills (Ponti al., 2022), we use Adapters consistent with all other experiments and chose learning rate ratio of 10 for skill matrix, which we found to be best after sweeping for {1, 10, 100) in both the settings. For exact implementation details of above methods, refer the reader to each of the respective works.
10,
It is also important to mention that SMEAR's memory footprint is comparable to that of other methods. Since each expert's 's size is moderate, we compute expert outputs by preparing an expert for each example and applying them to the examples in parallel utilizing torch. bmm operation. Thus, none of the methods need an extra weight-sized tensor. Training on T5-GLUE requires around 30GB of memory for SMEAR and other methods, with no significant differences observed in ResNet- SesNettDomainNet..
B.1 T5-GLUE
GLUE consists of nine datasets (SST-2 (Socher et al., 2013), CoLA (Warstadt et al., 2019)), MNLI (Williams et al., 2017), RTE (Bentivogli al., 2009), QQP (Shankar al., 2017), MRPC (Dolan & Brockett, 2005), STS-B (Cer et al., 2017), QNLI (Rajpurkar al., 2016), and WNLI (Levesque et al., 2012)) that cover a wide range of natural language processing tasks. Following convention, exclude WNLI and use the remaining eight datasets. We use the prompted form of these datasets available in PromptSource (Bach et al., 2022), which maps each example into a natural language requost-and-resposee form. During training, we randomly select a prompt templates for each example; during evaluation, we evaluate each example using all of its dataset's templates. In the T5-GLUE experiments in this paper, we concatenated all 8 datasets of GLUE and perform multitask training. T5 models were trained for 600k steps using a learning rate of 34-4, with 2k warmup steps, and batch size of 128. The AdamW optimizer was used with its default settings. We ran the ST-Gumbel estimator with
value of 10 and anneal rate of 1e-6 6 by sweeping in the range of {1, 10) and the anncal rate in the range of (1e-4. -5_ 1c. For the REINFORCE estimator, used the same values in Clark et al. (2022), le-2, B 5e and = 1e- The adapters here use swish non- lineriie in between.
B.2 ResNet-DomainNer
In the ResNee-DommannNet experiments, all the domains from DomainNet were concatenated to perform multitask training similar to T5-GLUE. ResNet models were trained for 100k steps with batch size of 128 and a learning rate of -3, with no warm up, using Adam optimizer. We used value of 10 and anneal rate of 1e-4 for the ST-Gumbel estimator by sweeping 7 in the range of (1, 10} and the anneal rate the range of (le-4, 15 le 18). The values of a, B, and 7 for the REINFORCE estimators are same as in T55-LLU experiments. The hyperparameter that weighs entropy regularization in Dselect-k chosen as 0.1 which worked best among {0.01, 0.1, The adapters also used a swish non-linearity in between.
C Expert dropout
Table illustrates the impact of expert dropout on methods that do not make use of metadata but learn adaptive routing, namely methods that are trained through gradient estimation and SMEAR. We conduct this ablation single seed to limit the amount. of computation. The results show that SMEAR benefits
19

---

Published in Transactions on Machine Learning Research (05/2024)
from improvement of 2.9% T5-GLUE, and Top-k achieves 1.2% and 0.2% improvement T5-GLUE and ResNee-DDmminNel respectively. As result, we include expert dropout for these two methods when discussed in the main text. However, expert dropout has negative impact on the performance of ST-Gumbel and REINFORCE methods, and thus, we exclude it for these two methods in the main text.
Routing
T5-GLUE
ResNet-DomainNet
Top-k
77.2
59.8
w/ Expert dropout 0.1
78.4 (+1.2)
60.0 (+0.2)
ST-Gumbel
78.3
58.3
w/ Expert dropout 0.1
77.2 (-1.1)
57.9 (-0.4)
REINFORCE
79.8
59.8
w/ Expert dropout 0.1
78.2 (-1.6)
59.8 (+0.0)
SMEAR
80.2
62.0
w/ Expert dropout 0.1
83.1 (+229)
62.0 (+0.0)
Table 1: Performance comparision of different adaptive routing methods w and w/o dropout on a single seed. The results indicate that SMEAR and Top-k method benefit from the expert dropout, while ST-Gumbel and REINFORCE are negatively affected.
D License
T5 is licensed under Apache 2.0. The ResNet model we used is licensed under BSD 3-Clause License. QNLI uses CC BY-SA 4.0 license. MultiNLI uses data sources of multiple different licenses (Williams al., 2017). CoLA, SST-2, RTE, MRPC, STS-B, QQP, and DomainNet allow non-commercial rescarch cases. Our code for T5-GLUE is based on Hyperformer code for ResNet-DomainNet is developed by us.
et al., 2021), which is shared under Apache 2.0. The
E Ethics Statement
We are not aware of any negative ethical implications of our work. Our work does not involve human subjects and is primarily focused on diagnosing issues with an efficient class of neural networks. While conditional computation has been used to design extremely large neural networks (Shazeer et al., 2017; Fedus et al., 2021; Du al., 2022) that have high computational costs (and, correspondingly, energy usage), our work primarily focuses smaller-scale models.
F Full results on T5-GLUE and ResNet-DomainNet
We show the full results of T5-GLUE in table 2 and ResNet-DomainNet in table
G Routing distribution in all routing blocks
Here, we present the routing distribution across all routing blocks in both T5-GLUE and ResNet-Domain learned by SMEAR. This is depicted in fig. 4, fig. 5, fig. 6, and fig. 7.
For other learned routing methods, refer to the corresponding figures for detailed routing distributions:
Latent Skills: fig. 8, fig. fig. 10, and fig. 11.
Top-k: fig. 12, fig. 13, fig. 14, and fig. 15.
ST-Gumbel: fig. 16, fig. 17, fig. 18, and fig. 19.
REINFORCE: fig. 20, fig. 21, fig. 22, and fig. 23.
20

---

Published in Transactions on Machine Learning Research (05/2024)
Average
81.61.0
80.2p.8
78.41.1
78.40.4
66.9p.9
78.51.2
81.016
78.2p.9
77.90.4
80.0n.s
81. $1.71.0
81.31.1
CoLA
mcc
33 33.8n.4
20.1
5.13.7
5.1
2.92.5
25.18.3
30.8s.3
2.02.0
1.317
14.1n9
33.76.1
33.16.5
QNLI
90.2
80 89.90.7
89. 89.90.5
90. 00.50.
80.0p.5
87.9p.9
89.31.
90.1n.9
89.1o.5
90.8 80.8n.7
89 69.80.6
89 6g.6p.
acc
MNLI
84.90.5
84.10.5
84.50.7
85.40.3
72.00.9
84.40.8
84.90.9
84.90.9
84.50.3
85.8n.4
84.70.5
84.40.5
acc
QQP
90.1n.2
90.20.2
89. 39.20.2
89.40.1
82.80.3
89.50.3
39.80.3
89.00.5
89.20.2
89.5n.2
1..
90.1o.3
acc
QQP
86. 86.9p. n..9
87.00.3
85. 6p.3
85. 7..
76.8 10.2
86.2 2..
86.4b. 4b.4
35.30.4
85.70.1
86. 1o.3
86. 80.8
86 8p.8
fl
STS-B
86. 6n.8
85. 55.30.
87.3 17.30.7
86.60.7
65.21.8
81.51.2
56.31.4
87.41.7
86. 36.80.8
87.2n.5
8.6
86.51.0
STS-B
pearson
87.0 17.0
85. 55.50.8
87 41.8
65. 42.4
12.2
86.61.5
87.71.6
86.91.0
87 4..6
85.71.5
86.
86
MRPC
aсc
86.92.2
86. 0..0
85 85.53.4
83.31.2
68.58.2
79.5p.s
25.
84.71.7
84.81.7
86. 12.02.1
87.714
85. (..8..
SST-2 MRPC
90.5
89 09.9
89.2 29.22.7
87.41.4
77.72.
85. 35.40.7
90.01.0
88.6 60.6
88.81.0
19.81.6
9..91.4
89.51.1
fl
90.9
92. 0.5
91. p1.00.2
92. 6
85.61.3
90.3p.5
90.8p.8
92.5p.
92.10.7
92. 6n.
91. 4
90.9p.7
acc
RIE
69.92.6
2..32.1
67.3 .3.3
70.23.2
58. 82.77
71.72.9
70.44.6
08.22.3
67.62.3
70.93.3
7..91.6
70.9
1x parameters
REINFORCE
1x compute
Latent Skills
ST- Gumbel
SMEAR 2x
Ensemble
Routing
SMEAR
Adamix
Hash
Top-k
Tag
Table 2: Full T5-GLUE results.
21

---

Published in Transactions on Machine Learning Research (05/2024)
Final Accuracy
62 62.0n. 0..0.
60 10.8
19.00.1
58 03.0
52. 42.1
61.4n.1
61.90.2
60.0n.1
58 53.50.2
60. 10.00.1
59. 19.30.2
60. 60.5u.1
62. 62.9h.1
62.8p.1
Sketch
56.0 56.0p.2
14.90.4
52.6 12.6p.2
51. 11.2
45. 45.7u.2
55. 55.1n.3
36.30.2
53.5p.1
11.904
53.6 23.6p.2
52. 12.50.4
p4.9
57. ...
57. 57.3g.1
Real
74. 1
72.90.1
72.30.1
72. 1..1
68. 13.50.1
74.1o.1
74.20.1
73.00.2
72.10.2
72.8..1
72.50.2
73. 30.
74.60.1
74.90.2
Quickdraw
62. 3o.
61 61.5o.1 10.0
59.0p.2
07.20.3
48. 68.6..3
61.70.2
61.60.3
10..2.33
58.10.5
60. 60. 40.3
09.50.3
60. 10.3..2
63. 73.70.2
63. 63.30.3
Painting
57. 8p.33
56.4 40.3
54.8p.1
54.10.2
49 80.8
58. .3.
58.90.1
35.8p.4
54.50.3
55.9p.2
55.10.3
56. 26.5p.2
58. 50.3
58. 58.40.5 40.5
Infograph
31.2p.3
29.8 29.80.3
27.90.3
27.00.2
23. 43.4
30.20.3
31.20.1
29.60.2
27.604
29.1
28 43.
29. p..3
32.30.0
31.8p.1
Clipart
64.2p.1
k3.30.3
60.20.3
58.9 08.90.2
53.5 u.3
62.8p.4
64.50.4
61.6p.2
59.90.3
61.3n.3
60.6p.2
62.7 82.7n.2
65.70.1
65.3p.1
1x parameters
REINFORCE
1x compute
Latent Skills
UT-Gumbel
SMEAR 2x
DSelect-k
Soft MoE
Ensemble
Routing
SMEAR
Adamix
Hash
Top-k
Tag
Table 3: Full ResNet-Domaing results.
22

---

Published in Transactions on Machine Learning Research (05/2024)
Clpart
6:19
0.17
0:18
0.11
0.14
0.23
0.12
0.1
0:28
0.16
0.17
0:18
0.23
0:15
0:17
0.18
0:18
0:13
0.18
0.25
0.16
0:25
0.12
0.09
0.23
0.14
0.13
0.12
0.13
0.26
0.08
0.16
0:34
0.13
0.14
0.14
0.12
0.12
3.16
0.11
0.88
0.11
0.18
0.22
0.18
0.18
0.39
0.14
Painings
0.18
0.12
0.12
6:15
0.14
6.3
0:12
0:25
0.21
0.11
0.14
0.18
0.13
0.15
0.27
0.15
0.17
0.14
0.13
0.11
0.39
0.15
0.11
0.11
Quickdrsw
0.35
0.28
0.08
6.1
0.1
0:16
6.03
0.07
0:18
6.5
0.65
0.08
0.5
0.04
0.2
2.08
0.11
0.61
0.04
0.04
0.06
6:18
0.07
0.14
0.16
0.29
0.12
0.21
0.21
9:12
0.14
0.15
0.14
0.72
0.14
0.19
0.12
0.15
0:11
0.14
0.21
0.15
0.12
0.18
0.27
0.17
0.07
Expart Bouting Blook 5
Expert Routing Block
Expert Routing Brock?
Expert Rearing Block
Clpart-
0.33
6:18
0:18
0.38
IMfgrrsh
Paining-
Quickdraw
6:11
0:37
0.08
0.06
Real
0:16
6:13
Sketch
0.18
0.12
0.15
0.14
0.12
0.33
0.41
0.09
0.17
0.16
0.09
0.09
3.00
02
0.2
0.25
0.18
0.09
0.18
0.17
0.19
0.31
0.08
Expert
Expert
Exper
Eppart
Sppert
Export
Expert
Expart
Dreer
Driet
Export 8
Exprrt 6
Exper
Epprt 2
Expert
Empert
Expert
Emplet
Expert
Expert
Expert
Explrt
Figure 4: Routing distribution learnt by SMEAR in the routing blocks of
Dselect-k: fig. 24
Ensemble: fig. 25, fig. 26, fig. 27, and fig. 28.
23

---

Published in Transactions on Machine Learning Research (05/2024)
Expart
0.04
0:59
0:06
0.01
0.22
0.14
012
0.14
0:33
0.6
0.2
0.19
0.0
0.01
0:02
0.02
0.0
0:09
5GT2
0:08
0.09
0.0
602
0.01
2.01
0:21
0.0
3.01
0.02
886
0.09
0:47
0.09
2.07
9:26
0.99
0.23
0.05
611
0.04
0.1
0-07
QURS
MPPC
5.05
04
0.04
1:17
20
0.20
0.01
5.07
063
0.04
0.03
0:25
0.29
0.02
04
0.34
0.00
0:53
0.0
0.05
0.22
9:00
2-
0.01
0.02
2:34
0:31
C.1
0.21
0.25
0.04
2/02
ST93
0.04
0:12
0.54
0.01
0.08
003
5:01
0.84
0.01
0:08
02
0.76
0.00
5:85
0.58
0.02
0.08
0:03
0.34
0.04
0:07
3GP
0:01
2.01
0.01
2/
2.02
0.02
0.83
0.02
0.23
0.25
0:01
0:01
0:01
MNLI
0.02
5.66
0:09
0.02
0.55
0.03
0.64
0.91
60
0.19
5.09
0.02
0:00
0.02
0:11
0:09
0:54
0.0
0.0
0:04
COlA
0
0:72
0:07
0:01
3/
0.11
0.0
1212
5.00
0.81
0.05
0.0
0.50
0/07
0.26
0.61
0.11
0:111
0.08
0.04
0.04
0.11
0:15
0.88
0.10
0.35
0.08
0.1
013
0:11
0.
Export Pouting Block
Duse thouting Biccks
Expert foudi
Slook
Expent Souting Stack
ATE-
0.03
0.0
0:51
0:30
50a
3:01
0.14
0:09
5.05
0:81
07
0.01
0:51
0.3
0.04
0.00
0.10
5:07
0.00
6.15
5:3
0.28
5.05
0.00
0.44
0:02
0.17
0/12
5:03
0.07
0.15
0.07
2ST2
0.01
9.0
0:21
0.0
0.2
0.21
0.01
0.01
0.04
0.21
0.0
9.0
0.1
0.0
0.84
0.88
0.0
0.0
0.05
0.02
0.01
0.01
0.01
2.01
0.0
0:08
0:01
0.02
0.0
MARC
9.0
0.0
3:71
0.29
0.0
0.0
0.0
0.0
0.07
0.15
0:28
0.05
9:02
0.34
0.03
0.01
0:01
0.0
0.71
MNLI
0:25
9.0
6.9
0/12
0:01
9:01
0:37
6.26
0:27
0.64
0:19
0:95
9.0
0:16
0.35
6.08
0.01
3:08
6:1
0:14
0.04
9.37
0.01
0:25
0:25
0:27
0.09
0.05
0.0
0:10
0.0
2.0
0:99
0.0
3.
0.2
0.0
0.0
0.0
0.0
0.0
0.0
0:00
0.21
0.0
0.
0.42
0.02
0:01
0.07
2:21
0.24
0.01
0.02
0.3
0.01
0.34
0.07
0.0
0,02
0.04
COLA
0.0
9.8
00
5.00
0.0
0:02
0.02
0.0
2:47
5.03
0.0
0.87
0.02 0.01
0.01
0.02
0.03
0.02
0.00
0:01
Export Bouting Block
Expert Routting Slock
Expert Souding Slous
Expent
Readdig
Block 12
RTE-
0.0
0.21
0.02
0.04
0:27
0.99
0.02
0.0
0.0
8:09
0.0
0.09
0.0
9.0
0.06
0.09
0:01
0:01
0.02
0:14
0.06
0.0
4153
0.0
9.0
0.2
0.0
0.0
1/
0.0
9.0
0.0
0.32
0.98
0:25
9.55
0.09
0.05
0.0
0.0
0.0
0.0
2GP
0.0
3.2
6.0
02
0.
0.2
0.0
0.0
2.0
0.0
0.0
0.0
3.2
0.0
0.0
10
0.01
0.01
0.52
2.0
2:01
9:37
0.00
0.0
0.0
0.0
0.0
0.0
0.0
MNLI-
0.0
9.0
0:24
3.0
9.14
0.27
0:36
5.0
0.0
0.0
9.0
0.0
0.0
0.11
5:08
0.36
0.2
0.09
0.02
0.12
0.82
1.0
0.0
0.0
0.0
0.0
0.0
GNL
0.0
10
0.0
0.0
0.0
8.0
0.07
0.01
0.05
0.52
0.02
0/01
0.0
0.0
008
0.12
0.08
0.09
002
0.01
0.07
0.1
2.01
0.02
0.01
0.0
2.01
2.0
0.04
0.02
0.02
0.50
6.0
0.95
0.01
MTCC
0.8
0:12
0:19
0:21
0:12
0.07
0.01
0.23
5.
0.2
0.0
0.8
3.0
0.0
0:77
0.00
3.0
0.01
0.25
8:6
0:83
0.0
0.01
0.22
0.0
0.01
5.0
6.0
0.24
6/15
ST53
0.04
0.55
0.04
5.06
0.22
0.31
0.34
0.0
0.0
5.0
0.0
0.0
0.01
0.00
0:27
0:12
0.02
0.01
8:40
0.02
0.0
0.0
0.0
5.0
3GP-
9.0
0.1
0.00
3.09
0:53
0.02
6.16
0:12
0.0
9.0
0.0
5.01
0.29
8:99
0:02
6.01
0.0
0.02
0.6
mm
COLA
0.0
0.0
0.02
0.
0.0
0.0
30
8.90
0.0
0.0
9.0
0.0
0.06
0.0
2.0
8:89
0.08
0.01
0.0
0.01
0.1
0.0
0.0
6.0
0.0
3:93
0.36
0.0
0.0
Expurt Routing Block
Export Routing Block 18
Excert Blcck19
Export Rooting Blckk 20
RTE:
0.37
0.22
0:11
0:02
0:25
0.03
0.1
0.1
5.0
2.01
02
b.02
3.2
0:70
0.0
0.0
2.0
0.01
0.0
8:87
0.5
0.12
2.0
0.0
0.01
0.0
0.00
6.0
0.0
0.39
0.0
0.01
3ST2
0.02
3.54
3.59
0.01
0.00
0.06
0.03
0.01
50
0.09
5:01
0.0
3.0
0:07
0.0
0.67
0.11
0.0
0.0
0.0
0.6
0.0
0.54
0.01
MERC-
5:05
0.01
0.05
2.01
0.03
5.24
3.0
0.0
0.0
0.00
0.0
0.0
0:01
0:08
86
0.38
080
9.0
9:87
9.0
0.0
0.6
0.0
0.0
MN/
2.1
0.15
0.03
5:13
0.27
0.24
0.0
0.0
0.52
0.21
0.25
0:2
2.0
0.0
0.0
3.0
0.0
0.0
0.0
0.0
0.0
0.0
GNL1
0:01
3.8
0.02
5.01
5:50
6.04
0.03
0.3
3.0
0.5
0.0
0.6
3.0
1.5
0.0
3.
2.5
0.0
0.0
10
0.5
0.0
8.0
6.5
0.0
5.
6.5
0.0
10
0.0
COLA
0.01
0.02
0.09
0.01
0.09
0.03
0.01
5.0
0.0
0.01
0.01
9.0
0:00
0.0
3.0
9.0
3.08
0.0
0:03
0.0
0/0
0.0
0.0
0.0
0.16
0.0
0.0
01
0.0
Export bouiing
Stock:11
Excort Rouding Slock 22
px Routiin Stock
Expo Bou Blckk 24
RTTE
0.0
2.0
0.01
0.00
0:03
0.01
02
0.9
8.0
3.0
0.0
0.74
0.20
0.0
0.79
2.04
0.01
5ST2-
S798-10
0.0
0.07
0:25
0.22
3.07
0.21
0.8
2:21
0.25
0.0
3.0
0.2h
0.7
2.0
0.0
0:57
2.0
1:12
8.0
0.5
0.01
0.01
8:00
32P
0.0
3.0
3.14
02
20
0.2
0.62
5.24
2.0
0.5
0.0
0.0
3.2
0.0
0.0
10
2.0
0.0
0.75
2.0
0.5
0.0
0.0
0.5
0.0
5:2
0.5
0.0
SUOT
0.0
MNLI
5:02
0:00
6:48
9.07
0.
8.8
19
0.
3.5
6.0
3.0
0.0
8:49
5.
0.0
0.02
8.0
6.5
0.01
6.01
8:59
0.02
0.02
0.
3.0
0.0
0.29
0.0
F
Enpet
Exter
Exper
Byper
F
2
3
4
Eeper
Exchet
Exper
Fower
Daper
Eipert Z
Drwwn d
Frone
F
Experr
Eepert
Dener
Exterr
Eypert
Figure 5: Routing distribution learnt by SMEAR in the encoder routing blocks (1-24) of T5-GLEE
24

---

Published in Transactions on Machine Learning Research (05/2024)
Expart
0:25
0.06
0.344
0.33
0.25
0.02
0.01
0:10
0.0
0.2
0.0
0/0
6.R.
0:23
0.0
0.0
5GT2
0.22
0.07
0:81
0.05
0.54
3.0
0:17
0.02
0.01
0:50
2.03
0.22
0.18
2.0
0.0
6:78
3.0
2:01
0.01
0:09
0.07
0.0
00
0,2
MPPC
0.02
0.09
0.02
0.00
5.06
0.51
0.1
0.55
5.02
2.01
2.04
0.01
0.23
0.21
0:55
04
2.0
0.27
0.0
2.0
0.0
0.0
5/0
0:79
0:27
0.01
245
0.0
0.0
ST93
0:26
0:21
0.00
0.04
0.04
0.04
25
0:15
6:11
9.0
04
0.0
0:01
0.01
o0
0:25
3GP-5
0.03
2.58
0.02
0.12
0.05
5.3
0.02
0.21
0.01
0.3
0.0
0.5
2:34
0.0
MNLI
0:06
0.53
013
0.44
0.02
0.00
0.92
0:50
9.0
0.0
00
0:04
0.0
COLA
0:05
0.26
0.01
0:17
504
6.03
0.24
0:45
3.0
0:81
0.01
0.9
0.59
0/07
0.14
5:43
0.0
0.0
0.0
3/
D2
0
0.F.
0.04
0.2
0.51
0.
0.0
D
0.2
Excer Sooetigutiigg Blook
29
Expert Booting Block
90
Exper foutting Block
Expon Routing Blcck
RTE-
5:02
0:01
0:51
0:05
5:33
0.07
0.7
0:02
5.00
0.81
0.0
0:72
0.0
0.19
0.0
0.0
3.0
0.0
0.0
8:79
8.
G.
0.2
0.0
0:09
0.07
0:12
0.0
0:28
5ST2
0.01
0.01
0.9
0.92
0.01
0.09
0.21
0.01
0.51
0.08
0.0
0.0
2/01
0.0
0.0
3.0
0.50
0.01
01
GI
0.0
0.99
0/
0.0
0.0
0.0
60
0G1
0.0
0:13
6:10
D/07
MARC
0:08
0.51
0:12
3.01
0.0
5.25
0.30
0.22
0.02
0.21
0.0
0:11
0.0
0.0
02
0.0
MNL1
6:02
3:28
6.
9.6
0.07
0.31
6.07
0.21
0:11
9.05
0.01
2:16
0.0
0:41
6:17
3.67
0.2
S/R
8:1
0.0
0.48
0.0
0.28
0.0
0.07
0.0
0.0
0.0
6/97
0.03
2.0
0.3
0:93
0.01
0.2
0.21
0.0
0.01
0.0
0.21
0.8
2:12
0.0
0.0
2.
2.0
0.0
0.0
8:50
0.0
0.0
2.0
0.02
0.0
0.01
0.0
0.0
0.0
0.0
9.99
0.0
COLA
0:00
0:17
0.09
0.27
0.0
0.59
5.0
0.06
0.0
0.0
0.14
0.D
0.00
6.2
Expert Roating Block
Expent Boading Block 34
Export Sourting Stock
Exper Routing Block
36
RTT
0.0
0.2
0.02
0.0
0:34
0:02
9.
0.0
0.0
5.81
9.0
0:17
0.0
0.0
4153
0.9
3.0
0.12
0.07
0.0
0.9
0.00
0.20
0.0
3.01
0.0
0.01
2.0
2
0.0
0.0
0.0
0.0
0.0
0.0
2GF
2:42
0.2
6.
0:10
3.0
5:19
0.0
0.0
2.8
0-2
0.14
0.0
3.2
0.0
0.2
0.0G
2.2
0.0
0.
2.
0.2
0.0
2.0
0.01
0.0
0.0
6.0
22
0.0
022
MNLI
0.0
6.56
0.2
0.99
3.0
0.04
0.0
0.02
0.00
0.07
9.0
0:33
0:24
0.02
3.07
0.0
1:15
0.0
0.2
0.0
0.0
3:24
0.0
5.20
2.0
0.98
0.0
8.0
0.0
80
0.0
0.02
0.0
0.0
5.50
0.0
0.62
0.0
0:03
0:17
0.0
0.0
0.0
0.0
0.10
0.79
0.02
MTCC
0.8
0.22
3.21
82
6:79
0.0
0.0
5.
3:17
0.12
0:41
3.0
0.0
0:32
5.0
6.71
5.65
0.0
5.0
0.5
5/8
0.0
0.
0.07
0.52
0.09
0
0/14
0.0
STS3
0.0
8.52
50
0.2
0.0
0.8
0.0
5.0
0.14
0.0
0.0
0:72
0.
0.0
0.3
3GP-
0.0
0.51
0.0
0.8
0.20
0:51
3.0
0.6
6.67
0.0
5.30
0.6
0.0
0.5
6.0
0.68
COLA
0.0
0:52
0:81
0.0
0.0
3.9
0.78
0.03
0.24
9.0
0.0
9.0
0.29
0.0
0.0
0/9
0.0
6/
0:11
0.9
0.2
6/0
0.0
0.0
0.0
Eepert Rouding Slack
Excert Rousing Slack
42
Expurt Rourtig Slack
Bouting Blckk
44
RTE
5.02
0.0
0:75
0:17
5.00
2.07
0.0
0.02
0.52
0.00
0.13
0.0
0.2
0.2
0.0
0.0
0:12
2.07
0.22
0:12
0:01
0.74
5.2
0.04
0.0
0.0
0.0
0.01
0.0
0.07
2ST2
0.0
9.
0000
0.0
0.0
0.0
0.04
0.0
0.0
0:76
0.2
0.00
0:83
0.01
3.01
8.0
0.
0.0
0.03
MERC
525
2.0
0:3
0.03
0.0
0.0
0.08
0.72
0.09
0.17
0.0
0.0
06
0:87
0.0
9.0
0.16
0:18
0:07
0.84
0.02
0.5
0.
MNI
5.25
3.0
0.59
0.2
034
2.1
0.78
0.0
2.34
0.0
0.28
2.21
3.02
0.7
0.18
0:15
0.02
0.0
0.0
0.27
0:72
GNLI-
0.8
3.8
3:9
5.01
0.01
0.2
0.0
0.01
0:55
0.02
9:42
0.8
3.2
0.0
0.0
0.0
0:31
5:05
8.26
0.05
0.05
0.08
8:36
0.66
0.5
0.0
5.0
6.5
12
0.0
COLA
0:9
9.0
0:33
0:02
3.
0.0
0.0
0.0
0.06
0:41
0.17
0.0
0.28
0.0
9.0
0.00
9:25
0.06
0:17
0.02
0.06
0.04
0.0
0.0
0.
02
0,0
6:19
0.0
ExpetBbuttty Bookk
Excort Bouting Block
Expost Routtr Stock
Expo Bouttp Blck 45
RTE
0.92
0.52
0.01
0.0
0.0
0.06
0.04
5.02
0.52
2.03
3.63
0.14
0.5
0.79
S798
0.0
3.0
0.09
5.27
0.8
0.23
0.05
0.27
0.79
0.26
0:7
0.11
2.0
0.0
0.0
8.0
0.5
0.0
0,3
32P
3:20
2.0
6.2
0:77
20
0.2
0.0
0.0
0.07
0.02
0.04
0.02
0.57
2.05
0.24
0.25
2.0
2:63
0.0
2.0
0:35
0.0
0.8
0.0
0.32
0.0
0.0
2.14
0.0
0.26
0:42
0.0
MNLI
60
8:2
0:00
5:78
3.13
8:16
8.8
5.06
0.07
0.14
074
0.7
3.0
0.0
6.6
8.
0.6
614
0:36
2NL1
0.0
0.04
002
0:18
0.01
0.0
Exter
Exper
Bypert
Popert
2
3
4
Eeper
Exsher
Epper
Dipert Z
Drwwr
7
Frant
Experr
Dener
Exteer
Eypert
Emaen
Figure 6: Routing distribution learnt by SMEAR in the decoder routing blocks (25-48) of T5-GLUE
25

---

Published in Transactions on Machine Learning Research (05/2024)
Expart Routing Black
Rexting Stack
Expars Routing Stock 51
Expent Rosting Black
R7T
0.94
3.01
0:53
0.09
0.09
0:01
0.01
0.0
0:21
0:17
0.3
6:2
0.1
0:01
6:0
0.0
0.2
0.00
0.09
0.23
0:01
6:98
0:15
0::1
2/02
5S12-
0.44
0.22
0444
0.07
0001
0.04
0.32
0.07
0.0
00
0.0
24
0.0
0.0
1.3
2.0
0.0
0.0
0.0
0.0
0.22
0.35
0.50
0.03
0.12
0.42
0:10
0:17
0:01
0:12
D.62
2/02
MRRC
0.01
0.0
04
3.0
0:42
0.0
32
0:4
6.3
6:17
0.0
0.17
0:15
01
0:45
PM
9198
0.42
0:02
0.05
5.13
50
0.0
0.0
0:32
0.0
0.0
0.0
0.09
0.08
0.00
12
22P
0:35
0:24
0:03
5.07
0:52
0:07
0.13
3.0
5.0
6.
8:60
0.14
0.06
0.32
0.0
0.0
0:32
0:15
5/11
0.04
0.04
Expart Routing Rtock
Rouriing Bandk
Faneet
Randi
Fxneet Rexting Rock s6
RTE-
0.0
0.2
0:12
2.0
0:33
5:03
0.03
0.29
0.01
0.24
0.02
5.07
0.0
0.0
0.0
0.64
0.
02
0.0
D:14
5ST2
0.0
0.0
0:21
8:57
0.0
0:42
0.0
0.0
0.2
5:12
0.02
0.01
5:01
0:67
0.20
0.00
0.0
0.0
6.
0.0
0.0
0.0
5.0
0:00
0.0
0.0
5/3
0
D.D
MXFC
0.0
0.0
S/D
0.0
0/2
0.24
3.0
5:19
0.00
0:21
6:09
0:22
0:11
6:14
5.24
0.09
0.0
0.
0.0
0.02
0.79
6.0
0.19
0.07
3.54
02
9.0
RU
G/D
0.0
ST53
0.0
0.0
24
0.0
0.
0.0
0.02
0:25
0:03
0.05
0.19
0:27
0.09
0.0
0.01
6/
0.0
0.0
04
0.0
3.0
0.0
3.0
G/D
0.0
3CP-
0.0
0.0
0:32
3.2
0.02
0.04
0.20
0.02
0.5
0.02
0.41
0.0
0:31
2:20
MNLI
GNLI
0.0
0-
0:9
0:66
0:21
0.07
0.94
0:54
0:16
0.0
0.2
0.66
Roading
RTE-
0.17
0:03
0.04
8:50
0.00
0.00
0.99
0:64
0:01
0:01
0.54
0.0
0.0
0.22
0.02
0:78
5GT2
0.01
0:53
0.26
0.52
0:01
0.01
0:01
0.00
0:83
0.01
0:14
0:98
0.03
0.26
0.05
0.01
1.9
0.0
0.0
0.0
0.0
32
0.0
27
0.0
9.0
0.0
0.52
MtFC
2:22
0.26
0.14
222
D.04
0.00
0.05
0.02
5:17
0.15
0.23
0.11
0.01
3:29
0.01
0.07
0.0
0.0
0.0
0.0
2.0
0.0
0.0
5-5
02
0.0
0:21
0:72
0.0
STR2
5.00
5:51
0.04
5:18
2:04
0.02
0:01
0.14
5:75
0.07
0:19
0.04
2.07
0:21
0.02
5.04
0.00
0.0
0.2
0.0
0.5
9:54
5.0
0.0
0.07
0.0
0:85
0.0
0.0
0.0
0.1
8.50
0:21
53
0.02
5.10
0:28
5:35
8.0
0:52
0.09
0:2
3.7
0.0
0.
0.0
MN.
0:27
0.52
282
Opaint
Erowr
Ecaer
Exper
Daker
Browr
Dowrr
Sown
Eppert
Cowe
Erper
Guber
Enperr
Exowe
Ergar
Gperr
Exper
Brpar
Erowr
Laser
Epprr
Exter Zuaer
Ecwwy
Expert
Expher 6
Empor
Figure 7: Routing distribution learnt by SMEAR in the decoder routing blocks (49-60) of T5- GL-G
Ooerd Pheiigg
Eqpe:Bad Back
Experilluting Beck
Opeen-
6:5
0.35
20
E:55
SD
6-55
0.25
2.3
SO
6.0
SD
0.23
0.2
0.30
0.26
2:41
0.44
0.25
6.5
Real-
0.2h
Saith
6:5
0.24
0.2
515
22
6:18
0.2%
0.29
0:29
S/D
1.2
0:25
0:5
2:24
Reating
Expe: Nouking
Experifbuure
Opper:
6:2
0:23
0.2
5:5
26
60
6:2
0.2
SD
0.
0.20
630
0.25
3.5
2:25
0u5
3.22
2.0
55
0.5
30
00
822
0.21
20
023
0.25
0.22
022
002
6.0
0:2
0.22
Fest-
2at
02
28
22
62
2:21
2:30
80
E/
0.21
06
60
48
80
E23
0:30
80
Saterr
Spper
Coppet
Severa
Appera
Geser
Spwer
Esper
Boser
Rovre
Figure 8: Routing distribution learnt by Latent Skills routing in the routing blocks of Res Domain
26

---

Published in Transactions on Machine Learning Research (05/2024)
0.23
0.
022
0:22
0.2
10.20
3.24
0.25
GOT
0.0
0.22
0:20
0:22
022
0:01
0.25
6:27
2.0
022
0:31
0:20
94
6-3
6:50
5.0
0.3
6.0
6.3
14:25
0:35
6:55
0.
0-2
0.0
9.35
0.6
9.0
00
0:24
0/3
0:10
5.26
3.6
3.94
6:63
0:33
0/2
GD
9.2
80
00
033
3:5
0:3
50
0.2
00
0.0
2.5
0:59
0:50
20
00
80
0.5
3.0
0.2
20
R2
0T5
5/2
022
2.5
50
0.0
0:33
0.5
C0
0:53
1033
114
00
2.5
0.25
22
222
0:22
0.2
0.0
222
622
0:32
0.2
C.
0:55
0.22
6.21
2.
D14
0.21
CO
0:23
0.29
6.0
0.2
0.22
0.0
0.2
C.01
Dupert Roudin
Eccert Roating Block
Eapar Rou Rock?
Rxpert Routin Block
69
00
12
6-3
0.2
69
4.0
033
6:5
0-2
0.0
18:55
0.
02
6o
0.32
0.
0153
0.0
MM2
C0
00
36
6.3
622
2.0
3.30
0.33
0.5
26
60
6-2
0.0
00
5.20
3.0
02
60
622
0.25
0.05
3.2
0:25
0.
2.0
6:22
T02
00
00
88
00
28
60
0:04
0.0
00
0.0
05
CC
1W
0.
0.0
CD
27
<10
2:25
00
26
2:23
0.25
50
2.
0.22
0-2
0.0
22
3.0
002
0.0
0.5
2.0
0.25
0:2
0.22
0.25
0.3
0.2
0:17
222
0.0
02
0.21
0.29
0.0
0:20
2.0
2.26
620
2.0
0.2
0:
colk
c0
00
9.4
6-3
0
3.0
0.3
6.
6.0
3.4
6O
0-3
0.6
3.2
0.23
0.22
0.50
3.0
0-3
0.3
0.24
2.00
2:10
6.0
6:0
3.0
0.3
Dapers Boadin
Capers Roxting
Souting
Caper Rocting
0:34
0-0
0.4
6-3
6G4
3.0
02
0.32
0.0
3.4
0.21
0:21
0.21
0.15
6.21
0.12
0.2
0.2
GO
0.2
0.2
0.2
0.22
C001
6:25
0.24
6O
0.2
0:25
0:25
88
9:24
00
2:
0:
00
5.0
DD
0.39
5.15
0151
00
0.
0:1
0.20
2:29
0.3
0:22
6.0
00
0:15
00
GO
00
5:29
26
0:51
C0
5.0
2.5
0.22
0.21
0:22
225
2.15
0:02
0.0
2.0
0.2
21
110-
22
28
25
0.25
D0
00
00
0:25
0:35
DCS
0:25
05
3.25
0.27
3.2
0.
02
0.25
0.0
5.25
3:24
022
0:23
0.6
0:3
0-22
0.2
SD
2:35
0.0
02
0.4
22
C0
6:20
244
6.3
0:30
2.0
04
0.32
0.0
0.4
0:22
0-
0.24
0.0
0.24
0.0
0:22
0.3
0.29
2.22
0.2
07
0.6G
0.03
6.0
0.0
0.0
6.
Dosrritcuting Bock
Diper: Foxtngg
Excant Rouking Bbok
Capet Tacting
0:17
00
2:55
6:17
0/66
0.17
022
6117
0.0
0:27
Q2
CG
0.32
0:22
.23
0.0
0:22
0.01
600
0.2
0.25
0.25
0.25
0.24
0.0
241
022
0.25
6O
M912
8.0
60
0.0
3.0
0.3
0.2
0:22
03
0:20
0.00
827
265
5.0
D0
5.00
2:24
0.24
116
0:255
0:55
1.255
0:35
DUS
60
0.21
0:85
0:30
0:25
0.
G.0
0:29
00
98
2.
0.29
0.0
9.5
9:29
0:2
0/23
1:20
0.
00
0.6
0:29
2.29
0.7
2.
0.5
0.0
RXZ
0.7
22
DX2
0:33
00
044
0:33
0.0
0.2
2.0
0.0
0.4
0:32
0:00
0.0
00
2.-4
2.0
0.2
0.2
22
0.5
0.0
2.0
0-2
GO
0.2
Dyper Routing Bockk
Supert Seeting oocia
Eclent Rolkiing Stoicto
Exsent RoLddig Slidc 20
02
2.2
215
0:20
0.15
2:27
0/12
0.22
0.0
2.
D.0
EAT
0U23
0244
024
0.22
0:11
0:23
0.23
2:07
0:17
0.22
0.5
032
0.0
23
2/1
0.32
0.2
WM2
022
00
3:5
00
5.0
3.5
332
0.33
0.
2:37
0.2
0.0
0.5
0.5
TD2
8.29
0:51
60
6:55
3.6
3.
0.3
0.2
0.0
0.32
0:30
2.25
2:00
0.0
02
0/2
38
0.2
02
0:25
0.29
R-2
0.0
0.
6:2h
9:
SR
5.2h
2.25
5.25
0.5
80
00
0.23
3.55
624
0.22
02
0.49
8.0
044
2:27
92
021
0.27
0.2
0.6
0::2
0.7
21
022
0.22
0.5
0.22
0.22
0.6
2.0
0.22
CAN-
20
00
6.2
0.0 0:02
0:21
8:43
0:7
24
0.22
02
0.22
0.12
2.2
022
21
022
0.2
0.2
22
5.0
0.0
2.0
0.2
DO
DoersRouttig g Bock 21
DasenRathgg Brokk
paper Souting @ 00420
Expert Rouding ng Block 24
co
0:29
002
3.00
5.0
0.25
0.24
0.5
0.0
0.
022
2.00
0200
003
0102
822
22
0.35
TD2
60
6:2
6.2
0:5
0.25
0.0
3.4
0.2
0.0
0.
0:27
GD
27
2.0
6:27
0.29
2.
me
6:19
60
6
03
2.6
6/6
0/3
00
3:15
0:25
D2
0/S
0-7
0.35
6nd-
00
0:52
002
6.2
0.0
033
2:0
0.00
5.0
24
332
832
231
DOT
8044
80
0:12
2.0
0,2
S2
RD
02
2.2
05
22
0.22
0.0
2
2.24
0.24
C:4
244
822
3.00
00
8.0
3.8
00
00
2.01
622
3.00
017
2.0
022
Aguerg
Banet
Saler
Lanets
Sabura
Guner
Laker Baneea
Equar
Lauera
7
Squrre
Poper
Sapest
Sours
Lawrts
Figure 9: Routing distribution learnt by Latent Skills in the encoder routing blocks (1-24) of T5-GLUE
27

---

Published in Transactions on Machine Learning Research (05/2024)
0.21
219
2.00
CO
0:21
0.22
0:19
0.0
0.2
0.0
0.00
6:00
0:27
0:12
D00
0:04
0.0
2.16
3.99
1.30
0.00
0:11
3:24
0:12
0.
0:01
1:27
0:35
3.0
0-2
01
5:24
01-31
0.00
0-6
9.5
0:00
DUS
6:20
Y02
017
0:03
R:7
0:19
0.00
0:13
0.
0.
5.0K
OU71
2:55
0:75
0.1
5.09
0.5
0
2.75
2:27
0.18
0
0.
RIOS
114
0.12
21
6:14
OUT
0.0
21
022T
0.21
0:17
6:77
0.03
C.5G
C.O
0:02
0:17
15
0.0
0.05
6:7
0:12
0.17
5.02
D:77
0:7
0.16
0.0
0.12
0:02
0:09
0.16
012
2.12
225
2.00
0:34
0.
Expert Roxting Blok
Excen Routing Block
Excert Roudiing Slock
Exeent Rouding Block
32
60
60
3:25
0:04
0-3
0.20
0.00
0.4
0-
1.3
0.01
5:00
000
SIM
0.24
AX14
0:13
0.06
0.0
0:35
0:92
MM2
0:42
+:2
2:1
6.3
0.
0:33
0-
60
0.06
1-12
6:17
0/6
2
0.15
0.0
0.04
0.6
2:10
C:14
0.
3:33
0.1
0.
3.2
0.05
0.31
2.0
6-11
0.2
6:17
21
0:22
1.06
R22
R02
21
0:16
25
0.0
CD
<10
00
02
27
2.0
0:1
C0
24
2.6
011
012
0.00
0.0
D:7
2.0
0:77
0:28
D02
2:28
02
0.22
0.02
0:12
0:04
0:33
0.60
0.00
0.2
0:07
2.0
0.15
0.37
0:72
0.00
cola
60
6:92
9/1
0:12
611
0.
6:13
8:17
0:00
6:0
1G
0.1
0.14
0.04
0.14
3:21
0:00
6:00
0.0
0.60
0.02
6.0
0.0
0.24
0.24
Expert
Block
Excutt Roating Block
Capert Routing D0C
Expert Routing
36
6:33
6:01
2.4
0.2
0-3
0.0
3.4
3:27
0:27
0:22
0.12
0.22
3.0
COI
0.21
0.2
22
0.12
0.02
0:01
0.01
0:35
0.2
20
00
9:24
0/22
0.25
8.0
0:25
0:21
0/5
0/02
0:15
0:55
2.30
3.0
20
015
0/
6.0
0.0
07
0/
2.88
0:00
215
0.72
0:20
01
0.2
0.21
0.04
2:4
022
2:25
22D
0.00
0:22
110
2.00
22
0U2
0424
0.0
DUOT
0:18
6:01
1:19
611
6:12
03
enk
0.18
0.
15
DOT
0.77
0:18
0.70
0.79
2.00
2:10
5.71
2:00
0.19
5.0
3.0
0.00
0:39
0.7
0.2
0.0
02
0.4
CO
6:12
62
6:01
2
0:44
0/18
0.15
0.22
0.02
0.12
0.21
0.22
0:15
2.01
0.3
3.0
0.3
8:04
333
6:14
0.4
2115
6:12
0.01
6/1
1
C.01
2.17
Expet Tooting Blck
Excet Routing Slack
Capert Routig
Ecant Rouling
CO
2:04
0.11
0:14
0.0
0:24
0.20
02
0.21
2.4
6.0
G/1
221
0:77
0.02
0.21
220
0:02
0:25
0.20
60
0.2
0.0
0.15
0.25
6.
0.0
0:25
022
0.25
02
222
M912
1:20
6:11
38
3:58
0/3
28
R-2
224
6954
0.
0:04
0.
0.0
0/3
SD
000
0UZ
0:21
5.71
5:11
32
012
8.51
0:22
DUTT
6:12
C0
6:5
622
2G
0.20
6.0
0.20
3:33
0:25
0.0
0.12
0:30
090
0:21
0.0
21
0:20
00
R11
0.2h
0:0
0.
R-2
0.
0-2
9.5
0.7
5.2
0:37
0:08
0.0
R:04
DUD
3.24
2:07
0.22
0:3
6:24
2424
0:13
2.
0.24
0.4
0:22
0:4
0.03
0.27
0.17
2.0
6:2
1
0C1
0.17
2.2
6:12
0.0
22
022
0.20
GO
ExcemRRuuthe Bock
DadenRouthhg Book
Capar So ting Rock 43
Expen Rouring Block
61
0.34
2:00
04
6.2
8.27
2:21
022
0.25
5.0
25
0:33
0:11
22
R:2
025
0.7
0.5
0.2
0.32
DO
0.57
2.0
0.5
2.1
0.0
004
0:1
C3
WM2
60
3.6
0.2
CO
3.8
6/1
00
0:33
0.5
0.0
TD2
8:7
0%
00
00
26
0.2
0.23
0.0
DUD
0-33
0:
3.25
8:
26
0
0:75
D:74
3:
S
00
0117
9:51
9:17
0.
1
D177
2.25
5.25
0.5
2
80
00
23
225
0
004
02
83
2.01
5.22
0:22
92
22
R-2
0.25
24
0.12
2:27
2.
62
0.21
0.21
32
5.0
0.0
2.0
0.2
22
0.2
CAN-
3:28
6:57
0.4
6.2
0.0
2.0
02
0:47
5.0
0.25
R:22
R:77
27
204
0.27
80
0.2
0:22
0.00
0,222
0.2
0074
0,20
5.0
0.0
0,2
0.29
DO
Expor Rowting Bock
45
Eosas Routing Block
Cipen Souting 8 B0C47
Expert Rouring Block
43
co
3.8
022
8.0
0.00
0030
00
0:02
0.29
0.46
0.2
00
TD2
C0
00
3.2
0:25
0:5
0.20
6.0
0.25
3:25
022
0.
0:02
0.24
0.24
2.24
0.23
6.25
0.25
022
6.
me
6:5
28
40
S/N
S:S
RD
3.0
87
SD
6:W
0117
1:17
1/14
00
2211
6.2
8.29
0:23
2:20
0.25
2.01
0.20
3.29
02
R22
8.25
80
523
2:7
0.20
RD
R2D
0,23
2.2
023
5D
8.25
0
022
0.22
622
32
244
00
0.18
3.1
8.11
02
0
366
3:30
634
00
0.11
R23
0
8.20
0:2
Equera
3
Bamet
Lanera
Sabura
Guner
Laker
Equar
7
Squrr
Poper
Sapest
Eaurs
Inpprt Speere
Loorts
Figure 10: Routing distribution learnt by Latent Skills in the decoder routing blocks (25-48) of T5-GLUE
28

---

Published in Transactions on Machine Learning Research (05/2024)
Block
Exentt Routin Bissuit
Erpar tlociin Block 92
0.02
5.25
60
0:22
0:2
0.0
02
222
CO
CG
R22
DO
0:02
00:27
0:43
2.00
0.0
0.0
0.2
0:53
OL27
00
DO
0.0
0.04
0024
682
3:33
0:5
60
00
00
0:07
0.2
00
0
00
0.0
100
0.0
0:35
0:3
0:00
3.0
0.0
0.5
0.
80
0.0
000
2.0
0.5
4190
222
5.0
50
500
2:23
50
0.3
0.01
0.05
0:32
0.
2.31
2
0:22
0:22
0.
0.0
110
00
2:2
0:
2.0
00
2.0
0.2
2.5
0.0
030
00
0.0
0:20
96
0:21
0.
6:91
A
0:11
0:11
0:29
0:30
0-3
00
0:14
0:05
0:21
2.0
0.
0.123
0.33
0.
0.0
0:27
0:19
0:32
0.0
0:01
6:0
0.01
0.24
0.25
2.04
0:2
0.
0.5
0:32
02
0O
C0
0.0
00
0:23
2:30
0:22
00
2.0
0.2
0J
0022
022
0.0
3:37
600
0:02
0.2
06
0.
S0
0
6:32
0/
2:27
CS2
00
3.0
CUP
1
21
62
00
60
0O
0.D
1180
C0
60
8:20
0
0:
02
0-
0:27
60
3:3
2:27
00
0.0
0-
2.0
00
00
2.0
0-2
0.2
GO
CO
00
22
0.2
0:00
2.D
0.2
900
2:33
00
60
00
0:32
92
32
C2
0.2
6.0
0
90
22
0:17
RUT
OUT
2.D
GUT
5:17
222
00
60
022
022
2.
C0
8:25
3:
60
80
60
5.
0.3
2.0
2.18
0
Q2
3.2
02
60
0
02
0:35
0.0
6.0
0:23
0:24
0:00
24
02
2.0
02
2.4
0:23
00
0.0
022
cdl
Excent Reatin Bacc
Super Sauding
Dapers Souatrg Bock
RoaatRottty Block
DO
5.0
0:30
2.0
0.2
0.
0.6
0.0
022
0:25
0.0
50
0.39
6.0
60
00
2:20
6:23
2:2
0.
2.0
Q2
Q2
0-
0.2
6.0
620
0.0
0.0
0:2
67
57
6/
R:17
0:17
0:19
R117
2:5
6:25
9:5
R/S
0:15
1:12
00
1:17
60
86
2:
6:20
00
60
6:55
5:33
50
9
80
00
3.0
652
2:5
921
0G
6:24
00
8G
60
600
6.0
3.6
6:55
3.0
20
00
80
R33
80
R52
0.33
3:3
0
00
R3
0.0
E26
80
0.0
80
5.D
0
0:2
$33
R0
2:75
0D
0:3
5.0
0.
Love
Bymer?
Dane
Sabee
Srner 2
Eyper
Dane
Spper
Eyper?
Samee
Popet
Daset
Dowe
Laner
Krpwe
Doter
Figure 11: Routing distribution learnt by Latent Skills in the decoder routing blocks (49-60) of T5-GLUE
Crper Pheilig ny
Expert Pouking Duck
Exper Rod my Blckk
Exper/Truting Zbcck
Olpan-
0.17
5:77
0:
0:17
3.17
6:13
D:77
0.17
0:15
0.17
1.27
317
0:17
0.78
0.2
0:15
0.14
0.37
6:77
014
6.17
0.17
0.11
0:17
116
0:19
0.16
6:11
Q14
8:17
0.1G
Pelting
0.17
8:37
11
0.17
0.17
6:12
0.66
0.13
8/12
0.18
2:37
0.55
3:17
0.17
0.10
8.17
0.12
0.10
2:10
0.13
8.19
Saitch
817
114
8112
6:19
0.12
0:19
8:1
817
8:T
017
0.57
0.12
0/14
317
6112
1.19
8:17
0:24
0.15
0113
Feaeiig
Expet Poxtre
Expert Kog
Olper:
0.77
0:12
0:18
6:11
0:21
1.
6/12
0.15
5:21
0:18
6:12
5.19
50
80
0.17
8:10
2/2
0.77
2:17
2.14
0.1G
0.15
6:16
0.13
0.20
2.30
0.
0.17
0.16
5.0
0.0
5:1
5/17
3.17
8:5
217
0.17
3127
632
6:2
3
0.13
8:10
0.18
1:15
0112
312
6/2
319
5.0
50
012
8:31
817
6114
412
6:34
A12
015
0:37
0:14
X:
km
6:14
421
50
80
Sater
Aaper
Expert
B
Dieet
Sqwer
Reark
Figure 12: Routing distribution learnt by Top-k routing in the routing blocks of ResNNt- CosNee-DDmainnNee
29

---

Published in Transactions on Machine Learning Research (05/2024)
0:02
03
D02
0.21
0.20
6.3
20
02
20
10
60
04
6-
000
5.0
6.0
60
9.
18
83
66
04
00
60
0.0
9.0
03
00
16
03
0.0
60
00
30
02
60
00
80
231
0.
0.2
0.79
5.0
24
15
93
20
04
00
80
0.0
3.0
93
50
10
03
0.0
0.
22
00
00
20
50
05
114
2.27
80
0.04
02
200
48
5.3
0.0
0.2
5.0
2.0
0.2
Dxpert Roudin
Ecarnt Routing Bloce
Eapart Rock?
Expert Routin Brock
6o
60
04
6-3
00
5.
60
0:54
0:00
0-2
0.24
0:11
0.6
9.0
60
0.3
6.0
0.0
3.4
60
00
261
0.
MM2
60
60
24
63
00
30
10
40
261
26
60
00
06
60
6.9
60
0.6
9.0
93
60
0.2
0.0
0.4
60
00
2.0
02
60
T02
00
80
88
64
00
10
20
2
26
60
20
0.20
00
0.0
60
0.
010
60
00
00
010
00
20
E07
039
00
200
02
28
00
0.
541
0.27
5.01
0.2
201
0.2
00
60
00
04
6-
00
3.0
60
6.0
3.6
60
0.6
0.
0.6
9.0
60
0.3
0.0
0.4
3.5
60
0.0
3.0
0.3
60
Booding
Cipers Souting
Souting
Capet Rocting
c0
00
04
0:
3.0
0-3
6.0
5.0
3.6
60
00
04
0.0
0.
0.6
0.11
0.0
0:00
60
0.3
0.0
3.2
60
0.0
0.0
60
00
00
88
10
0.
3.0
0.3
00
500
28
50
28
10
00
00
9.0
0.
600
0.
0.0
60
00
00
50
00
00
00
10
50
0.0
00
0.2
2.0
114
20
00
0.0
00
00
0.2
00
5.0
0.
00
00
0.2
0.0
2.0
20
c0
00
04
10
0.0
2.0
04
2.0
0.0
3.4
600
0.3
0.
0.0
00
0.6
3.0
0.3
6.0
0.2
0.0
0.4
0.11
0.0
0.0
0.0
0.00
6.
00
Desriiluuing Back
Fosting
Exantt Rouling Bboc
Capet Tocting
80
04
6.
0.0
10
20
0.0
3.6
6.0
84
0.0
00
0.6
3.0
0.
0.
02
0.0
0.6
2.
60
0.
2.0
60
00
19C2
50
80
38
50
8.1
88
60
0-
0.0
00
3.0
88
03
0.0
0.
80
00
20
00
0.0
02
0.0
50
00
80
5.0
0.0
0.2
0.0
0.6
3.0
0.
0.
0.0
50
20
00
04
6.5
0.0
2.0
1.0
20
5.0
0.4
6.0
2.0
0.0
00
0.5
2.0
9.2
0.0
022
0.0
0.5
2.2
0.0
0.0
0.0
0.
Dypen Bouting
Supert Seeting eccil
Eccent Rouuding Stoic
Exeen Rouding Siiiec
20
80
04
6.2
0.0
5.
0.2
10
5.0
0.0
9.2
8:2
0.0
00
0.5
2.0
02
0.0
12
02
0.0
0.5
2.71
2.25
0.6
2.0
02
IM2
0.0
TD2
0.0
03
3.0
02
50
00
0.0
00
0.2
0.0
3.0
0.
08
6:77
0.0
2.0
0.5
50
00
04
6.2
0.0
5.0
30
5.0
0.
6.0
0.21
0.00
0.0
80
0.5
2.0
9.
0.
15
82
0.0
0.5
3.1
2.99
0.0
2.0
0.2
0.0
0.0
CAN
20
00
24
0.2
0.0
2.0
0.2
10
5.0
2.
0.0
0.2
8:2
0.0
00
0.0
2.0
0,2
20
02
2.0
0.5
0022
3.8
0.0
2.0
0.2
0.0
0.2
DoersRouthhg Block
DesenRouthigae
paper Boutinglll e0122
Exaert Rouding Block 24
6.01
6.57
3.48
5.0
3.8
0.0
TD2
60
2:
0.2
0.0
0.
0.
0.00
02
00
50
60
6/2
8
4:9
81
5/
3.24
22
0-2
06
06
0/5
60
60
00
0.80
20
80
032
5.2
0.0
200
2/48
20
5.0
24
2.30
82
035
R22
0.
80
03
15
83
20
80
82
05
22
50
00
0.54
82
00
246
32
241
3.08
00
3.8
0.0
00
83
Squrr
5
Sanet
Saker
S
Sauura
Guner
Laker
Spurts
Power
Sapest
Equrs
Severe
Eaurta
Figure 13: Routing distribution learnt by Top-k in the encoder routing blocks (1-24) of T5-GLUE
30

---

Published in Transactions on Machine Learning Research (05/2024)
3.01
0.90
2.22
2.24
0.01
02
20
02
192
0
00
D02
0:00
00
3.0
4.
6.3
6.00
0:00
0-3
0-2
0.0
0.6
18
03
60
60
03
04
601
00
30
00
60
9.
102
2
00
24
222
0.
50
80
5.25
0:54
E00
20
R1
000
820
0.0
15
03
20
80
00
25
00
30
00
00
80
0.5
110
2.00
00
3
2.55
00
5.C4
0.2
0.0
02
2.0
02
0:22
0.01
0.
0.00
6O
0:00
0:51
0.0
0:72
6.0
2.0
0.2
Expert Rocting Blck
Exceet Routing Block
Eccart Roudinng Slock
Exeent Routing
60
00
0.
0:0
0.3
6.0
6.0
18:16
6:
0-2
0.0
0.0
0.6
9.0
60
0.3
6.0
01.
3.5
60
0.0
3.0
0.2
00
MMZ
60
60
94
3.9
0.0
3.0
043
0.0
6.0
0:14
9:23
6.5
0.3
3.62
0.0
0.
0.6
3.0
93
60
0.2
0.0
0.0
60
00
2.0
92
00
0.
T02
00
00
88
10
5.0
0.3
0.
8.0
0:
60
2.
8:2
0.0
00
3.0
60
0.3
0.0
010
60
0.0
00
010
2.00
00
3:28
0.01
0:01
00
0.2
200
02
28
0.24
0.
2.1
2:23
0.2
0o
5.3
2:1
0.2
0.0
0.2
2.0
02
60
0.
3.4
10
0:0
3.0
0.3
6.0
6.0
00221
0:42
0:21
0.22
0:01
0.0
0.0
0.6
9.0
60
0.3
0.0
0.4
3.5
6.
0.0
0.0
0.
00
Expart Block
Excunt Routing Block
Capart Routing oocas
Expert
60
8.78
3:41
6:12
0:0
2:61
0-3
6.0
8.0
0.27
6.0
0-2
0.6
60-2
0.73
0.0
1.0
3.0
0.
60
0.
3.2
6.
0.0
0:00
0.3
0.22
00
00
0.
88
2:28
3.0
0.3
0.0
5.0
5.06
D.0
2.0
00
1.0
3.0
80
00
00
60
0.
0.07
03
8:23
K85
00
844
0:21
5.0
0079
0.21
0.
0.
0:00
02
114
8M
5:35
0.00
000
6:79
0.40
03
00
0:28
98
0.0
5.07
0.20
0-2
0.27
000
0.2
0.0
0:07
0.2
6:23
60
6.74
2.4
0.24
0.0
3.0
0.2
2.0
2.01
0:14
6.0
0-3
0.0
6:30
0.0
1.0
3.0
0.3
0.0
0.0
0.2
0.0
0.5
3.2
0.0
0.0
0.03
0.2
0.22
0.
0.2
Excet fbcting Blck
Excet Routing Black
Capert Routhguuu
Ecarti Rouiing 2eak
0:40
0.
04
2:47
0.0
3.0
3.05
2.0
8.0
2.
6.0
02
0.6
0.00
60
0.05
0.22
07
2.2
6.0
2.0
0.2
6.0
1:12
60
38
3.66
5.1
28
0.0
0.8
0.22
0.37
3.0
08
80
0.12
00
2.55
2.05
0:04
5:09
2.0
8%
3.0
65
09
0.49
0:11
0.0
0.22
0.0
0.2
0:49
00
98
0:47
3.00
0.0
0.0
0-2
0.
10
0.6
R:0T
0.22
2.0
0.5
3:2
00
0.4
0:91
0.0
5.0
2.04
2.0
3.0
0.4
6.0
9.2
0.0
1.0
0.0
0:04
E:07
2.
0.21
2.0
0.00
2.2
6.0
2.0
0.2
0.0
0.0
ExcemRRouttng Bock
Rastenoouthigg Block
Sapar So Rock d3
Expert Routing blocc 61
0:5
00
04
0:24
053
2.0
0.2
2.0
2.0
5.04
0.0
022
5.0
0.25
80
0.5
2.0
92
0.2
2.0
22
5.0
2.0
0.2
0.0
MM2
2.04
0.02
(0.00
2.0
6.01
T82
0.23
0:21
5:00
0.02
066
2.0
0:01
mmB
0:55
0.
98
3.54
5.0
0.0
0:12
6.6
0/2
0.57
00
80
0.
0.2
2.0
8:01
0.34
00
04
0.24
0.32
2.0
0:21
2.0
8.0
5.05
6.0
0.21
8:61
8-2
0.32
2.01
0.5
2.0
93
0.
800
0.2
0.0
22
60
0.00
2.0
0.2
0.0
5.21
0.2
CAN
0.23
00
24
8:01
0.5
2.0
0.2
2.0
5.0
0:27
5.0
0.21
2:01
0.0
2,21
0.5
2.0
0.
20
0.0
0.
2.0
22
5.
0.00
2.0
0:07
0.0
0.
Expot Routing Bock 45
Easan Routing Block 46
Cipan Souting 6oC47
Expert Rouding Block
41
60
00
0.77
0:00
0.0
0.
0.0
3.0
0.98
TD2
60
00
0.11
0.8
3.04
230
0.0
00
02
027
690
2.0
02
50
60
35
6/
88
641
0:5
35
50
0/
3.74
86
80
D6
3.0
86
60
10
10
02
86
80
50
00
24
00
20
3:0
0.25
0.04
25
5D
83
28
236
00
80
0.
3.0
93
20
80
82
2.
22
50
14
2.0
82
80
80
32
241
03
34
0.7
581
00
00
02
Sp
Samet
Saker
S
Sauura
Guoer
Laker
Laurra
Squrre
Poper
Sapest
Egurs
Smurrc
Spvere
Lourts
Figure 14: Routing distribution learnt by Top-k in the decoder routing blocks (25-48) of T5- 155-LLB
31

---

Published in Transactions on Machine Learning Research (05/2024)
Diperi Riacing Bockkkkkk
Dopert Ractig
Exent Reutng Boocust
Dree thoting Blokk
00
0.0
60
00
850
0:71
02
00
0.0
2:77
0.00
0.0
2:00
0.0
02
2.0
0.0
80
00
80
0.0
80
04
60
00
60
0D
03
02
682
80
00
60
80
84
2.75
03
00
00
2.10
80
0.
5:00
00
0.0
00
80
00
80
00
019
00
08
80
90
80
00
00
05
K095
00
50
2:0
2.0
2.0
00
00
010
00
50
50
2:0
2.0
2.0
00
60
00
8:6
0:04
6:11
3.11
60
0.0
8:27
0.0
0.0
0.
0.
0.0
00
0:09
0.0
9.01
0.
6,7
3.0
2.0
02
0.
9.2
60
10
60
0.
0.0
60
0-3
50
60
60
90
0.
00
3.0
00
0.0
80
3.0
00
9.4
60
60
60
3.0
00
118
00
6.99
60
0.0
9:21
6.0
0.
80
00
60
60
9.
0.0
00
2.0
0.
18
0.0
80
2.0
00
0.4
00
60
10
60
0.0
0.
900
50
0:04
60
0.0
0:5
6.0
02
90
0.
00
60
0.0
500
20
84
201
9.
13
0.0
80
2.0
00
0.
00
6.0
13
80
0.0
0.0
200
02
60
6:47
0.0
0:00
0.0
3.0
9.8
0.00
3.0
5:20
0.02
261
00
3.0
0.
Expert Reatig Bocc
Duper Sauding
Capers Souting
DanamRoutiy Block
0.0
2.00
2.0
0.
0.
1:0
0.
600
00
0:21
9:11
0:44
8:55
0.0
0.0
C0
00
60
60
0:21
2:20
0:42
2:10
0.0
0.
0.
00
0.
THR
50
60
60
0:21
66
0:55
5:02
0.0
545
516
00
00
60
10
60
60
00
00
03
CO
86
60
60
80
60
2:66
0:04
600
00
0.0
6:77
418
00
0.0
80
3.0
03
00
16
60
60
00
80
03
3.0
08
20
D0
80
60
00
80
00
0:11
0:13
0:25
E:5
D
0.0
0.0
275
00
00
00
00
3.0
80
00
80
00
80
00
5.0
0.
Lovn
Diirtt
Sypeet
Oryeet
Evert
Byper?
Done
Dane
Eyper?
Bamne
Praet
Donr
Saner
Krowe
Doter
Figure 15: Routing distribution learnt by Top-k in the decoder routing blocks (49-60) of T5-GLUE
Oxper Pheiing
Exer: Reees Back
Expert/futiig Sbccu
Open
10
20
D0
BD
0.7
2.
8.0
0.0
5.0
0,0
20
2.2
20
00
3.0
0.0
0.
Saith
0.0
00
00
22
6/2
10
0.
22
800
Feaeiig
Exper Poxtre
Expe: Nouting
Experifbuute
Opper:
CD
0.
0.6
50
35
50
20
0.2
10
5.0
2.2
20
00
8.0
03
13
0.0
2.5
5.0
28
20
60
8D
8a
10
08
22
44
80
80
214
0:30
80
Spper
dowrd
Apper
Geses
Expert
Bowel
Diset
Reart
Figure 16: Routing distribution learnt by ST--Gmbel routing in the routing blocks of Res Net
32

---

Published in Transactions on Machine Learning Research (05/2024)
0O
6.3
20
02
20
02
60
60
94
10
00
5.0
03
60
6.1
10
60
66
04
00
00
0.
3.0
03
00
60
04
60
00
30
60
98
102
00
00
28
10
00
20
07
80
5.0
1.0
93
20
00
80
0.0
30
93
20
00
00
0.
22
50
00
20
00
00
0.
110
00
10
5.
10
00
02
02
4
00
03
2.
0.0
5.3
0.0
2.0
0.
Dupert Soudinglldk
Ecaert Routing Block
Eapart Bock
Expert Routin Block
0:65
0.
1.4
0:3
6:22
0:00
0.3
6.0
6.0
60
0-2
00.81
0.0
0.0
03
60
60
0.
0..
3.5
60
00
30
0.
60
MM2
60
6:01
0.22
0:73
0.0
3.0
0.3
6.0
6.0
5:00
60
03
04
60-2
0.02
0.
0.6
15
03
60
0.2
0.4
32
60
00
2.0
02
60
102
0:
60
004
0:31
02
0.3
0.0
5.0
00R2
D.
2.
00
00
20
60
010
60
0.0
00
010
2.00
00
0.22
0:28
0.0
5.0
0.57
00
02
200
02
28
0:25
0.3
0/12
0.01
0.0
5.3
5,34
5.
022
0:6
0.2
2.0
02
60
0.0
0.09
0:00
0:0
3.0
0.3
6.0
6.0
60
0-3
0.6
0.0
0.0
0.0
0.3
0.
60
0.3
0.4
3.8
6.0
0.0
3.0
0.3
60
Dupers Rocdin
Cipert Foxting
Souting
Capart
0.22
0.0
0.4
6:12
3.0
3.60
0.0
5.3
6.0
6-2
0.0
0.0
0.6
3.0
0.00
00
60
0.11
0.00
3.00
0.11
6.0
2.0
0.2
60
02
6:11
88
6:14
0:0
3.0
7:00
0.
5.0
5.0
8:5
0.0
00
3.0
80
00
00
6.0
5.0
05
60
0:54
00
2.2
0:21
2.0
5.0
2.25
0.24
0.0
00
0.
0.0
2.0
02
114
3.0
3:00
3.0
03
6:77
2:52
0:70
0.00
5.0
0.44
0-2
0.0
00
00
0.
0.0
2.0
0.2
00
0.0
6.62
044
0.00
0
2.0
2.0
0.0
5.00
6.0
0.2
0.24
0-2
0.0
0.
0.6
3.0
0.3
0.2
0.0
3.2
6.0
2.0
0.2
0.0
0.0
Deeriiauuing Back
Oieer Foxtngg
Ecant Rouking
Capett
20
00
04
6.
0.0
04
2.0
5.0
2.6
6.0
02
04
0.0
00
250
3.0
02
66
60
0.2
0.6
3.2
6.0
0.00
2.0
0.2
6:04
0.0
19C2
50
80
38
83
00
5.1
38
88
0.6
80
3.0
08
00
20
0002
3.0
5.0
00
00
0.0
0.
2.0
0.2
50
00
03
80
5.0
0.
0.
00
0:07
3.0
9.0
0.01
2.0
0.7
8:54
20
00
04
6.2
0.0
2.0
5.0
0.4
6.0
9.2
2.
0.0
00
1.0
2.0
0.2
2.
0.2
2.0
0.5
2.2
6.0
0.00
2.0
0.2
0.0
0.0
0.2
Dypen Souting Bockk
Supert Seuting occ
Ecaent Roldingg Stoicto
Exeen Rouding Siciic 20
20
00
04
6.2
0.0
0.2
2.
0.00
2.5
8.0
637
68
8:
0.0
00
0.5
002
0.04
20
0.2
2.0
0.5
22
50
0.0
2.0
0.2
IM2
0.
0.00
0.0
TD2
00
00
0:11
0.0
0.0
03
50
00
50
0:2
0.0
00
0.6
0/0
0.7
9.0
2.0
03
20
00
24
6.2
0.0
1.5
84
20
2.01
3.4
6.0
6.0
2.6
0-2
0.0
80
0.6
0:53
0.02
5.0
8.0
0.2
0.0
0.5
22
5.
0.0
2.0
0.2
8.
CAN
20
00
24
0.2
0.0
1.5
0.2
20
613
24
2.
0:2
0.0
80
0.0
0:32
0.08
2.
0.0
0.2
2.0
0.5
22
5.0
0.0
2.0
0.2
0.0
10
RoersRouthhg Block
Capars Souting 001-2
Exeert Rouding Sbock 24
00
0.0
0.0
0.14
0.
0622
0.02
00
T82
60
00
041
02
co
5.0
0.2
0.27
0.00
0.2
2.0
0:00
m2
58
80
16
6/2
8.
5.0
5.
38
60
28
6/2
0:21
0.6
3.0
10
60
0/
05
30
85
80
20
00
002
5.
0.0
221
82
20
5.0
24
00
883
28
R:2
0.77
80
0.5
2.0
92
10
82
0.
22
50
06
220
3.50
80
32
241
00
00
03
001
2.0
072
Dome11
Equrg
Egoor
S
Samet
Laker
Salura
Gumer
Laker Basvel
Squar
Laurr
Spurt
Power
Sapest
Equrs
Severe
Lawrts
Figure 17: Routing distribution learnt by ST--STTGGii Gumbel in the encoder routing blocks (1-24) of T5-GLUE
33

---

Published in Transactions on Machine Learning Research (05/2024)
6.3
0.0
0.
0.2
2.22
0:15
0:24
60
60
94
64
50
60
36
60
10
0.
00
0.
3.0
60
04
0.
0.6
3.11
0.54
611
8:58
6-
0:30
0.64
6107
T02
00
80
24
00
00
50
03
5.0
20
80
00
20
10
00
80
0.
3.
20
00
00
0.0
0.0
007
0.
R22
0.22
0
DXX
110
00
24
5.
25
0.0
00
0.
3.22
0.22
2.41
6/1
0.25
0.0G
00
0.
5.3
0.0
0.2
6.0
2.22
0.00
0:33
0.02
0.29
0.04
Export Rocting Bloc
Excet Roatin Block
Excart Routiing Block
Expert Rouding Block
60
00
0.4
0:0
5.0
0.3
6.0
6.0
34
60
04
00
0-
0.0
0.6
0.:7
12
5.01
0100
0:61
0.4G
3.5
6.
6:0
6:31
0.2
6:60
0.01
60
00
94
10
0.0
3.0
043
4.0
5.0
26
60
00
06
64
0.0
0.11
0.0
0.52
1:17
3.63
0.02
0.08
0.66
3.5
6:01
0.
015
0.
0.0
0.60
0.01
T02
00
00
88
10
0:
3.0
0.3
00
50
28
50
20
0.0
0.
00
0.00
S102
0:61
0.01
27
07
0.
0.
010
00
10
20
5.
0.
0.22
0.04
0.21
0.22
0.00
0.23
0.2
28
00
0.
25
02
0o
5.2
0.0
10
9
0.02
0.22
0.00
6.27
0.52
0.01
20
c0
00
0.4
10
0:0
3.0
0-
6.0
6.0
3.6
60
0-
0.0
0.0
00:7
0.60
2.01
5:00
0624
0.66
3.41
6:01
6:50
0.3
6.40
0.01
Expurt Block
Excet Routing Block
Capart Routing poc35
Expert Routing Block
c0
00
04
0.00
0:0
3.0
0-3
6.0
6.73
2.6
60
021
0.22
6-2
0.0
0.22
0.01
0.02
0.06
0:01
0.63
0:52
0.28
0.09
3.5
6.0
2.0
0.3
6.0
00
00
00
88
1:0
0:0
3.0
0.3
0.0
0:22
01
60
0.71
0.55
0/1
0.0
0.2
0.0
3.0
0:01
5:m0
0:21
01.93
011
6.0
5.0
05
60
60
00
00
28
2:0
2.01
0.
3.
25
002
DOT
5.72
2.07
0:22
0.22
2.0
0.
110
0.44
2.07
0:57
0:43
3.0
00
3.0
0.5
00
00
98
2.29
5.01
0.2
0.0
0.5
2.5
0-2
0.0
0:01
0.0
00
0.54
0.23
2.09
2.0
0.2
00
00
00
044
0.90
0.0
3.0
0.2
2.0
0.26
0.51
600
0:11
0.02
0.0
0.61
0.01
3.0
0.01
0.
6.0
0.2
0.24
2.04
3.2
6.0
2.0
0.2
0.0
0.0
Expet fbettng Bock
Excett Roating Black
Capert Routigg
Ecart Rouling
0.22
8:22
04
6:5
6:22
6:11
3.00
2.0
0.0
2.6
3.01
0.0
04
0.2
0.0
0.00
0.6
3.0
02
00
60
0.0
0.6
22
60
0.0
0.0
0.2
0.0
1912
0.51
38
6:12
5:32
800
0.
3.8
6.0
0.
0.0
6.1
3.6
8.6
0.
0.0
03
0.52
28
0:
2.05
20
5.77
0.
5.0
03
0:22
6:47
0.2
6:0
3:61
3.00
0.22
0.01
3.00
021
0.04
0.0
02
0:22
0.51
2:18
0:09
0:51
60
0:51
9:92
2.00
0:92
6.17
0.5
3.0
00
58
0.0
0.
0.04
2:22
0.4
6/1
0:25
2.04
2.0
3.0
0.4
6.0
9.2
0.
1.0
0.0
00
0.5
2.0
0.2
0.
8.0
2.0
0.5
2.
0.0
0.0
0.0
0.2
0.0
0.2
ExsertRouuting Bock
Rastenouuthg Block
Dapart Souttng d.
Expert Roartig Block
20
00
04
10
0.0
5.0
0.2
2.0
2.0
0.
D.0
022
0:2
0.0
00
0.5
2.0
0.2
8.0
0.2
0.0
0.5
22
227
2.04
0:32
0.01
0.25
0:14
IM2
38
3.9
0.0
031
0.00
0.32
6.01
TD2
0:22
0.
0.42
0:21
0.22
0:51
0:22
0.2
0:12
0:35
mnB
00
00
98
2.09
0.7
0.22
0.0
3.5
0.0
0.2
0.0
00
0.6
3.0
0.0
0.7
9.0
0.24
8.09
5:00
9:29
00
00
04
3.50
0.0
3.0
0.2
0.07
8.0
0.5
6.0
92
0.00
0:2
0.0
80
0.6
2.0
9.
0.0
0.2
0.0
0.5
2.11
2.28
0.09
0.32
6.01
0.14
5.12
2/16
CAN
00
00
2.
0.59
0.0
3.0
0.2
0.07
5.0
2.2
0.0
0.2
R:2
0.0
00
0.5
2.0
0.2
0:20
0.0
0.2
2.0
22
0.5
0.04
0.45
0:01
0.25
5:55
Export Rosting Booc 45
Eoses Routing Block
Cipen Souting 60c-77
Exeert Rouding Block
43
0.07
0.37
3.8
0.23
0.0
0.0
4.00
0.67
2:24
6:02
0.27
0.60
8:34
9:12
8:08
5.5%
TD2
3.00
6.21
603
3.26
0.25
0.34
3.6
0.03
0.2
0.0
8.1
0.02
2.0
0.07
0.24
0.04
0.0
50
00
42
3:58
0:22
0.64
03
0.14
38
244
0/
0/5
06
80
mnt
0:22
6:01
2:44
0/22
0:00
055
50
06
30
cD
60
20
83
002
3:3
804
575
82
0.28
0.42
24
355
83
8:2
00
80
0:02
0.08
25
022
8/0
0.2
225
22
50
06
2.0
10
20
00
02
241
6:6
007
0.1
6.34
5.
38
6:13
00
8.30
2.2
02
024
3.60
0.0
Dome11
Expor 3
Baneta
Saker
Salura
Guner
Laker Baneea
Equer
Lauer
7
Squrts
Poper
Sapest
Eour's
Sphere
Loorts
Figure 18: Routing distribution learnt by ST-Gumbel in the decoder routing blocks (25-48) of 75-GLUE
34

---

Published in Transactions on Machine Learning Research (05/2024)
Cppert Rech Black-kk
Drpert Roading
Ecant Reuing Besust
Exper tlotiin Block 92
00
6.20
60
00
00
02
0.02
0.0
00
8.0
80
00
2.0
0.0
0.0
80
2.0
00
00
0.
600
0:53
00
80
0.0
0,
2D3
682
80
0:71
80
8:23
00
010
000
00
00
80
80
80
30
80
2.0
00
0.
80
20
00
00
08
80
0:45
00
80
0.0
00
22
0090
00
559
5.0
2.0
2.0
03
00
02
60
010
0.2
5.50
2.77
9:27
2.0
0.0
00
2.0
0.
00
0.22
00
60
08
0.0
5:01
0.24
09
0.21
0.0
0.0
0.0
0.69
0.
0.0
0.0
0.0
0.22
0:50
0.11
0.05
0:10
0044
0.20
0.0
60
2.0
6-5
221
0.02
0:07
0:9
00
60
50
00
60
5.0
0.
0.
60
0.0
00
0.0
0.0
80
0.0
OCT
SIM
9:96
003
60
60
60
0.0
0.
1180
50
0.
60
50
00
60
02
00
0.0
0.0
0.0
90
0.0
02
2.0
5.5
60
6:08
22
6:03
0.06
00
6.0
10
60
0.0
0.0
0.0
900
50
0.
60
50
00
6.0
02
80
0.0
0.
15
0.0
90
3.0
02
2.0
0.
2.00
5:0
0:05
002
0.0
0.0
0:59
60
0.0
0.0
2.0
00
60
0.00
6:00
0.02
0:00
0.0
0.0
2.0
6.7
0.05
0.07
6.17
0.04
Expent Reatig Bocc
Saper Saading
Capers Southn
DanatRouthgg Block
0.0
50
5.0
0.57
2.0
0.
0.
00
60
0.0
00
0:
0:11
0.0
0.
C0
00
60
60
6:23
0.2
0:01
0.0
0.
0.
002
0.
00
00
0.0
THR
50
60
60
80
50
50
S00
0.0
0:08
0.
0m
0/
00
60
00
00
0
CO
86
60
50
80
60
03
50
6:1
021
667
3.0
4.42
001
00
80
0.0
0.01
3.0
0.3
00
10
60
00
60
0.0
3.0
20
00
80
60
00
80
03
00
220
2:5
R21
30
03
5.77
00
80
0.
00
0.0
03
00
15
E0
00
80
0.0
80
5.0
0.
Lovn!
Diirt
Orpeet
Biert
Bypnr?
Done
Sabae
Dane
Sppet
Eyperr
Sanee
Popet
Drset
Dober
Figure 19: Routing distribution learnt by ST-Gumbel in the decoder routing blocks (49-60) of T5-GLUE
Oxper Pheiing
Dapt Duck
Exer: Rodin Pack
Exper/Trutin Slcck
Openn
1D
20
00
2.
GD
0.2
5.0
2.5
Q2
6.0
0.0
5.0
0,0
20
Q2
8.05
2.0
2.
00
3.8
5.0
0.2
2.0
0.0
0.2
0.0
0.0
5.0
3.0
Saith
8:28
0.5
00
22
tum
6/2
20
03
22
08
8.0
800
1/1
Reatiig
Expet
Expt: Rouking
Opper:
0/
ED
50
95
50
600
50
10
60
2.0
50
0.2
5.0
6.0
0.0
0.0
6.24
10
00
38
1.0
500
03
0.
22
00
0.
5.0
0
573
2atch
28
10
22
10
80
88
60
08
22
42
80
80
0:25
Saper
S
dowrd
Sewra
Apper
d
Expert
B
Reart
Figure 20: Routing distribution learnt by REINFORCE routing in the routing blocks of ResNet- Net
35

---

Published in Transactions on Machine Learning Research (05/2024)
03
0.2
CO
6.3
0.0
0.
20
0.2
20
02
892
60
60
16
64
00
5.0
60
6.0
36
60
03
10
0.0
00
0.6
3.0
03
00
60
04
0.
60
00
20
02
60
102
00
80
18
0.0
50
07
00
5.0
20
00
03
28
10
00
00
0.0
3.0
93
20
00
00
0.5
60
80
20
00
50
00
05
000
00
5.
00
02
200
02
40
0o
0.
0.2
00
5.3
0.0
0.2
2.0
0.2
Dxpert Rock
Ecaert Routing Black
Eapar Rou Rock
Expert Routing Brock
6o
00
9.
6-3
6.
04
6.0
6.3
3.6
60
6-2
0.0
0.
3.0
03
60
60
0.2
6.00
0..
3.5
60
00
3.0
0.
MMZ
60
00
98
6.
14
2.0
04
40
5.0
2.
60
93
64
0.0
60
10
3.0
03
00
60
0.2
0.6
3.
60
00
2.0
0.2
60
T02
00
00
88
6.
20
03
00
50
28
60
0.0
00
30
20
00
60
0.0
0.
010
00
00
0.0
00
0.
200
02
28
00
0.
022
00
5.3
02
0.0
0.2
2.0
02
60
00
0.4
6-
3.0
04
6.0
6.0
3.6
60
0.3
6-
0.0
0.0
3.0
0.3
0.
60
0.3
0.4
3.5
60
0.0
3.0
0.3
60
Duper: Bock
Cipers Routing Routing Blocc
Souting
Capet
10
00
044
6.
00
3.0
04
2.0
0.0
3.6
60
0.6
64
0.0
0.
0.6
3.0
80
60
0.
0.0
0.6
3.2
60
0.0
0.0
0.
60
80
22
00
5.0
08
60
28
0:5
00
80
0.0
10
88
00
00
0.0
0.0
60
00
000
00
K05
00
26
20
5.0
02
0.0
00
0.
2.0
02
114
00
28
12
00
00
5.0
5.0
0:2
0.0
000
0.2
0.0
2.0
0.
0.
0.4
6.3
0.0
2.0
04
2.0
0.0
3.4
6.0
0.0
0-2
0.0
0.
0.6
3.0
2.
600
0.2
0.0
0.6
3.2
0.0
0.0
0.0
0.2
0.
Dosriituutin Back
Oieer Foxting
Exantt Rbuling Bbok
Capet
00
04
6.
0.0
04
10
0:55
26
6.0
00
000
0.0
00
0.6
3.0
26
60
0.
0.0
0.6
22
60
0.0
2.0
0.
1912
00
80
38
10
38
0/2
0.0
00
3.0
60
03
000
80
00
26
00
00
0.00
044
0.
02
0.0
02
50
00
38
0.19
0.0
0-2
0.
00
0.6
0.0
00
0.7
6.0
000
0.
80
0.
20
00
04
6.5
0.0
2.0
1.2
1.0
2.4
6.0
9.2
0.0
00
0.6
2.0
2.
0.0
0.2
0.0
0.5
2.2
0.0
0.0
0.0
0.2
02
Dypen Souting Bock
Supert Seuting eocil
Ecaen Roudiigg Stoict
Exeen Rouding Siiiec 20
50
10
6.2
0.0
5.
0.2
2.0
2.0
2.
D.
02
2.44
0.0
0.25
1.0
2.0
02
60
80
0.2
2.0
0.5
22
50
06
2.0
0.2
M92
040
6.10
0.0
T82
0.0
05
0.0
02
06
18
80
0.0
60
0.
6.0
3.0
00
0.
9.0
2.0
05
20
10
04
6.2
9.0
3.0
04
20
3.0
0.5
6.0
93
24
2:2
0.0
0:03
1.0
2.0
0.2
0.0
8.0
0.2
0.0
0.5
22
5.
0.0
2.0
0.2
80
CAN-
20
10
24
0.2
0.0
2.0
0.
2.0
5.0
2.
0.0
0,2
2.
0.00
0.0
1.0
2.0
0.2
20
0.0
0.2
2.0
1.5
22
5.0
0.0
2.0
0.2
0.0
10
DoerRoutthg g 21
DeaenRotthgg Blokk
Dapers Souting 20122
Exeert Rouding Bloc 24
co
00
5.0
3.8
0.2
0.0
0.
3.0
3.0
02
TD2
60
00
5.
24
02
00
0.2
2.0
02
50
60
35
6/
80
03
10
50
48
02
00
8.
0.00
0/2
0
CD
06
30
cm
20
00
24
00
50
82
15
50
24
15
83
22
82
00
80
0.
2.0
3.54
2.26
02
2.0
0.
002
5.0
0.
2.0
82
80
32
241
03
34
5.0
38
00
03
0.0
0.
Dome11
Eqoor
Gume
Loker Baneel
Squar
Sparts
Power
Sapest
Eour's
Amurrc
Sperre
Loorts
Figure 21: Routing distribution learnt by REINFORCE in the encoder routing blocks (1-24) of T5-GLUE
36

---

Published in Transactions on Machine Learning Research (05/2024)
022
00
6.3
04
0.0
0.2
20
Q2
60
60
94
64
0.
60
6.
10
60
83
66
04
00
00
0.6
9.0
03
60
0.3
0.
0.
3.4
60
00
20
03
60
T02
00
80
24
0.3
20
07
80
5.0
1.0
93
20
R3
00
00
0.0
3.0
93
10
80
0.
0.0
0.0
22
50
80
20
05
80
110
00
00
5.
10
00
0.2
200
02
00
0.
2.
00
5.3
0.0
00
0.2
0.0
2.0
0.2
Expert Rocting Blcc
Exces Routin Block
Excert Routing Slock
Expent Routing Blook
co
00
04
6-3
00
60
60
00
0-2
0.0
60
10
3.0
04
60
60
0.2
0.5
0.5
60
0.
3.0
0.2
00
60
60
24
63
00
30
10
40
10
26
60
00
06
0.0
60
10
3.0
03
00
60
0.2
0.4
02
60
00
200
02
00
02
T02
00
80
88
60
00
20
10
20
28
50
20
00
00
10
3.
20
00
010
60
000
00
010
00
20
0.0
00
0.
200
02
28
00
0.
co
10
0.0
0.2
2.0
02
60
00
044
6-
0
3.0
60
2.6
60
0-3
0-
0.0
0.
1.0
3.0
0.3
0.
60
0.3
0.4
3.5
60
0.0
0.0
0.
00
Expert Block
Excert Routing Block
Capart Routing pocas
Expert
c0
00
04
6.
3.0
04
0.0
0.0
3.6
60
00
0.
0.0
1.0
3.0
00
60
60
0.
0.6
3.2
60
0.0
60
00
00
00
08
0.
3.0
0.
00
5.0
28
50
20
0.0
10
3.0
80
00
0.0
0.0
60
00
05
60
00
00
6.6
00
5.
0.0
0.
0.5
02
50
110
00
00
0.
02
00
00
0.2
0.0
5.0
02
0.0
10
00
0.
0.
50
00
0.0
0.4
6.3
2.0
04
2.0
0.0
3.4
60
0.3
0.
0.0
1.0
3.0
0.3
0.
600
0.2
0.0
0.6
0.2
0.0
0.0
0.2
0.0
0.0
Excet focting S7
Excett Routing Black
Capert Routigoo
Ecart Rouling beck
000
10
04
6.2
0.0
04
2.0
0.0
9.6
6.0
0.0
00
0.6
3.0
02
06
60
0.
0.5
2.
60
0.0
2.0
0.
1912
0.0
38
000
8.1
88
60
0.2
0.0
00
3.6
80
03
03
000
80
28
00
0.
60
6.0
0.0
02
0.0
02
00
10
58
60
0.0
6.0
0.2
0.0
0.6
3.0
00
0.
0.0
0.
50
20
10
04
0.2
0.0
0.0
0.2
2.0
0.0
0.4
6.0
2.0
0.0
00
0.5
2.0
0.2
0.2
1.0
0.5
2.2
0.0
0.0
2.0
0.2
EiserBoouttig Bock
Dasernoouttn Block
Eapart Southn da
Expert Routing blocc
20
00
04
6.2
0.0
0.2
2.
0.0
2.
8:44
02
6.
0.05
80
0.5
2.0
20
60
0.2
2.0
0.5
2.
0.6
2.0
02
IM2
0.
3.8
0.51
0.0
TD2
00
00
0.54
0.42
3.0
03
50
00
03
5.0
3.29
0:2
0.39
00
0.6
00
0.7
0.0
0.0
2.0
05
20
00
04
6.
0.0
15
84
20
8.0
84
2:42
02
044
8:2
0.50
80
0.5
2.0
1.0
28
000
0.2
0.0
0.5
22
10
0.6
2.0
9.2
00
CAN
20
00
24
0.2
0.0
83
2.
5.0
24
3:77
0.2
2.
R:2
0.23
80
0.5
2.0
28
0.0
0.2
2.0
0.5
22
1.
0.0
2.0
0.2
0.0
0.
Expot Boating Bock 45
Eases Routing Block
Cxpen Souting 60C47
Exeett Rousing Bloc 43
60
00
34
0.
38
0.0
0.
03
0.0
TD2
50
00
02
60
24
02
00
0.2
0.0
50
80
35
62
80
10
88
60
10
35
50
02
no
D6
0.0
0/
10
60
04
50
06
30
cD
00
20
03
24
52
0.
15
83
20
16
25
50
83
22
82
00
80
05
3.0
93
10
82
0.
22
50
06
2.0
10
20
00
02
241
03
34
00
0.
00
03
0.0
5
Sanet
Saker
Eqorr
Gune
Laker Baneel
Laurrg
Squrts
Doper
Sapest
Eour'
Sphere
Lawrts
Figure 22: Routing distribution learnt by REINF FORCE in the decoder routing blocks (25-48) of T5- GLUE
37

---

Published in Transactions on Machine Learning Research (05/2024)
Coprr Rech Block-o
Dxpert Roxting
Eoxnt Routng Blesust
Drpe thoting Block
0.
60
00
00
60
02
00
0.0
0/
00
80
00
02
2.0
0.0
80
00
80
0.0
0.
80
15
80
80
00
80
0.
03
682
10
00
80
80
80
D0
00
00
80
15
00
80
00
0.0
00
80
00
80
0.0
00
80
80
80
00
00
0.0
50
K090
80
80
50
50
2.0
2.0
0.
00
60
00
010
00
60
50
200
2.0
0.
00
00
00
60
00
0.
0.0
0.0
60
0.0
0.0
0.0
0.0
0.0
2.0
00
0.
60
00
4.0
0.
1.0
22
00
02
80
60
50
00
60
50
60
50
90
2.0
00
60
00
0.0
0.
10
34
60
00
60
0.0
1180
50
0.
60
50
00
60
02
60
50
60
60
80
0.
2.0
0.
80
00
80
0.0
0.2
10
04
14
60
00
60
0.0
900
10
0.
60
50
00
6.0
92
80
00
60
60
00
80
2.
10
2.0
80
0.0
00
2.0
0.
10
02
12
60
00
80
0.0
0.0
00
60
0.
38
0.
2.0
00
2.0
0.
24
Expert Reatig Bacc
Dupert Sauding
Capers Southng
DaaatRosthg Block
0.0
50
5.0
2.0
0.2
0.0
0.0
0.0
03
00
0.0
0.
00
0.
0.0
C0
0.
60
2.0
0.2
0.0
6.0
00
00
0.0
T2
50
00
60
50
0/
80
15
00
0.
03
0.0
03
00
60
60
00
00
00
10
50
C0
00
60
13
00
60
93
80
60
00
00
60
0.0
00
3.0
00
80
0.0
00
03
00
34
60
60
00
60
30
10
30
08
20
00
80
00
80
02
00
20
80
0.
00
0.0
03
0.0
00
00
0.0
00
03
00
22
00
80
00
80
0.
10
5.0
03
Lovn
D1p012
Diiett
Bient
Bypnr?
Sabee
Erper
Danet
Sppeet
Eyper??
Banne
Prpert
Dyper
Daser
Drset
Doter
Figure 23: Routing distribution learnt by REINFORCE in the decoder routing blocks (49-60) of T5-GLUE
Orper Pheiing
Dast Rack Duck
Expr Rod Back
Exper Beckk
Oppam
20
0:1
112
0.2
5.0
2.5
3.
1.0
0.0
1.0
0,0
20
667
0.00
Q2
0.0
3.0
3.0
0:04
0.2
2.0
0.0
0.2
Saith
0/
0.5
00
4.47
6/2
11
8/3
22
05
80
8.0
60
Reveiig
Exper Poxtre
Expe::
Exper//fuute
Opper:
0/
SD
0G
50
D.D
50
5:41
5.
20
00
6.
0.2
5.0
2.2
12
5.0
0.12
2.2
20
00
10
8.0
0.
5.0
2.
0.
5.0
0.11
0:33
28
20
60
6/2
60
44
80
Spper
dewrd
Apper
Expert
Sowert
Roard
Figure 24: Routing distribution learnt by Dselect-k routing in the routing blocks of ResNet- ResNet-DomainnN DemainNNe
38

---

Published in Transactions on Machine Learning Research (05/2024)
Dxpert fleling Back1
Eesel Pocking Dock
Epe--Raiip Back
Experilauuig Peak4
Kayooh
Pakiin
82
0.29
9 000000
8:10
0.22
0.22
0:1
8:17
8.00
6:13
0.55
8:12
6.9
6:00
200
Rasl
0.14
8:27
0:21
0:00
6:14
6:27
0:29
81
0.20
0.09
0.00
0:24
0:2
2:32
0:05
6.5
8:04
0:11
0.27
884
2:24
Slitch
6:14
0:21
0.7
0:18
6:19
0:24
5:00
0:11
6:12
0:21
5.24
1116
6:12
0:27
0:21
0.
5:14
Eosrt Roaling Bickk
Expet Roxi Bock
Exper: Bo.d Beck
Experihfudte Block
0.29
0:21
0.22
6:36
6:16
6:11
6:13
0.24
Puliin
6 0X00OW
8:2
0.24
0.10
0.22
6:14
0.18
0.24
3.11
0:23
0.28
0.18
Raal
6113
6:15
0:24
0:42
6:1
0/07
D104
6:14
2:14
114
6:15
6:11
5:15
0:20
017
0/35
011
81
0:31
14
6:14
115
B:12
0:24
6:15
6:14
2:44
314
D:14
6/11
6:14
0:24
6:18
0:25
6.2
6%
Sabeer
Epper
Sapers
Gamer
Sewn
Bawee
Seper
Tower
Beser
Sewer
Rewor
Figure 25: Routing distribution learnt by Ensemble routing the routing blocks of MoANee--Dmainnae Net
39

---

Published in Transactions on Machine Learning Research (05/2024)
0.25
6:27
007
5.05
0.T
2.23
2.00
5.00
0.72
0.22
0.14
0,25
0.92
0.11
0.23
0.2.
6:04
0:44
0:15
0.25
M92
0.
6:0
3:29
6:37
0:04
0111
0.37
6:12
6.45
6:44
0:21
0104
6.14
6.6G
0:11
0.00
5:05
04:4
$100
11.70
6.60
0104
9.04
1117
6:00
0.29
S:77
15
0:15
0.16
2:27
2.00
12
0:27
2:71
3:77
0.
0.02
2.58
0.09
0.
0:36
114
EXY
0:75
2.30
0/05
7.7
DUT
0000
R:14
RX1
R:23
23
110
3:12
52
6:12
2.05
0.02
5.21
0:14
0.22
0:12
6:11
5.09
2.22
0.12
0.06
0.00
0:22
4
0:7
0.24
0:02
210
2.55
0:13
0.70
0.17
5.05
0.22
0:13
9.55
0.71
0:31
0.04
012
0.02
0.00
2.00
0.11
0.14
0:03
0.1
1:17
Dapert Soudin
Ecarrt Roating Block
Eapart Rou Rock
Rapert Routin Black
0:04
6:67
1:15
6:14
0:0
0:10
3.00
0:34
0.20
0:12
0/15
D:01
00011
0:45
0:04
0.11
6:36
1:15
0.66
3.5
0.11
6:0
0:21
MM2
0:24
6:22
0:03
0:06
C04
2.00
0.23
0.60
5.04
0.53
C:04
0.04
0:01
0.51
0.74
0.01
0:02
3.00
0:02
2:01
0.51
0.24
0.66
0.02
6:01
0.01
0:02
3.0M
0.01
0.01
T02
2:
DU0
0:04
8:29
0.1
0.99
0:42
0:25
0:11
C:00
0/22
0.52
01:11
0.04
0111
0::5
EUST
611
0:61
0101
0:04
0/23
<10
0:28
3:34
0:22
2.04
0.22
0:37
0:27
2:27
0.44
2.55
0.03
0.5G
0:22
2.05
0.31
0.02
0.06
2.7
DUTT
0.22
28'
0.27
5:0
0:23
2.29
0.79
0.21
0:21
2:02
0O1
0.22
0.02
0.22
2.
00
0.2
0.00
OUT
2:22
0.21
0:02
0:07
6:11
5:69
097
9:04
OL11
3.6G
0.52
0.06
0.04
0:14
0/17
3.44
0.20
6.40
0:02
6:01
0.02
2.01
0.3
0..4
3.CG
0.24
0.26
0.09
0.67
3.M
0-34
Daper: Soxding
Foxting
Souting
Capet
0.00
6:2
0:04
3:00
004
2.00
0-13
0.10
0.
0:11
2.44
0:03
0.44
5.03
0:02
6:00
0.51
0.16
6:00
1112
0UT1
2.11
0.22
0.2
3.0
0.2
60
c0
D004
3:08
0.03
07
11
01
0:00
0:21
0122
0.81
0.5
0:11
0.00
$102
0:14
0.00
0:11
0:01
0:01
5.0
0.
2.44
215
0.27
0.22
5.0
5.01
0.21
2.07
0.51
0.22
DUTT
2.0
0.2
0.0
114-
0:00
D:03
3:54
0.
0.51
0.0
0.72
0
0.02
0:21
6:H1
1:13
9:26
0/22
D100
0:95
0:07
0.2
2.58
0:51
0.01
0.07
9:
0.52
0.21
0:02
0.99
0.02
0:02
212
9.51
0.7
0.0
2.0
0.5
3.24
5:00
0.02
0.54
0.09
0:03
2.00
0.09
2.01
0.61
0.02
0:21
0:02
0.00
0.6
3.0
0.01
6.00
2.01
2011
0.5
0.2
0.22
0.01
0.0
0.2
0.0
Dosriituuing Blcc
Diper: Bostngg
Ecantt Rouking Bbok
Capet
0.27
0.18
D07
0.07
8:29
0:55
0.12
0.14
0.22
2.4
6.0
C67
0.0
0.
2.0
6.1
0.2
0.22
0.1
0:13
0.24
0012
201
0001
0.0
0.40
0:04
1912
0.22
RAT
D:51
0:63
3.46
8.0
0:00
0-2
0.0
6.6G
0:00
0
2.00
S100
0.06
0:01
0.0
0:23
8:
0:27
0.52
5.05
DUT
0.03
8:09
5.55
3.0
0.2
0:15
6:
0:22
0:5%
0:03
0:66
0.0
0:24
0:18
004
3.00
0.25
0.2
0.0
0.16
3:19
6:22
3.6
0.0G
0.24
0.15
0.0
0:04
D02
9:58
8.57
D/0
0.00
3.5
0.0
0-2
0.0
00
0.0%
9:76
61
0.29
0.56
0.0
0.5
3.49
3:9
2:11
6:11
6:25
0.2
0.47
0.0
0.4
6.0
0:21
0:50
80
0211
0:02
0.02
0.22
0.50
0.22
2.4
2.2
0.0
0.0
2.0
0.2
Dypen Soutiig Block
Supert Seeting occia
Eclent Roldiing Stoictt
Expert Roudiig Siide 20
0:43
2.22
002
0.1
0:25
2.17
2.00
0.25
0:72
5.05
2:07
002
DOT
0.25
0.04
5.55
0002
0.02
0.22
0.27
0.4
2:33
0.00
0.2
2:28
0.00
002
2.10
0.35
WM2
5.01
0.02
3.54
0.24
0.11
0.04
D101
0.11
0.60
0.23
0115
0.21
CUD1
0:20
TD2
8:28
0:09
5:50
0.23
6:0
0.32
3.00
0.00
0:11
0.02
0.01
0.0
0100
0:01
0.69
3:15
0.36
0.01
0:12
3.25
8:06
D:34
B:17
0:02
0:19
0.29
0.
0:27
3:54
8:15
9.2
0/15
0.
0.06
0.04
0:25
2.39
DUN
0.15
2.20
3.51
8:17
0177
4:4
5.11
6.42
8:21
0022
2.11
5.00
0.52
0.2
0.04
0.50
3.5
6.0
9.2
0.6
R-2
0.21
2.44
0:02
0.05
0.04
3.60
0.05
0.24
03
0.02
5.0
0.21
2.11
0.2
CAN-
2.47
2.05
D:24
0.1
0.7
2.00
0.75
0.50
0:21
3.50
0.2
2.
R/2
0.21
0.
0.07
0043
0.01
2:30
DUM
2.14
3.11
0.22
0.0
0-22
RUOT
0.73
S.07
0.2
DoersRouthh g
$1
DasenRotthigg Brock
Capers Souting 20120
Expert Rouding Block 24
3.52
6:0
3.
3:25
0.32
5.0
0.2
0.16
0.01
3.0
0.00
0:14
0.
0:04
0.03
8:
2.00
5.51
TD2
60
674
2
6.0
2.4
0.2
0.00
C.T
6:13
0.0
2.0
6.
60
DAX
80
3:00
S/
36
D/D
0/2
RX2
0-2
8.0
6:21
60
0/
2:16
0:00
C
DIN
20
2:7
24
6.2
0.0
221
03
20
5.0
10
SUD
92
22
R2
00
80
0.
2.0
622
20
80
82
2.0
2.25
201
0:07
0.0
2.0
5.07
244
00
38
021
241
03
00
5.0
00
3.0
03
360
02
3.22
022
4qurr
Saler
Gune
Laker Baneea
4quar
Laurra
7
Squrts
Doper1
Sapest
Eoue's
fipere
Laurts
Figure 26: Routing distribution learnt by Ensemble in the encoder routing blocks (1-24) of T5- (5--GLUE
40

---

Published in Transactions on Machine Learning Research (05/2024)
3.01
0.2
9.01
3.02
2.40
3:02
C
0.03
0.02
9.21
0.79
0.15
0:12
022
0.0
0:22
6:01
3.4
0:
0:40
0:01
0.32
0.00
6.65
8:
6:00
0:22
9.09
0:0
9:1
601
00
3.01
(:22
0:11
600
0
2:1
5:00
0:22
(121
Y02
50
011
25
0.2
00
0.0
0:14
3:50
0:09
20
00
8.0
0.0
3.0
03
20
D0
02
9.9
2
50
0:01
0:33
RUOT
C0
0.5
114
2
5.01
0000
2.00
5.04
3.02
0.22
2.03
0.0
0.2
0.0
0.C
3.22
0:02
0.22
0777
0.01
6.27
D02
0O
0.15
0.50
0.22
0:15
0.
0.2
0.21
221
2.0G
Expert Rocting Bloce
Exces Routing Block
Eccert Roudinng Sbock
Exeent Rouding Slock 32
60
00
0.4
9:29
0
1.27
0:11
0:11
0:00
6:2
0.
1.66
0.0
0:11
0.6
3.0
0:01
6.0
3.
60
0.0
3.0
0/2
CD
C0
0:44
244
D0T
0
2.61
0-2
0.41
0.0
5.02
0:11
4.01
0.64
0.31
0.60
0.6
000
0.3
00
60
0.2
0.0
0.6
3.2
60
0.0
0:7
0-2
C2
CG
0.2
T02
00
80
D00
6/3
1114
81
5.0
DU
20
02
0.0
00
0.0
5:19
R:1
00
0.0
<10
2:22
00
S1
3:23
CO
0:30
5:04
0:02
0.21
02
2.
0.2
0.5
0.
2
00
0.
0.07
0.27
2.
0:1
0.77
00
0.2
2.0
0.2
60
00
344
6-3
02
0.2
6G
0:01
5.02
3.00
0:21
0.2
0.0
0.11
0.0
3.2
0-3
60
0.3
0.0
0.6
3.5
60
0.0
0.0
0.3
Expart
Block
Excunt Routing Block
Capart Routing DOC25
Expert Routing Black
co
00
0.4
6-
27A
0.3
0.61
0.0
6.79
6O
0.2
0.0
0.2
0.3
3.0
0.
0.
6G
0.3
0.0
0.5
3.2
6.
0.
3.0
0.3
6.7
0.
t:
6/3
0.0
0:11
0.5
0.
0:81
81
5:0
8:00
0.0
D/01
01:11
00
0.0
0.0
0.
0.0
0.50
6.0
5.0
0.
0:9
00
266
0.52
5.0
0.2
0.2
0.
0.2
0.0
0.5
02
2.0
0.2
0.0
110
2
ST
0.20
0.02
0:75
0:0T
mm
00
00
2S
0.09
0.0
6.0
5.0
R-2
0.2
0D0
2.0
0.2
0:25
00
22
6.3
0
0.74
0-
20
0.0
01
6.0
0.0
0.
0.6
3.0
0.3
0.
0.
0.
0.0
0.5
0.2
0.0
0.
2.0
0.2
0.
0.2
Expet Thetin Bock
Excet Roating Black
Capert Routigg Back
Ecant Rouling
00
211
0:55
C60
2.0
2.00
200
0.17
24
2.12
02
0G
0.05
0.00
00
001
0.77
0.2
00
60
0.22
0.0
0.5
02
6D
0.0
241
0.2
6.0
19C2
00
00
38
0.0
60
0/2
0.0
00
3.0
0.0
0.5
00
0.70
00
0.0
0.
R22
118
00
G2
00
3.6
3.59
6:13
0.66
5.02
0.2
0.0
0.0
0.06
0.01
6.67
0.2
0.07
0/12
6:07
261
0.
0.27
67
00
00
98
0.2
60
0.7
3.5
0.0
0:21
0.
0:51
9.0
0.0
0.2
6
20
6.12
04
0.29
0.21
0.0
0.2
2.0
2.61
2.4
6.0
92
2.20
0.0
0.0
2.5
0.2
0.
0.0
0.2
0.0
0.5
2.2
0.0
0.0
2.0
0.2
0.00
EiserRRouting Rock 41
DaseeRouthhg Block
Capar So wwt.nnngg Rrock 43
Exsen Routing Block 61
2:5
00
04
6.2
0.0
2:7
0.2
0.22
0.0
25
DD
0.0
00
0.5
0:74
0.00
0.22
2.07
0.2
0.071
0.2
22
60
0.0
2.0
0.2
WM2
0.00
60
2.01
6.7
0.0
TD2
0.54
00
0.39
0.12
0.2
mm
00
8:00
9:37
6.2
9:27
5.00
6:76
9.
9:2
0.09
0:37
R:14
0.2
0.0
0.2
22
0:41
00
24
6.2
0.0
0:52
20
8.0
2.4
6.0
24
R-2
0.0
80
0.6
0:03
0.2
0:0F
2:00
9.2
0.22
DOT
22
5.0
0.0
2.0
0.2
10
CAN
011
00
24
0.2
0.0
0.2
20
5.0
24
82
R:2
0.0
80
0.0
0:55
RU0T
21
2:09
02
2.0
0.5
02
5.0
0.0
2.0
0.2
0.
0.2
Exput Rosting Bock 45
Easas Routin Block
Cipen Souting D B0C47
Exsert Routing Block
43
002
3.52
0.35
0:00
3:37
007
8/7
02
0.04
3.11
0:11
0.44
CD4
3.00
0.25
0.59
5:44
9.22
5.57
0:11
TD2
3.22
6.2
35
3.22
0:21
0.23
0.01
5.00
M
D01
0.00
0.3
0.
213
DUT
0.22
4:45
3:00
5:18
3.60
5.6
8:21
0.
9:06
5:14
RX2
3.66
99
8:27
4:11
PM
6n8
0.57
2:04
22
5.2
0.2
2.0
2.02
0.04
2.77
2.04
0.1
8:5
R02
222
0A4
0.5
3.0
20
0.42
0.50
2D
0.
002
3.54
0.15
0:02
BUOT
0.23
2:04
055
244
0.52
4:00
602
832
0.33
5.11
2.04
3.23
3.55
0.37
0.00
CIM
0.2
5.00
8:23
0X2
Aquera
3
S
Banet
Saler
Salura
Guner
Laker
Equar
Lauerg
Squrt
Poper
Sapest
Eours
fipere
Lawrts
Figure 27: Routing distribution learnt by Ensemble in the decoder routing blocks (25-48) of 755-GLUE
41

---

Published in Transactions on Machine Learning Research (05/2024)
Reading Rluuk
Roating
Pagant
6:72
8:09
0:31
2:30
0:21
5.0
2.19
S.5
1.10
EUST
0:
0.115
0:04
0:05
182
45
0:
60
0/
19:11
01:11
0:51
0.51
0:08
5.0
6.00
0:09
DAN
9:61
01.04
0:29
0:51
0:66
8:17
DVD
50
0.77
24
5.0
5:09
2:07
0:18
0.0
6:0
222
3.0
03
2.0
2:02
0.11
0.00
0-45
0
1.16
2.04
0.21
0.21
2:75
DO2
5:51
DOZ
118
50
0.71
83
60
OUT
0102
0:71
8:77
0.0
9.
D.0
0:0
5:00
3.0
0.0
0.39
5.5
R:38
RKY
D107
0.35
0:02
8:09
5:55
CVOR
0
32
2:50
2.0
0.
0.05
0.52
0.21
0:5
0.0
2.77
0:03
0.
0.03
0:54
6:07
Canet Roetrng Strch
Creert Raadrng
Daper Suttrellcc
0.00
0.12
3.22
0:15
0.21
0:22
2.61
9.21
OLT1
119
0.12
0.05
0:37
0:23
6:07
018
3.09
0.32
0.07
0:53
0:22
0:00
D03
5:0
2.70
0:52
0.2
0.02
0.12
0.0
0:38
3:
3:51
0:
3.00
0:18
3:44
9:21
5:58
0:44
0:02
6:00
0-02
0:01
0:00
0:21
60
BUSS
6:14
6:00
2:33
6:22
0:01
000
0:02
6-12
2.28
0:02
6:00
0.3
0:01
3.44
6.22
2:77
6:44
0:
0:55
0.28
0:12
5.67
0:02
0.13
K:TI
6:07
277
8:11
0:32
6:11
0:23
011
0.05
2.25
0:11
0.15
5.10
6:1
C
0:22
3.09
2:11
2.55
0:22
573
DOX
0.02
0:21
2:57
2:07
032
0:6
2:77
2.9
2:22
0.03
0:71
0:02
0.21
0.05
2:30
0.11
0:04
9:44
2:11
0:00
0:02
R:21
0.01
0.21
8:03
6.02
3.81
0.00
6.62
0.81
0.03
Expurt Fosting Bocx57
Daper Raudio 26
Caper Souting Duck Duck
DosenPoutng Block 20
0.27
3:67
0.11
0.07
0.21
0.01
3.61
8:23
0.52
6.04
0:53
0.23
5.00
2.50
0:22
0.14
0.2
2.77
0.31
5.61
2.51
5.06
0.03
0.03
0:02
0.22
0.1
5.09
2.14
3:30
0.21
0.51
0.61
0.02
5.02
6:1
6.44
3:14
6.66
0:22
6:22
0.1
0:28
3.12
0.21
0.04
0.5
0.16
65
6:11
0:23
0.19
6:11
3.99
6.22
0.01
2.40
0.00
0.
6.02
034
4.00
3.1
0.22
0:63
0:1
5:19
818
600
0121
8:9
2:00
821
mm
2.25
8:1
DAY
0:1
I12
D:22
8:1
S
Eype12
S
Baper
Dosee
Daser
Domve
Basert
Doper
Dyne
Cower
Dowe
Boser
Eamm
Figure 28: Routing distribution learnt by Ensemble in the decoder routing blocks (49-60) of T5- GLLU
42


Reviews:
[{"review_id": "7h4npsOfUG", "reviewer": "Reviewer_PJBA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The provided method shows novelty, allowing differentiable routing provides more opportunities to improve MoE training.\n2. The computational cost is not significantly increased.\n3. The routing mechanism avoids the use of discrete decisions and instead proceeds through a learned distribution, which enables the model to be trained end-to-end via standard backpropagation.\n4. SMEAR avoids adding significant computational cost by computing the output of a single merged expert, while maintaining the ability to learn adaptive routing strategies.\n\nWeaknesses:\n1. It is not clear how large the extra computation costs that the average operation introduces.\n2. Optimizing all experts during each forward propagation requires extra memory to store gradients, this extra cost should be considerred and analyzed.\n3. More experiments of larger models are recommended.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, authors hypothesize that the non-differentiable discrete routing decisions in gradient estimation techniques are root causes of this performance degradation. From this point, authors introduce Soft Merging of Experts with Adaptive Routing (SMEAR), using a single \"merged\" expert that is the weighted average of all experts' parameters to replace discrete routing. This strategy enable similar computational costs of SMEAR and standard gradient-based training. Authors empirically validate that SMEAR outperform other baselines, and provide qualitative analysis demonstrating that the experts learned via SMEAR exhibit specialization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weakness 1 and 2. Futhermore, to ensure reproducibility, it would be better to provide the source codes and experiment instructions.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EI7r1uJZpv", "reviewer": "Reviewer_MRC3", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:  \n- This paper is well-written and easy to follow. \n- The proposed method is simple and effective, which is based on a clear motivation.  \n- The experimental validation is extensive and sufficient.  \n\nWeaknesses:  \n- Although SMEAR claims to not incur significant additional computational costs compared to discrete routing, the paper acknowledges that there is a computational overhead associated with averaging the parameters of the experts. While this cost may be justified by the performance gains, a more detailed analysis or comparison with other methods could provide better insights.  \n- SMEAR assumes a common architecture among experts, which may limit its applicability to scenarios where experts have divergent architectures. This assumption should be carefully considered, as it might affect the generalizability of the proposed method to a broader range of tasks and domains.  \n- Although the paper compares SMEAR with models using gradient estimation and heuristic routing strategies, the experiments are also quite sufficient. However, additional comparisons with other state-of-the-art methods or benchmarks could provide a more comprehensive evaluation of its performance across various tasks or datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the adaptive routing problem which aims to learn a routing path to different expert subnetworks. The modular models often underperform compared to parameter-matched dense counterparts, and the using a non-learned heuristic routing strategy causes suboptimal performance, the authors propose SMEAR method which avoids discrete routing decisions by merging experts through a weighted soft merging process, further enabling standard gradient-based training without significantly increasing computational costs. Through extensive experiments, the performance SMEAR is validated to outperform those using other routing methods and demonstrate meaningful specialization of experts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses for details.", "broader_impact_concerns": "no ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CQcBsEd863", "reviewer": "Reviewer_jS8e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "=== Strengths === \n\n(S1): The problem of learning routing without resorting to gradient approximation is important. The approach proposed by the authors is reasonable and quite simple; its explanation is very clear. \n\n(S2): The paper outlines different routing variants very well, explaining the differences between them in a clear manner. The authors also do a good job in surveying and analysing existing works. The paper is generally quite polished and well-written.\n\n(S3): The paper includes many quantitative and qualitative results, which I find rather convincing. \n\n \n\n=== Weaknesses === \n\n(W1): While empirical timings shown in the paper indicate that the overhead from weight averaging in SMEAR is not too big, I am a bit confused about what effect does SMEAR have on efficient batching during training (or during batched inference). Methods such as the sparse Mixture-of-Experts from Shazeer et al effectively divide an incoming batch between experts, so that each expert roughly has to process a batch of size `batch_size / num_experts`; if the original `batch_size` is large enough this ensures the experts themselves can run efficiently. In SMEAR, it seems to me that each forward pass will instantiate `batch_size` different averaged experts, and then each of those experts will have to be evaluated on a single input. Did the authors needs to take special care to make this operation efficient, e.g. using an operation similar to `torch.bmm`? If so, this should be discussed explicitly. Relatedly, I wonder whether the timings from Figure 2 show batched inference, or are models shown there running a single example at a time.\n\n(W2): While reading the paper I was confused about the computational overhead of the weight averaging in SMEAR; being more used to models that route individual token embeddings, computing an averaged expert for each token seemed prohibitively expensive. Eventually I learned the routing is done on the image/sequence level. I think this could be mentioned or alluded to earlier in the paper to make for a smoother reading experience. Relatedly, I wonder what the consequences of routing on the sentence level really are – would SMEAR work much better if routing was more fine-grained? I also wonder if routing simply based on averaged activations is expressive enough, given e.g. the experience in Graph Neural Networks showing that weighted aggregation methods are often superior. \n\n \n\n=== Missing related work === \n\nI think the paper does a good job at discussing a large body of related works. Here I list some more papers that could also be discussed. \n\n- [1] is a work earlier than Latent Skills that proposes a similar routing strategy, also based on a binary task-expert allocation matrix. It compares with an MoE baseline and (somewhat similarly to the authors of the work under review here) finds that the routing learned by MoE is highly suboptimal, hypothesizing it is due to training difficulties. \n\n- Related to weight averaging, there is also a body of work on Stochastic Weight Averaging, which is averaging of weights of the same model across different points during training. This is similar to averaging different fine-tuned versions of the same base model, as in this case the averaged models also \"share a common past\". This is commonly used to improve generalization at no extra cost [2, 3]. \n\n \n\n=== Nitpicks === \n\n- In `{f_1, f_2, … f_n}` (bottom of page 2), perhaps there should be a comma after the `…`. \n\n- \"the router R acts an agent\" - missing \"as\".\n\n- Missing braces in two places in the description of DSelect-k \n\n- \"Beyond the simple baseline discussed above\" - not sure which baseline does this refer to (or should it be plural?) \n\n- Missing parenthesis around citation of Houlsby et al (Section 4). \n\n \n\n=== References === \n\n[1] Maziarz et al, \"Flexible Multi-task Networks by Learning Parameter Allocation\" \n\n[2] https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/ \n\n[3] Izmailov et al, \"Averaging Weights Leads to Wider Optima and Better Generalization\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes SMEAR - an alternative to discrete Mixture-of-Experts models. SMEAR routes each input example to a single soft expert computed as a weighted average of all experts within a layer, instead of performing discrete routing to route to a single expert. The authors show that this does not incur a large overhead if routing is done on a coarse level (entire image/sequence instead of individual tokens/pixels). Finally, the paper shows that this routing strategy performs well and learns sensible routing patterns.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would request that the authors clarify (W1) in some way, and add the references I mentioned if they believe they are indeed relevant. Resolving (W2) would also be good, although that is a smaller issue.", "broader_impact_concerns": "I believe this work poses no outstanding ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the adaptive routing problem which aims to learn a routing path to different expert subnetworks. The modular models often underperform compared to parameter-matched dense counterparts, and the using a non-learned heuristic routing strategy causes suboptimal performance, the authors propose SMEAR method which avoids discrete routing decisions by merging experts through a weighted soft merging process, further enabling standard gradient-based training without significantly increasing computational costs. Through extensive experiments, the performance SMEAR is validated to outperform those using other routing methods and demonstrate meaningful specialization of experts. The authors address reviewers' concerns well in their rebuttal. Thus, I would like to recommend accept as is.
Decision: Accept (poster)