Sparse MoEs meet Efficient Ensembles
Authors: James Urquhart Allingham
Florian Wenzel
Zelda E Mariet
Basil Mustafa
Joan Puigcerver
Neil Houlsby
Ghassen Jerfel
Vincent Fortuin
Balaji Lakshminarayanan
Jasper Snoek
Dustin Tran
Carlos Riquelme Ruiz
Rodolphe Jenatton
Venue: Accepted by TMLR ()

Abstract: Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, often exhibit strong performance compared to individual models. We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs). First, we show that the two approaches have complementary features whose combination is beneficial. This includes a comprehensive evaluation of sparse MoEs in uncertainty related benchmarks. Then, we present efficient ensemble of experts (E$^3$), a scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up to 45% fewer FLOPs than a deep ensemble. Extensive experiments demonstrate the accuracy, log-likelihood, few-shot learning, robustness, and uncertainty improvements of E$^3$ over several challenging vision Transformer-based baselines. E$^3$ not only preserves its efficiency while scaling to models with up to 2.7B parameters, but also provides better predictive performance and uncertainty estimates for larger models.

Published in Transactions Machine Learning Research (09/2022)
Sparse MoEs meet Efficient Ensembles
James Urquhart Allingham
joa23 joam3 ac. uk
Florian 2e
fin. www.ellanaaaata..o com
Zelda E Marict, Basil Mustafa
(amariet, Esmarieelbusiim @google.com com
Joan Puigcerver, Neil Houlsby
(jonigeerver Sgooge com
Ghassen Jerfel31 3.
ghassen@@oogle..omm com
Vincent Fortuin 1,4.+
ubf2 @com. ac.
Balaji Lakshminarayanan, Jasper Snoek
faalajjinn,ssoodl @google. com
Dustin Tran, Carlos Riquelme, Rodolphe Jenatton
(trandustin rjenatton @ggoogle.com com
Google Research, Brain Team; Universsiy of Cumbridge; noo affitiation; Wayme; 4 ETHH Zürich
Reviewed OpenReview: https: // openrevsew. net/ forum? id- 40Z33642QU
Abstract
Machine learning models based on the aggregated outputs of submodels, either at the acti- vation or prediction levels, often exhibit strong performance compared individual models. We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs). First, we show that the two approaches have complementary features whose combination beneficial. This includes comprehensive evaluation of sparse MoEs in uncertainty related benchmarks. Then, present efficient ensemble of experts (E*), scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up 45% fewer FLOPs than deep ensemble. Ex- tensive experiments demonstrate the accuracy, log-likelihood, few-shot learning, robustness, and uncertainty improvements E² over several challenging vision Transformer- based base- lines. F not only preserves its efficiency while scaling to models with up 2.7B parameters, but also provides better predictive performance and uncertainty estimates for larger models.
Introduction
Neural networks (NNs) typically use all their parameters to process an input. Sustaining the growth of such models reaching today up to 100B+ parameters (Brown et al., 2020) is challenging, e-g., due to their high computational and environmental costs (Strubell et al., 2019; Patterson et al., 2021). In this context, sparse mixtures of experts (sparse MoEss) employ conditional computation (Bengio et al., 2013) to combine multiple submodels and route examples to specific "expert" submodels (Shazeer al., 2017; Lepikhin et al., 2021; Fedus et. al., 2021; Riquelme et. al., 2021). Conditional computation can decouple the growth of the number of parameters from the training and inference costs, by only activating a subset of the overall model in an input-dependent fashion.
Paralleling this trend, the deployment. of ML, systems in safety-critical fields, e.g. medical diagnosis (Dusen- berry et al., 2020b) and self-driving cars (Levinson al., 2011), has motivated the development of reliable
Work done a Google Research intern. Work done while at Google Research.

---

Published in Transactions Machine Learning Research (09/2022)
MoMol: orr
Thaeroosen Block
Nonsi
MSAA
Nonas
TRAnsFonscn Blooss
M.M BLOCK
CLASSETIER
1
KLP (
Essesice (MEAN)
1u2,1
getegg (W2 (1i.)
22
A
(K =
1ss,
(12
h3.1
Aug
(
A
3
gatey (W2
awng (
Jot 2-Mox BLock cocar
(P2
A2
Figure I: End-to-end overview of E with E 6 experts, partitioned into = groups, with sparsity of K K= and "last-2" configuration. Top: E² contains sequence of transformer blocks, followed by alternating transformer and p(artitioncd)-
blocks. As in VIT, images are split into patches whose embeddings are
processed by each block. Here, we show 1 embedding for each of three images ( D. [). Bottom left: In a p-MoE block, replace the transformer block's MLP with parallel partitioned expert MLPs, see (2). The effect of the routing weights is not depicted. Embeddings are tiled (2) in the first p-MoE block only. Bottom right: The classifier averages predictions from the final liled representations (I)
deep learning, c.g., for calibrated and robust predictions (Ovadia et al., 2019). Among the existing approaches, ensembles of NNs have remarkable performance for calibration and accuracy under dataset shifts (Ovadia et al., 2019). These methods improve reliability by aggregating the prodictions of individual submodels, referred as ensemble members. However, this improvement comes at significant computational cost. Hence, naively ensembling NNs that continue to grow in size becomes less and less feasible. In this work, we try to overcome this limitation. Our core motivation is to improve the robustness and uncertainty esti- males of large-scale fine-tuned models through ensembling, but to do in tractable and thus practically useful manner, by carefully developing a hybrid approach using advances in sparse MoEs.
While sharing conceptual similarities, these two classes of models MoEs and ensembles have different propertics. Sparse MoEs adaptively combine their experts depending the imputs, and the combination generally happens at internal activation levels, Ensembles typically combine several models in a static way and al. the prediction level. Moreover, these two classes of models lend to be benchmarked tasks: few-shot classification for MoEs (Riquelme et al., 2021) and uncertainty- evaluation for ensem- bles (Ovadia et al., 2019; Gustafsson et al., 2020). For example, sparse MoEs are seldom, ever, applied the problems of calibration.
different.
Here, we study the interplay between spanse MoEs and ensembles. This results in two sets of contributions:
Contribution 1: Complementarity of sparse MoEs and ensembles. We show that sparse MoEs and ensembles have complementary features and benefit from each other. Specifically:
The adaptive computation in sparse MoEs and the static combination ensembles are orthogonal, with additive benefits when associated together. At the intersection of these two model familles is an exciting trade-off between performance and compute (FLOPs). That is, the frontier can be mapped out by varying the ensemble size and the sparsity of MoEs.
Over lasks where either sparse MoEs or ensembles are known to perform well, naive- and computa- tionally expensive-ensenilee of MoEs provide the best predictive performance. Our benchmarking effort includes the first evaluation of sparse MoEs on uncertainty-related vision tasks, which builds upon the work of Riquelme et al. (2021).
Contribution 2: Efficient ensemble of experts. We propose Efficiont Ensemble Exports (E²), see Figure I, an efficient ensemble approsch tailored to sparse MoEls:

---

Published in Transactions Machine Learning Research (09/2022)
improves over sparse MoEs across low-shot crror, likelihood and calibration error. g matches the performance of deep ensembles while using from 30% to 45% fewer FL.PSs
Ed gracefully scales up to 2.7B parameter models.
n. both simple requiring only minor implementation changes and convenient F. models can be line-tuned directly from standard sparse-MoE checkpoints. Code be found at https:// github. com/ google- reeeggeeeeeearcc/eeoo research/vmoe.
2 Preliminaries
We focus on classification tasks where we learn classifiers of the form f(x; @) based on some training data D = (a Bn) A pair (Zv, Ve) corresponds an input Is R. together with its label yn E (l,..., C} belonging to one of the C' classes, The model f(: is parametrized by 0 and outputs C-dimensional probability vector. We use to refer to the matrix element-wise product.
2.1 Vision Transformers and Sparse MoEs
Vision Transformers. Throughout the paper, we choose the model f to be a vision Transformer (ViT) (Dosovitskiy al., 2021). VIT is growing in popularity for vision, especially in settings where it was shown to outperform convolutional networks while requiring fewer pre-training re- sources. ViT operates at the level of patches. An input image is split into equal-sized patches (e.g., 32 x 32, 16 16, 14 14 pixels) whose resulting sequence (lincarly) embedded and processed by Trans- former (Vaswani et al., 2017). The opcrations in the Transformer then mostly consist of succession of multiheaded self-attention (MSA) ) and MLP layers. VIT is defined at different scales (Dosovitskiy et al., 2021): S(mall), B(ase), (arge) and H(uge); specifications in Appendix A.1. For example, ViT-L//6 stands for a large VIT with patch size 16 x 16.
Sparse MoEs and V-MoEs. The main feature of mparsely-gated mixture-of-expert models (sparse MoEs) lies in the joint use of sparsity and conditional computation (Bengio et al., 2013). In those mod- els, we only activate small subset of the network parameters for given input, which allows the total number of parameters 0 to grow while keeping the overall computational cost constant. The experts are the subparts of the network activated on a per-input fashion.
Central to our study, Riquelme et al. (2021) recently extended VIT to sparse MoEs. Their extension, referred to as V-MoE, follows the successful applications of sparse models in NLP (Shazeer et al., 2017). Riquelme al. (2021) show that V-MoEs dominate their "dense" ViT counterparts on a variety of tasks for the same computational cost. In the specific case of V-MoEs, the experts are placed in the MLP layers of the Transformer, a design choice reminiscent of Lepikhin et al. (2021) in NLP. Given the input h E R.D of such layer, the output of single MLP(h) is replaced by
MoE(h) = => ge (h) (h) with to (h)) topx (opp(s))ttmx (Wh)), c=1
(1)
where the routing weights (h))) combine the outputs of the E different experts (MLP,) ) To sparsely select the experts, tOpk sets all but the K largest weights to zero. The router parameters W E REXD are trained together with the rest of the network parameters. We call the layer defined by (1) an MoE layer. In practice, the weights (h)) are obtained by noisy version of the routing function (softtaxx oE)) with E ~ (0, I), which mitigates the non-differettiabliity of topx when combined with auxiliary losses (see Appendix A in Shazeer al. (2017)). Making non-dilferentiable operators smooth with some noise injection is an active area of research (Berthet et al., 2020; Abernethy, 2016; Duchi et al., 2012). We use the shorthand gate, (z) = topx tma((= + rε)) and take o = 1/EE as Riquelme et al. (2021).
In this paper, we consider the "last-n" setting of Riquelme et al. (2021) wherein only few MoE layers are placed at the end of the Transformer (n = 2 for the {S, B, L} scale and n = 5 for H). This setting retains most of the performance gains of V-MoEs while greatly reducing the training cost.

---

Published in Transactions on Machine Learning Research (09/2022)
Table 1: Overview of key properties of sparse MoEs, ensembles, and E E² acheives the best of both worlds. dense a base model upon which we add the sparse MoE or ensemble logic, e.g., VIT model this paper.
PREDICTIONS
COMBINATIONS
CONDITIONAL COMPUTATION
COST
Sparse MoEs
Single
Activation level
Yes, aduptively per-input
2 dense
Ensembles
Multiple
Prediction level
No, static
dense
Multiple
Activation & prediction level
Yes, adaptively per-input
A dense
2.2 Ensembles of Neural Networks
Ensembles. We build on the idea of ensembles, which is known scheme to improve the performance of individual models (Hansen & Salamon, 1990; Geman el. al., 1992; Krogh & Vedelssy, 1995; Opitz & Maclin, 1999; Dietterich, 2000; Lakshminaaayanau el al., 2017). Formally, we assume a set. of M model parameters θ 10.m J We refer to M as the ensemble size. Prediction proceeds by computing के Eoce f(x; 0), i.e., the average probability vector over the M models. To assess the diversity of the prodictions in the ensemble, we will use the KL divergence (f
f(r:; 0m/)) between the predictive distributions (ar;; 0m) and [1z;; ), averaged over the test input x, and all pairs (m, m') of ensemble members.
Batch ensembles. Wen et al. (2019) construct a batch ensemble (BE) as a collection of submodels, with the parameters 6m E θ sharing components. This mitigates the computational and memory cost of ensembling, while still improving performance. We focus on the example of a single dense layer in f with parameters U R assuming no bias. BE defines M copics of parameters {U,) J.M that Um U o (rmss)) where U are parameters shared across ensemble members, and I'm and 8m are separate D- and L-dimensional vectors for ensemble member m. Given an input, BE produces M putsuts which are averaged after applying all layers. Despite the simple rank-1 parametrization, BE leads to remarkable predictive performance and robustness (Wen et al., 2019). Notably, the elliciency of BE relies on both the parameter sharing and the tiling of the inputs to predict with the M ensemble members, two insights that we exploit in our paper.
2.3 Pre- Pre-training and Fine-tuning
Large-scale Transformers pro-trained on upstream tasks were shown to have strong performance when fine- tuned smaller downstream tasks, across a variety of domains (Devlin al., 2018; Dosovitskiy al., 2021; Radford et al., 2021). We follow this paradigm and focus on the fine-tuning of models pre-trained on JFT- 300M (Sun et al., 2017), similar to Riquelme et al. (2021). We will thus assumee the availability of already pre-trained ViT and V-MoE model checkpoints. Our assumption relies on the growing popularity of transfer learning, e.g. Kolesnikov et al. (2020), and the increasing accessibility of pre-trained models in repositories such as www. tensorflow. org/hub or www www-pytorch. org/hub. The fine-tuning of all the approaches we study here, including extensions of VIT and V-MoE, will be either directly compatible with those checkpoints or require only mild adjustments, e.g., reshaping or introducing new downstream-speeeffe parameters (see Appendix B). Also, unless otherwise mentioned, the performance we report will always be downstream, e.g. for ImageNet (Deng et al., 2009) CIFAR10/100 (Krizhevsky, 2009). In all our comparisons, we will use the downstream training floating point operations per second (FLOPs), or GFLOPs (i.e., 10 xFLOss, to quantify the computational cost of the different methods.
3 Sparse MoEs meet Ensembles
As illustrated in Table 1, sparse MoEs and ensembles have different properties. For instance, ensembles typically do not use conditional computation and just statically combine members at the prediction level. This contrasts with sparse MoE's where the different. experts are combined at internal activation levels while enjoying per-input adaptivity through the routing logie, see (1). In terms of cost, sparse MoEs are usually designed to match the inference time of their dense counterparts whereas ensembles, in their simplest forms, will typically lead substantial overhead. In this section, we study the extent which these properties are complementary and may benefit from cach other. In Section 5, we further evaluate this complementarity on tasks where either sparse MoEs or ensembles are known to perform well, e.g., seo-shot and (OOD) evaluations, respectively.

---

Published in Transactions Machine Learning Research (09/2022)
425
59
4075
150
-19
000
0.0
135
1.
(a) Log-Likellnn Likelihood (L.L)
(b) (downstream) GFL.OPs
(c) log GFLOPSIK GFLOFS
T. Mn Manaj
Figure 2:
Increasing static (M) and adaptive (K) ensembling on ImageNet, for V-MoE-S/322 Yel-
low/pur /purple indicates better/ worse performance. Increasing both static and adaplive ensembling is ben- eficial, the latter being more efficient.
To invostigate the interactions of the properties in Table 1, we study the performance of downstream deep ensembles (i.e., with ensemble members having the same upstream checkpoint) formed by M independent V-MoEs with E experts per MoE layer and sparsity level K (the larger K, the more selected experts). M controls the static combination, while K and E' impact the adaptive combination of experts in each sparse MoE model. We report in Figure the ImageNet performance and compute cost for ensembles with varying choices of K and M, while keeping E 32 fixed. We focus K rather than E to explore adaptive computation, as we found the performance quickly platcaus with E' (see Figure 12 in the Appendix). Also, by fixing E 32, we match more closely the setup of Riquelme et al. (2021). The architecture of the V-MoE is VIT-S/32 see details in Appendix A.7. We make the following observations:
Investigating the cumulative effects of adaptive and static ensembling. In the absence of ensembles (i.e., when consider M 1), and given fixed number of experts, the authors of Riquelme et al. (2021) already reported an increase in performance as K gets larger. Interest- ingly, we observe that for each value of K, it is also beneficial increase the ensemble size M. In other words, the static combination of ensembles is beneficial when applied to sparse MoEs. This observation is perhaps surprising since adaptive combination may already encapsulate the effect of static com- bination. Figure 3, and Appendix L.1, show that the combination of static ensembling and adaptivity beneficial to NLL a range of VIT familics. We also that the benefits of static ensembling similar for V-MoE and VIT (which does not have any adaptivity).
0.9
NLL (lower better)
0.8
0.6
Investigating ensembles of sparse MoEs with fewer experts.
In Appendix I.2, we compare the performance of V-MoE with E 32 exports and ensembles V-MoEs with fewer experts, namely (M = E 16) and (M as measured by NLL is better for (M = E = 8) than (M = 2, =1) which is turn better than (M 1, E 32). Thus, we conclude that reducing the namber of experts only mildly affects the combination of adaptive and static ensembling.
E 8). We that the performance e.g-,
downstream log(GFLOPs) (lower is better)
Figure 3:
ImageNet evalu-
ation for VIT
and V-
MoE. (K = 1) (+) ensembles of
Investigating the trade-off between FLOPs and performance. Without any computational constraints, the previous observation would favor approaches with the largest values ever, different values of (K, M) lead different computational costs, as measured here by FLOPs, with (K, M) (1, 1) being the cheapest. Figure shows, as expected, that number of FLOPss grows more quickly along the M axis than along the K axis. To capture the various trade-offs at play, in Figure 2c report the logarithm of the normalized gains in log likelihood aFLOP -PT when going from (K, .)) = 1) to other choices (). Interestingly, it appears more advantagoous to first grow K, i.c., the adaptive combination, before growing M.
size 2, (o.
Model
sizes: S/32 (+). B/32 L/32 (+),
and M. How-
VIT and V-MoE benefit from ensembling
1./16 (+), and H/14
equally, all scales.

---

Published in Transactions on Machine Learning Research (09/2022)
Summary. While simple ensembling sparse MoEs results strong prodictive performance, we lose their computational efficiency. We next show how to efficiently combine ensembling and sparse MoEs, exploiting the fact that statically combining sparse MoEs with fewer experts remains effective.
4 Efficient Ensemble of Experts
Equipped with the insights from Section 3, we describe efficient ensemble of experts (E*), with the goal of keeping the strengths of both sparse MoE' and ensembles. Conceptually, E,3 jointly learns ensemble of smaller sparse MoEs, where all layers without experts (e.g,, attention layers) are shared across the members.
4.1 The Architecture
There are two main components in g::
Disjoint subsets of experts. We change the structure of (1) by partilioning the set of E experts into M subsets of E/M experts (assuming that E is multiple of M). We denote the subsets by Em For example, & = {1, and & = (4,5.6] 6] for = 6 and
2. Intuitively, the ensemble members have separate parameters for independent predictions, while efficiently sharing parameters among all non-expert layers. Instead of having single routing function gate (W.) as in (1), we apply separate routing functions gatex m) each subset Em. Note that this does not affect the total number of parameters since W has E rows while each Wm. has E/M rows A similar partitioning of the experts was done in Yang et al. (2021) but not exploited to create different ensemble members, in particular not in conjunction with tiled representations, which we show to be required to get performance gains (see Section 4.2.1).
Tiled representation. To jointly handle the predictions of the M ensemble members, we tile the inputs by factor M, inspired by Wen al. (2019). This enables a simple implementation of E² on top of an existing MoE. In Appendix E, connect sparse MolEs and BE, illustrating that tiling naturally fits into the formalism of sparse MoEs. Because of the tiling, a given image patch has M different representations that, when entering an MoE layer, are each routed to their respective expert subsets &m Formally, consider some tiled inputs H E B xxxxxxx xD where B refers to the batch size (a batch contains image patches) and h,m e R the representation of the i-th input for the m-th member. The routing logic re can be written as
p-MoE(hi,m))
Σ Be (hemm) (h..(h with (he gate ( (mmhi.)) EEE...
(2)
where the routing weights are now W.. E ((E/M)xD; see Figure 1.
To echo the observations from Section 3, we can first see that E brings together the static and adaptive combination of ensembles and sparse MoEs, which we found to be complementary. However, we have seen that static ensembling comes at the cost of a large increase in FLOPss, thus we opt for an elficient ensembling approach. Second, we "split" the MoE layers along the axis of the experts, i.e., from E experts to M times E/M experts. We do so since we observed that the performance of sparse MoE's tends to plateau quickly for more experts. We note that E² retains the property of ensembles to output multiple predictions per input.
In generic implementation, inputs Xiiled = X;:..: X] e R(M-B)xP and the model processcs f(X 0). Since tiling in g has an effect only from the first MoE layer onwards, we postpone the tiling operation to that stage, thus saving otherwise redundant prior computations in non-Moli-layers. For example, for 1//16 and K . 2, we can save about 47% of the FLOPs. Further implementation details of E², and a discussion of the increased memory consump- Lion due to tiling, in Appendix C. Code be found at https //gps:///thtttt com/ geogle-research/ Finally, we note that although E² and BE share conceptual design similarities tiled representation and sharing of parameters they differ in fundamental structural ways, see Appendix F.
tile batch of B inputs X E BxP by factor M to obtain the tiled
4.2 Ablation Studies: Partitioning and Tiling
Our mothod introduces two changes to V-MoEs: (a) the partitioning the exports and (b) the tiling of the representations. In this section, we assess the separate impact of each of those changes and show that.

---

Published in Transactions on Machine Learning Research (09/2022)
Table 2: ImageNet performance (means = standard errors over 8 seeds) of E BB-B (K = M = 2), V-MoE
(K 4), and two ablations: only tiling and only partitioning. The noise in gatex is denoted by a.
NLL 4
ERROR +
ECE +
KL
V-MoE E
0,636
0.001
16.70
0.04
0.034
0.001
0.612
0.001
16.49
d: 0.02
0.013
d: 0.000
0.198
0.003
Tiling
0,637
# 0.002
16.74
0.00
0.028
0.001
0.000
#: 0,000
Tiling (a 2)
0.638
1: 0.001
16.72
0.03
0.033
1: 0.002
0.001
1: 0.000
Tiling (or 4)
0.638
o
16.74
0:08
0.033
0.001
0.002
0.000
Partitioning 0.640
16.72 0.05
0.034
0.001
indeed their combination that explains the performance gains. We summariiz the results of this study in Table 2 which shows ImageNet performance negative log likelihood (NLL), classification error, expected calibration error (ECE) (Guo et al., 2017), and Kulbbck-Leiin
divergence for different ablations of Ea. See Appendix N, for end-to-end overview diagrams in the style of Figure 1, for the these ablations. We E = 32 experts. We provide FLOP measurements for these ablations in Appendix M.I.
4.2.1 Partitioning without Tiling
We first compare g² with a variant of V-MoE where we only partition the set of experts (Partitioning). In this variant, each input h E IRP (note the dropping of the index m due to the absence of tillng) can select K experts in subset e,,, resulting in a total of K M selected experts per input. Formally, (2) becomes
M
partitiioning-onlyyookkkk = Σ Σ
ga(h)) (h,,(hs) with (go(hi))oce..- gattll)tee....ttee/e gate, (Wmh,))
cEEm
The eapert prototyping of Yang et al. (2021) leads to a similar formulation. As shown in Table 2, across all metrics, Partitioning is not competitive with E². We do not report KL since without tiling, Partitioning does not output multiple predictions per input.
4.2.2 Tiling without Partitioning
We now compare gd with the variant where only the tiling is enabled (Tiling). In this case, we have tiled inputs H E RBXMxD applied to the standard formulation of (1). Compared with (2), there is mechanism to enforce the M representations of the i-th input across the ensemble members, ie.., (MoE()..m M to be different. Indeed, without partitioning, each him could select K identical experts. As a result, we expect Tiling to output M similar predictions across ensemble members. This is confirmed in Table 2 where we observe that the KL for Tiling orders of magnitude smaller than for r.3. To mitigate this effect, we also kried to increase the level of noise in gatex (by a factor {2, 4)), cause the expert assignments differ across [ While we do see an increase in KL, Tiling still performs worse than E² across all metrics.
Interestingly, we can interpret Tiling as an approximation, via M samples, of the marginalization Ee....r (z; 0)] with respect to the noise {eihf= in the É MoE layers of [((:0) (further assuming the capacity constraints of the experts, as described in Riquelme et al. (2021), do not bias the M samples).
4.2.3 Tiling with Increasing Parameter Sharing
The results in Table 2 (as well as Appendix 1) suggest that the strong performance of n:3 is related to its high-diversity predictions. We hypothesise that this diversity is a result. the large number of non-shared parameters in each ensemble member, i.e., the partitioning of the experts. To test this hypothesis, we allow the subsets &m in g3 to have some degree of overlap (i.e., ensemble members share some experts), thus interpolating between g² and Tiling. For example, with total experts E = 32 and an ensemble size M 2, an overlap of 8 shared experts mcans that cach ensemble member has (32 8)/2 = 12 unique experts, and 12 + 8 = 20 in total. Table 3 shows that increasing the number of shared experts directly decreases diversity and thus NLL, Error, and ECE. We see the same trends for K (rather than K 2; Table 18).

---

Published in Transactions on Machine Learning Research (09/2022)
Table 3: ImageNet performance (means = standard errors over 8 seeds) of E B/32 (K = M = 2), E = 32 lotal experts, and varying expert overlap bet ween subeets &.
OVERLAP
NLL 4
ERROR 4
ECE +
KL 1
O (=B²)
0.612
0:001
16.49
0.02
0.013
0,000
0.198
0.005
0.617
4: 0.00s
16.55
i 0.09
0.016
0.001
0.167
4: 0.005
4
0.622
0.001
16.62
0.02
0.017
0.001
0.148
0.005
8
0.627
0.0
16.67
0.07
0.021
0.001
0.122
0.010
16
0,639
+ 0.004
16.82
0.07
0.030
0.00s
0.077
+ 0.011
32 (-Tiling)
0.637
16.74
0.028
0.000
Table 4: ImagoNet performance (mcans + standard errors over 8 sceds) of V-MoE-B/32 and a simple multi-prediction variant (Multi-pred)) whose last MoEE layer is changed in (3).
K
NLL +
ERROR +
ECE 4
KL.1 1
E (M- = 2)
0.622
0,001
16.70
0.00
0.018
0.000
0.217
-0.00
Multi-pred
0.636
+ 0.001
17.16
0.02
0.024
+ 0,000
0.032
0.001
V-MoE
2
0.638
16.76
0.033
TO (M- = 2)
2
0.612
0.001
16.49
0.013
0.198
0.003
Multi-pred
4
0.645
1 0.001
17.39
0.04
0.021
0.000
0.011 I 0.001
V-MoE
0.636
# 0.001
16.70
0.04
0,034
# 0.001
B (M = 2)
0.611
0,001
16.45
0.03
0,014
0,000
0.193
0.005
Multi-pred
8
0.650
+ 0.001
17.50
0.08
0.021
0,000
0.003
0.000
V-MoE
0.635
4: 0.002
16.72
0.06
0.028 0.001
4.2.4 Multiple Predictions without Tiling or Partitioning
As highlighted in Table 1, an ensemble of size M outputs M predictions for given imput (thercafter, averaged) while sparse MoEs only produce a single prediction. Thus, a natural question how much the gains of g are simply due to its ability to produce multiple predictions, rather than its specific tiling and partitioning mechanisms? To answer this, we investigate a simple preditpredittin variant of sparse MoEs (Mulli-pred) wherein the last. MoE layer of the form (1) is replaced by
Multi-pred-MoE(h) = (h) (1)) E RK with {ge(h)) = gate (Wh),
(3)
where we have assumed MLP (h) R., Instead of summing the expert outputs like in (1), we stack the K selected expert contributions (as a reminder, gateg zeroes out the E K smallest weights). Keeping track of those K contributions makes it possible to generate K different predictions per input as in the classifier of Figure 1, thus aiming at capturing model uncertainty around the true prediction.
Table 4 compares the ImageNet performance of this simple multiple-prediction method with the standard V-MoE and E². In all cases, Multi- pred under performs relative to g3, Indeod, despite improvements ECE, it is only for K = 2, that Multé-pred provides small gains in NLL relative to V-MoE, while its classification error is always worse. In fact, Multi-pred for K = 4 performs worse in terms of NLL, classification error, and diversity than for K = 2. The KL diversity metric indicates that the Multi-pred unable provide diverse predictions. This indicates that it is specifically tiling and partitioning in ES that provide good performance.
4.3 Comparison with Alternative Efficient Ensembling Strategies
In the previous subsection we saw that a simple approach to multiple predictions a V-MoF, model is unable to achieve good diversity in predictions and thus strong predictive performance. Following Havasi et al. (2020); Soflaci et al. (2020), a possible fix to this problem would be to have a multi-prediction and mulli- imput approach. Furthermore, other efficient ensembling strategies could provide alternative solutions this problem. Unfortunately, as we show in Table 5, while common efficient ensembling strategies such as BE (Wen et al., 2019), MC Dropout (Gal & Ghahramani, 2016), and MIMO (Havasi et al., 2020), do improve slightly on ViT/V-MoF, they are unable to match the performance of E.. In Appendix G, we provide

---

Published in Transactions Machine Learning Research (09/2022)
Table 5: ImageNet performance (means = standard errors over 8 seeds) of different efficient ensemble
approaches based on VTT-B/32 architecture.
M
NLL +
ERROR 4
ECE +
KL 1
GFLOPs
VIT BE VIT
0,688
0.005
18.65
0.022
0.000
78.0
2
0.682
0.003
18.47
0.03
0.021
i: 0.000
0.040
0.001
97.1
V-MoE
0.638
0.001
16.76
0.05
0.033
0.001
94.9
MC Dropout V-MoE
2
0.648
0.002
17.10
0..8
0.019
1: 0.001
0.046
0.000
97.2
MIMO V-MoE
2
2
0.636
0.002
16.97
0.04
0.028
0:001
0.000
00000
96.3
4
0.672
0.001
17.72
0.04
0.037
0.000
0.001
0.000
99.0
2
0.622
+ 0.001
16.70
0.08
0.018
# 00000
0.217
0.008
105.9
ImageNet
ImageNet
Mean Acroes Datasets
Mean Acroes Datasets
0.45
IntStoc (lower is better)
0.8
0.46
25
17
20
0.5
0.42
15
0.1
14
3.6
3.0
3.2
3.4
3.6
downstream leg(GFL.OPs) (lower better)
V-MoE
VIT
S/32
R/32
L/32
L/16
Hr14
Member
Members
Members
Figure 4: ImageNet NLL (left, center left) and mean 10-shot error across datasets (center right, right), with woomed-in plots of highlighted areas. Zoomed-in plots include additional ensemble baselines for comparison. Dashed lines show Pareto frontiers which tend to be defined by g..
detailed description of how we carefully apply these methods ViT ViT/ TV-/VE to ensure fair evaluation (sometimes even designing extensions of these methods). We also give results for additional K and M values.
Evaluation
We now benchmark g against V-MoE. As a baseline we also include results for dounstream ensembles of V-MoE and VIT. These ensembles offer a natural baseline against E² they also use single upstream check- point, are easy to implement, and provide consistent improvements upon V-Molf. In Appendix J, we compare with upstreom ensembles that require multiple upstream checkpoints (Mustafa al., 2020). All results cor- respond to the average over 8 (for (S, B, L) single models) or 5 (for H single models and all up/ doswerset ensembles) replications. In Appendices L and M we provide results for additional datasets and metrics as well as standard errors. Following Riquelme et al. (2021), we compare the predictive-performmnnt cost trade-offs for each method across a range of ViT familics. In the results below, E² uses (K, M) = (1,2), single V-MoE models use = 2, V-MoF, ensembles use = I, and all use = 32. Experimental details, including those for upstream training, downstream fine-tuning hyperparameter sweeps and (linear) few-shot evaluation can be found in Appendix A. Our main findings are as follows:
compute
5.1 V-MoE vs. ViT
Ensembles help V-MoE just as much as VIT. Ensembling was expected to benefit. VIT. However, Figures 3 to 8 suggest that ensembling provides similar gains for V-MoE in terms of few-shot performance, NLL, ECE, OOD detection, and robustness to distribution shift. We believe this has not been observed before. Moreover, downstream ensemble with four H/14 V-MoEs leads 88.8% accuracy on ImageNet (even reaching an impressive 89.3% for upstream ensemble that further benefits from multiple upstream chockpoints, see Table 15).
ViT consistently provides better ECE than V-MoE. Surprisingly, despite V-MolE tending have

---

Published in Transactions Machine Learning Research (09/2022)
ImageNet
ImageNet
ImageNet-C: (wverage)
ImageNNttAA
0.40
BCB (lower bettrr)
0.06
0.35
0.30
0,01
0,0000
0.23
0.02
0,0175
0.04
0.30
0.15
0.01
0.0125
0.02
0.30
downstream log(GFLOOPs) (lower better)
V-MoE
VIT
S/32
B/32
L/32
L/16
H/14
Member
Members
Members
Figure 5: ECE for ImageNet (left), with zoomed-in plot of the highlighted area (center left), and under distribution shift. (right, center right). Thiss is a metric for which ensembles are known to perform well whereas, to the best of our knowledge, the performance of V-MoE has not been evaluated. The zoomed-in plot includes additional ensemble baselines for comparison. Dashed lines show Pareto frontiers. CIFARI0 Places365
CIFARIO CIFARIO0
CIFARIO CIFARI00
CIFARIO SVHN
0.060
0.055
0.12
PPR295 (lower
0.20
0.11
0.015
0,12
0.30
0,140
0.01
0.135
0.00
0.030
downstream (og(GFLOPs) (lower better)
g
V-MoE
VIT
S/32
BV/22
L/32
L/16
Member
Members
Members
Figure OOD detection, measured by false positive rate 95% precision (Fort al., 2021), for models fine-tuned on CIFARIO, with zoomed-in plot of the highlighted area. This is metric for which ensembles are known to perform well whereas, to the best of our knowledge, the performance of V-MoE has not been evaluated. The zoomed-in plot includes additional ensemble baselines for comparison. Dashed lines show Parcto frontiers.
better NLL than VIT (Figure 3), their ECE is worse (Figure 5).
ECE is not consistent for different VIT VVT/V-MoE familics. We see the ECE, unlike other metrics presented in this work, tends not to provide consistent trends as we increase the VIT family size (Figure 5). This observation is consistent with Minderer et al. (2021), who noted similar behaviour for a range of models. They note that post-hoe temperature scaling can improve consistency.
V-MoE outperforms VIT in OOD detection. With L/32 being the only exception, V-MoE outper- forms VIT on range OOD detection tasks (Figure 6). While this may seem surprising, given the opposite accurate predictions about the scale of the uncertainty timates while V-MoE makes better predictions about the relative ordering of the uncertainty estimates. For smaller VIT families, V-MoE outperforms VIT in the presence of distribution shift. contrast the OOD detection results, Figure shows that for smaller ViT families V-MoE improves on the performance of VIT. However, as the VIT family becomes larger, this trend less consistent.
trend for FCE, it suggests that VIT makes
5.2 Efficient Ensemble of Experts
improves classification performance. As shown Figure 4, 3 either on, or very near to, the Pareto frontiers for NLL and 10-shot classification error, despite the fact that these are metrics for which ensembles and V-MoE, respectively, are known to perform well. Figure 8 shows that similar conclusions hold for CIFAR10/100 NLL.
3 performs best at the largest scale. The difference in predictive performance between r:3 and V-
10

---

Published in Transactions Machine Learning Research (09/2022)
ImageNes-C (avergge)
ImageNet-C (average)
ImageNet-A
ImageNet- V2
2.5
1.6
NL. (lower better)
1.10
2,0
1.05
1.00
1.2
1.5
0.95
1.0
1.0
5
0.8
0.85
3.0
3/4
downstream log(CFLLPs)) (lower betler)
F.3
V-MoE
VIT
S/32
R/32
L/32
L/16
H/14
Member
2 Members
Members
Figure NLL under distribution shift for models trained on ImageNet. For ImageNet-C, we provide zoomed-in plot of the highlighted arca. The zoomed-in plot includes additional ensemble baselines for comparison. Dashed lines show Parcto frontiers which tend to be defined by g3.
CIFARIO
CIFARIO
CIFARIO-C
CIFARI00
0,07
0.45
NLL better)
11.6
0.5
0.40
0.05
0.005
0.35
0.04
0,000
0.03
0.25
0.025
0.20
2.50
2.75
3,00
downsttream (og(GGLLOPs) (lower is hetter)
E
V-MoE
VII
8/32
B/32
L/32
L/16
Member
Members
Members
Figure 8: NLL for CIFARIO (left), with zoomod-in plot of the highlighted area (center left., CIFAR10- C (center right), and CIFAR100 (right). The zoomed-in plot includes additional ensemble basclines for comparison. Dashed lines show Pareto frontiers.
increases as the VIT family becomes larger (Figures 4 to 8, and Appendix D where we propose scheme to normalize performances across scales).
MoE ensembles
g3 becomes Pareto efficient for larger VIT familics the presence distribution shift. Figure 7 shows that E² is more robust to distribution shift for larger VIT families, despite less consistent V-MoE performance at scale. When averaged over the shifted datasets (ImageNet-CC, ImageNet-A, ImageNet- V2), 3² improves on V-MoE terms of NLL for all ViT families other than S/32, with improvements up 8.33% at the largest scale; see Appendix M.3.
E improves ECE over ViT and V-MoE. Despite V-MoE providing poor ECE, E does not suffer from this limitation (Figure 5). For most ViT families, E also provides better ECE than V-MoE ensembles. E does not provide consistent OOD detection performance. Firstly, Figure 6 shows that for small VIT families, 3 performs worse than V-MoE and (even VIT in some cases). Nevertheless, as above, the relative performance improves for larger VIT families such that g becomes Pareto efficient for two dataset pairs. Secondly, F. seems to perform better on the more dillicult near OOD) detection task (CIFARIO vs. CIFAR100). These results, although sometimes subtle, are consistent across OOD detection metries and dataset pairs, shown in Appendix L.
Summary. While no single model performs best in all of our evaluation settings, we do find that E performs well and is Pareto efficient in most cases. This is particularly true for the larger ViT families. One exception was OOD detection, where g performance was somewhat inconsistent. On the other hand, while V-MoE clearly outperforms VIT in terms of accuracy, the uncertainty estimates can better or worse depending the downstream application.

---

Published in Transactions on Machine Learning Research (09/2022)
6
Related Work
Mixture of Experts. MoEs (Jacobs et al., 1991; Jordan & Jacobs, 1994; Chen al., 1999; Yuksel et al., 2012; Eigen et al., 2014) combine the outputs of different submodels, or experts, in an input-dependent way. Sparse MoEs only select a few experts per input, enabling to greatly scale models while keeping the prediction time constant. Sparse MoEs have been used to build large language models (Shazeer et al., 2017; Lepikhin et. al., 2021; Fedus et al., 2021). Recently, sparse MoE's have been also successfully applied to vision problems (Riquelme et al., 2021; Yang et al., 2021; Lou et al., 2021; Xue al., 2021). Our work builds on the V-MoE archilecture proposed by Riquelme el al. (2021), which is based the vision Transformer (VIT) (Dosovitskiy et al., 2021) and showed improved performance for the same computational cost as ViT. We explore the interplay between sparse MoEs and ensembles and show that V-MoEs benefit from ensembling, by improving their predictive performance and robustness. While previous work studied ViT's calibration and robustness (Minderer et al., 2021; Fort et al., 2021; Paul & Chen, 2021; Mao et al., 2021), we are the first to study the robustness of V-MoE models.
Ensembles. Ensemble methods combine several different models to improve generalization and uncertainty estimation. Ensembles achieve the best performance when they are composed of diverse members that make complementary errors (Hansen & Salamon, 1990; Fort et al., 2019; Wenzel et al., 2020; D' D'Angelo & Fortuin, 2021; Lopes et al., 2021). However, standard ensembles are inefficient since they consist of multiple models, each of which can already be expensive. To reduce test time, Xie et al. (2013) and Hinton et al. (2015); Tran et al. (2020); Nam et al. (2021) use compression and distillation mechanisms, respectively. To reduce training time, ensembles can be constructed with cyclical learning-rate schedules to snapshot models along the training trajectory (Huang et al., 2017; Zhang et al., 2019). Our work builds on batch ensemble (Wen et al., 2019) where a single model encapsulates an ensemble of networks, a strategy also explored by Lee et al. (2015); Havasi et al. (2020); Antorán et al. (2020); Dusenberry et al. (2020a); Rame et al. (2021). Wenzel et al. (2020) extended BE to models with different hyperparameters.
Bayesian Neural Networks. BNNs are an alternative approach to deep ensembles for uncertainty quan- tification in neural networks. In BNN the weights are treated as random variables and the uncertainty in the weights is translated to uncertainty in predictions via marginalisation. However, because the weight pos- terior is intractable for NNs, approximation is required. Popular approximations include variational inference (Hinton & Van Camp, 1993; Graves, 2011; Blundell et al., 2015), the Laplace approximation (MacKay, 1992; Ritter et al., 2018), and MC Dropout (Gal & Ghahramani, 2016). However, many of these approximations make restrictive mean-field assumptions which hurt performance (Foong et al., 2020; Coker et al., 2022; For- luin et al., 2021). Unfortunately modeling full weight correlations for even small ResNets wiihhh relatively few parameters compared to the ViT models considered here is intractable (Daxberger et al., 2021).
7 Conclusions and Future Work
Our study of the interplay between sparse Molss and ensembles hasshown that these two classes of models are symbiotic. Efficient ensemble of experts exemplifies those mutual benefits- as illustrated by its accuracy, log- likelihood, few-shot learning, robustness, and uncertainty calibration improvements over several challenging baselines in a range of benchmarks. We have also provided the first, to the best of our knowledge, investigation into the robustness and uncertainty calibration properties of V-MoEs showing that these models are robust to datasct shift and are able to detect OOD examples. While our study has focused on downstream fine-tuned models, we believe that an extension to the upstream case and from-scratch training, would also result in a fruitful investigation. In fact, in Appendix K we provide some promising, but preliminary, results for from- scratch training. Similarly, although we have focused on computer vision, our approach should be readily applicable to the modelling of text, where sparse Mol's have been shown to be remarkably effective. With the growing prevalence of sparse MoEs in NLP (Patterson et al., 2021), the questions of understanding and improving the robustness and reliability of such models become increasingly important. Furthermore, the computational scale at which those models operate make those questions challenging to tackle. We beliove that our study, and approaches such as 1. make steps in those directions.
12

---

Published in Transactions Machine Learning Research (09/2022)
Acknowledgements
We would like to extend our gratitude to Josip Djolonga for having reviewed an earlier version of the draft and for helping with robustness senattees__etrics (Djolonga al., 2020). We would also like thank André Susano Pinto and Cedric Renggli for fruitful comments on the project, as well as Jakob Uszkoreit for insightful discussions. JUA ackuowledges lunding from the EPSRC, the Michael E. Fisher Studentship Machine Learning, and the Qualcomm Innovation Fellowship. VF acknowledges funding from the Swiss Data Science Center through a PhD Fellowship, as well as from the Swiss National Science Foundation through a Postdoc.Mobility Fellowship, and from St. John's College Cambridge through Junior Rescarch Fellowship.
References
Jacob D. Abernethy. Perturbation techniques in online learning and optimization. 2016.
Javier Antorán, James Urquhart Allingham, and José Miguel Cornández-Lobato. Depth uncertainty in neural networks. In Advances in Newral Information Processing Systems, 2020.
Monika Bansal, Munish Kumar, Monika Sachdeva, and Ajay Mittal. Transfer learning for image classification using 19g19: Caltech-101 image data set. Journal of Ambient Intelligence and Humanized Computing, pp- 1-12, 2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arzie preprind armiv:1308.9435 2013.
Quentin Berthet, Mathieu Blondel, Olivier Teboul, Marco Cuturi, Jean-Philippe Vert, and Francis R. Bach. Learning with differentiable perturbed optimizers. CoRR, abs/2200.008670 2020. URL https://arxiv org/abs/2002. 08676.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning, pp- 1613 1622, 2015.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderrann-iingg and Qiao Zhang, JAX: compos- able transformations of Wothon-NumP? programs, 2018. URL http://github ctwc google/jax.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arziv preprint arziv:2005. 110000...41166 14165, 2020.
Ke Chen, Lei Xu, and Huisheng Chi. Improved learning algorithms for mixture of experts in multiclass classification. Neural Networks, 1999.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Beau Coker, Wessel P. Bruinsma, David R. Burt, Weiwei Pan, and Finale Doshi Volez. Wide mean-field bayesian neural networks ignore the data. In Gustau Camps-Valte Francisco J. R. Ruiz, and Isabel Valera (eds.), Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pp. 5276-5333. PMLR, 28-30 Mar 2022. URL https: https://proceedings. www.ggs.m...ppp......... www.lr.prees/v1///oollooo htmml.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Francesco D' D'Angooo and Vincent Fortuin. Repulsive deep ensembles are bayesian. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances Newral Information Processing Systems 34: Annual Conference on Newral Information Processing Systems 2021, NewrIPS 2021, December 6-14, 2021, virtual, pp. 3451-3465, 2021. URL https:/ /atp:://rocceddnee Abatrttcttt.baaatt...aaaaaaaat.e html..
13

---

Published in Transactions on Machine Learning Research (09/2022)
Erik A. Daxberger, Eric T. Nalisnick, James Urquhart Allingham, Javier Antorán, and José Miguel Hernández-Looato. Bayesian deep learning via subnetwork inference. In Marina Meila and Tong Zhang (ods.), Proceedings of the 98th International Conference on Machine Learning, ICML 2021, 18-24 July 2021; Virtual Eeent, volume 139 of Procerdings of Machine Learning Research, pp. 2510 2521. PMLR, html.
2021. URL http::/ppooceee http://prroceddings www.g..........mm.. mlr. https///11d//xabeggeeee
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. CVPR, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Loe, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arziv preprint arsiv: 1 11t111810..0000 2018.
Thomas G Dietterich. Ensomble methods in machine learning. In International workshop on maltiple
classifier systems, pp. I 15. Springer, 2000.
Josip Djolonga, Frances Hubis, Matthias Minderer, Zachary Nado, Jeremy Nixon, Rob Romijnders, Dustin Tran, and Mario Lucic. Robustness Metrics, 2020. URL https://github: com homm//googlleeerrcee robustness _enustness_metrics.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Golly, Jakob Uszkorcit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
John C. Duchi, Peter L. Bartlett, and Martin J. Wainwright. Randomized smoothing for stochastic opti- mization. SIAM J. Optim., 2272):674-701, 2012. doi: 10.1137/110331659, URL https://dol. 1137/110831659.
Michael W Duscnberry, Ghasson Jerfel, Yoming Wen, Yi-an Ma, Jasper Snock, Katherine Heller, Balaji Takshminargyanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank-1 factors. In International conference machine learning, 2020a.
Michael W Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M Dai. Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference, and Learning, pp. 204-213. 2020b.
David Eigen, Mare' Aurelio Ranzato, and Nya Sutskever. Learning factored representations in a deep mixture of experts. In ICLR (Workshop Poster), 2014.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. orziv preprint arziv:2101.0900 2021.
Andrew Y. K. Foong, David R. Burt, Yingzhen Li, and Richard E. Turner. On the expressiveness of approximate inference in bayesian neural networks. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceediie Ttt.:////S.co00001000111111111a1
html.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspective. arsic preprint arnin:191 1912. 191200257, 2019.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of arnin, 2021.
detection.
Vincent Fortuin, Adrià Garriga-Alonso, Florian Wenzel, Gunnar Rätsch, Richard E. Turner, Mark van der Wilk, and Laurence Aitchison. Bayesian neural network priors revisited. CoRR, abs/2102.06371, 2021. URL
wrtpa////2///b///2..... 065711.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International conference machine learning, pp. 1050 1059, 2016.
14

---

Published in Transactions Machine Learning Research (09/2022)
Stuart Geman, Elic Bienenstock, and René Doursat. Noural networks and the bias/variance dilemma. Neurul computation, 4(1):1 58, 1992.
Alex Graves. Practical variational inference for neural networks. In J. Shawe- Taylor, R. S. Zemel, P. L. Bartlett, F. Percira, and K. Q. Weinberger (ods.), Advances in Neural Information Process- ing Systems 24, pp. 2348-2356. Curran Associates, Inc., 2011. URL http://papers mips. cc/paper/ 4329
pdf.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1821-1330. PMLR, 2017.
Fredrik K Gustafsson, Martin Danelljan, and Thomas B Schon. Exaluating scalable bayesian deep learning methods for robust computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognilion Workshops, pp.
2020.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions pattern analysis and machine intelligence, 12(10):933-101,, 1990.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: dala mining,
inference, and prediction. Springer, 2017.
Marlon Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe Liu, Jasper Snock, Balaji Lakshmi-- narayanan, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust pre- diction. In ICLR, 2020.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. aiXiv preprint arXiv:1903. 122611, 2019.
Dan Hendryeks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial exam- ples. arXiv preprint arXiv:1907.07119 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. NIPS Deep Learning and Representation Learning Workshop, 2015.
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the sisth anauall conference on Computational learning theory, pp. 5-13, 1993.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q WWeinberger. Snapshot ensembles: Train 1, get m for free. arriv preprint arrio: 704.00100 2017.
R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Newral Compulation, 1991.
Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Newrol Compulalion, 1994.
Jakob Nikolas Kather, Cleo-Aron Weis, Francesco Bianconi, Susanne M Melchers, Lothar Schad, Timo Gaiser, Alexander Marx, and Frank Gerrit Zöllner. Multi-class texture analysis in colorectal cancer his- tology. Scientific reports, 6(1):1 11, 2016.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual reprosentation learning. In ECCV, pp. 491-507. Springer, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained catego- rization. In 4th International IEEE Workshop on 3D Representation ond Recognition (3ARR-13), Sydncy, Australia, 2013.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
15

---

Published in Transactions Machine Learning Research (09/2022)
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances in neural information processing systems, pp. 231 238, 1995.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive un- certainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6412, 2017.
Stefan Lee, Seuthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads better than one: Training a diverse ensemble of deep networks. arziv preprint arzio: 151 1.06314, 14, 2015.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. ICLR, 2021.
Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Socren Kammel, J Zico Kolter, Dirk Langer, Oliver Pink, Vaughan Pratt, et al. Towards fully autonomous driving: Systems and algorithms. In 2011 IEEE Intelligent. Vehicles Symposium (IV), pp. 163 168. IEEE, 2011.
Raphael Gontijo Lopes, Yann Dauphin, and Ekin D. Cubuk. No one representation to rule them all: Overlapping features of training methods. CoRR, abs/2110. 12899, 2021. URL https://arxiv, org/abe/ 2110. 2119.12899. 12899.
Yuxuan Lou, Fuzhao Xue, Zangwei Zheng, and Yang You. Sparse-mlp: fully-mip architecture with conditional computation. arric, 2021.
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computation, 4 (3):448 472, 1992.
Xiaofeng Mao, Gege Qi, Yueleng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer, 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic. Revisiting the calibration of modern neural networks, arzio, 2021.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Deep ensembles for low-data transfer learning. arXiv preprint orX:22010006666 2020.
Giung Nam, Jongmin Yoon, Yoonho Lee, and Juho Lec. Diversity matters when learning from ensembles. In M. Ranzato, A. Beygelzimer, Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neurel Information Processing Systems, volume 34, pp. 8367 8377. Curran Associates, Inc., 2021. URL https: pdf.
//proceedings. neurips.... hww.rps.ccpppper////...
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Maria-Elent Nilsback and Andrew Zisserman, Automated flower classification over large number of classes. In 2008 Skath Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722 729. IEEE, 2008.
David Opitz and Richard Maclin. Popular ensemble methods: An empirical study. Journal of artificial intelligence research, 11:169 198, 1999.
Yaniv Ovadia, Emily Fertig, J. Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498 3505. IEEE, 2012.
16

---

Published in Transactions Machine Learning Research (09/2022)
David Patterson, Joseph Gonzalez, Quoe Lc, Chen Liang, Lluis-Miquel Munguia, Damiel Rothchild, David So, Mand Texier, and Jeff Dean. Carbon emissions and large neural network training. arziv preprint arriv:2104.10550 2021.
Sayak Paul and Pin- Yu Chen. Vision transformers are robust learners. arzie, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arzie preprint arri::2103.000 00000000, 2021.
Alexandre Rame, Remy Sun, and Matthieu Cord. Mixmo: Mixing multiple inputs for multiple outputs via doep subnetworks. arXiv preprint orXi::2133.0611 2021.
Benjamin Rocht, Rebocca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389 5400, 2019.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. arziv preprint arri::2106. 050744, 2021.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural nelworks. In 6th International Conference on Learning Representations, ICLR 2018- Co1-Confeee Track Proceedings, volume 6. International Conference on Reproscntation Learning, 2018.
Noam Shazeer, Azalia Mirhoscini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outragcously large neural networks: The sparsely-gated mixture-of-exparr layer. In ICLR, 2017.
Masoumeh Soffaei, Hongyu Guo, Ali Al-Bashabsheh, Yongyi Mao, and Richong Zhang. Aggregated learn- ing: A vector-quamizatioon approach to learning neural network classifiers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5810, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Tyy Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way prevent neural networks from overfitting. The journal of machine learning rescarch, 15(]): 1929 1958, 2014.
Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Usækoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. anXiv preprint arXi::1006. 10270, 2021.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. arziv preprint arrie:1906. 02204,,,,3 2019.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843 852, 2017.
Linh Tran, Bastiaan S. Veeling, Kevin Roth, Jakub Swiatkosski, Joshua V. Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserving ensemble diversity for model distillation. CoRR, abs/2001 04094, 2020. URL /ttps://arxiv, http:///aaxxvv..///a////////////
org//bs//000111oo.. 04694.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, 2017. URL https://proceeddngss neurips. .df.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds- 200-2011 dataset. 2011.
Yoming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficiont ensomble and lifelong learning. In ICLR, 2019.
17

---

Published in Transactions Machine Learning Research (09/2022)
Florian Wenzel, Jasper Snock, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robust- nes8 and uncertainty quantification. In Newral Information Processing Systems); 2020.
Jingjing Xie, Bing Xu, and Chuang Zhang. Horizontal and vertical ensemble with deep representation for classification. arziu, 2013.
Fuzhao Xue, Ziji Shi, Futao Wei, Yuxuan Lou, Yong Liu, and Yang You. Go wider instead of deeper. arain 2021.
An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, et al. Exploring sparse expert models and beyond. arziv preprint 0rziv21055 15082, 2021.
Yi Yang and Shawn Newsam. Bag-of-visual-woore and spatial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL international conference on advances in geogrophic information systems, pp. 270-279, 2010.
Seniha Esen Yuksel, Joseph N Wilson, and Paul Gader. Twenty years of mixture of experts. IEEE transactions neural networks and learning systems, 23(8):1177-1199, 2012.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. anXiv preprint or Nio:2100. 04 560, 2021.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. International Conference on Learning Representations, 2018.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Audrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In ICLR, 2019.
Bolei Zhou, Agala Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
18

---

Published in Transactions Machine Learning Research (09/2022)
Appendix Structure
This appendix structured as follows:
We provide experimental details, such as model specifications, upstream and downstream training hyperparameters, and few-shot evaluation settings, in Appendix A.
We discuss the compatibility and adaption of upstream V-MoB and VIT checkpoints to the various models considered in this paper, in Appendix B.
We provide implementation details for E in Appendix C.
We show that E. provides larger relative improvements versus V-MoE for larger models, in Ap- pendix D.
We draw connections between BE and sparse MoEs, in Appendix
We highlight differences between E and BE, in Appendix F.
We compare empirical results for g and various other efficient ensemble approaches applied to VIT and V-MoE, in Appendix G.
We provide sensitivity analyses for the load balancing loss strength, expert initialisation diversity, and expert group permutation, Appendix
investigate the roles of diversity and individual model performance overall ensemble perfor- mance, in Appendix I.
We compare ensembling only downstream with ensembling that takes place both up and downstream, Appendix J.
We provide preliminary results for training on ImageNet from scratch, without any pre-training, in Appendix K.
We extend the results of Sections 3 to 5 to additional datasets, metrics, model sizes, and other settings, in Appendix L.
We provide a range of tabular results, including flops numbers, parameter connts, and standard errors for the experiments in Section 5, Appendix M.
We provide additional overview diagrams, in the style of Figure for the various algorithms and ablations presented in the main text, in Appendix N.
A Experiment Settings
A.1 ViT Model Specifications
Following Dosovitskiy et al. (2021), recall the specifications of the ViT models of different scales in Table
Table 6: Specifications of ViT-S, ViT-B, VIT-L and VIT-H.
HIDDEN DIMENSION
MLP DIMENSION
# LAYERS
Small
512
2048
Base
768
3072
12
Large
1024
4096
24
Huge
1280
5144
32

---

Published in Transactions Machine Learning Research (09/2022)
A.2 Upstream Setting
For all our upstream experiments, we scrupulously follow the setting described in Riquelme et al. (2021), see their Section B.2 in their appendix. For completeness, we just recall that S/32 models are trained for 5 epochs while B/ (16, 32} and L/32 models are trained for 7 epochs. For L/16 models, both 7 and 14 epochs can be considered (Dosovitskiy et al., 2021; Riquelme et al., 2021); we opted for 7 epochs given the breadth of our experiments. Finally, the H/14 model are trained for 14 epochs.
In particular, the models are all trained on JFT-300M (Sun et al., 2017). This dataset contains about 305M training and 50 000 validation images. The labels have a hierarchical structure, with a total of 18 291 classes, leading average to 1.89 labels per image.
A.3 Downstream Setting
During fine-tuning, there number of common design choices we apply. In particular:
Image resolution: 384.
Clipping gradient norm at: 10.0.
Optimizer: SGD with momentum (using
B 0.9).
Batch size: 512.
For V-MoE models, we finetune with capacity ratio C' = 1.5 and evaluate with C = 8.
We use the following train/ train/valide validation splits depending on the dataset:
DATASET
TRAIN DATASET FRACTION
VALIDATION DATASET FRACTION
ImageNet
99%
1%
CIFAR10
98%
2%
CIFARIOO 100
98%
2%
Oxford-IIIT Pets
90%
10%
Oxford Flowers-102
90%
10%
All those design choices follow from Riquelme et al. (2021) and Dosovitskiy et al. (2021).
A.4 Hyperparameter Sweep for Fine-tuning
In all our fine-tuning experiments, we use the sweep of lyperparameters described Table 7. We use the recommendations from Dosovitskiy et al. (2021) and Riquelme et al. (2021), further considering several factors (0.5, 1.0, 1.5, 2.0} to sweep over different numbers of steps. Riquelme et al. (2021) use a half schedule (with the factor 0.5) while Dosovitskiy et al. (2021) take the factor 1.0.
We show in Table 8 the impact of this enlarged sweep of hyperparameters in the light of the results reported in Riquelme al. (2021). We notably tend to improve the performance of ViT and V-MoE (especially for smaller models), which thus makes the basclines we compare to more competitive.
A.5 Details about the (Linear) Few-shot Evaluation
We follow the evaluation methodology proposed by Dosovitskiy al. (2021): Riquelme et al. (2021) which we recall for completeness. Let us rewrite our model f with parameters 0 = (Q,00) as
f(: sof softmax( (attmaxx(@xx::::) 0'))
where Q RCxS corresponds to the parameters of the last layer with the S-dimensional representation ((:; 0') I

---

Published in Transactions Machine Learning Research (09/2022)
Table 7: Hyperparameter values for fine-tuning on different datasets. Compared with Dosovitskiy et al. (2021) and Riquelme et al. (2021), we further consider several factors {0.5, 1.0, 1.,, 2.0} sweep over different numbers of steps.
DATASET
STEPS
BASE LR
EXPERT DROPOUT
ImageNet
20 x {0.5, 1.0,
1.5, 2.0]
(0.0024, 0.003, 0.01, 0.03)
0.1
CIFARIO
5000 x {0.5. 1.0,
1.5, 2.0
{0.001, 0.003, 0.01, 0.03)
0.1
CIFAR100
5000 x {0.5, 1.0, 1.5, 2.0}
{0.001, 0.003, 0.01, 0.03)
0.1
Oxford-LITT Pets
500 x {0.5, 1.0,
1.5, 2.0]
{0.001, 0.003, 0.01, 0.03]
0.1
Oxford Flowers-100 102
500 X {0.5, 1.0, 1.5, 2.0}
{0.001, 0.003, 0.01, 0.03]
0.1
Table 8: Impact of using the enlarged sweep of hyperparameterss described in Table 7. We typically improve the results reported Riquelme et al. (2021), therefore strengthening the baselines we compare to. The table displays means and slandard errors over 8 replications, except. for H/14 that has 4 replications. L/16"; For L/16, we consider the setting where the upstream models are trained with epochs, as opposed to 14 epochs Riquelme al. (2021), hence the slightly worse accuracy reported in this paper.
MODEL SIZE
MODEL NAME ACCURACY (THIS PAPER)
ACCURACY (Riquelme et al., 2021)
S/32
VIT V-MoE (K=2)
76.31
1 0.05
73.73
78.91 0.08
77.10
B/32
VIT V-MoE (K=2)
81.35
+ 0.05
80.73
83.24
82.60
VIT V-MoE (K=2) VIT
84.62 0.05
84.37
1./32
84.95 + 0.05
85.04
84.30
84.15
B/16
V-MoE (K=2) VIT V-MoE (K-2) VIT
85.40 # 0.04
85.39
86.63 4: 0.08
87.12
L/16*
87.12
87.54
88.01 4 0.05
88.08
11/1
V-MoE (K=2)
88.11 0.13
88,23
In linear few-shot evaluation, construct lincar classifier to predict the target labels (encoded onc-hot vectors) from the S-dimensional feature vectors induced by o(; 0"); see Chapter in Hastie et al. (2017) for more background about this type of lincar classifiers. This ovaluation protocol makes it possible evaluate the quality of the representations learned by f.
While Dosovitskiy al. (2021): Riquelme et al. (2021) essentially focus on the quality of the representations learned upstream JFT by computing the (linear) few-shot accuracy on ImageNet, we interested in the representations after fine-tuning on ImageNet. As result, we consider a collection of 8 few-shot datasets (that does not contain ImageNet)):
Caltech-UCSD Birds 200 (Wah et al., 2011) with 200 classes,
Caltech 101 (Bansal et al., 2021) with 101 classes,
Cars196 (Krause et al., 2013) with 196 classes,
CIFAR100 (Krizhevsky, 2009) with 100 classes,
Colorectal histology (Kather et al., 2016) with 8 classos,
Describable Textures Datasct (Cimpoi et al., 2014) with 47 classes,
Oxford-IIIT pet (Parkhi et al., 2012) with 37 classes and
21

---

Published in Transactions Machine Learning Research (09/2022)
UC Mercod (Yang Newsam, 2010) with 21 classes.
In the experiments, compute the few-shot accuracy for each of the above datasets and we report the averaged accuracy over the datasets, for various number of shots in (1.5, 10, 25}. As commonly defined in fow-shot learning, we understand by shots sctting wherein we have access to s training images per class label in each of the dataset.
To account for the different scales accuracy across the datasets, we also tested to compute weighted av- erage, normalizing by the accuracy of reference model (ViT-B/32). This is reminiscent of the normalization carried out in Hendrycks & Dietterich (2019) according to the score of Alex We found the conclusions with the standard average and weighted average be similar.
A.5.1 Specific Considerations in the Ensemble Case
For an ensemble with M members, we have access to M representations {0(2;; 0, Mi for a given input a. We have explored two ways to use those representations:
Joint: We concatenate the M representations {ø(æ; M into single "joint" feature vector in
M-S. remembering that each 6(æ;; 0%,) E RS. We then train a single a linear classifier to predict the target labels from the "joint" feature vectors.
Disjoint: For each of the M representations {ø(œ; M=))= we separately train a linear classifier to predict the target labels from the feature vectors induced by o(x; 0y,). We then average the predictions of the M linear classifiers trained in this fashion.
In Table 9, we report comparison of those approaches. We aggregate the results over all ensemble models (namely, E² and upstream ViT/V-MoE; ensembles of size 2 and 4) and over 8 replications, for the VIT families S/32, B/32 and L/32.
The results indicate that "joint" and "disjoint" perform similarly. Throughout our experiments, use the "joint" approach because it eased some implementation considerations.
A.6 List of Datasets
For completeness, in addition to the few-shot datasets listed in Appendix A.5, we list the datasets used for downstream training and evaluation this work.
ImageNet (ISSVRC2012) (Deng et al., 2009) with 1000 classes and 1281167 training examples.
ImageNet-C (Hendrycks & Dietterich, 2019), an ImageNet test set constructed by applying 15 dif- ferent corruptions 5 levels of intensity to the original ImageNet test set. (We report the mean performance over the different corruptions and intensities.)
ImageNet-A (Hendrycks et al., 2019), an ImageNet test set constructed by collecting new data and keeping only those images which a ResNet-50 classified incorrectly.
Iea (Recht al., 2019), an ImageNet test set independently collected using the same methodology the original ImageNet dataset.
CIFAR10 (Krizhevsky, 2009) with 10 classes and 50000 training examples.
CIPARIO-C (Hendrycks & Dietterich, 2019), a CIFARIO test. set constructed by applying 15 different. corruptions at 5 levels of intensity to the original CIEAR10 test set. (We report the mean performance over the different corruptions and intensities.)
CIFAR100 (Krizhevsky, 2009) with 100 classes and training 50000 examples.
Oxford Flowers 102 (Nilsback & Zisserman, 2008) with 102 classes and 1020 training examples.
22

---

Published in Transactions Machine Learning Research (09/2022)
Oxford-IIIT pet (Parkhi al., 2012) with 37 classes and 3680 training examples.
SVHN (Netzer et al., 2011) with 10 classes.
Places363 (Zhou et al, 2017) with 365 classes.
Describable Textures Dataset (DTD) (Cimpoi al., 2014) with classes.
A.7 Sparse MoEs meet Ensembles Experimental Details
The setup for the experiments in Figures and 12 differs slightly the other experiments in this paper. Specifically, while for all other experiments used upstream V-MoE checkpoints with (K, /)) = (2, 32), for these experiments we matched the upstream and downstream checkpoints. We did this to avoid a checkpoint mismatch as a potential confounder in our results.
A.8 Multiple Predictions without Tiling or Partitioning Details
The naive multi-pred method presented in Section 4.2.4 was trained in almost the same manner as the vanilla V-MoE, the only difference being the handling of multiple predictions. This was accomplished by using the average ensemble member cross entropy as described for g² Appendix C. contrast, in order to compute the evaluation metrics presented in Table 4, first averaged predictions of the model and then used the average prediction when ca calculating each metric.
Table 9: Comparison of two approaches, "joint" and "disjoint", to compute the linear few-shot evaluation in the case of ensembles. For the ViT families S/32, B/32 and L/32, mean error across datasets is averaged over replications and over all the ensemble models of size and 4.
MODEL SIZE
METHOD
MEAN ERROR ACROSS DATASETS
1 SHOT
5 SHOTS
10 SHOTS
25 SHOTS
S/32
disjoint
51.01
+ 0.48
32.80
0.94
26.33 0.00
20.97 + 0.18
joint
51.12
+ 0.42
32.81
0.30
26.30
0.24
20.77
+: 0.17
B/32
disjoint
42.43
25.49
0.21
20.30
15.98
joint
42.59
25.74
20.54
16.06
L/32
disjoint
36.41
0.31
21.49
0.16
17.13
0.12
13.56 à 0.10
joint
36.48
0.30
21.66
0.18
17.34
0.10
13.56 + 0.08
B Compatibility and Adaptation of the Upstream Checkpoints
Throughout the paper, we make the assumption that we can start from existing checkpoints of ViT and V- MoE models (trained on JFT-300M; soe Appendix A.2). We next describe how we can use those chockpoints for the fine-tuning of the extensions of VIT and V-MoE that consider in this paper.
In all our experiments that involve V-MoEs, we consider checkpoints with and 32, which the canonical setting advocated by Riquelme et al. (2021).
B.1 Efficient Ensemble of Experts
In the case of E², the set of parameters is identical to that of V-MoE model. In particular, neither the tiled representation nor the partitioning of the experts transforms the set of parameters.
To deal with the fact that the single routing function gate (W-) of V-MoE becomes separate routing functions (gate (W.:)) ) one for expert subset En, we simply slice row-wise W E EDD into the M matrices Wm. m (E/M))

---

Published in Transactions Machine Learning Research (09/2022)
B.2 Batch Ensembles (BE)
We train BE starling from VIT checkpoints, which requires to introduce sewwnstream-speecili parameters. Following the design of V-MoEs, we place the batch-ensemble layers in the MLP layers of the Transformer.
Let consider dense layer in one of those MLPs, with parameters U RDx in absence of bias term. In BE, the parametrization of cach ensemble member has the following structure Un. = a (mms)) where (rm JM and {8m JM are respectively D- and I-dimensional vectors.
standard VIT checkpoint provides pre-trained parameters for U. We then introduce {rm Jo and [mm] m== at fine-tuning time, following the random initialization schemes proposed in Wen al. (2019); see details in the hyperparameter sweep for BE in Appendix G.I.
B.3 MIMO
We train MIMO models from V-MoE checkpoints. The only required modifications are the input and output parameters of the checkpoints. The linear input embedding must be modified to be compatible with input images containing M times as more channels, as required by the multiple-input structure MIMO. Similarly, the final dense layer in the classification head must modified have M times more output units, following the multiple-output structure in MIMO.
Concretely, the embedding weight Wn & R is replicated in the third (channel) dimension, resulting in dM R x3- where H and W are the height and width of the convolution and D is the Table 6). The output layer weight Wont E RDC is replicated the second (output) dimension, resulting OMI R.x.M where the number of classes. The oulput layer bias boun E g is replicated resulting in out RRCXM Finally, in order to preserve the magnitude of the activation for these layers, in and WMIMO,out are sealed by 1/M.
hidden dimension of the ViT family (specified
C Implementation Details of Efficient Ensemble of Experts
We provide details about the training loss and the regularizer used by F... We also discuss the memory requirements compared to V-MoE.
C.1 Training Loss
Since E outputs M prodictions 6m)) > for given input æ, need to adapt the choice of the training loss L. accordingly. Following the literature on efficient ensembles (Wen et al., 2019; Dusenberry al., 2020a; Wenzel et al., 2020), we choose the average ensemble-mentn cross entropy
Cly. a: 0)
Σ
cross-entropyyy, (ar; on))
M
m=l
instead of other alternatives such as the ensemble cross-entropy
1 M cross-entrapy (v.
Σ f(x; 0m)) M m=1
that was observed to generalize worse (Dusenberry et al., 2020a).
C.2 Auxiliary Losses
Inspired by previous applications of sparse MoEs in NLP (Shazeer al., 2017), Riquelme et al. (2021) employ regularizers, also referred to auriliary losses, to guarantce balanced usage of the E experts. Two auxiliary losscs-the importance and load losses, see Appendix A in Riquelme et al. (2021) for their formal definitions averaged herether to form the final regularization term that we denote by Ω.

---

Published in Transactions Machine Learning Research (09/2022)
As a reminder, let us recall the notation of the routing function
hE RD + gate (Wh) = topx (softmax( W + σs)) E IR
with W E ERED and AO(0 f). Consider a batch of B inpuls {h;),-1 that we represent by H BRXD
Finally, let us define
A HW DERXE R
where we emphasise that B is a matrix of Gaussian noise entries in R R The regularization term ? used by Riquelme et al. (2021) can be seen as a function that depends on and HW
In the context of efficient ensemble of experts, the set of E experts is partitioned into M subsets of E/M experts, denoted by U U see Section 4.1. With the introduction of the routing fune- lions (Wat) Iom with each W., R(E/M)XD, the matrix A becomes accordingly partitioned into [Am] bmm where each A. E BBX(E))M)
Since we want to enforce balanced usage of the E/M experts in each subset &m of the experts, we thus redefine the regularization as the average regularization separately applied to each part of the partition
Spartition (A, HW
Σ 9(Am. H. m=
M
We found this option to work better in practice. To guarantee fair comparison, also applied Spartition to the "Only partitioning" model in the ablation study of Section 4.2.1.
Following Riquelme al. (2021); the regularization parameter controlling the strength of 9 partition set. to 0.01 thronghout the experiments.
C.3 Memory Requirements versus V-MoE
Due to tiling, E² requires more memory than V-MoE. To be concrete, the memory complexity of V-MoE can be decomposed into two terms
mmemory params nemoryactivatione/)
for the forward and backward passes, respectively. For E², the complexity becomes
Libefore
Latter M
memory params
meraryparams + memory
Laotal
where M is the ensemble size, and, Luefores Latters and Leotal are the number of layers before tiling, after tiling,
and in total, respectively. Importantly, neither params nor memory activations depend on M. Thanks increase memory
to the "last-n" setting employed in the paper, we have latter & Libefores and thus due to tiling remains mild. More concretely, for ViT-L, we have Liotal 24, Lieftre = 21, and Latter 3 (with MoE layers placed at layers 22 and 24). Thus, for an ensemble of size M = 2, E would only increase memory, activatioos by 12.5%, while leaving memory params unchanged.
D Efficient Ensemble of Experts and V-MoE Relative Improvements per ViT Family
In Soction 5 we claim that g² performs best at the largest scalc. In this section we motivate that claim in more detail. Specifically, consider two metrics of improvement in performance. Firstly, we consider the percentage improvement in NLL for both g and V-MoE versus vanilla VIT. Secondly, we consider a normalised version of this improvement. We consider this second metric take into account the "difficulty" in further improving the NLL of larger ViT family models. Intuitively, the larger the ViT family, the better the corresponding NLL will be, and the more difficult will be to improve on that NLL.
The normalisation we apply is based on the gradient of the NLL with respect to FLOPs. Indeed, this gradient captures the typical variation NLL at a particular amount of FLOPs. The ratio of this gradient at the FI.OPs values (i.e., the instantaneous change in NLL at those FLOPs values) for two ViT families
25

---

Published in Transactions Machine Learning Research (09/2022)
mcasure of the relative difficulty in increasing the NLL. Thus, we can use this ratio to normalisc our results. To be more concrete, let us define the mapping
NLL = ((FLPP)) and its derivative & ((FOPP:) = dp(FLLOPS)
dFLOPs
We estimate @p and its gradient by fitting a linear model to the (NLL, FLOPs) pairs for each VIT family, using the data of the standard VIT models trained. We [1.0, log(FLOPss, log(FLOP)) 1ogg and solve for the parameters of the linear model via ordinary least squares. We determine the gradient of this function at cach FLOPs value using automatic diffcrentiation in JAX (Bradbury et al., 2018). See Figure 9 for the resulting fit and indication of the gradients.
feature expansion
1.0
Tangent
09
VI
08
NLL
0.6
0.5
04
1000
2000
3000
(downstream) GFLOPs
Figure Estimated o compared to the ImageNet NLL values for our ViT models. We also include the Langent at the points corresponding lo each VIT model to indicate the gradients al those points.
The normalised values are calculated as:
(FLOPss 1/14)
Normalised improvement(()) = improvement(v) x
(4)
& (FLOPs.)
where u one of the VIT families, i.e., S/32, B/32, L/32,, L/16, or H/14. Note that this normalisation leaves the improvement for H/14 the same. We tried to normalize with respect to other choices of VIT family, different from H/14. Our conclusions are robust, the sense that both the ordering and the monotonic behavior with respect to scale are preserved. Using the ratio for normalisation also has the advantage that the normalisation is less sensitive to the particular parameterisaatin of sp
Table 10: Percentage improvements in NLL for E² with (K, M) = (1,2) and V-MoE with K= vs. ViT for familics of increasing size. The top two rows show normalised improvements, see (4), which take into con- sideration the increased difficulty of improving NLL for larger ViT families whose performance is beginning to saturate. The boltom rows are the original percentage improvements without normalisation.
S/32
B/32
L/32
L/16
H/14
E vs. VIT
0.02%
0.09%
0.24%
2.35%
4.27%
Normalised
V-MoE vs. VIT
0.01%
0.06%
-0.04%
0.89%
0.02%
B vs. VIT 9.82%
9.53%
3.76%
5.38%
4.27%
Not normalised
V-MoF vs. VIT
7.98%
6.62%
-0.60%
2.05%
0.02%
Table 10 shows both the nirfmault--orrmalieed and original improvements (without normalisation). Looking first at the original improvements, we can see that while both E² and V-MoE have smaller improvements over VIT for larger familics, gs's s improvements decrease more slowly. Furthermore, by comparing the normalised improvements we see that gs improvements actually grow monotonically when taking difficulty into account. This is not the case for V-MoE.
26

---

Published in Transactions Machine Learning Research (09/2022)
E From Batch Ensembles to Sparse MoEs
Wen et al. (2019) have shown that, given a batch of B inputs X E RBx,, single forward pass can efficiently compute the predictions all the ensemble members (m) m- By appropriately liling the inpuls the network Xtteed (BMM by factor M, each internal operation per ensemble member can then be vectorized.
We take the previous example of dense layer with parameters U RDxL and assume the layer reccives the tiled inputs (Hm) MM where Hm E RBxD, We need to compute for each ensemble member HmUm Hm(U o (Fm Denoting by him IR the i-th input in Hm: we have
E
ge (h = iffe=== go(hs,m = otherwise
h Un E (he.m) (exphb.)) (ham) with =
and expert, (z)= = (r.s.)), Although (5) may appear convoluted way of writing the operations
in batch ensembles, it unveils connection with (1). Indeed, operations in batch ensembles can be seen as a specific sparse MoE, e.g., with binary routing weights depending only on the position in tiled inputs.
While Wen et al. (2019) primarily tiled the imputs for the sake of efficiency, also induces some form of conditional computation, an insight that we exploit in E².
Batch Ensembles versus Efficient Ensemble of Experts
Table 11: Summary the differences between BE and E².
SHARED PARAMETERS
UNIQUE PARAMETERS
TILNGG
TrAINInG EPOCHS
BE
Keruels in each linear layer
"Fast- weights" and biases Experts in subset Em
Start of the model Before 1st MoE layer
Typically =500% more Unchanged
All parameters outside MoE layer
While g is inspired in several ways by BE, it has also been specialised to sparse MoEs. This leads number of significant differences between the two algorithms. These differences are summarised in Table 11, and we elaborate upon them here:
Shared parameters: In BE, the shared parameters are the kernels each linear (e.g. dense convolution) layer. This makes up the majority of the parameters in BE. In E², the shared parameters are all of those not found in the expert layers.
Ensemble member specific parameters: BE uses pairs of "fast-weights" {rmm 8m) and bias terms for each linear layer. ne uses the parameters of all the E/M experts in the m-th subset & and the router for those expert, each MoE layer. This results much larger set ensemble member specific parameters, and thus higher level of predictive diversity (see Section 4.2.3).
Tiling: BE, tiling performed at the very beginning of the model, regardless of the model's 's internal structure. Tiling in D² occurs before the first MoE layer, thus saving redundant computation for "Last-n" V-MoE.
Number of training epochs: BE usually requires 50% more training epochs than vanilla model. Ed requires the same number of training epochs as V-MoE.
G Efficient Ensemble Comparisons
In this section, we compare efficient ensomble of experts (E²) to several popular efficient ensemble approaches, namely MIMO (Havasi et al., 2020), batch ensemble (BE) (Wen et al., 2019), and MC Dropout (Gal Ghahramani, 2016).

---

Published in Transactions Machine Learning Research (09/2022)
Table 12: ImageNet performance of different efficient ensemble approaches. The table reports the means = standard errors over 8 replications. models have ViT-B/32 architecture. K stands for the sparsity V-MoEs, M denotes the ensemble size while "BR" corresponds to the batch repelition in MIMO (Havasi et al., 2020).
M NLL +
ERROR +
ECE
KL 1
GFLOPs 1
VIT
0.688
0.003
18.65
0.08
0.022
0.000
78.0
BE ViT
2
0.682
0.003
18.47
0.05
0.021
0.000
0.040
0.001
97.1
0.675
0,003
18.40
0.09
0.017
0.000
0.035
0.001
135.4
0.642
A: 0. 0.002
16.90
0.05
0.029
A: 0.001
82.4
V-MoE
0,638
16.76
0.033
0.001
94.9
0.636
0.001
16.70
0.04
0.034
1: 0.001
120.1
0.635
0,002
16.72
0.06
0.028
4: 0.001
170.4
0.648
3: 0.002
17.10
0.05
0.019
1: 0.001
0.046
0.000
97.2
4
0.641
16.96
0.017
0.046
135.6
MC Dropout V-MoE
2
2
0.642
002
16.94
0.021
0.001
0.046
0.001
113.7
0.634
0,001
16.80
0.03
0.020
0.000
0.046
0.001
168.6
4
0.639 2
16.91
0.022
0.045
146.7
MIMO V-MoE (BR-
2
0.636
16.97
0.028
0.000
0.000
96.3
4
0.672
0.000 0.0001
17.72
0.037
0.000
0.001
0.000
99.0
MIMO V-MoE (BIR-2
0.638
0,001
17.14
0.08
0.031
+ 0.000
0.001
0.000
192.6
2
0.665 4
17.38
0.038
0.000
198.1
2
0,622
16.70
0,018
0.000
0,217
0.003
105.9
4
0.624
0.001
16.99
0.00
0.013
1: 0.000
0.164
0.001
153.0
E²
0.612
0.001
16.49
0.02
0.013
#: 0.000
0.198
0.063
131.1
2
4
0.620
16.86
0.015
0.170
203.3
0.611 2
0.001
16.45
0.03
0.014
0.000
0.193 0.003
181.4
Table 12 reports the ImageNet performance of those different techniques, when all models are based ViT-B/32 architecture. We start by highlighting the most salient conclusions of the experiment and defer to the next subsoctions the descriptions of the different competing techniques.
We make the following observations:
BE built upon VIT improves the performance of VIT in terms of NLL, classification error and ECE. However, the resulting increase in FLOPs makes BE less viable option compared to g
MC Dropout V-MoE is par with standard V-MoE in terms of NLL and classification error, while it improves the ECE. For all values of K, we observe that the performance tends to improve as the {2, 4). the resulting increase
number of samples, i.c., M, increases. However, already for FLOPs makes MC Dropout V-MoE less favorable option compared to E.
Perhaps surprisingly (see detailed investigations in Appendix G.3), MIMO V-MoF does not lead improvements compared with V-MoE. In fact, for higher ensembles sizes, MIMO V-MoE results in worse performance than standard V-MoE. Moreover, increasing the batch repetition parameter MIMO ("BR" in Table 12) further worsens the results. Interestingly, can see that MIMO does not manage to produce diverse predictions, as illustrated by the small values of KL.
Ed offers the best performance vs. FLOPs trade-offs, e.g., when looking at (K, = (1 and (K,M) = (2, 2). We notably obscrve that the diversity of the predictions in is orders of magnitude larger than that of the other ensemble approaches.
28

---

responsible for MIMO's poor performance when applied to Published in Transactions Machine Learning Research (09/2022)
We briefly recall the optimization explainod in Section 4.1 save redundant computations: In the "last- n" setting of Riquelme et al. (2021), it is sufficient to tile the representations only when entering the first MoE /rrooot layer/batehe batch-ensemble layer for respectively E² #*/MC Dropout V-MoE/BE. We apply this optimization all the efficient ensemble methods.
G.1 Batch Ensembles
Following the design of V-MoEs,, we place the batch-ensemble layers in the MLP layers of the Transformerr following the "last-n" setting of Riquelme et al. (2021); see Section 2.1.
The vectors of the rank-1 parametrization introduced at fine-tuning time (see Appendix B) need to be initialized and optimized. Following the recommendation from Wen al. (2019), we consider the following hyperparameters in addition to the common sweep described in Table 7:
Initialization: Either random sign vector with entrics in {-1,1) independently drawn with probability 1 or random Gaussian vector with entries independently drawn from (11, 0.5).
Learning-rate scale factor: The vectors of the rank-1 parametrization are updated with learning rate scaled by factor in [0.5, 1,
G.2 MC Dropout V-MoEs
For MC Dropout V-MoE, we take the available fine-tuned V-MoEs and emable dropout at prediction time. Indeed, described Table 7, all V-MoE models already have a dropout rate the experts.
G.3
MIMO V- V-MoEs
Following Havasi et al. (2020) we consider two MpMO-specific hyperparameters, in addition to the hyperpa- rameters listed in Table 7:
Input replication probability: (0.5,0.625, 0.6,5, 0.75)
Batch repetitions: {
Our preliminary investigations also considered lower input repetition probabilities and higher batch repeti- tions. However, lower imput repetition probabilities tended to result in poorer performance. While higher batch repetitions did help to some extent, the additional computational cost made it impractical.
Given the surprising result that ensemble size of M provides no performance improvement over the standard V-Molf and that increasing M further provides worse performance, there seems to be some incompatibility between MIMO and V-MoE. In fact, our investigations revealed that ViT is the source the problems since applying MIMO to vanilla VIT without experts resulted in the same trends as for V-MoE. Thus we hypothesise that the differences between ViT and ResNet the architecture to which MIMO was originally applied by Havasi al. (2020) VIT.
Difference 1: Class token. One of the differences between VIT and ResNet is that VIT makes of special learnable class token to classify an image (see Dosovitskiy et al. (2021) for details). ResNet on the other hand makes use of the representation from an entire image for classification. We tried two strategics to mitigate this difference:
1. We applied the global average pooling (GAP) and multi-head attention pooling (MAP) classification strategies introduced in Dosovitskiy et al. (2021) and Zhai al. (2021), respectively. In short, both of these methods make use of all the tokens from image for classification. However, neither of these strategics made a significant difference to the relative performance MIMO and VIT. fact, the choice of classification methad the least impactful hyperparameter in our sweep.

---

Published in Transactions Machine Learning Research (09/2022)
Rather than learning single class token, we learnt M class tokens. This stratogy resulted MIMO with M 2 outperforming VIT. However, for M >> 2 the improvement was small enough that VIT still outperformed MIMO.
The other major difference between VIT and ResNet is the building block for each model. While ResNets primarily composed of convolution operations, VIT makes heavy used of attention. We hypothesised that attention is less suited ta separating the information for M input images stored in the channel dimension of single image. We tried two strategies to mitigate this potential issue:
Difference Attention.
1. We applied the hybrid architecture, described in Dosovitskiy et al. (2021), in which the imput quence to VTT is formed by CNN feature mape. We used ResNet14 and ResNet50. In both cases, we found that strategy boosted the performance of ViT and MIMO equally.
Rather than concatenating images in the channel dimension, we concatenated them in the width dimension, resulting in 3 limes many patches for VIT to process. This stralegy was successful in the sense that the MIMO performance for > improved significantly. However, the significant additional computational cost made it infeasible solution.
Our findings suggest that MIMO and ViT are indeed somewhat incompatible. Unfortunately, none of our proposed solutions to this problem provided high enough predictive performance increases (or indeed low enough computational cost increases in cases) to warrant immediate further investigation.
H Sensitivity Analyses
H.1 Load Balancing
Figure 10 explores the effect of the load balancing loss on diversity and predictive performance. We see that for both V-MoE and g the predictive performance, as measured by NLL, Error, and ECE are largely insensitive to the strength of the load balancing loss. Similarly, the diversity of the predictions of g*-as measured by the KI, Kl.-millly depends on that regularization. Only when the load balancing loss strength becomes excessively large (4 orders of magnitude larger than standard values) do all the performance metrics plummet. In other words, this hyperparameter does not allow us to increase the diversity and thereby the predictive performance of our models.
V-MoF (K-2)
NT. (douerss batee)
Y-MoF (K+4)
Etror (lower letee)))
BCR (loncrisheite)) (eltie))
0.05
0.22
Fa (K-1, M-d)
1a (K=2, M-
0.04
0.61
0.19
6.7
0.6
0.01
0.15
(A)
Ingse
Ingze(A)
Figure 10: The effect of 11 the strength of the load balancing, see appendix A of Riquelme et al. (2021) on NLL, Error, ECE and diversity, for E and V-MoE. Results averaged over three random seeds. All models have ViT-B/32 architecture.
H.2 Expert Initialization
Figure 11 explores the influence of the initialization of the experts. In particular, we add Gaussian noise with varying standard deviations the initial weights of the expert MLPs. We notably show that more diverse initializations (larger standard deviations) do not translate to any clear performance gain. Note that this experiment takes place upstream, since downstream, the experts are already initialized (we finc-tune them from the upstream checkpoint).
30

---

Published in Transactions Machine Learning Research (09/2022)
Iasgoles (Slee Freor (bsectabbite)) (elttet
Imagoher 5-Stee Error (Breer (ererrichhhii bchitee))
Iraggein (0-Shen Timer (lreer horken))
ImapeNe 25-50 Stlli Eour (uwes bollle)
0.00
0:55
0.45
V-MoF-8-12
Initad mnrio standard deviation
Inmaliaatnnn staddrd deviation
Inmaliztioon santard deviation
Inmtttottion standand deviation
The influence of noise in the expert MLPs initial random weights. The models are trained on JFT-300M and measure the ImageNet few-shot Error as in Riquelme al. (2021). Results are for a single random seed.
Figure 11:
H.3 Expert Group Permutation
Table investigates the performance of E² where the ordering of the upstream V-MoE experts has been randomly permuted before fine-tuning' We see that the permutation of the experts has little-lo-no significant. impact on performance (as measured by NLL and error). The small discrepancies between error for = and NLL for K = 2 can be explained by noting that g aggregates results with different upstream chockpoints while E. (permuted) only takes one of these checkpoints and varies the permutations; this was done to isolate the effect of the permutations only. Note also that the standard errors for Ee and r:3 (permuted) are very close to each other, which indicates that permutation of experts has similar impact to random initialization.
Table 13: Comparison of B--B/32 with E 32 total experts and M = partitions, and E where the order of the experts has been permuted. For E²3 results are aggregated over random seeds whereas for E (permuted) results are aggregated over 5 permutations.
K
NLL
ERROR J
R
0.622
16.70
(peruuted)
0.620
0.001
16.80
0.04
2
0.612
0.001
16.49
0.02
(permuted)
0.608
0.000
16.47
0.03
The Roles of Ensemble Diversity and Individual Model Performance
Table 14 shows the individual member performance for each of our efficient ensemble variants well as upstream and downstream deep ensembles, for sizes M = 2 and M = 4. For each method and ensemble size, the first row shows the combined ensemble performance, and the following rows show the performance of the individual ensemble members.
For E²3 the gap between single members and their ensemble considerable. This is reminiscent of deep ensembles (both variants) and in stark contrast with what observe for the other efficient ensembles: BE ViT and MIMO V-MoE. The diversity of E. is comparable to that of upstream deep ensembles and much larger than downstream deep ensembles. For example, if we compare MIMO V-MoE (M=2) and g (M=2), the single members of E² are all worse in NLL, ACC, and ECE than the single members of MTMO, However, the performance gap between the individual members and the ensemble much larger for E than MIMO (where there is almost no difference). The diversity of the individual models in E² key to its strong performance. Thus, g outperforms MIMO.
We also permate columns the routing W., acrordingly keep the routing consistent.

---

Published in Transactions Machine Learning Research (09/2022)
In short, this suggests that E approximates doep ensemble of smaller V-MoE models. That is, with Es, are able to take advantage of the overparametertaation of the experts to create a rich set of ensemble members within a single model. Each ensemble member has large mumber of non-shared parameters, thus high induced diversity. In comparison, BE only has few vectors specific to each member.

---

Published in Transactions Machine Learning Research (09/2022)
Table 14: Comparison of the individual ensemble member performance and combined ensemble performance for BE VIT, MIMO V-MoE (K' BR=1), E² (K 1), as well upstream and downstream V-MoE (K 1) ensembles.
NI.I.
FRBOR 4
FCE +
KL 1
ensemble
0.682
0.003
18.47
0.05
0.021
0.000
0.040
0.00
M-2
member
0.693
0.008
18. 18.68
0.08
0.025
0.000
member I
0.693
18.68
0.025
BE VIT
ensemble
0.675
18.40
0.017
0.035
member 0
0.690
0.003
18.70
0.00
0.022
0.000
M=4
member 1
0.690
0.008
18.70
0.08
0.023
0.000
member 2
0.690
18.70
0.023
member 3
0.691
0.008
18.70
0.07
0.023
0.000
ensemble
0.636
0.002
16.97
0.00
0.028
0.000
0.001 4 0.000
M-2
member 0
0.636
0.002
16.97
0.04
0.028
0.001
member |
0.636
16.97
0.028
MIMO V-MoE
ensemble
0.672
17.72
0.037
0.000
0.001
member 0
0.672
0.001
17.74
0.00
0.037
0.0000
M-4
member
0.672
0.000
17.71
+ 0.08
0.037
0.000
member 2
0.672
17.72
0.037
member 3
0.673
0.00L
17.73
0.06
0.037
0.000
ensemble
0.622
0.001
16.70
0.018
2.0.000
0.217
0.000
M=2
member 0
0.67
0.003
17.74
de 0.08
0.038
000001
member 1
0.683
17.94
0.038
0.000
B
ensemble
0.624
16.99
0.013
0.000
0.164
member 0
0.677
0.002
18.04
0.05
0.034
0.003
M-4
member
0.685
0.001
18.23
0.05
0.034
0.001
member 2
0.691
18.35
0.035
0.001
member 3
0.697
0.002
18.47
0.08
0.035
00000
ensemble
0.588
0.001
15.74
0.017
1.0001
0.214
0.001
M=2
member 0
0.640
0.000
16.82
+ 0.04
0.030
0.000
member 1
0.645
0.002
16.97
0.029
0.002
Up-DE
ensemble
0.561
0.000
15.10
0.020
0.000
0.214
member 0
0.639
0.001
16.80
0.030
0.0000
M
member
0.641
0.00s
16.84
J: 0.05
0.030
0.000
member 2
0.642
16.90
0.031
0.000
member 3
0.647
0.002
17.05
0.08
0.028
0.0002
ensemble
0.620
0.001
16.44
0.023
0.000
0.073
0.001
M=2
member 0
0.642
0.00L
16.83
de: 0.03
0.030
0.000
member 1
0.643
16.84
0.030
0.000
Down-DE
ensemble
0.607
0.000
16.17
0.021
0.003
0.073
4: 0.000
member 0
0.641
0.001
16.82
0.030
0.0000
M=4
member
0.642
0.001
16.85
à: a...o.0. 00
0.030
0.000
member 2
0.643
0.000
16.89
0.031
0.000
member 3
0.643
0.001
16.93
0.07
0.031
0.000

---

J Upstream & Downstream versus Downstream-only Ensembles
Published Transactions on Machine Learning Research (09 22022)
In Section 5, and Appendix L include downstream deep ensembles (down-DE) of V-MoE, and in some cases ViT, as a baseline. This choice was motivated by the fact that like VIT, V-MoE, and E², down-DE requires only a single upstream chockpoint, which all of the methods more comparable. However, it is clear that using different upstream checkpoints and then further fine-tuning each of these with different random seeds to construct an upstream deep ensemble (up-DE) would result in more varied ensemble members and as a result, a better performing ensemble. This idea has recently been explored by Mustafa et al. (2020).
Thus, for completeness, we also investigate the effects of upstream ensembling on V-MoE. Table 15 compares the performance of upstream and downstream V-MoE (K = 1) ensembles of sizes M and M = 4. Across the range of metrics, for both ImageNet and ImageNet-C., for all VIT families, and for both values of M, see that up-DE outperforms down-DE. In fact, up-DE with M 2 is very often better than or equal to down-DE with M = 4. This is especially true for the diversity metrics, which indicates that diversity is indeed the driver for improved performance in up-DE. Not shown in the table is the very large computational cost associated with training upstream ensembles.
Table 15: Comparison of upstream and downstream ensembles of V-MoE with (K = 1).
IMAGEE
TNtACINNer--C
M
NLL J
ERROR
ECE $
KL. 1
Cos, Stzt.
NORM.
Dis. 1
NLL J
ERROR
ECE J
KL.
Cos. SIM.
NORM,
DrS. f
down- DE
11.35
0.018 u.s
0.079 -.00
0,974
wasss
0.458
cua
0.871 -
21.37
0.021
0.218 4.
0.925
caa
0.625 -
H/14
up-DE
2
0.391
11.12
0.016
0.126
0.963
0.625
0.839
20.66
0.022
0.355
0.892
0.809
down-DE
0.352
1.00
11:20
6.90
0.014 2000
0.083 1.000
0.973
2 0.00
0.500
Co00
0.851 2.00
20.97
1.00
0.021
0.221
2000
0.823
1.00
0.650 2.0
up-DE
0.375
10.66
0.013
0.129
0.963
0.652
0.792
19.01
0,032
0.361
0.892
0,850
down. DE 2
0.450
12.02
0.018
0.061
2000
0.970
0.419
1.010
24.43
0.021
-.
0.168
0,036
cian
0.530 0.00
T/116
up-DE
0.434
12:20
0.014
0.118
0.964
0.584
0:961
21.46
0.023
0,342
0,890
0.766
down-DE
4
0.140
1000
12.39
2.00
0.015
10000
0.061
10000
0.979
0,00
0.125
1.00
0.983
23.95 1.0.2
0.020
1.0.0c
0.266
1.00
0.9377
1,00
0.541 1.000
up-DE
0.418
0.118
1,000
0.964
5.000
0.603
1.0000
22.45
0.311
1.0000
0.800
down. DE
2
0.533
14.55
0.025 m.
0.092
0.968
0.479
1.184
27.98
0.029
0.190 a
0.025
0.556 =:0.00
1.332
up-DE
0.511
14.07
0.019
0.191
0.945
5.000
0.094
1.133
0.000
20.97 2.0.
0.022
0.449
0.861
0.820 a.0
down-DE
0,518
11:29
0.022
0.092
0,969
0,187
1.151
27.47
0.023
0.399
0.825
0.567
up-DE
0.188
1.00
13.52
1 6.92
0.016
1,00
0.190
0.916
1,000
0.722
1.0000
1.073
25.74
9.00
0.030
1,00
0.116
1000
0.862
0.857
6,000
down-DE
2
0.519
14.09
0.021
0.048
0.982
was
0.351
13316
30.02
0.030
0,132
0.943
0.448
B/16
up-DE
2
0.460
13.40
9.000
0.015
0.169
0.951
0.608
1.231
2:0004
28.41 m..
0.023
0.481
0.845 =.0:e:
0.808 -.0000
down-DE
0.511
13:95
2:000
0.019
0.048
0.982
0.354
1,208
29.67
0.026
0,132
0.943
0.453
up-DE
0.168 4
12.89
0.016
12000
0.168
0.951
0,690
1.000
1.166
27.08
0.037
0.179
0.8:6
0.879
dowa-DE
1.00
0.023
0:073
9.00
0,973
0.414
1,510
33.77 1..0
0.032
1,000
0.175 w.ss
0.925
1.0000
0.498
B/32
do
DE
2
0.607
0.588
2.000
16.17 2.00
15.74
0.017
0.021
8:00
0.214
0.073 2:0000
0.937
0.973
0.700
0.418
1.000
1.430
1.483
32.37
33.36
2.005
0.022
0.027
1/000
0.537
0.174 o:00
0.824
0.926
2:000
0.844
0.504 m.0
up-DR
0.561
15.10
0.020
0.214 1.00
0.937
0.739
1.357
30.92
0.036
0.637
0.824
0.884
down-Di
2
0.807
20.90
0..0
0.018
0.102
0.962
0.458
2.106 uasas
44.32
0.038
0.223 6.0.0
0.900
0.521
8/92
up-DE
0.763
19.85
0.016
0.305
0.911
0.773
2.004
42.92
0.025
0.683
0.767
0.856
down-DR
0.796
100
20.66
0.015
000
0.102
2.00
0.962
2 2.
0.462
2.076 o.0
44.16
0.031
1 1000
0.222
0:00
0.900
1:00
0.526
2.0000
up-DE
0.728
19.06
0.025
0.304
0.911
1.914
41.38
0.682
0.767

---

Published in Transactions Machine Learning Research (09/2022)
Table 16: ImageNet performance (means + standard errors over 3 seeds) of V-MoE and E for S/32 backbone, without pre-training.
K
NLL
ERROR
4
E (M = 2)
1.420
0.22 0.007
23.78
V-MoE
2
1.478
24.45 0.08
K Preliminary ImageNet Results without Pre-training
Training large-scale sparse MoE' on datasets such as ImageNet and in absence of any pre-training is difficult task. Indeed, the massive number of parameters causes models to severely overfit in that regime, In practice, we need combine various regularization techniques to control this overfitting.
To obtain preliminary results of g trained from scratch ImageNet, adapl the recipe proposed in the code released by Riquelme et al. (2021). The recipe itself based on the regularisation protocol of Steiner al. (2021) ("AugReg"), which trained performant dense ViT models from scratch on ImageNet, without pre-training Overall, compared to the default training schoma used for this paper, we add:
Mixup (Zhang et al., 2018),
Weight decay,
Dropout (Srivastava et al., 2014) on expert MLPs.
The baseline configuration V-MoE was tuned according the hyperparameter search space defined by Steiner et al. (2021), with extra sweep over expert MLP dropout in (..1. Selecting according to validation accuracy, the optimal V-MoE setting was medium2, i.c., mixup ratio 0.5 and RandAugment (Cubuk al., 2020) parameters and 15 augmentations applied of magnitude 15), alongside expert dropout 0.2. For ES, these regularization-reaaee precisely, we keep medium2 and tune only the learning rate (0.001, 0.003]; we thus suspect we could improve our current results for E by considering broader sweep of hyperparameters (e.g., re-tuning all the eggularization-rlattes hyperparameters).
hyperparameters were kept fixed and were not further tuned. More
In Table 16, we report the performance for both V-MoE and E² with a S/32 backbone. In terms of both NLL and classification crror, E² outperforms V-MoE. The model chockpoint, along with our code, can be found at https:://// http:/gittttt h/gggtttubb.../ hwtlb...ooooogoooo wwt.///oogl---ererrcccccccmom
Code can be found at https: //gtth////tttbt//g com httpsttt...oo//ooo/eeeeeeerr//a
Additional Experimental Results
In this section expand on various experiments presented in Sections 3 to 5. In experiments considering multiple ViT families we also include B/16 which was excluded from the main text for clarity.
L.1 Static versus Adaptive Combination
Here we continue the investigation into static versus adaptive combination from Scction 3.
Individual gains with respect to E, K and M. Figure 12 shows the effect of increasing the the various 'ensemble size' parameters for deep ensemble of V-MoEs. In particular, we investigate the static combina- tion axis M (the number of ensemble members), as well as the two adaptive axes (the number of experts chosen per patch) and E (the total number of experts).
When investigating the effect of K, we fix E = 322 and average over M E (1,.,88. Similarly, when investi- gating M, fix E 32 and average K E f1, When investigating the effect of E we fix K

---

Published in Transactions Machine Learning Research (09/2022)
and average over M E [I,.., 8). As result of this procedure the exact values of the curves are not directly comparable. However, can still examine the relative trends.
Specifically, we note that while the variation in K and M curves is roughly of the same size, the variation in the E curve is smaller, We also note that there is very little variation beyond E 8 (note the difference in the scales of the axes for the curves). These observations motivate the design of E², where we split. the sub- models along the E axis, in order better take advantage of the experts.
NLL vs. 'ensemble size'
E
4
8
16
32
0.84
K
M
0.82
E
0.80
0.78
K/M
Figure 12: Comparison for the impact on ImageNet NLL of variations in K, E and M. The underlying model VIT-S/32.
Extended Results for the Cumulative Effects of Static and Adaptive Combination. In Figure 13 we exlend the ImageNet NLL results, presented in Figure 3, lo a range of other datasets and performance metrics. We see that in most cases, the trends the same as before. That is, (static) ensembling tends to improve the performance of ViT and V-MoE equally. The two exceptions are ECE and OOD detection for VIT-S/32 where we see that larger ensemble sizes can result in decreased performance. These results indicate that the for small ViT models, larger ensembles can have slightly lower quality uncertainty estimates. The trend for ImageNet-CC performance is also not as consistent with ensembling sometimes helping ViT or V-MoE less (as indicated by the changes in ordering on the y-axis).
L.2 An Additional Motivating Experiment Deep Ensembles of V-MoEs with Fewer Experts
As an additional motivation for combining sparse MoEs and ensembles, Table 17 compares the performance
of a V-MoE with E 32 total experts and ensembles of V-MoEs with (M 2, E 16) and (M 4, E 8). for both K = I and K = 2. We see that, in terms of NLI, (M = 4, E 8) is better than (M 2,
E = 16) which is in turn better than (M = E = 32). We see similar results for Error and ECE.
We note that this result is especially remarkable since the individual upstream (and later, down- stream) models are such that E is worse than = 16 which in turn performs worse than E = 32; ensembling thus manages to counterbalance the poorer individual model performance. This suggests that the efficient ensembling i.e., the combination of multiple models within a single model of sparse MoEs could lead to strong performance while reducing computational costs.
L.3 Extended Results for the Tiling with Increasing Parameter Sharing Ablation
In Table 18, we extond the results for our parameter sharing ablation in Soction 4.2.3, from K 2 to K 1. We that the results remain the same in this case.
36

---

Published in Transactions Machine Learning Research (09/2022)
Table 17: Comparison of upstream deep ensembles of V-MoE-B/32 models with fewer experts.
K
M
NLL 4
ERROR 4
ECE +
KL t
1
32
0.642
0.009
16.90
0.029
0.001
2
0.588
15.97
0.015
0.211
0.577 4
8
15.82
0.017
0.228
V-MoE
1
32
0.638
0.001
16.76
0.033
2
2
16
0.583
15.75
0.016
0.211
4
0.580
15.94
0.015
0.184
Table 18: Extension of Table 3, showing the impact of parameter sharing in F², for K
OVERLAP
NLL +
ERROR +
ECE +
KL
E=E*)
0.622
0.001
16.70
0.03
0.018
+ 0.000
0.217
0.003
2
0.627
# 0.008
16.83
= 0.07
0.022
0.001
0.194
0.005
4
0,634
0.002
16.92
0.024
0.178
8
0.642
0.001
17.04
0.10
0.028
0.001
0.151
0.009
16
0.659
0.008
17.28
0.12
0.036
001
0.103
0.009
L.4 Extended Results for Few-shot Learning
In Figure 14, extend the few-shot learning results of Figure 4 to also include and 25-shot. Additionally, we show results for the weighted aggregation strategy mentioned in Appendix A.5.
We confirm the result that few-shot performance for E² gets better, relative the other baselines, with larger VIT families. Additionally, we see that E performance seems to get better, again relative to the other baselines, with more shots. This phenomenon can most easily be noticed by comparing the results for S/32 across the different numbers of shots. Finally, see that the trends with and without the weighted mean are the same.
L.5 Extended Results for OOD Detection
Here we extended the OOD results of Figure 6. Specifically, we add CIFAR100 as an in-distribution dataset and Describable Textures Dataset (DTD) (Cimpoi et al., 2014) OOD dataset. We also add area under the receiver operating characteristic (AUC (ROC)) and area under the precision-recall curve (AUC (PR)) as metrics. Figures 15 and 16 contain the results with CIFAR10 and CIFAR100 as the in-distribution datascts, respectively.
As Figure 6, we see that E performs better (relative the other baselines) for larger VIT families. Furthermore, E² seems to perform better in near OOD detection (i.e. CIFAR10 versus CIFAR100, and vice versa) than far OOD detection. Finally, we see that these trends are consistent across OOD metrics.
L.6 Extended Results for ImageNet
In this section we extend the results for ImageNett and the corrupted variants presented in Figures 4, 5 and In addition to NLL, classification error, ECE (for standard ImageNet), and Brier score, Figure 17 provides classification error and ECE for all ImageNet variants.
Most of the trends observed in Section remain true:
3 Lends to be Pareto elficient the presence distribution shift.
For smaller VIT families, V-MoB outperforms VIT in the presence of distribution shift.
37

---

Published in Transactions Machine Learning Research (09/2022)
improves ECE over VIT and V-MoE.
E improves classification performance.
ViT consistently provides better ECE than V-MoE.
However, there are exceptions:
ImageNet-A classification error. All models (including v²) under-perform relative to ViT-S/32 and V/T-H/14.
ECE for ImageNet-C,, ImageNet-A, and ImagoNet-V2. V2. Interestingly for the non-standard ImageNet variants, and in particular for ImageNet-A, there is a strong correlation between lower ECE and larger VIT families.
We also find that the results for classification error and Brier score follow those for NLL closely.
L.7 Additional CIFAR10, CIFAR100, Flowers, and Pets Results
Here we extend the results for ImageNet and the corrupted variants presented in Figures 4, and 7 to four additional datasets. Figures 18, 19, 20 and 21 provide results for CIFAR1O, CIFAR100, Oxford Flowers 102, and Oxford IITT Pet, respectively. As in Appendix L.6, we find that the results are similar to those in Section 5.
Compared to ImageNet, for CIFARIO, CIFARIO-C, and CHFARI00, F: seems to perform even better relative to the other baselines. Note, for example, that g² is Pareto efficient (even for S/32) in the cases of CIFAR10- C and CIFARIO0 NLL. As in Appendix L.6, we that the ECE has stronger downward trend with respect tO increased VIT lamily size for shifted test data.
For Flowers and Pets, where we only have results for smaller VIT families, E² seems under perform. However, the performance for 1./32 is better than for S/32 and B/32 which suggests that the results for these datasets are consistent with the other datasets presented this work and, therefore, that we should expect re's predictive performance to keep improving with larger models.
L.8 Efficient ensemble of experts and V-MoE with larger values of K and M
Figure and Figure 23 show the effect of varying on g and V-MoE, and the effect varying M on E respectively. We make the following obscrvations:
In almost all cases, increasing K or M does not result in Pareto efficient models.
For V-MoE, increasing K seems to help in most cases, except for ECE performance where it usually hurts.
For n², going from to = seems to help most cases but going from
usually hurts. Going from K to K still helps but lesser extent than from K 1 to K 2.
For E², increasing M either doesn't make a consistent and significant difference or hurts (e.g. in OOD detection).
These conclusions should, however, be considered with cantion. Recall that the upstream checkpoints used for fine-tuning all V-MoE and g models in this work are V-MoE models with K 2. Thus, the results in this experiment are confounded by upstream and downstream checkpoint mismatch for all E² models and all V-MoE models with K * 2. This phenomenon was observed in Riquelme et al. (2021) for V-MoE models. Lc., performance downstream V-MoE models with mismatched values of K were relatively worse than those with matched values of K; their Appendix E.4 (Figures 33-35). We also hypothesise that it is

---

Published in Transactions Machine Learning Research (09/2022)
difficult to train downstream g models with larger values M from upstream V-MoE models because each subset of the experts some common expert specialisations will need to be duplicated. Our ensemble members are fine-tuned from an upstream V-MoE with predefined total number of experts (E = 32) meaning that increasing M decreases the number of experts available each ensemble member (with R/M experts per member). This could also impact the performance of g with larger sizes of M.
L.8.1 Upstream vs. Downstream Mismatch
Table 19: ImageNet performance of E. models fine-tuned from V-MoE-B//2 checkpoints with K = 2 or K 4, and 32. Kupstream results are averaged over 8 random seeds, while Kupatream results are averaged over 3 seeds.
K
M
K Kapstream
NLL +
Error 4
2
0.622
16.70
0.03
1
2
3
0.622
0.001
16.81
4: 0.05
0.624
0.001
16.99
0.08
0.622
0.000
16.93
0.01
Here we investigate whether upstream versus downstream mismatch partially explains the counter-intuitive result that increasing M can result in worse performance for E². We train downstream E² models with (K =x=1, 1, M = 2) and (K = 1, M 4) from upstream V-MoE checkpoints with K = 2 and K 4. Table 19 shows the results. We see that (K = 1, M = 2) does not seem to benefit from increasing K Kupstream from model (NLL upstream is 8.18 for K and 8.27 for K = 2). On the other hand, (K = 1, M = 4) docs benefit from having K=4 upstream.
2 4, despite the fact that the upstream K model better than the upstream
This confirms that the mismatch between upstream and downstream models is one of the factors explaining the results of g for growing M. However, we also see that the best performing model is (Kupstrem = 2, K 1, M = 2), which suggests that there are other confounding factors, such as those mentioned above.

---

Published in Transactions Machine Learning Research (09/2022)
NLL (lower is better)
Error (lower is better)
ECE (lower better)
25.0
0.9
0.035
0.8
0.030
20.0
ImageNer
0.7
0.025
17.5
0.6
0.020
15.0
0.5
0.015
12.5
0.010
0.4
0.005
NLL (lower is better)
Error (lower is better)
ECE (lower is better)
50
ImageNot-C (avorago)
45
0.05
2.0
40
0.04
35
1.3
30
25
1.0
0.02
20
5-Shot Error (lower is better)
10-Shot Error (lower better)
35-Shot Error (lower better)
30
25.0
35
Mean Across Datasets
22.5
30
20.0
17.5
20
20
12.5
15
10.0
FPR (lower is better)
AUC (ROC) (higher is better)
AUC (PR) (higher is better)
0.25
0.99
0.990
IO CIFARI00
0.985
0.20
0.98
0.975
0.97
0.10
0.970
0.05
0.96
0.965
0.960
2
2
downstream log(GFLOPs) (lower is better)
L/32
B/16
V-MoE
ViT
Member
Members
Members
Figure 13: Extended results for Figure selection other tasks and metrics. see that in most cases, ensembles tend to help ViT and V-MoE (K 1) equally.

---

Published in Transactions Machine Learning Research (09/2022)
1-Shot Erro (lower better)
5-Shoc Error (lower hetter)
10-Shot Error (lower is better)
25-Shot Error (lower hetter)
30
25.0
Mean Accoss Datasets
20.0
17.5
12.5
10.0
Welghted Mean Across Datneet
84-
94
dowsstream (og(GFLOPs) (lower is better)
S/32 F
B/32
L/32
B/16
L/16
11/4
V-MoE
ViT
Member
2 Members
Figure 14: Extended few-shot results from Figure 4 with an additional aggregation method and numbers of shots.

---

Published in Transactions Machine Learning Research (09/2022)
FPR (lower is better)
AUC (ROC) (higher is better)
AUC (PR) (higher is better)
0.99
0.990
CIFAR vs. CIFARIO0
0.985
0.20
0.98
0.980
0.15
0.975
0.97
0.10
0.970
0.05
0.96
0.965
0.020
1.000
1,00000
0.99975
CI DTD
0.015
0.998
0.99950
0.010
0. 9.99925
0. 0.99900
0.994
0.005
0. 0.99875
0.000
0.992
0. 0.99850
0.14
0.95
0.12
CIFAR vs. Places363
0.990
0.85
0.08
0.985
0.80
0.75
0.04
0.70
0.975
0.02
0.65
0.998
0.995
CIFARIO SVHN
0.996
0.03
0.994
0.02
0.992
0.985
0.01
0.990
0.00
0.980
0.988
2
downstream log(GFFLOP)) (lower better)
S/32
B/32
L/32
B/16
L/16
V-MoE
VIT
Member
2 Members
Figure 15: Extended OOD detection the results from Figure with an additional OOD dataset and more metrics.

---

Published in Transactions Machine Learning Research (09/2022)
FPR (lower is better)
AUC (ROC) (higher is better)
AUC (PR) (higher better)
0.96
0.6
CIFARIO0 CIFARIO
0.5
0.92
0.92
0.90
0.4
0.3
0.30
0.99
0.9975
CIFAR vs. DTD
0.25
0.98
0.9950
0.97
0.9925
0.20
0.96
0,9900
0.15
0.95
0.9875
0.94
0.10
0.93
0.05
0.9825
0.975
0.6
CIFARIO0 vs. Places365
0.950
0.7
0.5
0.925
0.4
0.900
0.5
0.875
0.850
0.2
0.825
0.3
0.50
0.95
0.90
CIFART00 vs. SVHN
0.94
0.45
0.88
0.86
0.91
0.84
0.90
0.82
0.30
0.89
2
downstream log(GFFLOP) (lower better)
S/32
B/32
L/32
B/16
L/16
V-MoE
VIT
Member
2 Members
Figure 16: Extended OOD) detection the results from Figure 6 with CIFAR100 as the in-distributtin dataset, an additional OOD dataset, and more metrics.

---

Published in Transactions Machine Learning Research (09/2022)
NIL (lower better)
Ercur (lower is better)
ECE (lower is better)
Drier (lower is bettec)
0.040
0,500
0.7
0.025
0.299
0.029
0.5
0.200
0.015
0.000
0.159
2.25
0.69
0.08
2.00
1.75
0.59
ImngeNal-C
0.94
0.40
0.63
1.00
0.75
100
0.35
ImageWeeA
5-
1.0
4
0.20
0.9
0.8
3
0.35
0.7
0.00
0.50
1.4
0.08
Imng<Na-V2
1.2
0.40
0.06
0.05
0.30
0.53
dowustream logrGFLLPPs (lower is better)
R/32
B/32
L/32
R16
L.16
H/14
V-Mol
Members
Figure 17: Extended results from Figures 4, and 7 with additional metrics.

---

Published in Transactions Machine Learning Research (09/2022)
NL. (lower is better)
Ervor (lower better)
ECE (lower is better)
Brier (lower is batter)
0.96
0.007
0.050
CIFARIO
0.05
1.5
0.006
0.025
0.005
1.0
0.004
0.015
0.00
0.010
0.5
16
14
0.07
0.20
CIFARIO-C
12
0.06
10
0.05
0.3
0.04
0.05
0.10
0.2
downstrrea leg(GFLLOPs) (lower is hetter)
S/32
B/32
L/32
B/16
L/16
V-MoE
ViT
Memher
2 Members
Figure 18: Results for CIFARIO and CIFARIO-C.
NLL (lower
Error (lower better)
BCE (lower better)
Brier (lower better)
0.45
0.04
0.18
0.10
CIFAR100 100
0.03
0.16
10
0.14
0.30
0.25
0.10
0.30
downstream log(GHLOPs) (lower is better)
S:32
B/32
L/32
B/16
L/16
V-MoE
VIT
Member
2 Members
Figure 19: Results for CIFAR100.

---

Published in Transactions Machine Learning Research (09/2022)
NLL (lower better)
Eroor (lower is better)
ECE (lower is better)
Brier (lower is better)
0.08
0.04
Oxford Floaess
0.15
0.02
0.10
0.02
0.05
0.00
1.5
2.0
2.5
1.5
2.0
2.5
1.5
2,0
2.5
1.5
2.0
2.5
downstovam (og(GFFLOPs) (lower is better)
V-MoE
VIT
S/32
B:32
L/32
Member
Members
Figure 20: Results for Oxford Flowers 102.
NLL (lower better)
(lower better)
ECE (lower better)
Brier flower better)
0.14
0.30
Oxford Pats
0.25
0.02
5
0.08
0.15
0.01
1.5
2.0
2.5
1.5
2.0
2.5
dowzstocam (og(GFLLPP))) (lower is betler)
V-MoE
ViT
S132
B:32
L/32
Member
Member
Figure 21: Results for Oxford III Pet.

---

Published in Transactions Machine Learning Research (09/2022)
NLL (lower is better)
Error (lower is better)
ECE (lower better)
24
0.9
22
0.8
ImageNet
20
0.7
18
0.5
14
NLL (lower is better)
Error (lower is better)
ECE (lower is better)
50
0.07
2.25
ImageNot-C (avorago)
0.06
2.00
0.05
40
1.75
0.04
35
0.03
1.25
30
0.02
5-Shot Error (lower is better)
10-Shot Error (lower better)
25-Shot Error (lower better)
30.0
24
Mean Across Datasets
27.5
22
30
25.0
20
22.5
18
20.0
16
17.5
14
20
FPR (lower is better)
AUC (ROC) (higher is better)
AUC (PR) (higher is better)
0.99
0.990
IO CIFARI00
0.25
0.985
0.98
0.980
0.15
0.975
0.97
0.10
0.96
0.965
0.960
1.5
2.0
2.5
1.5
2.0
2.5
1.5
2.0
2.5
downstream log(GFLOPs) (lower is better)
V-MoE
ViT
S/32
B/32
L/32
Member
2 Members
Figure 22: Results for V-MoE with K E {1, 2, 4, 8) and E² with E (1,2,4). Models with larger values K have larger FLOPs.

---

Published in Transactions Machine Learning Research (09/2022)
NLL (lower is better)
Error (lower is hetter)
ECE (lower is better)
24
0.9
0.055
22
0.030
ImageNot
20
0.7
18
0.015
0.5
14
0.010
NLL (lower is better)
Error (lower is better)
ECE (lower is better)
2.4
50
ImngeNat-C (average)
2.2
45
0.05
2.0
400
0.04
1.6
35
30
1.2
0.02
5-Shot Emorr (lower better)
10-Shot Error (lower is better)
25-Shot Error (lower is better)
30.0
24
Mcan Across Datasets
35
27.5
30
25.0
20
22.5
18
20.0
17.5
14
FPR (lower is better)
AUC (ROC) (higher is better)
AUC (PR) (higher is better)
0.99
II vs. CIFAR/00
0.25
0.98
0.20
0.97
0.15
0.97
0.96
0.05
0.96
0.95
1.5
2.0
1.5
2.0
1.5
2.0
downstream log(GFLOPs) (lower is better)
V-MoE
VIT
S/32
B/32
L/32
Member
Members
Members
Figure 23: Results for E² with 4 and K e (1,
48

---

Published in Transactions Machine Learning Research (09/2022)
M
Results Tables
M.1 FLOPs Numbers
Table 20 provides the downstream training FLOPs for various E², V-MoE, and ViT confiuurations. These numbers correspond to the x-values of the points in the figures presented in Section and Appendix L. Table 21 provides the percentage difference FLOPs between the E.; V-MoE and down-DE models most commonly used in this work. Note that the percentage differences for H/14 do not follow the trend of the other sizes, e.g, that the percentage difference between E² and V-MoE gets smaller for larger sizes, due to the fact that for H/15 we use a last-5 configuration rather than the last-2 configuration used for the other VIT familics. Table 22 provides the downstream training FLOPs for the various ablation study models presented in Section 4.2.²
Table 20: Downstream training GFLOPs for the various F.,, V-MoE, and VIT baselines used in this work.
K
M
S/32
B/32
B/16
L/32
L/16
H/14
2
36.69
105.89
452.62
320.92
1356.46
4183.28
58.03
152.98
403.77
E
2
47.98
131.06
552.65
365.42
1533.74
4
80.66
203.31
492.77
70.61
181.40
454.43
24.01
77.97
334.48
271.83
1151.71
3033.60
VIT
2
48.02
155.93
668.95
543.66
2303.43
6067.21
4
96.03
311.87
1337.9 .90
1087.33
4606.85
12134.41
1 26.02
82.35
351.91
279.55
1182.08
3179.90
52.04
164.70
703.81
559.10
2364.16
6359.81
104.07
329.41
1407.63
1118.21
4728.32
12719.61
V-MoE
2
1
31.66
94.98
401.98
301.75
1270.78
3617.94
42.95
120.11
501.99
346.25
1448.06
8
1
65.59
170.44
435.26
Table 21: Percentage difference in downstream training FL.PPs for n. with (K, M) = (1,2) compared with
V-MoE with K = and an ensemble of two such V-MoE members.
S/32
B/32
B/16
L/32
L/16
H/14
E vs. V-MoE
41.01
28.58
28.62
14.80
14.75
31.55
3 vs. down-DE
-29.49
-35.71
-35.69
-42.60
-42.62
-34.22
M.2
Parameter Counts
Table 23 compares the parameter counts for ViT and V-MoE/rt models in each ViT family.
M.3
Summary for NLL under Distribution Shift
Table 24 shows the percentage improvement for E² versus V-MoE in NLL (i.e., NLLy-Mo NLL,
100, with positive values thus indicating that ga improves upon V-MoB) averaged over ImageNet- C, ImageNet-A, and ImageNet-V2, V2, for given VIT family and (K, M) in ((1, 2), (2, 2)} (compared to V-MoE with K = 2 and
STlly-Moo
"Note that the g and V-MoE results here are different to those in Table 20 due to a difference in implementation. We used a simplified, but less computationally efficient expert-routing implementation for all the ablation studies. As a result. the V-MoE FLOPs in Table 20 are lower and cannot be fairly compared with the ablation models presented here. We thus re.benchmarked xd and V-MoF obtain comparable results.

---

Published in Transactions Machine Learning Research (09/2022)
Table 22: Downstream training GFLOPs comparison for the ablation study models in Section 4.2.
K
M
GFLOPs
V-MoE
2
96.895
V-MoE
123.644
V-MoE
173.133 8
E
1
2
109.781
2
2
138.457
E
196.870
Tiling
138.460 2
Partitioning
97.885 2
Overlap = 2
2
2
138, 138.457
Overlap 2
138.458
Overlap 8
2
2
138.459
Overlap =
2
138. 138.460 2
Multi-prod
2
96.330
Multi-pred
4
122.526
Multi-pred
175.889 8
Table 23: Parameler counts for VIT vs V-MoE and E.
S/32
B/32
B/16
L/32
L16
H/14
VIT
36.5M
102.1M
100.5M
325.3M
323. 123.1M
655.8M
V-MoE/E3
166.7M
395.0M
393.3M
845.8M
843.6M
2688.6M
K 4, respectively). We see that for all VIT families except S/32, E outperforms V-MoE. This result not unexpected, since ensembles tend to provide improved performance under distribution shift relative to single models (Lakshminarayanan et al., 2017; 7; Ovadia et al., 2019; Havasi et al., 2020).
Table 24: Percentage improvement for vs V-MoE in NLL under distribution shift, averaged over ImageNet- C, ImageNet-A, and Ien
S/32
B/32
B/16
1./32
L/16
H/14
E K = 1, 2) V-MoE (K -1.15
0.52
0.31
5.34
2.78
8.33
K = 2, M = 2) vs. V-MoE (K = 4)
-0.21
3.91
1.33
7.37
2.27
M.4 Auxiliary Tables with Standard Errors.
The tables in this section provide the mean values and corresponding standard errors for many of the results depicted in figures throughout Scction 5 and Appendix
50

---

Published in Transactions on Machine Learning Research (09 (2022)
Table 25: ImageNet comparison of V-MoE, downstream ensembles there-of, and E with 2 experts per input. in each case.
IMAGOENNr Ennos 4
ANAGNNNN--C (AVERACE))
IMtAcINNi-AA Ennon
IRtACI-NET-V Esnon
K
M
NLL 4
ECE
NLL
EnnOR 1
ECE 1
NLL
ECE 1
NLL 4
ECE
AFLOPs (X6 4
2
0,408 2.m
11:63
0.012 xamr
0.865
21.16
0.018 s
2.270
0.101 co0
0.745
19.50
0.035
15.63
11/14
down-DE
1
0.428 2
0.40K3 100
11.35
0,018
0.871
21.37
0.921
2.273
47.93
0.108
0.758
19.28
0.044
70.78
V-MoE
2
11.89
0,050
0.936
22:41
0.038
2,517
50.34
0.167
0.811
20.25
0.065
2
0.448
12.00
0,030 9.0
1.023
24.89 2
0.024 2000
2.836
57.97
0.160
0.838 1.00
21.30
1.000
0.061
0.74
L/16
down. DE
1
2
0.450
12.62
0.016
1.010
24.43
0.021
2.796
37.06 ..032
0.136
0.818
21.06
0.044
86,34
V-MoE
0/161
12.58
0.025
1,058
25.27
-1:0
1900
37.85
1:3
0.178
0.818
21.33
0.057
le!
2
0.535
14.70
0.027
1.198
28.28 g.0
0.032
4.170
74.74
0.286 3.30
0.989
24.62 6:1
0.0663
6.35
L,32
down- DE
0.533
14.55
0.025
1.184
27.98
0.029
4.139
74.29
0.254
0.982
24.42
0.061
85,29
V-MoE
0.563 6.30
15.05
0.039
1.261
29:12
0.052
4.394
-337
0.3011
1.048
25.22
0.083
1
2
0.519
14.25
0.020
1.352
31.20
0.022
3,835
72.25 1.
0.223
0.974 1m
21.10 1.4.
0.051
12.60
B/16
down-DE
0.519
14.09
0.021
1.316
30.02
3.618
0.203
0,914
235.39
0.054
75.08
V-MoE
2
0.553 1
14:00
0.022
1.372
31.27
0.031
3.875
70.79
9.30
0.245
0.959
24.09
0.056
set
2
0.622
16.70
0.018
1.532
34:73
0.022
5,080
84.31
0.301
1.143
27.97
0.055
11.54
D/32
down-DE
2
0.620
16.44
0.023
1.510
5.455
33.79
0.032
4.891
$1.98
6:3
0.284 co0a
1.126
27.24
0.001
T3.49
V-MoE
2
0.635 1
16.76
0.033
1.502
31.30
0.050
5.032
81.71
0.317
1.150
27.50
0.076
2
0.818
21.18
0.035
2.109
45.79
0.0330
6.419
0.346
1.427
34.111
0.054
15.87
8/32
D.
2
0.807
20.90
0.018
2.106
44.52
0.038
6.063
92..3%
0.335
1.393
33.29
0.062
64.34
V.MoE
2
1
0.829
21.09
0.005
2.182
44.95
0.068
6.227
92.09
0.377
1.437
33.42
0,086
11
Table 26: ImageNet comparison of V-MoE, downstream ensembles th and E² with 4 experts per input in each case.
IMAGGENter
IMAGINNE--C (AVRRAGEE)
IMAGGEETTA ERROR
INCAGENNT-VV V2 ERROO
DOwNSTTIMM
K
M
NLL.
ECE
N
FUBOR J
ECR
NLL.
RCE 1
NLJ.
RCE
ARLOPs (%6)
2
2
0:451 coee
12.61 g.
0.023
1.028-
24.00 c..
0.828 2..
2.896
58.26 gom
0.171
2.00
0.849
2..0
21.40 2m.
0.066
5.92
L/18
down-DD
4
0.440
12.39
0.015
0.983
23.95
0.020
2.712
56.36
0.120
0.803
20.83
0.039
225.53
V-MoE
4
1
0,465 Lose
12.86
cm
0.025
1.000
25.28
0.035
2.976
58.26 1.
0.188
0.850
21.58
0.060
E²
2
2
0.529 soess
10.63
0.019
k.m
1.188
28 28.23
0.025
4.095
71.58
0.253
0.962
21.41
0.055
5.54
1,332
de
0.518
14.29
1.154
27.47
0.028
4.033
78.79
0.237
0.959
24.08
0.054
V-MoE
0.506 coea 1
15.08 cm
0.041
1.267
1.00
29.19
0.053
4.428
75.52 m.8
0.306
1.050
25.21
0.084
5.00
g
2
2
0.510
11.13
0.016
1.528
30.81
0.020
3.713
71.12
0.211
0.915
23.75
0.041
10.09
B/16
down-DE
0.511
13.95 2m31
0.019
2:000
1.293
29.67 2.14
0.038 2.000
3.544
68.49 1:
0.190 2000
0,9130 2.00
28.17
0.050
180.41
V-MoE
4
0.532 1
soea
14.21
0.029
1.350
30.41
0.028
3.726
66.58
0.208
0.973
29.69
0.069
0.612
16.49
0.013
1.491
33.85
0.019
4.572
82,50
0.275
1.099
0.045
9.12
D/32
down-DE
4
0.007
16.17 m.3
0.021
1.483
33.36 835
0.027
4.787
82.11 maar
0.276
cocs
1.099
coes
26.98
+.000
0.065
174.26
V-MoE
1
0.636
16.70
0.081
1.555
$1.55
0.051
5.031
61.62
0.322
1.150
27.19
0.079
0,805
20.97
0.013
1.00
2,112
45.05
6.283
0.312
1.308
55.71
0.051
11.70
8/32
de
4
0.796
20.66
2010
0.015
2.076
44.16
0.031
5.900
92.25 m:00
0.324
20000
1.372
2.000
$2.88
0.054
142.29
V.MoE
0,820
20.97
0.030
2.133
44.53
0.060
6.142
0.366
1.417
33.30
0,081

---

Published Transactions on Machine Learning Research (09 (2022)
Table 27: ImageNett comparison of V-MoE and VIT.
IMAGRNET
IYAGENRI-C ERROR 1
IMAGENEI-A
IMAGIRNT-VV V2
K
NLL 4
ERBOR 4
ECE 4
NLL 4
ECE 4
NLL J
ERROR
ECE J
NLL
ERROR
ECE
1/14
V.MoE
0.426
0.026
2000
0.932
wore
22.43 a.
0.033
ware
2.494
50.22
0.806
20.08
VI
0.426
11.99
0.023
0.929
22.15
0.031
2.332
16,99
0.149
0.788
20.08
0.058
L/16
V-MoE
0.464
1000
12.91
2000
0.022
0.000
1.050
6.04
25.19
2006
0.030
10000
2.914
58.02
0:0
0.167
1:0000
0.844
1.00
21.42 a
0.053 1.000
saa 0.473
0.020
w.ss
60.92
22.04
-
0.055
L/32
V-MoE
0.559
15.10
0.036
1.246
29.02
0.046
4.320
75.25
0,288
1,032
25.13
0,077
VIT
0.556
15.38
0.025
3:909
1.255
29.57
0.038
2.00
4.286
75.32
0.287
2:0000
1.002
25.32
0.067
2:000
B/16
V-MoF
0.533
even
14.33
0.026
a 0.000
1.355
30.59 1
0.040
mm
3.727
= 2.0a
67.72
0.222
8: 2:000
0.967
23.63
am
0.060 2:000
VIT
0.565
15.70
0.021
1.515
31.32
0.012
4.219
75.85
0,280
1,020
25.77
0.061
B/32
V-MoE
1
0.6-12
Looes
16.90
1.900
0.029
0.000
1.568
0.00
31.68 1.000
0.045
1m
5.039
c.00
82-49
0.307
1.0000
1.151
1.00
27.83
0.071
1.0.00
VIT
0.688
0.022
1.089
--100
57.00
1.209
29.80
0.067
2.0000
V.MoE
1
0.834
21.43
0.029
2,171
45.44
0.056
6.199
92.47
0.355
1.433
33,84
0.078
8/32
VIT
0.907
0.003
23.69
0.65
0.016
3.00
2.300
2.0.00
48.75
9.13
0.055
1.00
6.639
1.0
94.06
0.375
0.000
1.529
1.0.00
36.42
0.00
0.066
0.000

---

Published in Transactions Machine Learning Research (09/2022)
Table 28: CIFAR10 comparison of V-MoE, downstream ensembles there-of, and B² with 2 experts per input in each case.
CIFAR10 ERROR
CIFAR10-C ERROR
DOWNSTREAM AFLOPs (56
NLL 1
ECE
NLL
ECE
Ba
0.022
2 marr
0.198
2 2:033
5.65
0.027
6.74 56.04
L/16
down-DE
0.026
0.59
0.004
0.227
0:033
6.18
0.032
0.003
V-MoE
0.028
0.61
0.004
0.220
6.17
0.032
Ea
0.026
0.69
0,003
0.246
7.01
0.033
6.35
L/32
down-DE V-MoE
0.034
0.81
0.005
0.252 0.033
7.00
0.032
85.29
0.042
0.90
0.006
0.325
8.05
0.048
0.000
ES
0.036
0:0000
0.84
0.005
0.295
8.28
0.038
12.60
1/166
down-DE
0.030
0:0000
0.81
0.0.
0.003
0.000
0.247
0.000
7.60 0.36
0.028
2.000
75.09
V-Mo
0.040
0..0a
0.97
0.006
c.000
0.303
8.08
c.0
0.047
0.041
+ 0.000
1.11
0.00
0.004
c.0o0
0.340
10.01 a.s
0.040
11.54
18/32
down-DE V-MoE
0.036
2.00
0.003
a 0.000
0.345
1. 0.000
10.87
:: 0.15
0.042
=: 0.000
73-49
0.042
0.006
0.405
11:30
0.063
66.000
Ed
0.060
0.476
0.049
0.002
15.86
8/32
down-DE
0.056
1.75
0.005
0.491
15.42
0.064
64.34
V-MoE
0.058
1.80
0.007
0.514
15.48
0.076
Table 29: CIFAR10 comparison of V-MoE, downstream ensembles there-of, and E² with 4 experts per input
in each case.
CIFAR10 ERROR
CIFARIÔ-C ERROR
DOWNSTREAM
K
M
NLL 4
ECE
NLL
ECE
AFLOPs (%)
E
0.026
0.58
0,003
0.247
6.58
0.036
5.92 226.53
L/16
down-DE V-MoE
0.025
0.57
0.004
0.219
6.12
0.030
0.034
0.75
0.002
0.005
0.278
6.000
7.71 0.10
0.043
0.000
E
0.033
0.80
0.00
0,004
0.242
0.0000
6.65 c.ua
0.031
0.063
5.54
1./32
down-DE
0.030
0.78
0.004
0.232
2 9:0m
6.74
0.028
222.94
V-MoE
0.035
0.200
0.81
0.00
0.005
0.000
0.295
0,000
7.63
0.04
0.044
0.00000
0.034
4 0.200
0.81
0.00
0.004
c.000
0.253
8,06
4 1.u
0.037
6.000
10.09
13/16
down-DE
A mirro
0.242
0.003
180.41
V-MoE
0.032
0.000
0.79
0.00
0.005
0.266
0,000
7.55
0.30
0.040
0.000
Ed
0.034
0.002
0.00
0.003
0.306
0.20
0.037
0.002
9.12
B3/2
down-DE
01.355
1.06
0,003
0.338
10.77
0.040
174.26
V-MoE
0.042
1.12
0.006
0.410
0.007
11.29
0.066
E
0.058
1.82
0.004
0.472
15.01
0.051
11.69
8/32
HEwn-DE
0.055
1.76
0.004
0.486
15.36
0.061
142.28
V-MoE
0.039
1.80
0.00
0.007
0.536
15.64
0.084
Table 30: CIFAR10 comparison of V-MoE and VIT.
CIFAR 10 ERROR
CIFAR10-C ERROR
K
M
NLL
ECE 1
NLL
ECE |
L/16
V-MoE
1
0.029
0.001
0.59
0.02
0.004
0.000
0.236
0.011
6.25
0.20
0.034
0.002
ViT
0.034
1 0
0.69
0.08
0.005
1 0.000
0.325
0.000
7.53
= 0.40
0.050
0.004
L/32
V-MoE
0.040
+ 0.
0.86
0.08
0.006
0.000
0.290
0.018
7.35
0.99
0.043
VIT
0.030
0.78
0.005
0.281
7.28
0.043
V-MoE
0.030
0.85
0.003
0.249
7.53
0.030
B/16
VIT
0.031
0.001
0.87
0.05
0.004
+ 0.000
0.293
0.000
7.99
0.18
0.043
0.002
B/32
V-MoE
0.038
000
1.14
0.08
0.004
0.359
0.000
11.09
0.17
0.046
VIT
0.050
+ 0.002
1.38
+ 0.04
0.007
0.000
0.330
00010
8.97
0.19
0.051
0.00000000 0.002
S/32
V-MoE
0.059
001
1.84
0.09
0.006
000
0.497
0.007
15.53
0.90
0.066
00000
VIT
0.065
2.08
0.006
0.518
15.04
0.24
0.072

---

Published in Transactions on Machine Learning Research (09 (2022)
Table 31: CIFARIO OOD comparison of V-MoE, downstream ensembles there-of, and E with 2 experts per input in each case.
CIPARIO vs. CIFAR100
CIFARIO vs. DID AUC (BOC)
CIFARIO va. PLACEaO66
CIPARIO vs. SVIN
K
M
AUC (PR)
AUC (ROC)
FPRE95
AUC (PR)
FPRS99
AUC (PR)
AUC (ROG)
FPRA95
AUC (PR) 1
AUC (BOC)
FPR
0.9905
0.9902
0.5899
0.9935
0.005
0.9103
0.9536
0.029
0.9963
0.9978
0.0007
1
down-DE
0,9896
0,9990
0.9996
0.9421
0.9959
0.9968
0.0007
V-MoE
0.9695
0.9890
0.0389
0.9808
1000
0.9888
00011
1000
0.7891
0.0887
0.0635
0.9966
0.0980 -10L
0.0005
0.9875 2000
0.9875
0.0427
0,9098
0.9900
0.0011
0.9196
0.9034
0.0321
0.9055
0.9970 -....
0.0012
1/332
0,98522
0,3842
0,0571
0.0894
0.9006
0U0032
U.7388
0,3846
0.9839
0.9960
0.0012
V-MoR
0.9535
0.0346
0.0017
0,98221 -000
0,9500
0.08866
0.98944
0.9667 6000
0.0015
0.7061
0,9834
0.1222
0,9014
1.0000
0,9942
0.0029
B/16
Gown DE
0,9860 4.3000
0.9859
0.0540
0,9990
C
0.9993
0,0009
0.5807
0.9922
0.0386
0.9851
0.0974 -1882
0.0027
V-MoE
0,9822
0,9800
0,0800
0.0094
0.9964
0U0123
IL7300
019841
I11168
0.9020
0.9953
0.0637
0,9773
0,9738
0,1254
0.9022
0 1955
0.0038
0.7438
0.9794
0,0802
0.988]
0,9918
0.0071
dow
0.9813
0
0.9989
0.8507
0,9900
0,0514
0,9924
0.0054
V-MoE
0,0798 6000
0,9750
0.0875
0.9697
0.8986
0.0017
0.8725
0.9904
0.0482
0.9918
0.8952
0.0047
0,9617
0.9367
0,2029
0.9892
50000
0.9956
0.0118
0,7431
0.9750
0.1270
-1000
0.9821
0.9899
0.0388
8/32
COWT-DE
0.9696
0.9669
0.1856
0,9996
0.9981
0,0037
0.813G
0.9845
0.0816
0.9879
0.9933
0.0164
V-Mol
0.9067
0.1870
0.9979
0,6010
0,9830
0,0827
0.9678
0.9927
0.0192
Table 32: CIFARIO OOD comparison of VIT and V-MoE
CIPARIO vS. CIPARTOO
CIPARTO DTD
CIEAR1O vs. PLACEI366
CIPARTO SVHN
K
M
AUC (PR) 1 AUC [8OC]
PPRE9G
AUC (PR) 1
ALC (ROC) 1
PPRE555
AUC (PR) 1
AUC (HOC) 7
FPREOS 1
AUC (PR) 1
AUC (ROO) 1
PPRNIS 1
L/16
V-MoR
1
0.9891
0,9885
0.0379
0.9899
0.9997
0
0,9400
cower
0.9960
0.0162
0.9972
0,9984
0.0007
VIT
0,9830
0,0541
0,0006
0.2078
0,7334
0.5857 13000
I19947
DJKE22
1/33
V-M
VIT
0,9834
0.9842
0,9850
0.9847
0.0573
0.0GG1
come
0.9008
1.000
0.0987 0000
0.9976 1982
0.0016
0.9164
0.7089
0.9838
0.9682 2
0.0279
0,0020 s
0.9940
0.9042
0.9967
0.9066
0.0015
0,0028
B/16
V-M
VIT
1
0.9856
0,9801
0.9855
0.9798
0.0554
0,0857
0. 0.9998
0,9996
0.9992
0.9679
0.0015
0,0346
0.8536
0.8598
0.9912
0,9895
0.0413
0
0.9919
0,9928
0.9972
0.9961
0.0027
0,0018
D/32
V-MoE
0.9814
0.0855
0.9989
0.9902
0.0806
VI
0,9752 1O00
0.9716
0.1685
0,0085
0.9915
1991
0.0136
0.0307
0.5134
à 1
0.1021
0,9572
0.9920
0.0148
8/32
V-MoRR
0.9885
0.9658
0.1922
0.9006
0.9977
0.0845
0.8092
0.9841
0.0821
0.9874
0,9929
0.0185
VII
0,9629
0.5688
0,2422
0.5063
0,0134
0.7177
0.9779
0,1110
0138007
0.9889
0,1426

---

Published in Transactions on Machine Learning Research (09 (2022)
Table 33: CIFARIOO OOD comparison of V-MoE, rownstream ensembles there-of, and E with 2 experts per inpul in each case.
CIPAR1OO vs. CIFAR10
CIFARLOO vs. DTD AUC (BOC)
CIFARIDU vs. PLAUSE365
CIPARIOC vs. SVHN
K M
AUC (PR)
AUC (ROC) 1
FPRE95
AUC (PR) 1
FPRS95
AUC (PR) C
AUC (ROG)
FPRM96
AUC (PR) 1
AUC (BOC)
FPRS995
0.9507
0.9490
0.2889
0.5009
0.9847
0.0822
0.7281
0,9542
0.2909
0.9031
0.9482
0.3071
1
down-DE
0.9481
0.9681
0.7619
0.9664
0.2156
0.9071
0.9507
0.2932
V-MoE
0,9091
0.9413
0.2901
0.9977
0.9687
0.0674
11007
0.7773 6.0
0.9680
0.2150 4.80
0.9002
0.8461 -100
0.3223
0.9423
0.9379
0.5501
0.9058
0.9792
0.1165
0.6853 1
0.9394
0.3670
0.8848
0.9398 2.00
0.3428
1/332
0,9980
0.9387
0.3362
0,9958
0.9796
0,1170
0.6710
0.9436
0.3465
0.8677
0.9460
0.3633
V-MoR
0,9530
0.9790
0.8767
0.3434
0.9200
0.9121 -.00
0,8656
0.9638
0,9710
00001
0.1642
0.5475
0.9102
0.4652
0.8730
0:9319
0.3699
B/16
Gown DE
0.9244 443000
0.9275
0.3506
0.9932
0.9777
0.1144
0.6023
0.9313
0.3377
0.8726
0.8340
0.35226
V-MoE
0,9150
0,9211
0.3729
0,9950
0.9768
0.1193
0.6029
0.9331
0.3682
0.8704
0.9283
0.2015
0,9075
0,6915
0,5358
0.9005
0.9551
0.2181
0.5355
0.8811
0.5465
0.8583
0.9221
0-3051
down-
0.9188
0.9158
0.9711
0.1517
0.5525
0.4276
0.87558
0.9292
0.5955
V-MoE
0.9182 1.00
0.9151
0.4884
0.9013
0.958)
0.218]
100M
0.0640
0.8969
0.5230
0.8641
1.0m)
0.8252
0.4249
0.8677
0.8528
0.6086
0,9878
c
0.9430
0.2677
0,3628
0.8519
0,5718
0.8279
12007
0,8004
0.4543
8/32
COWT-DE DE
0.9607
0.8001
0.5203
0,9892
0.9G32
0,1900
0.418G
0.8814
0.4667
0.8162
0.8012
0.4835
V-Mol
0.5063
0,9694
0.9628
0.4129
0.6762
0.7009
0,8133
0.6U51
55
Table 34: CIFARIOO OOD comparison of V-MoE and VIT
CIFAR1B0 vs. CIPAR1O
CIFAR100 vs. DTD
CIPARIO0 vs. PLACES365
CIPARIDO vs. SVHN
K
M
AUC (PR) 1 AUC [8OC]
PPRE9G
AUC (PR) 1
ALC (RDC) 1
PPRE05
AUC (PR) 1
AUC (ROC) 7
FPRE0OS 1.
AUC (PR) 1
AUC (ROO)
PPRNOS 1
L/16
V-MoR
1
0.9454
0,9481
2000
0.2682
0.9878
0,9882 gooe
0.0658
0.7681 2.00
0.9687
0.2141
0.9115 2
0,0633
0,2784
VIT
0.9411
0.9449
0,2784
0,9070
0.3654
0.7552
0.9608
1.2566
118097
0,300
1/33
V-M
VIT
0,9358
0.9285
0.9306
0.9323
0.3131
0.3201
come
0.9007
0.9961
0.0838
0.9806
0.0911
0.7541
0.615T
0.9G34
0.9481 1000
0.2292
0,330 so0e
0.8863
0.8496
0.9459
0.9294
0,3063
0.391
8/16
V-M
VIT
1
0.9206
0.9177
0.9241
0.9171
0.3572
0,3825
0.9951
0,9906
0.9776
0.9071
0.1094
0,2199
1.00
0.5824
0.0913
0.9331
0.8080
0.3532 -.00
0.4927
0.8720
0.8325
0.9309
0.9204
0.3705
0,0220
D/32
V-MoE
0.9168
0.971d
0.9178
0.4258
0.4026
VI
0.9058 1.R
0.9041
1.00
0.4180
0.0027
0.9663
0.1631
0.5306
0.9112
0.4176
0.8579 uo0a
0.9116
0.4328
8/32
V-MoRR
0.8878
0.8631
2000
0.5281
0.9895
0.9524
0.1978
0.4024
0.8778
0.4765
0.8224 1
0.8945
0.4729
VII
0.8644
0.5541
0,5716
0.5287
0,2976
0.3140
0,5340
ILT082
0.5088
0,8894
0.4588

---

Published
Table 35: Few-shot comparison of B², V-MoE and ensembles thereof.
MEAN Acnoss DATASETS
WEIGHTED MEAN ACRceS DATASETS 10-SHOT ERBOR
DOWNSTREAM AFLODs (%6) 4
R
M
L-SHOT ERROR
S-SHOT Eanon
10- SHOT ERROR
25-Sno7 EAROR 1
1-SHOT ENBOR
5-SHor Ennon
25-SHO7 ERROR 4
Transactions on Machine Learning Research (09 (2022)
17.87
14.29 a.2
10.84 cm
78.51
80.65 aaz
81.26
15.63 75.79
H/11
dowa-DE
30.84
Low
17.77
11.43 1910
11.06 1:0
78.36
1::00
80.63 1
81.31 1
81.00 1/1
V-MoE'
18.77 1.44
15.19 warr
78.93
1:00
80.89 1
81.18 L
81.83 1.0
R
34.08 2-1
19.57 2:1
15.21 2014
11.75 c
79.50 9
81.09 201
81.48 230
81.76 k.
6.74
1.116
dF.
92.98
19.65
15.44
11.92
79.09
81.10
N1.54
81.79
86.04
V-Mol
33.98 1:20
20.42 -022
16:20 1.3
12.73 -33
79.50 1.0
81.30 -.31
81.72 +.32
81.97 -...3
Ex dowa-DE V-Mot
36.71
22.14
17.79
13.07
80.51
81.77
82.11
82.26
6.35
L/32
36.15 1.00
22.18 1:
17.81 1:334
14.00 1.0
80.45 1:00
81.79 1:00
82.13 1.0
82.27 1:10
85.29
36.56
25.00
18.86
15.02
80.56
52,00
82,97
82:50
38.60 L:0
21.93 1.
17.43 1.3
13.02 Low
81.14 1.0
81.74 1.00
82.05 100
82.19 1/10
12.60
B/16
down-DE
2
36.96
23.36 2.00
18.99 2019
14.84
81.39
82.09
82,40 2.33
82.46
75.09
V-MoF
37.76 1.1
23.16 9.99
18:79 2.04
14.94 2.14
80.97 1:400
82.04 0.00
82.36 g.033
92.48 2.94
E
2
43.39 -.3
26.51 6.3
21.18 1.37
16.82 1/0
82.86
82,91 8.0
82,93
82.90 1/1.0
11.54
B//2
down-DD DE
c.u
26.48
21.61 ..3
17.20 c.m.
82.24 6...
82.89
83.03 -3:0
82.09 x..0
73.40
V-Mol'
2
12.50
27:41 2.00
22.73 1.00
18.60 1.3
82.66
83.16 1.000
83:29 1.0.
83:29 1.0.s
91
31.28
27.54
21.85
85.89 1.0e
81.81
81.39 L.0.
1/10
15.87
8/32
down-DE
2
48.27 m-0
32.19
26.35 a.
21.95 1:000
84.62 k.0
84.35
84.19 =340
84.04 5.1.0
64.34
V-MoF.
40.37
33.51
28.00
23.25
85.04
84.60
84.54
84.33
99
Table 36: Few-shot comparison of V-MoE and VIT.
MEAN ACROSS DATASETS
WEIGHTED MEAN ACROSS DATASETS
K M
1- 1-SHOT ERROR +
3-SHOT EAROR
10 SHOT ERROR J
25 25-SHOT ERBOR
1 1-SHOT ERROR
5- 3-SHOT ERROR J
10 10-SHOT ERROR 1
25. SHOT ERBOR J
V-MoE:
33.04 ni
19.23
15.63 cor
12.06 n:000
79.05 n
81.00 cire
81.59 nire
81.82 non
H/14
VIT
35.78 2.
16.78 om
13.62 6.0
79.97 ean
81.37 a.
81.87 a ons
82.16 sin
V-MoE
34.15 van
20.32
16.14
12.71 1
79.54
81.27 6.t
81.71
81.97
L/16
VIT
36.52
21.79
19
17.51
14.07 2.04
80:38
81.67 2.000
$2.03
82.27
L/32
V-MoE
1
36.83 9.9
23.05 6.0.
18.78 6.a
14.95 1990
80.67 1.000
82.01 6.00
82.35
6.04
82.48 1.9.22
VIT
35.22 6.33
23.97
19.56 1.
15.75 624
81.10 6.0
82.25 1.000
$2.53
1.000
82.65
B3/16
V-MoE
39.22
24.42
15.94 4.0
81.47 4.00
82.36
82.70 4.0.
VIT
41.29
25.03
20.08
15.87
82.16
82.51
82.65
82.68
V-MoF: VIT
1
42.37
27.60
22.89
18.46
82.64 2:111
83.19
83.33
83.26
B/32
44.60 0:30
28.20 6:17
22.97 6:19
18.86 1:
83.35 6:08
83.35 6.04
83.35
83.35
S/32
V-MoE
49.60 w00
33.34
27.88
23.30 6.0.4
85.09 6.00
84.33
VET
54.16
36,25
29.88
24.42
86.32
85,26
84,90
84.54

---

Published in Transactions Machine Learning Research (09/2022)
N
Additional Algorithm Overview Diagrams
(11 [11-5)
TRANSSORMER Brock
MSS
Nora
yo MoM
THANSPODKER BLOCK
M-MoE BLOCK
CLASSIFFE
Mup (ha)
DISPATOR
gube (Wa Pb.)
MLP (Peg)
(KC 2)
1
SOFTHAX
12g
hg
(1bg)
fgg
hg hg
gateg (Wgh.))
Mung (Pe))
Figure 24: End-to-end overview of the Partitioning method, from Section 4.2.1, with E=6 experts, parti- lioned into = groups, with sparsity K 22 and a "last-2" configuration. Top: Partilioning contains sequence of transformer blocks, followed by alternating transformer and p(artitionod)-MoE blocks. As in ViT, images are split into patches whose embeddings are processed by each block. Here, we show embed- ding for each of three images ( D. M). Bottom left: In a MoE block, we replace the transformer block's MLP with parallel partitioned expert MI.Ps. The effect of the routing weights is not depicted. Bottom right: The classifier makes predictions from the final representations (m). Notice that without a tiling mechanism, there only a single prediction per input.
SOPTHAX
1
Huny (
Ar
BSSESILL (MaA)
1
MPP (
hg
1
N3,1
HuP (R
A3,1
2
Prg
(Whh) (h)))
(K =0
A
kg
HIP (haga)
hs.u
1st MoE BLOCK ONLY
a22
Figure 25: End-to-end overview of the Tiling method, from Section 4.2.2, with = experts, sparsity of K 2, and "last-2" configuration. Top: Tiling contains a sequence of transformer blocks, followed by alternating transformer and MoE. blocks. As VIT, images are split. into patches whose embeddings are processed by each block. Here, show 1 embedding for each of three images 1, D). Bottom left: In a MoE block, we replace the transformer block's 's MLP with parallel partitioned expert MLPs. The effect of the routing weights not depicted. Embeddings are tiled (8) in the first p-MoE block only. Bottom right:
The classifier averages predictions from the final tiled representations (M). Notice that without partitioning, some patches and their corresponding tiled versions can be routed to the same experts, resulting in a reduced diversity in predictions.

---

Published in Transactions Machine Learning Research (09/2022)
[n x
TRANSFORMMR BLOCk
Nona
MSA
Nossa
Moi
T'KNSPPOSMER BLoCK
MOR BLOCK
CLASBIFIER
ScePNIIN
1
11.2
[MRAN)
(P.)
h
A2
g00 (Wh;)
CommENE
h2
L.
(K=2)
22
(h,)
hg
h3
Facrpr Lase
k3.2
Mol:
Figure 26: End-to-end overview of the simple Multi-pre MoE, from Section 4.2.4, with E 3 experts, sparsity of K 2, and "last-2" configuration. Top: The multi-pred MoE contains sequence of transformer blocks, followed by alternating transformer and MoE blocks. As in VIT, images are split into patches whose embeddings are processed by each block. Here, we show embedding each of three images (L, E, D). Bottom left: In all but the last MoE block, recombine the predictions usual. Bottom right: The classifier averages predictions from the final representations


Reviews:
[{"review_id": "wyKUooObco", "reviewer": "Reviewer_Qd4u", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the system is explained well and the benefits clearly demonstrated on multiple tasks measuring error vs computation.  Ablations demonstrate the independent effects of tiling and partioning.  Measuring KL divergence between groups notably confirms the intuition that ensembling helps when predictions are diverse; the overlap study particularly drives this point home.\n\nThere were a few more minor points I didn't entirely understand.  First, how is the tiling of M representations handled in transformer attention?  Is attention applied within each of the M groups, or can they mix with one another?  Are transformer weights in non-final MLPs shared between the M groups or unshared?  And how is the computational cost of these parts outside the MoE affected by tiling replication?\n\nSec 5 figs 4, 5, 6, 7:  The zoomed-in (red outline) plot and full plot on the left include different points.  It seems some were included in one but not the other.  Maybe to keep the figures less busy, although sometimes there are points missing in the zoomed-in plot as well, e.g. brown triangle in fig4.  It's also not clear in the figure whether E3 uses M=2 or 4, I only see one box size in each plot.  I found this inconsistency confusing, it would be better to either include all points or explain why some are missing in the caption.\n\n\n\nAdditional questions:\n\nIs it possible to measure KL-divergence between experts in addition to between groups?  This could extend the diversity measurements to V-MoE without partitioning to provide interesting comparisons (currently these are marked n/a in the tables).  The gating weights may pose an issue here, but perhaps these could be incorporated in an interpolation so that full weight on 1 expert results in 0 diversity measure between experts, while equal weights yields the full distance comparison.\n\nFigure 2 is interesting, illustrating static and adaptive ensembling (M and K), but I didn't see a similar plot for the E3 system.  It could be nice to have another plot like this to show the effect of K vs M:  since K*M total experts are selected, there is a question of how to select K and M for a computation budget, perhaps using a smaller MLP where K or M might be larger.\n\nDoes the order of experts in initialization affect the system?  The paper states that sparse MoE checkpoints are used.  Does this mean the model is initialized with trained V-MoE weights for E experts, but applied using M partitioned groups?  What if the order of the E experts is permuted so that different experts are in each group?\n\nI don't see the number of experts E mentioned for the experiments in tables 4 and 5, though the text around table 2 and many other places says E=32.  Was this the case here as well?  If so, it could make sense to say something along the lines of \"we set the number of experts E=32 unless specified otherwise\".\n\ntable 2:  I think it could also be good to include plain (non-tiled, non-partitioned) MoE as a baseline, e.g. selecting the same number K*M of all E experts with a single gating; though, V-MoE is compared in other tables.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a system \"E3\" combining MoE and ensemble averaging, inside of a vision transformer.  Input tokens are replicated (\"tiling\"), and independently processed in groups (\"partitioning\") by transformers with MLPs replaced by MoEs.  This is similar to V-MoE, with smaller sets of MoEs applied independently in groups and averaged at the end.  V-MoE using top-K of E experts has similar computation in the MoE part of the system to E3 using M tilings each choosing top-K/M of E/M experts --- but the latter can average over more diverse predictions, as exhibited by the measurements of KL divergence between groups.  Extensive experiments show the method outperforms vanilla ViT and V-MoE in classification for imagenet variants and few-shot classification tasks, particularly in the largest computational budgets, and mixed results in OOD.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "See above questions.  Addressing the missing points / inconsistencies in figures for Sec 5 is most important.  No others are critical, but would further strengthen and clarify several points.", "broader_impact_concerns": "none", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pfJ7eSYJbW", "reviewer": "Reviewer_JkZs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strong aspects\n- the writing/presentation is good and it is a pleasure to read this paper\n- the empirical experiments are thorough\n\n\n## Weak aspects\n- The motivation for combining MoE and ensemble is not clear to me. Sec 3 is not surprising: like the naive deep ensemble, simple ensembling of sparse MoEs results in strong predictive performance while losing computational efficiency. What are the new insights? In my opinion, Sec 3 is not a qualified motivation for the proposal E^3. You should make clear the necessity of combining these two methods, for example, given an MoE with 16 experts and an ensemble of two MoEs with 8 experts respectively (assuming the two models have similar FLOPs), you show that the ensemble significantly surpasses the separate MoE. Then we are motivated to introduce ensemble into MoE, and will figure out how to realize this efficiently in the next section. \n\n- The technical novelty is limited. The concept of using partitioning and tiling to implement an efficient ensemble has been explored by multiple related works like the (cited) batch ensemble, MIMO, etc. I would better regard it as a trick to implement ensemble instead of a considerable technical contribution.\n\n- The improvements of E^3 upon V-MoE in aspects of Error and NLL in Table 4 and 5 are not significant. The OOD detection performance in  Figure 6 and the NLL under distribution shift in Figure 7 seem to prove that V-MoE is at least on par with E^3, with only marginally added GFLOPs.\n\n- Though I know ViT architectures are powerful and widely used nowadays. However, I think conditioning your method on such a specific architecture is restrictive. It seems not pretty hard to apply this method to other architectures but adding relevant experiment results is definitely beneficial. \n\n- An issue caused by tilling is the rapidly-increasing memory cost. It is better to clarify this point and report some details.\n\n- A minor issue: why can replacing topK(softmax(Wh)) with topK (softmax(W h+ σε)) mitigate the non-differentiability of topK? I see a citation but I recommend the authors revise this part carefully.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies the interplay of ensembles of neural networks and sparse mixture of experts (sparse MoEs). It first performs a comprehensive evaluation of sparse MoEs in uncertainty-related benchmarks and then proposes E^3 -- a scalable and simple ensemble of sparse MoEs. The extensive experiments are extensive, confirming some of the made claims.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I expect the authors to address my concerns listed above and will re-evaluate this submission after revision.", "broader_impact_concerns": "Not applicable", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KFJwttbJPK", "reviewer": "Reviewer_n8qH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The empirical investigation on ensembling and MoE is valuable to the community.\n- While the proposed method introduces additional hyperparameters, it can be fine-tuned from existing MoE checkpoints.\n- Consistent emperical improvements are observed in terms of NLL and ECE.\n- The ablation studies in 4.2.* validate various design choices.\n- MoE is presented very well for non-MoE-experts such as myself.\n\nWeaknesses:\n- Various tables and figures are missing metrics that could improve clarity of presentation and the ability to better contrast the method with other approaches. I would highly recommend adding FLOPs to Tables 2-5. Moreover, I would recommend a version of Figures 3/4 which looks at accuracy instead of NLL. It is not clear why NLL is a good metric.\n- Many of the experiments in the main text are with M=2, which I would appreciate more discussion about. It seems from Appendix Table 12 that increasing M hurts performance (despite increasing FLOPs) which I don't understand and perhaps warrants further investigation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper attempts to combine the best of both ensembles and MoE (mixture of experts). This investigation is motivated by the observation that increasing the number of experts in MoE provides orthogonal improvements to training networks independently and averaging their predictions (i.e., ensembling). This motivates a proposed modification of tiling (i.e., making copies) of the input then running MoE but with disjoint groups of experts. Layers without experts remain shared.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "- Adding a FLOPs comparison in Tables 2-5 and considering accuracy instead of NLL in Figures 3/4 (or, motivating why NLL is a good metric).\n- Investigation or further discussion as to why M>2 seems to hurt performance despite increasing FLOPs, motivating why M=2 is a good choice.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigated MoE and ensemble averaging within the vision transformer. Though a similar idea has been explored before in V-MoE, the authors have given a thorough comparison between the proposed algorithm and V-MoE, and demonstrated the algorithm's advantages in terms of few-shot error and in-distribution ImageNet NLL,  in-distribution, and OOD ECE,  NLL for Cifar10/100/-C. Reviewers agree that the empirical findings in this paper are meaningful to the research community.  The authors have updated the manuscript accordingly by including the new discussions and experimental results. 
Decision: Accept (poster)