Sparse MoEs meet Efficient Ensembles
Authors: James Urquhart Allingham
Florian Wenzel
Zelda E Mariet
Basil Mustafa
Joan Puigcerver
Neil Houlsby
Ghassen Jerfel
Vincent Fortuin
Balaji Lakshminarayanan
Jasper Snoek
Dustin Tran
Carlos Riquelme Ruiz
Rodolphe Jenatton
Venue: ICLR 2022 Submitted (2022)
Keywords: Ensembles
Sparse MoEs
Robustness
Uncertainty Calibration
OOD detection
Efficient Ensembles
Large scale
Computer vision

Abstract: Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, lead to strong performance. We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs). First, we show that these two approaches have complementary features whose combination is beneficial. Then, we present partitioned batch ensembles, an efficient ensemble of sparse MoEs that takes the best of both classes of models. Extensive experiments on fine-tuned vision transformers demonstrate the accuracy, log-likelihood, few-shot learning, robustness, and uncertainty calibration improvements of our approach over several challenging baselines. Partitioned batch ensembles not only scale to models with up to 2.7B parameters, but also provide larger performance gains for larger models. 

Under review conference paper ICLR 2022
SPARSE MOEs MEET EFFICIENT ENSEMBLES
Anonymous authors Paper under double-bling review
ABSTRACT
Machine learning models based on the aggregated outputs of submodels, either at the activation or prediction levels, lead to strong performance. We study the in- terplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs). First, we show that the two approaches have complementary features whose combination is beneficial. Then, we present partitioned butch ensembles, efficient ensemble of sparse MoEs that takes the best of both classes of models. Extensive experiments on fine-tuned vision Trans- formers demonstrate the accuracy. log- likelihood, few-shot learning, robustness, and uncertainty improvements of our approach over several challenging baselines. Partitioned batch ensembles not only scale models with up 2.7B parameters, but also provide larger performance gains for larger models.
INTRODUCTION
Neural networks typically use all their parameters to process an input. Sustaining the growth of such models--rechhing today up to 100B parameters (Brown et al., 2020)-is challenging, e.g., due their high computational and environmental costs (Strubell et al., 2019; Patterson al., 2021). this context, sparse mixtures of experts (sparse MoEs) employ conditional computation (Bengio et al., 2013) to combine multiple submodels (experts) and route examples to certain experts (Shazeer al., 2017: Lepikhin al., 2021; Fedus et al., 2021; Riquelme et al., 2021: Yang et al., 2021). Conditional computation can decouple the growth of the number of parameters from the training and inference costs, by only activating subset of the overall model in input-dependent fashion.
Paralleling this trend, the deployment of ML systems in safety-critical fields, e-g., medical diag- nosis (Dusenberry et al., 2020b) and self-driving cars (Levinson et al., 2011), has motivated the development of reliable deep learning. e.g., for calibrated and robust predictions (Ovadia et al., 2019). Among the approaches, ensembles of neural networks have remarkable performance for cal- ibration and accuracy under dataset shifts (Ovadia et al., 2019), These methods improve reliability by aggregating the predictions of individual submodels (ensemble members).
While sharing conceptual similarities, these two classes of models-Moo and ensembles have different properties. Sparse MoEs adaptively combine their experts depending on the inputs, and the combination generally happens at internal activation levels. Ensembles typically combine several models in a static way and at the prediction level. Moreover, these two classes of models tend to be benchmarked on different tasks: few-shot classification for MoEs (Riquelme al., 2021) and mncertainty-relatee evaluation for ensembles (Ovadia et al., 2019; Gustafsson et al., 2020).
CONTRIBUTIONS: In this paper, we study the interplay between sparse Mols and ensembles. This results in two sets of contributions.
Contribution 1: Complementarity of MoEs and ensembles. We show that sparse MoEs and en- sembles have complementary features and benefit from each other. Specifically:
The adaptive computation in sparse MoEs and the static combination in ensembles are orthogonal, with additive benefits when associated together. Their association results in insightful performance versus FLOPs trade-offs while varying the ensemble size and sparsity.
In sparse MoEs, combining models at the prediction level leads to improved uncertainty estimates. Over tasks where either sparse MoEs or ensembles are known perform well, naive-ame and com- putationally
ensembles of MoEs provide the best predictive performance. Our bench- marking effort includes the first evaluation of sparse MoEs on uncertainty-relatee vision tasks, which builds upon the empirical work of Riquelme et al. (2021).

---

Under review conference paper ICLR 2022
>-Moc Block
n=xix Vir Block
Noxz
MSA
Norsd
FBB Moe
MMMull BLoCK
V BLOCK
CLASSINER
Block
h
(P.,))
A1.1
Disnercut
EXSEMBLE
A2.
Ar
ha.u
McPu(h, .
Av
(2
lilm
(LP
Buke ( (2
1st 1-3008 Block onlo
Figure I: End-to-end overview of PBE with 6 experts, = partitions, sparsity of K 2, and a "last-2" configuration. Top: pBE contains sequence of ViT blocks, followed by alternating p- MoE and ViT blocks. Images are split into patches whose linear embeddings are processed by each block. Here, we show embedding for each of three images (.. D. D In practice, we have many embeddiings including special class embedding. a a-MMoE block, we replace the VIT block's MLP with parallel partitioned expert MLPs, see (3). Em- beddings are tiled (22 in the first P-MoE block only. The effect of the routing weights is not depicted. Bottom right: the classifier uses the class embeddings (-) to make predictions. Ensembling of the predictions for the emheddings and corresponding tiled versions happens only at test time.
in Dosovitskiy et al. (2021). Bottom left: in
Contribution Partitioned batch ensembles. We propose partitioned batch ensembles (pBE), see Figure 1, an efficient ensemble approach tailored to sparse MoEs. Specifically:
pBE improves over sparse MoEs across metrics including few-shot performance, likelihood and calibration error. PBE matches the performance of deep ensembles for 30%-43% fewer FLOPs.
pBE gracefully scales up to vision Transformers with to 2.7B parameters.
pBE is both simple (requiring only minor implementation changes) and convenient because stan- dard sparse-MoE checkpoints can used directly to initialize pBEs for fine-tuning.
2
PRELIMINARIES
We focus on classification tasks where we learn classifiers of the form f(æ; 0) based on some train- ing data D = {( Vo, J A pair (ar,, Un) corresponds to input 2,, E R²" together with its label yu (1...., C} belonging to one of the C classes. The model f(: 0) parametrized by 0 and outputs a C-dimensional probability vector. We use to refer to matrix element-wise product.
2.1 VISION TRANSFORMERS AND SPARSE MOES
Vision Transformers. Throughout the paper, we choose the model f to be a vision Transformer (ViT) (Dosovitskiy et al., 2021). ViT is growing in popularity for vision, especially in transfer- learning settings where was shown to outperform convolutional networks while requiring fewer pre-training resources. ViT operates at the level of patches. An input image is split into equal-sized patches (e.g., 32 x 32, 16 x 16, or 14 x 14 pixels) whose resulting sequence is (linearly) emhedded and processed by Transformer (Vaswani et al., 2017). The operations in the Transformer then mostly consist of succession of multiheaded self-attention (MSA) and MLP layers. ViT is defined at different scales (Dosovitskiy et al., 2021): S(mall), B(ase), L(arge) and H(uge); see specifications Appendix For example, ViT-L/16 stands for a large VIT with patch size 16 x 16.
Sparse MoEs and V-MoFs. The main feature of sparsely-gated mixture-of-experts models (sparse MoEs) lies in the joint use of sparsity and conditional computation (Bengio et al., 2013). In those models, only activate small subset of the network parameters for given input, which allows the total number of parameters to grow while keeping the overall computational cost constant. The subparts of the network that are activated on a per-input fashion are known as experts.
Central to our study, Riquelme et al. (2021) recently extended ViT to sparse MoEs. Their extension, referred to V-MoF, follows the successful applications of sparse models in NI.P (Shazeer et al.,

---

Under review as conference paper ICLR 2022
2017). Riquelme et al. (2021) show that V-MoEs dominate their "dense" ViT counterparts on a variety of tasks for the same computational cost. In the specific case of V-MoEs, the experts are placed in the MLP layers of the Transformer, a design choice reminiscent of Lepikhin et al. (2021)
NLP. Given the input h E IR of such a layer, the output of a single MLP(h) is replaced by
E
(hE(h)= Σ
De (h) (L) with (gc(h)) = top (sofftsa
(1)
where the routing weights (oe (h))ez combine the outputs of the E different experts (MPP).-. To sparsely select the experts, tOpx sets all but the K largest weights to zero. The router param- eters W E R xx D are trained together with the rest of the network parameters. We call the layer defined by (1) an MoE layer. In practice, the weights {g. (h)) ==1 are obtained by a noisy version (wwffmaaa(xeee differentiability of tOPx when combined with auxiliary losses (Shazeer al., 2017). We use the shorthand gate x (z) = topx (softmaxx + re)) and take r = 1/E (Riquelme et al., 2021).
of the routing function topx wtt.ammaaWW.. + dε)) with e ~ No, I). which mitigates the non-
In this paper, we take the "last-n" setting of Riquelme et al. (2021) wherein only a few MoE layers are placed at the end of the Transformer (n 2 for the {S. B. L} scale and 5 for H). This setting retains most of the performance gains of V-MoEs while greatly reducing the training cost.
2.2 ENSEMBLES OF NEURAL NETWORKS
Ensembles. We build on the idea of ensembles, which is a known scheme to improve the perfor- mance of individual models (Hansen & Salamon, 1990: Geman et al., 1992: Krogh & Vedelsby, 1995; Opitz & Maclin, 1999; Dietterich, 2000; Lakshminarayanan et al., 2017). Formally. we as- sume a set of M model parameters θ {0m 1 We refer to A as the ensemble size. Pre- diction proceeds by computing M Zace f(z: 9). ie.. the average probability vector over the M models. To assess the diversity of the predictions in the ensemble, we will use the KL divergence (f (l. (x;; mm))) )) between the predictive distributions ff fxx;; fm) and f(x;; 0 aver- aged over the test input æ; and all pairs (m, m²) of ensemble members.
Batch ensembles. Ensembles differ in the way Ô is defined. Central to our study, batch ensem- bles (BE) (Wen et al., 2019) build the ensemble as a collection of submodels, with the parameters Om E θ sharing components. This mitigates the computational and memory cost of ensembling, enabling one to improve the performance of the original model at little extra cost. We focus the example of a single dense layer in f with parameters U E R RDx,, assuming no bias. BE defines M copies of parameters {Um]m= M so that Un. = U o (T) where U are parameters shared across ensemble members, and rm and 8m are separate 1)- and I--dimensional vectors for ensemble member m. Given an input, the BE produces applying all layers. Despite the simple rank-1 1 parametrization, BE leads to remarkable predictive performance and robustness (Wen al., 2019). Notably, the efficiency of BE relies on tiling the inputs to simultaneously predict with the M ensemble members, an insight that we also exploit.
outputs, and the M outputs are averaged after
2.3 UPSTREAM PRE-TRAINING AND DOWNSTREAM PINE-TUNING
Large-scale Transformers pre-trained on upstream tasks were shown to have strong performance when fine-tuned on smaller downstream tasks, actoss a variety of domains (Devlin et al., 2018; Dosovitskiy et al., 2021: Radford et al., 2021). We follow this paradigm and focus on the fine- tuning of models pre- preitriied on JFT-300M (Sun et al., 2017), similar to Riquelme et al. (2021). We will thus assume the availability of already pre-trained ViT and V-MoE model checkpoints. Our assumption relies on the growing popularity of transfer learning, e.g. Kolesnikov et al. (2020), and the increasing accessibility of pre-trained models in repositories such as www wensorfnoofoooo fow. org/ / hub www. www.prytccccc orggorgghhoo hub.. The fine-tuning of all the approaches we study here, including extensions of VIT and V-MoE, will be either directly compatible with those checkpoints or require only mild adjustments. e.g., reshaping or introducing new wwwnstream-speciine parameters (see Appendix C). Also, unless otherwise mentioned, the performance we report will always be downstream, c.g., for ImageNet (Deng et al., 2009) or Cifarl (Krizhevsky, 2009). In all our comparisons, we will use the downstream training floating point operations per second (FLOPs), or GFLOPs (i.e., 10 xF)OPs), to quantify the computational cost of the different methods.

---

Under review conference paper ICLR 2022
Table I: Overview of key properties of sparse MoEs and ensembles. dense is base model upon which we add the sparse MoE or ensemble logic, e.g., a VIT model in this paper.
PREDICTIONS
COMBINATIONS
CONDITIONAL COMPUTATION
COST
Sparse MoEs Ensembles
Single Multiple
At activation level At prediction level
Yes, adaptively per-input
dense dense
No, static
3 SPARSE MOES MEET ENSEMBLES
As illustrated in Table 1, sparse MoEs and ensembles have different properties. For instance, en- sembles typically do not use conditional computation and just statically combine members at the prediction level. This contrasts with sparse MoEs where the different experts are combined at inter- nal activation levels while enjoying per-input adaptivity through the routing logic; (1). In terms of cost, sparse MoEs are usually designed to match the cost of their dense counterparts whereas ensembles, in their simplest forms, will typically lead to substantial overhead. In this section, we study the extent which these properties are complementary and may benefit from each other. In Section 5, we further evaluate this complementarity on tasks where either sparse MoEs or ensembles are known to perform well, e.g., fho-shot and distriibbbtinn (OOD) evaluations, respectively. More details about the experiments in this section be found in Appendix B.6.
3.1 STATIC VERSUS ADAPTIVE COMBINATION
42
2
14
50
(a) Log- Likeliikl (LL)
(b) (downstream) GFLOPs
(c) log GH (1LOP5(1.))
M.
Figure 2: The effect of increasing static (M) and adaptive (K) ensembling. ImageNet performance
for ViT-S/32 models. Yellow indicates better performance; purple indicates worse performance.
We first focus on the interplay between the (a) static combina- tion in ensembles and (b) the adaptive combination of experts in sparse MoEs. To this end, we study performance of down- stream deep ensembles (i.c., with all ensemble members having the same upstream checkpoint) formed by M independent V- MoEs with E experts per MoE. layer and a sparsity level K (the larger K, the more selected experts). The parameter M con- trols the static combination, while K and É impact the adap- tive combination of experts in each sparse MoE model. port in Figure 2 the ImageNet performance and compute cost for ensembles with varying choices of K and M, while keep- ing 32 fixed. We focus on K rather than explore adaptive computation, as we find that the performance changes for E plateau relatively quickly (see Figure 8 in the Ap- pendix). Also, by fixing Æ 32, we match more closely the experimental setup of Riquelme et al. (2021). The architecture of the V-MoE is ViT-S/32; details in Appendix B.6.1. We make the following observations:
NLL. NLL. (lower is hetter)
0.88
0.6
re-
0.4
the axis
downstream leg(GFLOPg) (lower is hetter)
Figure 3: VIT (o) and V-MoE (4) ensembles of size M (1,
(denoted by markers of increas- ing size) for S/32 (+), B/32 L/32 (+), L/16 (+), and H/14 on ImageNet.
Cumulative effect. the absence of ensembles (M = 1), and given fixed number of experts, Riquelme et al, (2021) already reported increase in performance K gets larger. Interestingly, we observe that for each value

---

Under review conference paper ICLR 2022
Table 2: Feature-level vs. prediction-level ensembling. ImageNet performance of V-MoE and a naive multi-head variant (means + standard errors over 5 replications). All models have a ViT-B/32 architecture. For the multi-head variant the last MoE layer modified as (2).
K
NLL +
ERROR
ECE 4
KL
V-MoE
2
0.638
1.000
16.76
000
0.033
1000
Naive Multi-head
2
0.633
1000
16.85
nor
0.025
1,000
0.033 0.00
V-MoF
4
0.636
16.70
0.034
Naive Multi-head
4
0.638
0100
17.23
0.020
0.012
of K, it is also beneficial to increase the ensemble size M. In other words, the static combination of ensembles is beneficial when applied sparse MoEs. This observation is perhaps surprising since adaptive combination may already encapsulate the effect of static combination. Figure 3, and Appendix H.1. show that the combination of static and adaptive ensembling is beneficial to NLL for range of VIT families. We also see that the benefits of static ensembling are similar for V-MoE and ViT (which does not have any adaptive ensembling).
Taking FLOPs into account. Without any computational constraints, the previous observation would favor approaches with the largest values of K and M. However, different values of (K, M) lead different computational costs, as measured here by FLOPs, with (K, MM = (1, 1) being the cheapest. Figure 2b shows, as expected, that the number of FLOPs grows more quickly along the M axis than along the K' axis. To capture the various trade-offs play, in Figure 2c report LLcK -LL
(at0PPxxxxx-GGLLPPpp.po GFLOPs..,) when going from the logarithm of the normalized gains in log likelihood GFLÖPs (r)
(K, M) 1I. to other choices of (K, M). Interestingly, it appears more advantageous to first grow K. i... the adaptive combination, before growing M.
3.2 FEATURE-LBVEL VERSUS PREDICTION- ENSEMBLING
As highlighted Table I, an ensemble of size M outputs M predictions for a given input (there- after, averaged) while sparse MoEs only produce a single prediction. We study the impact of this differentiating property. To this end, propose simple variant of sparse MoEs wherein the last MoF. layer of the form (1) is replaced by
mult t ieaeed heaadMoE((A (h) (g. MLPo(h))a. E RK*Q (c)( = gatex (Wh), (2)
where we have assumed MLP,(h) E R., Instead of summing the expert outputs like (1), we stack the K selected expert contributions (as reminder, gateex zeroes out the E K smallest weights). Keeping track of those K contributions makes it possible to generate K predictions per input as in the classifier of Figure thus capturing model uncertainty around the true prediction.
Table 2 compares the ImageNet performance-negattne log likelihood (NLL), classification error and expected calibration error (ECE) (Guo et al.. 2017)-of this naive multi-head method with the standard V-MoE. For K 2, the multi-head method provides small but statistically significant gains in NLL and ECE. However, its classification error is worse. On the other hand, for = 4. both the NLL and classification eгror for multi-head are worse than V-MoE, despite even larger improvement in ECE. In fact, the multi-head for K 4 performs worse in terms of NLL, classification error, and diversity than for = 2. Note that the KL diversity metric indicates that the multi-head variant is unable to provide diverse predictions (c.g., a downstream ensemble of two V- MoEs with K 1 provides much higher diversity of 0.073, see Table 10). Following Havasi et al. (2020); Soflaei et al. (2020), a possible fix to this problem would be to consider multi-head and multi-input approach, but we show in Appendix F that this strategy is not effective in our context.
Thus, while prediction level ensembling can be beneficial for uncertainty culibration of MoEs, a different strategy is required such that the error is not worse. We propose a better approach next.
4 PARTITIONED BATCH ENSEMBLE
Equipped with the insights from Section 3, we describe partitioned batch ensemble (pBE), with the goal of keeping the strengths of both sparse Moïs and ensembles. Conceptually, we can view pBE

---

Under review conference paper ICLR 2022
as jointly learning an ensemble of smaller sparse MoEs, where all the layers that do not contain experts are shared across the members, e.g., the self-attention layers. As its name indicates, pBE is inspired by batch ensemble in that (a) ensemble members have shared parameters-hent the sharing tailored sparse MoEs and (b) we reuse the idea of tiled representations.
4.1 THE ARCHITECTURE
There are two main components in PBE:
Disjoint subsets of experts as ensemble members. We change the structure of (1) by partitioning the of E experts into M sets of E/M experts (we assume that E is multiple of M). We denote this partition by UM E for example & {1. 3, and &g 5,66,, 6) for The M sets of E/M experts play the role of the M ensemble members. Intuitively, the ensemble members have separate parameters for independent predictions, while efficiently sharing parameters among all non-expert layers.
and M
Instead of having a single routing function gate, (W.) like in (I), we apply separate routing functions {gatex (Wm, M to each member of the partition. Note that this does not affect the total number of parameters since W has E rows while each W., has E/M rows. A similar partitioning of the experts was proposed in Yang et al. (2021) but not exploited to create different ensemble members, in particular not in conjunction with tiled representations, which we show to be required get performance gains (see comparison in Section 4.2.
Tiled representation. To jointly handle the predictions the M ensemble members, we tile the inputs by a factor M, as proposed in Wen et al. (2019). Tiling naturally fits into the formalism of sparse MoEs, as illustrated by the connection we draw between BE and sparse MoEs (Appendix I). This enables a simple implementation of pBE on top of existing one of sparse MoEs.
Because of the tiling, a given image patch has M different representations that, when entering an MoE layer, are each routed within their respective parts of the partition UM 1 Formally, con- sider some tiled inputs H E R BBMM XD where B refers to the batch size and him E R is the representation of the i-th input for the m-th member. The routing logic in pBE can written as
ORE-Moo((hi.))
Σ cEEm
Je.m(hinn -LPP (him)) (ge,m Jeee...--...-- gateg Wmhh.)) (3)
where the routing weights are now parametrized by W.. E R(F/M)XD see Figure 1. To echo the observations from Section 3, we can first see that PBE brings together the static and adaptive combination of ensembles and sparse MoEs, which we found to be complementary. However, we have seen that static ensembling comes at the cost of a large increase in FLOPs, thus we opt for an efficient ensembling approach. Second, we "split" the MoE layers along the axis of the experts, i.e., from E experts M times E/M experts. We do since we observed that the performance of sparse MoEs tends to plateau quickly as the number of experts grows. Finally, pBE retains the important property of ensembles to output multiple predictions per input, which we also saw to be beneficial for uncertainty calibration.
generic implementation. we tile batch of B inputs X E R. P by factor M obtain the tiled inputs X = [X X] E RMM (M and the model processes f( ((xiite; 0). Since tiling PBE has an effect only from the first MoE layer onwards, we postpone the tiling operation that stage, thus saving all prior computations in non MoE-layers that would have been redundant otherwise. For example, for I/16 and K M 2, we can save about 47% of the FL.OPs. We apply the same optimization for all the efficient ensembles methods we compare in Appendix F. We provide further implementation details of pBE in Appendix D.
4.2 ABLATION STUDIES: PARTITIONING AND TILING
Our method introduces two changes to V-MoEs: (a) the partitioning of the experts and (b) the tillng of the representations. In this section, we assess the separate impact of each of those changes and show that it is indeed their combination that explains the performance gains. We summarize the results of the ablation in Table 3 where we show the ImageNet performance of the different variants of V-MoE. All models have ViT-B/32 base architecture and K- M 2.

---

Under review conference paper ICLR 2022
Table 3: ImageNet performance (means + standard errors over 8 replications) of pBE and two abla- tions, disabling either the tiling the expert partitioning. All models have a ViT-B/32 architecture. The level of noise in gatex is denoted by a.
N
ERROR
ECE L
KL f
pBR
0.612 2000
16.49 ou
0.013
nots
0.198
0:0000
Only tiling (a x 1)
0.637 gome
16.74
a cm
0.028 w00
0.000
11:000
Only tiling 2)
0.638 2000
16.72 = om
0.033
0001
0.001
=: 0.000
Only tiling 4)
0.638
16.74
0.033
0.002
Only partitioning 0.640 0.00
16.72
0.034
4.2.1 PARTITIONING WITHOUT TILING
We first compare PBE with variant of V-MoE where we only partition the of experts ("Only partitioning"), that variant, each input h, E RD (note the dropping of the index m due to the absence of tiling) can select K' experts in each part of the partition U E resulting in total of K x M selected experts per input. Formally, (3) becomes > Eeer, (hs) MLPP ML.,hh;) The expert prototyping of Yang et al. (2021) leads similar formulation. Äs shown in Table 3, "Only partitioning" is not competitive with pBE across all metrics. We do not report the KL since without tiling, "Only partitioning" does not output multiple predictions per input.
4.2.2 TILING WITHOUT PARTITIONING
We now compare PBЕ with the variant where only the tiling is enabled ("Only tiling"). In that case, we have tiled inputs H E B BxMxD applied to the standard formulation of (1). Compared with (3), there is no mechanism enforce the M representations of the i-th input across the ensemble mem- bers, i.c., (Mo,(h, M to be different. Indeed, without partitioning, cach ham could select K identical experts. As a result, we expect "Only tiling" to output M similar predictions across ensemble members. We capture this intuition Table 3 where we observe that the KI. for "Only tiling" is orders of magnitude smaller than for pBE.
To mitigate this effect, we also tried to increase the level of noise in gateş (by factor {2, 4/), cause the expert assignments differ across (hr. IM m While we do see an increase in KL, "Only tiling" still performs worse than PBE across all metrics. Interestingly, we can interpret "Only tiling" as an approximation, via M samples, of the marginalization Eer....,, 0)] with respect to the noise (et)f_, in the MoE layers of [1; 0) (further assuming the capacity constraints of the experts, as described in Riquelme et al. (2021), does not bias the M samples).
5 EVALUATION
We now benchmark PBE against V-MoE. As baseline we also include results for downstream ensembles of V-MoE and VIT. These ensembles offer natural baseline against pBE they also use a single upstream checkpoint, are easy to implement, and provide consistent improvements upon V-MoE. In Appendix G. we compare with upstream ensembles that require multiple upstream checkpoints (Mustafa al., 2020). In Appendix F, compare with other efficient ensembling ap- proaches: MIMO (Havasi et al., 2020), BE (Wen et al., 2019), and MC Dropout (Gal & Ghahramani, 2016). All results correspond to the average over (for {S. B. L.} single models) or 5 (for H single models and all up/downstream ensembles) replications. In Appendix H we provide standard errors as well as results for additional datasets and metrics. Following Riquelme et al. (2021), we com- vs. compute cost trade-offs for each method across range of VIT
pare the predictive-performmne familics. In the results below, pBE uses (K, .) (1, (1, and V-MoE uses K 1. Experimental details, including about our upstream training. downstream fine-tuning, hyperparameter sweeps and our (linear) few-shot evaluation can be found in Appendix B. Our main findings are as follows:
(a) V-MoE versus VIT. Predictive performance and robustness. Ensembles help V-MoE just as much as VIT. Ensembles were expected benefit ViT models. However, Figure 3 and Figure suggest that ensembling provides similar gains for V-MoE models in terms of both few-shot performance and NLL. We believe this has not been observed before. Moreover, downstream ensemble with four H/14 V-MoEis leads to 88.8% accuracy ImageNet

---

Under review conference paper ICLR 2022
Ims NeN
ImagoNet
Moo Actrec Datacets
Men Actis Datzouls
She linee (Jower it better)
NLT [1..0orr 14 batee)
6.7
6.6
9.40
20
0.5
15
3.6
downstrrem bogtH better)
PBB
V.MaE
VIT
S:32
B:32
L822
L/16
Hi14
Model
Modads
Modds
Figure ImageNet NLL (left, center left) and mean 10-shot enorr across datasets (center right, right). We provide zoomed-in in plots of the highlighted areas. The dashed lines show Pareto frontiers.
ImageNet
VI CIFARLC0
CIFARIO va. SVAEN
CIFARIO vs. Place3365
0:34
FPR895 (lover bchaatr)
BCT (boaurr is better)
0:03
0.03
0.10
0.02
0.01
0M6
0:01
donustream leg((FLOPs) (Jower (lower better)
pbB
V-Mol
Vii
8/32
1832
L52
L/16
H/14
Model
Mocals
Modals
Figure Quality of uncertainty estimates. ImageNet ECE (left), near (center left) and far (cen- ter right, right) OOD detection, measured by false positive rate at 95% precision (Fort et al., 2021). These are metries for which ensembles known to perform well whereas, to the best of our knowl- edge, the performance of V-MoB not been evaluated. The dashed lines represent Pareto frontiers.
(even reaching an impressive 89.3% for an upstream ensemble, see Table 10).
ViT consistently provides better ECE than V-MoE. Surprisingly. despite V-MoE tending have better NLL than ViT (Figure 3), their ECE is worse (Figure 5).
ECE is not consistent for different ViT/V-MoE families. We see the ECE, unlike other metrics increase the ViT family size (Figure 5). V-MoE outperforms VIT in OOD detection. With I/32 being the only exception, V-MoEE out- performs ViT on a range of OOD detection tasks (Figure 5).
presented in this work, provides consistent trends
For smaller ViT families, V-MoE outperforms ViT in the presence of distribution shift. In contrast to OOD detection results, Figure shows for smaller ViT families V-MoE improves on the performance of VIT, however, as the ViT family becomes larger, this trend reverses.
(b) Partitioned Batch Ensembles. Predictive performance and robusiness.
pBE improves classification performance. As shown Figure pBE is either very near to the Pareto frontiers for NLL and 10-shot classification error, despite the fact that these are metrics for which ensembles and V-MoE, respectively, are known to perform well. Furthermore, Figures 14 and 15 show that even starker conclusions hold on 10f000
PBE performs best at the largest scale. The difference predictive performance between pBE and V-MoE or ensembles thereof increases as the ViT family becomes larger (Figures 4, 5 and 6). See Appendix E for further motivation of this point.
pBE tends to be Pareto efficient in the presence distribution shift. Figure shows that pBE is more robust to distribution shift for larger ViT families, despite the opposite being true for V-MoE. PBE improves ECE over VIT and V-Mol.. Despite V-MoE providing poor ICCE, pBE does not suffer from this limitation (Figure 5). Furthermore, for most ViT families, PBE also provides better ECE than V-MoE ensembles.
PBE does not provide consistent OOD detection performance. Firstly, Figure 5 shows that for small ViT families, PBE performs worse than V-MoE and (even ViT in some cases). Nevertheless, as above, the relative performance improves for larger VIT families such that pBE becomes Pareto efficient for two dataset pairs. Secondly, pBE seems to perform better the more difficult

---

Under review conference paper ICLR 2022
ImageNeetCC
Imagg (avenugd
ImageNatAA
ImageNa-V2 V2
2.1
NLT (1//rrr is batiel)
2.0
1.5
1.00
0.95
1.0
1.a
0.33
3.0
downstream Lgg(G better)
PBB
V.MaE
VIT
S:32
B/32
L-22
L/16
H114
Midel
MoCads
Moddi
Figure 6: NLL in the presence distribution shift for models trained on ImageNet. For ImageNet- C, we provide a zoomed-in in plot of the highlighted area. The dashed lines represent Pareto frontiers. We provide results for additional distribution shift datasets and metrics in Appendix H.
OOD detection task (Cifarl0 vs. Cifar100). These results, although sometimes subtle, are consistent across OOD detection metrics and dataset pairs, as shown in Appendix H.
RELATED WORK
Mixture of Experts. MoEs (Jacobs et al., 1991; Jordan & Jacobs, 1994; Chen el al., 1999; Yuksel al., 2012; Eigen al., 2014) combine the outputs of different submodels, experts, in input- dependent way. Sparse MoEs only select few experts per input, enabling to greatly scale models while keeping the prediction time constant. Sparse MoEs have been used to build large language models (Shazeer et al., 2017: Lepikhin et al., 2021; Fedus et al., 2021). Recently, sparse MoEs have been also successfully applied to vision problems (Riquelme et al., 2021; Yang et al., 2021; Lou et al., 2021; Xue et al., 2021). Our work builds on the V-MoE architecture proposed by Riquelme al. (2021), 1), which is based on the vision Transformer (ViT) (Dosovitskiy
al., 202 While previous work studied ViT's calibration and robustness (Minderer et al., 2021; Fort et al., 2021; Paul & Chen, 2021; Mao et al., 1), we are the first to study the robustness of V-MoE models.
Ensembles. Ensemble methods combine several different models to improve generalization and un- certainty estimation. In their simplest form, they can be inefficient because they consist of multiple models themselves potentially expensive. To reduce test time, Xie et al. (2013) and Hinton et al. (2015) respectively use compression and distillation mechanisms. To reduce training time, ensem- bles can be constructed with cyclical learning-rate schedules to snapshot models along the training trajectory (Huang et al., 2017; Zhang et al., 2019). Our work builds on batch ensemble (Wen et al., 2019) where a single model encapsulates al. (2015); Havasi et al. (2020); Antorán al. (2020); Dusenberry et al. (2020a); Rame al. (2021), Wenzel et al. (2020) extended BE to combine models with different hyperparameters.
ensemble of networks, strategy also explored by Lee
CONCLUSIONS AND FUTURE WORK
Our study of the interplay between sparse MoEs and ensembles has shown that these two classes models are symbiotic. Partitioned batch ensemble exemplifies those mutual benefits-ae as illustrated by its accuracy, log-likelihood, few-shot learning, robustness, and uncertainty calibration improve- ments over several challenging baselines a range of benchmarks. While our study has focused on downstream fine-tuned models, we believe that an extension to the upstream case would also result in fruitful investigation. Similarly, although have focused computer vision, our approach should be readily applicable to the modelling of text, where sparse MoEs have been shown to be remarkably effective. With the growing prevalence of sparse MoEs in NLP (Patterson et al., 2021). the questions of understanding and improving the robustness and reliability of such models become increasingly important. Furthermore, the computational scale at which those models operate make those questions even more challenging to tackle. We believe that our study, and approaches such as pBE, make steps in those directions.

---

Under review conference paper ICLR 2022
Ethics Statement. Our research lies the intersection of two topics where hope that our work can make positive contributions.
First, following the conclusions of Patterson et al. (2021), we develop an approach based on sparse MoEs that were shown reduce the environmental footprint of standard "dense" models. Second, for any decision-making process, it critical to ML systems. In particular, this desirable property has growing importance within a context where those systems being widely deployed in safety-critical fields such as self-driving cars and medical diagnosis. think that approaches such as pBE can help make progress in this area.
to reliably trust. the uncertainty output by
Reproducibility Statement. We are fully aware that (i) relying on the proprietary JFT-300M dataset for our upstream models, as well as (ii) not having already open-sourced code are two obsta- cles for reproducibility. We are focusing together with the release checkpoints pre-trained on ImageNet-21k, We are actively working to release those with the camera-ready version of the paper.
lightweight open-sourced version of V-MoE and pBE
We would like stress the fact that have provided many details as possible about the perimental settings (see Appendix B)
hyperparameter sweeps. Furthermore, we aimed provide well-considered and fair experimental setups for all of the baselines used in this work. For instance, to illustrate the fairness of our approach, experimental design choices tend to make the baselines we compare to more competitive; see Table 6. Moreover, we performed evaluation our models using the open-sourced library
(Djolonga et al., 2020).
Finally, we have also reported results that could be regarded as "negative" with the hope inform other researchers about those findings. For instance, Appendix F.3, systematically described in detail the failures we encountered while trying to apply MIMO to VIT and V-MoE. Similarly, we have transparently reported- in the core paper-ree results of pBE for OOD detection where our approach seems perform than the baselines in several settings.
10

---

Under review conference paper ICLR 2022
REFERENCES
Javier Antorán, James Urquhart Allingham, and José Miguel Hernández-Lobat Depth uncertainty in neural networks. In Advances Neural Information Processing Systems, 2020.
Monika Bansal. Munish Kumar, Monika Sachdeva, and Ajay Mittal. Transfer learning for image classification using veg19: Caltech-101 image data set. Journal of Ambient Intelligence and Humanized Computing, pp. 1-12, 2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons conditional computation. arxiv preprint arxiv: 1308.3333 2013.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnee and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. ctwmco///////o. http://ooglleooooooo jax.
Tom B Brown, Bemjamin Mann, Nick Ryder, Melamie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arxiv preprint 200iv:0005.11165. 2020.
Ke Chen, Lei Xu, and Huisheng Chi. Improved learning algorithms for mixture of experts in multi- class classification. Neural Networks, 1999.
M. Cimpoi, S. Maji. L. Kokkinos. S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2014.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arxiv preprint arxiv:1810.04855, 2018.
Thomas G Dietterich. Ensemble methods machine learning. International workshop on multi- ple classifier systems, pp. 1-15. Springer, 2000.
Josip Djolonga. Frances Hubis, Matthias Minderer, Zachary Nado, Jeremy Nixon, Rob Romijnders, Dustin Tran, and Mario Lucic. Robustness Metrics, 2020. URL https: /tt///gtttt github..h.... com/
google-resea wwwph//ooosttttsssetttt.. org
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image worth 16x 16 words: Transformers for image recognition scale. In ICLR, 2021.
Michael W Dusenberry, Ghassen Jerfel, Yeming Wen, Yi-an Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan, and Dustin Tran. Efficient and scalable bayesian neural nets with rank- 1 factors. In International conference on machine learning, 2020a.
Michael W Dusenherry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M Dai. Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference. and Learning. pp. 204-213, 2020b.
David Eigen, Marc'Aurelic Aurelio Ranzato, and Ilya Sutskever. Learning factored representations in deep mixture of experts. In ICLR (Workshop Poster), 2014.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Sealing to trillion parameter models with simple and efficient sparsity. arxiv preprint arxiv:2101.0396 2021.
Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of ou -f-sstribbuttionn delection. arxiv, 2021.

---

Under review conference paper ICLR 2022
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In International conference on machine learning, pp. 1050-1059, 2016.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 1(1): 1992.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks, In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Fredrik K Gustafsson, Martin Danelljan, and Thomas Schon. Evaluating scalable bayesian deep learning methods for robust computer vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition Workshops, pp. 318-319, 2020.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions pattern analysis and machine intelligence, 12000) 1990.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer, 2017.
Marton Havasi, Rodolphe Jenatton, Stanislav Fort, Jeremiah Zhe I Liu, Jasper Snoek, Balaji Lakshmi- narayaman, Andrew Mingbo Dai, and Dustin Tran. Training independent subnetworks for robust prediction. In ICLR, 2020.
Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustmess common cor- ruptions and perturhations. arXiv preprint 1903. 2019.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. arXiv preprint 0eXXv:9907.071770 2019.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. NIPS Deep Learning and Representation Learning Workshop. 2015.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E Hopcroft, and Kilian Q Weinherger. Snapshot ensembles: Train 1, get m for free. arxiv preprint
2017.
R. A. Jacobs, M. Jordan, Nowlan, and G. E. Hinton. Adaptive mixtures of local experts. Neural Computation, 1991.
Michael I. Jordan and Robert A. Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural Computation, 1994.
Jakob Nikolas Kather, Cleo-Aron Weis, Francesco Bianconi, Susanne M Melchers, Lothar R Schad, Timo Gaiser, Alexander Marx, and Frank Gerrit Zöllner. Multi-class texture analysis in colorectal cancer histology. Scientific reports, 6(1):1-11, 2016.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, pp. 491- 507. Springer, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei. 3d object representations for fine tineggrained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013,
Alex Krizhevsky, Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances in neural information processing systems, pp. 231-238. 1995.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
12

---

Under review conference paper ICLR 2022
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arxiv preprimt 1511v:/511.003111 2015.
Dmitty Lepikhin, HyoukJoong Lee. Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. In ICI.R, 2021.
Jesse Levinson, Jake Askeland, Jan Becker, Jennifer Dolson, David Held, Soeren Kammel, J Zico Kolter, Dirk Langer, Oliver Pink, Vaughan Pratt, et al. Towards fully autonomous driving: Sys- tems and algorithms. In 2011 IEEE Intelligent Vehicles Symposium (IV), pp. 163-168, IEEE, 2011.
Yuxuan Lou, Fuzhao Xue, Zangwei Zheng, and Yang You. Sparse-mlp: A fully-mip architecture with conditional computation. arxiv, 2021.
Xiaofeng Mao, Gege Qi, Yuefeng Chen, Xiaodan Li, Ranjie Duan, Shaokai Ye, Yuan He, and Hui Xue. Towards robust vision transformer, 2021.
Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tram, and Mario Lucic. Revisiting the calibration of modern neural networks. arxiv, 2021.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Deep ensembles for low-data transfer learning, arXiv preprint arXiv:2010. 0,06666, 2020.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Maria-Blena Nilshack and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp.
JEEE, 2008
David Opitz and Richard Maclin. Popular ensemble methods: An empirical study, Journal of artificial intelligence research, 11:169-198, 1999.
Yaniv Ovadia, Emily Fertig, Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayaman, and Jasper Snock. Can you trust your model's 's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, 2019.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEEE, 2012.
David Patterson, Joseph Gonzalez, Quoe Le, Chen Liang, Lluis-Miquel Munguia, Daniel Rothchild, David So, Maud Texier, and Jeff Dean. Carbon emissions and large neural network training. arxiv preprini arxiv:2104. 10350, 2021.
Sayak Paul and Pin- Yu Chen. Vision transformers are robust learners. arxiv, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. arxiv preprint arxiv:2 2021.
Alexandre Rame, Remy Sun, and Matthieu Cord. Mixmo: Mixing multiple inputs for multiple outputs via deep subnetworks. arXiv preprint arXiv:2103.061:: 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400, 2019.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa. Maxim Neumann, Rodolphe Jenatton, André Su- sano Pinto, Daniel Keysers. and Neil Houlsby. Scaling vision with sparse mixture of experts. arxiv preprint arxiv:210..05000 1at::2106005000 2021.
13

---

Under review conference paper ICLR 2022
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, Jeff Dean. Outrageously large neural networks: The sparsely-gated cuxture-of-expert layer. In ICLR, 2017.
Masoumeh Soflaei, Hongyu Guo, Ali Al-Bashabsheh, Yongyi Mao, and Richong Zhang. Aggregated learning: A vector-quamiiztionn approach to learning neural network classifiers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5810-5817, 2020.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. arxiv preprint arxiv: 1906.02243, 2019.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017. URL https: ptrpp://procseeding neurips. ec/ wcppappee/211///1..... pdf.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach efficient ensemble and lifelong learning. In ICLR, 2019.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. In Neural Information Processing Systems), 2020,
Jingjing Xie, Bing Xu, and Chuang Zhang. Horizontal and vertical ensemble with deep representa- tion for classification. arxiv, 2013.
Fuzhao Xue, Ziji Shi, Futao Wei, Yuxuan Lou, Yong Liu, and Yang You. Go wider instead of deeper. arxiv, 2021.
An Yang, Junyang Lin, Rui Men, Chang Zhou, Le Jiang, Xianyan Jia, Ang Wang, Jie Zhang, Jiamang Wang, Yong Li, et al. Exploring sparse expert models and beyond. arxiv preprint arxiv:2105. 15082, 2021.
Yi Yang and Shawn Newsam. Bag-of- la--of-visullwordd. and spatial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems, pp. 270-279, 2010.
Seniha Esen Yuksel, Joseph Wilson, and Paul D Gader. Twenty years of mixture of experts. IEEE 2012.
Iransactions neural networks and learning systems,
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv: 106.00000 (46.04560, 2021.
Ruqi Zhang, Chumyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In ICLR, 2019.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions Patfern Analysis and Machine Intelligence, 2017.
14

---

Under review conference paper ICLR 2022
HIDDEN DIMENSION
MLP DIMENSION
LAYERS
Small
512
2048
Base
768
3072
12
Large
1024
4096
24
Huge
1280
5144
32
Table 4: Specifications of ViT-S, ViT-B, VIT-L and ViT-H.
A VrT MODEL SPECIFICATIONS
Following Dosovitskiy al. (2021), we recall the specifications of the ViT models of different scales Table 4.
EXPERIMENT SETTINGS
B.1 UPSTREAM SETTING
For all upstream experiments, scrupulously follow the setting described in Riquelme al. (2021), see their Section B.2 their appendix. For completeness, we just recall that S/32 models are trained for 5 epochs while (16 32} and L/32 models are trained for 7 epochs. For L/16 models, both 7 and 14 epochs be considered (Dosovitskiy et al., 2021; Riquelme al., 2021); opted for epochs given the breadth our experiments. Finally, the H/14 model are trained
epochs.
In particular, the models are all trained on JFT-300M (Sun et al., 2017). This dataset contains about 000 validation images. The labels have hierarchical structure, with total of
305M training and 50000 18 291 classes, leading on average 1.89 labels per image.
B.2 DOWNSTREAM SETTING
During fine-tuning, there number common design choices we apply. particular:
Image resolution: 384.
Clipping gradient norm at: 10.0.
Optimizer: SGD with momentum (using half- nalf-preciison, 0.9).
Batch size: 512.
For V-MoE models, we finetune with capacity ratio C = 1.5 and evaluate with C 8.
We use the following train/validation split partitions depending on the dataset:
DATASET
TRAIN DATASET FRACTION
VALIDATION DATASET FRACTION
ImageNet
99%
1%
CIFARI0
98%
2%
CIFAR100
98%
2%
Oxford-IIIT Pets
90%
10%
Oxford Flowers-102
90%
10%
All those design choices follow from Riquelme al. (2021) and Dosovitskiy al. (2021).
B.3 HYPERPARAMETER SWEEP FOR FINE-TUNING
all our fine-tuning experiments, use the sweep of hyperparameters described in Table S. We the recommendations from Dosovitskiy et al. (2021) and Riquelme et al. (2021), further considering several factors (0.5, 1.0, 1.3, 2.0} to sweep over different numbers of steps. Riquelme et al. (2021)
half schedule (with the factor 0.5) while Dosovitskiy et al. (2021) take the factor 1.0.

---

Under review conference paper ICLR 2022
Table 5: Hyperparameter values for fine-tuning tne-tuning on different datasets. Compared with Dosovitskiy al. (2021) and Riquelme et al. (2021), further consider several factors {0.5, 1.0, 1.5, 2.0} sweep over different numbers of steps.
DATASET
STEPS
BASE LR
EXPERT DROPOUT
ImageNet
20000 X [0.5. 1.0, 1.5, 2.0)
(0.0024. 0.003, 0.01, 0.03)
0.1
CIFARTO
5000 X [0.5. 1.0,
1.5, 2.0)
(0.001, 0.003, 0.01, 0103)
0.1
CIFAR100
5000 X [0.5, 1.0, 1.5. 2.0
[0.001, 0.003, 0.01, 0.03
0.1
Oxford-IITT Pets
500 [0.5, 1.0,
1.5. 2.0
(0.001. 0.003, 0.01, 0.03)
0.1
Oxford Flowers- 102
500 [0.5, 1.0,
1.5, 2.03
0.000, 0.01, 0.03)
0.1
Table 6: Impact using the enlarged sweep of hyperparameters described Table 5. We typically improve the results reported in Riquelme et al. (2021), therefore strengthening the baselines we compare to. The table displays means and standard errors over 8 replications, except for H/14 that has 4 replications. L/16": For L/16, we consider the setting where the upstream models are trained with 7 epochs, as opposed to 14 epochs in Riquelme et al. (2021), hence the slightly worse accuracy reported in this paper.
MODEL SIZE
MODEL NAME
ACCURACY (THIS PAPER)
ACCURACY (Riquelme et al., 2021)
ViT V-MoE (K=2)
76.31
73.73
S/32
78.91
:
77.10
81.35 + 0.08
80.73
B/32
V-MoE (K=2)
83.24 + 0.05
82.60
ViT V-MoE (K=2)
84.62 + 0.05
84.37
I.32
84,95 1 0.00
85.04
ViT V-MoE (K=2) VIT
84.30
84.15
B/16
85.40
85.39
86.63 :
87.12
L/16"
V-MoE (K=2) VIT
87.12 # 0.04
87.54
88.01 1 0.00
88.08
H/14
V-MoE (K=2)
88.11 : 0.113
88.23
We show in Table the impact of this enlarged sweep of hyperparameters in the light of the results reported in Riquelme et al. (2021). We notably tend to improve the performance of ViT and V-MoE (especially for smaller models), which thus makes the baselines we compare to more competitive.
B.4 DETAILS ABOUT THE (LINEAR) FEW-SHOT EVALUATION
We follow the evaluation methodology proposed by Dosovitskiy et al. (2021): Riquelme et al. (2021) which we recall for completeness. Let rewrite our model f with parameters 0 {Q, 0')
fa: 0) sof tootmaaiQ@el 0'))
where Q e CC88 corresponds the parameters of the last layer of with the S-dimensional representation p(r;;)) R
In linear few-shot evaluation, we construct linear classifier to predict the target labels (encoded as one-hot vectors) from the S-dimensional feature vectors induced by o(:; 0'): see Chapter in Hastie et al. (2017) for more background about this type linear classifiers. This evaluation protocol makes it possible to evaluate the quality of the representations & learned by f.
While Dosovitskiy et al. (2021); Riquelme al. (2021) essentially focus the quality of the rep- resentations learned upstream on JFT by computing the (linear) few-shot accuracy on ImageNet, we are interested in the representations after fine-tuning on ImageNet. As a result, we consider a collection of 8 few-shot datasets (that does not contain ImageNet):
Caltech-UCSD Birds 200 (Wah al., 201 with 200 classes,
Caltech 101 (Bansal et al., 2021) with 101 classes,
Cars 196 (Krause al., 2013) with 196 classes,
16

---

Under review conference paper ICLR 2022
C0 (Krizhevsky, 2009) with 100 classes,
Colorectal histology (Kather et al., 2016) with 8 classes,
Describable Textures Dataset (Cimpoi et al., 2014) with 47 classes,
Oxford-IIIT pet (Parkhi et al., 2012) with 37 classes and
UC Merced (Yang Newsam, 2010) with classes.
In the experiments, we compute the few-shot accuracy for each of the above datasets and we report the averaged accuracy over the datasets, for various number of shots in {1, 10, 25). As commonly defined in few-shot learning, we understand by s shots a setting wherein we have access s training images per class label in each of the dataset.
To account for the different scales of accuracy across the 8 datasets, we also tested to compute a weighted average, normalizing by the accuracy of reference model (ViT-B/32). This is reminiscent of the normalization carried out in Hendrycks & Dietterich (2019) according to the score of AlexNet. We found the conclusions with the standard average and weighted average to be similar.
B.4.1 SPECIFIC CONSIDERATIONS THE ENSEMBLE CASE
For an ensemble with M members, we have access to M representations (<(a;; O MM for a given input æ. We have explored two ways to use those representations:
Joint:
We concatenate the representations {di(;; 0', ) m- into a single "joint" feature vector in R XS remembering that each <(æ: (%)) R We then train single linear classifier to predict the target labels from the "joint" feature vectors.
separately train a linear classifier to predict the target labels from the feature vectors induced by o(æ; 0' We then average the predictions the linear classifiers trained this fashion.
Disjoint: For each of the M representations {Ø(x; 6 (.))
In Table 7, we report comparison of those approaches. We aggregate the results over all ensemble models (namely, pBE and upstream ViT/V-MoE ensembles of size and 4) and over 8 replications, for the ViT families S/32, B/32 and L/32.
The results indicate that "joint" and "disjoint" perform similarly. Throughout our experiments, we use the "joint" approach because it eased some implementation considerations.
B.5 LIST OF DATASETS
For completeness, in addition to the few-shot datasets listed in Appendix B.4, we list the datasets used for downstream training and evaluation in this work.
ImageNet (ILSVRC2012) (Deng et al., 2009) with 1000 classes and 1281 188167 training ex- amples.
ImageNet-C (Hendrycks & Dietterich, 2019), an ImageNet test set constructed by applying
15 different corruptions at 5 levels of intensity to the original ImageNet test set. (We report the mean performance over the different corruptions and intensities.)
ImageNet-A (Hendrycks et al., 2019), an ImageNet test set constructed by collecting new data and keeping only those images which a ResNet-50 classified incorrectly.
ImageNet-V2 (Recht et al., 2019), an ImageNet test set independently collected using the same methodology the original ImageNet dataset.
Cifar10 (Krizhevsky, 2009) with 10 classes and 50000 training examples.
Cifar10-C (Hendrycks & Dietterich, 2019), Cifarl0 10 test constructed by applying 15 different corruptions at levels intensity to the original Cifarl0 test set. (We report the mean performance over the different corruptions and intensities.)
Cifar100 (Krizhevsky, 2009) with 100 classes and training 50000 examples.
Oxford Flowers 102 (Nilsback & Zisserman, 2008) with 102 classes and 1020 training examples.
17

---

Under review conference paper ICLR 2022
Table 7: Comparison of two approaches, "joint" and "disjoint". to compute the linear few-shot evaluation in the case of ensembles. For the ViT families S/32, B/32 and L/32, the mean error across datasets is averaged over 8 replications and over all the ensemble models of size 2 and
MODEL SIZE
METHOD
MEAN ERROR ACROSS DATASETS
SHOT
SHOTS
10 SHOTS
25 SHOTS
S/32
disjoint
51.01
32.80
0:3
26.33
20.97
joint
51.12
0.4
32.81
+:0.0
26.30 024
20.77
= 0.17
B/32
disjoint
42.43
-041
25.49
1:021
20.30 1.55
15.98
0.11
joint
42.59
0.40
25.74
0:18
20.54
0.13
16.06
=: 0.10
L/32
disjoint
36.41
-031
21.49
1.15
17.13
13.56
1 0.10
joint
36.48
21.66
0:13
17.34
0.00
13.56
Oxford-IIl pet (Parkhi et al., 2012) with 37 classes and 3680 training examples.
SVHN (Netzer al., 20 with 10 classes.
Places365 (Zhou et al., 2017) with 365 classes.
Describable Textures Dataset (DTD) (Cimpoi al., 2014) with 47 classes.
B.6
ABLATION DETAILS
B.6.1 STATIC VERSUS ADAPTIVE ABLATION DETAILS
The setup for the experiments in Figures 2 and differs slightly the other experiments in this paper. Specifically, while for other experiments we used upstream V-MoE checkpoints with (K, E) (2, 32), for these experiments we matched the upstream and downstream checkpoints. We did this avoid checkpoint mismatch potential confounder in our results.
B.6.2 FEATURE-LEVEL VERSUS PREDICTION-LEVE ENSEMBLING ABLATION DETAILS
The "Naive Multi-Head" method presented in Section 3.2 was trained in almost the same manner as the vanilla V-MoE, the only difference being the handling of multiple predictions. This accom- plished by using the average ensemble member cross entropy as described for pBE Appendix
On the other hand, order to compute the evaluation metrics presented in Table 2, we first averaged predictions of the model and then used the average prediction when calculating each metric.
C COMPATIBILITY AND ADAPTATION OF THE UPSTREAM CHECKPOINTS
Throughout the paper, we make the assumption that we can start from existing checkpoints of VIT and V-MoE models (trained on JFT-300M see Appendix B. We next describe how those checkpoints for the fine-tuning of the extensions of ViT and V-MoE that we consider in this paper.
use
In all our experiments that involve V-MoEs, we consider checkpoints with K 2 and E 32, which the canonical setting advocated by Riquelme al. (2021).
C.I PARTITIONED BATCH ENSEMBLES (PBE)
In the case of PBE, the set of parameters is identical to that of a V-MoE model. In particular, neither the tiled representation nor the partitioning of the experts transforms the set of parameters.
To deal with the fact that the single routing function gate (W-) of V-MoB becomes separate routing functions (gatex( (Wn.)) m= one for each part of the partition, we simply slice row-wise W E j E into the M matrices W m E (REMM))x
18

---

Under review conference paper ICLR 2022
C.2 BATCH ENSEMBLES (BE)
We train BE starting from VIT checkpoints, which requires to introduce coanssteam--ppcciiee pa- rameters. Following the design of V-MoEs, we place the batch-ensemble layers in the MLP layers of the Transformer.
us consider dense layer in one of those MLPs, with parameters U E IR Px absence of bias term. In BE, the parametrization of each ensemble member has the following structure U U (rm=m) where (rm and {8m Jal are respectively D- and I-dimmnnional vectors.
A standard ViT checkpoint provides pre-trained parameters for U. We then introduce {rm JM and fm 1M mm=1 at fine-tuning time, following the random initialization schemes proposed in Wen et al. (2019); see details in the hyperparameter sweep for BE in Appendix F.1.
C.3 MIMO
train MIMO models from V-MoE checkpoints. The only required modifications are to the input and output parameters of the checkpoints. The linear input embedding must be modified to be compatible with input images containing M times more channels, required by the multiple- input structure of MIMO. Similarly, the final dense layer the classification head must be modified to have M times more output units, following the multiple-output structure in MIMO.
Concretely, the embedding weight W E XX is replicated in the third (channel) di- mension, resulting MIMM
XX-MX, where and W are the height and width of the convolution and D is the hidden dimension of the ViT family (specified in Table 4). The output layer weight W ous E IR is replicated in the second (output) dimension, resulting in WMIMO,out R # where C the number of classes. The output layer bias bous E R is replicated resulting in bRIMO,,ou.. RCXM, Finally, in order to preserve the magnitude of the activation for these layers, WMIMO,i and W out are scaled by 1/M.
D IMPLEMENTATION DETAILS OF PBE
We provide details about the training loss and the regularizer used by pBE.
D.I TRAINING LOss
Since PBE outputs M predictions (f ,,.)) for given input æ. we need to adapt the choice of the training loss L accordingly. Following the literature on efficient ensembles (Wen al., 2019; Dusenberry et al., 2020a; Wenzel et al., 2020), we choose the average ensemble-member cross entropy
1 M M
Cly, x:; 0)
Σ
cross-entropy(v. (2; 6m))
instead of other alternatives such as the ensemble cross-entropy
cross-entropy M Σ f(z; 0m)
m=1
that was observed to generalize (Dusenberry et al., 2020a).
D.2 AUXILIARY LOSSES
Inspired by previous applications of sparse MoEs in NLP (Shazeer et al., 2017), Riquelme et al. (2021) employ regularizers, also referred auxiliary losses, to guarantee balanced usage of the experts. Two auxiliary losses-the importance and load losses, see Appendix A in Riquelme et al. (2021) for their formal definitions-ant are averaged together to form the final regularization term that denote by Ω.
As reminder, let us recall the notation of the routing function
h E IR gate (Wh) (op +re) R
19

---

Under review conference paper ICLR 2022
with W& e REXD and e NN I). Consider batch of B inputs {h:) that we represent by H BRBDD Finally, let us define
A HWT +oE ERRE.
where emphasise that B E is matrix of Gaussian noise entries in R The regularization term S used by Riquelme al. (2021) can be seen as function that depends on A and HW!
the context of partitioned batch ensemble, the set of experts is partitioned into groups E/M experts, whose partition is denoted by - see Section 4.1. With the introduction of the M routing functions {gate x (Wm:)) m==1 with each Wm. E (B/MM)x the matrix A becomes accordingly partitioned into {Am) J== where each A. E BR(A/)
Since we want to enforce balanced usage the E/M experts in each part &n of the partition, we thus redefine the regularization as the average regularization separately applied to each part of the partition
M Σ Q(A HWT)-
Opariition (A, HWW
M
We found this option to work better in practice. To guarantee a fair comparison, we also applied Aparition to the "Only partitioning" model in the ablation study of Section 4.2.1.
Following Riquelme al. (2021), the regularization parameter controlling the strength of 9 Qprtition was set to 0. throughout the experiments.
E PBE AND V-MOE RELATIVE IMPROVEMENTS PER VIT FAMILY
In Section we claim that pBE performs best the largest scale. In this section we motivate that claim in more detail. Specifically, we consider two metrics of improvement in performance. Firstly, we consider the percentage improvement in NLL for both PBE and V-MoE versus vanilla VIT. Secondly, we consider normalised version of this improvement. We consider this second metric to take into account the "difficulty" in further improving the NLL of larger ViT family models. Intuitively, the larger the VIT family, the better the corresponding NLL will be, and the more difficult it will be to improve on that NLL.
The normalisation we apply is based on the gradient of the NLL with respect FLOPs. Indeed, this gradient captures the typical variation of NLL at a particular amount of FLOPs. The ratio of this gradient at the FLOPs values (i.e., the instantaneous change in NLL at those FLOPs values) for two ViT families is a measure of the relative difficulty in increasing the NLL. Thus, we can use this ratio to normalise our results. To be more concrete, let us define the mapping
NLL = (FFLOPs) and its derivative &f (FFLOPs) = dp(FFLOPs)
dFLOPs
We estimate yp and its gradient by fitting a linear model to the (NLL,, FLOPs) pairs for each VIT family, using the data of the standard ViT models trained. We use feature expansion [1.0, log(FL. log (FLOPs) (oLOFL)))) and solve for the parameters via ordinary least squares. We determine the gradient of this function at each FLOPPs value using automatic differentiation in JAX (Bradbury et al., 2018). See Figure 7 for the resulting fit and an indication of the gradients.
the linear model
The normalised values are calculated as:
Normalised improvement(() = improvement (v) x o (FLOPs mi) o (FFOOS)).
(4)
where v is one of the VIT families, i.e., S/32, B/32, I/32, I/16, or H/14. Note that this normalisation leaves the improvement for H/14 the same. We tried to normalize with respect to other choices of VIT family, different from H/14. Our conclusions are robust, in the sense that both the ordering and the monotonic behavior with respect to scale are preserved. Using the ratio for normalisation also has the advantage that the normalisation is less sensitive to the particular parameterisation of so.
Table 8 shows both the Looking first at the original improvements, can that while both pBE and V-MoE have smaller
and original improvements (without normalisation).
20

---

Under review conference paper ICLR 2022
1.0
Tangent
VIT
0.8
NLL
0.7
0.6
05
1000
2000
2000
(downstream) GFLOPs
Estimated oo compared to the ImageNet NLL values for our ViT models. We also include the tangent at the points corresponding to each ViT model to indicate the gradients at those points.
Figure 7:
Table 8: Percentage improvements NLL for PBE with (K, M) = (1, and V-MoB with = vs. ViT for families of increasing size. The top two rows show normalised improvements, see (4), which take into consideration the increased difficulty of improving NLL for larger VIT families whose performance beginning to saturate, The bottom two rows are original percentage provements without normalisation.
S/32
B/32
L/32
L/16
H/14
PBE vs. VIT
0.02%
0.08%
0.22%
2.21%
4.27%
Normalised
V-MoB vs. ViT
0.01% %
0.06%
-0.03%
0.84%
0.02%
pBE vs. VIT 9.82%
9.53%
3.76%
5.38%
4.27%
Not normalised
V-MoE vs. ViT
7.98%
6.62%
-0.60%
2.05%
0.02%
improvements ViT for larger families, pBE's improvements decrease more slowly. Further- more, by comparing the normalised improvements we see that pBE's improvements actually grow monotonically when taking difficulty into account. This is not the case for V-MoE.
EFFICIENT ENSEMBLE COMPARISONS
In this section, we compare partitioned batch ensembles (pBE) to several popular efficient ensemble approaches, namely MIMO (Havasi et al., 2020), batch ensemble (BE) (Wen et al., 2019), and MC Dropout (Gal& Ghahramani, 2016).
Table 9 reports the ImageNet performance of those different techniques, when models are based on ViT-B/31 architecture. We start by highlighting the most salient conclusions of the experiment and defer the next subsections the descriptions of the different competing techniques.
We make the following observations:
BE built upon VIT improves the performance of ViT in terms NLL, classification er- ror and ECE However, the resulting increase in FLOPSs makes BE a less viable option compared to pBE.
MC dropout V-MoB is on par with standard V-Moß in terms of NLL. and classification while improves the ECE. For all values of K, we observe that the performance tends improve
number samples, i.e., M,, increases. However, already for M {2, 4), H the resulting increase in FLOPs makes MC dropout V-MoE a less favorable option compared to pBE.
Perhaps surprisingly (see detailed investigations in Appendix F33), MIMO V-MoE does not lead to improvements compared with V-MoE. In fact, for higher ensembles sizes, MIMO V-MoE results in worse performance than standard V-MoE. Moreover, increasing the batch
21

---

Under review conference paper ICLR 2022
Table 9: ImageNet performance of different efficient ensemble approaches. The table reports the means + standard errors over replications. All models have ViT-B/32 architecture. K stands for the sparsity in V-MoEs, M denotes the ensemble size while "BR" corresponds the batch repetition in MIMO (Havasi et al., 2020).
K
M
NLL
ERROR
ECE
KL
GFLOPS
VIT
0.688
0.08
18.65
0.022
0.000
72.5
VIT
2
0.682
1,006
18.47
1.00
0.021
0,0000
0.040
00001
94.8
4
0.675
0.00
18.40
0.017
0.035
0001
132.1
0.642
a 0.02
16.90
0.05
0.029
0.001
V-MoE
0.638
0.00
16.76
0.05
0.033
0.001
0.636
=: 0.001
16.70
= 0.04
0.034
0001
107.0
0.635
0.022
16.72 0.06
0.028
0.001
151.7
2
0.648
17.10
0.019
0.046
95.9
0.641
16.96
0,017
0.046
138.4
MC dropout V-Mol
0.642
0.00
16,94
=:
0.021
0.046
119.6
0.634
0.00
16.80
0.03
0,020
0.000
0.046
0.001
183.5
0.639
0.02
16.91
0.06
0.022
0.001
0.045
0.001
164.0
MIMO V-MoE (BR=1)
2
0,636
0.002
16.97
0.028
0,000
0.000
90.2
4
0,672
17.72
0.037
0,001
0.000
92.8
MIMO V-MoE (BR=2)
2
0.638
6:00
17.14
0.031
0.001
0000
180.3
0.665 4
17.38
0.038
0.000
185.4
0.622
1 1.00
16.70
1. 0.03
0.018
0.000
0.217
J. 0.00
94.5
4
0.624
0.001
16,99
0.03
0.013
0.000
0.164
0.01
136.3
PBE
2
0.612
0.000
16.49
0.00
0,013
0.000
0.198
0.001
116.8
4
0,620
0.001
16.86
0.015
0.170
0:01
181.0
4
2
0.611
0.001
16.45
0.014
0.193 0.003
161.4
repetition parameter of MIMO ("BR" Table further worsens the results. Interestingly, we can see that MIMO does not manage to produce diverse predictions, as illustrated by the small values of KI.
pBE offers the best performance FLOPS trade-offs, e.g., when looking (K, (1, 2) and (K, M) = (2, 2). notably observe that the diversity of the predictions PBE is orders of magnitude larger than that of the other ensemble approaches.
We briefly recall the optimization explained in Section 4.1 to save redundant computations: In the "last-ni" setting of Riquelme et (2021), sufficient to tile the representations only when entering the first MoE layer/dropout layer/batch-ensembl layer for respectively pBE/MC dropout V-MoE/BE. We apply this optimization to all the efficient ensemble methods.
E.1
BATCH ENSEMBLES
Following the design of V-MoEs, we place the batch-ensemble layers in the MLP layers of the Transformer, following the "last-n" setting of Riquelme et al. (2021); see Section 2.1.
The vectors of the rank-1 parametrization introduced at fine-tuning time (see Appendix C) need be initialized and optimized. Following the recommendation from the following hyperparameters in addition to the common sweep described in Table 5:
al. (2019), we consider
Initialization: Either random sign vector with entries in {-1. independently drawn with probability 1 or random Gaussian vector with entries independently drawn from (11,0.5)
Learning-rate scale factor: The vectors of the rank-1 parametrization are updated with a learning rate scaled by factor in {0.5, 1, 2)
22

---

Under review conference paper ICLR 2022
F.2 MC DROPOUT V-MOES
For MC dropout V-MoE, we take the available fine-tuned V-MoEs and enable dropout at prediction time. Indeed, as described in Table 5, all V-MoE models already have a 0.1 dropout rate in the experts.
E.3 MIMO V-MoEs
Following Havasi et al. (2020) we consider two MIMO-specific hyperparameters, in addition to the hyperparameters listed in Table 5:
Input replication probability: {0.5, 0.625, 0.75}
Batch repetitions: {1,2]
Our preliminary investigations also considered lower input repetition probabilities and higher batch repetitions. However, lower input repetition probabilities tended result poorer performance. While higher batch repetitions did help to some extent, the additional computational cost made it impractical.
Given the surprising result that an ensemble size of M 2 provides no performance improvement over the standard V-MoE and that increasing M further provides worse performance, there seems to be some incompatibility between MIMO and V-MoE. In fact, our investigations revealed that ViT the source of the problems since applying MIMO vanilla VIT without experts resulted in the same trends as for V-MoE. Thus we hypothesise that the differences between ViT and ResNet- the architecture to which MIMO originally applied by Havasi et al. (2020) responsible for MIMO's poor performance when applied to ViT.
Class token. One of the differences hetween ViT and ResNet is that ViT makes use a special learnable class token to classify an image (see Dosovitskiy et al. (2021) for details). ResNet on the other hand makes use of the representation from an entire image for classification. We tried two strategies to mitigate this difference:
Difference 1:
1. We applied the global average pooling (GAP) and multi-head attention pooling (MAP) classification strategies introduced Dosovitskiy et (2021) and Zhai et al. (2021), re- spectively. In short, both these methods make use of all the tokens from an image for classification. However, neither of these strategics made significant difference to the rel- ative performance MIMO and ViT. fact, the choice of classification method was the least impactful hyperparameter in our sweep.
2. Rather than learning single class token, we learnt M class tokens. This strategy resulted in MIMO with M outperforming VÝT. However, for M > the improvement was small enough that ViT still outperformed MIMO.
other major difference between ViT and ResNet the building block for each model. While ResNets are primarily composed of convolution operations, ViT makes heavy used of attention. We hypothesised that attention is less suited separating the information for input images stored in the channel dimension of a single image. We tried two strategies mitigate this potential issue:
Difference 2: Attention.
We applied the hybrid architecture, described in Dosovitskiy et al. (2021), in which the input sequence to ViT is formed by CNN feature maps. We used ResNet14 and ResNet50, In both cases, found that strategy boosted the performance of ViT and MIMO equally.
Rather than concatenating images the channel dimension, we concatenated them in the width dimension, resulting in 3 times as many patches for ViT to process. This strategy successful in the sense that the MIMO performance for M improved significantly. However, significant additional computational cost made it an infeasible solution.
Our findings suggest that MIMO and ViT are indeed somewhat incompatible. Unfortunately, none of our proposed solutions to this problem provided high enough predictive performance increases (or indeed low enough computational cost increases in some cases) to warrant immediate further investigation.
23

---

Under review conference paper ICLR 2022
G UPSTREAM & DOWNSTREAM VERSUS DOWNSTREAM-ONLY ENSEMBLES
Section 5, and Appendix include downstream deep ensembles (down-DE) of V-MoF, and some cases VIT. baseline. This choice was motivated by the fact that like VIT. V-MoE, and pBE. down-DE requires only single upstream checkpoint, which all of the methods more comparable. However, clear that using different upstream checkpoints and then further fine-tuning each these with different random seeds to construct an upstream deep ensemble (up-DE) would result in varied ensemble members and result, better performing ensemble. This idea has recently been explored by Mustafa al. (2020).
Thus, for completeness, we also investigate the effects of upstream ensembling on V-MoE. Table 10 compares the performance of upstream and downstream V-MoE (K 1) ensembles of sizes M = 2 and M 4. Across the range of metrics, for both ImageNet and ImageNet-C, for all ViT families. and for both values of M, we see that up-DE outperforms down-DE. fact, up-DE with M = 2 is very often better than equal to down-DE with M 4. This is especially true for the diversity metrics, which indicates that diversity is indeed the driver for improved performance up-DE. Not shown in the table is the very large computational cost associated with training upstream ensembles.
24

---

Table 10: Comparison of upstream and downstream ensembles of V-MoE with (K = 1).
Under review as conference paper ICLR 2022
IMAGENET
IMAGENET-C
M
NLL I
ERROR
ECE
KL
Cos. SIM.
NORM. DIS.
NLL
ERROR
ECE
KL ~
Cos. SIM. 4
NORM.
1
down-DE
2
0.403
11:35
0,018
0,079
0.974
0.488
0.871 a-0012
21.37
0,021
0,218
0.925
0.628
H/14
up-DE
0.391
1:000
11.12
a:300
0.016
0.126
0.963
0,625 0.022
0.839 a:1
20.66
0:022
0.022
00000
0.355
0.892
0.809
#-3000
down-Dr:
4
0.792 1:000
11.20
0.014
0.083
0.973
0.509
0.851
20.97
0.021
0.221
0.923
0.650
up-DI:
0.375 4
3,0000
10.66
20000
0.013
0000
0.129
0000
0.963
0000
0.652 1000
0.792 3:0000
19.61
2,000
0.032
6.000
0.361
200e
0.892
2.000
0.850
2: 3300
down-DF
2
0,450
1:0000
12.62 1:
0.016
0000
0.061
0.979
0000
0.419
1.010 1:000
24.43
0.021
6.000
0.168
0000
0,936
00000
0.539
1,300
L/16
up-DF
2
0.434
1,0000
12.23
0.014
0.118
0000
0,964
0,584 0000
0.961 3:000
23.46 3:103
0.023
0.342
2000
0.890
2,0000
0.766
3:300
down-DE
4
0,440
12.39
1:306
0.015
0.061
0000
0.979
D000
0,425 000002
0.983 0:0006
23.95
0:032
0.020
0000
0.166
0.937
0,547
#:3002
up-DE
0.418 4
11.86
0.013
0.118
0.964
0,603
0.916
22.45
0.034
0.341
0.890
0,800
down-DE
2
0.533
0-00022
14.55
+:304
0.025
0.092
0.969
0.479
+: 0:04
1.184 0.0.03
27.98 0:004
0.029
0.199
0.925
0.556
#:3002
1J32
up-DE
0.511
1:0.0
14.07
3.3
0.019
0.191
0.945
0,694
1.133
26.97
2:044
0.022
0.449
0.861
0.820
6:3000
DF:
0.486 4
0,518
14.29
1:00
0.022
0.092
000
0.969
00000
0.487
2000
1.154 3.000
27.47
0.023
6,000
0.199
$000
0.925
g0000
0.567
2:300
up-DE
2:0000
13.52
3:10
0.016
0.190
0000
0,946
0000
0.722 2: 8:00
1.073
2: 2:000
25.74 2: 8:
0.030
6000
0.446
0000
0.862
=: 0.000
0.857
3:300
down-Dr:
2
0.519
14.09
0.021
0.048
0,982
0.351
1.316
30.02
0.030
0,132
0.943
0.448
B/16
down-DE
up-DI:
2
0.489
0.511
1-0.12
2000001
13.95 3:300
13.40 1:30
0.019
0.015
0.00
0.048
0.169
0000
0.982
0,951
0.000
0.354
0.668
à:
1.293
1.231 2:0000
0.0.00
29.67
28.41 2:
0.026
0.023
6.000
0,132
0.481
0.845
0,943
L: 20000
0.838
0.453
3:000
6-0002
up-DE
0.468 4
0:0:00
12.89
1:300
0.016
0,168
0.951
0.690
0:0000
1.166
1.0.02
27.08
1:305
0.037
10000
0.479
00000
0.846
bxbe
0.879
0:3000
25
dowa-DE
0.620
16.44
1:4
0.023
0.073
0.973
+ D000
0,414
+: 0:00
1.510
8:0035
33.79 2:20
0.032
-0000
0.175
0.925
0.498
#:3002
up-DE
2
0.588
15.74
0.017
0,214
0.937
0.709
1.430
32.37
0.022
0.537
0,824
0,844
B/32
down-Dr:
4
0.607
2:000
16.17 1:100
0.021
2000
0.073
0000
0.973
0.00
0.418 axes
1.483
1:000
33.36 1/
0.027
6,000
0.174
0000
0.926
2.000
0.504 3:300
up-DE
0.561 4
15.10
0.020
0.214
0.937
0.739
1.357
34192
0.036
0.537
0.824
0.884
down-Di DF:
2
0.807
2.0000
20.90
0.018
0000
0.102
0000
0,962
0000
0.458 5:000
2.106 3:0010
44.52 1
0.038
0,223
=000
01900)
2.000
0.521
3:3000
S/32
up-DE
2
0.763
19.85
1:1
0.016
0000
0,305
0000
0,911
0000
0.773 4: 1:0000
2.004
2:0030
42.92
0.025
2 6.00
=0000
0.767
2.0000
0.856
down-DE
4
0.795
20.66
0.015
0.102
0.962
0.462
2.076
44.16
0.031
0,222
0,900
0,526
2:300
up-DE
4
0.728
19.06
3:04
0.025
1000
0.304
0.911
0.808
00002
1.914
0.0033
41.38
2:05
0.034
00000
0.682
-000
0.767
0.891
3:3002

---

Under review conference paper ICLR 2022
H ADDITIONAL EXPERIMENTAL RESULTS
this section we expand on various experiments presented in sections 3 and experiments considering multiple VIT families we also include B/16 which was excluded from the main text for clarity.
H.1 STATIC VERSUS ADAPTIVE COMBINATION
Here continue the investigation into static versus adaptive combination from Section 3.1.
K and M. Figure 8 shows the effect of increasing the the various "ensemble size' parameters for a deep ensemble V-MoEs. In particular, we investigate the static combination axis M (the number of ensemble members), as well as the two adaptive axes K (the number of experts chosen per patch) and E (the total number of experts).
Individual gains with respect
When investigating the effect K, we fix E investigating M, we fix E 32 and average over K E {I..., 8). When investigating the effect of E we fix K 2 and average the curves are not directly comparable. However, we can still examine the relative trends.
and average over M (1....8. Similarly, when
M (1,... 8). As result of this procedure the exact values of
Specifically, we note that while the variation K and curves is roughly variation in the E curve is smaller. We also note that there is very little variation beyond E = 8 (note the difference in the scales of the for the curves). These observations motivate the design of pBE, where we split the sub-models along axis, order to better take advantage of the experts.
same size, the
NLL 'ensemble size'
E
0.84
K
0.82
E
0.80
2
K/M
Figure 8: Comparison the impact on ImageNet NLL of variations K, E and M. The underly- ing model is ViT-S/32.
Extended Results for the Cumulative Effects of Static and Adaptive Combination. In Fig- ure 9 we extend the ImageNet NLL results, presented in Figure 3, range other datasets and performance metrics. We see that in most cases, the trends are the same as before. That is, (static) ensembling tends to improve the performance of VIT and V-MoE equally. The two exceptions ECE and OOD detection for ViT-S/32 where we see creased performance. These results indicate that the for small ViT models, larger ensembles can have slightly lower quality uncertainty estimates. The trend for ImageNet-CC performance also not as consistent ensembling sometimes helping VIT V-MoE less (as indicated by the changes in ordering on the y-axis).
larger ensemble sizes can result de-
26

---

Under review conference paper ICLR 2022
NLL (lower is better)
Error (lower better)
ECE (lower better)
0.035
22.5
0.030
ImagoNet
20.0
0.7
0.025
17.5
0.6
0.5
0.015
12.5
0.010
0.4
10.0
NLL (lower is betler)
Error (lower betler)
ECE: (lower is betler)
50
45
0.05
2.0
ImageNet-C
0.04
35
30
0.03
0.02
5-Shot Error (lower is better)
10-Shot Error (lower is better)
25-Shot Error (lower is better)
30
25.0
Mean Across Dalasets
35
22.5
30
25
20.0
17.5
25
20
15.0
15
FPR@95 (lower better)
AUC (ROC) (higher better)
AUC (PR) (higher better)
0.25
0.99
0.990
CIFARIO CIFAR100
0.985
0.20
0.980
0.97
0.05
0.96
0.965
0.960
downstream log(GFL.OP) (loweri is better)
S/32
B:32
L/32
B/16
L/16
H/14
V-MoB
ViT
Model
2 Models
4 Models
Figure 9: Extended results for Figure 3 to selection of other tasks and metrics. We see that in most cases, ensembles tend to help ViT and V-MoE equally.

---

Here we extended the OOD results of Figure 5. Specifically, we add Cifar100 as an in-distribution datasct and Describable Textures Dataset (DTD) (Cimpoi et al., 2014) as an OOD dataset. We also add area under the receiver operating characteristic (AUC (ROC)) and area under the precision-recall Cifar10 and Cifar 100 as Under review conference paper ICLR 2022
H.2 EXTENDED RESULTS FOR FEW-SHOT LEARNING
Figure 10, we extend the few-shot learning results of Figure 4 to also include 5, and 25-shot. Additionally, we show results for the weighted aggregation strategy mentioned in Appendix B.4.
We confirm the result that few-shot performance for pBE gets better, relative to the other baselines, with larger VIT families. Additionally, we see that pBE performance seems to get better, again relative to the other baselines, with more shots. This phenomenon can most easily be noticed by comparing the results for S/32 across the different numbers of shots. Finally, we see that the trends with and without the weighted mean are the same.
better?
(lower bedee))
beteer)
better)
30
25.0
Mon Acms Dattcats
80-
22.5
45
17.5
731
150
35
10
Waighed Mor Actie Datants
66
14-
83
60-
X1
dowmstream leg(GFLO beter)
2/15
H/14
PAE
V-Moll
VIT
Model
2 Modill
Figure 10: Extended few-shot results from Figure 4 with additional aggregation method and numbers of shots.
H.3 EXTENDED RESULTS OOD DETECTION
curve (AUC (PR)) as metrics. Figures 11 12 contain the results the in-distribution datasets, respectively.
As in Figure 5, we see that pBE performs better (relative to the other baselines) for larger VIT families. Furthermore, PBE seems perform better in near OOD detection (i.e. Cifarl0 versus Cifar100, and vice versa) than far OOD detection. Finally, we see that these trends are consistent across ODD metrics.
28

---

Under review conference paper ICLR 2022
F (lower better)
AUC (ROC) (bigher better)
ALC (PR) (higher is better)
0.99
0.25
CIFAR CIFARIO0
0.20
0.98
0.15
0,975
0.97
0.10
0.970
0.05
0.965
0.960
0.020
1.000
1.00000
CIFARIO DTD
0.998
0,99950
0.010
0.99925
0.99900
0.005
0.99875
0.14
0.95
0.995
CIFARIO Placo665
0.90
0.990
0.985
0.06
0.75
0.980
0.04
0.70
0.02
0.955
0.65
0.998
0.04
0.995
CIFARIO SVHN
0.996
0.03
0.994
0.02
0.985
0.00
0.980
downstream log(GFLOPS) (lower is better)
S/32
B/32
L/32
B/16
L/16
pBE
V-MoE
ViT
Model
Models
Figure Extended OOD detection results from Figure an additional OOD dataset more metrics.
29

---

Under review conference paper ICLR 2022
FPR (lower is better)
AUC (ROC) (bigher better)
AUC (PR) (higher is better)
0.96
0.96
0.6
CIFAR CIFARIO
0.5
0.92
0.92
0.90
0.88
0.3
0.86
0.30
0.99
0.9975
CIFARIOO DII
0.25
0.9950
0.97
0.9925
0.20
0.96
0.9900
0.15
0.9875
0.94
0.9850
0.93
0.9825
0.975
0.8
0.6
CIFAR100 Places665
0.950
0.7
0.6
0..
0.900
0.875
0.2
0.825
0.50
0.95
CIFARIOO SVHN
0.45
0.93
0.86
0.91
0.84
0.35
0.30
0.89
0.80
downstream log(GFLOPS) (lower is better)
S/32
B/32
L/32
B/16
L/16
pBE
V-MoE
ViT
Model
Models
Figure 12: Extended OOD detection the results from Figure with Cifar100 as the in-distribution datasct, an additional OOD dataset, and more metrics.

---

Under review conference paper ICLR 2022
H.4 EXTENDED RESULTS FOR IMAGENET
this section we extend the results for ImageNet and the comrupted variants presented in Figures 5 and 6. In addition to NLL (and ECE for standard ImageNet), Figure 13 provides classification and ECE for all ImageNet variants.
Most trends observed in Section remain true:
pBE tends be Pareto efficient the presence of distribution shift.
For smaller VIT families, V-MoE outperforms VIT the presence distribution shift.
pBE improves ECE over VIT and V-MoE.
PBE improves classification performance.
ViT consistently provides better ECE than V-MoE.
However, there are some exceptions:
ImageNet-A classification error. models (including pBE) under-perform relative ViT-S/32 and ViT-H/14.
ECE for ImageNet-C, ImageNet-A, and ImageNet-Va. V2. Interestingly for the non- standard ImageNet variants, and particular ImageNet-A, there strong correlation between lower ECE and larger VIT families.
H.5 ADDITIONAL CI CIFAR FLOWERS, AND PETS RESULTS
Here extend the results for ImageNet and the corrupted variants presented in Figures and 6 additional datasets. Figures 14, 15, 16 provide results Cifar Cifar100, Oxford Flowers 102, and Oxford IITT Pet, respectively. As in Appendix H.4, we find that the results are similar those in Section 5.
Compared to ImageNet, for Cifar10, CifarlO-C, and Cifar100, pBE seems perform even better relative to the other baselines. Note, for example, that pBE is Pareto efficient (even for S/32) in the cases of Cifar 10-C and Cifar 100 NL.I. As in Appendix H.4, we see that the ECE has stronger downward trend with respect increased ViT family size for shifted test data.
Flowers and Pets, where we only have results smaller ViT families, pBE seems under perform. However, the performance for L/32 is better than for S/32 and B/32 which suggests that the results for these datasets consistent with the other datasets presented in this work and, therefore, that we should expect pBE's predictive performance keep improving with larger models.
H.6 PBE AND V-MOE WITH LARGER VALUES OF K AND M
Figure 18 and Figure 19 show the effect of varying K pBE and V-MoE, and the effect of varying M on PBE, respectively. We make the following observations:
almost all cases, increasing K or M does result in Pareto efficient models.
For V-MoE, increasing K seems to help in most cases, except for ECE performance where it usually hurts.
For PBE, going from K to K seems help most cases but going from K to K 4 usually hurts. Going from K = to = 4 still helps but to a lesser extent than from K = to K- 2.
For pBE, increasing M either doesn't 't make consistent and significant difference hurts (e.g. in OOD detection).
These conclusions should, however, be considered with caution. Recall that the upstream check- points used for fine-tuning all V-MoE and pBE models in this work are V-MoE models with K 2. Thus, the results in this experiment match for all PBE models and all V-MoE models with * 2. For example, we hypothesise that it is more difficult to train downstream pBE models with larger values of M from upstream V-MoE models because in each partition common expert specialisations will need to be duplicated.
confounded by upstream and downstream checkpoint mis-

---

Under review conference paper ICLR 2022
(lower is better)
Error (lower better)
BCE (lower better)
24
0.040
0.035
20
ImageNet
0.025
0.6
0.020
14
0.015
0.4
10
2.25
2.00
ImageNet-C
40
0.05
1.75
1.50
35
0.04
30
1.25
0.03
1.00
0.75
100
90
0.35
ImageNet-A
80
0.30
0.25
70
0.20
0.09
1.4
ImngeNet-V2
0.07
30
1.2
25
0.05
0.04
2
downstream log(GFLOPS) (lower is better)
S/32
B/32
L/32
B/16
L/16
H/14
pBE
V-MoE
ViT
Model
Models
Figure 13: Extended results from Figures 4, 5 and with additional metrics.

---

Under review conference paper ICLR 2022
NLL (lower is better)
Error (lower is better)
ECE (lower better)
0.07
0.008
0.06
0.007
CIFARIO
0.006
1.3
0.04
1.0
0.03
0.02
0.5
16
0.08
0.07
CIFAR
0.06
0.05
0.3
0.03
2
downstream log(GFLOPP) (lower is better)
S/32
B/32
L/32
B/16
L/16
pBE
V-MoE
VIT
Model
Models
Figure
Results for Cifarl0 and 10far10-C.
NLL (lower better)
Error (lower is better)
ECE (lower is better)
0.45
0.04
0.40
CIFARIOD 100
0.03
10
0.30
0.02
8
0.20
downstream log(GFLOPP) (lower is better)
S/32
B/16
L/16
pBE
V-MoB
ViT
Model
2 Models
Figure 15: Results for Cifar100.

---

Under review conference paper ICLR 2022
NLL (lower is better)
Error (lower is better)
ECE (lower is better)
0.20
0.04
Oxford Flowers
0.03
0.15
0.10
2
0.01
0.05
0.00
2.0
2.0
1.5
2.5
downstream (Gg(OPL)Ss) (lower is better)
PBE
V-MoE
VIT
S/32
B/32
L/32
Model
2 Models
Figure 16: Results for Oxford Flowers 102.
NLL (lower better)
Error (lower is botter)
ECE (lower is better)
0.30
0.04
Oxford Pets
0.25
0.20
6
5
0.15
0.01
2.0
2.5
2.0
1.5
2.5
downstream log(GFLOPP) (lower is better)
PBE
V-MoB
VIT
S/32
B/32
L/32
Model
2 Models
Figure 17: Results for Oxford IIIT Pet.

---

Under review conference paper ICLR 2022
NLL (lower is better)
Error (lower better)
ECE (lower better)
24
0.040
22
ImagoNet
20
0.030
0.025
18
0.020
14
0.010
NLL (lower is better)
Error (lower better)
ECE (lower better)
50
0.07
2.25
0.06
45
ImageNet-C
0.05
1.75
0.04
35
1.50
1.25
30
0.02
5-Shot Error (lower is better)
10-Shot Error (lower is better)
25-Shot Error (lower is better)
30.0
Mean Across Dalasets
27.5
25.0
22.5
20.0
16
17.5
14
FPR@95 (lower better)
AUC (ROC) (higher better)
AUC (PR) (higher better)
0.99
0.990
CIFARIO CIFARJ00
0.25
0.985
0.20
0.980
0.15
0.975
0.97
0.970
0.10
0.96
0.965
0.05
0.960
1.5
2.0
2.0
2.5
2.0
2.5
downstream log(GGLLOPs) (lower is better)
pBE
V-MoB
VIT
8:32
B/32
1/32
Model
Models
Figure 18: Results for V-MoB with K 2,,2,4,8) and pBE with K {1,2, 4). Models with larger values of K have larger FLOPs.

---

Under review conference paper ICLR 2022
NLL (lower is betler)
Error (lower better)
ECB (lower is better)
24
0.035
22
0.030
ImageNet
20
0.020
16
0.015
14
0.010
NLL (lower is better)
Error (lower better)
ECE (lower is better)
2.4
0.06
45
0.05
ImageNet-C
1.8
40
0.04
35
30
1.2
5-Shot Error (lower is better)
10-Shot Error (lower is better)
25-Shot Error (lower is better)
30.0
24
Mean Datasets
27.5
22
25.0
20
22.5
18
25
17.5
FPR (lower is better)
AUC (ROC) (bigher better)
AUC (PR) (higher is better)
0.99
0.99
0.30
IO CIFAR100 100
0.25
0.98
0.98
0.20
0.97
0.15
0.97
0.10
0.96
0.96
1.5
2.5
2.5
2.0
downstream log (lower is better)
PBE
V-MoE
VIT
S/32
B/32
L/32
Model
2 Models
Figure 19: Results for pBE with M = and K E ( (1,

---

Under review conference paper ICLR 2022
H.7 FLOPS NUMBERS
Table 11 provides the downstream training FLOPs for various PBE, V-MoE, and VIT configura- tions. These numbers correspond to the x-values of the points in the figures presented in Section 5 and Appendix H. Table provides the percentage difference in FLOPs between the pBE, V-MoE and down-DE models most commonly used in this work. Note that the percentage differences for H/14 do not follow the trend of the other sizes, e.g. that the percentage difference between PBE and V-MoB gets smaller for larger sizes, due the fact that for H/15 than the last-2 configuration used for the other ViT familics.
last-5 configuration rather
Table II: Downstream training GFLOPs for the various pBE, V-MoE, and ViT baselines used this work.
K
M
S/32
B/32
B/16
L/32
L/16
H/14
PBE
1
2
32.71
94.51
403.92
287.14
1210.91
3967.77
PBE
1
4
51.65
136.35
360.62
PBE
2
2
42.69
116.79
492.62
326.41
1367.99
PBE
2
4
71.65
181.00
439.89
PBE
4
2
62.68
161.44
405.67
VIT
22.32
72.46
310.67
252.61
1069.53
2962.57
VIT
2
44.63
144.93
621.34
505.22
2139.06
5925.13
VIT
4
89.26
289.86
1242.68
1010.45
4278.13
11850.27
V-MoE
23.22
73.55
313.95
249.70
1055.14
3008.02
V-MoE
2
46.44
147.10
627.89
499.39
2110.29
6016.04
V-MoE
4
92.87
294.19
1255.79
998.78
4220.58
12032.08
V-MoE
2
1
28.23
84.74
358.70
269.51
1134.41
3436.74
V-MoE
4
1
38.20
107.01
447.39
308.96
1291.49
V-MoE
8
1
58.20
151.67
388.05
Table 12: Percentage difference downstream training FLOPs for pBE with (K, M) compared with V-MoE with K = and an ensemble of two such V-MoE members.
(1,2)
S/32
B/32
B/16
L/32
L/16
H/14
PBE vs. V-MoE
40.88
28.51
28.66
14.99
14.76
31.91
pBE down-DE
-29.56
-35.75
-35.67
-42.50
-42.62
-34.05
H.8 AUXILIARY TABLES STANDARD ERRORS.
The tables in this section provide the mean values and corresponding standard errors for many the results depicted in figures throughout Section 5 and Appendix H.
37

---

Table 13: ImageNet comparison of V-MoB, downstream ensembles there-of, and pBB with 2 experts per input in cach case.
Under review as conference paper ICLR 2022
IMAGENET ERROR 1
IMAGENET-C ERROR 1
IMAGENET-A A
IMAGENET V2 ERRUR
DOWNSTTREM AFLOPs (16) J
K
M
NLL
ECE
NLL L
ECE 4
NLL 4
ERROR 4
ECE 1
NLL 1
ECE J
PBE
2
0,408
1:000
11.63
0.012
0.865
21.46
0.018
2.276
49.09
0.101
0.745
19.50 1:30
0
15.45 75.05
H/14
down-DE
0.403
11.35
0.018
1:000
0.871
21.37
0.021 1101
2.273
47.93
0:000
0.108
0001
0.758 00000
19.28
1:2
0.044
500000
V-MoE
2
0,428
11.89
0.030
0.934
22.41
0.038
2,517
50.34
2:11
0.167
0.811 40000
20.25
4:105
0.065
PBE down-DE DE V-MoE
0.448
0:000
12.60
0.020
1:300
1.023
2000
24.89
0.024
1:00
2.836
0005
57.97
1:0.1
0.160
0001
0.838 1000
21.30
0.051
10000
6.74 86.03
L/16
0,450
12.62
0.016
1.010
24.43
0.021
2.796
57.06
=:000
0.136
0.818
21.06
0.044
0.464
1:930
1:300
1.058
1.000
25.27
0.034
2.945 0006
57.85
1.90
0.178
0.848
1/000
21.33
0.057
1,000
PBE
2
0,535
1,000
14.70 200
0.027
1,300
1.193 1.000
28.28
0.032 1.00
4.170
74.73
1,00
0.266
1000
0.989
1.000
24.62 1
0.063
1000
6,54
LJ32
down- DE
0,533
1:000
14.55
2.0
1:300
1.184
2.000
27.98
0.029
4.139
2002
74.29
1:977
0.254
0.982
1/100
24.42
0.061
1,000
85.29
V-MoE
2
0,563
1:0.00
15.05
0.03
0.039
9:3000
1.261
0.00
29.12
0.052
1.000
4,394
0004
75.39
8:0.1
0.301
0002
1,046
0.0002
25.23
5:336
0.083
000000
PBE
0.519
0:000
14.26
0,020
3:3000
1.352
8.00
31.20
0.022
3.835 0009
72.25
0:0.22
0.223
0002
0.974
24.10
5:330
0
B/16
down-DE
0.519
4:000
14.09
0.021
1:300
1,316 8.
30.02 1::0
0.030 =000)
3.618 2015
66,99
2:103
0,203
2 20022
0.945
5/100
23.39
5:3:0
0.054
20000
75.05
V-MoE
0.533
2:0000
14.60
1:44
0.022
8:00
1.372
6.00
31.27
2:000
0.034
3,875
70.79
5:0.30
0.235
0.959
24.09
#:3:1
0.056
PBE
2
0.622
2:000
16.20
0.018
3:00
1,532 m.0
34.73 2.00
0.022 2000
5 5.080
0005
84,91
0.301
2001
1.143 4:000
27.97
#:3:3
0.055
00000
11.54
B/32
down DE
0.620
16.44
0.023
1,510
33,79
0.032
4,891
81.98 1/9.5
0.284
1.116
27.24
1
0.061
73.59
V-MoF
0.638
1:000
16.76
0.033
1 3:0
1.562
34,40
0.050
5.032 000
81.71
1::01
0.317
1.150
1/000
27.50
3:3
0.076
10000
PAE down- DE
0.818
21.18
0.015
2.169
45,79
0.030
6.419
93.98
0.345
1,437
34.03 g
0,054
15.88
38
S/32
0.807
0:000
20.90
0.33
0.018
8:000
2.106 2.000
44.52
0:00
0.038
1001
6.063
92.33
0.335
1.393
1:0000
33.29
5:230
0.062
500000
64.55
V-MoB
2
0,829 2:000
21.09
0.035
4:000
2.162
44,95
0,066
6:227
92.09
0.373
1,437
40000
33.42
0:30
0.086
Table 14: ImageNet comparison of V-MoE, downstream ensembles there-of, and pBE with 4 experts per input in each case.
IMAGENET ERBOR J
IMAGENET-C
IMAGENET-A ERROR
IMAGENET-V V2 ERROR
DOWNSTREAM AFLOPS J.
K
M
N
ECE
NIL.
ERROR 1
ECE.
NI.L
ECE L
NI..
ECE_ J
pBR
2
0.451
12.61 1m
0.023
1,028 gm
24.90
g0o
0.028
1:00
2.886
58.26
0.171
0.849
21.40 g:10
0.055
g.000
5.92 226.80
L/16
Jowa-DE
4
0,440
12.39
0.015
0,983
23,95
0.020
2.712
56.36
0.120
0
20,83
0,039
V-MoE
1
0.465
12.86
3.3
0.026
0000
1.060 1.00
25.28
0.035
1300
2.976
58.26
0.0
0.186
6000
0.856
1000
21.53
1:37
0.060
1.000
PBE
2
0.529
14.63
0.019
1.188
28.23
0,025
4,095
74.38
0.253
0.962
24.41
0,055
5.65 223.27
L32
down-DE
4
0.518
0002
14.29
2:30
0.022
0000
1.154 2:004
27.47
000
0.023
1:000
4.033
= 0006
73.79 0:00
0.237
= 2002
0.959
24.03 #:33
0.054
50000
V-MoE
1
0.566
15.08
0.041
1,267
29.19
0005
0.053
4.428
75.52
0.306
1,050
25:21
5:335
0.084
PBE
2
0.510
14.13
4:30
0.016
00000
1.328 0:0.0
30.81
0.020
1:0
3,743
e D
71.12
0.211
000
0:945
23.75
5:235
0.044
80000
10.11
B/16
down-DF
4
0.511
2000
13.95
1m
0.019
1,293
1:000
29.67
0,026
1:0
3.544
66.49
0.190
0.930
23.17
0.050
1000
180.69
V-MoE
0.532
14.21
334
0.029
1.0.d
30.44
0:0
0.046
3a0
3.726
66.88
0.238
0002
0.973 0000
23.69
3:300
0.069
PBF
2 2
0.612
16.49 am
0.013 20000
1.491 1::00
33.85
0.019
1:00
4.872
82,80
1000
0.275
1.099
27.36
2:000
0.045
2000
9.14
B/32
down
0.607
16.17 1:300
0.021
1:006
33,36
0.33
0.027
1.000
4.787
$2.11
0.276
0.000
L,199
1.0000
26.9%
1:311
0.055
174.91
V-MoE
1
0.636 0001
16.70
0.034
1.555
2:0.00
34.33
0.051
190
5.031
81.62
0.322
0,02
1,150
27.49
4:300
0.079
20000
PBE
2
0.805
20.97
0.013 20000
2.112 2:0.00
45.05
=:00
0.028
1:0
6.283
93.68 s:m
0.342
0001
1,408
33.71
=:300
0.051
11.73
S/32
down-DE
4
0.795
oues
20.66
1:3
0.015
2.076
2:00012
44.16
0201
0.031
1:0
5,990
92.25 suu
0.324
0.01
1.372
=:000e
32.83
2:235
0.054
50000
143.10
V-MoF.
0.820
20.97
0.030
2.133
44.53
0,060
6.142
91.62
0.365
1.417
33.30
0.081

---

Under review as conference paper ICLR 2022
Table 15: ImageNet comparison of V-Mol and VIT.
IMAGENET ERROR
IMAGENET-C ERROR L
IMAGENET-A ERROR 1
IMAGENET-V V2
K
M
NLL
ECE L
NLL 0
ECE 1
NLL 1
ECE
NLL J
ERROR 1
ECE
H/14
V-MoE
0.426
0.000
11.81
000
0.026
0000
0.932
0.00
22.43 2:06
0.033
2:000
2.494 0:05
50.22 1/
0.153
2000
0.806
20.08 2
0.059
0.000
0.000
11.99
0.5
0.023
0,00
0000
22,45
2:000
0.031 20000
200
0.149
22
0.788
0.058
0.000
I/16
V-MoE
0,464
12.91
0.022
1,50
25.19
0.030
0:000
2,914 6:00ss
58.02
0.167
0,844
21.42
0.053
39
VIT
0.473
0.004
13.37
0.020
1.114
26.68
#:0.7
0.032
3,094
60.92
0.203
0.864
22.04
0.055
L/32
V-MoE
0.559
1.000
15.10
000
0.036
1.246
29.02
cum
0.046
0.000
4.320
1.0.03
75.25
0.288
1
1.032
DUNCS
25.13
2.00
0.077
6000
VIT
0.556
0.000
15.38
0.025
1000
1.255
(UXM
29.57
4: c.t
0.038
0000
4.286
0:0.21
75.32
0.287
1.002
25.32
2.00
0.067
6300
B/16
V-MoE
0.533
= 0.000
14.33
0:4
0.026
0000
1,355
30.59
4:0.0
0.040
00000
3.727
#:0.544
67.72
0.222
:: 6000
0.965
30000
23.63
=:1.1
0.060
6.00
VIT
0.565
15.70
0.021
1,515
34.52
0.042
4,219
75.85
0.280
1.020
25.77
0.061
B/32
V-MoE
0.642
10000
16.90
0.029
1,568
34.68
0,045
5,039
82.49
0.307
1.151
27.83
0.071
VIT
0.688
18.65
0.022
1.689
38.02
0.045
0.000
5,358
0:0004
87.00
0,342
1u:0
1.209
29.89
0:30
0.067
0.003
S/32
V-MoE
0.834
0.00
21.43
0.029
1. not
2.171
J. Doons
45.44
1.000
0.056
3.000
6.199
1.0.03
92.47
0.355
J.no
1.433
33.84
0.078
0.000
VIT
0.907
23.69
0.016
0.00
2.309
48.75
0.055
6.639
0.375
1.529
36.42
0.066

---

Under review conference paper ICLR 2022
Table 16:
Cifar10 comparison of V-MoE, downstream ensembles there-of, and pBE with experts
per input in each case.
CIFAR1O ERROR
CIFARIO-C ERROR
DOWNSTREAM
M
NLL
ECE
NLL
BCE
6.74 AFLOPs (%)
PRF
2
0,022 1:0
0.55
am
0
0.198
gear
5.68
0.027
2
0.026 1300
0.00
0.004
0.227
6.18
020
0.032
V.MoE
2
0.028 1.30
0.61
1.00
0.004
1000
0.220
1012
6.17
023
0.032
0000
PBE
0.026 1:300
0.003
0.246
7.01 6015
0.033
L/32
down-DE
2
0.034 1:300
0.81
0.005
0000
0.252
7.00
0:2
0.032
0000
85.29
V-MoE
0.042 3:30
0:00
0.006
0,325
8.05
0.048
PBE
2
0.036
3:3
0.84
0:002
0.005
00000
0,295
1:00
8.28
0.038
0002
12,60
B/16
down-DE
0.030
0.81
2:900
0.003
0.247
7.60
0.028
75.05
V-MoE
0.040
3:00
0.97
0.002
0.006
0000
0,303
8,08
0.0
0.047
0002
2
0.041
2:00
0.004
0.340
10.01
0,040
11.53
B/32
down DE
0.036
1:300
1.07
1.005
0.003
1000
0.345
10.87
0.0
0.042
1002
73.59
V-MoE
0.042
1.11
0.006
0.405
11.30
0.063
PBE
2
0.060
1:0
1.92
1:000
0.003
00000
0.476
15.30
1
0.049
0000
15.87
S/32
down-DE
0.056
1.75
0.005
0.491
15.42
0.064
64.49
0.058
0.007
0000
0.514
15.48
0.076
Table 17: Cifarl0 comparison of V-MoE, downstream ensembles there-of, and pBE with 4 experts per input in each case.
CIFAR10 ERROR
CIARRO-CC ERROR
DOWNSTRLAS AFLOPS (5)
M
NLL.
ECE
NILL
BCE L
2
0.026
3:0
0.003
0.247 1101
6.56
0,036
5.92
1/16
down-DE
0.025
1:19
0.57
1:00
0.004
100
0,219
2100
6.12
2014
0.030
2000
226.80
V-MoE
0.034
0.75
0.005
0,278
7.71
0.043
PBE
2
0.033
190
0.80
2:00
0.004
0080
0,242
2100
6.68
0.031
2000
5.64
L/32
down DE
4
0.030
0.78
0,004
0.232
6.74
0.028
223.27
V-MoE
0.035
1.0
0.81
1:000
0.005
1008
0,295
k:w
7.63
2011
0.044
0.00
PBE
2
0.034
0.81
0.004
0.283
8.06
0.037
10.11
B/16
Jown-DE
0.029
8:3000
0.80
8:005
0.003
1000
0.242 1M
7.54 2::3
0.026
0000
180.69
V-MoE
0.032
0.79
0.005
0.266
7.55
0.040
PBE
2
0.034
1:3000
1.00
8:002
0.003
0000
0,306
1:00
9.44
0:23
0.037
0002
9.13
B/32
down-DE
4
0.035
1.06
2:00
0
0,338
2196
10,77
2000
0,040
174,91
MoM
0.042
1.3
1.002
0.006
0000
0,410 L1877
11.29
0.066
0000
PBE
2
0.058
1.82
2:001
0.004
0.472
2tom
15.01
2:0
0.051
11.73
0.055
1.0
1.001
0.004
10000
0.486
15.36
1.0
0.061
0000
V-MoE
0.059
3.00
1.80
0:005
0.007
0000
0.536
1.12
15.64
0.22
0.084
0.00
Table 18: Cifar comparison of V-MoE and ViT.
CIFAR10 ERROR
CIFARIO-C ERROR 4
K
M
NLL 4
ECE
NLL 4
ECE 1
L/16
V-MoE
1
0.029
0000
0.59 am
0.004
1:00
0.236
1: om
6.25 2:m
0.034
00000
VIT
0.034 0:00
0.69
cm
0.005
0.0000
0.325
6020
7.53
0:0.0
0.050
0004
1/32
V-MoE
1
0.040
0.00
0.86
0.02
0.006
0:0000
0.290
a: 0013
7.35
0:022
0.043
0.03
VIT
0.030
0.00
0.78
mm
0.005
1:0000
0.281
à oome
7.28 1015
0.043
0.00
B/16
V-MoE
I
0.030
0000
0.85
0.003
0:000
0.249
0.04
7.53
0.030
VIT
0.031
6.00
0.87
0.004
+00000
0.293
0.099
7.99
0.0.5
0.043
0.03
B/32
V.M
0.038
1.14
0.004
0:00
0.359
11.09
0.046
ViT
0.050
6.000
1.38
0.007
0.000
0.330
8.97
0.10
0.051
S/32
V-MoL VIT
1
0.059
6-000
1.84
0.006
0.497
6.003
15.53
0:20
0.066
0.065
ovoon
2.08
0.006
0.518
15.04
0.072
40

---

Under review as conference paper ICLR 2022
Table 19: Cifar 10 OOD comparison of V-MoE, downstream ensembles there-of, and pBE with 2 experts per input in each case.
CI vs. CIFARIDO
CIFARIO DED
CIFARID vs. PLACES365
CIFARIO SVHN
K
M|
AUC (PR)
AUC (ROC) 1
FPR 95
ATIC (PR)
AUC (ROC) 1
95
|UCC (PR)
AUC (ROC) ?
FPR J
ACCC(PR)
AUC (ROC)1
6PR
PBE
0.9905
0
0,0313
0,9899
0,9993
0.0005
0.9103
0.9936
0,0294
0.9963
0.9978
0,0007
0,9896
0.9896 -0305
0.0357
0.9999
0.9996
0.0001
0.9421
10000
0.0162 1.000
0.9982
-0000
L-0033
V-MoE
2
0,9895
1.00
0,9890 000000
0.0389
000003
0,9998
c.00
0.9988 1100
0.0011
0000
0.7891
1000
0.9887
00000
0.0435 1
0.9966
53000
0,9980
00000
0.0005
0000
PDE
0.9675
1.000
0.9873
00000
0,0427
1000
0.9998
c.00
0.9990
10000
0
0000
0.9196
0.9934
0002
0.0321
c303
0.9950
83000
0,9970
L0200
0.0012
L/32
down. DE
0.9852
0.9842
0.057]
0.9994
0.9966
0.0032
0.7388
0.9846
0.0561
0.9939
0.9960
0.0012
V-Mol.
0,9539 02
0.9839
2.00
0.0589
1/000
0,9998
0.9987
0.0013
2500
0,8967
2/000
0.0346
0.9944
0.9965
2/000
0.0017
PRF down-DE
0.9823 0 9m3m
0.9800
0,0846
g/mms
0,9994
0.9967
0.0015
0.7661
0.9834
1000
0.8602
0,9014
2/100
0.9942
0.0029
LOORS
B/16
0,9860
0,9859
0,0540
0,9999
0,9993
0.0009
0.8807
0,9922
040388
0.9951
0,9974
0.0027
V-MoE
0.9622
0,9809
0,0800
0,9994
60000
0.9964
0.0023
8.000
0.7399
0.9841
00000
0.0568
0.04
0.9925
83004
0.9953
0.0037
100000
PDE down. DE
0.9773 1.300
0.9738
6.000
0. 0.1254
0,9992
8.0000
0.9957
4.000
0.0038
4000
0.7438
0.9794
00005
0.0802
c
0.9881
30000
0.9918
600000
0.0071
B/32
0.9820
0
0,0810
2-0003
0.9998
0,9989
0.0014
0.8567
0.9900
0.0514
0.9924
5000
0.9957
0.0054
V-Mol.
0.9798
0.9790
0.0675
0.9997
0.9986
0.0611
0.8725
0.9904
0.0482
0.9919
0.9952
0.0047
PRE
0.9617 2:300
0.9567
20000
0.2629
1-000
0,9992
0.9956
0.0118
2000
0.7431
2/00m
0,9750
2.000
0,1230
0.0824
0,9899
2/000
0,0388
8/32
dowa-DE
0.9691 1:000
0.9669
20000
0,1856
2/00044
0.9996
000000
0.9UM1
1:000
0.0037
2.00
0.8135
0.9845
0.0815
c.00s
0,9857
2000
0
0.0164
200000
V-MoE
0,9691
0,9667
0,1870
0.999
0,9979
0,0043
0.5010
0,9839
0,0827
0.9873
0.9927
0.0192
Table 20: Cifar 10 OOD comparison of ViT and V-MoE
CIFARLO vs. CIFARIOO
CIFARIO DTD AUC (ROC) 1
CIFARIO PLACES365
CIFARIO SVIIN
M
AUC (PR) 7
AUC (ROC) 1
FPR695
AUC (PR) f
FPR
AUC (PR) f
AUC (ROC) 1
FPR
AUC (PR) 1
AUC (ROC) 1
EPR G:95
1/16
V-MoB
VIT
0,9891
0.9839
00008
0,9895
0.9845
0,0379
0.0541 00003
0,9999
0.9996
0000
0,9997
0.9978 0300
0,0001
0.0018
0.7334
0,9400
1.0022
0,9960
0.9857 33000
0.0162
0000
0.9947
0,9972
2000
5.0000
0.9984
0.9967
-3002
0.0007
0244
L/32
V-VaB
0.9854
2000
0.9850
0.0573
2000
0.9996 2:
0:99776 2m
0.0030 g00m
0.7489
0.9862 1m0m
0.0520
g00m
0.9946
20000
0.9967
2:00
0,0015
VIT
0,9842
0.9847
0,0551
0,9998
0.9987
0,0016
0,9164
0.9938
0.0279
0,9942
0.9966
0.0028
B/16
V-MoB
VIT
0.9856
0.9801 50000
0.9855
0.9798
0.304
0,0857
0002
0,9998
0.9996
00000
0.9979
0.9992
6.300
0,0046
0000
0.83M8
0.8536
000007
0.9912
0.9895
-.000
0.0511
0002
0.9926
0,9949
000000
0.9972
0.9961
1300
0.0027
0.0048
D2NES
B/32
V.MoE
VIT
0,9814
0.9752
32000
0,9806
0.9716
0. 14445
0,0853
600227
0.9985
0,9998
0,9989
0:9915
0.300
0,0017
0.0186
0.6507
0.8590
000000
0,9902
0.9734
5.3005
0. P.1021
0,0506
0000
0.9872
0.9923
100000
0.9958
0.9920
1.000
0.0061
0.0148
8/32
V-MoB
0,9685
000008
0.9658
8.300
0.1922
000005
0,9996
600000
0.9977
-13000
0,0045
000000
0.8092
80000
0.9841
23300
0
00002
0.9874
000000
0.9929
1.00
0.0185
0.000
VIT
0.9629
0.00
0.9588
0..004
0,2422
600022
0.9993
0.9963
0.0134
000033
0.7177
0.9775
-13000
0.1110
00015
0.9807
4.0000
0.9689
0.0426

---

Under review as conference paper ICLR 2022
Table 21: Cifar 100 OOD comparison of V-MoE, downstream ensembles there-of, and pBE with 2 experts per input each case.
CIFARIOO 10D CIFARIO
CIFAR vs. DTD
CIFARIOO vs. PLACES365
CIFAR SVHN
K
M|
AUC (PR)
AUC (ROC) 1
FPR J
ATIC (PR)1
AUCC (ROC) 1
95 L
(PR)
(ROC) ?
FPR J
ACC(
AUC (ROC)
FPR
PBE
0.9507
0,9490
0.2559
0,9969
0,9647
0.0862
0.728]
0,9542
0.2969 c300
0.9011
0.9452
0.3071
DE
0.9455
0.9481 -0001
0,9881
0.0665
0.7619
10005
0.9664
40002
0.2158 cmm
0.9507
V-MoE
2
0 0.004
0.9443
0001
0.2901
10007s
0.9977
C.00
0,9887
1030
0.0674 0001
0.7773
1000244
0.9689
10000
0.2150
0.9002
33004
0.9461
000024
0,3223
00024
PDE
0,9423
1.000
0.9379
1000
0.3591
0.9958
0,9792
1:00
0.1165 000
0.6653
1.0027
0.9394
0000
0.3670
0.5548
50000
0.9398
0.3428
0000
L/32
down. DE
0.9380
0,9387
0.3362
0.9958
0.9796
0.1176
0.6710
0.9436
0.3465
0.8877
0.9460
0.3033
V-MoR.
0,9275 11300
0,9330
0.3004
10007
0.9958
0.9790
0.1306
0.6773
2/000
10007
0,3489
1:
0.9399
2/100
0.3404
2.000
PRF down-DE
0.9200 1:300
0.9121
0.4656
1mmm
0,9938
0.9710 0:00
D. D.1542
0.5475
g/1mm
0,9102
1000
0.4653 mmm
0.8730
0.9319
0.3699
2.000
B/16
0.9242
0,9275
0,3596
0.9952
0.9777
0,1144
0.6023
0,9343
0.3577
0.8726
0,9340
0.3526
V-MoE
0.9159
0
0,3729
0/00223
0.9950
0.9768
1.300
0.1193
0.6029
0.9331
0000
0.3682
(3K0)
0.5704
0,9283
000022
00007
PDE down. DE
0.9075 8.002
0.8945
80015
0.5358
00000
0,9906
c.00-
0.9554
0:000
0.2481
48000
0.4355
1-000L
0
00000
0.5465 c.00
0.8583
80006
0.9221
L-0024
0.4051
000000
B/32
0,9188
0.9158
01011
0.4248
0.9942
0.9719
2:00044
0,1517
0.5528
0.9176
0.4276
0
0.9292
0.39.59
V-Mol.
0.9192
0,9151
0.4444
0,9580
0.2481
0.4449
0.5230
0.8641
0,9252
0,4249
PRE
0.8677 2000
0.8528
200011
0.6086
1/000
0.9878
0.9430
0.2677
2:00
0.3628
2.007
0,8519
2000
0.5716 cans
0.8279
0,9004
2/0074
0,4543
2.0000
8/32
dowa-DE
0.8697 8:00
0.8651
2000
0,5203
2:005
0,9892
0.005
0,9532
0.1909
6000
0.4185
0.8814
0.4657
camm
0.8162 3/004
0
0.4K35
V-MoE
0,5687
0,8663
0,5206
0.9894
0,9523
0,2076
0.4129
0.8752
0.5033
0.8133
0.8552
0.5031
42
Table 22: Cifar 100 OOD comparison of V-MoE and VIT
CIFARLOO vs. CIFARIO
CIFAR100 vs.
CIFAR100 Vs. PLACES365
CIFARIOO SVIN
M
AUC (PR) 7
AUC (ROC) 1
FPR
AUC (PR) f
AUC (ROC) 1
FPR
AUC AUC((R) f
AUC (ROC) 1
FPR
AUC (PR) 1
AUC (ROC) 1
EPRG95
1/16
V-MoB
VIT
0,9411
0,9454
0000
0,9481
0,9449
0202
0.2682
0.2734 00003
0.9970
0,9976
000000
0,9882
0.9554
0,0658
0,0853 0000
0,7631
0.7552
0.00
0.9667
0.9608
33047
0.2141
0.2566 0000
0.8697
0,9115
000000
0,9533
0.9326
0.2794
0200
L/32
V-VaB
0.935M
0.9368
2130
0,3431
20000
0.9961
0.9806 11
0.1148 2000
0.6757
2.000
D.9461
1
0.3308 2000
0.8863
0.9459 110m
0,3063 0007
VIT
0,9285
0.9323
0.3201
0.9967
0,9833
0
0,7541
0,9634
0.2292
0,8495
0.9234
0.3949
B/16
V-MoB
VIT
0.9206
0,9177 00013
0.9241
0.9171
0.004
0.3572
0,3925
8.000
0
0.9905
00000
0.9776
0.9571
6.002
0,2199
0000
0.4913
0.5924
5:007
00088
0,9334
0.8980
1.304
0.3532
0.4927
00005
0,8720
0.8525
500044
00005
0.9309
0.9204
-3002
0,3705
0.4226
02122
B/32
V.MoE
VIT
0.9038
0.9166 32017
0.9145
0.9044
L0302
0.4211
0.4180
6-0043
0.9927
0.9940
000000
0
0.9663
0.1631
0,1562
0.5306
0.5433
0.9112
0,9178
3.3066
0.4258
0.4176
<0owo
0.8730
0.8379
6.00m
0.91
0.9276
0.4026
0.4328
00100
8/32
V-MoB
0.8678
000812
0.8631
0.305
0.5281
8000003
0,9893
8.0000n
0.9524
6.300
0.1978
800022
0.4024
0000
0.8778
-3300
0.4763
0.8224
000005
0.8945
-30044
0.4729
00005
VIT
0.8644
02011
0.8541
1.3000
0,5716
0.9836
0.9287
-1300
0.2976
60002
0.3144
0,8349
-1.004
0.5982
0,8068
0.8894
0,4568
60.004

---

Under review as conference paper ICLR 2022
Table 23: Few-shot comparison of pBE, V-MoE and ensembles thereof.
MEAN ACROSS DATASETS
WEIGHTED MEAN ACROSS DATASETS
DOWNSTREAM SFLOPS (%)
1-SHOT ERROR
5-SHOT ERROR
10-SHor ERROR
25-SHOT ERROR
1-SHOT ERROR
5-SHOT ERROR 10-SHOT ERROR 4
25-SHOT ERROR
pBE
31.47
17.87
14.29
10.64 c:3
78.51
80.65
81,28
81.51
15:45
down-DF
30.84
100
200
2000
78.36 1.
2014
2:00
V-MoE
2
32.47 2002
18.77 2:04
15.19
12.09 cM
78.93
80.89
2.11
81.48
81.83 2:00
PBE
34,08 2002
19.57
15,21
11.75 cc
79.50 8:07
81,09
81,48
81.76
6.74
L/16
down DE
32.98 100
19.65 1o0
15.44 1
11.92 6:2
79.09 2.6.3
81.10 2::0
81.54
81.79 1:001
86.03
V.Mol.
2
33,98 1
20.42
16:20
12.73
79.50
81,30
81,72
81.97
PBR
36.71 1000
2005
1000
2.007
81.77 2::0
200
2:00
L/32
down DE
36.15 0.22
22,18 0.0
17.81 1.014
14,00 c3a
S0.45 1.3
81.79 6.04
82.13
6.03
K2.27 1.000
85.29
MoM
36.56
23.00
18.86
15.02
801.6
82,00
82.37
82.50
PBE
38.60 1.0/
21.93 3,00
17.43 1.003
13.62 g.
81.14 6:6
81.74 2:40
82.05 2.03
82.19 1/00t
12.61
B/16
down- DE
38.96
23.36
18.99
14,84 c:1
$1,39 c.0
82.09
82,40
82.46
75.05
MoE
37.76 ous
80.97
82.48
PBE
2
43.39 1.0
26.51 1.001
21.18 0.7
16.82 1.9
82.86 C:1
82,91 6.4
82,93
6.44
82,90 1-0/2
11.54
B/32
down-DE
41,09 4:00s
26.48
17.20
82.24 2:035
82,89
83.03
82.99
73.59
M-MoE
42.50 azo
27.44 0.0
22.73
0.0
18.60 8.2
82,66 c:4
83.16
83.29
-685
83.29 2:004
PBE down-DE
52.91
34.28
27.54
21.85
85,89
84.81
$4.39
84,01
15,88
S/32
48.27 am0
2000
21.95 PM
84.62 1
84.04 2:00
64.50
V-MoE
2
49.37 au
33.51
28.00
0014
23.25 833
85.04 0.00
84.69
84,54
84.33
2/00
Table 24: Few-shot comparison of V-MoE and ViT.
MEAN ACROSS DATASETS
WEIGHTED MEAN ACROSS DATASETS
M 1-SHOT ERROR
5-SHOT ERROR
10-SHOT ERROR J
25-SHOT ERROR
I-SHOT ERROR 1
5-SHOT ERROR 10-SHOT ERROR
25-SHOT ERROR J
V-MoE
33.04 600
19,23
15.63
12.06 uus
79.05 4:322
81.00
81.59
81.82
H/14
VIT
35.78 and
20.61 1015
16.78 and
13.62 g0M
79.97 2014
81.37
81.87
82,16
L/16
V-MoE
34.15 1
20.32 1.0
16.14 1.n.
12.71 2
79.54
1:000
81.27 1. 1m
81.71 1am
81.97 1.
VIT
36.52
21.79
17,51
14:07 2004
80.38 0:0
81.67
82.03
82.27
1,32
V-MoE
36.83
23.05
18.78
14.95
80.67
0.000
82.01 Lu
82.35
82.48
VIT
38.22 0:1
23.97 2014
19.56 #0.0
15.75 20.4
81.10 0:000
82.25 D.0
82.53
608
82.65 6.0
B/16
V-MoE
39.22 bzr
24.42 4:033
20.01 +am
15.94 u22
81.47 a:02s
82.36 +t bus
82.64
it bubs
82,70
VIT
41.29
25.03
20.08
15.87 2007
82.16
82.51
82.65
82.68
B/32
V-MoE
42.37 #:031
27.60 a:0.0
22.89 2:0.1
18,46 =0000
82.64
#:001
83.19
83.33
83,26
VIT
44.60 Loz
28,20 1.0.1
22.97 1.0.1
18.86 0:5
83.35 1.00
83.35
83.35
83.35
S/32
V-MoE
49.60
33.34
27.88
23.30
85.09
0.00
84.64
84.50
84.33
1.D0
ViT
54.16
0:16
36.25
29.88
24.42 203
86.32
0:05
85.26
84,90
84,54
6008

---

Under review conference paper ICLR 2022
I FROM BATCH ENSEMBLES TO SPARSE MoEs
Wen et al. (2019) have shown that, given a batch of B inputs X E BXPP a single forward pass can efficiently compute the predictions of all the ensemble members ( mmm]m= By appropriately tiling the inputs of the network Xiled E IR BX by a factor M, each internal operation per ensemble member can then be vectorized.
We take the previous example of a dense layer with parameters U E RDD and we assume the layer where H.. RR We need to compute for each ensemble member Hm Um = Hm U (Tm3m))) Denoting by ham E R the i-th input in H.. we have
receives the tiled imputs Hm 1
-=1 where
F. T Un = = Da (hemm)) expert.(h..)) with M E, { (be.)) = 1 if e m, (ge(h),m) = 0 otherwise
(5)
and expert,(z) I (r. Although (5) may appear convoluted way of writing the operations in batch ensembles, unveils connection with (I). Indeed, operations in batch ensembles can be seen as a specific sparse MoE, e.g. with binary routing weights depending only on the position in the tiled inputs. While Wen et al. (2019) primarily tiled the inputs for the sake of efficiency, it also induces some form of conditional computation, an insight that exploit here.
J NEW RESULTS FOR THE REBUTTAL
J.1 NEW RESULTS FOR THE FEATURE- LEVEE VERSUS PREDICTION-LEVE ABLATION
Table 25 shows updated and expanded results for the feature-level versus prediction-level ab- lation, described in Section 3.2. The original conclusion-ice ensembling at the prediction level is helpful for calibration but that the naive multi-head approach provides worse error, NLL, and diversity-Seems to hold for the K- 8 case.
Note that the results for K 2 and K differ slightly from the original version in Table 2 due a change in experimental setup. In the original version, we matched K for the up- and downstream models in the multi-head case (i.e., the pretrained and fine-tuned models both use either K 2 or K 4). We did this be as fair as possible the multi-head model. While it known, from Riquelme et al. (2021), that V-MoE models are fairly robust to the choice of upstream model (e.g.. KuPSTREAM can be set to 2 and we can just change K DowNSTREA. which is the approach we take throughout our paper-see see Appendix C), this was not known for the multi- due to time constraints we are unable to train an upstream model with K = 8. Thus, for the updated results we have opted for single upstream model with K = 2 in all cases. As side observation, we can notably observe that the multi-head variant is more sensitive than V-MoE the choice of the upstream model, c.g., the performance worsens from (Kupstream KpOwNStREAM) = (4, 4) to ( (KUPStREAm. DOWNSTTEAM) (2, 4).
model. Unfortunately,
Table 25: New feature-level vs. prediction-leve ensembling ablation results. ImageNet performance of V-MoE and a naive multi-head variant (means + standard errors over 8 replications). All models have VIT-B/32 architecture. For the multi- variant the last MoE layer is modified as in (2)
K
NLL
ERROR 1
ECE
KL 1
V-MoF
2
0.638
16.76
0.033
200
Naive Multi-head
2
0.636
0.001
17.16
0.00
0.024
0.000
0.032
:: 0000
V-MoE
4
0.636
ovoon
16.70
0.034
100
Naive Multi-head
0.645 4
0.001
17.39
0.00
0.021
0.000
0.011
0.001
V-MoE
8
0.635
0.002
16.72
0.028
Naive Multi-head
8
0.650
0.001
17.50
0.08
0.021
0.000
0.005
J.2 NEW RESULTS FOR THE STATIC VERSUS ADAPTIVE ABLATION
Figure 20 repeats the experiment of Figure 2a with two new random seeds. The new results show smoother improvement in I.I. as K increases. In particular, we see that there is nothing special
44

---

Under review conference paper ICLR 2022
about K or K 6. This indicates that the anomalous drop at K 6 in the original results was simply due to noise in the training of the upstream model. We note that, like Riquelme et al. (2021), we have observed that training noise is more pronounced in the smallest (i.e., S/32) models.
0076
M
K
Figure 20: Replication of Figure 2a, averaged over two new random seeds, showing the effect on LL of increasing static (M) and adaptive (K) ensembling. ImageNet performance for ViT-S/32 models. Yellow indicates better performance; purple indicates worse performance.
J.3 NEW LOAD BALANCING ABLATION
Figure shows a new experiment exploring the effect of the load balancing loss on diversity and predictive performance. We see that for both V-MoE and pBE the predictive performance, as mea- sured by NLL, Error, and ECE are largely insensitive to the strength of the load balancing loss. Similarly, the diversity of the predictions of pBE measured by the KL ELilil depends on that regularization. Only when the load balaneing loss strength hecomes excessively large (4 orders of magnitude larger than standard values) do all the performance metrics plummet. This hyperpa- rameter does not allow us increase diversity and thereby the predictive performance our models.
J.4 NEW EXPERT INITIALIZATION ABLATION
Figure 22 shows a new experiment exploring the influence the initialization of the experts. In particular, we add Gaussian noise with varying standard deviations to the initial weights of the ex- pert MLPs. We notably show that more diverse initializations (larger standard deviations) do not translate to any clear performance gain. Note that this new experiment takes place upstream, since downstream the experts are already initialized (we just fine-tune them from the upstream check- point).
J.5
NEW PARAMETER COUNTS TABLE
Table 26 compares the parameter counts for ViT and V-MoF/pBE models in each VIT family.
S/32
B/32
B/16
L/32
L16
H/14
VIT
36,5M
102, 1M
100,5M
325.3M
323, 1M
655.8M
V-MoR/pBE
166.7M
395.0M
393.3M
845.8M
843.6M
2688.6M
Table 26: Parameter counts for VIT V-MoE and pBE.
45

---

Under review conference paper ICLR 2022
1.00
24
V-MoE (K
V-MoB (K-1)
NLL (lower is better)
(lower better)
0.90
pBE (K-1, M-2)
22
0.85
PBE (K=2, M-2)
21
20
0.75
19
18
17
0.60
16
log1o(A)
log1o(A)
ECB (lower bettes)
KL (higher is better)
0.04
0.20
0.19
0.18
0.17
0.16
-2
2
log1o(A)
lag1a(A)
Figure 21: The effect of x, which controls the strength of the load balancing loss as described appendix A of Riquelme et al. (2021), NLL, Error, ECE and diversity, for pBE and V-MoE. The resuls are averaged over three random seeds. All models have VYT-B/32 architecture.
46

---

Under review conference paper ICLR 2022
ImageNet 1-Shot Error Eror (lower is better)
ImageNet 5-Shot Error (luwer is helter)
0.55
0.48
V-MoB-S/32
Initialization standard deviation
Initialization standard deviation
ImageNet 10- Shot Error (lower is better)
ImageNer 255 Shot lirorr (lower is helter)
0.60
0.55
0.50
10-8
10
10-4
10-7
Initialization standard deviation
Initialzzation standard deviation
Figure 22: The influence of the noise standard deviation of the expert MLPs' initial random weights. The models are trained on JFT-300M we measure ImageNet few-shot Error as in Riquelme et al. (2021). )). Results are for a single random seed.


Reviews:
[{"review_id": "yLMXq6Qoeyz", "reviewer": "Reviewer_cawf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\nThe paper provides confirmation that MoEs and ensembles have complementary (or additive) benefits. Moreover, the proposed approach (partitioned batch ensembles), which combines MoEs and ensembling, achieves the benefits of both techniques simultaneously. \n\nThe paper provides the first study of the robustness of vision sparse MoE’s (parallel submission).\n\nWeaknesses:\n\nThe proposed approach is incremental. In the words of the authors, the method “introduces two changes to V-MoEs: (a) the partitioning of the experts and (b) the tiling of the representations.”\n\nThe paper focuses on the comparison of specific architectures and training procedures. In this regard, conclusions one may be able to draw from the paper are too constrained. For example, can the benefit of “static” ensembles be achieved by “adaptive” ensembling? While this paper’s results suggest perhaps not, I don’t think this is conclusive. [The distinction between static and adaptive may be generalized in such a way as to enable finer control on aspect and degree of adaptation with implications to, e.g., the diversity and uncertainty calibration at the output of the ensemble.] Further, e.g., (Wen et al. 2020) claim that “diversity of initialization entirely determines the diversity of ensembling system.” The present paper notes ways to achieve diversity given a particular distinction between static vs adaptive ensembling, and training procedure. There are other dimensions (e.g., initialization, example balancing) which have direct relevance to diversity and thus, uncertainty calibration, which are not explored in this paper.\n\nSection 3.1 is an ablation study just like section 4.2 is. I find the discussion going back and forth to be detrimental to clarity and efficiency of communication.\n\nSection 3.2 compares a naive multi-head method -- without specifying how it is trained -- with a “standard” vision (sparse) MoE. I don’t find the results there clear enough to be compelling. For example, whether the multi-head variant is able to provide diverse predictions is more likely to do with the training procedure than with the architecture -- this is related, e.g., to the “load balancing” during training used in (Riquelme et al. 2021). Besides, the claims in section 3.2 are conflicting: we are told naive multi-head improves ECE but also that a different strategy is required for uncertainty calibration.\n\nThe experimental results (section 5) are simply stated. For example, most of the time (static) ensembling helps but when it doesn’t (e.g., out of distribution performance) the paper provides no discussion or insight.\n\nSome acronyms are used without introduction (the introduction appears after the first usage), e.g., OOD and NLL in page 4.\n\nIt’s not clear to me how the static ensembling portion of the proposed approach is “efficient” (section 4.1). For example, Batch Ensemble (Wen et al. 2020) generates the family of ensemble weights via element-wise multiplication of a shared matrix and “fast-weights.” In my understanding, in the case of the proposed approach any sharing of parameters is done within the ensemble members.\n\nHow are the dashed lines in figures 4 and 5 pareto frontiers? ", "summary_of_the_paper": "The paper investigates the benefits of combining (Sparse) Mixture of Experts (MoE) and ensembling. Sparse MoE’s employ conditional computation to reduce computational and environmental costs of DNNs while maintaining (or increasing) performance. On the other hand, ensembling models has been shown to achieve the highest robustness in the presence of dataset shift, e.g., higher accuracy and better estimation of uncertainty.\n\nThe present submission empirically demonstrates that Sparse MoE’s can be ensembled to attain the benefits of both techniques simultaneously, i.e., conditional computation (with its implications to scalability) together with robustness in the presence of dataset shift. The main components of the approach are: 1) Disjoint MoE’s as ensemble members. 2) Tiling of representations which enables all ensemble members to compute the output for a batch in a single forward pass.\n\nA number of experiments compare predictive performance vs computational cost for the proposed approach (pBE), vision Sparse MoE’s (V-MoE) and Vision Transformers (ViT). The results show that, under some conditions, pBE displays the known benefits of Sparse MoE’s and ensembling, which leads to performance gains w.r.t. ViT and V-MoE under metrics like accuracy, negative log-likelihood (NLL) and expected calibration error (ECE).", "summary_of_the_review": "Overall I lean towards rejection. While the paper presents experimental results likely not found elsewhere, in my perception, that appears to be the sole reason for the paper. Specifically, the paper presents experimental results on the robustness of V-MoE’s (vision sparse MoE’s) and adds ensembling on top of the V-MoE model to increase robustness. However, no further insight or development (e.g., in architecture or optimization) is provided.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bfuB_Cnnwm6", "reviewer": "Reviewer_KpkQ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. PBEs share the attention layers and use an ensemble of sparse MOEs for the MLP layers of Vision Transformers. Any reasons for this design choice, and not going the other way, i.e. sharing the MLPs and using Sparse MOEs for the self-attention layers ?\n2. Results in Table 2 suggest prediction level ensembling can be beneficial for uncertainty calibration. The Table reports the results for K = 2 and K = 4. Does this claim hold true for K > 4 as well ?\n3. The multi-head MoE presented in Section 3.2 stacks the K selected expert predictions. Are these K predictions averaged in order to compute the NLL and Error scores reported in Table 2 ?\n4. Fig 2 a: For every value of M, the log likelihood score increases till K=5, and drops at K=6. Any reasons behind this ?\n5. The idea of Batch Ensembling and Tiling isn't new, and has been used in previous works [A].\n\n[A] Yeming Wen, Dustin Tran, and Jimmy Ba. Batch ensemble: an alternative approach to efficient\nensemble and lifelong learning. In ICLR, 2019.\n", "summary_of_the_paper": "The authors show empirically that Sparse MOEs and Ensembles have complementary features, and suggest that combining the two should lead to improved performance. Authors build on the Vision Transformer (ViT) for their experiments. To efficiently combine Sparse MOEs and Ensembles, the paper presents Partitioned Batch Ensembles (PBE), where the parameters of the self-attention layers are shared, and an ensemble of Sparse MOEs are used for the MLP layers of the Transformer blocks.\n\n", "summary_of_the_review": "I only have minor concerns with the paper, and I have stated those concerns in the Main Review. Other than that, the paper is novel and the experiments demonstrate the usefulness of the proposed Partitioned Batch Ensembles (PBE).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ENNBBFEvMoC", "reviewer": "Reviewer_5srE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written and the experiments are comprehensive. The main concerns about this paper are as follows.\n1. The paper lacks novelty. It just combines the MoE with Ensemble in a trivial manner. The tiling technique is quite simple and it is just the classical way of doing ensemble across models.\n2. The improvement is quite limited. In Table 3, I find although the proposed method achieves gains across different metrics e.g. NLL, Error, and ECE, the improvement in terms of accuracy is less than 0.5% which is quite small and cannot justify the effectiveness of the method. Furthermore, when compared to ViT and V-MoE in Figure 4,5,6, I can only find the curve for NLL instead of Error, which I think is more critical. \n3. The number of parameters should be reported and compared to ViT, since MoE is a parameter-consuming method and we cannot ignore this factor.\n4. The FLOPs cost by tiling and partitioning is another concern. More details about how to compute FLOPs for the proposed method and V-MoE should be clarified. For V-MoE, it is possible to first compute the coefficient for each expert and then synthesize the experts before extracting features. Therefore, it can save a large scale of computational costs. But due to tiling and partitioning, each branch should cause its own computation, which I think will cost a lot of extra computation and let me doubt the efficiency of the proposed method. I wonder whether the author computes the FLOPs with the optimal way.", "summary_of_the_paper": "This paper considers the combination of MoE and Ensembles for image recognition tasks to obtain both improvement of classification accuracy as well as stability of prediction. The proposed method is evaluated with extensive expereiments from large-scale image recognition to OOD and the results indicate better performance for the proposed method.", "summary_of_the_review": "The paper still has a big space for further improvement. I recommend rejecting it this time.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper experiments with a combination of Sparse MoEs and Ensembles on the Vision Transformer (ViT), showing improved performance. To efficiently combine Sparse MoEs and Ensembles, the paper presents Partitioned Batch Ensembles (PBE), where the parameters of the self-attention layers are shared, and an ensemble of Sparse MOEs are used for the MLP layers of the Transformer blocks.

While reviewers agree that the proposed approach is interesting, they also point out several weaknesses, such as the limited novelty of the proposed method (a simple combination of existing techniques) and small experimental gains. They also pointed out several weakness related to the experimental part. While the authors responded in a very detailed manner to several of these points and presented several additional experiments, I feel this paper will benefit from consolidating all these new results and going through another round of reviews.
Decision: Reject