BatchEnsemble: an Alternative Approach to Efficient Ensemble and Lifelong Learning
Authors: Yeming Wen
Dustin Tran
Jimmy Ba
Venue:  (2020)
Keywords: deep learning
ensembles

Abstract: 
Ensembles, where multiple neural networks are trained individually and their predictions are averaged, have been shown to be widely successful for improving both the accuracy and predictive uncertainty of single neural networks. However, an ensemble’s cost for both training and testing increases linearly with the number of networks, which quickly becomes untenable.
In this paper, we propose BatchEnsemble, an ensemble method whose computational and memory costs are significantly lower than typical ensembles. BatchEnsemble achieves this by defining each weight matrix to be the Hadamard product of a shared weight among all ensemble members and a rank-one matrix per member. Unlike ensembles, BatchEnsemble is not only parallelizable across devices, where one device trains one member, but also parallelizable within a device, where multiple ensemble members are updated simultaneously for a given mini-batch. Across CIFAR-10, CIFAR-100, WMT14 EN-DE/EN-FR translation, and out-of-distribution tasks, BatchEnsemble yields competitive accuracy and uncertainties as typical ensembles; the speedup at test time is 3X and memory reduction is 3X at an ensemble of size 4. We also apply BatchEnsemble to lifelong learning, where on Split-CIFAR-100, BatchEnsemble yields comparable performance to progressive neural networks while having a much lower computational and memory costs. We further show that BatchEnsemble can easily scale up to lifelong learning on Split-ImageNet which involves 100 sequential learning tasks

Published as a conference paper at ICLR 2020
BATCHENSEMBLE: AN ALTERNATIVE APPROACH TO EFFICIENT ENSEMBLE AND LIFELONG LEARNING
Yeming Wen 1.23. Dustin Tran & Jimmy Bal University of Toronto, Vector Institute, "Google Brain
ABSTRACT
Ensembles, where multiple neural networks are trained individually and their predictions are averaged. have been shown to be widely successful for improving both the accuracy and predictive uncertainty of single neural networks. However, an ensemble's cost for both training and testing increases linearly with the number of networks, which quickly becomes untenable.
In this paper, we propose BatchEEnemble', putational and memory costs are significantly lower than typical ensembles. BatchEnsemble achieves this by defining each weight matrix be the Hadamard product of a shared weight among all ensemble members and a rank-one matrix per member. Unlike ensembles, BatchEnsemble not only parallelizable across devices, where one device trains one member, but also parallelizable within a device, where multiple ensemble members are updated simultaneously for given mini-batch. Across CIFAR-1O, CIFAR-100, WMT14 EN-DE/EN-FR translation, and out-of-distribution tasks, Batchen yields competitive accuracy and uncertainties typical ensembles; the speedup test time is 3X and memory reduction is 3X at an ensemble of size 4. We also apply BatchEnsemble to lifelong learning, where on Spli--CIFAR-I00 BatchEnsemble yields comparable perfor- mance to progressive neural networks while having much lower computational and memory costs. We further show that BatchEnsemble easily scale up lifelong learning on Split-ImageNet which involves 100 sequential learning tasks.
ensemble method whose com-
INTRODUCTION
Ensembling one of the oldest tricks in machine learning literature (Hansen & Salamon, 1990). By combining the outputs of several models, an ensemble can achieve better performance than any of its members. Many researchers demonstrate that a good ensemble is one where the ensemble's 's members are both accurate and make independent errors (Perrone & Cooper, 1992; Maclin & Opitz, 1999). In neural networks, SGD (Bottou, 2003) and variants such as Adam (Kingma Ba, 2014) are the most common optimization algorithm. The random noise from sampling mini-batches of data in SGD-like algorithms and random initialization of the deep neural networks, combined with the fact that there is a wide variety of local minima solutions in high dimensional optimization problem (Ge et al., 2015; Kawaguchi, 2016; Wen et al., 2019), results in the following observation: deep neural networks trained with different random seeds can converge to very different local minima although they share similar error rates. One of the consequence is that neural networks trained with different random seeds will usually not make all the same errors on the test set, i.e. they may disagree on prediction given the same input even if the model has converged (Fort et al., 2019).
Ensembles of neural networks benefit from the above observation achieve better performance by averaging or majority voting on the output of each ensemble member (Xie et al., 2013; Huang et al., 2017). It is shown that ensembles of models perform at least as well as its individual members and diverse ensemble members lead to better performance (Krogh & Vedelsby, 1995). More recently, Lakshminarayanan et al. (2017) showed that deep ensembles give reliable predictive uncertainty estimates while remaining simple and scalable. A further study confirms that deep ensembles generally
Partial work done as part of the Google Student Researcher Program. Email: ywen wwwnncc.torooto.ccin Phttps:/ //g thub. com/ /oog] /ddward2

---

Published as a conference paper at ICLR 2020
achieves the best performance outtof-distribution uncertainty benchmarks (Ovadia al., 2019;
Gustafsson et al., 2019) compared to other methods such as MC-dropout (Gal & Ghahramani, 2015). Despite their success benchmarks, ensembles are limited in practice due to their expensive com- putational and memory costs, which increase lin- early with the ensemble size in both training and testing. Computation-wise each ensemble mem- ber requires separate neural network forward pass of its inputs. Memory-wise, each ensemble member requires an independent copy of neural network weights, each up millions (sometimes billions) of parameters. This memory requirement also makes many tasks beyond supervised learning prohibitive. For example, in lifelong learning, natural idea is to use a separate ensemble member for each task, adaptively growing the total number of parameters by creating new independent set of weights for each new task. No previous work achieves competitive performance lifelong learning via ensemble methods, as memory is major bottleneck.
Relative cost
30
testing cost
memory cost
20
naive
0
10
15 20
25
30
Ensemble size
Figure 1: The test time cost (blue) and memory cost of BatchEnsemble (orange) w.r.t the ensem- ble size. The result is relative to single model cost. Testing time cost and memory cost of naive ensemble are plotted in green.
this paper, we aim to address the computational and memory bottleneck by building a more parameter efficiintt-ensmmlle method: BatchEnsemble. We achieve this goal by exploiting a novel ensemble weight generation mechanism: the weight of each ensemble member is generated by the Hadamard product between: one shared weight among all ensemble members. b. one rank-one matrix that varies among all members, which we refer to as fast weight the following sections. Figure 1 compares testing and memory cost between BatchEnsemble and naive ensemble. Unlike typical ensembles, BatchEnsemble is mini-batch friendly, where it not only parallelizable across devices like typical ensembles but also parallelizable within device. Moreover, it incurs only minor memory overhead because a large number of weights are shared across ensemble members.
Our contribution:
Empirically, we show that BatchEnsemble has the best trade-off among accuracy. running time, and classification
memory on several deep learning architectures and learning tasks: with ResNet32 (He et al., 2016) and WMT14 EN-DE/EN-FL machine translation with Trans- former (Vaswani et al., 2017). Additionally, we show that BatchEnsemble is effective calibrated prediction on out-of-distribution datasets; and uncertainty evaluation contextual bandits. Finally, we show that BatchEnsemble can be successfully applied in lifelong learning and scale up to 100 sequential learning tasks without catastrophic forgetting and the need of a memory buffer. Section further provides diversity analysis as tool to understand why BatchEnsemble works well in practice.
2 BACKGROUND
In this section, describe relevant background about ensembles, uncertainty evaluation, and lifelong learning for our proposed method, Bascbbne
2.1 ENSEMBLES FOR IMPROVED PERFORMANCE
Bagging, also called bootstrap aggregating, is an algorithm to improve the total generalization performance by combining several different models (Breiman, 1996). Strategies to combine those models such as averaging and majority voting are known as ensemble methods. It is shown that ensembles of models perform at least as well as each of its ensemble members (Krogh & Vedelsby, 1995). Moreover, ensembles achieve the best performance when each of their members makes independent emrrrs (Goodfellow et al., 2015; Hansen & Salamon, 1990).
Related work on ensembles: Ensembles have been studied extensively for improving model per- formance (Hansen & Salamon, 1990; Perrone & Cooper, 1992; Dietterich, 2000; Maclin & Opitz, 1999). One major direction in ensemble research is how to reduce their cost at test time. Bucila et al. (2006) developed method to compress large, complex ensembles into smaller and faster models which achieve faster test time prediction. Hinton et al. (2015) developed the above approach further by distilling the knowledge in ensemble of models into one single neural network. Another
2

---

Published as a conference paper at ICLR 2020
major direction in ensemble research is how reduce their cost at training time. Xie et al. (2013)
forms ensembles by combining the output of networks within a number of training checkpoints, named Horizontal Voting Vertical Voting and Horizontal Stacked Ensemble. Additionally, models trained with different regularization and augmentation can be used as ensemble to achieve better performance in semi-supervised learning (Laine & Aila, 2017). More recently, Huang et al. (2017) proposed Snapshot ensemble, in which single model is trained by cyclic learning rates (Loshchilov & Hutter, 2016; Smith, 2015) so that it is encouraged to visit multiple local minima. Those local minima solutions are then used as ensemble members. Garipov et al. (2018) proposed fast geometric ensemble where it finds modes that can be connected by simple curves, and each mode can taken as one ensemble member. The aforementioned works are complementary to BatchEnsemble, and one could potentially combine these techniques to achieve better performance. BatchEnsemble is efficient in both computation (including training and testing) and memory, along with a minimal change to the current training scheme such as learning rate schedule. For example, the need of cyclic learning rates in Snapshot Ensemble makes it incompatible to Transformer (Vaswani et al., 2017) which requires warm-up and inverse square root learning rate.
Explicit ensembles are expensive another line of work lies on what so-called "implicit" ensembles. For example, Dropout (Srivastava et al., 2014) can be interpreted as creating an exponential number of weight-sharing sub-networks, which are implicitly ensembled in test time prediction (Warde-Farke et al., 2014). MC-dropout can be used for uncertainty estimates (Gal & Ghahramani, 2015). Implicit ensemble methods are generally cost-free in training and testing.
2.2 ENSEMBLES FOR IMPROVED UNCERTAINTY
Several measures have been proposed to assess the quality of uncertainty estimates, such as calibra- tion (Dawid, 1982; Degroot & Fienberg, 1983). Another important metric is the generalization of predictive uncertainty estimates to out-of-distribttion datasets (Hendrycks & Dietterich, 2019). The contextual bandits task was recently proposed to evaluate the quality of predictive uncertainty, where maximizing reward is of direct interest (Riquelme et al., 2018); and which requires good uncertainty estimates in order to balance exploration and exploitation.
Although deep neural networks achieve state-of- performance on a variety of tasks, their predictions often poorly calibrated (Guo al., 2017). Bayesian neural networks (Hinton & Neal, 1995), which posit a distribution over the weights rather than point estimate, are often used for model uncertainty (Dusenberry et al., 2019). However, they require modifications the traditional neural network training scheme. Deep ensembles have been proposed as a simple and scalable alternative, and have been shown to make well-calibrated uncertainty estimates (Lakshminarayanan et al., 2017). More recently. Ovadia et al. (2019) and Gustafsson et al. (2019) independently benchmarked existing methods for uncertainty modelling on a broad range of datasets and architectures, and observed that ensembles tend to outperform variational Bayesian neural networks in terms of both accuracy and uncertainty, particularly on OOD datasets. Fort et al. (2019) investigates the loss landscape and postulates that variational methods only capture local uncertainty whereas ensembles explore different global modes. It explains why deep ensembles generally perform better.
2.3 LIFELONG LEARNING
In lifelong learning, the model trains on a number of tasks in a sequential (online) order, without access to entire previous tasks' data (Thrun, 1998; Zhao & Schmidhuber, 1996). One core difficulty of lifelong learning is "catastrophic forgetting": neural networks tend to forget what it has learnt after training on the subsequent tasks (McCloskey, 1989; French, 1999). Previous work on alleviating catastrophic forgetting can be divided into two categories.
In the first category, updates on the current task are regularized so that the neural network does not forget previous tasks. Elastic weight consolidation (EWC) applies a penalty on the parameter update based on the distance between the parameters for the new and the old task using the Fisher information metric (Kirkpatrick et al., 2016). Other methods maintain memory buffer that stores a number of data points from previous tasks. For example, gradient episodic memory approach penalizes the gradient on the current task that it does not increase the loss of examples in the memory buffer (Lopez-Paz & Ranzato, 2017; Chaudhry et al., 2018). Another approach combines experience replay algorithms with lifelong learning (Rolnick al., 2018; Riemer et al., 2018).
3

---

previous tasks by Published as a conference paper at ICLR 2020
In the second category, one increases model capacity new tasks are added. For example, progressive neural networks (PNN) copy the entire network for the previous task and add new hidden units when adopting to new task (Rusu et al., 2016). This prevents forgetting construction (the network on previous tasks remains the same). However, leads to significant memory consumption when faced with large number of lifelong learning tasks. Some following methods expand the model in more parameter efficient way at the cost of introducing an extra learning task and not entirely preventing forgetting. Yoon et al. (2017) applies group sparsity regularization to efficiently expand model capacity; Xu & Zhu (2018) learns to search for the best architectural changes by carefully designed reinforcement learning strategies.
METHODS
As described above, ensembles suffer from expensive memory and computational costs. In this section, we introduce BatchEnsemble, an efficient way to ensemble deep neural networks.
3.1 BATCHENSEMBLE
In this section, we introduce how to ensemble neural networks in an efficient way. Let W be the weights in neural network layer. Denote the input dimension as m and the output dimen- sion n, ie. W E _mx= For ensemble, assuming the ensemble size is M and each ensem- ble member has weight matrix W,: Each ensemble member owns a tuple of trainable vectors and 8; which share the same dimension output and input (n and m) respectively, where i ranges from 1 to M. Our algorithm generates a family of ensemble weights W, by the following:
mullippleddd by Indoponcent rank weighta...
pledd eroemble member.
W. W o Fú, where F, S.T."
(1)
weight
For each training example in the mini-batch, receives an ensemble weight W, by element- wise multiplying W, which we refer as "slow weights", with rank-one matrix F,, which we refer as "fast weights." " The subscript repre- sents the selection of ensemble member. Since W is shared across ensemble members, we term it as "shared weight" in the following paper. Fig- ure visualizes BatchEnsemble. Rather than modulating the weight matrices, modulate the neural networks' intermediate fea- tures, which achieves promising performance in visual reasoning tasks (Perez et al., 2017).
W Wirrey"
W
W2 W Yyfa"
also
Figure 2: An illustration how to generate the ensemble weights for two ensemble members.
Vectorization: We show how make the above ensemble weight generation mechanism paralleliz- able within a device, i.e., where one computes a forward pass with respect to multiple ensemble members in parallel. This achieved by manipulating the matrix computations for mini-batch (Wen et al., 2018). Let x denote the activations of the incoming neurons in neural network layer. The next layer's activations are given by:
Yn Φ (W: znn)
(2)
= & (IW o (xTTT) T In
(3)
= & ((WT(xn si)) o or:).
(4)
where & denotes the activation function and the subscript n represents the index in the mini-batch. The output represents next layer's activations from the ith ensemble member. To vectorize these computations, we define matrices Ii and whose rows consist of the vectors r; and for all examples in the mini-batch. The above equation vectorized as:
Y (((X &)W) R).
(5)
4

---

Published as a conference paper at ICLR 2020
where X is the mini-batch input. By computing Eqn. 5, obtain the next layer's activations for each ensemble member in a mini-batch friendly way. This allows us to take full advantage of parallel accelerators to implement the ensemble efficiently. To match the input and the ensemble weight, can divide the input mini-batch into M sub-batches and each sub-batch receives ensemble weight W., i (1,...,Mf
In our experiments, we take the average of predictions of each en- semble member. Suppose the test batch size is B and there are M ensemble members. To achieve an efficient implementation, one repeats the input mini-batch M times, which leads to an effective batch size B M. This enables all ensemble members to compute the output of the same input data points in single forward pass. It eliminates the need to calculate the output of each ensemble member sequentially and therefore reduces the ensemble's 's computational cost.
Ensembling During Testing:
3.2 COMPUTATIONAL COST
The only extra computation in BatchEnsemble over single neural network is the Hadamard product, which is cheap compared to matrix multiplication. Thus, BatchEnsemble incurs almost no additional computational overhead (Figure 1).² One limitation of BatchEnsemble is that if we keep the mini- batch size the same as single model training, each ensemble member gets only a portion of input data. In practice, the above issue can be remedied by increasing the batch size so that each ensemble member receives the same amount of data as ordinary single model training. Since BatchEnsemble is parallelizable within a device, increasing the batch size incurs almost no computational overhead in both training and testing stages on the hardware that can fully utilize large batch size. Moreover, when increasing the batch size reaches its diminishing return regime, BatchEnsemble can still take advantage from even larger batch size by increasing the ensemble size.
The only memory overhead in BatchEnsemble is the set of vectors, {ri, fri,...,,',, and {si...... 8m which are cheap to store compared to the weight matrices. By eliminating the need to store full weight matrices of each ensemble member, BatchEnsemble has almost no additional memory cost. For example, BatchEnsemble of ResNet-32 of size 4 incurs 10% more parameters while naive ensemble incurs 3X more.
3.3 BATCHENSEMBLE AS AN APPROACH TO LIFELONG LEARNING
The significant memory cost of ensemble methods limits its application to many real world learning scenarios such as multi-task learning and lifelong learning, where one might apply an independent copy of the model for each task. This is not the case with BatchEnsemble. Specifically, consider total of T tasks arriving in sequential order. Denote D, = (zz, y:. t) as the training data in task t where t E {1, .......7 T and is the index of the data point. Similarly, denote the test data set as Ti (Ii, Di, 1). At test time, we compute the average performance on T, across all tasks seen so far as the evaluation metric. To extend BatchEnsemble lifelong learning, we compute the neural network prediction in task t with weight W, W (res,) ) in task t. In other words, each ensemble member is in charge of one lifelong learning task. For the training protocol, we train the shared weight W and two fast weights T1, 81 on the first task,
min (W $1, r1, W.
(6)
where J, is the objective function in the first task such as cross-entropy in image classification. On subsequent task t, we only train the relevant fast weights Tt, St.
min La (st, rz; Dz).
BatchEnsemble shares similar advantages as progressive neural networks (PNN): it entirely prevents catastrophic forgetting as the model for previously seen tasks remains the same. This removes the need of storing any data from previous task. In addition, BatchEnsemble has significantly less memory consumption than PNN as only fast weights are trained to adapt to new task. Therefore, BatchEnsemble can easily scale to up to 100 tasks as we showed in Section 4.1 on split ImageNet. Another benefit of BatchEnsemble is that if future tasks arrive in parallel rather than sequential order,
In Figure 1, note the computational overhead of Ratchbee at the ensemble size indicates the
additional cost of Hadamard products.
5

---

Published as a conference paper at ICLR 2020
Table 1: Computational and memory costs on Spli-CIFARR00 on LeNet. Numbers are relative to vanilla neural network.
Vanilla
BatchE
DEN
PNN
RCL
Computational
!
1.11
9.58
1.12
26.41
Memory
1
1.10
5.31
4.16
2.52
one can train on all the tasks at once (see Section 3.1). We are not aware of any other lifelong learning methods can achieve this.
Limitations: BatchEnsemble is one step toward toward full lifelong learning agent that both immune to catastrophic forgetting and parameter-efficien On existing benchmarks like split-CIFAR and split- ppattlmmaeeNet Section 4.1 shows that BatchEnsemble's rank-1 perturbation per layer pro- vides enough expressiveness for competitive tfat-of-thhear accuracies. However, one limitation of BatchEnsemble is that only rank-1 1 perturbations are fit to each lifelong learning task and thus the model's 's expressiveness is valid concern when each task is significantly varied. Another limitation is that the shared weight is only trained on the first task. This implies that only information learnt for the first task transfer subsequent tasks. There is explicit transfer, for example, between the second and third tasks. One solution is to enable lateral connections to features extracted by the weights of previously learned tasks, done in PNN. However, we found that no lateral connections were needed for Split-CIFAR R 100 and Split-ImageNet. Therefore we leave the above solution to future work further improve BatchEnsemble for lifelong learning.
Computational cost compared to other methods: Dynamically expandable networks (Yoon et al., 2017) and reinforced continual learning (Xu & Zhu, 2018) are two recently proposed lifelong learning methods that achieve competitive performance. These two methods can be seen an improved version progressive neural network (PNN) (Rusu et al., 2016) in terms of memory efficiency. As shown in Xu & Zhu (2018), all three methods result to similar accuracy measure in Split-CIFARI00 100 task. Therefore, among three evaluation metrics (aceuracy, forgetting and cost), we only compare the accuracy of BatchEnsemble to PNN in Section 4. and compare the cost in this section. We first compute the cost relative to PNN on Split-CIFAL 100 on LeNet and then compute the rest of the numbers base what were reported in Xu & Zhu (2018). Notice that PNN has much computational overhead on Split-CIFAR 00 because the number of total tasks is limited to 10. Even on the simple setup above, BatchEnsemble gives the best computational and memory efficiency. BatchEnsemble leads to more lower costs on large lifelong learning tasks such as Split-lmageNet.
4 EXPERIMENTS
Section 4.1 first demonstrates BatchEnsemble's effectiveness as an alternative approach to lifelong learning on Split-CIFAR and Splie-ImageNet. Section 4.2 and Section 4.3 next evaluate BatchEnsem- ble on several benchmark datasets with common deep learning architectures, including image classifi- cation with ResNet (He et al., 2016) and neural machine translation with Transformer (Vaswani et al., 2017). Section 4.4 demonstrates that BatchEnsemble can be used for calibrated prediction. Finally, showcase its applications in uncertainty modelling in Appendix and Appendix D. Detailed description of datasets we used is in Appendix A. Implementation details are in Appendix B.
4.1 LIFELONG LEARNING
We showcase BatchEnsemble for lifelong learning on Split-CIFAR100 and Split-ImageNet. Split- CIFAR 100 proposed in Rebuffi et al. (2016) is harder lifelong learning task than MNIST permu- tations and MNIST rotations (Kirkpatrick et al., 2016), where one introduces new set of classes upon the arrival of a new task. Each task consists of examples from disjoint set of 100/T classes assuming tasks in total. To show that BatchEnsemble able to scale to 100 sequential tasks, we also build own more classes (and thus more tasks) and higher image resolutions are involved. More details about these two lifelong learning datasets provided in Appendix A.
dataset which shares the same property Split-CIFAR 100 except
We consider T 20 tasks on Split-CIFARIOO following the setup of Lopez-Pazz & Ranzato (2017). We used ResNet- 18 with slightly fewer number of filters across all convolutional layers. Note that for

---

Published as a conference paper at ICLR 2020
0.25
Avg validation acc
EWC
0.8
0.70
0.20
BetchE
0.15
0.60
0.4
0.10
2.6
0.50
0.05
0.2
Nechod:
BN-Tuned
Upperbound
0.40
0.00
Task number
Accuracy
Forget
Mem
(a) Averaged validation accuracy on Split- ImageNet
(b) Various of measures on Split- Split-CIFAR100
Figure Performance for lifelong learning. (a): Validation accuracy for each Split- ImageNt task. Standard deviation is computed over 5 random seeds. (b): BatchEnsemble and several other methods on Split-CIFAR 100. Batchensemble achieves the best trade-off among Accuracy (t), Forget (J), and Time & Memory (4) costs. VAN: Vanilla neural network. EWC: Elastic weight consolidation (Kirkpatrick et al., 2016). PNN: Progressive neural network (Rusu et al., 2016). BN- Tuned: Fine tuning Batch Norm layer per subsequent tasks. BatchE; BatchEnsemble. Upperbound: Individual ResNet-50 per task.
the purpose of making of the task descriptor, we build different final dense layer per task. We compare BatchEnsemble to progressive neural networks (PNN) (Rusu et al., 2016), vanilla neural networks, and elastic weight consolidation (EWC) Split-CIFAR Xu & Zhu (2018) reported similar accuracies among DEN (Yoon et al., 2017), RCL. (Xu & Zhu, 2018) and PNN. Therefore we compare accuracy only PNN which has an official implementation and only compare computational and memory costs to DEN and RCL in Table 1.
Figure 3b displays results on Split- Split-CIFARIOOS 100 over three metrics including accuracy, forgetting, and cost. The accuracy measures the average validation accuracy over total 20 tasks after lifelong learning ends. Average forgetting over all tasks also presented in Figure 3b. Forgetting on task measured by the difference between accuracy of task right after training on it and at the end of lifelong learning. It measures the degree catastrophic forgetting. As showed in Figure 3b, BatchEnsemble achieves comparable accuracy as PNN while having 4X speed-up and 50X less memory consumption. also preserves the no-forgetting property of PNN, Therefore BatchEnsemble has the best trade-off among all compared methods.
For Split- Netee,,, we consider T = 100 tasks and apply ResNet-50 followed by final linear classifier per task. The parameter overhead of BatchEnsemble on Split-ImageNet over 100 sequential tasks is 20%: the total number of parameters 30M v.s. 25M (vanilla ResNet-50). PNN is not capable of learning 100 sequential tasks due to the significant memory consumption; other methods noted above have also not shown results at ImageNet scale. Therefore we adopt two our baselines. The first baseline is "BN-Tuned", which fine-tunes batch normalization parameters per task and which has previously shown strong performance for multi-task learning (Mudrakarta et al., 2018). To augment the number of filters in BN-Tuned so that both methods have
make a fair comparison, the same number of parameters. The second baseline is a naive ensemble which trains an individual Re per task. This provides a rough upper bound on the BatchEnsemble's expressiveness per task. Note Batehbn and both baselines are immune to catastrophic forgetting. So we consider validation accuracy on each subsequent task as evaluation metric. Figure 3a shows that BatchEnsemble outperforms BN-Tuned consistently. This demonstrates that Batchen is practical method for lifelong learning that scales to a large number sequential tasks.
4.2 MACHINE TRANSLATION
In this section, we evaluate Batehene on the Transformer (Vaswani et al., 2017) and the large-scale machine translation tasks WMT14 EN-DE/EN-FR. We apply BatchEnsemble to all self- attention layers with an ensemble size of 4. The ensemble a self-attention layer can be interpreted as each ensemble member keeps their own attention mechanism and makes independent decisions. We conduct our experiments on W English- Seglish-Grrman dataset and WMT14 English-Frened Freench dataset with Transformer base (65M parameters) and Transformer big (213M parameters). We maintain
7

---

Published as a conference paper at ICLR 2020
BigSingle
BigSingle
4.1
BigBE BesaSingle
Loss
BaseSingle
Big
4.0
BasaBE
BazaBE
0.25
0.50
0.75
0.8
Steps
te5
Steps
1e5
(a) English-German
(b) English-
Figure 4: Comparison between BatchEnsemble and single model WMT English-Germat and English-French. Training stops after the model reaches targeted validation perplexity. BatchEnsemble gives faster convergence by taking the advantage of multiple models. (a): Validation loss of WMT16
English-German task. (b): Validation loss WMT14 English-Frened task. Big: Tranformer big model. Base: Transformer base model. BE: Batchens Single: Single model.
exactly the same training scheme and hyper-parameters between single Transformer model and Batchen Transformer model.
As the result shown in Figure 4, BatchEnsemble achieves much faster convergence than single model. Big BatchEnsemble Transformer is roughly 1.5X faster than single big Transformer on WMT16 Boglish-German In addition, the BatchEnsemble Transformer also gives a lower validation perplex- ity than big Transformer (Table 2). This suggests that BatchEnsemble is promising for bigger Trans- former models. We also compared BatchEnsemble
Table Perplexity on Newstest2013 with big Transformer. BatchEnsemble with ensemble size
Single
MC-drop
BatchE
EN-DE EN-FR
4.30
4.30
4.26
2.76
2.77
2.74
to dropout ensemble (MC-drop in Table 2). Transformer single model itself uses dropout layers. We run multiple forward passes with different sampled dropout maskd during testing. The sample size is 16 which is already 16X more expensive than BatchEnsemble. As Table 2 showed, dropout ensemble doesn't give better performance than single model. However, Appendix B shows that while BatchEnemble's test BLEU score increases faster over the course of training, Ensembe which gives lower validation loss does not achieve better BLEU score over the single model.
4.3 CLASSIFICATION
We evaluate Batchensemble on classification tasks with CIFAR- 10/100 dataset (Krizhevsky, 2009). We run our evaluation on ResNet32 (He et al., 2016). To achieve 100% training accu- racy on CIFAR 100, we use 4X more filters than the standard ResNet-32. In this section, compare to MC-dropout (Gal & Ghahramani, 2015), which is also a memory efficient ensem- ble method. We add one more dense layer fol- lowed by dropout before the final linear classifier so that the number of parameters of MC-dropout the same as BatchEnsemble. Most hyper-parameters are shared across the single model, BatchEnsem- ble, and MC-dropout. More details about hyper-parameters are in Appendix B. Note that increase the training iterations for BatchEnsemble to reach its best performance because each ensemble member gets only a portion of input data.
Table 3: Validation accuracy on ResNet32. En-
semble with size 4. MC-drop stands for Dropout ensemble (Gal & Ghahramani, 2015).
Single MC-drop
BatchE
NaiveE
C10 C100
95.31
95.72
95.94
96.30
78.32
78.89
80.32
81.02
We train both BatchEnsemble model and MC-dropout with 375 epochs on CIFAR-10/100, which 50% more iterations than single model. Although the training duration longer, BatchEnsemble is still significantly faster than training individual model sequentially. Another implementation that leads to the same performance is to increase the mini-batch size. For example, if we use 4X large mini- batch size then there is no need to increase the training iterations. Table 3 shows that BatchEnsemble reaches better accuracy than single model and MC-dropout. We also calculate the accuracy naive ensemble, whose members consist of individually trained single models. Its accuracy be viewed as the upper bound of efficient ensembling methods. For fairees,, we also compare BatchEnsemble naive ensemble of small models in Appendix F.
8

---

Published as a conference paper at ICLR 2020
0.9
0.6
0.3
Method
Vanilla
Dropout 8
Ensemble
BEdrop_6
0.3
BotchEnsemble
0.2
Skew intensity
Method
Venilla
Dropout 8
Ensemble
BEdrop_ 8
BatchEnsemble
0.2
0.1
0.0
Skew intensity
boxplots showing a comparison of ECE under all types of corruptions CIFAR-10 10. Each box shows the quartiles summarizing the results across all types of skew while the error bars indicate the min and max across different skew types. Ensseble/BatchEnsembbbe Naive/Batch ensemble of 4 ResNet32x4 models. Dropout-8: Dropout ensemble with sample size 8. BEDrop-8: BatchEnsemble of models + Dropout ensemble with sample size 8. similar measurement can be found in Ovadia al. (2019).
Figure 5: Calibration on CIFAR-10 corruptions:
4.4 CALIBRATION ON CORRUPTED DATASET
In this section, we measure the calibrated prediction of BatchEnsemble on corrupted datasets. Other uncertainty modelling tasks such as contextual bandits are delegated to Appendix C and Appendix D.
Other than unseen classes, corruption another type of -iitr--diittibttionn examples. is common that the collected data is corrupted or mislabelled. Thus, measuring uncertainty modelling under corruption is practically meaningful. We want our model to preserve uncertainty or calibration in this case. In this section, we evaluate the calibration of different methods on recently proposed CIFAR-10 corruption dataset (Hendrycks Dietterich, 2019). The dataset consists of over 30 types of corruptions to the images. Notice that the corrupted dataset is used as testset without training on it. Given the predictions on CIFAR- corruption, we can compare accuracy and calibration measure such as ECE loss for single neural network, naive ensemmble, and BatchEnsemble. Ovadia et al. (2019) benchmarked a number of methods on CIFAR-10 corruption. Their results showed that naive ensemble achieves the best performance both accuracy and ECE loss, outperforming other methods including dropout ensemble, temperature scaling and variational methods significantly. Dropout ensemble is the state-of-the-art memory efficient ensemble method.
The scope of this paper is on efficient ensembles. Thus, in this section, we mainly compare BatchEnsemble dropout ensemble CIFAR-10 corruption. Naive ensemble also plotted as an upper bound of our method. As showed in Figure 5, BatchEnsemble and dropout ensemble achieve comparable accuracy on corrupted dataset on all skew intensities. Calibration is a more important metric than accuracy when the dataset is corrupted. We observed that BatchEnsemble achieves better average calibration than dropout as the skew intensity increases. Moreover, dropout ensemble requires multiple forward passes to get the best performance. Ovadia et al. (2019) used sample size 128 while we found no significant difference between sample size 128 and 8. Note that even for sample size is 8, it is 8X more expensive than BatchEnsemble in the testing time cost. Finally, we showed that combining BatchEnsemble and dropout ensemble leads to better accuracy and calibration. It is competitive to naive ensemble while keeping memory consumption efficient. It also an evidence that Bateben is an orthogonal method to dropout ensemble; combining these two potentially obtain better performance.

---

Published as a conference paper at ICLR 2020
5 DIVERSITY ANALYSIS
As mentioned in Section 2, more diversity among ensembling members leads to better performance. Therefore, beyond accuracy and uncertainty metries, we are particularly interested in how much diversity rank-1 1 perturbation provides. We compare BatchEnsemble to dropout ensemble and naive ensemble over the newly proposed diversity metric (Fort et al., 2019). The metric measures the disagreement among ensemble members on test set. We computed it over different amount of training data. See Appendix for details on diversity metric and plots.
In this section, we give an intuitive explanation of why Batchhnsemble leads to more diverse members with fewer training data. If only limited training data is available, the parameters of the neural network would remain close to their initialization after convergence. In the extreme case where only one training data point is available, the optimization quickly converges and most of the parameters are not updated. This suggests that the diversity of initialization entirely determines the diversity of ensembling system. Naive ensemble has fully independent random initializations. BatchEnsemble has indeediddeeeneent random initializations. In comparison, all ensemble members of dropout ensemble share the same initialized parameters. Therefore, both naive ensemble and BatchEnsemble significantly outperform dropout ensemble in diversity with limited training data.
More importantly, Figure 8 provides insightful advice on when Bascbbe achieves the best gain in practice. We observe that diversity of Batchensemble is comparable to naive ensemble when training data is limited. This explains why Easemble has higher gains on CIFAR-IO1 1-0 than CIFAR-10, because there only 500 training points for each class CIFAR-100 whereas 5000 on CIFAR-10 10. Thus, CIFAR- 100 has more limited training data compared to CIFAR- 10. Another implication that BatchEnsemble can benefit more from heavily over parameterrazee neural networks. The reason is that given the fixed amount of training data, increasing the number of parameters essentially converges the case where the training data limited. practice, the best way to make full use of increasing computational power is to design deeper and wider neural networks. This suggests that BatchEnsemble benefits more from the development of computational power; because it has better gain over- werrparameeterized neural networks.
CONCLUSION
We introduced BatchEnsemble, efficient method for ensembling and lifelong learning. BatchEnsemble can be used to improve the accuracy and uncertainty of any neural network like typical ensemble methods. More importantly. BatchEnsemble removes the computation and memory bottleneck of typical ensemble methods, enabling its successful application to not only faster ensem- bles but also lifelong learning up to 100 tasks. We believe BatchEnsemble has great potential to improve in lifelong learning. Our work may serve as starting point for new research area.
ACKNOWLEDGMENTS
YW was supported by University of Toronto Fellowship, Faculty of Arts And Science and Vector Scholarships in Artificial Intelligence (VSAI).
10

---

Published as a conference paper at ICLR 2020
REFERENCES
Léon Bottou, Stochastic learning, In Summer School on Machine Learning, pp. 146-168. Springer, 2003.
Leo Breiman. Bagging predictors. Machine Learning, 24:123-140, 1996.
Denny Britz, Anna Goldie, Minh-Thang Luong, and Quoc V. Le. Massive exploration of neural machine translation architectures. CoRR, abs/1703.03906, 2017.
Cristian Bucila, Rich Caruana, and Alexandru Niculescu-Mizil Model compression. In KDD, 2006.
Arslan Chaudhry, Marc' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny, Efficient lifelong learning with A-GEM. ArXiv, abs/1812.00042, 2018.
A. Philip Dawid. The well-calibrated Bayesian. 1982.
Morris Degroot and Stephen E. Fienberg. The comparison and evaluation of forecasters. 1983.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR 2009, 2009.
Thomas G. Dietterich. Ensemble methods in machine learning. Multiple Classifier Systems, 2000.
Michael W Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew Dai. Analyzing the role model uncertainty for electronic health records. arXiv preprint arXiv:1906.03842. 2019.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. ArXiv, abs/1912.02257, 2019.
Robert M. French. Catastrophic forgetting in connectionist networks. Trends Cognitive Sciences, 13528-135, 1999.
Yarin Gal and Zoubin Ghahramani. Dropout Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In ICML, 2015.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P. Vetrov, and Andrew Gordon Wilson.
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs. In NeurIPS, 2018.
Rong Ge, Furong Huang. Chi Jin, and Yang Yuan. Escaping from saddle points-online stochastic gradient for tensor decomposition. Conference on Learning Theory, pp. 797-842, 2015.
Ian J. Goodfellow, Yoshua Bengio, and Aaron C. Courville, Deep learning. Nature, 521:436-444, 2015.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In ICML, 2017.
Fredrik K. Gustafsson, Martin Danelljan, and Thomas B. Schön. Evaluating scalable Bayesian deep learning methods for robust computer vision. ArXiv, abs/1906.01620 2019.
Lars Kai Hansen and Péter Salamon. Neural network ensembles. IEEE Trans. Pattern Anal. Mach. Intell., 120113-1001, 1990.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representattonss 2019. URL httph:ttp///cm htattts:://pperevviii
net / forumoriiiooooooo HJ2
Geoffrey E. Hinton and Radford M. Neal. Bayesian learning for neural networks. 1995.

---

Published as a conference paper at ICLR 2020
Geoffrey E. Hinton, Oriol Vinyals, and Jeftrey Dean. Distilling the knowledge in neural network. CoRR, abs/1503.02531, 2015.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. ECCV, 2016.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snapshot Ensembles: Train 1, Get M for Free. CoRR, 7tss///04000099, 2017.
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating deep network training by reducing internal covariate shift. In ICML, 2015.
Kenji Kawaguchi. Deep learning without poor local minima. In Advances in neural information processing systems, pp. 586-594, 2016.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 141.v:1412.6980, 2014.
James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinst Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 114 13:3521-3526, 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Anders Krogh and Jesper Vedelsby. Neural network ensembles, cross validation, and active learning. In Advances in neural information processing systems, pp. 231-238, 1995.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. CoRR, abs/1610.02242, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continuum learning. In NIPS, 2017.
Iya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with restarts. CORR, abs/1608. 03988, 2016.
Richard Maclin and David W. Opitz. Popular ensemble methods: An empirical study. J. Artif. Intell. Res., 11:169-198, 1999.
M. W. McCloskey. Catastrophic interference in connectionist networks: The sequential learning problem. 1989.
Pramod Kaushik Mudrakarta, Mark Sandler, Andrey Zhmoginov, and Andrew G. Howard. K for the price of 1: Parfieter-efficient multi-task and transfer learning. ArXiv, Ahs//1800.10 10703, 2018.
Mahdi Pakdaman Nacini, Gregory F. Cooper, and Milos Hauskrecht, Obtaining well calibrated prob- abilities using Bayesian binning. Proceedings of the ... AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence, 2015:2201129900 2015.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D. Sculley, Sebastian Nowozin, Joshua V. Dillon, Balaji Lakshminarayanan, and Jasper Snock. Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift. ArXiv,
02530. 2019.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. FiLM: Visual reasoning with a general conditioning layer. AAAI, 2017.
Michael P. Perrone and Leon N. Cooper. When networks disagree: Ensemble methods for hybrid neural networks. 1992.
12

---

Published as a conference paper at ICLR 2020
Sylvestre- Alvise Rebuffi, Alexander I Kolesnikov, Georg Sperl, and Christoph H. Lampert. iCaRL: Incremental classifier and representation learning. 2017 IEEE Conference on Computer Vision and Pattem Recognition (CVPR), pp. 5533-5542, 2016.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. ArXiv, abs/1810.11910, 2018.
Carlos Riquelme, George Tucker, and Jasper Snoek. Deep Bayesian bandits showdown. 2018.
David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Greg Wayne. Experience replay for continual learning. ArXiv, abs/1811 1.1 1682, 2018.
Alejandro Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. CoRR, abs/l412.65500 2015.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. ArXiv, Abs//600.04677/ 2016.
Leslie N. Smith. No more pesky learning rate guessing games. CoRR, abs/ 2015.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, Journal of Machine Learning Research, 15:1929-1958, 2014.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Highway networks. CoRR, abs/1505.00387, 2015.
Sebastian Thrun. Lifelong learning algorithms. In Learning to Learn, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
David Warde Warde-Farley, Ian J. Goodfellow, Aaron C. Courville, and Yoshua Bengio. An empirical analysis of dropout in piecewise linear networks. CoRR, abs/1312.6197, 2014.
Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, and Roger B. Grosse. Flipout: Efficient pseudo-
independent weight perturbations on mini-batches. In International Conference Learning Representations, 2018
Yeming Wen, Kevin Luk, Maxime Gazeau, Guodong Zhang, Harris Chan, and Jimmy Ba. Interplay between optimization and generalization of stochastic gradient descent with covariance noise. ArXiv, 1b2.1992.0 2019.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V. Le, Mohammad Norouzi. Wolfgang Macherey. Maxim Krikun, Yuan Cao, Qin Gao, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rud- nick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. Google's neural machine translation system: Bridging the gap between human and machine translation. CoRR, abs/1609.00 (609.08166 44, 2016.
Jingjing Xie, Bing Xu, and Chuang Zhang. Horizontal and vertical ensemble with deep representation for classification. CoRR, Abs//330.22541 2013.
Ju Xu and Zhanxing Zhu. Reinforced continual learning. In NeurIPS, 2018
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. ArXiv, abs/1 708.011077 2017.
Jian Hua Zhao and Jürgen Schmidhuber. Incremental self-immprveement for life-time multi-agent reinforcement learning. 1996.
13

---

Published as a conference paper at ICLR 2020
A DATASET DETAILS
We consider two CIFAR datasets, CIFAR-10 and CIFAR-100 (Krizhevsky, 2009). Each consists of training set of size 50K and test set of size 10K. They are natural images with 32x32 pixels. In our experiments, we follow the standard data pre-processing schemes including zero-pacee aero-padding with pixels on each sise, random crop and horizon flip (Romero et al., 2015; Huang et al., 2016; Srivastava et al., 2015).
CIFAR:
WMT: In machine translation tasks, we consider the standard training datasets WMT16 English- German and WMT11 English-Frenchh WMT16 English- German dataset consists of roughly 4.5M and
sentence pairs. We follow the same pre-pr pre-preeessing schemes in (Vaswani et al., target tokens are processed into 37K shared sub-word units based on byte-pair encoding (BPE) (Britz et al., 2017). Newstest2013 and Newstest2014 are used as validation set and test set respectively. WMT14 English-French consists of much larger dataset sized at 36M sentences pairs. We split the tokens into a 32K word-piece vocabulary (Wu et al., 2016).
Split-CIFARII The dataset has the same set of images as CIFAR-100 100 dataset (Krizhevsky, 2009). randomly splits the entire dataset into T tasks so each task consists 100/"T classes of images. To leverage the task descriptor in the data, different final linear classifier is trained top of feature extractor per task. This simplifies the task to be 100/T class classification problem in each task. i.e. random prediction has accuracy T0100. Notice that since are not under the setting of single epoch training, standard data pre-processing including padding, random crop and random horizontal flip are applied to the training set.
The dataset has the same set of images as ImageNet dataset (Deng et al., 2009). It randomly splits the entire dataset into T tasks so each task consists of 1000/T classes of images. Same as Split-CIFARI00 each task has its own final linear classifier. Data preprocessing (He et al., 2016) is applied to the training data.
B IMPLEMENTATION DETAILS
In this section, we discuss some implementation details of BatchEnsemble.
28
Weight Decay: In the BatchEnsemble, the weight of each ensemble member is never explicitly cal- culated because we obtain the activations directly by computing Eqn. 5. To maintain the goal of no additional computational cost, we can instead regularize the mean weight W over ensemble members, which can be efficiently calculated as W = A S R, where W is the shared weight among ensemble members, S and I are the ma- trices in Eqn. 5. We can also only regularize the shared weight and leave the fast weights unregularized because it only accounts for a small portion of model parameters. practice, we find the above two schemes work equally.
BLEU
26
24
Biggie
BigBE
22
0.00
0.25
0.50
0.75 Steps
1.00
1.25
1.50
Figure 6: BLEU on English- task.
Diversity Encouragement, Additional loss term such as KL divergence among ensemble members can be added to encourage diversity. However, find it sufficient for BatchEnsemble have desired diversity by initializing the fast weight (s; and r; in Eqn. 1) to be random sign vectors. Also note that the scheme that each ensemble member is trained with different sub-batch of input can encourage diversity as well. The diversity analysis is provided in Appendix G.
Machine Translation: The Transformer base is trained for 100K steps and the Transformer big is trained for 180K steps. The training steps of big model are shorter than Vaswani et al. (2017) because we terminate the training when it reaches the targeted perplexity on validation set. Experiments are run on 4 NVIDIA P100 GPUs The BLEU score of Big Transformer on English-German task is in Figure 6. Although BatchEnsemble has lower perplexity as we showed in Section 4.2, we didn't 't observe better BLEU score. Noted that the BLEU score in Figure is lower than what Vaswani et al. (2017) reported. is because in order to correctly evaluate model performance at given timestep. didn't 't use the averaging checkpoint trick. The dropout rate of Transformer base is 0.1 and 0.3 for
14

---

Published as a conference paper at ICLR 2020
Transformer big Inglish-German while remaining 0.1 on English-Frencha For dropout ensemble, we ran a grid search between 0.05 and 0.3 in the testing time and report the best validation perplexity.
We train the model with mini-batch size 128. We also keep the standard learning rate schedule for ResNet. The learning rate decreases from 0.1 to 0.01, from 0.01 to 0.001 at halfway of training and 75% of training. The weight decay coefficient is set to be 10-4 We use ensemble size of 4, which means each ensemble member receives 32 training examples if we maintain the mini-batch size of 128. It is because Batch Normalization (loffe & Szegedy, 2015) requires least 32 examples to be effective on CIFAR dataset. As for the training budget, we train the single model for 250 epochs
Classification:
C PREDICTIVE UNCERTAINTY
In this section, we evaluate the predictive uncer- tainty of Basbbbe on out-of-distribution tasks and ECE loss.
(a) Histogram of the predictive entropy on exam- ples from known classes, CIFAR-10 (left) and unknown classes, CIFAR-100 (right).
Similar to Lakshminarayanan et al. (2017),
12.5
Single
Single
we first evaluate BatchEnsemble on out-of-
10.0
NaiveE 2.0
NaiveE
distribution examples from unseen classes. It is known that deep neural network tends to make over-confident predictions even if the predic- tion is wrong or the input comes from unseen
7.5
1.5
5.0
classes. Ensembles of models can give better uncertainty prediction when the test data is out of the distribution of training data. To measure the uncertainty the prediction, calculate the predictive entropy of Single neural network, naive ensemble and Batchensemble. The result is presented in Figure 7a. As we expected, sin- gle model produces over-confident predictions on unseen examples, whereas ensemble meth- ods exhibit higher uncertainty on unseen classes, including both BatchEnsemble and naive ensem- ble. It suggests our ensemble weight generation mechanism doesn't 't degrade uncertainty modelling.
2.5
0.0
0.0
0.0
0.5
1
2
H (log(p(y(z)))
(b) Expected Calibration Error. Ensemble of size Lower ECE reflects better calibration.
Single
MC-drop
BatchE
NaiveE
C10 C100
3.27
2.89
2.37
2.32
9.28
8.99
8.89
6.82
We also calculate the Expected Calibration Error (Naeini et al., 2015) (ECE) of single model, naive ensemble and BatchEnsemble on both CIFAR-10 and CIFAR-I00 in Table 7b. To calculate ECE, group model predictions into interval bins based on the predictive confidence (each bin has size M.. Let B,n denote the set of samples whose predictive probability falls into the interval ( (m=1 for m E (l.... (1. Let acc Bm) and conf Bm,) be the averaged accuracy and averaged confidence of the examples in the bin B,. The ECE can de defined the following.
M
Bm
ECE
Σ
(m.) conf( Bm))
(8)
where n is the number of samples. ECE as a criteria of model calibration, measures the difference in expectation between confidence and accuracy (Guo et al., 2017). It shows that BatchEnsemble makes more calibrated prediction compared to single neural networks.
D UNCERTAINTY ON BANDITS
In this section, conduct analysis beyond accuracy, where show that BatchEnsemble can be used for uncertainty modelling in contextual bandits.
For uncertainty modelling, evaluate our BatchEnsemble method the recently proposed bandits benchmark (Riquelme et al., 2018). Bandit data comes from different empirical problems that highlight several aspects of decision making. No single algorithm can outperform every other
15

---

Published as a conference paper at ICLR 2020
Table 4: Contextual bandits regret. Results are relative the cumulative regret of the Uniform algorithm. We report the mean and standard error of the mean over 30 trials. Ensemble size with 4, 8. We remove the methods with mean rank greater than 10.
M.RANK
M.VALUE
MUSHROOM
STATLOG
FINANCIAL
JESTER
WHEEL
Naiveensemble4
5.30
34.64
13.44 + 3.83
7.10 1.15
11.31 + 1.48
72.73 + 6.32
68.63 + 21.97
Nneeele
6.50
34.91
13.59 + 3.13
7.15 0.98
11.64 + 1.57
73.54
+ 6.14
68.65 + 19.32
BatchEnsemble
34.52
5.21
11.53 5.06
10.24 1: 2.66
72.65
1 6.27
62.94 1 26.12
BatchEnsemble8
5.70
33.95
13.48 3.36
9.85 3.67
13.17 2.87
71.84
6.47
61.41 1: 26.18
Dropout
8.20
36.73
15.05
+ 8.23
9.31 0.19
13.53
71,90
73.86
22.48
LinFullPost
9.40
49.60
97.42 +
19.00 1.08
10.24 + 0.92
78.40 =.85
42.94 + 12.68
MultitaskGP
5,90
34.59
12.87 1: 4.70
8.04 1.44:,777
8.50 0.80
74.03 -: 5.96
69.52 1 18.55
RMS
9.40
39.18
16.31 6.13
10:44 5.02
11.75 - 2.64
73.38 = 4.70
84.02 + 24.67
Uniform
16.00
100.00
100.00
100.00
100.00
100.00
100.00
algorithm on every bandit problem. Thus, average performance of the algorithm over different problems used evaluate the quality of uncertainty estimation. The key factor to achieve good performance in contextual bandits is to learn reliable uncertainty model. In our experiment, Thompson sampling samples from the policy given by one of the ensemble members. The fact that Dropout which is implicit ensemble method achieves competitive performance on bandits problem suggests that ensemble can be used as uncertainty modelling. Indeed, Table 4 shows that BatchEnsemble with ensemble size 8 achieves the best mean value on the bandits task. Both BatchEnsemble with ensemble size and outperform Dropout in terms average performance. We also evaluate BatchEnsemble CIFAR- 10 corrupted dataset (Hendrycks & Dietterich, 2019) in Appendix C. Figure shows that BatchEnsemble achieves promising accuracy, uncertainty and cost trade-off among all methods compared. Moreover, combining Basembee and dropout ensemble leads to better uncertainty prediction.
E DIVERSITY ANALYSIS
E.1 DIVERSITY METRIC
Final performance metries such accuracy and uncertainty score we provided in Section 4 obscures many insights of our models. In this section, we provide visualization of some commonly used diversity metrics proposed in Fort al. (2019). The diversity score used in the experiments below quantify the difference of two functions, by measuring fraction of the test data points on which when two functions are making identical predictions, and I when they differ on every single example in the test set. We also normalize the diversity metric by the error rate account for the case where random predictions provide the best diversity. There are other diversity metrics we can such as the KL-divergence between the probability distributions. doesn't 't make significant difference in our experiments, so we chose the fraction of disagreement for simplicity. We compare BatchEnsemble to dropout ensemble and naive ensemble. For BatchEnsemble and naive ensemble, we first select a model as our base model. We calculated the diversity measure of other ensembling members against the base model. We also plotted the diversity the base model as reference, which is trivially zero. For dropout ensemble, we sample a number of dropout masks and take the prediction of the first dropout mask the base model. The disagreement fraction of the rest dropout masks can be computed against the base model.
their predictions disagree. This metric
In Figure 8, we plotted the diversity measures of Batchensemble, dropout ensemble and naive ensemble. According to the ensembling theory 2, the more diversity among ensembling members leads better total accuracy. As Table showed, naive ensemble achieves the best accuracy and then followed by BatchEnsemble and dropout ensemble. Therefore, we expect the diversity measure of Batchen is in the middle of naive ensemble and dropout ensemble. Figure 8a confirms our hypothesis. Note that naive ensemble, which only differs from random initializations, is very effective at sampling diverse and accurate solutions. This is aligned to the observation in Fort et al. (2019). Notice that naive ensemble incurs 4X more memory cost and 3X more inference time than Rank-1 1 Net. Thus, we can conclude that BatchEnsemble achieves the best trade-off among accuracy, diversity and efficiency in all methods compared. The diversity gap between naive ensemble and BatchEnsemble due to the limited expressiveness of rank-1 perturbation. This provides scope for future research direction.
16

---

Published as a conference paper at ICLR 2020
Fractton of changes 111.0 (cacuurcyy
ResNet32 CIFAR-10
Fraction of tabels changes / /11.0 (0-ccuracyy
ResNet32 CIFAR-10
1.0
1.0
0.8
0.8
0.4
0.4
0.2
0.2
0.949
0.950
0.951
0.952
0.953
0.954
0.965
0.916
0.018
0.920
0.922
0.924
0.926
Validation accuracy
Validation accuracy
(a) Trained on Full CIFAR-10.
(b) Trained on 50% CIFAR-10.
ResNet32 on CIFAR-10
ResNet32 ÇIFAR-IO
Froction of labels charges (C1 (..ccuusccy
Traction of labes changes 111,6 accuracy)
Naive Ensemble:
1.0
0.0
Dropour Fasemble
RatchFnieeblee
0.8
0.4
0.2
0.2
0.0
0.810
0.815
0.820
0.825 0.830 0.835 0.940
0.545
0.67
0.68
0.69
0.70
0.71
0.72
0.73
0.74
Validation accuracy
accuracy
Trained 20% CIFAR-10.
(d) Trained 10%
CIFAR-10.
Figure& 8: Comparison among BatchEnsemble, naive ensemble and dropout ensemble over diversity metric. Each point the plot represents trained model where x-axis represents its accuracy on validation set and y-axis represents its diversity against the base model. The base model trivially has diversity. We plot the diversity of models trained on different proportions of training data, respectively 100%, 50%, 20% and 10%.
E.2 DIVERSITY METRIC ON PARTIAL TRAINING SET
There are two sources of uncertainty: aleatoric uncertainty and epistemic uncertainty. Epistemic uncertainty accounts for the uncertainty in the model we train. We focus more epistemic uncertainty because the aleatoric uncertainty difficult to measure. The simplest to magnify epistemic uncertainty to reduce the number of training data points. Under this case, want model to be more uncertain to reflect the lack of training data. Ovadia et al. (2019) showed that more diversity among ensembling member leads larger uncertainty score. There, under the case where only limited training data is provided, we hope that our ensembling method can produce more diverse member, compared training full dataset.
We repeated the experiments in Appendix E. with the exactly same diversity metric on models trained with proportional CIFAR-10 dataset. Figure 8b, Figure 8c and Figure 8d plotted the diversity measure with 50%, 20%, and 10% CIFAR-10 training data, respectively. The results showed that BatchEnsemble is on par with naive ensemble under limited training data. Figure 8b, shows that when we reduce the number of training data to half, BatchEnsemble achieves comparable diversity naive ensemble. It outperforms dropout ensemble by a significant margin. Figure Se and Figure 8d confirm the conclusion under even fewer training data points. Given the fact that diversity reflects epistemic uncertainty, we want our model have more diversity when only limited training data available. However, Figure 8 showed that dropout ensemble has the same diversity 100%, 50%, and 20% training data. This a significant flaw of dropout ensemble. In conclusion, BatchEnsemble leads to much more diverse member than dropout ensemble under the of limited training data. To supplement Figure 8, we provide the ensembling accuracy of BatchEnsemble, naive ensemble and dropout ensemble trained proportional training in Table 5.
17

---

95.59 Published as a conference paper at ICLR 2020
Table 5: Validation accuracy on ResNet32 with proportional training data. Ensemble with size MC-drop stands for dropout ensemble (Gal & Ghahramani, 2015). Single represents the accuracy of the base model in naive ensemble Figure 8.
Single
MC-drop
BatchEnsemble
NaiveEnsemble
CIFAR-10
95.22
95.40
95.61
96.09
CIFAR-10 (50%)
92.20
92.43
92.93
93.01
CIFAR- (20%)
82.32
84.5
86.08
86.17
CIFAR-I 10 (10%)
69.37
71.3
76.75
76.77
F COMPARISON TO NAIVE ENSEMBLE OF SMALL MODELS
In this section, we compare BatchEnsemble to naive ensemble of small models on CIFAR-10/100 dataset. To maintain the same memory consumption as BatchEnsemble, we trained 4 independent ResNet14x4 models and evaluate the naive ensemble on these models. This setup of naive ensemble still has roughly 10% memory overhead to BatchEnsemble. The results are reported in Table 6. shows that naive ensemble of small models achieves lower accuracy than BatchEnsemble. illustrates that given the same memory budget, BatchEnsemble is a better choice over naive ensemble.
Table 6: Supplementary result to Table 3. NaiveSmall is naive ensemble of 4 ResNer1 models. Vanilla, MC-drop and BatchEnsemble are still ResNet32x4 as in Table 3.
Vanilla
MC-drop
BatchEnsemble
NaiveSmall
CIFAR1O CIFAR 100
95.31 78.32
95.72 78.89
95.94 80.32
79.09
G PREDICTIVE DIVERSITY
As discussed in Section 2, ensemble benefits from the diversity among its members. We focus the set of test examples on CIFAR-IO where single model makes confident incorrect predictions while ensemble model predicts correctly. We used the final models reported in Section 4.3. In Figure we randomly select examples from the above set and plot the prediction map of single model, each ensemble member and mean ensemble. As we can see, although some of the ensemble members make mistakes thoes examples, the mean prediction takes the advantage of the model averaging and achieves better accuracy on CIFAR-10 classification task. We notice that BatchEnsemble preserves the diversity among ensemble members naive ensemble.
18

---

Published as a conference paper at ICLR 2020
0123436789 0125455759
0123456783
0323436783
0323438783
0123455789
Figure 9: Visualizing prediction diversity among BatchEnsemble (top row) and naive ensemble (bottom row) members on selected test examples on CIFAR-10. The y-axis label denotes mean prediction of ensemble (Mean), individual ensemble member prediction (from El to E4) and single model prediction (Single). Correct class is labelled as red. BatchEnsemble preserves the model diversity as naive ensemble.
19


Reviews:
[{"review_id": "rkljAl1hKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper aims to improve the efficiency of ensembles of neural nets in traditional supervised learning and life-long learning (learning on a series of tasks). The main idea is to let all the neural nets in an ensemble share the same weights W for each layer, and the weights for each neural net is generated by the Hadamard product of W and a specific rank-one matrix of the same size as W that is different across members in the ensemble. In experiments, they evaluate the method with some baselines on life-long learning, traditional classification, NMT tasks, and uncertainty modeling.\n\nThe paper relates the proposed method to several different learning problems and applications and lists many potential advantages in these applications: it covers a lot of things. However, it lacks in-depth discussion to several key problems, rigorous analysis or complete experimental study to support the main claims, for example:\n\nWhy can the simple method achieve a more compelling trade-off between accuracy and efficiency/memory costs comparing to a large single model or a naive ensemble of small models? Any mathematical or information-theoretical explanation behind that?\n\nIt is easy to understand that the ensemble defined here can improve efficiency and reduce memory cost. But as an alternative to the naive ensemble, we also expect the performance to not suffer from severe drawbacks. How to control efficiency-performance trade-off in the proposed method?\n\nHow were the baselines for each experiment selected? How to determine the specific setting in each experiment (any reason behind choosing the parameters in the settings)?\n\nIn the life-long learning settings, the shared weights W is only trained on the first task and then keeps fixed: this can leads to both large variance and bias. Why does it simply work well without causing any serious problems?\n\nThe rank-one extension of a shared model W enforces a very strong regularization to the model for each task. Will the method work promisingly when the tasks are more different from each other or harder to solve? For example, what if we increase the classes in each task? Is the rank-one extension still flexible and expressive enough to handle this situation?\n\nThese are some of the most important questions needed to be answered in the first place before showing higher evaluation metrics and listing the potential advantages of the proposed method. But it is not clear to me at all how they can be answered according to the contents in the current paper. I notice that the authors mentioned the last two questions at the end of Section 3, but no explanations/discussions were given.\n\nOther major concerns:\n\n1) Mathematically, comparing to single model Wx, the proposed ensemble method equals to applying a dimension-wise scaling to the input x and a dimension-wise scaling to the output Wx, and the scaling factors vary across different tasks. Hence, the proposed structure is exactly the same as fine-tuning two groups of batch normalization scaling factors before and after applying transformation W. It does not make much sense in the experiments that the performance of BN-Tuned in Figure 3a is much worse than the proposed method since they share exactly the same structure and math (note the memory and computational costs are also the same). The paper does not give an explanation about this. Moreover, the baseline BN-Tuned is only compared on only one of those datasets in the paper. It should be one of the most important baselines and needs to be compared in all experiments.\n\n2) On each benchmark dataset (except the last one), only 1-2 baselines are compared and most baselines are not state-of-the-art methods or not methods specifically designed for the problem (e.g., many are dropout and its variants). This makes the comparisons not convincing, especially considering that the experimental settings are determined by the authors and might be chosen for the best performance of the proposed method.\n\n3) At least two baselines should be included in all experiments: 1) single model with the equal number of model parameters, and 2) naive ensemble not sharing parameters across member models. However, each experiment only includes one or even none of these two baselines.\n\n4) Memory and training/test computational costs need to be reported for each experiment. However, the currently reported results are incomplete here and there.\n\n5) Comparing to the currently limited number of baselines on the incomplete evaluation metrics, the proposed method does not show significant improvements, for example, the results in Figure 4, Table 1 and Table 2.\n\n6) The proposed method requires the models for different tasks should have exactly the same architecture. This could be a strong limitation in many scenarios. For example, when different tasks have significantly different numbers of classes.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1xd6xjpFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new efficient ensembling method that has smaller memory footprint than naive ensembling and allows a simple parallelization on one device. The authors’ idea is based on sharing weights between individual ensembling models. The weights of each model can be represented as element-wise product of two matrices: shared one and matrix with rank 1 that can be efficiently stored.\n\nThe idea is quite interesting despite its simplicity. The experimental part is quite broad. I would like to highlight the lifelong learning as the strongest experimental result achieved by the authors. Despite the significant improvement on top of the baselines, this approach has one drawback described by the authors themselves. This method is difficult to generalize for the case of very diverse tasks despite its scalability. Nevertheless, I would not consider it as a large problem.\n\nI have a concern regarding ensembling. Do I understand correctly that in Figure 1 the method achieves almost constant test time cost only in the case of one device parallelization? If yes, then Figure 1 is slightly misleading and the description of this figure should be improved.\nIn the classification section the authors compare their approach only with MC-dropout. I would recommend adding other ensembling methods that have small memory footprint: e.g. [1], and can be better than MC-dropout. The same is true for machine translation section. \n\nThe authors emphasize that their approach is faster than consequently training independent models. However,  since these models are independent, it is possible to train them in parallel on multiple devices. The restriction to one device during training seems in general a bit artificial.\n\n\n[1]  Stefan  Lee,  Senthil  Purushwalkam,  Michael  Cogswell,  David  Crandall,  and  Dhruv  Batra.   Whym  heads  are  better  than  one:  Training  a  diverse  ensemble  of  deep  networks.arXiv  preprintarXiv:1511.06314, 2015b\n\n\nOverall, it is an interesting paper, that has several drawbacks.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxoU252KH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an ensemble method for neural networks, named BatchEnsemble, that aims to provide the benefits of improved accuracy and predictive uncertainty of traditional ensembles but with a significantly lower computational cost and memory cost. The method works by maintaining a shared “slow” weight matrix per layer, along with an ensemble of rank-1 “fast” weight matrices that are combined individually with the slow matrix via a Hadamard product in order to generate the network ensemble. The fast matrices can be stored as a pair of vectors, incurring a much smaller memory cost than a full rank matrix, and the prediction of an ensemble member can be vectorized such that the forward pass through the whole ensemble can be parallelized within a single GPU, yielding a computational speedup over traditional ensembles. The method is evaluated across a host of experimental settings, including image classification, machine translation, lifelong learning and uncertainty modelling. \nOverall, I recommend this paper to be accepted because:\n(i) the method proposed is simple to understand and implement, \n(ii) it yields clear computation and memory benefits over a traditional ensemble, \n(iii) the method is motivated by a good literature review, putting the approach and experiments conducted in context, \n(iv) while in terms of performance the experimental results are mixed, many different settings are evaluated, they are conducted fairly and they are transparently described, with the limitations are clearly acknowledged for the most part.\n\nSpecific comments / questions\n* Issues with BatchEnsemble as a lifelong learning method. When applied to lifelong learning, the slow weights are only tuned for the first task - as acknowledged by the authors, this means that forward transfer is only possible from the 1st task to all subsequent tasks and, more concerningly, it could severely limit the expressiveness of the ensemble for subsequent tasks that can only make a rank-1 adjustment to each layer. On the split-cifar and split-imagenet tasks this interestingly does not seem to be an issue, but one could imagine that it could be for tasks that differ more.\n    * Was the task split and order randomised for each run in Figure 3a and 3b? Would be interesting to know if the choice of first task matter for performance. Also, did the authors try not training the slow weights at all for the lifelong learning experiments? This would show how much the transfer from the first task helps the subsequent ones.\n    * In Figure 3b, it’s strange that EWC has a similar/ slightly higher forgetting than a vanilla neural network - do the authors have an explanation for this? Was the regularisation coefficient tuned for EWC?\n    * The proposed solution for enabling transfer beyond the 1st task is to enable lateral weights from features from previous tasks, as in progressive neural networks, but this would undermine the parameter efficiency of the model.\n* Machine translation experiments.\n    * BatchEnsemble on the attention layers of a transformer speeds up training in machine translation, but has little effect on final performance of the model versus a single transformer.\n    * Were any measures taken to equalise the number of parameters in the single transformer versus the ensemble method?\n    * Was a naive ensemble trained on the machine translation tasks for comparison?\n* Image classification experiments.\n    * It is hard to fairly compare the BatchEnsemble performance to the single model performance here given the 50% extra training iterations, but its encouraging that BatchEnsemble outperforms MC-dropout and comes close to the performance of a naive ensemble.\n* Predictive uncertainty / diversity. BatchEnsemble seems to perform well for uncertainty modelling in contextual bandits relative to a number of baselines.\n* How can the method be used as a basis for future work? It would be good to see some discussion of whether and how BatchEnsemble could be combined with other neural network ensemble methods.\n\nMinor comments not affecting review:\n* Section 4.4, paragraph 2, line 1 “uncertainty” misspelt.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed an improved ensemble method called BatchEnsemble, where the weight matrix is decomposed as the element-wise product of a shared weigth metrix and a rank-one matrix for each member.  The effectiveness of the proposed methods has been verified by experiments on a list of various tasks including image classification, machine translation, lifelong learning and uncertainty modeling.  The idea is simple and easy to follow.  Although some reviewers thought it lacks of in-deep analysis, I would like to see it being accepted so the community can benefit from it.
Decision: Accept (Poster)