Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper proposes a new efficient ensembling method that has smaller memory footprint than naive ensembling and allows a simple parallelization on one device. The authors’ idea is based on sharing weights between individual ensembling models. The weights of each model can be represented as element-wise product of two matrices: shared one and matrix with rank 1 that can be efficiently stored.

The idea is quite interesting despite its simplicity. The experimental part is quite broad. I would like to highlight the lifelong learning as the strongest experimental result achieved by the authors. Despite the significant improvement on top of the baselines, this approach has one drawback described by the authors themselves. This method is difficult to generalize for the case of very diverse tasks despite its scalability. Nevertheless, I would not consider it as a large problem.

I have a concern regarding ensembling. Do I understand correctly that in Figure 1 the method achieves almost constant test time cost only in the case of one device parallelization? If yes, then Figure 1 is slightly misleading and the description of this figure should be improved.
In the classification section the authors compare their approach only with MC-dropout. I would recommend adding other ensembling methods that have small memory footprint: e.g. [1], and can be better than MC-dropout. The same is true for machine translation section. 

The authors emphasize that their approach is faster than consequently training independent models. However,  since these models are independent, it is possible to train them in parallel on multiple devices. The restriction to one device during training seems in general a bit artificial.


[1]  Stefan  Lee,  Senthil  Purushwalkam,  Michael  Cogswell,  David  Crandall,  and  Dhruv  Batra.   Whym  heads  are  better  than  one:  Training  a  diverse  ensemble  of  deep  networks.arXiv  preprintarXiv:1511.06314, 2015b


Overall, it is an interesting paper, that has several drawbacks.

review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents an ensemble method for neural networks, named BatchEnsemble, that aims to provide the benefits of improved accuracy and predictive uncertainty of traditional ensembles but with a significantly lower computational cost and memory cost. The method works by maintaining a shared “slow” weight matrix per layer, along with an ensemble of rank-1 “fast” weight matrices that are combined individually with the slow matrix via a Hadamard product in order to generate the network ensemble. The fast matrices can be stored as a pair of vectors, incurring a much smaller memory cost than a full rank matrix, and the prediction of an ensemble member can be vectorized such that the forward pass through the whole ensemble can be parallelized within a single GPU, yielding a computational speedup over traditional ensembles. The method is evaluated across a host of experimental settings, including image classification, machine translation, lifelong learning and uncertainty modelling. 
Overall, I recommend this paper to be accepted because:
(i) the method proposed is simple to understand and implement, 
(ii) it yields clear computation and memory benefits over a traditional ensemble, 
(iii) the method is motivated by a good literature review, putting the approach and experiments conducted in context, 
(iv) while in terms of performance the experimental results are mixed, many different settings are evaluated, they are conducted fairly and they are transparently described, with the limitations are clearly acknowledged for the most part.

Specific comments / questions
* Issues with BatchEnsemble as a lifelong learning method. When applied to lifelong learning, the slow weights are only tuned for the first task - as acknowledged by the authors, this means that forward transfer is only possible from the 1st task to all subsequent tasks and, more concerningly, it could severely limit the expressiveness of the ensemble for subsequent tasks that can only make a rank-1 adjustment to each layer. On the split-cifar and split-imagenet tasks this interestingly does not seem to be an issue, but one could imagine that it could be for tasks that differ more.
    * Was the task split and order randomised for each run in Figure 3a and 3b? Would be interesting to know if the choice of first task matter for performance. Also, did the authors try not training the slow weights at all for the lifelong learning experiments? This would show how much the transfer from the first task helps the subsequent ones.
    * In Figure 3b, it’s strange that EWC has a similar/ slightly higher forgetting than a vanilla neural network - do the authors have an explanation for this? Was the regularisation coefficient tuned for EWC?
    * The proposed solution for enabling transfer beyond the 1st task is to enable lateral weights from features from previous tasks, as in progressive neural networks, but this would undermine the parameter efficiency of the model.
* Machine translation experiments.
    * BatchEnsemble on the attention layers of a transformer speeds up training in machine translation, but has little effect on final performance of the model versus a single transformer.
    * Were any measures taken to equalise the number of parameters in the single transformer versus the ensemble method?
    * Was a naive ensemble trained on the machine translation tasks for comparison?
* Image classification experiments.
    * It is hard to fairly compare the BatchEnsemble performance to the single model performance here given the 50% extra training iterations, but its encouraging that BatchEnsemble outperforms MC-dropout and comes close to the performance of a naive ensemble.
* Predictive uncertainty / diversity. BatchEnsemble seems to perform well for uncertainty modelling in contextual bandits relative to a number of baselines.
* How can the method be used as a basis for future work? It would be good to see some discussion of whether and how BatchEnsemble could be combined with other neural network ensemble methods.

Minor comments not affecting review:
* Section 4.4, paragraph 2, line 1 “uncertainty” misspelt.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper aims to improve the efficiency of ensembles of neural nets in traditional supervised learning and life-long learning (learning on a series of tasks). The main idea is to let all the neural nets in an ensemble share the same weights W for each layer, and the weights for each neural net is generated by the Hadamard product of W and a specific rank-one matrix of the same size as W that is different across members in the ensemble. In experiments, they evaluate the method with some baselines on life-long learning, traditional classification, NMT tasks, and uncertainty modeling.

The paper relates the proposed method to several different learning problems and applications and lists many potential advantages in these applications: it covers a lot of things. However, it lacks in-depth discussion to several key problems, rigorous analysis or complete experimental study to support the main claims, for example:

Why can the simple method achieve a more compelling trade-off between accuracy and efficiency/memory costs comparing to a large single model or a naive ensemble of small models? Any mathematical or information-theoretical explanation behind that?

It is easy to understand that the ensemble defined here can improve efficiency and reduce memory cost. But as an alternative to the naive ensemble, we also expect the performance to not suffer from severe drawbacks. How to control efficiency-performance trade-off in the proposed method?

How were the baselines for each experiment selected? How to determine the specific setting in each experiment (any reason behind choosing the parameters in the settings)?

In the life-long learning settings, the shared weights W is only trained on the first task and then keeps fixed: this can leads to both large variance and bias. Why does it simply work well without causing any serious problems?

The rank-one extension of a shared model W enforces a very strong regularization to the model for each task. Will the method work promisingly when the tasks are more different from each other or harder to solve? For example, what if we increase the classes in each task? Is the rank-one extension still flexible and expressive enough to handle this situation?

These are some of the most important questions needed to be answered in the first place before showing higher evaluation metrics and listing the potential advantages of the proposed method. But it is not clear to me at all how they can be answered according to the contents in the current paper. I notice that the authors mentioned the last two questions at the end of Section 3, but no explanations/discussions were given.

Other major concerns:

1) Mathematically, comparing to single model Wx, the proposed ensemble method equals to applying a dimension-wise scaling to the input x and a dimension-wise scaling to the output Wx, and the scaling factors vary across different tasks. Hence, the proposed structure is exactly the same as fine-tuning two groups of batch normalization scaling factors before and after applying transformation W. It does not make much sense in the experiments that the performance of BN-Tuned in Figure 3a is much worse than the proposed method since they share exactly the same structure and math (note the memory and computational costs are also the same). The paper does not give an explanation about this. Moreover, the baseline BN-Tuned is only compared on only one of those datasets in the paper. It should be one of the most important baselines and needs to be compared in all experiments.

2) On each benchmark dataset (except the last one), only 1-2 baselines are compared and most baselines are not state-of-the-art methods or not methods specifically designed for the problem (e.g., many are dropout and its variants). This makes the comparisons not convincing, especially considering that the experimental settings are determined by the authors and might be chosen for the best performance of the proposed method.

3) At least two baselines should be included in all experiments: 1) single model with the equal number of model parameters, and 2) naive ensemble not sharing parameters across member models. However, each experiment only includes one or even none of these two baselines.

4) Memory and training/test computational costs need to be reported for each experiment. However, the currently reported results are incomplete here and there.

5) Comparing to the currently limited number of baselines on the incomplete evaluation metrics, the proposed method does not show significant improvements, for example, the results in Figure 4, Table 1 and Table 2.

6) The proposed method requires the models for different tasks should have exactly the same architecture. This could be a strong limitation in many scenarios. For example, when different tasks have significantly different numbers of classes.
review_title: Official Blind Review #1

Meta-review: This paper proposed an improved ensemble method called BatchEnsemble, where the weight matrix is decomposed as the element-wise product of a shared weigth metrix and a rank-one matrix for each member.  The effectiveness of the proposed methods has been verified by experiments on a list of various tasks including image classification, machine translation, lifelong learning and uncertainty modeling.  The idea is simple and easy to follow.  Although some reviewers thought it lacks of in-deep analysis, I would like to see it being accepted so the community can benefit from it.
Decision: Accept (Poster)