Training independent subnetworks for robust prediction
Authors: Marton Havasi
Rodolphe Jenatton
Stanislav Fort
Jeremiah Zhe Liu
Jasper Snoek
Balaji Lakshminarayanan
Andrew Mingbo Dai
Dustin Tran
Venue: ICLR 2021 Poster (2021)
Keywords: Efficient ensembles
robustness

Abstract: Recent approaches to efficiently ensemble neural networks have shown that strong robustness and uncertainty performance  can be achieved with a negligible gain in parameters over the original network. However, these methods still require multiple forward passes for prediction, leading to a significant runtime cost. In this work, we show a surprising result:
the benefits of using multiple predictions can be achieved 'for free' under a single model's forward pass. In particular, we show that, using a multi-input multi-output (MIMO) configuration, one can utilize a single model's capacity to train multiple subnetworks that independently learn the task at hand. By ensembling the predictions made by the subnetworks, we improve model robustness without increasing compute. We observe a significant improvement in negative log-likelihood, accuracy, and calibration error on CIFAR10, CIFAR100,  ImageNet, and their out-of-distribution variants compared to previous methods.

Published as a conference paper at ICLR 2021
TRAINING INDEPENDENT SUBNETWORKS FOR ROBUST PREDICTION
Marton Havasi"
Rodolphe Jenatton Google Research jjenatton@google... com
Stanislav Fort
Department of Engineering University of Cambridge mh7 4 cam ac. uk
Stanford University
edu
Jeremiah Zhe Liu Google Research Harvard University jereliu@google.com
Jasper Snoek
Balaji Lakshminarayanan Google Research balajiin@googlle. com
Google Research jsnoek8google.com
Andrew M. Dai Google Research adai aaii@google.com com
Dustin Tran
Google Research
trandustiin@ppogle.ccm
ABSTRACT
Recent approaches to efficiently ensemble neural networks have shown that strong robustness and uncertainty performance can be achieved with a negligible gain parameters over the original network. However, these methods still require multiple forward passes for prediction, leading to a significant computational cost. In this work, show surprising result: the benefits of using multiple predictions can be achieved 'for free' under a single model's forward pass. In particular, we show that, using multi-input multi-output (MIMO) configuration, one utilize single model's capacity to train multiple subnetworks that independently learn the task at hand. By ensembling the predictions made by the subnetworks, we improve model robustness without increasing compute. We observe significant improvement in negative logelio,,,,od, accuracy, and calibration error on CIFAR CIFAR100, ImageNet, and their out-of out-of-distrioe difdiitrrbutionn variants compared to previous methods.
INTRODUCTION
Uncertainty estimation and dutoo-ddistributinn robustness are critical problems in machine learning. In medical applications, confident misprediction may be misdiagnosis that is not referred to physician as during decision-making with a "human-in-the tn-the-loopp.. This can have disastrous conse- quences, and the problem is particularly challenging as patient data deviates significantly from the training set such as demographics, disease types, epidemics, and hospital locations (Dusenberry al., 2020b; Filos al., 2019).
Using a distribution over neural networks is a popular solution stemming from classic Bayesian and ensemble learning literature (Hansen & Salamon, 1990; Neal, 1996), and recent advances such as BatchEnsemble and extensions thereof achieve strong uncertainty and robustness performance (Wen al., 2020; Dusenberry al., 2020a; Wenzel et al., 2020). These methods demonstrate that significant gains can be had with negligible additional parameters compared to the original model. However, these methods still require multiple (typically, 4-10) forward passes for prediction, leading to significant runtime cost. In this work, we show a surprising result: the benefits of using multiple predictions can be achieved "for free" under a single model's forward pass.
The insight we build on comes from sparsity. Neural networks are heavily overparameterized models. The lottery ticket hypothesis (Frankle Carbin, 2018) and other works on model pruning (Molchanov et al., 2016; Zhu & Gupta, 2017) show that one can prune away 70-80% of the connections in
Work done Google Research inter

---

Published as a conference paper at ICLR 2021
yy p(Dog)ne1%%
My
Vii p(Cal)-97%
Xii P(Cai)-95%
Ksi P(Car)=550
y: P/Cat)+81%
By p(Cat)-81%
(a) Training
(b) Testing
Figure I: In the multi-input output (MIMO) configuration, the network takes M inputs and gives M outputs. The hidden layers remain unchanged. The black connections are shared by all subnetworks, while the colored connections are for individual subnetworks. (a) During training. the inputs are independently sampled from the training set and the outputs trained classify their corresponding inputs. (h) During testing, the same input is repeated times and outputs are averaged in an ensemble to obtain final prediction.
neural network without adversely affecting performance. The remaining sparse subnetwork, called the winning ticket, retains its predictive accuracy. This suggests that neural network has sufficient capacity to fit 3-4 independent subnetworks simultancously. We show that, using a multi- multi- concurrently train multiple independent subnetworks within
output (MIMO) configuration, one network. These subnetworks co-habit the network without explicit separation. The advantage of doing this is that test time, we can evaluate all of the subnetworks at the same time, leveraging the benefits of ensembles in a forward pass.
Our proposed MIMO configuration only requires two changes a neural network architecture. First, replace the input layer: instead of taking single datapoint as input, take M datapoints as inputs, where M is the desired number of ensemble members. Second, replace the output layer: instead of single head, use M heads that make .M predictions based on the last hidden layer. During training, the inputs are sampled independently from the training set and each of the M heads is trained to predict its matching input (Figure la). Since, the features derived from the other inputs are not useful for predicting the matching input, the heads learn to ignore the other inputs and make their predictions independently. At test time, the same input is repeated M times. That is, the heads make M independent predictions on the same input, forming an ensemble for a single robust prediction that can be computed in single forward pass (Figure lb).
The core component of an ensemble's 's robustness such as in Deep Ensembles is the diversity of its ensemble members (Fort et al., 2019). While it is possible that single network makes confident misprediction, it is less likely that multiple independently trained networks make the same mistake. Our model operates the same principle. By realizing multiple independent winning lottery tickets, we are reducing the impact of one of them making a confident misprediction. For this method to be effective, it is essential that the subnetworks make independent predictions. We empirically show that the subnetworks use disjoint parts of the network and that the functions they represent have the same diversity as the diversity between independently trained neural networks.
Summary of contributions.
1. We propose a multi-input multi-output (MIMO) configuration to network architectures, enabling multiple independent predictions in single forward pass "for free." Ensembling these predic- tions significantly improves uncertainty estimation and robustness with minor changes to the number of parameters and compute cost.
2. We analyze the diversity of the individual members and show that they are diverse as independently trained neural networks.
3. We demonstrate that when adjusting for wall-clock time, MIMO networks achieve new state-of- the-art on 10, CIFAR 100, ImageNet, and their out-of-distiibution variants.
2 MULTI-INPUT MULTI-OUTPUT NETWORKS
The MIMO model is applicable in supervised classification or regression setting. Denote the set of training examples X ("), y (x)) ())) J where (n))) is the th datapoint with the corresponding label (m) and N is the size of the training set. In the usual setting, for input æ,, the output
2

---

Published as a conference paper at ICLR 2021
-1.0
member
-3.8
0.3
Dvye tho role
0.02
warelt Infi to su!
-120
Ve5
0.07
P00
0.02
033
-0.04
.0.0
-1.3
1.
35
550
1300
1500
2002
500
1063
1030
2200
500
1000
1502
2020
#5O0T6
<potte
Figure 2: Illustration of MIMO applied to a synthetic regression problem. (left) Example MIMO learning
M = 3 diverse predictors. As M increases, predicting with MIMO comes with higher bias but a smaller variance (two middle panels respectively). Despite the slight increase in bias, the decrease in variance translates into improved generalization performance (right).
of the neural network ý is probability distribution pe (v((a)), which captures the uncertainty in the predictions of the network. The network parameters 0 are trained using stochastic gradient descent (SGD) to minimize the loss L(0) on the training set, where the loss usually includes the negative log-likellng ligliikei and regularization term R (such as the L2 regularization): L(0) = Exx,y)xx(- log (y) + R(0).
In the MIMO configuration, the network takes M inputs and returns M outputs (Figure 1), where each output is a prediction for the corresponding input. This requires small changes to the architecture. At the input layer, the M inputs {xi,..., ...} are concatenated before the first hidden layer is applied and the output layer, the network gives M predictive distributions (Y1 [.,,..., 2t )..)...... Po (y ay [....... .MM) correspondingly. Having M input and M outputs require additional model parameters. The additional weights used in the MIMO configuration account for just 0.03 % increase in the total number of parameters and 0.01 % increase in Hooting--piint point operations (FI.Ps).2
The network is trained similarly to traditional neural network, with few key modifications to account for the M inputs and M outputs (Figure la). During training, the inputs X1.... XyM are sampled independently from the training set. The loss is the sum of the negative log-likelihoods of predictions and the regularization term:
M
(0) =
E (2) vi
Σ M
log (0 II....., RMd + +((0),
ex
(œM WM JEX
which is optimized using stochastic gradient descent. Note that the sum of the log-likelihoods equal the log-likelihood of the joint distribution EM log po (y ( aM) log (9).. YM 1 xx) since the input-output pairs are independent. Hence a second interpretation MIMO is that simply training a traditional neural network over M M-tuples of independently sampled datapoints.
At evaluation time, the network used make prediction on a previously unseen input æ'. The IM x' (Figure lb). Since all of the inputs are x', each of
input x' is tiled M times, so the outputs independently approximate the predictive distribution (y m x .e......"" x') ) zs o(y'(x') (for 1.... M). As ensemble, averaging these predictions improves the predictive performance, leading to the combined output po (y' (y' MM ΣM Po (y "e""...,''),
Unlike Bayesian methods requiring multiple weight samples, or even -efeeetter-fffiemtt methods like BatchEnsemble, MIMO's advantage is that all of the ensemble members can be calculated in single forward pass. As a result, MIMO's wall-clock time is almost equivalent to a standard neural network.

---

Published as a conference paper at ICLR 2021
MIMO architecture
Dssggrement
Disagreement subnet
Disagresment
Weight direction
0.03
37
1.7
07
35
15
05
0,713
orgo Eri
Not yrd
Weightspace direction
Weight space direction 1
Weigt space direction
Weight direction
Naive multihead architecture
Test accs, 3 subnets
6.06
Dsagrwement w/ submet
Disagresment subnet
Disagreement w/ subret
Weight space--recttoo diacciin2
Eais
6658
5:66
6.35
orgh 1210
0:25
0508
Weight space direction 2
Weight space direction
Weight spacae direction
Weight space direction
Figure 3: Accuracy landscape and function space landscape comparison of individual subnetworks MIMO (top row) and the naive multiheaded architecture (bottom row). (left): The test accuracy the weight space section containing M = 3 trained subnetworks and the origin. For the MIMO architecture, the individual subnetworks converge to three distinct low-loss basins, while naive multihead leads to the same mode. (middle- left right): The blue, red and green panels show the disagreement between the three trained subnetworks for the same section the weight space. For MIMO architecture, subnetworks often disagree, while naive multihead architecture they are all essentially equivalent.
2.1 ILLUSTRATION OF MIMO ON A SYNTHETIC REGRESSION EXAMPLE
Before applying MIMO large-scale vision models, first illustrate its behavior simple one-dimensional regression problem. consider the noisy function from Blundell et al. (2015) (see Figure 2, left), with a training and test set of N 64 and 3000 observations respectively. We train multilayer perceptron with two hidden- lalden-layers, composed (32, 128) units and ReLU activations.
For different ensemble sizes M (1....,5), mean squared error Exg. If denote by f's the regressor with M ensemble members learned over X, we recall that E = E(a' E over training sets of size N. We make two main observations.
5).
evaluate the resulting models in terms of expected
[Ex [Ex((far (a' (a',..., .') "] where Ex denotes the expectation
First, in the example of M in Figure (left), we can see that MIMO can learn diverse set of predictors. Second, the diverse predictors obtained by MIMO translate into improved performance, as seen in Figure 2 (right). Moreover, bias and variance components (Sec. 2.5 in Hastie et al. (2009)). More formally, we have
regression setting, we can decompose & into its (squared)
Exs =
E (f x') w'> (ae' w/ Eo
E
IB (Z ....,,,...........,, x') fM ( ()
(2rl )
(squared) bins
where fM = [M]] The bias-variance decomposition nicely captures the strength MIMO. While learning neural network over M-tuples induces slight bias compared standard model with M = i.e. the individual members perform slightly worse (Figure 2, middle-left), this is compensated by the diverse predictions of the ensemble members that lead lower variance (Figure middle-right). MIMO yields an improvement when the model has sufficient capacity to fit M > 1 diverse, well-performing ensemble members.
We use bold font to denote random variables and italic to denote their instantiations, for example, random
variable and its instantiation
standard ResNer28-11 has 36.479M parameters and takes 10.559G FL.OPs to evaluate, while MIMO configuration, it has 36.492M parameters and takes 10.561G FLOPPs to evaluate. We provide an interactive notehook that repraduces these experiments.
4

---

Published as a conference paper at ICLR 2021
0.43
0.3
0.22
100
0,0
Duagesaa
Dia.
Naive multiheul
0.000
0.000
0.1
TreeNet
0.010
0.010
1o1
BanchEneuuble
0.144
0.021
0.0
02 0.3 0.3 2200 00
Vertaie Ma)
Juy Mi
MTMO
Deep Ensecable
0.032
0.032
0.086
0.086
Var
6001 6.02 2.03
LCRE sco
Figure 4: Analyzing the subnetworks on the CIFARIO dataset. (left): Histogram of the conditional variances of the pre-activations w.r.t each input (M 2. ResNet28-10), (middle-left): Seatter plot of the conditional variances of the pre aetaviiiationss w.r.t. cach input. Almost all the pre-activations only have variance with respect to one of the inputs: the subnetwork they that are part of (M = 3. ResNet28-10, 10). (middle-right): Training trajectories of the subnetworks. The subnetworks converge to different local optima (M = 3. SmalICNN). (right): Diversity of the members (Dp) in different efficient ensemble models (ResNet 28-10).
3 UNDERSTANDING THE SUBNETWORKS
The mapping of each input-output pair in MIMO configuration is referred to as subnetwork. In this section, we show that the subnetworks converge to distinct local optima and they functionally behave independently trained neural networks.
3.1 LOSS-LANDSCAPE ANALYSIS
Using multiple inputs is the key to training diverse subnetworks. The subnetworks learn independently, since features derived from each input are only useful for the corresponding output. In contrast, in naive multiheaded architecture, where the input is shared, but the model has separate M outputs, the outputs rely on the same features for prediction, which leads to very low diversity.
To showcase this, we replicate a study from Fort et al. (2019). We look at the SmalICNN model (3 convolutional layers with 16, 32, 32 filters respectively) trained on CIFAR-10, and linearly interpolate between the three subnetworks in weight space. In the case of MIMO, we interpolate the input and output layers, since the body of the network is shared, and for the naive multiheaded model, we only interpolate the output layers, since the input and the body of the network is shared. Analogously to Deep Ensembles, the subnetworks trained using MIMO converge to disconnected modes weight space due to differences in initialization, while in the case of the naive multiheaded model, the subnetworks end up in the same mode (Figure 3, left). Figure (right) shows the disagreement i.e. the probability that the subnetworks disagree on the predicted class. MIMO's disconnected modes yield diverse predictions, while the predictions of the naive multiheaded model are highly correlated.
3.2 FUNCTION SPACE ANALYSIS
We visualize the training trajectories of the subnetworks in MIMO in function space (similarly to Fort et al. (2019)). As we train the SmalICNN architecture (M 3), we periodically save the predictions on the test set. Once training is finished, we plot the t-S projection (Maaten & Hinton, 2008) of the predictions. We observe that the trajectories converge to distinct local optima (Figure 4).
For quantitative measurement of diversity in large scale networks, we look at the average pairwise similarity of the subnetwork's 's predictions at test time, and compare against other efficient ensemble methods. The average pairwise similarity is
DD E ID (po (x..///xx........ poyy.
x))]
where D is a distance metric between predictive distributions and (x,y) E X. We consider two distance metrics. Disagreement: whether the predicted classes agree, Ddisagreement (P1, P,, P2) I(arg max, (0) = arg max, Pa(i)) and Kullback-Leiblr divergence: FPI,(P,, P2) Er, [log (1) log Pa(w). When the ensemble members give the same prediction at all test. points, both their disagreement and KL divergence are O.
The first efficient ensemble approach we compare against is the aforementioned naive multiheaded model, where the input and the body of the neural network is shared by the ensemble members, but each member has its own output layer. Next, TreeNer (Lee et al., 2015), where the input and the first two residual groups are shared, but the final residual group and the output layer trained separately for each member. Finally, BatchEnsemble (Wen et al., 2020), where the members share network parameters up rank-1 perturbation, which changes information flow through the full network.
5

---

Published as a conference paper at ICLR 2021
Test Accuracy (%)
Test llkellhood
Test Accuracy
Test llkellhood
96
80
75
94
-0.20
-1.0
93
Ensemble
Ensemble
70
Ensemble
Erisebb
Subnewwork
-0.25
Subnetwork
65
Subnetwork
-1.2
Subnstwork
2
3
M
(a) CIFARIO
(b) CIFAR100
Figure 5: The performance of the subnetworks and the ensemble of the subnetworks as the number of subnetworks (M) varies. M = 1 is equivalent to a standard neural network (R8-Net-28-10).
We find that the naive multiheaded model fails induce diversity: the predictions of its subnetworks are nearly identical on all test points as shown in Figure (right). TreeNet and BatchEnsemble have more diversity, although there is still significant correlation in the predictions. MIMO has better diversity than prior efficient ensemble approaches and it matches the diversity of independently trained neural networks.
These results allow us to pinpoint the source of robustness: The robustness of MIMO comes from ensembling the diverse predictions made by the subnetworks, thus MIMO faithfully replicates the behaviour of a Deep Ensemble within one network.
3.3 SEPARATION OF THE SUBNETWORKS
To show that the subnetworks utilize separate parts of the network, we look at the activations and measure how they react to changes in each of the M inputs. Namely. we calculate the conditional variance each pre-activation in the network with respect to each individual input. For input X1:
Var(o:|X2) = E (ar((a x2)] (M = 2), and Var(a, X2, X3) X2 at
E [Var(a;(X1, (2,, X2.K5
22, x3)] (M = 3)
where o; is the value of i-th activaivv in the function of the M inputs. For reference, there are 8190 pre-activations in a ResNet28-100 10. We can estimate the conditional variance by fixing and calculating the variance of a; w.r.t. X2 E X (when M = 2. X2, X3 E X when M 3) and finally averaging over the possible values X1 E X. The conditional variance is analogously defined for X2..... XMM the conditional variance of an activation is non-zero w.r.t. an input, that means that the activation changes as the input changes and therefore consider it part of the subnetwork corresponding to the input. the subnetworks are independent within the network, we expect that the conditional variance of each activation is non-zero w.r.t. of the inputs, the subnetwork to which it belongs, and close-to zero w.r.t. all the other inputs.
When we plot the conditional variances, this is exactly what we see. In Figure 4 (left), we see the histogram of the pre-activations in the network. Each point has two corresponding values: the conditional variance w.r.t. the two inputs. As we can see. all activations have non-zero conditional variance w.r.t. one of the inputs and close-to zero w.r.t. the other. Figure 4 (middle-left) shows the scatterplot of the activations for M 3. We see that, similarly to M 2, almost all activations have non-zero conditional variance w.r.t. one of the inputs and close-to zero conditional variance w.r.t. the others. Since almost all activations are part of exactly one of the subnetworks, which we can identify by calculating the conditional variances, we conclude that the subnetworks separate within the network. This implies an extension to Frankle Carbin (2018): the subnetworks realize separate winning lottery tickets within single network instance.
3.4 THE OPTIMAL NUMBER OF SUBNETWORKS
natural question that arises is the ideal number of subnetworks M to fit in a network. Too few subnetworks do not fully leverage the benefits of ensembling, while having too many quickly reaches the network capacity, hurting their individual performances. Ideally, we are looking to fit as many subnetworks as possible without significantly impacting their individual performances.
Figure shows the performance both the individual subnetworks and the ensemble as M varies. M = 1 is equivalent to traditionally trained network i.e. the performance of the subnetwork matches the performance of the ensemble since there is only one subnetwork. As M grows, we can see that the performance of the subnetworks slowly declines as they utilize more and more of the network

---

Published as a conference paper at ICLR 2021
Test Accurasy (%)
Test Log likelhoodd
Test Accuraiy 196)
Test Log likellhodd
96.5
-2.32
-0.13
76
93.5
-2.35
76
-0.95
95.0
-5.16
75
Reshett0
-1.00
Rsshet.0
94.5
74
Wide ResNut50
W ResNub5
94.0
-0.18
0.0
022
0.4
0.5
0.5
1.0
0.0
0.2
2.4
0.6
0.8
1.0
batch ropetitions
batch repetition
(a) Input repetition
(b) Batch repetition
Figure 6: (a) Performance of MIMO (M 2) as a function of p on ImageNet. p O, the subnetworks are independent and they are limited by the network capacity. With p > O, the subnetworks are able to share features and better utilize the network capacity. Wide ResNet has 2x more filters. (b) Repeating examples in the same batch improves convergence and yields a slight boost in performance.
capacity. The performance of the ensemble, however, peaks between M = 2 and M = 4, where the benefits of ensembling outweigh the slight decrease in performance of the individual subnetworks. Interestingly, the accuracy peaks earlier than the log-likelihood, which suggests that ensembling is more beneficial for the latter.
In Appendix C, we further illustrate how MIMO exploits the capacity of the network. In particular, we study the performance of MIMO when the regularization increases (both in terms of LI and L2 regularization), i.e. when the capacity of the network is increasingly constrained. In agreement with our hypothesis that MIMO better utilizes capacity, we observe that its performance degrades more quickly as the regularization intensifies. Moreover, the larger M, the stronger the effect. Interestingly, for the regularization, we can relate the performance of MIMO with the spursity the network, strengthening the connection to Frankle & Carbin (2018).
3.5 INPUT AND BATCH REPETITION
MIMO works well by simply adding the multi-input and multi-output configuration to an existing baseline, and varying only one additional hyperparameter (Section 3.4's number of subnetworks M). We found two additional hyperparameters can further improve performance, and they can be important when the network capacity is limited.
Input repetition Selecting independent examples for the multi-input configuration during training forces the subnetworks not to share any features. This is beneficial when the network has sufficient capacity, but when the network has limited excess capacity, we found that relaxing independence is beneficial. For example, ResNet50 ImageNet (He et al., 2016) does not have sufficient capacity to support two independent subnetworks (M = 2) in MIMO configuration.
Our proposed solution is to relax independence between the inputs. Instead of independently sampling Xi and X2 from the training set during training, they share the same value with probability p. That is, X1 sampled from the training set and X2 is set to be equal to x, with probability p or sampled from the training set with probability - p. Note that this does not affect the marginal distributions of x and X2, it merely introduces a correlation in their joint distribution.
Figure 6a shows the performance as varies. At O, the subnetworks are independent but their performance is limited by the network capacity. As P grows, the subnetworks share increasingly more features, which improves their performance. However, as p approaches 1, the subnetworks become highly correlated and the benefit of ensembling is lost. Unlike ResNet50, Wide ResNet50 has more capacity and benefits less from input repetition (roughly 78-79% vs 74-77%)
Batch repetition For stochastic models, most notably MC dropout and variational Bayesian neural nets, drawing multiple approximate posterior samples for each example during training can improve performance as reduces gradient noise w.r.t. the network's model uncertainty, e.g., Dusenberry et al. (2020a). We achieve similar effect by repeating examples in the minibatch: this forms new minibatch size of, e.g., 512 5 (batch size and number of batch repetitions respectively). Like the choice of batch size which determines the number of unique examples in the SGD step (Shallue et al., 2018), varying the number of repetitions has an implicit regularization effect. Figure 6b shows performance over the number of batch repetitions, where each batch repetition setting indicates box
7

---

Published as a conference paper at ICLR 2021
plot over sweep of 12 settings of batch size, learning rate, and ensemble size. Higher repetitions typically yield slight boost.
BENCHMARKS
We described and analyzed MIMO. In this section, we compare MIMO on benchmarks building on Uncertainty Baselines." This framework allows us to benchmark the performance and to compare against high-quality, well-optimized implementations of baseline methods (see framework for further baselines than ones highlighted here). We looked at three model/dataset combinations: ResNer28- 10/CIFARIO, ResNet28-10/CIE 10/CIFAR and ResNet50/lmageNe., MIMO's code open-sourced.
Baselines Our baselines include the reference implementations of deterministic deep neural network (trained with SGD), MC-Dropout, BatchEnnsemll, and ensemble models, as well as two related models, Naive multihead and TreeNet. Thin networks use half the number of convolutional filters while wide models use double. See Appendix B for the details on the hyperparameters.
Metries To measure robustness, we look at accuracy, negative log-likelihood (NLL), and expected calibration error (ECE) the IID test set well corrupted test set where the test images perturbed (e.g. added blur, compression artifacts, frost effects) (Hendrycks & Dietterich, 2019). Appendix includes ImageNet results for 5 additional ofitt--disttibution datasets. To measure computational cost, we look at how long it takes to evaluate the model on a TPUv2 core, measured in ms per example.
Name
Accuracy (1)
NLL. (L)
ECE ())
cAcc (1)
cNLLL (L)
<ECE (1)
Prediction
Forward
time (1)
passes (L)
Deterministic
0.159
0.023
76.1
1,050
0,153
0.632
Dropout
95.9
0.160
0.024
68.8
1.270
0,166
0.656
Naive mutlibead (M
95.9
0.022
0.969
0.141
0.636
MIMO (M = 3) (This work)
96.4
0.123
0.010
76.6
0.927
0.112
0.639
TreeNet
95.9
0.158
0.018
15.5
0.963
0.137
BatchEnsemble (M
96.2
0.143
0.021
77.5
1,020
0.129
2 2.552
4
Thin Ensemble (M 4)
96.3
0.115
0.008
17.2
0.840
0,089
0.823
Ensemble (M 4)
96.6
0.114
0.010
77.9
0.810
0.087
2.536
Table 1: ResNee28-10/CIFARII The best single forward pass results are highlighted in bold.
Predhction
IF Forward
Name
Accuracy (t)
NLL (L)
ECE (1)
cAce (f)
cNLL (1)
<ECE (1)
time (1)
passes (L)
Deterministic
79.8
0.875
0.086
51.4
2,700
0.239
0.632
Monte Carlo Drupout
79.6
0.830
0.050
42.6
2.900
0,202
0.656
Naive muilibead (M 3)
79.5
0.048
2.339
0.156
0.636
MIMO (M = 3) (This work)
82.0
0.690
0.022
53.7
2.284
0.129
0.639
TreeNet (M7
80.8
0.177
0.041
53.3
2,295
0,176
0.361
13
BatchEnsemble (M
81.5
0.740
0.056
54.1
2.400
0.191
2.552
4
Thin Ensemble (M 4)
81.5
0.691
0.017
53.7
2.190
0.111
0.823
Ensemble (M 4)
82.7
0.666
0.022
54.1
2,270
0.138
2.536
Table 2: ResNe22--10CIFFR 100: The best single forward pass results are highlighted in bold.
Name
Accumayy (?)
NLI.
BCF(()
cAsc (f)
cNI.I.())
oBCCE())
Prediction
(1)
passes
Forward
Deterministie
76.100
0.943
0.002
40.500
3.200
0.105
0.540
Naive mutlihead (M 3)
76.611
0.929
0.043
40.616
3.250
0.122
0.638
MIMO (M (p 0.6) (This work)
77.500
0.887
0.037
43,300
3.030
0.106
0.635
TreeNet (MV =
73.139
0.522
0.017
42.220
3,052
0.073
0.528
BatchEascmble (M
76.700
0.941
0.049
41.800
3.180
0.110
2 2.592
4
Ensemble (if
5.000
0.877
0.031
42.100
2.000
0.051
2.624
0.072
1.674
Wide MIMO (M (p 0.6) (This work)
79,300
0.843
0.061
45.791
3.048
0.147
1.706
Table 3: ResNer50/lmagNNet: The best single forward pass results are highlighted in bold.
The metrics show that MIMO significantly outperforms other single forward pass methods on all three benchmarks, It approaches the robustness of a Deep Ensemble, without increasing the computational costs.
// thub. http:///s/..tttocooooooooooo/co/ 1mm
8

---

Published as a conference paper at ICLR 2021
5 RELATED WORK
Multi-headed networks have been previously studied by Lee et al. (2015); Osband et al. (2016); Tran al. (2020). this approach to efficient ensembles, the input and part of the network are shared by the members while the final few layers and the outputs are separate. The advantage of the approach is that the computational cost reduced compared to typical ensembles, since many layers are shared, but the ensemble diversity (and resulting performance) is quite lacking. MIMO's multi-input configuration makes a significant impact as each ensemble member may take different paths throughout the full network. Further, MIMO has lower computational cost than multi-headed approaches, since all of the layers except the first and last are shared.
Related efficient ensemble approaches include BatchEnsemble, Rank-1 BNNs, and hyper batch ensembles (Wen et al., 2020; Dusenberry et al., 2020a; Wenzel et al., 2020). In these methods, most of the model parameters the model, but the evaluation cost still requires multiple forward passes. Interestingly, like MIMO, these methods also apply a multi-input multi-output configuration, treating an ensemble of networks as single bigger network; however, MIMO still outperforms BatchEnsemble. We believe important insights such as Section 3.5's input independence may also improve these methods.
shared among the members, which reduces the memory requirement of
Finally, there are simple heuristics which also retain efficient compute such as data augmentation, temperature scaling, label smoothing, contrastive training. These methods are orthogonal to MIMO and they can provide performance boost, without increasing the computational cost.
CONCLUSIONS
We propose MIMO, a novel approach for training multiple independent subnetworks within network. We show that the subnetworks separate within the model and that they behave independently trained neural networks. The key benefits of MIMO are its simplicity, since it does not require significant modifications to the network architecture and it has few hyperparameters, and also its computational efficiency, since it can be evaluated in single forward pass. Our empirical results confirm that MIMO improves performance and robustness with minor changes the number of parameters and the compute cost.
ACKNOW
Marton Havasi is funded by EPSRC. We thank Ellen Jiang for helping with writing.
REFERENCES
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In Proceedings of the 32nd International Conference on International Conference on Machine Learning- Volume 37, pp. 1613-1622. 2015.
Michael W Dusenberry, Ghassen Jerfel, Yeming Wen, Yi-an Ma, Jasper Snoek, Katherine Heller, Balaji Lakshminarayanan. and Dustin Tran. Efficient and scalable Bayesian neural nets with rank-1 factors. In ICML, 2020a.
Michael W Dusenberry, Dustin Tran, Edward Choi, Jonas Kemp, Jeremy Nixon, Ghassen Jerfel, Katherine Heller, and Andrew M Dai. Analyzing the role of model uncertainty for electronic health records. In Proceedings of the ACM Conference on Health, Inference, and Learning, pp. 204-213, 2020b.
Angelos Filos, Sebastian Farquhar, Aidan N Gomez, Tim GJ Rudner, Zachary Kenton, Lewis Smith, Milad Alizadeh, Arnoud de Kroon, and Yarin Gal. A systematic comparison of Bayesian deep learning robustness in diabetic retinopathy tasks. arXiv preprint arXiv: 10481, 2019.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv preprint arXiv:1912.0275: 2019. 2019.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 1803.03635, 2018.

---

Published as a conference paper at ICLR 2021
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions pattern analysis and machine intelligence, 12(10):993-100) 1990.
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The elements of statistical learning: data mining, inference, and prediction. Springer Science & Business Media, 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dictterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why M heads are better than one: Training diverse ensemble of deep networks. arXiv preprint arXiv:1511.065 2015.
Laurens van Maaten and Geoffrey Hinton. Visualizing data using t-SNE, Journal of machine learning research, 9NNov):5579-2605 2008.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016.
Radford M. Neal. Bayesian Learning for Neural Networks. Springer-Verlate New York, Inc., 1996.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. In NeurIPS, 2016.
Christopher Shallue, Jaehoon Lee, Joseph Antognini, Jaseha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism neural network training. arXiv preprint
2018
Linh Tran, Bastiaan S Veeling, Kevin Roth, Jakub Swiatkowski, Joshua V Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserving ensemble diversity for model distillation. arXiv preprint 1rXiv:2001.04694 2020.
Yeming Wen, Dustin Tran, and Jimmy Ba. BatchEnsemble: an alternative approach to efficient ensemble and lifelong learning. International Conference on Learning Representutions, 2020.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustness and uncertainty quantification. In Neural Information Processing Systems, 2020.
Michael Zhu and Suyog Gupta. To prune, not prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv: 2017.
10

---

Published as a conference paper at ICLR 2021
A PSEUDOCODE
Algorithm 1 Train(X)
1: for = 1... .....N Nuwr do
2:
(#1,M, y1:M) (
3:
Po(yi (1111111:1)) ....at)...PPP....
Po (1:M) - M1M((.
4:
(0) > -Md
po + R(()
5:
+- 0 1000-VVLi (0)
D e is the learning rate.
6:
end for
Algorithm 2 Evaluate(x')
1: (y X1:M x²).. po (y ( x') MIMO(XI.M x')
2: return M ΣMd (o XI:M x²)
B HYPERPARAMETERS
For the ResNet28- 10/CIFAR models, we use batch-size of 512, a decaying learning rate of 0.1 (decay rate 0.1) and L2 regularization 2c-4. The Deterministic, Dropout and Ensemble models are trained for 200 epochs while BatchEnsemble, Naive multihead and TreeNet are trained for 250 epochs.
For the ResNet50\maaeeNe models, we use batch-size of 4096 and decaying learning rate of 0.1 (decay rate 0.1) and L2 regularization le-4. The Deterministic, Dropout and Ensemble models are trained for 90 epochs, the BatchEnnemblle model is trained for 135 epochs and Naive multihead and TreeNet are trained for 150 epochs.
Regarding model specific hyperparameters, Dropout uses 10% dropout rate and a single forward pass at evaluation time. Both Ensemble and BatchEnsemble models use M members, since this provides most of the benefits of ensembling without significantly increasing the computational costs. The TreeNet architecture
MIMO For MIMO, we use the hyperparameters of the baseline implementations wherever possible. For the ResNet28- 10/CIFAR models, we use a batch-size of 512 with decaying learning rate of 0.1 (decay rate 0.1), L2 regularization 3e-4, 250 training epochs, and batch repetition of 4. For the ResNet50/mmageNer models, we use a batch-size of 4096 with decaying learning rate of 0.1 (decay rate 0.1), L2 regularization 1e-4, 150 training epochs, and batch repetition of 2. This makes the training cost of MIMO comparable to that of BatchEnsemble and Ensemble models. For the experiments. we use M 3 subnetworks because it performs well in both accuracy and log-likelinee ligeliidl.oodd For ResNet50/lmageNet the model has lower capacity so we used M 2 with p = 0.6.
C MIMO BETTER EXPLOITS THE NETWORK CAPACITY: PERFORMANCE VERSUS REGULARIZATION STRENGTH
In this section, we further illustrate how MIMO better exploits the capacity the network, through the lens of its sensitivity to regularization.
Our experimental protocol guided by the following rationale
Regularization controls the capacity of the network: the higher the regularization, the more constrained the capacity.
MIMO makes better of the capacity of the network: the more ensemble members, the more capacity is exploited.
As result, MIMO should be sensitive the constraining of the capacity of the network. And the more ensemble members (i.e., the larger M), the stronger the effect should be.
We consider the ResNer28-10/CIFARRI and paper where additionally vary the I.1 (respectively 1.2) regularization while keeping the other 1.2
settings used in the main
11

---

Published as a conference paper at ICLR 2021
cifar10
cifar10
1.0
test log log-likellhood
test accuracy
0.8
ens. size
-0.5
ens. size 1
ens, size
ens. size
0.6
ens, size
-1.0
ens. size 3
0.4
ens. size
1.5
ens. size
ens. size
ens. size 5
0.2
ens. size
-2.0
ens. size 6
6
5
-3
6
-3
l0g)0(f1)
log2o((1)
cifar100
cifar100
test accuracy
ens. size
test log -ikklllhodd
-1
ens. size
0.6
ens. size
-2
ens. size
ens. size
ens. size
0.4
ens. size
ens. size
0.2
size
size
ens. size
ens. size
-4
log)
log1o({1)
Figure 7: Accuracy and log-likelihood versus varying L1 regularization for ResNer28- 10 on CIFARIO (top row) and CIFAR100 (bottom row). Since MIMO better exploits the capacity of the network, its performance is more sensitive to the constraining of the capacity the regularization increases. The larger the ensemble size, the stronger the effect.
(respectively L1) term equal to zero. We display in Figures 7-8 the accuracy and log- liglliiel over those regularization paths (averaged over three repetitions of the experiments).
As previously hypothesised, we observe that MIMO is indeed more sensitive to the constraining of the capacity of the network as the regularization increases. Moreover, the larger the ensemble size, the stronger the effect. In the case of the L11 regularization, we can show how the accuracy and log-likelihood evolve with respect to the sparsity the network We report those results Figure where we can observe the same phenomenon.
D ADDITIONAL IMAGENET OOD RESULTS
In the following table, we evaluate trained ResNet-50 models on datasets. ImageNet, ImageNet-C, ImageNet-A, and ImageNetV2 each display three metrics: negative log-likelihood, accuracy, and expected calibration еттоr respectively. ImageNet-C further includes mCE (mean corruption error) in parentheses. ImageNet- MaageNet-ViidRobbeee Rib-Rousst, YTTB-Robust, and ObjectNet use their own pre-defined stability metrics.
These experiments were expensive run, so we were only able to obtain them for smaller set of methods. We find these results are consistent with the main text's benchmarks, showing MIMO consistently outperforms methods not only on corrupted images, but also across distribution shifts.
Name
ImageNet
ImageNet-C
ImageNet-A
Deterministic
0.939 / 76.2% /0.032
3.21 4...5% / 003 (75.4%)
8.09 0.7% / 0.425
MIMO (M = 2- 0.6)
0.887 / 77.5% / 0.037
3.03 43.3% / 0.166 (71.7%)
7.76 1.4% / 0,432
Wide MIMO (M 2. 0.6)
0.843 79.3% / /0066
3.1 / 45.0% 0.150 (69.6%)
7.33% 0.46
Name Deterministic
ImageNetV2
ImageNet- Vid Robust
YTTB RTTB-R
ObjeciNet
1 1.58 64.4% /0.074
29.9%
21.7%
25.9%
MIMO (M 2
1.51 / 65.7 % 0..08
31.8% %
22.2%
28.1%
Wide MIMO (M = 2, 0.6)
1.49 / 67.9% 0.109
35.3%
22.9%
29.5%
Table 4: ResNec50//maeNett & ImageNet OOD: The best single forward pass results are highlighted in bold.
A weight is considered to he zero if its absolute value larger than 10" 10-4
12

---

Published as a conference paper at ICLR 2021
cifar10
cifar10
test log log-likelihood
test accuracy
0.8
ens. size
-0.5
ens. size 1
ens, size
ens. size
0.6
ens. size
-10
ens. size 3
0.4
ens. size
1.5
ens. size
size
size
0.2
ens. size
-2.
ens. size 6
-3.0
-2.5
-2.0 -1.5
-1.0
-0.5
-2
-1
log10(f))
log2o(t2)
cifar100
cifar100
0.8
size
test log willlllellooooooooooooooo
-1
size
test accuracy
0.6
ens. size
-2
ens. size
ens.
ens.
ens. size
3
ens. size
0.2
ens.
ens. size
ens. size
ens. size
0.0
-3.5 -3.0
-2.5
-2.0 -1.5
-3.5
-3.0
-2. -2.0 -1. l0g]o(t2)
l0910(t2)
Figure 8: Accuracy and log log-likelihood versus varying L2 regularization for ResNet28- 10 on CIFARIO (top row) and CIFAR100 (bottom row), Since MIMO better exploits the capacity of the network, its performance is more sensitive the constraining of the capacity the regularization increases. The larger the ensemble size, the stronger the effect.
cifar10
cifar10
1.0
0.8
test log- logeliikllhhood
-0.5
test accuracy
ens. size
ens. size
0.6
ens. size 2
-1.0
ens. size
ens. size
ens. size
0.4
size
-1.5
size
ens. size 5
ens. size
0.2
size
-2.0
ens. size
-2
2
-2
logiol Knonzero weights)
1ogio(%nonzerro weights)
cifar100
cifar100
size 1
-1
size
test log likelihoodd
ens. size 2
ens. size 2
test accuracy
0.6
ens. size
-2
ens. size
ens. size 4
ens. size 4
0.4
ens. size 5
-3
ens. size
size 6
size
0.2
0.0
6
-2 log1o(%nonzee)) weights)
2
-4
-2
2
lonno(%oo weights)
Figure 9: Accuracy log-likelihood versus varying sparsity of ResNer28-10 on CIFAR10 (top row) and CIFAR100 (bottom row). Since MIMO better exploits the capacity of the network, its performance more sensitive to the sparsification of the network (as induced by an increasing LI regularization). The larger the ensemble size, the stronger the effect.
13


Reviews:
[{"review_id": "_Px3yjPuMoE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The ensemble method MIMO is proposed in this paper to reduce the inference delay and keep the prediction diversity. Only one model with sufficient capacity exists in this method while multiple implicit subnets are embedded in the parent model. Each subnet has individual I/O, so only one forward pass of the parent model is needed to process all subnets and make the ensemble.\n\nQuality: Medium to high. Pros: 1) The dissections of the subnets are impressive. They design the experiments to survey the loss plane, the parameter/activation projection of different subnets. 2) With the proposed training paradigm and proper test setting, the accuracy improvement can be seen and also uncertainty estimation. 3) They report the SOTA results of accuracy, uncertainty, and robustness on various datasets and their OOD variants when considering the inference latency. Cons: 1) They only report the inference time of one sample, but the total computation costs (e.g. MACs or FLOPS) are omitted. 2) The multiple branch networks are not only used in ensemble and broader usages exist. I know at least three works which involves multi-branch architecture (output-wise or layer-wise) in robustness or knowledge distillation and some of them are also using the ensemble of multiple predictions. The lack of citations in the related work seriously declines the quality of this work.\n\nClarity: High. Pros: 1) The method framework has a brief and clear explanation (Figure 1). 2) The training methods are conveyed in every detail, including some techniques like “input repetition”. 3) Some critical plots reporting accuracy using error bars or box plots to display the performance variance. Cons: Some format mistakes of symbols are existing. For example, the authors mix the usage of the normal and italic font of “x” when referring to samples in different expressions; the chaotic usage can even happen in the same equation for the first one of Section 3.3.\n\nOriginality: Low to medium. Pros: This method successfully uses the multi-branch architecture to reduce the inference delay in ensemble, which is a rather novel idea. Cons: As mentioned above, many similar multi-branch architectures have been used in different but related topics. Consider all of these works, the originality of this work has to be downgraded.\n\nSignificance: Low to medium. Pros: The shining points of this work are making ensemble for “free”. They do decrease the inference delay significantly. Cons: 1) The authors attempt to distract our attention on the computational costs of this MIMO architecture and try to make an illusion that it’s convenient in computing. The latency is decreased at the cost of batch size. Other ensemble methods have a longer delay, but they can process a batch of images. This method fills the batch with the same image which implicitly decreases the batch size. Furthermore, no measurement is used on computation costs or other related aspects. I think it’s deceptive and tricky. 2) As shown above, the originality of this work is not as solid as their experiments. \n\n[ Detailed comments]\n1. What are the structural considerations for the related work of Section 5 not to be explained in Section 2?\n2. In Figure 6, the ‘#’ marked on the abscissa of (b) is redundant\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "QmT9IAdEHfO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors assessed how these subnetworks can be as diverse as independently trained networks. The contribution of this paper is in proposing an approach to improve uncertainty estimation and robustness with minor changes (1 percent) to the number of parameters and compute cost. \n\nSTRENGTHS: \nTraining multiple independent subnetworks within a network, with minimal increase in number of parameters. \nThe use of MIMO makes this approach simple, while it can be evaluated in a single forward pass.  \n\nCONCERNS:\nThe authors claim that the benefits of using multiple predictions can be achieved ‘for free’, while their proposed model increases the number of parameters (even though by 1 percent)\nThe paper has examined the accuracy and disagreement of the subnetworks, but a detailed evaluation on number of parameters is missing (i.e. where the 1% increase in parameters comes from).\nAn experiment on more diverse datasets would be also helpful, such as OpenImages. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5o6eaFZLv_", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY:\nThis paper describes a multi-input multi-output (MIMO) strategy for training several subnetworks inside a same and single neural network for robust prediction. The approach consists in jointly training the heads to make predictions for their corresponding inputs. The strategy is simple and demonstrates strong results. The experimental study reveals that subnetworks functionally behave as independent networks, hence resulting in a strong and robust ensemble. \n\nSTRENGTHS:\n- The experimental study is very thorough. I really appreciated the investigation in Section 3, which indeed convincingly shows that the subnetworks behave as independent. \n- The method is compared on standard benchmarks, across a wide range of metrics. Experimental results show better performance than single forward pass methods.  Performance is reasonable with respect to a simple solution consisting in training an actual ensemble of 4 networks.\n- The paper is well written and easy to follow.\n\nWEAKNESSES:\n- I believe the approach to be original, but its similarities/differences with other multi-input multi-output (such as BatchEnsemble) could have been discussed much further to better appreciate the originality of MIMO.\n- Although independence between subnetworks is shown empirically, I cannot help but wonder how/why it emerges from the architecture! This is an exciting phenomenon that ought to be better understood. Nevertheless, I believe the actual independence of subnetworks should be nuanced at places (e.g., in the title or in the abstract), as this highly depends on the capacity of architecture and on the problem to solve -- as shown in the experiments themselves. \n- Some (hypothetical) theoretical explanations regarding the emergence of independence would have made the paper stronger, although I realize this would be a whole new paper by itself.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "0FluA_PN3gR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to train a single network with M input examples and M corresponding predictions, and the M input examples are mixed to produce the M corresponding predictions. Although only a single network is learned, it implicitly consists of multiple sub-networks due to the nature of multiple inputs and multiple outputs in training. In testing, the single testing example can be replicated M times as inputs, so that M outputs are produced by the trained network. The multiple outputs enable efficient ensembing for robust prediction. \n\nI find the proposed idea very clever. The empirical results on toy data and real data are interesting and compelling. It demonstrates that a single network has the capacity to contain multiple sub-networks, which is an interesting discovery in itself. \n\nI do have two main reservations. The first one is the lacking of some basic, not necessarily rigorous, theoretical formulation and analysis. The second one is about its practical potential. Apparently M has to be rather small, due to the limited capacity of a single network. Its advantage over training multiple networks may not be dramatic. \n\nI am also a bit concerned that the network is trained on M independent examples (although the proposed method does allow for occasional identical examples), but is tested on M identical copies of the same testing example. \n\nI am also unclear about the nature of mixing M independent training examples, and the effect of doing that. That is why I feel some theoretical understanding is needed. \n\nWhat if we permute the M training examples and check the difference of the corresponding outputs? \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple but effective method to obtain ensembles of classifiers (almost) for free. 
Essentially you train one network on multiple inputs to predict multiple outputs. The authors show that this leads to surprisingly diverse networks - without a significant increase in parameters - which can be used for ensembling during test time. 
Because of its simplicity, I can imagine that this approach could become a standard trick in the "deep learning tool chest". 

-AC
Decision: Accept (Poster)