Inner Ensemble Networks: Average Ensemble as an Effective Regularizer

Abstract: We introduce Inner Ensemble Networks (IENs) which reduce the variance within the neural network itself without an increase in the model complexity. IENs utilize ensemble parameters during the training phase to reduce the network variance. While in the testing phase, these parameters are removed without a change in the enhanced performance. IENs reduce the variance of an ordinary deep model by a factor of 1/mL−1, where m is the number of inner ensembles and L is the depth of the model. Also, we show empirically and theoretically that IENs lead to a greater variance reduction in comparison with other similar approaches such as dropout and maxout. Our results show a decrease of error rates between 1.7\% and 17.3\% in comparison with an ordinary deep model. We also show that IEN was preferred by Neural Architecture Search (NAS) methods over prior approaches.

Reviews and author responses:

The authors propose a drop-in replacement for CNN and FC layers that use multiple instances of the same layer and apply the average operation to those layers’ output. For inference, the weights are averaged themselves, so the params count in inference is the same as without the inner ensembles. The authors show that IEN ensembles of size m decrease the overall variance of a deep network by a factor of $\frac{1}{m^{L-1}}$. They also perform experiments with various image models and finally show theoretically and empirically that their method leads to a greater decrease in variance and error rate than similar methods (maxout, dropout).

Pros:
+ A simple and practically viable variance reduction scheme
+ Good experimental design for the main set of experiments that shows clear advantages of the proposed scheme

Cons:
- The theoretical analysis lacks rigor. There are hidden  assumptions in the methods used to achieve the conclusions. Please see the comment section below for more details.
- The proposed method being simple means a lot of related works used methods similar to the proposed one. The related work section lacks a clear explanation for what distinguishes the paper from the others and makes it novel (for example, comparing to Abbasian et al., 2013)

General comments/questions:
* (Section 3) It seems that $x_l$ should be $n_l$ dimensional and $y_l$ should be $d_l$ dimensional for variance in (4) to be correct
* (Section 4.1) The derivation from (6) to (7) is too important to be hidden inside the appendix as it hides the assumptions made in A.2. Better move it to the main part of the paper.
* (A.2) There are two parts here aiming at one goal: one is (24)-(29) and one from (sgK). Better keep only one of them for clarity.
* (A.2) The equations in (24)-(29) are not generalized. In (24) and in (sgK) x is assumed to have zero mean. The authors have to prove (24)-(29) without this assumption. This can be done for specific activation functions, for example, RELU, but for RELU (24) doesn’t stand true without any additional assumptions.   
* (Section 4.1) Equation (11) should probably use n_l
* (Table 1) What is $Base_{\widetilde{w}}$?
* (Table 1) This table has the $IEN+FC_{\tilde{w}}$ results but no $IEN+FC$ results. Can those be added for the sake of completeness? 
* (Table 1, Figure 2) Nit: the extensive use of colors makes the paper harder to process for colorblind people 
* (Section 2) (Abblasian et al., 2013) seems to be the first paper introducing Inner Ensembles. The authors should have a more extensive explanation of what is new in their paper and what distinguishes it from that existing one.
* (Section 2) (Opitz et al., 2017) does not force the use of a special loss function instead of a main loss but instead suggests using additional loss function to achieve better performance through making weights more diverse. Potentially Inner Ensemble Networks may benefit from using a similar loss (but maybe not the $IEN_{\widetilde{w}}$ versions). Adding those additional losses may be an interesting new set of experiments to run. 
* (Table 2) It is unclear why the outer ensemble outperforms single model IEN (not the $IEN_{\widetilde{w}}$). The paper will benefit from a discussion on why this may be happening. Is it because IEN makes model weights too close to each other?  
* (G) The dropout results are very poor. It suggests the models were not operating in a normal state. This may be the same issue as discussed in https://arxiv.org/abs/1603.05027. The https://arxiv.org/abs/1605.07146 suggests that proper placement of dropout can fix this problem.. In any case, comparing Dropout+IEN and Dropout is misleading as all Dropout experiments look catastrophically broken
* (Section 5) A more detailed description of CNN setup with IEN would be for the paper’s benefit.   
* (Table 1) Was maxout also used as a drop-in replacement without hyperparams tuning? It may be beneficial to also tune both maxout and IEN to distinguish IEN advantage of being a drop in replacement and possible IEN advantage when both methods are tuned to their maximum potential


On rating:
The Inner Ensemble Networks method looks like a simple, clean and easily reproducible way to reduce variance within a network. The authors extensively test their method empirically and show that this method also decreases error rates for various image-based models. However, the presented theoretical analysis lacks rigor and has to be updated to be formally correct.

Update: taking authors comments and improvements into account I've updated the rating from 6 to 7.


**Summary**
This paper proposes "Inner Ensemble Network" that ensembles intermediate layers in a deep neural network, which would be trained simultaneously during the training. On the inference time, the ensembled portion of the network would be substituted by the averaged layer, enabling lighter-weighted inference while preserving the benefit from the ensemble. Similar to the classic ensemble methods, this method can reduce the variance of the training and achieve better than the models without the ensemble.

**Originality and significance aspect**
This is probably the most weak aspect of this paper. The proposed methodology, Inner Ensemble Networks (or IEN) is very similar to the idea of average pooling layers that are widely used in computer vision architectures.  It is also fairly similar to some variants of online ensemble methods such as [1, 2, 3]. [2] is actually already cited in this paper; however, my point is that there is not enough novelty on this paper compared to these prior works. Authors claim this paper is simpler to apply than [2], but they do not have any empirical evidence that their proposed method is better than [2]. What if [2] works much better because their *sophisticated* loss function actually works out nicely?

From my understanding, IEN works just as the average pooling layers of duplicate layers and probably only novelty here is that they substitute the ensembled layers with the averaged layer similar to drop-out technique. I feel this is not enough for a full conference paper even though this paper may be still useful in practice.

[1] Liu, Yong, and Xin Yao. "Simultaneous training of negatively correlated neural networks in an ensemble." IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 29.6 (1999): 716-725.
[2] Opitz, Michael, Horst Possegger, and Horst Bischof. "Efficient model averaging for deep neural networks." Asian Conference on Computer Vision. Springer, Cham, 2016.
[3] Lee, Stefan, et al. "Why M heads are better than one: Training a diverse ensemble of deep networks." arXiv preprint arXiv:1511.06314 (2015

**Quality aspect**
The paper could be polished further in their figures and equations. For example the bottom line of the right side table of Figure 1 is missing and the left picture is not that informative. Many of the equations are not typesetted correctly. I think authors may consider using LaTex for tye type setting. The parenthesis in the equations (e.g. eq 4-11) should match the inner element’s size.

For the experiments, I think authors should actually use the classic version of ensemble (call outer-ensemble in Section 5.2 or in Table 2) as their baseline instead of a single model. It is because IEN costs almost 2x during the training and that is similar to the classic ensemble. If authors are concerned about the inference computation time, they can consider a distilled model from the classic ensemble method, which is a fairly standard technique.

**Clarity aspect**
The paper is mostly clear. 

**Recommendation**
Overall, the paper is interesting; however, I cannot recommend this paper mainly due to weak novelty and weak empirical support.

----
**Post rebuttal comment**
I thank the authors for detailed rebuttal and new empirical results. I also have read other reviewer's comment, and decided to keep my original score. The main concern of this paper, the weak novelty, still remains (also pointed out by Reviewer#1/4). 



-Regards the connection with Opitz and other works:

 We want to highlight that Opitz[2] lies within a group of work where they focus on doing ensembles using the shared parameters in the head (last layer) of the network. This line of work cares about introducing a loss function that encourages the last layer, unlike our where it can be applied everywhere in the model. For the empirical comparison with Opitz[2]: It is really difficult to compare our method with Opitz et.al[2] empirically because their code is not open sourced, we also traced down the papers that mention[2] to see if they are able to replicate the results or not, or even have the code open source but none reported an empirical number rather than discussing the idea of [2]. We also contacted the authors of [2] to see if they are able to share their code with us or not, yet to the moment we didn't receive a response. Yet, we tried to replicate the base model that is mentioned in [2] and followed the written settings with tracing back to Theano python library as the [2] mentioned it was implemented using it. We tried our best to replicate the base model in the paper in which we are at 80.49 test accuracy but the paper reports a 80.86 test accuracy. We trained IEN on the base model and we obtained an accuracy of 82.71 that's 2.74% increase in accuracy while Opitz[2] on the reported results[ Base: 80.86, Opitz Loss 82.3] achieves 1.7% accuracy increase. For [1] it introduced the negative correlation loss, in which they train separate individual networks that use a loss function to encourage the decorrelation between individual networks. This is a kind of an outer ensemble, [1] also is the work of [4] same authors where they analyzed the negative correlation more. In [2] they compared the negative correlation with their base model and it only resulted in 0.04% increase in the accuracy. [3] Proposes a specific architecture called TreeNet that can branch from a middle point into a couple of individual architectures, yet their proposed architecture isn't generic as ours and can't act as a single model in the end. Ours is a general operator that can be applied on any other architecture without requiring a specific geometry, specific loss function and hyper-parameters settings. We added Table#8 in Page 9 that summarizes these numbers. For convenience we show this table in text format: 

|                            | Accuracy | Accuracy Increase from Baseline |
| :------------------------------------- | :------- | :------------------------------ |
| Opitz Baseline                                     | 80.86   | -                               |
| DivLoss on Opitz Baseline              | 82.30   | 1.7%                            |
| Negative Correlation on Opitz Baseline   | 80.90   | 0.03%                          |
| Reproduced Baseline                    | 80.49   | -                               |
| IEN on the reproduced Baseline         | 82.71   | 2.74%                           |


[1] Liu, Yong, and Xin Yao. "Simultaneous training of negatively correlated neural networks in an ensemble." IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 29.6 (1999): 716-725.
[2] Opitz, Michael, Horst Possegger, and Horst Bischof. "Efficient model averaging for deep neural networks." Asian Conference on Computer Vision. Springer, Cham, 2016. 
[3] Lee, Stefan, et al. "Why M heads are better than one: Training a diverse ensemble of deep networks." arXiv preprint arXiv:1511.06314 (2015).
 [4] Liu, Yong, and Xin Yao. "Ensemble learning via negative correlation." Neural networks 12.10 (1999): 1399-1404.

We fixed the equation parenthesis and updated figure#1 to be more informative.  
*Updated Opitz baseline

Dear Reviewer#2,
Thanks a lot for your review and taking care of the fine details of the proof and pointing us to related work! Thanks a lot for your time and effort!

- Regards the connection with (Abblasian et al., 2013):

 Thanks for pointing us to the work of[1], though we share the same title of inner ensemble but we do operate differently. In our work we care about regularization and reducing the variance within a deep model itself and provide a plug and play tool that can be applied to any deep learning architecture no matter what it looks like. We also provide a theoretical guarantee for the variance reduction. The work of Abbasian et al.[1] focuses on the decisions made by the algorithm that defines the architecture and the model choice itself. For example when it is applied on a Bayesian neural network, [1] it builds the structure of the network during the learning phase of the algorithm. They use algorithms  that search for the best network structure among all possible ones. The internal decisions of the algorithm are being ensembled to have a better decision, same applies for the proposed K-means in their work. In other words[1] work is about enhancing the decision of algorithms that controls the constructions of a machine learning model by inner ensembling the decisions made by these algorithms. 

 [1]Abbasian, Houman, et al. "Inner ensembles: Using ensemble methods inside the learning algorithm." Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, Berlin, Heidelberg, 2013.

-Regard (Opitz et al., 2017): 
Indeed they do not force using a specific loss function as stated. Also, we tried to replicate Opitz work on their Baseline. Our main issue that the code is not open sourced as discussed with Reviewer#3, Yet to our best we reproduced a baseline that’s close to Opitz baseline using the information that’s provided in the paper. We showed that ours IEN results in 2.74% accuracy increase versus 1.70% accuracy increase when the Opitz loss is utilized. The results can be found int Table#8 on page 9. In future work we will consider such an approach but for now we are presenting a plug and play tool that can be used on prior architecture without a change in loss function or the design of the architecture itself.
For convenience we show this table in text format: 

|                            | Accuracy | Accuracy Increase from Baseline |
| :------------------------------------- | :------- | :------------------------------ |
| Opitz Baseline                                     | 80.86   | -                               |
| DivLoss on Opitz Baseline              | 82.30   | 1.70%                            |
| Negative Correlation on Opitz Baseline   | 80.90   | 0.03%                          |
| Reproduced Baseline                    | 80.49   | -                               |
| IEN on the reproduced Baseline         | 82.71  | 2.74%                           |

[1] Opitz, Michael, Horst Possegger, and Horst Bischof. "Efficient model averaging for deep neural networks." Asian Conference on Computer Vision. Springer, Cham, 2016. 

*Updated Opitz baseline

-Regards comparison with regular (outer) ensemble

Thank you for highlighting this point of comparison with classic ensembles (outer ensemble) in our paper. The main purpose of IEN is to show a technique that is borrowed from classic ensemble and apply it within the deep model itself. We are not competing against classic ensembles, rather we show an alternative to different competitive deep learning regularizers such as dropout and maxout that is inspired from classic ensemble. In Table 2 we showed the results against an outer ensemble of a base model (4 models) to an ensemble of IENs models. This table results shows that outer ensembles still have it’s benefits and IEN can be added to it to have better accuracy. Thus, we show that both outer and inner ensembles can co-exist and the deep learning community can benefit from both. We also highlighted in our work as mentioned by your comment that the final model benefits from ensemble mechanism yet it preserves the same model parameters while not having a training time as a regular outer ensemble. We thank you for this point and will add all of this response to the final version. 

* We fixed the equation parenthesis and updated figure#1 to be more informative. 



Regards “My concern is the motivation of why variance reduction is needed in deep neural networks is not highlighted…”: 

Indeed many work on deep ensemble encourages diversity of the model which accounts to a larger model variance. In our case IEN happens to be a regularization method that tends to improve the model accuracy yet it holds the property (behaves like) of the known regular (outer ensemble), thus we used variance analysis tools to show the regularization effect of IEN as it decreases the variance. Also, our assumption that the single model with lower variance as we showed in the analysis to have a better performance is supported by empirical results that connects with analysis correctly. Even in the case of our variance analysis for Maxout we noticed some explosion (bad accuracy) which aligns with our findings in maxout too. This is to our best understanding for the raised concern and we hope that our answer is satisfactory, if not please share with us more details so we can enrich the work. In the final version we will compile all of these points and highlight the variance reduction motivation. 


We sincerely thank all the reviewers for their time and effort reviewing this work and analyzing it. The reviews were extremely helpful in assuring the performance of the method and completing the comparison with prior works. We highlight the pros mentioned by the reviewers that our method is simple and viable for variance reduction supported with a set of empirical results supported by theoretical analysis that matches the empirical results. 
We highlight a summary of our response: 
- We proved that ours is not equivalent to the base model as R1 suggested. 
- We discussed the connection with prior works and added empirical comparison with Optiz as per R2 and R3.
- We discussed our connection with avgpooling and supported our claims empirically as suggested by R3. 
- We showed IEN performance on out of distribution datasets such as CIFAR-10C/100C and connected with a close optimization method SWA where we show the benefits of IEN as R4 suggested. 
 
We posted an updated version (not final) of the paper where we used page 9 to add empirical results requested by the reviewers to clarify different aspects of the paper and support our discussion. We also made the proof more rigorous while fixing some equations typesettings beside updating figure#1 to be more clear. 

Dear Reviewer#4:

Thank you for taking time reviewing our work. We also thank you for pointing out different mechanisms that ensure our proposed operator generalization! 


- Regards out of distribution datasets [CIFAR-10C, CIFAR-100C]: 

We thank you for this suggestion that will give a better view regarding the generalization of our method. We report the mean accuracy results on CIFAR-10C and CIFAR-100C out of distribution datasets. The models were only evaluated on these datasets and not trained on it as suggested in [1]. The results are in Table 5 on page 9 of the updated paper. We notice that in the weather corruptions such as frost and fog on our method IEN outperform the base and maxout. In the Blur case such as zoom and defocus, IEN performs better when it's used in the DenseNet architecture while in the ResNet architecture the base performs better in most of the cases. On the other hand maxout tends to perform better in the Noise group. The Digital noise group tends to swing between the 3 options IEN, maxout and base. Overall, each method apparently performs well in a specific group depending on the model used and the dataset in hand. We also would like to highlight that all of the models were trained on hyper-parameter settings that work best for the base model and nothing was tuned for IEN or Maxout. We thank you again as these findings are indeed interesting! 

[1]Hendrycks, Dan, and Thomas Dietterich. "Benchmarking neural network robustness to common corruptions and perturbations." arXiv preprint arXiv:1903.12261 (2019).

- Regards the comparison with SWA: 

Thank you for pointing this out, it's really helpful. SWA is an optimizer that uses the idea of averaging multiple points in the space to achieve a better performance, while IEN is an operator that can be applied architecture wise. They both share the same idea in testing time; they act as a single model but both add extra time in the training to achieve a better performance. Though SWA requires a specific learning rate mechanism, ours IEN can be applied out of the box without a change of the training mechanism. We see both methods can co-exist and be utilized to achieve a better optima. We ran SWA baseline code using the same settings on both PreResNet and WideResNet. The results are shown in Table#6 on page 9 of the updated paper. We notice that IEN is better than SGD and whenever IEN applied with SWA we achieve better results. For example IEN boost the WideResNet by 0.8% in terms of accuracy, while if both IEN and SWA applied we achieve 3.25% increase in model accuracy compared to 1.64% when SWA is used alone. It's like what you suggested: SWA  takes care of the global network while IEN takes care of the local network mean. For convenience we show a text table version of Table#6:

| DNN(Budget)               | SGD                                                        | SGD + IEN            | SWA 1.5 Budgets | SWA 1.5 Budgets + IEN      |
| :------------------------ | :--------------------------------------------------------- | :------------------- | :--------------- | :-------------------------- |
| PreResNet110(150)     | 76.77 +- 0.38                                         | 77.05 +- 0.14   | 79.10 +- 0.21   | 79.19 +- 0.25          |
| WideResNet28x10 (200)     | 80.82 +- 0.23                                             | 81.47 +- 0.14   | 82.15 +- 0.27   | 83.45 +- 0.13          |

[1]Izmailov, Pavel, et al. "Averaging weights leads to wider optima and better generalization." arXiv preprint arXiv:1803.05407 (2018).

Dear Reviewer#3:

Thanks for your review and especially highlighting the resemblance with avg pool giving us a chance to clarify this point and the connection with relative work. 

 -  Regards the connection with avg pooling: 

Thanks for highlighting the connection with avg pooling. Though avg pooling and ours IEN uses an average operator, ours differs from avg pooling in that it executes on the pixel level of the same channel on multiple m outputs. In avg pooling it does a pooling on a patch of pixels within the same channel. Thus average pooling does compute in a spatially separated manner unlike ours. Also, ours proposes a new initialization method that decreases the variance of the deep model. Same comparison can be applied to max pooling versus maxout. We also did an empirical evaluation where we used an avgpooling layer after each CNN layer, we made sure that the avgpooling does not change the dimension of the features wherever it applied so it does not change the network dimensions. In Table #7  We notice that using avgpooling leads to higher error rates versus the base model and the model where IEN is applied, in some cases it almost does 100% worse than IEN and Base models. We added this empirical comparison to assure the benefits of IEN vs avg pooling. For convenience we share a text version of the table: 

| Dataset   |                   | IEN                                                           | Base             | Avg Pool          |
| :-------- | :---------------- | :------------------------------------------------------------ | :--------------- | :---------------- |
| CIFAR-10  | ResNet56      | 6.93 +- 0.26                                         | 8.38 +- 1.2     | 12.61 +- 0.90    |
|           | ResNet110         | 6.27 +- 0.32                                             | 6.38 +- 0.48    | 14.58 +- 0.05    |
| CIFAR-100 | ResNet56      | 29.34 +- 0.49                                            | 29.97 +- 0.71   | 39.93 +- 1.80    |
|           | ResNet110     | 28.17 +- 0.11                                            | 27.83 +- 0.64   | 41.75 +- 2.21    |


-Regards performance against outer ensembles: 

We believe that outer ensemble outperforms a single IEN model because the outer ensemble models are all trained in a decorrelated way. We recall that the error of the network consists of bias, variance and covariance. In case of using IEN which is an inner ensemble we are able to tackle the variance but still bound the overall covariance/correlation. Yet, outer ensembles don't have this bound. We also show in Table 2 that an outer ensemble of IEN outperforms an outer ensemble of base models in most of the cases, this shows that IEN is a method to be used with outer ensemble and not a replacement, yet it's economical in terms of training time and final model inference. 


- We removed the sgK part to make it more clear in the proof A.2
- We will move a compact version of A.2 in the final version of the paper into section 4.1 after the end of the discussion period. Thanks for pointing this out. 
- For the color blind point, we replaced it with an italic font to make it friendly for the color blind people. 
- For the dimension problem, thanks for noting this dimension issue, now it's fixed. 
- We generalized equations (24)-(29) and proved it for a non-zero mean case. The proof can be found in the updated paper Appendix A.2. 
- Base_w is the weights of m separately trained base models that are averaged together, it's there to show that the weight reduction mechanism of IEN only applies to it unlike base and maxout. 
- We added the results of IEN+FC for completeness of the results table. 
- For the dropout results: We applied your suggestion of using  [1] for the best location of placing dropout within residual connection and setting the probability to p=0.3, we added the results of this experiment in the table in appendix G. We still notice that IEN outperforms dropout.
 [1]Zagoruyko, Sergey, and Nikos Komodakis. "Wide residual networks." arXiv preprint arXiv:1605.07146 (2016).

- “(Table 1) Was maxout also used as a drop-in replacement without hyperparams tuning” Yes, both IEN and Maxout was used as a drop in replacement without any hyper-parameters tuning in order to have a fair comparison. In the final version of the paper include a tuned version that shows the maximum potential of our method, yet tuning takes a whole that's not suitable for the time frame of this discussion period. Thank you again!


Dear Reviewer#1, 

Thank you for this insight about a possible direction to have the IEN effect with less parameters and thank you for the openness to discuss this point. We will discuss this proposal on two different terms, the first term is an empirical one while the other is a theoretical one. The proposal you suggested differs from our IEN proposal by the number of parameters $m$. We ran experiments using your proposal [learning rate = 1/m, w = sum_j w_j] and two extra proposals from our side to have a complete view of this mechanism the first is[ learning rate = original learning rate, w = 1/m sum_j w_j] and the second proposal is [learning rate = original learning rate, w= sum_j w_j]. In page [9] of the updated paper Table#4 we have the results of the empirical evaluation on bothResNet56,110 and DenseNet40-12,100-12. We report mean error rate and std. The lower the error rate the better. We notice that our proposal indeed is much better by a huge factor rather than the other proposals in most of the cases. Though the proposals look equally to ours in terms of math yet they do differ in one aspect which is the number of parameters. As our method utilizes the extra parameters during the training time to reach a better optimization point aka they have a higher dimension to move freely within it, all the other proposals do not have this extra dimension to move in. This aligns with the finding of [1] where they showed theoretically and empirically an overparameterized deep model indeed outperforms it's counter with much less parameters. We thank you a lot for this suggestion as it opens a horizon for us in our future work where we will attempt to have the same results but with the same training parameters. We will add this discussion into our final version.

 For convenience a text version of Table#4 is shown below: 

| Dataset  | Variation     | ResNet56          | ResNet110          | DenseNet40-12     | DenseNet100-12                   |
| :------- | :------------------- | :---------------- | :----------------- | :---------------- | :------------------------------- |
| CIFAR10  | IEN (ours)           | 6.49 +- 0.04     | 5.94 +- 0.32      | 6.89 +- 0.20     |  5.04 +- 0.07                   |
|              | Reviewer#1 proposal     | 89.29 +- 0.03 | 72.66 +- 1.46    | 8.52 +- 0.21     | 6.42 +- 0.08                        |
|              | First variation        | 92.3 +- 0.36     | 90.9 +- 0.06      | 7.33 +- 0.16     | 5.29 +- 0.21                    |
|              | Second variation     | 93.08 +- 0.35    | 84.48 +- 0.74    | 7.10 +- 0.05     | 5.07 +- 0.15                    |
| CIFAR100 | IEN (ours)           | 28.20 +- 0.19    | 27.34 +- 0.28     | 29.37 +- 0.19    | 23.67 +- 0.37                        |
|               | Reviewer#1 proposal     | 64.23 +- 0.22    | 40.27 +- 0.38       | 32.22 +- 0.44    | 28.55 +- 0.09                        |
|               | First variation        | 69.86 +- 0.31    | 66.7 +- 0.25     | 29.48 +- 0.67    | 23.84 +- 0.25                        |
|               | Second variation     | 70.81 +- 0.15    | 56.44 +- 0.11     | 29.30 +- 0.53    | 24.12 +- 0.04                   


[1]Brutzkus, Alon, and Amir Globerson. "Why do larger models generalize better? A theoretical perspective via the XOR problem." International Conference on Machine Learning. PMLR, 2019.


Summary:

In this paper, the authors proposed Inner Ensembe Networks (IEN), which trains an ensemble of layers and therefore reduces model predictive variance. Experiments on image classification demonstrate that the proposed method is effective in reducing model variance and it also leads to an improved accuracy, compared to other variance reduction baselines such as maxout and dropout.

Pros:

The theoretical analysis (under some assumptions) in Section 4.3 and 4.4 illustrates the connection between IEN and dropout & maxout. Therefore, the improvement of the proposed method (IEN) on variance reduction over dropout and maxout is theoretically justified. The authors also show the boundary case where IEN achieves the same variance reduction as dropout and maxout. 

The empirical evaluation matches the theoretical analysis. Figure 2 demonstrates one of the core contributions of this work: the proposed method IEN reduces the model variance even further than dropout and maxout.

Figure 3 demonstrates how ensemble size affects the ranking of all methods and all architectures considered in this work. It shows that the improvement of IEN is consistent among all network selections and ensemble sizes. The authors also showcase the benefit of IEN can be carried to deep ensembles (outer-ensemble defined in this paper) and NAS cells.

Cons:

My concern is the motivation of why variance reduction is needed in deep neural networks is not highlighted. This paper implicitly assumes that lower model variance leads to better performance (in terms of accuracy metrics). However, the connection between model predictive variance and averaged model accuracy remains unknown (especially in deep ensembles). Recently many papers aim to improve ensemble diversity. It improves the ensemble performance but it also leads to larger model variance. On the other hand, regularization can be connected to better generalization performance, but this paper spends more paragraphs on the variance reduction effect. 

The authors mentioned that reducing the variance in the prediction helps the ensemble of networks generalize better to unseen data. This implies that the proposed method IEN is supposed to outperform other baselines on out-of-distribution dataset (CIFAR-10C and CIFAR-100C). It would be more convincing if the authors can add empirical evaluations on out-of-distribution datasets.

The proposed method shares some similarities to Stochastic Weight Averaging (SWA) [1]. SWA also leverages the mean of multiple copies of the network weights, which also reduces the model variance. One main difference is SWA is a global mean of network weights while IEN takes the local (layer) averaged weights. The paper would be more inspiring if the authors can discuss the connection to SWA and make a comparison in the empirical evaluations.

Overall, the paper proposed an interesting method to efficiently reduce model variance, which leads to an improved performance. But the cons outweight the pros in its current version.

[1]: Izmailov, Pavel et al. “Averaging Weights Leads to Wider Optima and Better Generalization.” UAI (2018).


The paper proposes replacing each linear layer with a linear ensemble (average) of m (m>1) linear layers.  Authors argue that this helps to reduce network internal variance and achieve better accuracy.

I think it is possible to see that the proposed model is equivalent to the baseline model with a different hyper parameter setting (specifically the random initialization and learning rate), and therefore the observed variance reduction and improved performance are a result of better hyper-parameter choice in the proposed model.  The equivalence between the baseline model (with single linear transformation in every layer) and proposed model (m linear transforms in every layer whose output is averaged) can be seen by observing that:

Baseline:  y = w.x
Proposed: y = (1/m) sum_j w_j x = (sum_j (1/m)w_j) x

We see the following relation between initial weights of baseline and proposed model: w = \sum_j (1/m)w_j.   Furthermore, the gradients of objective w.r.t. w and w_j also follow a linear relationship: grad_{w_j} = (1/m)*grad_w, and due to this gradient scaling, the effective learning rate of the proposed model is smaller by a factor (1/m) as compared to the learning rate of the baseline model. Gradient of objective w.r.t. layer input x is same in the two models as long as the gradients w.r.t. output y are same and weights follow the relationship w = \sum_j (1/m)w_j.

Thus I would think that if the baseline model follows hyper-parameter settings that are equivalent to the proposed model — i.e. weight initialization such that w = \sum_j w_j (leading to reduced variance of initial weights), and a learning rate that is smaller by factor (1/m) — then it should see variances and performance that is akin to that of the proposed model.

I'd like to hear author's response and discuss if my reasoning is flawed.

Meta-review: The paper proposes inner ensemble method where output of inner layers are replaced by an ensemble average of them during inference time to reduce inference time and reduce variance. The authors include experiment results showing performance improvement of their method and use the theoretical analysis of the dropout and maxout to justify the goodness of the proposed method. 

pros.
-The authors consider a useful and interesting problem.

-The proposed method is simple and easy to plugin. 

-The results show performance improvements in a number of cases.


cons. 
-However there are various concerns raised by the reviewers that I find are not well-justified.

-The use of inner ensembles are not well justified.

-The baseline in the experiments miss the usual ensemble methods instead the authors use single model which does not have the same number of parameters.

-novelty and significance- The authors do not clearly distinguish their method from earlier results and how it differs and improve over existing work. Given the current state of the paper, the novelty is not significant.

-clarity of theoretical analysis is lacking.

Overall I suggest the authors improve the manuscript considering concerns and suggestions listed above and through reviewers and submit to an upcoming venue.





Decision: withdrawn-rejected-submissions