Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Authors assessed how these subnetworks can be as diverse as independently trained networks. The contribution of this paper is in proposing an approach to improve uncertainty estimation and robustness with minor changes (1 percent) to the number of parameters and compute cost. 

STRENGTHS: 
Training multiple independent subnetworks within a network, with minimal increase in number of parameters. 
The use of MIMO makes this approach simple, while it can be evaluated in a single forward pass.  

CONCERNS:
The authors claim that the benefits of using multiple predictions can be achieved ‘for free’, while their proposed model increases the number of parameters (even though by 1 percent)
The paper has examined the accuracy and disagreement of the subnetworks, but a detailed evaluation on number of parameters is missing (i.e. where the 1% increase in parameters comes from).
An experiment on more diverse datasets would be also helpful, such as OpenImages. 

review_title: This paper presents an approach to use a multi-input multi-output configuration for training multiple subnetworks with independent tasks. The authors claim that by ensembling the predictions (output of subnetworks) they can improve model robustness without additional computational cost. 

final_score: rating: 7: Good paper, accept
review_content: SUMMARY:
This paper describes a multi-input multi-output (MIMO) strategy for training several subnetworks inside a same and single neural network for robust prediction. The approach consists in jointly training the heads to make predictions for their corresponding inputs. The strategy is simple and demonstrates strong results. The experimental study reveals that subnetworks functionally behave as independent networks, hence resulting in a strong and robust ensemble. 

STRENGTHS:
- The experimental study is very thorough. I really appreciated the investigation in Section 3, which indeed convincingly shows that the subnetworks behave as independent. 
- The method is compared on standard benchmarks, across a wide range of metrics. Experimental results show better performance than single forward pass methods.  Performance is reasonable with respect to a simple solution consisting in training an actual ensemble of 4 networks.
- The paper is well written and easy to follow.

WEAKNESSES:
- I believe the approach to be original, but its similarities/differences with other multi-input multi-output (such as BatchEnsemble) could have been discussed much further to better appreciate the originality of MIMO.
- Although independence between subnetworks is shown empirically, I cannot help but wonder how/why it emerges from the architecture! This is an exciting phenomenon that ought to be better understood. Nevertheless, I believe the actual independence of subnetworks should be nuanced at places (e.g., in the title or in the abstract), as this highly depends on the capacity of architecture and on the problem to solve -- as shown in the experiments themselves. 
- Some (hypothetical) theoretical explanations regarding the emergence of independence would have made the paper stronger, although I realize this would be a whole new paper by itself.  
review_title: A simple and effective way to train a single network as an ensemble of networks

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to train a single network with M input examples and M corresponding predictions, and the M input examples are mixed to produce the M corresponding predictions. Although only a single network is learned, it implicitly consists of multiple sub-networks due to the nature of multiple inputs and multiple outputs in training. In testing, the single testing example can be replicated M times as inputs, so that M outputs are produced by the trained network. The multiple outputs enable efficient ensembing for robust prediction. 

I find the proposed idea very clever. The empirical results on toy data and real data are interesting and compelling. It demonstrates that a single network has the capacity to contain multiple sub-networks, which is an interesting discovery in itself. 

I do have two main reservations. The first one is the lacking of some basic, not necessarily rigorous, theoretical formulation and analysis. The second one is about its practical potential. Apparently M has to be rather small, due to the limited capacity of a single network. Its advantage over training multiple networks may not be dramatic. 

I am also a bit concerned that the network is trained on M independent examples (although the proposed method does allow for occasional identical examples), but is tested on M identical copies of the same testing example. 

I am also unclear about the nature of mixing M independent training examples, and the effect of doing that. That is why I feel some theoretical understanding is needed. 

What if we permute the M training examples and check the difference of the corresponding outputs? 

review_title: A clever idea and interesting empirical results

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary: The ensemble method MIMO is proposed in this paper to reduce the inference delay and keep the prediction diversity. Only one model with sufficient capacity exists in this method while multiple implicit subnets are embedded in the parent model. Each subnet has individual I/O, so only one forward pass of the parent model is needed to process all subnets and make the ensemble.

Quality: Medium to high. Pros: 1) The dissections of the subnets are impressive. They design the experiments to survey the loss plane, the parameter/activation projection of different subnets. 2) With the proposed training paradigm and proper test setting, the accuracy improvement can be seen and also uncertainty estimation. 3) They report the SOTA results of accuracy, uncertainty, and robustness on various datasets and their OOD variants when considering the inference latency. Cons: 1) They only report the inference time of one sample, but the total computation costs (e.g. MACs or FLOPS) are omitted. 2) The multiple branch networks are not only used in ensemble and broader usages exist. I know at least three works which involves multi-branch architecture (output-wise or layer-wise) in robustness or knowledge distillation and some of them are also using the ensemble of multiple predictions. The lack of citations in the related work seriously declines the quality of this work.

Clarity: High. Pros: 1) The method framework has a brief and clear explanation (Figure 1). 2) The training methods are conveyed in every detail, including some techniques like “input repetition”. 3) Some critical plots reporting accuracy using error bars or box plots to display the performance variance. Cons: Some format mistakes of symbols are existing. For example, the authors mix the usage of the normal and italic font of “x” when referring to samples in different expressions; the chaotic usage can even happen in the same equation for the first one of Section 3.3.

Originality: Low to medium. Pros: This method successfully uses the multi-branch architecture to reduce the inference delay in ensemble, which is a rather novel idea. Cons: As mentioned above, many similar multi-branch architectures have been used in different but related topics. Consider all of these works, the originality of this work has to be downgraded.

Significance: Low to medium. Pros: The shining points of this work are making ensemble for “free”. They do decrease the inference delay significantly. Cons: 1) The authors attempt to distract our attention on the computational costs of this MIMO architecture and try to make an illusion that it’s convenient in computing. The latency is decreased at the cost of batch size. Other ensemble methods have a longer delay, but they can process a batch of images. This method fills the batch with the same image which implicitly decreases the batch size. Furthermore, no measurement is used on computation costs or other related aspects. I think it’s deceptive and tricky. 2) As shown above, the originality of this work is not as solid as their experiments. 

[ Detailed comments]
1. What are the structural considerations for the related work of Section 5 not to be explained in Section 2?
2. In Figure 6, the ‘#’ marked on the abscissa of (b) is redundant


review_title: Limited novelty

Meta-review: This paper proposes a simple but effective method to obtain ensembles of classifiers (almost) for free. 
Essentially you train one network on multiple inputs to predict multiple outputs. The authors show that this leads to surprisingly diverse networks - without a significant increase in parameters - which can be used for ensembling during test time. 
Because of its simplicity, I can imagine that this approach could become a standard trick in the "deep learning tool chest". 

-AC
Decision: Accept (Poster)