Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a simple implicit model ensemble method based on feature-wise linear modulation (FiLM), where they learn M sets of affine parameters for batch norm layers. Experiments show higher diversity than DE, high accuracy and ECE among implicit ensemble methods, and good OOD detection capabilities.
strengths_and_weaknesses: The proposed method is conceptually simple in addition to being parallelizable.

The M sets of affine parameters are learned concurrently, and the M ensemble members would collapse to the same function if these parameters were the same. Do you have an intuition for why this does not happen? In other words, why is the different random initialization sufficient for diversity at the end of training, given that the functions share all other parameters?

I don’t understand why Figure 1 happens; why should the pairwise diversity of FiLM-Ensemble be bigger than that of DeepEns if nothing in the objective is directly encouraging diversification? If anything, it seems like it should be lower because of the shared parameters. Also, why does diversity increase with the number of members if these metrics are averages of pairwise metrics? As you add more models, it intuitively seems that some of them should “overlap” and thus have lower diversity.

Table 2 is unconvincing to me. The other networks (MIMO, BE, etc.) are doing worse than a single network, making it seem like the hyperparameters were not adequately tuned for these methods. For example, MIMO and BE perform better than a single deterministic network in the MIMO paper.

Line 107: “…depend bot on” typo
Line 116: “thid” typo
questions: In table 1, what causes the difference in inference time between MIMO and FiLM-Ensemble for ResNet-18? Aren’t both methods similar to feeding forward a batch of 16 datapoints (in terms of computation cost)?

Why do other methods perform so poorly in Table 2? If these methods were adequately tuned, could you report results on CIFAR10/100 with ResNet-28-10 and compare them to the results in the MIMO paper?
limitations: To my knowledge, this work has no potential negative societal impact other than what was already present in the existing literature.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to use feature wise linear modulation [1] to construct implicit deep ensembles. In FilM [1], the authors propose to replace batch norm layer in a deep architecture with a conditional batch norm layer, with two parameters (shifting and scaling coefficients), which are used to transform features given a conditional input. In this work, the authors propose to use film layer, but to use multiple sets of affine parameters to instantiate different ensemble members. 
The authors show for a couple of datasets (cifar10/100, retina glaucoma detection, not imagenet), some networks have performance better than other implicit ensemble approaches with respect to deep ensemble performance both in terms of accuracy and expected calibration error. 

Refsparame
[1] Perez et al. 2018. FilM
strengths_and_weaknesses: Strengths:
- The paper is well written, where the authors explicitly state the contributions, comparison metrics and results.
- The authors compare their proposed implicit ensemble model to a wide variety of existing implicit ensemble models.
Weaknesses:
- The paper would greatly benefit from a discussion on how the proposed film ensemble is different from MIMO or batch ensemble (BE) approaches. These could be added with minor equations in the appendix. Currently, the authors only state in line 163 "[BE]... is closest to our work".
- From the paper, it is unclear to me how difficult it is to learn the film parameters for this model. In my experience, finding the best hyper parameters for batch ensemble models requires min hyper search, is this also true here?
-  The authors mentioned that we can expect to trade-off accuracy vs gain given $\rho$ which controls the variance of the weight initialization. From Fig 3, it is unclear if there is a pareto optimal solution (tradeoff between accuracy and ece) or fig 3 is simply an artifact for a particular dataset. Adding additional fig for other datasets would be helpful.
- The authors state some failure modes for other implicit ensemble approaches, i.e. "batch ensemble generally exhibits a negative correlation between ensemble size and test set accuracy", "mimo shows very poor performance.." These results to my knowledge are not aligned with published work. For this part I would suggest to for the authors to better discuss the limitations of their baseline implementations. Conversely, if the authors do believe that this a standard result/common knowledge, they can explicitly state so/cite similar findings. This could be in addition to the discussion of how difficult it is to optimize the hyperparameters for film/other methods to construct implicit ensembles.  
- The datasets considered are low dimensional/ low resolution. Have the authors tried to reproduce their results on larger datasets, i.e. imagenet?
questions: - How easy is it to learn the hyperparameters of film ensembles?
- Have the authors tried to reproduce their results on larger datasets, i.e. imagenet?
- How do the authors expect this approach to perform in tasks where fine tuning/using pretrained networks is necessary?  

limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes the use of a set of FiLM layers instead of distinct neural networks as a way to build an ensemble. This has the benefit of drastically reducing the number of weights used by the ensemble, since it only relies on cheap channel-wise scale and shift vectors. The method has been compared with a variety of baselines (e.g. Deep Ensembles, MIMO, MC-Dropout, etc), using a few backbones (ResNets and VGG), on image-classification datasets such as CIFAR10, CIFAR100, and Glaucoma Detection. The empirical results show performances that are comparable with leading methods in terms of classification accuracy and calibration error with the benefit of a lower parameter count.
strengths_and_weaknesses: Strengths
---------

- The method is novel and original, the use of FiLM layers to create an ensemble provides substantial memory savings.

- The method is compared against strong baselines (e.g. Deep Ensembles).

- The paper is well written, easy to follow, and clear.


Weaknesses
----------

- The empirical evaluation has been performed on relatively small networks (e.g. ResNet18 and ResNet34) and simple datasets (e.g. CIFAR10 and CIFAR100). It would be useful to verify if the same performance gains can be obtained in larger models and datasets. 

- Limited variety of networks, the method has been mainly tested on ResNets. The number of parameters in a FiLM layer is related to the number of channels in that layer; in recent architectures (e.g. MobileNet and EfficientNet) the proportion of number of channels vs. number of layers can vary drastically w.r.t. ResNets. This seems quite relevant, since the total number of FiLM parameters used in the model can affect the performance, and change the gap w.r.t. the baselines (in particular Deep Ensembles). It would be useful to verify if the method works well also on other architectures.

- The use of the Expected Calibration Error (ECE). A recent line of work exposed several problems of the ECE score [1,2,3,4]. ECE is a biased estimate of the true calibration, and it only estimates miscalibration in terms of the maximum assigned probability, disregarding the full predicted probability. Other metrics have been proposed to fix those issues, for instance the Overconfidence Error(OE), Static Calibration Error (SCE), the Adaptive Calibration Error (ACE), and the Thresholded Adaptive Calibration Error (TACE), see [3] for reference. 

- The use of inference time in Table 1 can be misleading, as this can vary based on the machine overload and low-level implementation details. It would be better to use the total number of Multiply-accumulate operations (MACs) instead.

References
----------

[1] Ashukha, A., Lyzhov, A., Molchanov, D., & Vetrov, D. (2020). Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06470.

[2] Kumar, A., Liang, P., & Ma, T. (2019). Verified uncertainty calibration. arXiv preprint arXiv:1909.10155.

[3] Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., & Tran, D. (2019, June). Measuring Calibration in Deep Learning. In CVPR Workshops (Vol. 2, No. 7).

[4] Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J., & Schön, T. (2019, April). Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics (pp. 3459-3467). PMLR.

questions: 1) I would like to see a discussion of the concerns that I have pointed out in the "weaknesses" section.

2) The parameters of the FiLM layers are sampled from the uniform distribution bounded by the values reported in Equation 4. I was wondering if a similar procedure has been used to produce the members of the Deep Ensemble. This is crucial, as differences in the initialization strategy could affect the performance of the baselines.

3) Assigning a different set of initial weights to the member of the ensembles is just one way to enforce diversity. Did the authors try other techniques?
limitations: The authors have adequately addressed the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a parameter efficient approach to ensembles based on FiLM layers. The idea is that each of the ensemble members share all of the parameters in the network with the exception of a set of the FiLM layer parameters that are specific to each member. The FiLM parameters in each ensemble member are initialized with different random values at the start of training and evolve independently over the training process. At test time, the logits of each of the ensemble members are averaged to make a prediction. The method is parameter efficient since the FiLM parameters make up only a tiny fraction of the overall network parameter count. Despite only a small number of parameters differing in each ensemble member, the authors show that the diversity of the ensemble members is high, and overall accuracy and calibration is competitive with methods that are more expensive in terms of memory and computation time.
strengths_and_weaknesses: **Strengths**
1. The paper is clear and well written.

2. The concept of using FiLM layers as the basis for an ensemble method is fundamentally good. The method is simple and effective.

3. Results overall are excellent and compare favorably to leading competitive methods.

**Weaknesses**
1. The paper evaluates on relatively small and outdated networks (VGG, ResNet18/34). The paper would be improved if larger and more up to date networks (i.e. EfficientNets, Vision Transformers, etc.) were used in the evaluation.

2. The datasets used to evaluate the method are relatively small. The paper would be improved if larger and potentially more challenging datasets/benchmarks were used (e.g. ImageNet, VTAB)

3. There are no error bars in the tables, despite the fact that each experiment was repeated 3 times with different random seeds.

4. While the out of distribution experiments compare to other ensemble methods, the paper would be improved if it compared to state-of-the-art (or even standard) OOD methods.

5. The only calibration metric used was ECE. The paper would be improved if additional metrics were calculated (Brier sore, etc.).

There are some typos:
- Line 56: na ive
- Line 107: bot
- Line 116: thid
- Line 119: and **will** release the source code
questions: The method is quite simple and the paper was clear. As a result, there are no questions.
limitations: Despite the response in the checklist, I could not find where the authors described limitations of the proposed method. Some identified limitations of the work are described above in the **Weaknesses** section (i.e. the method was tested on relatively small networks and datasets). 

The authors did not discuss any negative societal aspects of the work, which is unfortunate as image classification technology has a dark side (i.e. surveillance, military, etc.).
review_title: null

Meta-review: metareview: This paper constructs implicit ensembles by adding multiple affine transformations to batch normalization layers. The proposed approach is elegant and complements existing implicit ensemble techniques. There are some concerns that the experimental evaluation does not include larger models or datasets - which is important because other implicit ensemble models (e.g. MIMO) exhibit sensitivity to model/dataset size. It would also be useful to discuss how this method could be used in conjunction with pre-trained models. Nevertheless, the idea is well-presented, simple, and effective, and therefore it will be of interest to the NeurIPS community.
confidence: Less certain
Decision: Accept