FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation
Authors: Mehmet Ozgur Turkoglu
Alexander Becker
Hüseyin Anil Gündüz
Mina Rezaei
Bernd Bischl
Rodrigo Caye Daudt
Stefano D'Aronco
Jan Dirk Wegner
Konrad Schindler
Venue: NeurIPS 2022 Accept (2022)
Keywords: efficient ensembles
subnetwork
feature-wise linear modulation
uncertainty
robustness

Abstract: The ability to estimate epistemic uncertainty is often crucial when deploying machine learning in the real world, but modern methods often produce overconfident, uncalibrated uncertainty predictions. A common approach to quantify epistemic uncertainty, usable across a wide class of prediction models, is to train a model ensemble. In a naive implementation, the ensemble approach has high computational cost and high memory demand. This challenges in particular modern deep learning, where even a single deep network is already demanding in terms of compute and memory, and has given rise to a number of attempts to emulate the model ensemble without actually instantiating separate ensemble members. We introduce FiLM-Ensemble, a deep, implicit ensemble method based on the concept of Feature-wise Linear Modulation (FiLM). That technique was originally developed for multi-task learning, with the aim of decoupling different tasks. We show that the idea can be extended to uncertainty quantification: by modulating the network activations of a single deep network with FiLM, one obtains a model ensemble with high diversity, and consequently well-calibrated estimates of epistemic uncertainty, with low computational overhead in comparison. Empirically, FiLM-Ensemble outperforms other implicit ensemble methods, and it comes very close to the upper bound of an explicit ensemble of networks (sometimes even beating it), at a fraction of the memory cost.

FiLM-Ensemble: Probabilistic Deep Learning via Feature-wise Linear Modulation
Mehmet Ozgur Turkoglu ETH Zurich
Alexander Becker ETH Zurich
Hüseyin Anil Gündüzz LMU Munich
Mina Rezaei LMU Munich
Bernd Bischl
Rodrigo Caye Daudt ETH Zurich
Stefano D'Aronco ETH Zurich
Jan Dirk Wegner ETH Zurich University of Zurich
LMU Munich
Konrad Schindler
ETH Zurich
Abstract
The ability to estimate epistemic uncertainty is often erucial when deploying chine learning in the real world, but modern methods often produce overconfident, uncalibrated uncertainty predictions. A common approach quantify epistemic uncertainty, usable across a wide class of prediction models, is to train a model ensemble. In naïve implementation, the ensemble approach has high compu- tational cost and high memory demand. This challenges in particular modern deep learning, where of compute and memory, and has given rise to a number of attempts to emulate the model ensemble without actually instantiating separate ensemble members. We introduce FiLe-enneebble a deep, implicit ensemble method based on the concept of Feature-wise Linear Modulation (FiLM). That technique was originally developed for multi-task learning, with the aim of decoupling different tasks. We show that the idea can be extended to uncertainty quantification: by modulating the network activations of a single deep network with FiLM, one obtains a model ensemble with high diversity, and consequently well-calibrated estimates of epis- temic uncertainty, with low computational overhead in comparison. Empirically, FiL.M --lsemll outperforms other implicit ensemble methods, and comes very close to the upper bound of explicit ensemble of networks (sometimes even beating it), at a fraction of the memory cost.
single deep network is already demanding in terms
Introduction
A key component for reliable and trustworthy machine learning algorithms that output not only accurate predictions of the target variables, but also well-calibrated estimates of their uncertainty (Gal and Ghahramani 2016). The overall uncertainty of predictor is usually decomposed into two parts Der Kiureghian and Ditlevsen 2009]. Aleatoric uncertainty is inherent in the data, for instance due class overlap sensor noise. On the contrary, epistemic uncertainty characterises the uncertainty of the model weights, due to a lack of knowledge about parts of the input space that are insufficiently represented in the training set. Uncertainty caused by distributional shifts between the training and test data is sometimes conceived as a third source of uncertainty [Malinin and Gales 2018), but in practice often modelled as part of epistemic uncertainty.
Measuring epistemic uncertainty for complex models such as deep neural networks is not trivial: by definition one cannot derive it from the training data, since it concerns the behavior of the model
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

in regions of the input space that are not represented in the training set. Different methods have
been explored [Welling and Teh 2011 Graves 2011 Blundell et 2015 akshminarayanan et al.) 2017a, Huang and Belongie| 2017 Gal and Ghahramani 2016 however the de facto standard remain deep ensemble models [Lakssminarayanan et al. [017a]: J its basic form, such a deep ensemble simply a collection of independently trained networks that can be regarded as Monte Carlo samples from the model space. To obtain ensemble members with reasonably low correlation, one can exploit the stochastic nature of the optimisation procedure, with different (random) weight initialisation and different (random) batches during stochastic gradient descent. The expectation is that, once trained, the ensemble members will agree for inputs near the training samples, since the loss function favours similar outputs at those locations. Whereas they may disagree in unseen regions of the data space. Thus, the spread of their predictions is a measure of epistemic uncertainty. The ensemble idea is conceptually very simple, but nevertheless yields uncertainties that are well calibrated, i.e., they are in line with the actual prediction errors (in expectation).
The drawback of deep ensembles their large computational cost. Both computation and memory consumption grow directly proportionally with the number of ensemble members, during training as well as during inference. This makes them impractical in hardware-constraaneed settings, and leads to widening gap models continue to grow in size, while at the same time applications like mobile robotics, augmented reality and smart sensor networks increase the need for mobile and embedded computing.
To improve efficiency, several researchers have explored ways to mimic deep ensembles without explicitly duplicating the underlying network. Possible strategies include the reuse and recombination of network modules [e.g.. /Wen et al.] 2020 Havaas et al., 202T]. injection of noise at inference time [e.g.. Gal and Ghahramani] 2016]. as well as hybrid variants [e.g.. DDuraovv et al.] [021] Empirically, these models do speed up training and/or inference, but still exhibit a significant performance gap compared to the naïve, explicit ensemble, both w.r.t. prediction quality and w.r.t. the calibration of the predicted (epistemic) uncertainties.
[e.g..[Wen
The lottery ticket hypothesis Frankle and Carbin 2018 and other network pruning studies, e.g., by Han et al. [2015). Lee et al.] [2019). Mallya and Lazebnik| [2018), underline that neural networks are heavily over- peer-parmmeteized. Their parameters are used inefficiently, and they can be pruned significantly without large performance drops. In lifelong learning and multi-task learning it is essential to use the network efficiently, in order to limit computational overhead when introducing new tasks. There are recent works that achieve good performance in these tasks by introducing modulation (respectively, adaptation) strategies. In particular, [i et al. (2018), Takeda et al. [202]] propose an efficient lifelong learning domain adaptation method for multi-task learning, utilizing single feature-wise linear modulation (FiLM).
Inspired by that line of work, we propose a new, efficient ensemble method, FiLM-Enseminn Our method adapts feature-wise linear modulation as an alternative way to construct an ensemble for (epistemic) uncertainty estimation. compared to the naïve ensemble approach, while performing almost on par with it, sometimes even better. In nutshell, individual member is defined via its own set of linear modulation parameters for the activations, whereas all other network parameters are shared among ensemble members- and therefore only need to be stored and trained once. Thanks to this design, our method requires only very small number of additional parameters on top of the base network, e.g., converting a single ResNet- 18 model to an ensemble of 16 models increases the parameter count by 1.3%, compared to an increase by 1500% We further show that
greatly reduces the computational overhead
can be described as an implicit model ensemble in which each
when setting up a naive ensemble, see Table I more diverse ensemble members compared other efficient ensemble methods. For instance, on the Cifar- 10 benchmark it achieves diversity scores > 6.9% and up to 9.2% (depending on ensemble size), against 6.8% for naive ensemble, see Fig. Our contributions can be summarized follows.
results in
We propose
a novel parameter- and time-efficient deep ensemble method.
FiL.M-Eneemlle designed in such way that it can be readily combined with many popular deep learning models, by simply replacing batch normalisation (batchnorm) layers with conditional batchnorm layers.
We show that F-LM--eseeblee provides an excellent trade-off between accuracy and calibra- tion performance, improving over existing ensemble methods.

---

2 FiLM-Ensemble
Perez et al. [0118 achieved the linear feature modulation by varying the affine parameters of the batch normalization layers and 3. For each batchnorm layer in the network, the parameters are predicted according to some conditioning input z (for instance, different prediction tasks):
(z)
B, = (z)
The size of Yn and A, is Dr. where D, is the feature dimension layer n. These parameters then used to linearly modulate the activations at the n-th layer F,::
FILM n Yh. Bm) 7n (x) F, + Bn(
(2)
with o being the Hadamard (element-wise) product taken w.r.t. the feature dimension.
In our scenario we aim to use the affine parameters to instantiate different ensemble members, i.e., the conditioning variable is simply an index m E {I...., M) that identifies the ensemble member. The functions 9n(2) and (2) degenerate look-up tables, we can dispose of them and simply learn M different sets of affine parameters for every batchnorm layer F,,
(ILM(Fn.nr)) (K)) Yn Fn P
(3)
To achieve ensemble members with diverse parameters, resort Xavier initialization, i.e., all n and 3 are sampled from uniform distribution bounded between:
V
VD,
where Dr is the number of features (channels) in the n-th layer, and p is an initialization gain factor. In our setting P is tunable hyperparameter that allows one to control the trade-off between predictive accuracy and calibration of the model, see Section 3.5 In general, increasing P leads to more diverse ensemble members and thus favours calibration. Note that as p O, all ensemble members start from similar initial values for 7 and 3T and the FiLM-Ensemble gradually collapses to a single model.
During training, we feed each input sample x every ensemble member and obtain predictions ym = Te (x), which depend both on the member-specific parameters (3mh > and on the shared parameters 0. All those parameters are optimized together to minimise the chosen loss function. Here we focus classification and use standard cross-entropy (CE) loss.
At inference time the final prediction is obtained by averaging the M predictions of the ensemble:
Vm
M
2.1 Implementation Details
We implemented I-dimensional and 2-dimensional FiL-M-Enseblle layers, to be used in combination with 1 1-dimensional and 2-dimensional convolution operations, respectively. The forward passes through different ensemble members can easily be parallelized by replicating both the input tensor and the FiLM parameters Y, 33 along the batch dimension and applying the affine transformation Eq. (the same holds for the Bnsebbee method), In this way ensemble members run simultaneously on a single device, thus optimally utilizing modern tensor computing hardware without having to load several instances of the classification network into memory. We have implemented FiLM-E Ensemble in Py PoTch and release the source code
For all experiments, we optimize the model parameters with standard stochastic gradient descent, with momentum μ = 0.9 and weight decay X x= 0.0005 for regularization. We train for 200 epochs with batch size 128. The learning rate initially set 0.1 and decays according to a cosine annealing schedule [Loshchilov and Hutter [017]. The initialisation gain is set to p 2 for all experiments, except for genome sequences (see Table 3). where p E {4,8, 16, 32). Please refer to the supplementary material for additional, dataset-specífic details.
htttpe://githubb.com/rrsssss Enssebl

---

3 Experiments
In the following, we first empirically examine the ability of FiLM-Ensemble to learn independent ensemble members, and compare it to deep ensembles. Then analyze the predictive accuracy and uncertainty calibration of our method, as well as its computational cost and memory consumption. We compare its performance against several baseline methods and state-of-thes the-art alternatives variety of different datasets, and using different backbone architectures. All experimental results are averaged over three runs with different random seeds.
We evaluate our proposed method on diverse set of classification tasks, including popular image classification benchmarks, masee medical diagnosis, and genome sequence analysis. CIFAR- 10 and CIFAR-100 (Krizhessky) 2009] are widely used testbeds for image classification, and deep learning in general. They consist of clean images of objects from 10, respectively 100, different semantic classes. Each of the two datasets contains 60,000 images, of which 10,000 are reserved for testing. The semantic classes uniformly distributed in the datasets and stratified across the train/test spit. Retina Glaucoma Detection [Diaz- Pinto et al. 2019 is real-world clinical dataset that includes microscopic retina images from 956 patients with the neuropathic disease Glaucoma, and from 1401 subjects with normal (healthy) retinas. Fach input sample is a single RGB image; we resize all images to 128 128. The image augmentation applies by combination of crop, horizontal flip, and color jitter. REFUGE 2020 (Orlando et al.) [2200] was a challenge at the MICCAI-2020 conference, aimed retinal Glaucoma diagnosis. The dataset consists of 800 microscopic retina images with size 1411 x 1411 pixels, collected from different clinics. We use this dataset in conjunction with the models trained on the previous dataset [Diaz-Pinto et al. 2019). to evaluate the ability detect samples with the help of the predicted uncertainties (Table 4). 6mA Identification (Li al. 2021]. is a 1 1-dimensional sequential genome dataset. It consists of DNA sequences of rice plants, along with binary labels that indicate whether the sequence is a N6-methyladenine (6mA) site. 6mA is an important DNA modification associated with several biological processes, such as regulating gene transcription, DNA replication and DNA repair [Campbell and Kleckner 1990, Cheng et al. 2016 Pukkila et al. 1983]. Each sequence consists of 41 nucleotides. As there are different types of nucleotides, each one-hot encoded sequence is represented by 41 x 4 matrix. In total there are 269,500 training samples and 38,500 test samples.
We compare FiLM MiLMe-nnemblee against (i) a single model without any ensembling, as the elementary baseline, (ii) naive deep ensemble LLakshminarayaaan et al. 2017b], and (ii) MC-Dropout [Gal and Ghahramani] 2016). Furthermore, we compare with other state-of-the
methods, including: (iv) Masksemble Durasovv et al., [222]] which can be seen as an extension of MC-Dropout, (v) MIMO (" ("multi-input multi- ootpuutttttt [Havasi et al. [021] which defines a multi- head architecture where each head acts as one ensemble member, and (vi) Batch Ensembee (Wen et al.] 2020 which creates an efficient ensemble by expanding the layer weights using low rank matrices. Arguably, this layer-wise modification of an underlying, common representation is the approach that comes closest to our work.
3.1 Diversity Analysis
Diversity is the key to constructing powerful ensembles: nothing is to be gained from highly correlated ensemble members that return similar outputs for (almost) any input (Zhang and Ma 2012). In order to analyze the diversity (respectively, the degree of independence) among members, we compute two distance metrics between the members' predictive distributions. Let fi and fj denote two different ensemble members for a classification task. We first measure the disagreement score D, defined as the fraction of all test samples for which the two members return different answers, averaged over all possible pairs of members:
M
M
2
D M(M -
[f(xx) + fx(xx)]
(6)
1=1 k=1
with [-] being the Iverson bracket. Second, we also measure the Kullback-Leibler (KL) divergence between the two predictive distributions p(ffs) and p(f,), again averaged over all pairs:
2
p(f((x)) log (xx()
KL M(M
p(5z(xx))
1=

---

Table 1: Memory and inference complexity comparison (CIFAR-10/100 datasets): Number of additional trainable parameters have 16 ensemble members for different backbones. The inference time (mult-adds) shown corresponds to the mean GPU time (number of multiply-add operations)
required run a forward pass for batch of size with 16 ensemble members. The bottom section comprises methods whose forward and backward passes are implemented in parallel over ensemble
members, Measurements are done on an NVIDIA GeForce GTX 1080
Method
Parameters (L)
Inference time (ms) (1)
Mult-adds (B) (J)
Backbone
VGG-11
ResNet-18
VGG-11
ResNet-18
VGG-11
ResNet-18
MC-Dropout
0.0%
0.0%
16x 2.5
16x2.4
16x0.15
16x0.56
Deep Ensemble
1500%
1500%
16x 2.3
16x 1.8
16x0.15
16x0.56
Masksemble
0.0%
0.0%
20.8
6.6
2,45
8,89
MIMO
1.1%
0.9%
2.7
0.18
0.58
BatchEnsemble 1.4%
5.2%
2.8
5.2
2.44
8.89
FinMM-eneebll 0.9%
1.3%
2.8
5.7
2.45
8.89
For both metrics, higher numbers correspond to higher diversity. See Fig. We observe that the average diversity increases with the number of ensemble members. In both metrics, FiLM--nsemble achieves higher scores than the naīve, explicit deep ensemble. Meaning that the predictions of FiLM- Ensemble are less correlated than those an equivalent number of networks trained independently with different random seeds. Also, note that with the increasing number of members, improvements in diversity metrics for the naïve explicit deep ensemble are negligible.
Disagreement
members
KL-divergence members
ferpfiss
DompFin
4.0
+
ALIKEro
RLM-3ns
Disagreement (%6)
A.
-L-divergenco
0.21
7.8
0.00
70-
319
10
10
number of members
number members
Figure 1: Diversity analysis: CIFAR-IOVGG-LI experiment. With increasing number of members, FiLMM-nnemble achieves more diverse representations. See Section 3.1
3.2 Computational Cost
We go on to measure the efficiency of our proposed method, both in terms of parameter count and in terms of resources needed for forward pass. [ section) reports the numbers of additional numbers of parameters compared to single network (i.e., without uncertainty calibration). Due to their design, MC-Dropout and Masksemble do not require any additional parameters. Among the remaining methods, overhead terms of parameter count, while performing significantly better, as we will see below. In terms of inference complexity (Tab. [ center and right). -iLe--nsemble turns out to be very competitive. Only MIMO with a ResNet-18 backbone is significantly faster and lighter, however this comes at considerable price in terms of accuracy and calibration, see below.
has the lowest (VGG-11) or second-lowest (Resnet- 18)t--18)
3.3 CIFAR-10/ CIFAR-100
perform several experiments on widely used benchmarks computer vision, CIFAR-10 and CIFAR-100 100. As performance metrics, we plot the test set accuracy and the expected calibration error
5

---

Table 2: Classification performance on CIFAR-I00 with M = 4, using ResNet-18/34 as backbone. Best score for each metric in bold, second-best underlined.
Backbone
Resnet-18
Resnet-34
Method
Acc (t)
ECE (L)
Acc (1)
ECE (J)
Single Network
78.0 +0.4
0.046 +0.001
79.3 =0.2
0.089 #0.006
Deep Ensemble
81.6 +0.3
0.041 +:0002
82.0 ::0.1
0.044 +00002
MC-Dropout
75.5 +0066
0.064
+=0.003
72.2 +00.2
0.079
+00004
MIMO
4-2.6
0.083 +-0.117
56.2 +:488
0.132 +0.055
Masksemble
72.5 +0.5
0.075 +00004
70.1 +122
0.067 +0.004
BatchEnsemble
77.7 0.1
0.052 0
78.3 0.1
0.056
= 0.002
+0.2
0.038 +0.000
80.2 #0.1
0.045 #0.001
(ECH) against the ensemble size M in Fig. [ for all compared methods. In terms of accuracy (left subfigure). E-Eeemlle is outperformed only by the explicit deep ensemble, across all tested values of M E f2, 4, 8, 16). These two surpass all other methods by a clear margin. Surprisingly, we observe that Batchhnsemble generally exhibits a negative correlation between ensemble size and test set accuracy, with M 4 performing best among all settings. MIMO shows very poor performance this experimental setting in terms of accuracy (and also in ECE). We speculate that its shared backbone probably has a tendency to fragment into largely independent ensemble members of low channel depth, which lack the necessary capacity when using comparatively small networks like VGG-11 or Resnet-18. With regard to ECE (right subfigure), see all methods improving with growing ensemble size M. --Lemmlbeeble achieves better calibration then the widely used deep ensemble and MC-dropout methods, with significant margins at large ensemble sizes. Masksemble and BatchEnsemble achieve very good calibration for sizes M E [4).. but at the cost of significant drops in classification performance. An attractive feature of FiLM- Eeseeblle is that it offers a simple mechanism for trading off accuracy against calibration, by tuning the gain factor p. See Section 3.5
In Tab. 2 we quantitatively compare all tested methods on the CIFAR- 100 dataset, both with a standard Resnet- 18 backbone and with a larger Resnet-34. We observe that with both architectures, and in terms of both predictive accuracy and calibration, FiL.M-Ensemlle always either ranks first, or it ranks second behind the inefficient, explicit deep ensemble. Also, note that when the capacity of the network is increased (Resnet- >> Resnet-34), calibration performance improves for some the implicit methods, but the cost of reduced accuracy. Overall, the experiments with both variants of CIFAR and all three model architectures confirm that our method presents an excellent trade-off between predictive accuracy, uncertainty calibration and computational efficiency.
Accuracy vs members
ECE vs members
Single
0.37
DespE
Single
93-
NBNO
MC-Orop op
NC: Drop
Accuracy (%)
82.
30xch E%
Mess-Ers Ers
0.05
Batch End
Mask-Ens
ECE
0.53
90-
0.01
2
10
34
number members
number of members
Figure Accuracy and ECE for CIFAR-10 10, with varying ensemble sizes, using VGG-11 as backbone.

---

Table 3: Classification performance for retinal Glaucoma images. Best score for each metric in bold, second- best underlined.
Accuracy (1)
(1)
member (M)
16
16
Single
84.4
0.084
Deep Enserable
85.6
$5.7
86.0 0.2
86.8
0.811 0.002
0.078 0.002 0.091 0.001
0.066 : 0.003
ESSD Fuung et al
2020
86.9 = 0.1
0.047 + 0.002
SNGP
0.064
PNML Bunas St
85.6 0.1
0.061 0.001
MC- MC-Dropout
67.0 0.2
78.4
80.0 0.4
0.4
0,002 0.001
0.046
0,009
0.053 0.011
0.055 :: 0.0088
Masksemble
82.7 + 0.5
0.5
80.2 0.6
81.7
0.064
0.004
0.149
+.007
0.021 0.010
0.062 + 0.012
BatchEnsemble
81.5 0.1
0.1
86.8 0.2
97 1
0.033
0.003
0.063 0.009
0.071
0.002
0.066 : 0.002
RLLMEEnsemble
3.4 Retinal Glaucoma Detection
Glaucoma is currently the leading reason of irreversible blindness in the world. Detection of glaucomatous structural damages and changes is a challenging task in the field of ophthalmology. We evaluate our proposed diagnosing Glaucoma and quantifying the uncertainty associated with the prediction, see Table achieves the best classification result across all ensemble sizes, and
as well as the baselines described above, on the ask of
The proposed also the best overall result, with M 16. Whereas there is clear trend with respect to uncertainty calibration.
3.5 6mA Identification
With the 6MA identification task show that FiLM-Ensemble can also be readily combined with existing models for 1-dimensional sequential genome data. We use the 6mA-rice-La dataset and ID-CNN architecture whose
have already been optimized for this dataset [Li et al. 202T]. FiLM-Ensemble improves the accuracy and the calibration of that model, see Fig 3 Our method performs par with the explicit deep ensemble and better than single instance of the model tuned for the specific task. More importantly, one can reach a significantly better calibration
(lower ECE) by increasing the gain p, with only minimal accuracy drop 0.25 percent points.
Please refer to the Section ?? for more results.
Accuracy vs Gain
ECE vs Gain
0.030
Accuracy (%)
0.025
93.6
ECE
93.4
0.015
Sinlle
E
Snggle
FILM Ensemble 8
FILM EnllEnneeblee
53.0
0.005
20
10
Figure Performance of FiLMM-Eneebll with varying gain p. c.f. Section
3.5
3.6 Out-of-Distribution Detection
Domain shift often occurs in medical datasets; thus, detecting is an important task in clinical diagnosis. Model uncertainty can be used for OOD detection. For our experiment, we regard the retinal Glaucoma dataset [Diaz-Pinte et al. 2019 as the within- distribution samples and perform OOD detection with the REFUGE dataset [Orlando et al. 2020 As performance metric, we report AUROC (Area Under the Receiver Operating Characteristic) scores, Table 4 HLLM-Ensemble can detect OOD test samples with significantly higher accuracy than other
(OOD) samples

---

Table 4: OOD detection for retinal Glaucoma images. Best score in bold, second-bent underlined.
Method
AUROC (%) (1)
# member
2
8
16
Single
68.42
Deep Ensemble
76.89 + 0.1
77.91 + 0.2 78.22 + 0.1
78.06 + 0.1
FSSD
Huang et al.
2020
76.42 + 0.2
SNGP
Liu al. 2020
71.25 0.6
PNML
Bibas et al.
2021
76.68 + 0.3
MC-Dropout
68.03 0.3
69.79 + 0.2
77.94 0.6
72.22 + 0.2
MIMO
57.33 1.4
59.49
+ 1.2
61.74
60.52 3.1
Masksemble
71.22 0.5
70.83 0.8
72.04
70.95
1.4
Batcbensemble
74.38 0.1
72.61
+ 0.3
15.44
75.04
FiL-Meeneeble
77.022:..
77.92
+ 0.2
79.43 0.1
79.85 0.2
efficient ensemble methods, standard state-of-the-art OOD detection methods, and even the explicit deep ensemble. This suggests that for challenging test samples, which are not adequately represented in the training data, the uncertainties estimated with FiLM-ennemble are better calibrated.
4 Related Work
4.1 Epistemic Uncertainty Quantification
A large corpus of related work addresses the estimation of epistemic (model) uncertainty in neural networks. At the heart of such modeling is often the concept of marginalization instead of optimiza- tion, i.e., integrating out (possibly uncountably infinite) set of models weighted by their posterior probability, instead of committing to a point estimate of that distribution. A multitude of methods have been proposed to implement approximate Bayesian inference w.r.t the model weights, given the training data and an appropriate prior, process that is not analytically tractable in general Kendall and Gal 2017].
Ranganath et al. 2014 Bundell et al.] [013] seek to learn an approximate posterior distribution which a member of simpler family of variational distributions. This variational distribution can often be learned using backpropagation
For instance, methods based on variational inference [Graves 2011
[Bunneell et al., 2015) and can then be sampled from, or sometimes even used for exact inference. Markov chain Monte Carlo sampling (MCMC) approaches (Neal] 1996 Welling and Teh 2011 Chen et al. 2014] construct a Markov chain which has the exact posterior distribution as its stationary distribution, that can then be employed for sampling. However, in practise, these approaches often fail to sufficiently explore high-dimon high-dimensional, multi-modal loss landscapes they common in deep learning [Gustafsson et al. 2020
4.2 Ensembles and Sub-networks
In a method referred to as deep ensembles [Lakshminarayanan et al. 2017a set of M neural network models are randomly and independently initialized, and are subjected to stochastic mini-batch sampling during SGD training. The models generally converge to different minima in the parameter space, and can be considered samples from an approximate posterior [Wilson and Izmailov 2020 Deep ensembles often achieve the best calibration Gustalsson et al. 2020 Ashukha et al.] 2020], but suffer from high computational complexity as they require training, storing, and running inference on several full instances the network.
Gustafsson et al. 2020 Izmailov et al. 2021
and predictive accuracy OOaddia et al. 2019
Many alternative methods have recently been proposed in an attempt to reduce either the computa- tional cost or the storage cost of deep ensembles. Monte Carlo (MC) Dropout (Gal and Ghahramani 2016) runs multiple forward passes on the dropout layers in order to obtain multiple predictions and obtain an uncertainty estimate. Although the method requires fewer computations compared to deep ensembles, it also leads to less accurate uncertainty estimates. Snapshot ensembles (Huang

---

et al. 2 use a cyclic learning rate schedule in order to find multiple local minima, they then store multiple copies of the network create the ensemble. While this approach reduces the computation cost during training it does not alleviate the storage cost.
Batchensembles (Wen et al. 2020) employ multiple low rank matrices, which be stored efficiently, in order to modulate the parameters of a neural network and thus mimic an ensemble of network models. Masksembles [Durasov et al. 2021] aim improve the performance of MC Dropout by carefully selecting the dropout masks, used to drop certain features, such that they lead to better uncertainty quantification. Another approach that uses multiple sub-networks is proposed in Haaaai et al. (202T]. In this case additional, independent layers are added at the beginning and at the end of the network, in order to obtain multiple prediction with an single backbone model. Although such method seems to reduce the computational resources required at training and inference time, the gain limited to larger backbones, such as Wide-ResNet (e.g., ResNer28- 10), whereas for widely used standard architectures like VGG (e.g., 16),, or ResNet (e.g., -34), the approach is not very effective, as shown in our experiments.
Although many methods have been proposed that aim at reducing the computational cost of deep ensembles, none of them appears to clearly outperform most others. In summary, the question how to effectively model uncertainty still remains open.
4.3 Feature-wise Linear Modulation
The idea of controlling the batch normalization parameters modulate network's function has been explored by many different authors to accomplish different tasks. Conditional batch normalization
(CBN) was proposed by de Vries et al., who achieved good performance in VQA experiments by using an MLP to estimate residuals for normalization parameters used in pre-trained ResNet (De) Vries al. 2017b]. Perez et al. (2018) proposed FiLM also for solving VQA tasks. [Strub al. [2018] modify FiLM to produce normalization parameters in several stages instead of all at once. This formulation is better able handle longer conditioning information, such as dialogues instead of questions, and achieved excellent results on the GuessWhat?! visual dialogue task [De Vries et al.,
2017a].
Such modulation operations have also been used for image style transfer. It has been observed that the statistics of a feature map. which are associated with and can be controlled by the parameters produced by FiLM, are directly related to the style of an image [Huang and Belongie [017]. Dumoulin et al. 2017 succeeded in capturing the styles of artistic paintings, as well as combining extracted styles to create new ones, by using conditional instance normalization, which can be seen as variation of FiLM. FiLM.,Ghiasi Ghiasi a [2017) expand on this work by jointly training a style prediction network and style transfer network, which also operate based on conditional instance normalization. Finally, Brock et al. (2018) used FiLM for natural image synthesis using generative adversarial networks (GANs). They report that this allowed for a reduction in computation and memory costs, as well as 37% increase in training speed.
Yang et al. [018]) have used FILM to modulate the layers of segmentation network to perform video object segmentation. This made it possible avoid the fine-tuning process that was used by competing methods, which resulted in a 70x speed-up while achieving similar accuracy. Feature modulation has also been applied for various other tasks. Oreshkin et al. (2018) used FiLM for task- dependent metric scaling, which allowed them to achieve excellent results in few-shot classification. Vuorio al. (219)) use FiLM for meta-learning via task-aware modulation. The authors note that
FiLM outperforms attention-based modulation in this context, and is more stable. Finally, Vinyals) et al. 2019) used FiLM their AlphaStar neural architecture for multi-agent reinforcement learning. To our knowledge, FiLM has so far not been used for (implicit) model ensembling or uncertainty quantification.
5 Limitations & Future Work
The work presented this paper gives rise to several questions that can be explored in future work. Using pre-trained models is standard procedure in deep learning applications. It would be useful to explore how Finseble performs when used in conjunction with pre-trained models. This may not be straightforward since HILM-Ensemble (and also other implicit ensemble methods,

---

e.g., BatchEnsemble, MIMO) is based on variations between ensemble members, which may be reduced if all members are similarly initialized. Furthermore, selff-attention-bue based models, e.g.. Transformers [Vaswani et al. 2017] and Vision Transformers [ViT's, Dosovitskiy et al. 2021] have recently become very popular; therefore it is natural to adapt Finemle work with such models. Note that layer normalization is standard in Transformers instead of batch normalization, which prevents the straightforward application of the presented method in this case. Also, a number of measures designed to enhance implicit ensembles are orthogonal to our approach and could potentially be combined with FiLM-Ensemble to further improve its performance and uncertainty calibration, while minimally increasing the computational costs. It appears trraight-forwand the selection of independent examples for each member during training [as in MIMO, Havasi et al. data augmentation strategies [such as, e.g., Ramé
add
202T. temperature scaling [Guo et al. 2017
et al. 2021
6 Conclusion
In this paper we present Fil.M Fisemmen,e,be, novel implicit deep ensembling method. We achieve high efficiency using a simple yet effective idea - feature- beature-wise wise linear modulation which has been shown to be effective in different domains, such as image style transfer, model-agnostic meta-learning, or multi-task learning. Our extensive evaluation shows that FiL.M-Eneeble outperforms, or is on par with, state-of-the-art ensemble methods many different experimental settings.
Broader Impact
Machine learning has recently witnessed a steep increase in model sizes and associated computational costs, and as a consequence a rapid growth in energy consumption. For instance, training state-of-the- language models like GPT-3 would amount to at least 1400 MWh, or 4.6 million S. Therefore, recently the term Green A/ has been introduced, referring to Al research that yields novel results while taking into account the computational cost, encouraging reduction resources spent. To this extent, we believe that the presented method can foster more efficient ensemble methods and be helpful towards a greener AI.
Acknowledgements We would like to thank Liyuan Zhu for his invaluable contributions to experi- ments. H.A.G. received funding from German Federal Ministry of Education and Research (BMBF, Grant "GenomeNet" 031L0199B), B. B. and M. R. were supported by the Bavarian Ministry of Economic Affairs, Regional Development and Energy through the Center for Analytics Data Applications (A (ADACenter) within the framework of BAYERN DIGITAL II (20-3410-2-9-8) and the German Federal Ministry of Education and Research and the Bavarian State Ministry for Science and the Arts.
References
A. Ashukha, A. Lyzhov, D. Molchanov, and D. Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. In International Conference Learning Representations, 2020.
K. Bibas, M. Feder, and Hassner. Single layer predictive normalized maximum likelihood for out-of- distribution detection, Advances in Neural Information Processing Systems, 34:1179-1191, 2021.
C. Blundell, Comebise, Kavukcuoglu, and Wierstra. Weight uncertainty neural network. In International Conference on Machine Learning, volume 37, 2015.
A. Brock, Donahue, and Simonyan. Large scale gan training high fidelity natural image synthesis. In International Conference on Learning Representations, 2018
J. L. Campbell and N. Kleckner. coli oric and the dnaa gene promoter are sequestered from dam methyltrans- ferase following the passage of the chromosomal replication fork. Cell, 62(5):967-979, 1990.
T. Chen, E. Fox, and C. Guestrin. Stochastic gradient Hamiltonian Monte Carlo. In International Conference on Machine Learning, volume 32, 2014.
L. Cheng, J. Sun, W. Xu. L. Dong. Y. Hu, and M. Zhou. Oahg: an integrated resource for annotating human genes with level ontologies. Scientific reports, 6(1): 2016.
10

---

H. De Vries, F. Strub, S. Chandar, O. Pietquin. H. Larochelle, and A. Courville. Guesswhat?! visual object discovery through multi-modal dialogue. In IEEE Conference Computer Vision and Pattern Recognition, 2017a.
H. De Vries, F. Strub, J. Mary, H. Larochelle, O. Pietquin, and A. C. Courville. Modulating early visual processing by language. In Advances in Neural Information Processing Systems, volume 30, 2017b.
A. Der Kiureghian and O. Ditlevsen. Aleatory or epistemic? does it matter? Structural Safety, 31((): 105-112 2009.
A. Diaz-Pinto, A. Colomer, V. Naranjo. S. Morales, Xu, and A. F. Frangi. Retinal image synthesis and semi- supervised learning for glaucoma assessment. IEEE Transactions Medical Imaging, 38(9):2211-22) 2019.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
V. Dumoulin, J. Shlens, and M. Kudlur. A learned representation for artistic style. ICI.R, 2017.
N. Durasov, T. Bagautdinov, P. Baque, and P. Fua. Masksembles for uncertainty estimation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13539-13548, 2021. 2021.
J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint A.K3:08803303633 2018.
Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference Machine Learning, 2016.
G. Ghiasi, H. Lec, M. Kudlur, V. Dumoulin, and J. Shlens. Exploring the structure of a real-time, arbitrary neural artistic stylization network. arXiv preprint arXiv:1705.06830 2017.
A. Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, volume 24, 2011.
C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, 2017.
K. Gustafsson, M. Danelljan, and T. B. Schon. Evaluating scalable bayesian deep lcarning methods for robust computer vision. In /EEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020.
Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
M. Havasi, Jenatton, S. Fort, J. Z. Liu, Snoek, B. Lakshminarayanan, A. M. Dai, and D. Tran. Training independent subnetworks for robust prediction. In ICLR, 2021.
G. Huang, Y. G. Pleiss, Liu, J. E. Hoperoft, and Q Weinberger. Snapshot ensembles: Train get m for free. In International Conference on Learning Representations, 2017.
H. Huang, 7. Li, L. Wang, S. Chen, B. Dong, and X. Zhou. Feature space singularity for out-of-distribution detection, arXiv preprint arXiv:201 1.1654, 2020.
X. Huang and S. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In IEEE International Conference on Computer Vision, 2017.
P. Izmailov, S. Vikram, M. D. Hoffman, and A. G. G. Wilson. What bayesian neural network posteriors really like? In International Conference on Machine Learning, volume 139, 2021.
A. Kendall and Y. Gal. What uncertainties do need in bayesian deep learning for computer vision? In Advances Neural Information Processing Systems, 2017.
A. Krizhevsky, Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30, 2017a.
B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017b.
11

---

N. Lee. T. Ajanthan, and P. H. Torr. Snip: Single-shot network pruning based on connection sensitivity. ICLR, 2019.
Li N. Wang, J. Shi, X. Hou, and J. Liu, Adaptive batch normalization for practical domain adaptation. Pattern Recognition, 80: 10:109-117, 2018.
Z. Li, H. Jiang, L. Kong, Y. Chen, K. Lang, X. Fan, L. Zhang, and C. Pian. Deepóma: deep learning framework for exploring similar patterns in dna n6-methylldennig sites across different species. PLoS computational biology, 1008767, 2021.
J. Liu, Z. Lin, S. Padhy, D. Tran, T. Bedrax Weiss, and B. Lakshminarayanan. Simple and principled uncer- tainty estimation with deterministic deep learning via distance awareness. Advances in Neural Information Processing Systems, 33:7498-7512. 2020.
L. Loshchilov and F. Hutter. SGDR: stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017.
A. Malinin and M. Gales. Predictive uncertainty estimation via prior networks. In Advances in Neurul Information Processing Systems, volume 31, 2018.
A. Mallya and S. Lazebnik. Packnet: Adding multiple tasks to a single network by iterative pruning. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages 7765-7773, 2018.
R. M. Neal. Bayesian Learning for Neural Networks. Springer-Verlag Berlin, Heidelberg, 1996.
B. Oreshkin, Rodríguez I lópez, and A. Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in Neural Information Processing Systems, 31, 2018.
J. I. Orlando, H. Fu, J. B. Breda, K. van Keer, D. R. Bathula, A. Diaz-Pinto, R. Fang, P.-A. Heng. J. Kim, J. Lee, et al. Refuge challenge: A unified framework for evaluating automated methods for glaucoma assessment from fundus photographs. Medical Image Analysis, 59: 101500, 2020.
Y. Ovadia, E. Fertig, J. Ren, Z. Nado, D. Sculley, S. Nowozin, J. Dillon, B. Lakshminarayanan, and J. Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, volume 32, 2019.
E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville. FILM: Visual reasoning with a general conditioning layer. AAAI Conference on Artificial Intelligence, volume 32, 2018.
J. Pukkila, J. Peterson, G. Herman, Modrich, and M. Meselson. Effects of high levels of dna adenine methylation on methyl-directed mismatch repair escherichia coli. Genetics, 104(4):571-583 1983.
A. Ramé, Sun, and M. Cord. Mixmo: Mixing multiple inputs for multiple outputs via deep subnetworks. In IEEE/CVF International Conference on Computer Vision, 2021.
Ranganath, Gerrish, and Blei. Black box variational inference. Internationnl Conference on Artificial Intelligence and Statistics, 2014.
Strub, M. Scurin, Perez, De Vries, J. Mary, P. Preux, and C. Pictquin. Visual reasoning with multi-hop feature modulation. In European Conference on Computer Vision, 2018
M. Takeda, Benitez, and K. Yanai. Training multiple and mixed tasks with single network using feature modulation. In International Conference on Puttern Recognition, pages 719-735. Springer, 2021.
A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
O. Vinyals, Bahuschkin, W. M. Czamecki, M. Mathieu, A. Dudzik, J. Chung, D. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, 575 (7782):350-354, 354, 2019.
R. Vuorio, S.-H. Sun, H. Hu, J. Lim. Multimodal model-agnostic meta-learning via task-aware modulation. In Advances in Neural Information Processing Systems, volume 32, 2019.
M. Welling and Y. W. Tch. Bayesian lcarning via stochastic gradient Langevin dynamics. In International Conference on Machine Learning 201
Y. Wen, D. Tran, and J. Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference Learning Representations, 2020.
12

---

A. G. Wilson and P. Izmailov. Bayesian deep learning and probabilistic perspective of generalization. In Advances Neural Information Processing Systems, volume 33, 2020.
L. Yang, Y. Wang, X. Xiong, J. Yang, and A. K. Katsaggelos. Efficient video object segmentation via network modulation. In IEEE Conference Computer Vision and Pattern Recognition, 2018.
C. Zhang and Y. Ma. Ensemble machine learning: methods and applications. Springer, 2012.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] Purely methodological contribution that, when added to learning system, does not alter its potential societal impacts. If anything, better uncertainty calibration reduces the risk of negative effects caused by undetected prediction errors.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [No] license information for the used datasets public.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "naCQbljJvHo", "reviewer": "Reviewer_xEc2", "summary": "The paper proposes the use of a set of FiLM layers instead of distinct neural networks as a way to build an ensemble. This has the benefit of drastically reducing the number of weights used by the ensemble, since it only relies on cheap channel-wise scale and shift vectors. The method has been compared with a variety of baselines (e.g. Deep Ensembles, MIMO, MC-Dropout, etc), using a few backbones (ResNets and VGG), on image-classification datasets such as CIFAR10, CIFAR100, and Glaucoma Detection. The empirical results show performances that are comparable with leading methods in terms of classification accuracy and calibration error with the benefit of a lower parameter count.", "questions": "1) I would like to see a discussion of the concerns that I have pointed out in the \"weaknesses\" section.\n\n2) The parameters of the FiLM layers are sampled from the uniform distribution bounded by the values reported in Equation 4. I was wondering if a similar procedure has been used to produce the members of the Deep Ensemble. This is crucial, as differences in the initialization strategy could affect the performance of the baselines.\n\n3) Assigning a different set of initial weights to the member of the ensembles is just one way to enforce diversity. Did the authors try other techniques?", "limitations": "The authors have adequately addressed the limitations of their work.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n---------\n\n- The method is novel and original, the use of FiLM layers to create an ensemble provides substantial memory savings.\n\n- The method is compared against strong baselines (e.g. Deep Ensembles).\n\n- The paper is well written, easy to follow, and clear.\n\n\nWeaknesses\n----------\n\n- The empirical evaluation has been performed on relatively small networks (e.g. ResNet18 and ResNet34) and simple datasets (e.g. CIFAR10 and CIFAR100). It would be useful to verify if the same performance gains can be obtained in larger models and datasets. \n\n- Limited variety of networks, the method has been mainly tested on ResNets. The number of parameters in a FiLM layer is related to the number of channels in that layer; in recent architectures (e.g. MobileNet and EfficientNet) the proportion of number of channels vs. number of layers can vary drastically w.r.t. ResNets. This seems quite relevant, since the total number of FiLM parameters used in the model can affect the performance, and change the gap w.r.t. the baselines (in particular Deep Ensembles). It would be useful to verify if the method works well also on other architectures.\n\n- The use of the Expected Calibration Error (ECE). A recent line of work exposed several problems of the ECE score [1,2,3,4]. ECE is a biased estimate of the true calibration, and it only estimates miscalibration in terms of the maximum assigned probability, disregarding the full predicted probability. Other metrics have been proposed to fix those issues, for instance the Overconfidence Error(OE), Static Calibration Error (SCE), the Adaptive Calibration Error (ACE), and the Thresholded Adaptive Calibration Error (TACE), see [3] for reference. \n\n- The use of inference time in Table 1 can be misleading, as this can vary based on the machine overload and low-level implementation details. It would be better to use the total number of Multiply-accumulate operations (MACs) instead.\n\nReferences\n----------\n\n[1] Ashukha, A., Lyzhov, A., Molchanov, D., & Vetrov, D. (2020). Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06470.\n\n[2] Kumar, A., Liang, P., & Ma, T. (2019). Verified uncertainty calibration. arXiv preprint arXiv:1909.10155.\n\n[3] Nixon, J., Dusenberry, M. W., Zhang, L., Jerfel, G., & Tran, D. (2019, June). Measuring Calibration in Deep Learning. In CVPR Workshops (Vol. 2, No. 7).\n\n[4] Vaicenavicius, J., Widmann, D., Andersson, C., Lindsten, F., Roll, J., & Schön, T. (2019, April). Evaluating model calibration in classification. In The 22nd International Conference on Artificial Intelligence and Statistics (pp. 3459-3467). PMLR.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "na9owMBC72d", "reviewer": "Reviewer_4hVr", "summary": "The paper proposes a parameter efficient approach to ensembles based on FiLM layers. The idea is that each of the ensemble members share all of the parameters in the network with the exception of a set of the FiLM layer parameters that are specific to each member. The FiLM parameters in each ensemble member are initialized with different random values at the start of training and evolve independently over the training process. At test time, the logits of each of the ensemble members are averaged to make a prediction. The method is parameter efficient since the FiLM parameters make up only a tiny fraction of the overall network parameter count. Despite only a small number of parameters differing in each ensemble member, the authors show that the diversity of the ensemble members is high, and overall accuracy and calibration is competitive with methods that are more expensive in terms of memory and computation time.", "questions": "The method is quite simple and the paper was clear. As a result, there are no questions.", "limitations": "Despite the response in the checklist, I could not find where the authors described limitations of the proposed method. Some identified limitations of the work are described above in the **Weaknesses** section (i.e. the method was tested on relatively small networks and datasets). \n\nThe authors did not discuss any negative societal aspects of the work, which is unfortunate as image classification technology has a dark side (i.e. surveillance, military, etc.).", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper is clear and well written.\n\n2. The concept of using FiLM layers as the basis for an ensemble method is fundamentally good. The method is simple and effective.\n\n3. Results overall are excellent and compare favorably to leading competitive methods.\n\n**Weaknesses**\n1. The paper evaluates on relatively small and outdated networks (VGG, ResNet18/34). The paper would be improved if larger and more up to date networks (i.e. EfficientNets, Vision Transformers, etc.) were used in the evaluation.\n\n2. The datasets used to evaluate the method are relatively small. The paper would be improved if larger and potentially more challenging datasets/benchmarks were used (e.g. ImageNet, VTAB)\n\n3. There are no error bars in the tables, despite the fact that each experiment was repeated 3 times with different random seeds.\n\n4. While the out of distribution experiments compare to other ensemble methods, the paper would be improved if it compared to state-of-the-art (or even standard) OOD methods.\n\n5. The only calibration metric used was ECE. The paper would be improved if additional metrics were calculated (Brier sore, etc.).\n\nThere are some typos:\n- Line 56: na ive\n- Line 107: bot\n- Line 116: thid\n- Line 119: and **will** release the source code", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "i6KGJ2iKvgA", "reviewer": "Reviewer_uuAJ", "summary": "This paper proposes to use feature wise linear modulation [1] to construct implicit deep ensembles. In FilM [1], the authors propose to replace batch norm layer in a deep architecture with a conditional batch norm layer, with two parameters (shifting and scaling coefficients), which are used to transform features given a conditional input. In this work, the authors propose to use film layer, but to use multiple sets of affine parameters to instantiate different ensemble members. \nThe authors show for a couple of datasets (cifar10/100, retina glaucoma detection, not imagenet), some networks have performance better than other implicit ensemble approaches with respect to deep ensemble performance both in terms of accuracy and expected calibration error. \n\nRefsparame\n[1] Perez et al. 2018. FilM", "questions": "- How easy is it to learn the hyperparameters of film ensembles?\n- Have the authors tried to reproduce their results on larger datasets, i.e. imagenet?\n- How do the authors expect this approach to perform in tasks where fine tuning/using pretrained networks is necessary?  \n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written, where the authors explicitly state the contributions, comparison metrics and results.\n- The authors compare their proposed implicit ensemble model to a wide variety of existing implicit ensemble models.\nWeaknesses:\n- The paper would greatly benefit from a discussion on how the proposed film ensemble is different from MIMO or batch ensemble (BE) approaches. These could be added with minor equations in the appendix. Currently, the authors only state in line 163 \"[BE]... is closest to our work\".\n- From the paper, it is unclear to me how difficult it is to learn the film parameters for this model. In my experience, finding the best hyper parameters for batch ensemble models requires min hyper search, is this also true here?\n-  The authors mentioned that we can expect to trade-off accuracy vs gain given $\\rho$ which controls the variance of the weight initialization. From Fig 3, it is unclear if there is a pareto optimal solution (tradeoff between accuracy and ece) or fig 3 is simply an artifact for a particular dataset. Adding additional fig for other datasets would be helpful.\n- The authors state some failure modes for other implicit ensemble approaches, i.e. \"batch ensemble generally exhibits a negative correlation between ensemble size and test set accuracy\", \"mimo shows very poor performance..\" These results to my knowledge are not aligned with published work. For this part I would suggest to for the authors to better discuss the limitations of their baseline implementations. Conversely, if the authors do believe that this a standard result/common knowledge, they can explicitly state so/cite similar findings. This could be in addition to the discussion of how difficult it is to optimize the hyperparameters for film/other methods to construct implicit ensembles.  \n- The datasets considered are low dimensional/ low resolution. Have the authors tried to reproduce their results on larger datasets, i.e. imagenet?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "415TqY66v9E", "reviewer": "Reviewer_gCqT", "summary": "This paper proposes a simple implicit model ensemble method based on feature-wise linear modulation (FiLM), where they learn M sets of affine parameters for batch norm layers. Experiments show higher diversity than DE, high accuracy and ECE among implicit ensemble methods, and good OOD detection capabilities.", "questions": "In table 1, what causes the difference in inference time between MIMO and FiLM-Ensemble for ResNet-18? Aren’t both methods similar to feeding forward a batch of 16 datapoints (in terms of computation cost)?\n\nWhy do other methods perform so poorly in Table 2? If these methods were adequately tuned, could you report results on CIFAR10/100 with ResNet-28-10 and compare them to the results in the MIMO paper?", "limitations": "To my knowledge, this work has no potential negative societal impact other than what was already present in the existing literature.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method is conceptually simple in addition to being parallelizable.\n\nThe M sets of affine parameters are learned concurrently, and the M ensemble members would collapse to the same function if these parameters were the same. Do you have an intuition for why this does not happen? In other words, why is the different random initialization sufficient for diversity at the end of training, given that the functions share all other parameters?\n\nI don’t understand why Figure 1 happens; why should the pairwise diversity of FiLM-Ensemble be bigger than that of DeepEns if nothing in the objective is directly encouraging diversification? If anything, it seems like it should be lower because of the shared parameters. Also, why does diversity increase with the number of members if these metrics are averages of pairwise metrics? As you add more models, it intuitively seems that some of them should “overlap” and thus have lower diversity.\n\nTable 2 is unconvincing to me. The other networks (MIMO, BE, etc.) are doing worse than a single network, making it seem like the hyperparameters were not adequately tuned for these methods. For example, MIMO and BE perform better than a single deterministic network in the MIMO paper.\n\nLine 107: “…depend bot on” typo\nLine 116: “thid” typo", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept