RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
FiLM-Ensemble
Perez et al. ( 2018 ) achieved the linear feature modulation by varying the affine parameters of the batch normalization layers \bm{\gamma} and \bm{\beta} .
For each batchnorm layer n in the network, the parameters are predicted according to some conditioning input z (for instance, different prediction tasks):

\bm{\gamma}_{n}=g_{n}(z)\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \bm{\beta}_{n}=h_{n}(z) (1)

The size of \bm{\gamma}_{n} and \bm{\beta}_{n} is \mathbb{R}^{D_{n}} where D_{n} is the feature dimension at layer n .
These parameters are then used to linearly modulate the activations at the n -th layer \mathbf{F}_{n} :

\textrm{FiLM}(\mathbf{F}_{n}|\bm{\gamma}_{n},\bm{\beta}_{n})=\bm{\gamma}_{n}(z)\circ\mathbf{F}_{n}+\bm{\beta}_{n}(z)\;, (2)

with \circ being the Hadamard (element-wise) product taken w.r.t. the feature dimension.

In our scenario we aim to use the affine parameters to instantiate different ensemble members, i.e., the conditioning variable z is simply an index m\in\{1,...,M\} that identifies the ensemble member.
The functions g_{n}(z) and h_{n}(z) degenerate to look-up tables, so we can dispose of them and simply learn M different sets of affine parameters for every batchnorm layer \mathbf{F}_{n} :

\textrm{FiLM}(\mathbf{F}_{n}|\bm{\gamma}^{m}_{n},\bm{\beta}^{m}_{n})=\bm{\gamma}^{m}_{n}\circ\mathbf{F}_{n}+\bm{\beta}^{m}_{n}\;

To achieve ensemble members with diverse parameters, we resort to Xavier initialization, i.e., all \bm{\gamma}_{n}^{m} and \bm{\beta}_{n}^{m} are sampled from a uniform distribution bounded between:

\pm\frac{\sqrt{3}}{\sqrt{D_{n}}}\rho\;, (4)

where D_{n} is the number of features (channels) in the n -th layer, and \rho is an initialization gain factor.
In our setting \rho is a tunable hyperparameter that allows one to control the trade-off between predictive accuracy and calibration of the model, see Section 3.5 . In general, increasing \rho leads to more diverse ensemble members and thus favours calibration. Note that as \rho\rightarrow 0 , all ensemble members start from similar initial values for \bm{\gamma}_{n}^{m} and \bm{\beta}_{n}^{m} and the FiLM-Ensemble gradually collapses to a single model.

During training, we feed each input sample \mathbf{x} to every ensemble member and obtain predictions y_{m}=f_{\bm{\theta},\bm{\gamma}^{m},\bm{\beta}^{m}}(\mathbf{x}) , which depend both on the member-specific parameters (\bm{\beta}^{m},\bm{\gamma}^{m}) and on the shared parameters \theta .
All those parameters are optimized together to minimise the chosen loss function. Here we focus on classification and use a standard cross-entropy (CE) loss.

At inference time the final prediction \hat{y} is obtained by averaging the M predictions of the ensemble:

\hat{y}=\frac{1}{M}\sum_{m=1}^{M}y_{m}\;
Implementation Details
We implemented 1-dimensional and 2-dimensional FiLM-Ensemble layers, to be used in combination with 1-dimensional and 2-dimensional convolution operations, respectively.
The forward passes through different ensemble members can easily be parallelized by replicating both the input tensor and the FiLM parameters \bm{\gamma},\bm{\beta} along the batch dimension and applying the affine transformation Eq. 3 (the same holds for the BatchEnsemble method).
In this way all ensemble members run simultaneously on a single device, thus optimally utilizing modern tensor computing hardware without having to load several instances of the classification network into memory.
We have implemented FiLM-Ensemble in PyTorch and release the source code. 1 1 1 https://github.com/prs-eth/FILM-Ensemble

For all experiments, we optimize the model parameters with standard stochastic gradient descent, with momentum \mu\!=\!0.9 and weight decay \lambda\!=\!0.0005 for regularization. We train for 200 epochs with batch size 128.
The learning rate is initially set to 0.1 and decays according to a cosine annealing schedule ( Loshchilov and Hutter , 2017 ) .
The initialisation gain is set to \rho\!=\!2 for all experiments, except for genome sequences (see Table 3 ), where \rho\in\{4,8,16,32\} .
Please refer to the supplementary material for additional, dataset-specific details.
Diversity Analysis
Diversity is the key to constructing powerful ensembles: nothing is to be gained from highly correlated ensemble members that return similar outputs for (almost) any input ( Zhang and Ma , 2012 ) .
In order to analyze the diversity (respectively, the degree of independence) among members, we compute two distance metrics between the members’ predictive distributions.
Let f_{i} and f_{j} denote two different ensemble members for a classification task. We first measure the disagreement score \mathcal{D} , defined as the fraction of all s test samples for which the two members return different answers, averaged over all possible pairs of members:

\mathcal{D}=\frac{2}{M(M-1)}\sum_{i=1}^{M}\sum_{j=i+1}^{M}\sum_{k=1}^{s}\frac{1}{s}\big{[}f_{i}(\mathbf{x}_{k}) eq f_{j}(\mathbf{x}_{k})\big{]}\;, (6)

with [\cdot] being the Iverson bracket. Second, we also measure the Kullback–Leibler (KL) divergence between the two predictive distributions p(f_{i}) and p(f_{j}) , again averaged over all pairs:

KL=\frac{2}{M(M-1)}\sum_{i=1}^{M}\sum_{j=i+1}^{M}\sum_{k=1}^{s}p(f_{i}(\mathbf{x}_{k}))\log\frac{p(f_{i}(\mathbf{x}_{k}))}{p(f_{j}(\mathbf{x}_{k}))}\;

For both metrics, higher numbers correspond to higher diversity.
See Fig. 1 .
We observe that the average diversity increases with the number of ensemble members. In both metrics, FiLM-Ensemble achieves higher scores than the naïve, explicit deep ensemble. Meaning that the predictions of FiLM-Ensemble are less correlated than those of an equivalent number of networks trained independently with different random seeds. Also, note that with the increasing number of members, improvements in diversity metrics for the naïve explicit deep ensemble are negligible.
Computational Cost
We go on to measure the efficiency of our proposed method, both in terms of parameter count and in terms of resources needed for a forward pass. Table 1 (left section) reports the numbers of additional numbers of parameters compared to a single network (i.e., without uncertainty calibration). Due to their design, MC-Dropout and Masksemble do not require any additional parameters.
Among the remaining methods, FiLM-Ensemble has the lowest (VGG-11) or second-lowest (Resnet-18) overhead in terms of parameter count, while performing significantly better, as we will see below. In terms of inference complexity (Tab. 1 , center and right), FiLM-Ensemble turns out to be very competitive. Only MIMO with a ResNet-18 backbone is significantly faster and lighter, however this comes at a considerable price in terms of accuracy and calibration, see below.
CIFAR-10 / CIFAR-100
We perform several experiments on widely used benchmarks in computer vision, CIFAR-10 and CIFAR-100. As performance metrics, we plot the test set accuracy and the expected calibration error (ECE) against the ensemble size M in Fig. 2 ,
for all compared methods. In terms of accuracy (left subfigure), FiLM-Ensemble is outperformed only by the explicit deep ensemble, across all tested values of M\in\{2,4,8,16\} . These two surpass all other methods by a clear margin. Surprisingly, we observe that BatchEnsemble generally exhibits a negative correlation between ensemble size and test set accuracy, with M=4 performing best among all settings. MIMO shows very poor performance in this experimental setting in terms of accuracy (and also in ECE). We speculate that its shared backbone probably has a tendency to fragment into largely independent ensemble members of low channel depth, which lack the necessary capacity when using comparatively small networks like VGG-11 or Resnet-18.
With regard to ECE (right subfigure), we see all methods improving with growing ensemble size M .
FiLM-Ensemble achieves better calibration then the widely used deep ensemble and MC-dropout methods, with significant margins at large ensemble sizes. Masksemble and BatchEnsemble achieve very good calibration for sizes M\in[4\ldots 16] , but at the cost of significant drops in classification performance.
An attractive feature of FiLM-Ensemble is that it offers a simple mechanism for trading off accuracy against calibration, by tuning the gain factor \rho . See Section 3.5 .

In Tab. 2 we quantitatively compare all tested methods on the CIFAR-100 dataset, both with a standard Resnet-18 backbone and with a larger Resnet-34. We observe that with both architectures, and in terms of both predictive accuracy and calibration, FiLM-Ensemble always either ranks first, or it ranks second behind the inefficient, explicit deep ensemble.
Also, note that when the capacity of the network is increased (Resnet-18 \rightarrow Resnet-34), calibration performance improves for some of the implicit methods, but at the cost of reduced accuracy.
Overall, the experiments with both variants of CIFAR and all three model architectures confirm that our method presents an excellent trade-off between predictive accuracy, uncertainty calibration and computational efficiency.
Retinal Glaucoma Detection
Glaucoma is currently the leading reason of irreversible blindness in the world. Detection of glaucomatous structural damages and changes is a challenging task in the field of ophthalmology. We evaluate our proposed FiLM-Ensemble, as well as the baselines described above, on the ask of diagnosing Glaucoma and quantifying the uncertainty associated with the prediction, see Table 3 . The proposed FiLM-Ensemble achieves the best classification result across all ensemble sizes, and also the best overall result, with M\!=\!16 . Whereas there is no clear trend with respect to uncertainty calibration.
6mA Identification
With the 6mA identification task we show that FiLM-Ensemble can also be readily combined with existing models for 1-dimensional sequential genome data. We use the 6mA-rice-Lv dataset and a 1D-CNN architecture whose hyper-parameters have already been optimized for this dataset ( Li et al. , 2021 ) . FiLM-Ensemble improves the accuracy and the calibration of that model, see Fig. 3 . Our method performs on par with the explicit deep ensemble and better than a single instance of the model tuned for the specific task. More importantly, one can reach a significantly better calibration (lower ECE) by increasing the gain \rho , with only a minimal accuracy drop by <0.25 percent points.
Please refer to the Section B.2 for more results.
Out-of-Distribution Detection
Domain shift often occurs in medical datasets;
thus, detecting out-of-distribution (OOD) samples is an important task in clinical diagnosis. Model uncertainty can be used for OOD detection.
For our experiment, we regard the retinal Glaucoma dataset ( Diaz-Pinto et al. , 2019 ) as the within-distribution samples and perform OOD detection with the REFUGE dataset ( Orlando et al. , 2020 ) . As performance metric, we report AUROC (Area Under the Receiver Operating Characteristic) scores, Table 4 . FiLM-Ensemble can detect OOD test samples with significantly higher accuracy than other efficient ensemble methods, standard state-of-the-art OOD detection methods, and even the explicit deep ensemble. This suggests that for challenging test samples, which are not adequately represented in the training data,
the uncertainties estimated with FiLM-Ensemble are better calibrated.
Epistemic Uncertainty Quantification
A large corpus of related work addresses the estimation of epistemic (model) uncertainty in neural networks.
At the heart of such modeling is often the concept of marginalization instead of optimization , i.e., integrating out a (possibly uncountably infinite) set of models weighted by their posterior probability, instead of committing to a point estimate of that distribution. A multitude of methods have been proposed to implement approximate Bayesian inference w.r.t. the model weights, given the training data and an appropriate prior, a process that is not analytically tractable in general ( Kendall and Gal , 2017 ) .

For instance, methods based on variational inference ( Graves , 2011 ; Ranganath et al. , 2014 ; Blundell et al. , 2015 ) seek to learn an approximate posterior distribution which is a member of a simpler family of variational distributions. This variational distribution can often be learned using backpropagation ( Blundell et al. , 2015 ) and can then be sampled from, or sometimes even used for exact inference.
Markov chain Monte Carlo sampling (MCMC) approaches ( Neal , 1996 ; Welling and Teh , 2011 ; Chen et al. , 2014 ) construct a Markov chain which has the exact posterior distribution as its stationary distribution, that can then be employed for sampling. However, in practise, these approaches often fail to sufficiently explore high-dimensional, multi-modal loss landscapes as they are common in deep learning ( Gustafsson et al. , 2020 ) .
Ensembles and Sub-networks
In a method referred to as deep ensembles ( Lakshminarayanan et al. , 2017a ) , a set of M neural network models are randomly and independently initialized, and are subjected to stochastic mini-batch sampling during SGD training. The models generally converge to different minima in the parameter space, and can be considered samples from an approximate posterior ( Wilson and Izmailov , 2020 ; Gustafsson et al. , 2020 ; Izmailov et al. , 2021 ) . Deep ensembles often achieve the best calibration and predictive accuracy ( Ovadia et al. , 2019 ; Gustafsson et al. , 2020 ; Ashukha et al. , 2020 ) , but suffer from high computational complexity as they require training, storing, and running inference on several full instances of the network.

Many alternative methods have recently been proposed in an attempt to reduce either the computational cost or the storage cost of deep ensembles.
Monte Carlo (MC) Dropout ( Gal and Ghahramani , 2016 ) runs multiple forward passes on the dropout layers in order to obtain multiple predictions and obtain an uncertainty estimate. Although the method requires fewer computations compared to deep ensembles, it also leads to less accurate uncertainty estimates.
Snapshot ensembles ( Huang et al. , 2017 ) use a cyclic learning rate schedule in order to find multiple local minima, they then store multiple copies of the network to create the ensemble. While this approach reduces the computation cost during training it does not alleviate the storage cost.

BatchEnsembles ( Wen et al. , 2020 ) employ multiple low rank matrices, which can be stored efficiently, in order to modulate the parameters of a neural network and thus mimic an ensemble of network models.
Masksembles ( Durasov et al. , 2021 ) aim to improve the performance of MC Dropout by carefully selecting the dropout masks, used to drop certain features, such that they lead to better uncertainty quantification.
Another approach that uses multiple sub-networks is proposed in Havasi et al. ( 2021 ) . In this case additional, independent layers are added at the beginning and at the end of the network, in order to obtain multiple prediction with an single backbone model.
Although such method seems to reduce the computational resources required at training and inference time,
the gain is limited to larger backbones, such as Wide-ResNet (e.g., ResNet28-10), whereas for
widely used standard architectures like VGG (e.g., -16), or ResNet (e.g., -34), the approach is not very effective, as shown in our experiments.

Although many methods have been proposed that aim at reducing the computational cost of deep ensembles, none of them appears to clearly outperform most others. In summary, the question how to effectively model uncertainty still remains open.
Feature-wise Linear Modulation
The idea of controlling the batch normalization parameters to modulate a network’s function has been explored by many different authors to accomplish different tasks. Conditional batch normalization (CBN) was proposed by de Vries et al. , who achieved good performance in VQA experiments by using an MLP to estimate residuals for normalization parameters used in a pre-trained ResNet ( De Vries et al. , 2017b ) . Perez et al. ( 2018 ) proposed FiLM also for solving VQA tasks. Strub et al. ( 2018 ) modify FiLM to produce normalization parameters in several stages instead of all at once. This formulation is better able to handle longer conditioning information, such as dialogues instead of questions, and achieved excellent results on the GuessWhat?! visual dialogue task ( De Vries et al. , 2017a ) .

Such modulation operations have also been used for image style transfer. It has been observed that the statistics of a feature map, which are associated with and can be controlled by the parameters produced by FiLM, are directly related to the style of an image ( Huang and Belongie , 2017 ) . Dumoulin et al. ( 2017 ) succeeded in capturing the styles of artistic paintings, as well as combining extracted styles to create new ones, by using conditional instance normalization, which can be seen as a variation of FiLM. Ghiasi et al. ( 2017 ) expand on this work by jointly training a style prediction network and a style transfer network, which also operate based on conditional instance normalization. Finally, Brock et al. ( 2018 ) used FiLM for natural image synthesis using generative adversarial networks (GANs). They report that this allowed for a reduction in computation and memory costs, as well as a 37% increase in training speed.

Yang et al. ( 2018 ) have used FiLM to modulate the layers of a segmentation network to perform video object segmentation. This made it possible to avoid the fine-tuning process that was used by competing methods, which resulted in a 70 \times speed-up while achieving similar accuracy.
Feature modulation has also been applied for various other tasks. Oreshkin et al. ( 2018 ) used FiLM for task-dependent metric scaling, which allowed them to achieve excellent results in few-shot classification. Vuorio et al. ( 2019 ) use FiLM for meta-learning via task-aware modulation. The authors note that FiLM outperforms attention-based modulation in this context, and is more stable. Finally, Vinyals et al. ( 2019 ) used FiLM in their AlphaStar neural architecture for multi-agent reinforcement learning.
To our knowledge, FiLM has so far not been used for (implicit) model ensembling or uncertainty quantification.
Limitations & Future Work
The work presented in this paper gives rise to several questions that can be explored in future work.
Using pre-trained models is standard procedure in deep learning applications. It would be useful to explore how FiLM-Ensemble performs when used in conjunction with pre-trained models. This may not be straightforward since FiLM-Ensemble (and also other implicit ensemble methods, e.g., BatchEnsemble, MIMO) is based on variations between ensemble members, which may be reduced if all members are similarly initialized.
Furthermore, self-attention-based models, e.g., Transformers ( Vaswani et al. , 2017 ) and Vision Transformers (ViTs, Dosovitskiy et al. , 2021 ) have recently become very popular; therefore it is natural to adapt FilM-Ensemble to work with such models. Note that layer normalization is standard in Transformers instead of batch normalization, which prevents the straightforward application of the presented method in this case.
Also, a number of measures designed to enhance implicit ensembles are orthogonal to our approach and could potentially be combined with FiLM-Ensemble to further improve its performance and uncertainty calibration, while minimally increasing the computational costs. It appears straight-forward to add the selection of independent examples for each member during training (as in MIMO, Havasi et al. , 2021 ) , temperature scaling ( Guo et al. , 2017 ) or data augmentation strategies (such as, e.g., Ramé et al. , 2021 ) .
Conclusion
In this paper we present FiLM-Ensemble, a novel implicit deep ensembling method. We achieve high efficiency using a simple yet effective idea – feature-wise linear modulation – which has been shown to be effective in different domains, such as image style transfer, model-agnostic meta-learning, or multi-task learning.
Our extensive evaluation shows that FiLM-Ensemble outperforms, or is on par with, state-of-the-art ensemble methods in many different experimental settings.
Broader Impact
Machine learning has recently witnessed a steep increase in model sizes and associated computational costs, and as a consequence a rapid growth in energy consumption. For instance, training state-of-the-art language models like GPT-3 would amount to at least 1400 MWh, or 4.6 million $.
Therefore, recently the term Green AI has been introduced, referring to AI research that yields novel results while taking into account the computational cost, encouraging a reduction in resources spent.
To this extent, we believe that the presented method can foster more efficient ensemble methods and be helpful towards a greener AI.

Acknowledgements We would like to thank Liyuan Zhu for his invaluable contributions to experiments. H.A.G. received funding from German Federal Ministry of Education and Research (BMBF, Grant “GenomeNet” 031L0199B). B. B. and M. R. were supported by the Bavarian Ministry of Economic Affairs, Regional Development and Energy through the Center for Analytics - Data - Applications (ADACenter) within the framework of BAYERN DIGITAL II (20-3410-2-9-8) and the German Federal Ministry of Education and Research and the Bavarian State Ministry for Science and the Arts.
A.1 CIFAR-10 / CIFAR-100
Please refer to Section 2.1 .
A.2 Retinal Glaucoma Detection
We followed the same training procedure as for CIFAR-10/100, please refer to Section 2.1 .
Resnet-18 is used as a backbone. Models are trained with 70% of the dataset for 150 epochs and tested on the test set (30% of the dataset).
A.3 6mA Identification
A 1-dimensional CNN architcture is used, whose hyperparameters such as kernel size and the number of layers are optimized by Li et al. ( 2021 ) . The CNN consists of 5 convolutional blocks, where each block contains a 1-dimensional convolution, ReLU activation, batch normalization, and dropout with a rate of 0.5. The convolutional layers have a filter size of 256, kernel size of 10, a stride of 1, and the first convolutional layer have a padding of 5. On top of the last convolutional block, there is a linear layer for predicting the binary labels. Binary cross-entropy is used as a loss.
All models are trained for 20 epochs.
The initial learning rate of 0.01 is used, as in Li et al. ( 2021 ) . Cosine-annealing is used as a learning rate scheduler.
B.1 EfficientNet as Backbone & More Calibration Metrics
We run more experiments using more modern architecture: EfficientNet whose proportion of the number of channels vs. the number of layers can vary drastically, compared to Resnets.
In this experiment we use two extra calibration metrics: (i) the Brier score Brier et al. ( 1950 ) and (ii) the Static Calibration Error (SCE) Nixon et al. ( 2019 ) . SCE can be considered an extension of ECE but more accurately account for calibration by considering all classes, instead of just the one with the highest confidence.
Table 5 shows that FiLM-Ensemble can also be effectively used in conjunction with EfficientNet architecture. In addition, other calibration metrics are also in favor of FilM-Ensemble.
B.2 Calibration-Accuracy Trade-off
As in Section 3.5 , we show that one can reach a significantly better calibration (lower ECE) by increasing the gain \rho , with only a minimal accuracy drop. In this case, we use Resnet-34 with 2 ensemble members on Cifar-100 dataset. See Fig. 4 .
Also note Fig. 5 is an extension of Fig. 3 with various number of ensemble members.
Figure 1: Diversity analysis: CIFAR-10/VGG-11 experiment. With increasing number of members, FiLM-Ensemble achieves more diverse representations. See Section 3.1 .; Diversity analysis: CIFAR-10/VGG-11 experiment. With increasing number of members, FiLM-Ensemble achieves more diverse representations. See Section 3.1 .
Figure 2: Accuracy and ECE for CIFAR-10, with varying ensemble sizes, using VGG-11 as backbone.; Accuracy and ECE for CIFAR-10, with varying ensemble sizes, using VGG-11 as backbone.