Introduction
A key component for reliable and trustworthy machine learning are algorithms that output not only accurate predictions of the target variables, but also well-calibrated estimates of their uncertainty ( Gal and Ghahramani , 2016 ) .
The overall uncertainty of a predictor is usually decomposed into two parts ( Der Kiureghian and Ditlevsen , 2009 ) . Aleatoric uncertainty is inherent in the data, for instance due to class overlap or sensor noise. On the contrary, epistemic uncertainty characterises the uncertainty of the model weights, due to a lack of knowledge about parts of the input space that are insufficiently represented in the training set. Uncertainty caused by distributional shifts between the training and test data is sometimes conceived as a third source of uncertainty ( Malinin and Gales , 2018 ) , but in practice often modelled as part of epistemic uncertainty.

Measuring epistemic uncertainty for complex models such as deep neural networks is not trivial: by definition one cannot derive it from the training data, since it concerns the behavior of the model in regions of the input space that are not represented in the training set.
Different methods have been explored ( Welling and Teh , 2011 ; Graves , 2011 ; Blundell et al. , 2015 ; Lakshminarayanan et al. , 2017a ; Huang and Belongie , 2017 ; Gal and Ghahramani , 2016 ) , however the de facto standard remain deep ensemble models ( Lakshminarayanan et al. , 2017a ) . In its basic form, such a deep ensemble is simply a collection of independently trained networks that can be regarded as Monte Carlo samples from the model space. To obtain ensemble members with reasonably low correlation, one can exploit the stochastic nature of the optimisation procedure, with different (random) weight initialisation and different (random) batches during stochastic gradient descent.
The expectation is that, once trained, the ensemble members will agree for inputs near the training samples, since the loss function favours similar outputs at those locations. Whereas they may disagree in unseen regions of the data space. Thus, the spread of their predictions is a measure of epistemic uncertainty.
The ensemble idea is conceptually very simple, but nevertheless yields uncertainties that are well calibrated, i.e., they are in line with the actual prediction errors (in expectation).

The drawback of deep ensembles is their large computational cost.
Both computation and memory consumption grow directly proportionally with the number of ensemble members, during training as well as during inference.
This makes them impractical in hardware-constrained settings, and leads to a widening gap as models continue to grow in size, while at the same time applications like mobile robotics, augmented reality and smart sensor networks increase the need for mobile and embedded computing.

To improve efficiency, several researchers have explored ways to mimic deep ensembles without explicitly duplicating the underlying network.
Possible strategies include the reuse and recombination of network modules (e.g., Wen et al. , 2020 ; Havasi et al. , 2021 ) , injection of noise at inference time (e.g., Gal and Ghahramani , 2016 ) , as well as hybrid variants (e.g., Durasov et al. , 2021 ) .
Empirically, these models do speed up training and/or inference, but still exhibit a significant performance gap compared to the naïve, explicit ensemble, both w.r.t. prediction quality and w.r.t. the calibration of the predicted (epistemic) uncertainties.

The lottery ticket hypothesis ( Frankle and Carbin , 2018 ) and other network pruning studies, e.g., by Han et al. ( 2015 ); Lee et al. ( 2019 ); Mallya and Lazebnik ( 2018 ) , underline that neural networks are heavily over-parameterized. Their parameters are used inefficiently, and they can be pruned significantly without large performance drops.
In lifelong learning and multi-task learning it is essential to use the network efficiently, in order to limit computational overhead when introducing new tasks. There are recent works that achieve good performance in these tasks by introducing modulation (respectively, adaptation) strategies. In particular, Li et al. ( 2018 ); Takeda et al. ( 2021 ) propose an efficient lifelong learning / domain adaptation method for multi-task learning, utilizing single feature-wise linear modulation (FiLM).

Inspired by that line of work, we propose a new, efficient ensemble method, FiLM-Ensemble. Our method adapts
feature-wise linear modulation as an alternative way to construct an ensemble for (epistemic) uncertainty estimation. FiLM-Ensemble greatly reduces the computational overhead compared to the naïve ensemble approach, while performing almost on par with it, sometimes even better.
In a nutshell, FiLM-Ensemble can be described as an implicit model ensemble in which each individual member is defined via its own set of linear modulation parameters for the activations, whereas all other network parameters are shared among ensemble members – and therefore only need to be stored and trained once.
Thanks to this design, our method requires only a very small number of additional parameters on top of the base network, e.g., converting a single ResNet-18 model to an ensemble of 16 models increases the parameter count by 1.3%, compared to an increase by 1500% when setting up a naïve ensemble, see Table 1 .
We further show that FiLM-Ensemble results in more diverse ensemble members compared to other efficient ensemble methods. For instance, on the Cifar-10 benchmark it achieves diversity scores >6.9\% and up to 9.2\% (depending on ensemble size), against 6.8\% for a naive ensemble, see Fig. 1 .
Our contributions can be summarized as follows.

We propose FiLM-Ensemble, a novel parameter- and time-efficient deep ensemble method.

FiLM-Ensemble is designed in such a way that it can be readily combined with many popular deep learning models, by simply replacing batch normalisation (batchnorm) layers with conditional batchnorm layers.

We show that FiLM-Ensemble provides an excellent trade-off between accuracy and calibration performance, improving over existing ensemble methods.