RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Thanks to this design, our method requires only a very small number of additional parameters on top of the base network, e.g., converting a single ResNet-18 model to an ensemble of 16 models increases the parameter count by 1.3%, compared to an increase by 1500% when setting up a naïve ensemble, see Table 1 .
Evidence type: table
Caption: Table 1: Memory and inference complexity comparison (CIFAR-10/100 datasets): Number of additional trainable parameters to have 16 ensemble members for different backbones. The inference time (mult-adds) shown corresponds to the mean GPU time (number of multiply-add operations) required to run a forward pass for a batch of size 1 with 16 ensemble members. The bottom section comprises methods whose forward and backward passes are implemented in parallel over ensemble members. Measurements are done on an NVIDIA GeForce GTX 1080 Ti.
Context: Inspired by that line of work, we propose a new, efficient ensemble method, FiLM-Ensemble. Our method adapts
feature-wise linear modulation as an alternative way to construct an ensemble for (epistemic) uncertainty estimation. FiLM-Ensemble greatly reduces the computational overhead compared to the naïve ensemble approach, while performing almost on par with it, sometimes even better. In a nutshell, FiLM-Ensemble can be described as an implicit model ensemble in which each individual member is defined via its own set of linear modulation parameters for the activations, whereas all other network parameters are shared among ensemble members – and therefore only need to be stored and trained once.
Verdict: Supported
Claim: A first observation is that one can transfer these weights to deeper ResNets without significantly affecting the test accuracy of the model: it remains above 94.5\% on CIFAR-10 and 72\% on ImageNet.
Evidence type: table
Caption: Table 2: Test accuracy (ResNet)
Context: In addition, we also want our pretrained model to verify assumption 2 so we consider the following setup. On CIFAR (resp. ImageNet) we train a ResNet with 4 (resp. 8) blocks in each layer, where weights are tied within each layer.
Verdict: Supported
Claim: In the experiments of LiPopt, we only used LiPopt-2.
Evidence type: table
Caption: Table 1: \ell_{\infty} -FGL estimation of various methods: DGeoLIP and NGeoLIP induce the same values on two layer networks. DGeoLIP always produces tighter estimations than LiPopt and MP do.
Verdict: Supported