RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Figure 2 demonstrates that MPGD can lead to successful optimization of the widening loss whereas the baseline GD and GD with Gaussian perturbations lead to poor solutions.
Evidence type: figure
Caption: Figure 2: Evolution of GD with Gaussian perturbations (left plot) vs. that of MPGD with \gamma=0.7 , \beta=0.5 (middle plot), using \mu=0.02 , \sigma=0.05 , and \eta=0.01 ( m=100 ). Here we see that MPGD leads to successful optimization of the widening valley loss whereas that with Gaussian perturbations fails to converge. Moreover, MPGD effectively reduces the trace of loss Hessian (see right plot), steering the GD iterates to flatter region of the loss landscape.
Context: For the experiments, we start optimizing from the point (u_{0},0) , where u_{0}\sim 5\cdot\mathcal{U}(d) with d=10 , and use the learning rate \eta=0.01 . We study and compare the behavior of the following schemes: (i) baseline (vanilla GD), (ii) GD with uncorrelated Gaussian noise injection instead, and (iii) MPGD.
Verdict: Supported
Claim: We see that LUI is the only method capable of converging to optimal policies both before and after the change in environment.
Evidence type: figure
Caption: Figure 5 : Environment change on the Chain Problem at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=0.8 in the exploration.
Context: Finally, we investigate the behaviour of the learning methods when after a fixed number of trajectories the probabilities of the true MDP change, as introduced in Section 5.4 . Figure 5 shows the performance of the robust policy for each learning method on the Chain environment, results for the same experiment on the Betting Game can be found in Appendix F .
After {\dagger}\in\{10^{2},\dots,10^{5}\} trajectories, we change the environment by swapping the transition probabilities, for three different bounds on the strength intervals. We similarly bound the MAP-estimation priors for MAP and UCRL. PAC is omitted from this experiment as PAC guarantees lose all meaning when changing the underlying distribution. After the change in environment, the new optimal policy has to use the opposite action from the previously optimal policy.
Verdict: Supported
Claim: For instance, on the Cifar-10 benchmark it achieves diversity scores >6.9\% and up to 9.2\% (depending on ensemble size), against 6.8\% for a naive ensemble, see Fig. 1 .
Evidence type: figure
Caption: Figure 1: Diversity analysis: CIFAR-10/VGG-11 experiment. With increasing number of members, FiLM-Ensemble achieves more diverse representations. See Section 3.1 .
Context: Inspired by that line of work, we propose a new, efficient ensemble method, FiLM-Ensemble. Our method adapts
feature-wise linear modulation as an alternative way to construct an ensemble for (epistemic) uncertainty estimation. FiLM-Ensemble greatly reduces the computational overhead compared to the naïve ensemble approach, while performing almost on par with it, sometimes even better. In a nutshell, FiLM-Ensemble can be described as an implicit model ensemble in which each individual member is defined via its own set of linear modulation parameters for the activations, whereas all other network parameters are shared among ensemble members – and therefore only need to be stored and trained once. Thanks to this design, our method requires only a very small number of additional parameters on top of the base network, e.g., converting a single ResNet-18 model to an ensemble of 16 models increases the parameter count by 1.3%, compared to an increase by 1500% when setting up a naïve ensemble, see Table 1 . We further show that FiLM-Ensemble results in more diverse ensemble members compared to other efficient ensemble methods.
Verdict: Supported
Claim: In both metrics, FiLM-Ensemble achieves higher scores than the naïve, explicit deep ensemble.
Evidence type: figure
Caption: Figure 1: Diversity analysis: CIFAR-10/VGG-11 experiment. With increasing number of members, FiLM-Ensemble achieves more diverse representations. See Section 3.1 .
Context: For both metrics, higher numbers correspond to higher diversity. See Fig. 1 .
We observe that the average diversity increases with the number of ensemble members.
Verdict: Supported
Claim: Also, note that with the increasing number of members, improvements in diversity metrics for the naïve explicit deep ensemble are negligible.
Evidence type: figure
Caption: Figure 1: Diversity analysis: CIFAR-10/VGG-11 experiment. With increasing number of members, FiLM-Ensemble achieves more diverse representations. See Section 3.1 .
Context: For both metrics, higher numbers correspond to higher diversity. See Fig. 1 .
We observe that the average diversity increases with the number of ensemble members. In both metrics, FiLM-Ensemble achieves higher scores than the naïve, explicit deep ensemble. Meaning that the predictions of FiLM-Ensemble are less correlated than those of an equivalent number of networks trained independently with different random seeds.
Verdict: Supported
Claim: With regard to ECE (right subfigure), we see all methods improving with growing ensemble size M .
FiLM-Ensemble achieves better calibration then the widely used deep ensemble and MC-dropout methods, with significant margins at large ensemble sizes.
Evidence type: figure
Caption: Figure 2: Accuracy and ECE for CIFAR-10, with varying ensemble sizes, using VGG-11 as backbone.
Context: We perform several experiments on widely used benchmarks in computer vision, CIFAR-10 and CIFAR-100. As performance metrics, we plot the test set accuracy and the expected calibration error (ECE) against the ensemble size M in Fig. 2 ,
for all compared methods. In terms of accuracy (left subfigure), FiLM-Ensemble is outperformed only by the explicit deep ensemble, across all tested values of M\in\{2,4,8,16\} . These two surpass all other methods by a clear margin. Surprisingly, we observe that BatchEnsemble generally exhibits a negative correlation between ensemble size and test set accuracy, with M=4 performing best among all settings. MIMO shows very poor performance in this experimental setting in terms of accuracy (and also in ECE). We speculate that its shared backbone probably has a tendency to fragment into largely independent ensemble members of low channel depth, which lack the necessary capacity when using comparatively small networks like VGG-11 or Resnet-18.
Verdict: Supported
Claim: This is illustrated in Figure 1 where we show that a deep ResNet can easily break the topology of the input space, which is impossible for a Neural ODE.
Evidence type: figure
Caption: Figure 1: Trajectory of ResNets with 300 layers. Left: we learn x\to\frac{x^{2}}{2} , trajectories are smooth and do not intersect. Right: we learn x\to\frac{-x^{2}}{2} , trajectories are not smooth and intersect.
Context: Neural ODEs also provide a theoretical framework to study deep learning models from the continuous viewpoint, using the arsenal of ODE theory (Teh et al., 2019 ; Li et al., 2019 ; Teshima et al., 2020 ) . Importantly, they can also be seen as the continuous analog of ResNets. Indeed, consider for N an integer, the Euler scheme for solving Eq. ( 2 ) with time step \frac{1}{N} starting from x_{0} and iterating x_{n+1}=x_{n}+\frac{1}{N}\varphi_{\Theta}(x_{n},\frac{n}{N}) . Under mild assumptions on \varphi_{\Theta} , this scheme is known to converge to the true solution of Eq. ( 2 ) as N goes to +\infty . Also, if \Theta=(\theta_{n}^{N})_{i\in[N-1]} and \varphi_{\Theta}(.,\frac{n}{N})=f(.,\theta_{n}^{N}) , then the ResNet equation Eq. ( 1 ) corresponds to a Euler discretization with time step \frac{1}{N} of Eq. ( 2 ). However, for a given ResNet with fixed depth N and weights, the activations in Eq. ( 1 ) can be far from the solution of Eq. ( 2 ).
Verdict: Supported
Claim: {N} makes the network behave badly at large depth, while it still works well with our scaling \frac{1}{N} , as shown in Figure 3 (a).
Evidence type: figure
Caption: Figure 3: (a) Test accuracy on CIFAR-10 as a function of the number of blocks in each layer of the ResNet. Within each layer, weights are tied ( 3 runs ). (b) Failure of the adjoint method with a ResNet-101 on ImageNet (the approximated gradients are only used in the third layer of the network, that contains 23 blocks). (c) Relative error between the approximated gradients using adjoint method and the true gradient, whether using a ResNet or a HeunNet. Each point corresponds to one parameter.
Context: We first train a ResNet-101 (He et al., 2016a ) on CIFAR-10 and ImageNet using the same hyper-parameters. Experimental details are in appendix B and results are summarized in table 1 , showing that the explicit addition of the step size \frac{1} {N} does not affect accuracy. In strike contrast, the classical ResNet rule without the scaling \frac{1}
Verdict: Supported
Claim: First, this results shows that the architecture needs to be deep enough, because it scales in \frac{1}{N} : for instance, we fail to train a ResNet-101 (He et al., 2016a ) on the ImageNet dataset using the adjoint method on its third layer (depth 23 ), as shown in Figure 3 (b).
Evidence type: figure
Caption: Figure 3: (a) Test accuracy on CIFAR-10 as a function of the number of blocks in each layer of the ResNet. Within each layer, weights are tied ( 3 runs ). (b) Failure of the adjoint method with a ResNet-101 on ImageNet (the approximated gradients are only used in the third layer of the network, that contains 23 blocks). (c) Relative error between the approximated gradients using adjoint method and the true gradient, whether using a ResNet or a HeunNet. Each point corresponds to one parameter.
Context: In Prop. 3 we showed under assumption 2 , that is if the residuals are bounded and Lipschitz continuous with constant independent of the depth N , then the error for computing the activations backward would scale in \frac{1}{N} as well as the error for the gradients (Prop. 4 ).
Verdict: Supported
Claim: The true backpropagation gives the same curves for the ResNet and the HeunNet.
Evidence type: figure
Caption: Figure 4: Comparison of the best test errors as a function of depth when using Euler or Heun’s discretization method with or without the adjoint method.
Context: We then apply a batch norm, a ReLU and iterate relation ( 1 ) where f is a pre-activation basic block (He et al., 2016b ) . We consider the zero residual initialisation: the last batch norm of each basic block is initialized to zero. We consider different values for the depth N and notice that in this setup, the deeper our model is, the better it performs in term of test accuracy. We then compare the performance of our model using a ResNet (forward rule ( 1 )) or a HeunNet (forward rule ( 8 )). We train our networks using either the classical backpropagation or our corresponding proxys using the adjoint method (formulas ( 6 ) and ( 9 )). We display the final test accuracy (median over 5 runs) for different values of the depth N in Figure 4 .
Verdict: Supported
Claim: Despite that they outperform GCN, SGC and GAT, they still perform worse than implicit models with infinitely deep layers (i.e., EIGNN and IGNN) since finite layers for aggregations cannot effectively capture underlying dependencies along extremely long chains (e.g., with length > 30).
Evidence type: figure
Caption: (a) Results of binary classification; (b) Results of multiclass classification; Averaged accuracies with respect to the length of chains.
Context: We first consider binary classification ( c=2 ) with 20 chains of each class as in Gu et al. [ 10 ] . We conduct experiments on graphs with chains of different lengths (10 to 200 with the interval 10). The averaged accuracies with respect to the length of chains are illustrated in Figure 1(a) . In general, EIGNN and IGNN always outperform the other baselines. GCN, SGC and GAT provide similarly poor performances, which indicates that these three classic GNN models cannot effectively capture long-range dependencies. H2GCN has an even worse performance than GAT, SGC, GCN, which is caused by its ego- and neighbor-embedding separation focusing more on ego-embedding rather than distant information. APPNP, JKNet, GCNII are either designed to consider different range of neighbors or to mitigate oversmoothing [ 16 , 33 ] .
Verdict: Supported
Claim: Figure 3 shows that across different noise levels, EIGNN consistently outperforms IGNN.
Evidence type: figure
Caption: Figure 3 : Accuracy on datasets with different feature noise.
Context: On synthetic chain datasets, we add random perturbations to node features as in Wu et al. [ 31 ] .
Specifically, we add uniform noise \epsilon\sim\mathcal{U}(-\alpha,\alpha) to each node’s features for constructing noisy datasets.
Verdict: Supported
Claim: We find that the BIC value for RR-RL 2 is substantially lower compared to that of the hybrid model ( 5562.63 against 6158.91 ) when aggregated across all participants; all other models provide a less adequate fit to human choices.
Evidence type: figure
Caption: Figure 2: Model comparison results on the two-armed bandit data from Gershman ( 2018 ). (a) Bayesian information criterion (BIC) values for the aggregated data of all participants. Lower values correspond to a better fit to human behavior. (b) Posterior probabilities for each model and participant. Higher values correspond to a better fit to human behavior.
Context: Having established that different styles of exploration emerge in RR-RL 2 depending on its description length, we next set out to test how well it explains human choices. In order to do so, we conducted a Bayesian model comparison (Bishop, 2006 ) . A detailed summary of our comparison procedure is provided in Appendix B . We used the Bayesian information criterion (BIC, Schwarz, 1978 ) as an approximation to the model evidence. The resulting BIC values for each candidate model are illustrated in Figure 2 (a).
Verdict: Supported
Claim: Matching the main result of the experimental study, we find that strategic directed exploration increases with description length, while strategic random exploration remains unaffected.
Evidence type: figure
Caption: Figure 4: Illustration of strategic directed and random exploration in the horizon task. (a) Human data from Somerville et al. ( 2017 ). During adolescence, people start to engage more in strategic directed exploration, whereas strategic random exploration remains constant over time. (b) Data simulated from RR-RL 2 with varying description lengths. Like in the human data, we observe an increase in strategic directed exploration, but no change in strategic random exploration.
Context: Results: We trained RR-RL 2 with a targeted description length of \{100,200,\ldots,10000\} nats on the same distribution used in the original experimental study. Figure 4 (b) visualizes how strategic directed and random exploration change as the description length of RR-RL 2 increases.
Verdict: Supported
Claim: Ant-dir : We trained models for performing the task on an episode of 200 timesteps and found that MetODS can adapt in a few time-steps, similar to memory-based models such as RL^{2} , (Figure 3 -f) thanks to its continual adaptation mechanism.
Evidence type: figure
Caption: Figure 3 : a-b) Schemas of the Harlow and Mujoco Ant-directional locomotion task. c-d) Evolution of accumulated reward over training. In the Harlow task, we conduct an ablation study by either reducing the number of recursive iterations (S=1) or removing the trainable plasticity weights \bm{\alpha} resulting in sub-optimal policy. In Ant-dir we compare our agent training profile against MAML and RL 2 . e) We can interpret the learned policy in terms of a Hopfield energy adapting with experience. We show horizontally two reward profiles of different episodes and the energy E_{\bm{W}_{t}}(v_{1},v_{2})=-v^{T}_{1}\bm{W}_{t}v_{2} along two principal components of the vector trajectory \bm{v_{t}}. In the first episode, the error in the first presentation (red square) transforms the energy landscape which changes the agent policy, while on the other episode, the model belief does not change over time. Note the two modes for every energy map, which allows the model to handle the potential position permutation of the presented values. f) Average rewards per timestep during a single episode of the Ant-dir task.
Context: \diamond MuJoCo
Verdict: Supported
Claim: We can observe that classification errors rise with increasing noise levels and region sizes.
Evidence type: figure
Caption: Figure 3 : TTA classification errors under different levels of shift severity levels and scopes.
Context: To investigate the trade-off between the shift scope (dense vs. sparse) and severity, we simulate different levels of corruption severity and corrupted region sizes and evaluate a classical TTA method TENT [ 15 ] on these configurations. Following [ 45 ] , we inject impulse noise to the CIFAR10 dataset, with noise levels ranging from 1 to 10 to simulate various severity levels. To control the shift’s scope, we crop regions of various sizes and introduce corruption only to this region. Figure 3 displays classification error curves under various shift severity levels and region sizes.
Verdict: Supported
Claim: We plot the proportion of conserved correct predictions for the panoramic camera rotation in Fig. 7 . First, we observe that both networks struggle to a higher extent with images taken on opposite to the position, with the light source behind the camera.
Evidence type: figure
Caption: Figure 7: Counterfactual study of all sizes of ConvNext and Swin networks for panoramic 360 ∘ camera rotation.
Verdict: Supported
Claim: As shown in Figure 4(a) , we found that the performance was optimal when the \alpha value was 1.0, and increasing or decreasing \alpha resulted in decreased performance.
Evidence type: figure
Caption: (a) Impact on instruction tuning; (b) Impact on unlearning; Impact of strength coefficient \alpha on performance
Context: For instruction tuning, we tested values within the range of [0.5, 2] and evaluated them on the first 50 data points of AlpacaEval.
Verdict: Supported
Claim: As shown in Figure 4(b) , increasing the value of \alpha can prevent the model from outputting more sensitive information, but it may also lead to the loss of necessary information.
Evidence type: figure
Caption: (a) Impact on instruction tuning; (b) Impact on unlearning; Impact of strength coefficient \alpha on performance
Context: For unlearning, we found that adjusting the value of \alpha can serve as a balance between forgetting and retaining .
Verdict: Supported
Claim: With the increase of data size, the performance increases.
Evidence type: figure
Caption: Figure 6: The Cumulative Reward of Anonymous in Validation Dataset. The performance of DAgger models is related to the number of iteration rounds.
Context: We show the performance of models with different settings in the validation set in Figure 6 . With the same data size, the model using DAgger performs better.
Verdict: Supported
Claim: We also see that the representations we learn lead to competitive results with respect to BERT and RoBERTa despite Sentence-BERT not being suited to classification.
Evidence type: table
Caption: Table 4: Classification results in terms of F1 score on the eight data sets.
Context: Similar to the clustering setting, we compare results from a Logistic Regression (LR) applied on the original sentence embeddings with and without the filtering operation we introduced. We also use fine-tuned BERT and RoBERTa (2 epochs) as baselines; we use the base versions due to computational restrictions. We report the results in Table 4 . We see that this operation leads to better performances on the classification task on the majority of the datasets with respect to the filterless Sentence-BERT but this performance increase is not as pronounced as for the clustering task.
Verdict: Supported
Claim: Across 57 translation directions, chrF2 improves by 1.3 (M2M-100) and 1.1 (SMaLL-100) points with source-contrastive decoding.
Evidence type: table
Caption: Table 1: Automatic evaluation results. Averages over different sets of translation directions.
Context: We report results using source-contrastive decoding ( C_{src} ), and combining source-contrastive and language-contrastive decoding ( C_{src+lang} ) in Table 1 . 10 10 10 See Appendix A for full results.
Verdict: Supported
Claim: Language-contrastive decoding brings additional gains of 0.4 (M2M-100) and 0.2 (SMaLL-100) points.
Evidence type: table
Caption: Table 1: Automatic evaluation results. Averages over different sets of translation directions.
Context: We report results using source-contrastive decoding ( C_{src} ), and combining source-contrastive and language-contrastive decoding ( C_{src+lang} ) in Table 1 . 10 10 10 See Appendix A for full results. Across 57 translation directions, chrF2 improves by 1.3 (M2M-100) and 1.1 (SMaLL-100) points with source-contrastive decoding.
Verdict: Supported
Claim: The remaining prompts types ( Discourse, Mask ) are comparable to Corpus .
Evidence type: table
Caption: Table 3: Mean perplexity (PPL) calculated using the GPT-J-6B model. Only the strings enclosed in square brackets are considered during calculation in order to provide a fair comparison with similar token length. For Corpus, PPL is calculated using the provided gold completion.
Context: To further analyze the outputs, we calculate the perplexity (PPL) of the generated predictions to determine their plausibility (Wilcox et al., 2020 ) . Here, we choose the model with the best \textit{WHR}_{5} score on the MKR-NQ benchmark, and calculate the mean perplexity over all queries for each prompt type (5 completions for each query). PPL is calculated as the exponentiated average negative log-likelihood of a sequence, with exponent base e . As a point of reference, we calculated the average perplexity of the provided completion of the original non-negated dataset (denoted Corpus ). From the reported perplexities ( Table 3 ), we can see that Default output are the most plausible (with PPL markedly lower than Corpus ), while Contrasting is the least natural.
Verdict: Supported
Claim: It also has an equivalent amount or more for almost any ethnicity, except for Asians.
Evidence type: table
Caption: Table 7: Percentage of questions with changed answers between the biomedical and generic model as compared to a question with no demographic information about the patient. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation.
Context: Similar to our analysis between QAGNN and BioLinkBert above, our analysis between the biomedical and generic models can be split into the amount of answers and accuracy that changes when the dimensions change. From Table 7 it is visible that the generic transformer has more than double the amount of answers change for each gender.
Verdict: Supported
Claim: The reason is that the model struggles in distinguishing between refutes and NEI claims in these datasets, as reflected by Table 5 .
Evidence type: table
Caption: Table 5: Class-wise F1 of 3-class fact verification for the zero-shot generalization setup (left) and the in-domain training setup (right). S: supports; R: refutes; N: NEI.
Context: Many works Jiang et al. ( 2020 ); Saakyan et al. ( 2021 ) do not consider NEI claims due to their ambiguity. To explore whether our previous observations also hold for the task of binary fact verification , we evaluate the generalization results for all 11 datasets using only the supports and refutes claims for training and evaluation, shown in Table 3 . In this setting, artificial claims also generalize well to natural claims in other domains. In 6 of the 7 datasets with natural claims, the best generalization score is from a model trained on artificial claims. This also holds for the evidence length: datasets with sentence-level evidence tend to generalize better than document-level datasets. Finally, compared with the three-class result in Table 2 , generalization improves a lot on Climate-FEVER, SciFact, and PubHealth.
Verdict: Supported