alabnii/sciclaimeval-shared-task huggingface dataset:
Claim: We see that FuncMol slightly improves VoxMol and both models perform worse compared to the equivariant point-cloud based baselines.
Evidence type: table
Evidence caption: Table 1: QM9 results w.r.t. test set for 10000 samples per model. \uparrow / \downarrow indicate that higher/lower numbers are better. The row data are randomly sampled molecules from the validation set. We report 1-sigma error bars over 3 sampling runs.
Context: Table 1 report the metrics described in Section ˜ 5.1 .
Verdict: Supported
Claim: We note that sampling time of FuncMol is an order of magnitude better than baselines.
Evidence type: table
Evidence caption: Table 1: QM9 results w.r.t. test set for 10000 samples per model. \uparrow / \downarrow indicate that higher/lower numbers are better. The row data are randomly sampled molecules from the validation set. We report 1-sigma error bars over 3 sampling runs.
Context: Table 1 report the metrics described in Section ˜ 5.1 . We see that FuncMol slightly improves VoxMol and both models perform worse compared to the equivariant point-cloud based baselines.
Verdict: Supported
Claim: Thereafter, Swin becomes comparatively much more robust as occlusion becomes severe.
Evidence type: figure
Evidence caption: Figure 3: Random patch drop occlusion study on ConvNext and Swin networks on the ImageNet-1k validation set. Swin Transformers are slightly more robust to this type of artificial occlusion than ConvNext networks when the information loss is small, although they become comparatively much stronger as the information loss is increased.
Context: For (1), we provide evidence that this difference between ResNets and DeiTs is not due to the nature of transformer vs. conv layers. In order to do so, we re-run the random patch drop experiments in [ 47 ] on all pairs of ConvNext and Swin networks. We present results for different levels of information loss in Fig. 3 . For all networks we observe a drop in performance as information loss increases. We show that ConvNext networks do not suffer the same critical failure mode that DenseNet121, ResNet50, SqueezeNet and VGG19 exhibit in Naseer et al. [ 47 ] . The top accuracy of these four classic ConvNets collapses to half after about 10% of patches are occluded and to 0 after about 40% information loss. ConvNext shows that it is much more resistant, conserving a large part of its accuracy even after 50% information loss. Next, we observe that ConvNext networks are slightly less robust than Swin Transformers for low amounts of information loss (under 50%).
Verdict: Supported
Claim: For Base-22k and Large, the difference becomes large after the scale multiplier is under 0.6.
Evidence type: figure
Evidence caption: Figure 5: Counterfactual study of all sizes of ConvNext and Swin networks for object scale.
Context: In order to compare the robustness of ConvNext and Swin networks to scale, we first generate images of all main object models with unit object scale. Although objects have different volume in this state (e.g. microwaves are larger than spatulas), they take a fair amount of space in the frame and don’t present much of a challenge to large architectures. Specifically, top-5 accuracies for Swin-L and ConvNext-L are 90.5% and 93.8% respectively. We then perform counterfactual simulation testing by plotting the stability of correct predictions when the object scale multiplier is reduced from 1. We show these results in Fig. 5 . We observe that for Small, Base-22k and Large-22k networks, ConvNext vastly outperforms Swin for smaller objects.
Verdict: Supported
Claim: With task-specific encoders (right), the cross-task class centroids are further apart than those with one global encoder (left).
Evidence type: figure
Evidence caption: Figure 3 : Need for task-specific distribution encoders: Cosine distance between the centroids of class-specific latent variables produced without ( left ) and with ( right ) the use of task-specific encoders on CIFAR100 (10 tasks, 10 classes per task). Centroids of class-specific latent variables are more separable on the right.
Context: where \{z^{t}_{i}\}_{m=1}^{M} are the M task-specific MC samples. Task-specific encoders help us sample latent variables that are more discriminative across tasks. This is depicted in Fig. 3 using the cosine distance between the embeddings of class-specific samples.
Verdict: Supported
Claim: As depicted in the figure, AgA with sign selection is aligned towards the fastest update direction, resulting in AgA progressing approximately 6 steps ahead of AgA without Sign at the end of the trajectory.
Evidence type: figure
Evidence caption: (a) Cleanup: AgA with different \lambda and AgA/Sign; (b) Cleanup: comparison between AgA and baselines; (c) Harvest: AgA with different \lambda and AgA/Sign; (d) Cleanup: comparison between AgA and baselines; Comparison of Episodic Social Welfare: our proposed AgA methods along with baselines on Cleanup (figure a and b) and Harvest (figure c and d) tasks. Figure (a) and (c) provide an illustration of varying alignment parameter \lambda values in AgA, as well as AgA sans sign selection (AgA/Sign), as applied to Cleanup and Harvest tasks respectively. Subfigures (b) and (d) exhibit the social welfare obtained by AgA utilizing the optimal \lambda value, in comparison to those secured by Simul-Ind, Simal-Co, CGA, SGA and SVO. Bold lines represent the average of the social welfare across three seeds, with the encompassing shaded areas denoting the standard deviation.
Context: In addition, Fig. 3 presents a critical comparison between the trajectories of AgA (in red) and AgA without sign alignment (AgA w/o Sign, depicted in purple), as outlined in Corollary 4.3 . This side-by-side comparison covers 40 steps and features highlighted points every tenth step. It provides a visual illustration of the efficiency introduced by sign alignment starting from the 14th step in the trajectories of AgA and AgA without sign alignment. Remarkably, norm gradients are represented by blue arrows, indicating the direction of the fastest updates.
Verdict: Supported
Claim: Approximated gradients, however, lead to a large test error at small depth, but give the same performance at large depth, hence confirming our results in Prop. 4 and 6 .
Evidence type: figure
Evidence caption: Figure 4: Comparison of the best test errors as a function of depth when using Euler or Heun’s discretization method with or without the adjoint method.
Context: We then apply a batch norm, a ReLU and iterate relation ( 1 ) where f is a pre-activation basic block (He et al., 2016b ) . We consider the zero residual initialisation: the last batch norm of each basic block is initialized to zero. We consider different values for the depth N and notice that in this setup, the deeper our model is, the better it performs in term of test accuracy. We then compare the performance of our model using a ResNet (forward rule ( 1 )) or a HeunNet (forward rule ( 8 )). We train our networks using either the classical backpropagation or our corresponding proxys using the adjoint method (formulas ( 6 ) and ( 9 )). We display the final test accuracy (median over 5 runs) for different values of the depth N in Figure 4 . The true backpropagation gives the same curves for the ResNet and the HeunNet.
Verdict: Supported
Claim: The underfitting delta model had a win rate of 54 when \alpha was 1.0, which increased to 64 when \alpha was 2.0.
Evidence type: figure
Evidence caption: (a) Impact on instruction tuning; (b) Impact on unlearning; Impact of strength coefficient \alpha on performance
Context: For instruction tuning, we tested values within the range of [0.5, 2] and evaluated them on the first 50 data points of AlpacaEval. As shown in Figure 4(a) , we found that the performance was optimal when the \alpha value was 1.0, and increasing or decreasing \alpha resulted in decreased performance. This is similar to causing the delta model to overfit or underfit, so we explored whether adjusting the \alpha value during inference could counteract the overfitting and underfitting during training. Specifically, the delta model performed best after training for four epochs, and we selected the checkpoint after training for two epochs as the underfitting scenario and the checkpoint after training for eight epochs as the overfitting scenario. We found that adjusting the \alpha value could counteract the overfitting or underfitting during training.
Verdict: Supported
Claim: We also see that acquiring based on the lookahead posteriors incurs only a minimal increase in error compared to the sampling-based method, despite being far more efficient.
Evidence type: figure
Evidence caption: Figure 7: Average RMSE of reconstructions for greedy active feature acquisition.
Context: Figure 7 presents the root-mean-square error, averaged across the test instances, when imputing \mathbf{x}_{u} with different numbers of acquired features. We see that our models are able to achieve lower error than EDDI.
Verdict: Supported
Claim: Better sample selection due to jittering subsequently leads to significantly better performance in regression (Fig. 4 (c)).
Evidence type: figure
Evidence caption: Figure 4: Jittering analysis. (a) When trained without jittering, feature extractors easily overfit the noisy training data (yellow-shaded region), while jittering-regularized feature extractors robustly learn from the noisy training data. (b) Overfitted feature extractors (yellow-shaded region) on noisy samples increase their likelihood, leading to a higher selection rate and ERR.
It exhibits nearly twice higher ERR (a lower value is better).
(c) Most importantly, jittering regularization improves performance in regression.
Context: Fig. 4 (a) shows that with jittering, the feature extractor exhibits higher accuracy on the clean test data due to its regularization effect. In the sample selection stage (Fig. 4 (b)),
the feature extractor trained without jittering easily overfits the noise, resulting in over-selection and higher ERR (§ 4.2 ). In contrast, the jittered feature extractor achieves a relatively low selection rate with halved ERR, indicating that the noisier samples are filtered out.
Verdict: Supported
Claim: ConFrag achieves the lowest ERR while maintaining above-average selection rates, resulting in the best MRAE.
Evidence type: figure
Evidence caption: Figure 5: Selection/ERR/MRAE comparison between ConFrag and strong baselines of CNLCU-H, BMM, DY-S, AUX and Selfie on IMDB-Clean-B. We exclude the performance during the warm-up.
Context: Selection/ERR/MRAE comparison. Fig. 5 compares ConFrag to five selection and refurbishment baselines of CNLCU-H, BMM, DY-S, AUX, Selfie on IMDB-Clean-B using the selection rate, ERR, and MRAE. Ideally, a model should attain a high selection rate and a low ERR. It is worth noting that the relative importance of ERR and selection rate may vary depending on the dataset and the task.
Verdict: Supported
Claim: The contrastive fragment pairing demonstrates superior performance to other pairing methods.
Evidence type: figure
Evidence caption: Figure 6: Analysis with 40% symmetric noise. (a) Comparison between the proposed contrastive pairing and other pairings on IMDB-Clean-B.
(b) Comparison between fragment numbers on SHIFT15M-B and IMDB-Clean-B.
Context: Fragment pairing. Fig. 6 (a) compares contrastive pairing to alternative pairings
using MRAE as a metric.
Verdict: Supported
Claim: In Fig. 5(a) , the accuracy is poorer in range [1,10] , grows in range [10,20] , and saturates thereafter.
Evidence type: figure
Evidence caption: (a) Accuracy-runtime tradeoff; (b) Parameter count comparison; Ablations on CIFAR100 showing: (a) performance trade-off with the number of MC samples M , (b) the number of trainable parameters in different finetuning methods.
Context: We vary the number of MC samples M from 1 to 50.
Verdict: Supported
Claim: On the other hand, increasing training data appears to have only a minor influence on robustness.
Evidence type: figure
Evidence caption: (a); (b); (c); NWER on non-adversarial and adversarial perturbations plotted against the number of parameters (a & c) and hours of training data used to train the DNN (b). Figures (a) and (b) plot all the models, while (c) plots three families of models (indicated by the color). The models within the family share similar architectures, and training datasets.
Context: To determine if the prevailing practice of training DNNs with more parameters on larger datasets is yielding improvements in robustness, we plot NWER against the number of model parameters and the size of the training data of all the candidate models. These plots are shown in Figures 4(a) and 4(b) . We note that increasing model size is correlated with improved robustness (lower NWER), however, the effect is more inconsistent and weaker for adversarial perturbations. To further isolate the impact of the model size we control the architecture and training data and plot the NWER of models from the same family in Figure 4(c) , which have similar architectures and training datasets. We note that larger models are more robust in the Whisper and Wav2Vec-2.0 families, but, surprisingly, not in the HuBert family.
Verdict: Supported
Claim: An intriguing observation is that the decodability of both attributes decreases as the depth of the neural network increases, which is also observed in [ 10 ] .
Evidence type: figure
Evidence caption: (a); (b); Exploring the Effect of depth modulation: (a) illustrates how the linear decodability of features decreases as neural network depth increases, while (b) dives into the training dynamics of MLPs with varying depths under ERM.
Context: We employ the concept of feature decodability to assess the extent to which the specific features of a given dataset can be reliably decoded from the models with varying depths. Hermann et al . [ 10 ] demonstrated that the visual features can be decoded from the higher layers of untrained models. Additionally, they observed that the feature decodability from an untrained model has a significant impact in determining which features are emphasized and suppressed during the model training. Following their approach, we specifically focus on assessing the decodability of bias and core attributes from the penultimate layer of untrained models. In order to evaluate the decodability of an attribute in a dataset, we train a decoder to map the activations from the penultimate layer of a frozen, untrained model to attribute labels. The decoder comprises a single linear layer followed by a softmax activation function. The decoder is trained using an unbiased validation set associated with the dataset, where each instance is labeled according to the attribute under consideration. Subsequently, the linear decodability of the attribute, measured in accuracy, is reported on the unbiased test set. We investigate the decodability of digit and color attributes in the CMNIST dataset from MLP models with varying depths, including 3, 4, and 5 layers, and the results are depicted in Fig. 2(a) .
Verdict: Supported
Claim: Hence, the disparity in decodability between color and digit attributes becomes significantly more pronounced in a 5-layer MLP in comparison to a 3-layer MLP.
Evidence type: figure
Evidence caption: (a); (b); Exploring the Effect of depth modulation: (a) illustrates how the linear decodability of features decreases as neural network depth increases, while (b) dives into the training dynamics of MLPs with varying depths under ERM.
Context: As observed in Fig. 2(b) , the initial phases of training for both networks emphasize color attribute (since bias is easy to learn), resulting in a notable enhancement in the decodability of color in both models. Also, as the training progresses, the decodability of the digit is higher in a 3-layer model when compared to a 5-layer model.
Verdict: Supported
Claim: As depicted in Fig. 4 , prior to the training, the deep branch demonstrates lower linear decodability for both the digit (core attribute) and color (bias attribute) in comparison to the linear decodability observed in the shallow branch.
Evidence type: figure
Evidence caption: Figure 4 : Training dynamics of DeNetDM during initial stages of training monitored in terms of linear decodability of bias attribute (color) and core attribute (digit).
Context: In Sec. 3.1 , we discussed the variability in the linear decodability at various depths and how the disparity in linear decodability between biased attributes and target attributes forms a significant motivation for debiasing. To further validate this intuition and identify the elements contributing to its effectiveness, we delve into the analysis of the training dynamics of DeNetDM during the initial training stages. We consider the training of Colored MNIST with 1% skewness due to its simplicity and ease of analysis. Figure 4 shows how linear decodability of attributes varies across different branches of DeNetDM during training.
Verdict: Supported
Claim: In particular, LISA requires much less activation memory consumption than LoRA since it does not introduce additional parameters brought by the adaptor.
Evidence type: figure
Evidence caption: Figure 3: GPU memory consumption of LLaMA-2-7B with different methods and batch size 1.
Context: In Figure 3 , it is worth noticing that the memory reduction in LISA allows LLaMA-2-7B to be trained on a single RTX4090 (24GB) GPU, which makes high-quality fine-tuning affordable even on a laptop computer.
Verdict: Supported
Claim: As shown in Figure 4 , LISA provides almost 2.9\times speedup when compared with full-parameter tuning, and \sim 1.5\times speedup against LoRA, partially due to the removal of adaptor structures.
Evidence type: figure
Evidence caption: Figure 4: Single-iteration time cost of LLaMA-2-7B with different methods and batch size 1.
Context: On top of that, a reduction in memory footprint from LISA also leads to an acceleration in speed.
Verdict: Supported
Claim: Specifically, Figure 5 highlights the model’s performance in various aspects, particularly LISA’s superiority over all methods in aspects of Writing, Roleplay, and STEM.
Evidence type: figure
Evidence caption: Figure 5: Different aspects of LLaMA-2-70B model on MT-Bench.
Context: As shown in Table 5 , LISA consistently produces better or on-par performance when compared with LoRA. Furthermore, in instruction-tuning tasks, LISA again surpasses full-parameter training, rendering it a competitive method for this setting.
Verdict: Supported
Claim: On top of that, LISA displays consistently higher performance than LoRA on all subtasks, underscoring LISA’s effectiveness across diverse tasks.
Evidence type: figure
Evidence caption: Figure 5: Different aspects of LLaMA-2-70B model on MT-Bench.
Context: As shown in Table 5 , LISA consistently produces better or on-par performance when compared with LoRA. Furthermore, in instruction-tuning tasks, LISA again surpasses full-parameter training, rendering it a competitive method for this setting. Specifically, Figure 5 highlights the model’s performance in various aspects, particularly LISA’s superiority over all methods in aspects of Writing, Roleplay, and STEM.
Verdict: Supported
Claim: The chart also contrasts the yellow LoRA line with the purple Vanilla line, revealing that in large models like the 70B, LoRA does not perform as well as expected, showing only marginal improvements on specific aspects.
Evidence type: figure
Evidence caption: Figure 5: Different aspects of LLaMA-2-70B model on MT-Bench.
Context: As shown in Table 5 , LISA consistently produces better or on-par performance when compared with LoRA. Furthermore, in instruction-tuning tasks, LISA again surpasses full-parameter training, rendering it a competitive method for this setting. Specifically, Figure 5 highlights the model’s performance in various aspects, particularly LISA’s superiority over all methods in aspects of Writing, Roleplay, and STEM. On top of that, LISA displays consistently higher performance than LoRA on all subtasks, underscoring LISA’s effectiveness across diverse tasks.
Verdict: Supported
Claim: It is also intriguing to observe from Figure 5 that Vanilla LLaMA-2-70B excelled in Writing, but full-parameter fine-tuning led to a decline in these areas, a phenomenon known as the “Alignment Tax” (Ouyang et al., 2022 ) .
Evidence type: figure
Evidence caption: Figure 5: Different aspects of LLaMA-2-70B model on MT-Bench.
Verdict: Supported
Claim: Notably, AgA with \lambda=0.1 underperforms, even when contrasted with AgA’s ablation study.
Evidence type: figure
Evidence caption: (a) MMM2: AgA with various \lambda and AgA/Sign; (b) MMM2: comparison between AgA and baselines; The comparison of win rates on the mixed-motive MMM2 map of SMAC.
The left figure shows results of varying alignment parameter \lambda values in AgA, as well as AgA sans sign selection (AgA/Sign),
The right figure compare the win rate of AgA (\lambda=100) with baselines.
The bold lines illustrate the mean of the win rate over 3 seeds, while the encompassing shaded regions depict the standard deviation.
Context: Results. Fig. 5(a) compares average win rates for various \lambda parameters in AgA, also encompassing AgA without sign selection, as stated in Corollary 4.3 . The results suggest comparable performance for \lambda=1 and 100 , with a lower value of 0.1 trailing noticeably.
Verdict: Supported
Claim: However, in our exploratory experiment on Mixtral 8x7B [ 17 ] , we find simply raising the number of activated experts (blue lines in Figure 1 ) does not lead to stable improvements and may even hurt performance on different tasks.
Evidence type: figure
Evidence caption: Figure 1 : Performance comparison between increasing the value of top- k ( i.e. , ensemble routing) and SCMoE. SCMoE surpasses the performance of ensemble routing across various benchmarks.
Context: In this paper, we investigate the impact of unchosen experts 1 1 1 Unchosen experts refer to the experts not selected by default routing ( e.g. , top-2 routing in Mixtral 8x7B). on the performance of MoE models and explore their suitable usage. A direct hypothesis is that incorporating more experts improves MoE models and helps solve more difficult problems [ 3 ; 36 ; 14 ] .
Verdict: Supported
Claim: For instance, top-3 routing yields best results on GSM8K, while top-4 routing achieves the highest accuracy on HumanEval and StrategyQA.
Evidence type: figure
Evidence caption: Figure 1 : Performance comparison between increasing the value of top- k ( i.e. , ensemble routing) and SCMoE. SCMoE surpasses the performance of ensemble routing across various benchmarks.
Context: As revealed by Figure 1 , using default top-2 routing is not optimal for all tasks.
Verdict: Supported
Claim: Furthermore, SCMoE can enhance the major@20 accuracy from 75.59 to 78.31 (+2.72) on GSM8K.
Evidence type: figure
Evidence caption: Figure 5 : Experimental results on combining SCMoE with self-consistency on GSM8K using Mixtral 8x7B.
Context: Using self-consistency [ 35 ] for multiple sampling and taking a majority vote to determine the final answer is a common method to improve LLMs’ performance. Therefore, we explore whether SCMoE can combined with self-consistency. For vanilla self-consistency, we use temperature sampling with temperature \tau=0.7 to reach the best baseline performance. For self-consistency with SCMoE, we simply employ \beta=0.5 , rank-3 routing as weak activation, according to the best hyperparameters setting from Table 8 . It is worth noting that since SCMoE already has a mask \alpha=0.1 to limit the sampling range of the vocabulary, we do not perform any additional temperature processing on the final logits. As shown in Figure 5 ,
SCMoE (67.94) yields comparable results with major@5 (66.87).
Verdict: Supported