alabnii/sciclaimeval-shared-task huggingface dataset:
Claim: Table 1 shows that our probabilistic inference module consistently outperforms its deterministic counterpart in terms of Avg and Last accuracy.
Evidence type: table
Evidence caption: Table 1 : Performance comparison of different methods averaged over three runs. Best scores are in bold . Second best scores are in blue . The results for L2P, DualPrompt, and PROOF are taken from [ 92 ] . See App. Table 14 for std. dev. scores.
Context: To understand our probabilistic inference modules further, we examine their performance against the deterministic variant of ours (Ours w/o VI).
Verdict: Supported
Claim: If we compare the two-layer network results from GeoLIP and Sampling in Table 1 , which is a lower bound of true Lipschitz constant, the ratio is within 1.783 .
Evidence type: table
Evidence caption: Table 1: \ell_{\infty} -FGL estimation of various methods: DGeoLIP and NGeoLIP induce the same values on two layer networks. DGeoLIP always produces tighter estimations than LiPopt and MP do.
Context: We have also shown that the two-layer network \ell_{\infty} -FGL estimation from GeoLIP has a theoretical guarantee with the approximation factor K_{G}<1.783 ( Theorem 3.3 ).
Verdict: Supported
Claim: Table 2 shows that the adversary applies lower dropout probabilities to less informative words such as ‘unknown’ tokens that replace all out-of-dictionary words, and so offer little information about the next word.
Evidence type: table
Evidence caption: Table 2 : Analysis of the Adversary. For selected SNLI (upper) and Yahoo (lower) sentences, word dropout scores are from a trained adversary and normalized per sentence. Darker colouring indicates a higher dropout probability. Boxed words are those selected to be dropped. <sos> = start of the sequence token; <eos> = end of the sequence token; _unk = unknown token.
Context: We obtain further insight into what the adversary learns by analyzing the word dropout scores for different sentences.
Verdict: Supported
Claim: We also show that adversarial training learns a useful generative model with meaningful latent space by interpolating between sentences (Table 3 ).
Evidence type: table
Evidence caption: Table 3 : Sentence interpolation (Yelp dataset). Representations of two sentences (top, bottom) are obtained by feeding them through an adversarially trained VAE encoder. Three linearly interpolated representations are passed to the VAE decoder and sentences generated by greedy sampling (middle) .
Context: We obtain further insight into what the adversary learns by analyzing the word dropout scores for different sentences. Table 2 shows that the adversary applies lower dropout probabilities to less informative words such as ‘unknown’ tokens that replace all out-of-dictionary words, and so offer little information about the next word. Depending on the data semantics, the adversary selects different types of words: for SNLI, verbs tend to be picked that explain the activity, e.g. working and participating ; for Yahoo, words are identified that carry question semantics, e.g. what , how , if , and when . Figure 4 (a) shows a quantitative analysis of dropout saliency map across different part-of-speech (POS) tags. Verb (verb), interjections (intj), and nouns (noun) have higher saliency scores (higher chances of being dropped) compared to punctuation (punc), determiners (det), and the start (sos) and end tokens (eos) which are relatively easier to predict given previous words.
Verdict: Supported
Claim: Lastly, we note that using larger values of \gamma naively does not guarantee better test performance: one has to fine tune the parameters \beta , \mu , \sigma , \eta appropriately to achieve favorable trade-off between training stability and test performance.
Evidence type: table
Evidence caption: Table 1: Shallow neural nets trained on the airfoil data set. The results in parenthesis are achieved with the variant ( 11 ).
All the results are averaged over 5 models trained with different seed values.
Context: For the training, we use a fully connected shallow neural network of width 16 with ReLU activation and train for 3000 epochs with the learning rate \eta=0.1 , using mean square error (MSE) as the loss and choosing \beta=0.5 . Table 1 reports the average root MSE (RMSE) and the RMSE gap (defined as test RMSE - train RMSE) evaluated for models that are trained with 5 different seed values for this task. We can see that MPGD leads to both lower test RMSE and RMSE gap when compared with vanilla GD (baseline) and GD with uncorrelated Gaussian perturbations (see the results not in parenthesis in Table 1 ; here \mu=0.01 , \sigma=0.02 ). Using the form of the perturbations in ( 11 ) instead can also give lower RMSE gap (see the results in parenthesis in Table 1 ; here \sigma=\mu=0.01 ). Overall these results support our generalization theory for MPGD.
Verdict: Supported
Claim: Table 2 shows that MPGD can lead to better test performance and lower accuracy gap when compared to the baseline (full batch GD) and Gaussian noise-perturbed GD.
Evidence type: table
Evidence caption: Table 2: ResNet-18 trained on CIFAR-10 for 1000 epochs. Here, accuracy gap = training accuracy - validation accuracy. The results in parenthesis are achieved with the variant of MPGD ( 11 ). All the results are averaged over 5 models trained with different seed values.
Verdict: Supported
Claim: We also show that our performance boost does not come from the additional training dataset of ActivityNet in Table 3 .
Evidence type: table
Evidence caption: Table 3: Ablation experiments. We investigate the effectiveness of each component of our method as well as compare to vanilla multi-dataset training method.
“Vanilla” means using cross entropy (CE) loss in training.
“w/o informative los” means using CE and projection loss.
The numbers are top-1/top-5 accuracy, respectively.
Training data:
(e) Kinetics-400;
(f) SSv2;
(g) MiT;
(h) ActivityNet.
Context: We then compare our method with state-of-the-art on these datasets. We train a higher resolution model with larger spatial inputs (312p) and achieves better performance compared to recent multi-dataset training methods, CoVER [ 59 ] and PolyVit [ 38 ] , on Kinetics-400, and significantly better on MiT and SSv2, as shown in Table 1 . Note that our model does not use any image training datasets, and our model computation cost is only a fraction of the baselines.
Verdict: Supported
Claim: Also, longer training schedule does not lead to better performance on some datasets, including SSv2, suggesting vanilla multi-dataset training is unstable.
Evidence type: table
Evidence caption: Table 3: Ablation experiments. We investigate the effectiveness of each component of our method as well as compare to vanilla multi-dataset training method.
“Vanilla” means using cross entropy (CE) loss in training.
“w/o informative los” means using CE and projection loss.
The numbers are top-1/top-5 accuracy, respectively.
Training data:
(e) Kinetics-400;
(f) SSv2;
(g) MiT;
(h) ActivityNet.
Context: Does our proposed robust loss help? We compare our model training with vanilla multi-dataset training, where multiple classification heads are attached to the same backbone and the model is trained simply with cross-entropy loss. The vanilla model is trained from a K400 checkpoint as ours. As shown in Table 3 , we try training the vanilla model with both the same training schedule as ours and a 4x longer schedule. As we see, there is a significant gap between the overall performance of the vanilla model and ours, validating the efficacy of our proposed method.
Verdict: Supported
Claim: In terms of performance on ActivityNet, we observe that both training methods achieve good results, which might be because ActivityNet classes are highly overlapped with Kinetics-400 (65 out of 200).
Evidence type: table
Evidence caption: Table 3: Ablation experiments. We investigate the effectiveness of each component of our method as well as compare to vanilla multi-dataset training method.
“Vanilla” means using cross entropy (CE) loss in training.
“w/o informative los” means using CE and projection loss.
The numbers are top-1/top-5 accuracy, respectively.
Training data:
(e) Kinetics-400;
(f) SSv2;
(g) MiT;
(h) ActivityNet.
Context: Does our proposed robust loss help? We compare our model training with vanilla multi-dataset training, where multiple classification heads are attached to the same backbone and the model is trained simply with cross-entropy loss. The vanilla model is trained from a K400 checkpoint as ours. As shown in Table 3 , we try training the vanilla model with both the same training schedule as ours and a 4x longer schedule. As we see, there is a significant gap between the overall performance of the vanilla model and ours, validating the efficacy of our proposed method. Also, longer training schedule does not lead to better performance on some datasets, including SSv2, suggesting vanilla multi-dataset training is unstable.
Verdict: Supported
Claim: As shown in Table 3 , the performance on MiT and SSv2 suffers by a large margin, indicating that the projection design helps boost training by better utilizing multi-dataset information.
Evidence type: table
Evidence caption: Table 3: Ablation experiments. We investigate the effectiveness of each component of our method as well as compare to vanilla multi-dataset training method.
“Vanilla” means using cross entropy (CE) loss in training.
“w/o informative los” means using CE and projection loss.
The numbers are top-1/top-5 accuracy, respectively.
Training data:
(e) Kinetics-400;
(f) SSv2;
(g) MiT;
(h) ActivityNet.
Context: How important is the projection loss? We then experiment with removing the projection heads (Section 3.2 ) during multi-dataset training. The model is trained with the original cross-entropy loss and the informative loss.
Verdict: Supported
Claim: Finally, we show that ViTCA does not dramatically suffer when no explicit positioning is used—in contrast to typical Transformer-based models—as cells are still able to localize themselves by relying on their stored hidden information.
Evidence type: table
Evidence caption: Quantitative ablation for denoising autoencoding with ViTCA (unless otherwise stated via prefix) on CelebA [ 47 ] . Boldface and underlining denote best and second best results. Italicized items denote baseline configuration settings. † Trained with gradient checkpointing [ 44 ] , which slightly alters round-off error during backpropagation, resulting in slight variations of results compared to training without checkpointing. See Appendix A.2 .
Context: As shown in Tab. 2 , ViTCA benefits from an increase to most CA and Transformer-centric parameters, at the cost of computational complexity and/or an increase in parameter count. A noticeable decrease in performance is observed when embed size d\!=\!512 , most likely due to the vast increase in parameter count necessitating more training. In the original ViT, multiple encoding blocks were needed before the model could exhibit performance equivalent to their baseline CNN [ 13 ] , as verified in our ablation with our ViT. However, for ViTCA we notice an inverse relationship of the effect of Transformer depth, causing a divergence in cell state. It is not clear why this is the case, as we have observed that the LN layers and overflow losses otherwise encourage a contractive F_{\theta} . This is an investigation we leave for future work. Despite the benefits of increasing h , we use h\!=\!4 for our baseline to optimize runtime performance.
Verdict: Supported
Claim: Note that resistance against the attack to both in- and out-sets is much harder than other cases since the perturbation budget has effectively been doubled.
Evidence type: table
Evidence caption: Table 1: OOD detection AUROC under attack with \epsilon=\frac{8}{255} for various methods trained with CIFAR-10 or CIFAR-100 as the closed set. A clean evaluation is one where no attack is made on the data, whereas an in/out evaluation means that the corresponding data is attacked. The best and second-best results are distinguished with bold and underlined text for each column.
Context: OOD detection under adversarial attack: To perform a comprehensive study, AUROC is computed in four different settings for each method. First, the standard OOD detection without any attack is conducted (Clean). Next, either in- or out-datasets are attacked (In/Out). Finally, both the in- and out-sets are attacked (In and Out).
Verdict: Supported
Claim: Moreover, we check the effect of attacking the generated images, which leads to a lower AUROC score.
Evidence type: table
Evidence caption: Table 3: Ablation study on our method. Other choices for the feature extractor, training method, and generated data attacking are tested on CIFAR-10, but none of them is as effective as the setting used in the ATD.
Context: Ablation study: ATD uses HAT as the feature extractor and performs adversarial training on open and closed sets to robustify the discriminator. As an ablation study using CIFAR-10 as the in-distribution data, we replace HAT with a standard trained model to check its effectiveness. Also, in another experiment, we replace the discriminator adversarial training with the standard training. The results in Table 3 demonstrate that these settings are not as effective as ATD in achieving a robust detection model. As another ablation, we consider removing the feature extractor and generating images instead of features. Based on the results, the trained discriminator with this method is also not as robust as the ATD.
Verdict: Supported
Claim: VeLoRA lowers the memory requirement of SSF [ 26 ] by 16% with only a minor degradation (0.1 pp ) in accuracy.
Evidence type: table
Evidence caption: Results on a subset of the VTAB-1k benchmark. All methods use a ViT-Base-224/16 model pre-trained on ImageNet-21k. The batch sizes and ranks are the same across all tasks.
Context: We conduct experiments evaluating the performance of VeLoRA for full-tuning and how it complements other PEFT methods. In Tab. 1 we reproduce a large set of results for LoRA [ 17 ] , SSF [ 26 ] , and Hydra [ 20 ] on a subset of the VTAB-1K benchmark, where the sub-token size for each experiment is given in the Appendix. Unlike what is more common in the PEFT literature [ 19 , 20 ] , we do not perform any task-specific hyperparameter tuning that would change the memory, such as batch size and rank, and to also avoid any potential overfitting to the specific task. For all experiments we used the authors provided implementations for the adapters and integrated them into the same training framework for a fair comparison. We observe that VeLoRA improves the performance compared to full-tuning by 1.5 percentage points ( pp ), while lowering the memory requirements. We also observe that when combined with PEFT methods, VeLoRA can come with improvement in memory and performance.
Verdict: Supported
Claim: It lowers the memory requirement of Hydra [ 20 ] by 7% while improving the accuracy by 0.1 pp .
Evidence type: table
Evidence caption: Results on a subset of the VTAB-1k benchmark. All methods use a ViT-Base-224/16 model pre-trained on ImageNet-21k. The batch sizes and ranks are the same across all tasks.
Context: We conduct experiments evaluating the performance of VeLoRA for full-tuning and how it complements other PEFT methods. In Tab. 1 we reproduce a large set of results for LoRA [ 17 ] , SSF [ 26 ] , and Hydra [ 20 ] on a subset of the VTAB-1K benchmark, where the sub-token size for each experiment is given in the Appendix. Unlike what is more common in the PEFT literature [ 19 , 20 ] , we do not perform any task-specific hyperparameter tuning that would change the memory, such as batch size and rank, and to also avoid any potential overfitting to the specific task. For all experiments we used the authors provided implementations for the adapters and integrated them into the same training framework for a fair comparison. We observe that VeLoRA improves the performance compared to full-tuning by 1.5 percentage points ( pp ), while lowering the memory requirements. We also observe that when combined with PEFT methods, VeLoRA can come with improvement in memory and performance. VeLoRA lowers the memory requirement of SSF [ 26 ] by 16% with only a minor degradation (0.1 pp ) in accuracy.
Verdict: Supported
Claim: Finally, it lowers the memory requirements of LoRA [ 17 ] by 4% while improving the accuracy by 0.6 pp
Evidence type: table
Evidence caption: Results on a subset of the VTAB-1k benchmark. All methods use a ViT-Base-224/16 model pre-trained on ImageNet-21k. The batch sizes and ranks are the same across all tasks.
Context: We conduct experiments evaluating the performance of VeLoRA for full-tuning and how it complements other PEFT methods. In Tab. 1 we reproduce a large set of results for LoRA [ 17 ] , SSF [ 26 ] , and Hydra [ 20 ] on a subset of the VTAB-1K benchmark, where the sub-token size for each experiment is given in the Appendix. Unlike what is more common in the PEFT literature [ 19 , 20 ] , we do not perform any task-specific hyperparameter tuning that would change the memory, such as batch size and rank, and to also avoid any potential overfitting to the specific task. For all experiments we used the authors provided implementations for the adapters and integrated them into the same training framework for a fair comparison. We observe that VeLoRA improves the performance compared to full-tuning by 1.5 percentage points ( pp ), while lowering the memory requirements. We also observe that when combined with PEFT methods, VeLoRA can come with improvement in memory and performance. VeLoRA lowers the memory requirement of SSF [ 26 ] by 16% with only a minor degradation (0.1 pp ) in accuracy. It lowers the memory requirement of Hydra [ 20 ] by 7% while improving the accuracy by 0.1 pp .
Verdict: Supported
Claim: Our method further reduces the memory needed for training to 2.23GB, an improvement of 18% compared to LoRA, and 45% compared to GaLore, while still reaching higher results than either of them.
Evidence type: table
Evidence caption: Comparison of our method with full fine-tuning, GaLore and LORA on GLUE benchmark using pre-trained RoBERTa-Base. Our method reaches the best overall results while showing significant memory improvements, especially compared to GaLore. We bold the best results from the considered PEFT methods. The GPU memory is measured on-device.
Context: We now evaluate our method with M=16 on various language tasks, using RoBERTa-Base [ 28 ] in the GLUE benchmark, and compare it with full fine-tuning, LoRA [ 17 ] and GaLore [ 53 ] , presenting the results in Tab. 2 .
We observe that both GaLore and LoRA lower the memory requirements compared to fine-tuning from 4.64GB to 4.04GB, respectively to 2.71GB, at a cost of accuracy degradation with GaLore performance lowered by 0.39 pp , while LoRA accuracy drops by 0.67 pp .
Verdict: Supported
Claim: Our method enhances the speed of both forward and backward computations compared to the previous state-of-the-art approach, benefiting both gradient-based and gradient-free optimization techniques.
Evidence type: table
Evidence caption: Table 2 : Time Performance. Our method achieves one order of magnitude speedup across all resolutions compared to PhiFlow in both forward simulation and backward gradient propagation.
Context: In this experiment, we demonstrate the performance efficiency of our framework through a comparison with PhiFlow. While PhiFlow operates with a TensorFlow-GPU backend, our framework is implemented in CUDA C++ and features a high-performance Geometric-Multigrid-Preconditioned-Conjugate-Gradient Poisson solver [ 17 ] . We benchmark both the one-step forward simulation time and gradient back-propagation time at different resolutions, as shown in Table 2 . The experiment runs on a workstation with a NVIDIA RTX A6000 GPU. Our high-performance framework consistently outperforms PhiFlow by an order of magnitude across all resolutions, thanks to the custom CUDA kernels and the rapid convergence of the Poisson solver.
Verdict: Supported
Claim: The experimental results, as shown in Table 2 , reveal that enhancing the strong activation of SCMoE can further boost MoE models’ performance.
Evidence type: table
Evidence caption: Table 2: Experimental results of different strong activations.
We set the weak activation with rank- 2 routing.
For each benchmark,
we select the top- k routing yielding the best performance in Figure 1 as the ideal strong activation.
The specific hyperparameter settings can be found in Table 9 .
Context: As revealed by Figure 1 , using default top-2 routing is not optimal for all tasks. For instance, top-3 routing yields best results on GSM8K, while top-4 routing achieves the highest accuracy on HumanEval and StrategyQA. This leads us to consider whether enhancing the strong activation in SCMoE can further unlock MoE models’ potential. To this end, we adjust the strong activation of Mixtral 8x7B to top-3 for GSM8K, and to top-4 for StrategyQA, MBPP, and HumanEval, while keeping the weak activation with rank-2 routing as before.
Verdict: Supported
Claim: The results in Table 3 show that SCMoE increases the decoding time by a factor of 1.30x compared to greedy.
Evidence type: table
Evidence caption: Table 3: Averaged decoding latency for each method. CS is short for contrastive search and CD is short for contrastive decoding. We set k = 3 for ensemble routing, while for dynamic routing we set threshold = 0.5. The speeds are tested on 4 A100 40G with batch size = 1.
Context: We further evaluate the impact of SCMoE on decoding latency and compare it with other methods on Mixtral 8x7B. Specifically, we first input 32 tokens to each method and then force them to generate a sequence of 512 tokens to calculate the latency.
Verdict: Supported
Claim: Specifically, compared to greedy baseline, SCMoE demonstrates improvements across all tasks: it enhances mathematical reasoning by 1.82 on GSM8K, commonsense reasoning by 2.58 on StrategyQA, code generation by 2.00 on MBPP, and 1.22 on HumanEval.
Evidence type: table
Evidence caption: Table 4: Experimental results on GSM8K, StrategyQA, MBPP and HumanEval with DeepSeekMoE-16B. We report the best results for each method here. The performance of each method with different hyperparameters can be found in the Appendix Table 10 .
Context: We further explore the adaptability of SCMoE to other MoE models. We conduct experiments on DeepSeekMoE-16B [ 5 ] . DeepSeekMoE-16B employs fine-grained expert segmentation and shared expert isolation routing strategies, which is different from Mixtral 8x7B [ 17 ] . We detail the hyperparameters settings of experiments in Appendix C . It is worth noting that contrastive decoding needs a suitable model to serve as an amateur. However, DeepSeekMoE-16B does not have a smaller model with the same vocabulary, so DeepSeekMoE-16B does not have the contrast decoding baseline. As depicted in Table 4 , SCMoE effectively unleashes the potential of DeepSeekMoE-16B.
Verdict: Supported