RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: In terms of performance on ActivityNet, we observe that both training methods achieve good results, which might be because ActivityNet classes are highly overlapped with Kinetics-400 (65 out of 200).
Context: Does our proposed robust loss help? We compare our model training with vanilla multi-dataset training, where multiple classification heads are attached to the same backbone and the model is trained simply with cross-entropy loss. The vanilla model is trained from a K400 checkpoint as ours. As shown in Table 3 , we try training the vanilla model with both the same training schedule as ours and a 4x longer schedule. As we see, there is a significant gap between the overall performance of the vanilla model and ours, validating the efficacy of our proposed method. Also, longer training schedule does not lead to better performance on some datasets, including SSv2, suggesting vanilla multi-dataset training is unstable.
Caption: Table 3: Ablation experiments. We investigate the effectiveness of each component of our method as well as compare to vanilla multi-dataset training method.
“Vanilla” means using cross entropy (CE) loss in training.
“w/o informative los” means using CE and projection loss.
The numbers are top-1/top-5 accuracy, respectively.
Training data:
(e) Kinetics-400;
(f) SSv2;
(g) MiT;
(h) ActivityNet.
Claim: SpeechSim/Noise combined with Vendi Score obtains the second highest correlation, following HuBERT.
Context: We report our results in Table 5 . We see that generally all representations have somewhat low levels of correlation.
Caption: Table 5 : Background noise diversity. Average Spearman correlations between the number of different classes of noise and diversity scores induced by speech representations.
Claim: However, it performs only at 67.58% accuracy on ambiguous/unanswerable questions, lagging 3.7% behind Claude 3.5 sonnet.
Context: Table 5 shows our baseline method’s (DIN-SQL) performance on SQL prediction of various LLMs given the interaction between the user and the assistant. Overall, Mixtral-large-v2 and Claude 3.5 Sonnet achieve the highest average accuracy of 71.95% and 72.15% on the ambiguous/unanswerable questions. Claude 3.5 sonnet achieves the highest performance of 79.21% on the answerable questions (original Spider dev set). The open-source model Llama-3.1 70B performs competitively on the answerable questions achieving 76.31% accuracy, only 2.9% lower than Claude 3.5 sonnet.
Caption: Table 5: Execution accuracy of SQLs predicted with DIN-SQL using different LLMs on various categories of ambiguous, unanswerable, and answerable questions. The "All" column shows the overall average accuracy across all categories, while the "Avg. Excluding Answerable" column shows the average accuracy excluding the answerable questions from the Spider dataset.
Claim: Interestingly, a strong bacterial growth inhibition was observed at a concentration of 2.75 µg/mL during all the incubation times ( Figure 1A ).
Context: To analyze the extracellular antimicrobial activity of cystatin C against P. gingivalis , bacteria were incubated with different concentrations of the peptide (1.25, 1.5, 1.75, 2, 2.5, 2.75 µg/mL) and increasing time periods (1, 12, 24, 48, and 96 h). Cystatin C exhibited a dose-and time-dependent antimicrobial activity after 12 h of incubation, as shown in Figure 1A . Additionally, maximum microbicidal activity was exhibited at concentrations between 2.25 and 2.75 µg/mL after 12 h. Concentrations of 2.25 µg/mL, 2.5 µg/mL, and 2.75 µg/mL inhibited bacterial growth at 60%, 80%, and 95%, respectively, after 24 h of incubation and compared to the control group ( p < 0.05). Furthermore, bacterial inhibition with these three concentrations was also observed after 48 h of incubation.
Caption: Figure 1: Effect of cystatin C on P. gingivalis growth and macrophage viability.
(A) 5 × 106 CFU/mL of P. gingivalis were cultured in the presence of cystatin C at different concentrations, for 1, 12, 24, 48, and 96 h. Cells without stimulation were used as a control group. when compared to the control group. The significance represents a p < 0.05 (*). (B) 1 × 105 cells were co-incubatedat different concentrations of cystatin C for 24 h. No significant differences were found when comparing different concentrations of cystatin C with control group p& γ τ 0.05 (ns). Data represent the mean ± SD of five independent experiments (n = 5).
Claim: Note that resistance against the attack to both in- and out-sets is much harder than other cases since the perturbation budget has effectively been doubled.
Context: OOD detection under adversarial attack: To perform a comprehensive study, AUROC is computed in four different settings for each method. First, the standard OOD detection without any attack is conducted (Clean). Next, either in- or out-datasets are attacked (In/Out). Finally, both the in- and out-sets are attacked (In and Out).
Caption: Table 1: OOD detection AUROC under attack with \epsilon=\frac{8}{255} for various methods trained with CIFAR-10 or CIFAR-100 as the closed set. A clean evaluation is one where no attack is made on the data, whereas an in/out evaluation means that the corresponding data is attacked. The best and second-best results are distinguished with bold and underlined text for each column.
Claim: If we compare the two-layer network results from GeoLIP and Sampling in Table 1 , which is a lower bound of true Lipschitz constant, the ratio is within 1.783 .
Context: We have also shown that the two-layer network \ell_{\infty} -FGL estimation from GeoLIP has a theoretical guarantee with the approximation factor K_{G}<1.783 ( Theorem 3.3 ).
Caption: Table 1: \ell_{\infty} -FGL estimation of various methods: DGeoLIP and NGeoLIP induce the same values on two layer networks. DGeoLIP always produces tighter estimations than LiPopt and MP do.
Claim: By repeating the wrapping instructions of ChatGPT thrice, we observed a decrease in the BiasF_{o} (ii) score presented in Fig. 6 . Combining (i) and (ii) suggests that this strategy is an effective mitigation.
Context: We found that repeating instructions generally increases FI scores (i) across most formats except “Placeholder”, which can consequently lessen the mode’s token bias towards format instructions ( Section 5.1 ). Using our two proposed criteria for effective format bias mitigation in Section 3.2 , it is worth examining if this approach reduces BiasF_{o} , thereby being an effective mitigation. Our answer is yes.
Caption: Figure 6: More demonstrations and repeating format instructions mitigate format bias. Finetuning mostly eliminates the format bias. The performance is reported using ChatGPT on MMLU (Appx.- Tab. 18 for num. results).
Claim: With regard to ECE (right subfigure), we see all methods improving with growing ensemble size M .
FiLM-Ensemble achieves better calibration then the widely used deep ensemble and MC-dropout methods, with significant margins at large ensemble sizes.
Context: We perform several experiments on widely used benchmarks in computer vision, CIFAR-10 and CIFAR-100. As performance metrics, we plot the test set accuracy and the expected calibration error (ECE) against the ensemble size M in Fig. 2 ,
for all compared methods. In terms of accuracy (left subfigure), FiLM-Ensemble is outperformed only by the explicit deep ensemble, across all tested values of M\in\{2,4,8,16\} . These two surpass all other methods by a clear margin. Surprisingly, we observe that BatchEnsemble generally exhibits a negative correlation between ensemble size and test set accuracy, with M=4 performing best among all settings. MIMO shows very poor performance in this experimental setting in terms of accuracy (and also in ECE). We speculate that its shared backbone probably has a tendency to fragment into largely independent ensemble members of low channel depth, which lack the necessary capacity when using comparatively small networks like VGG-11 or Resnet-18.
Caption: Figure 2: Accuracy and ECE for CIFAR-10, with varying ensemble sizes, using VGG-11 as backbone.
Claim: Both JS Divergence and Cosine Similarity methods show similar patterns, with high recall (0.93) but lower precision.
Context: The results show RACLETTE’s Combined method achieving the highest recall of 0.95, indicating superior ability to identify relevant cases, though this comes with a trade-off in precision at 0.63. Conversely, RoBERTa leads in precision at 0.72, but with lower recall at 0.84. The KL Divergence variant of RACLETTE stands out for its balanced performance, maintaining strong scores across all metrics (precision: 0.71, recall: 0.90, F1: 0.79, accuracy: 0.77).
Caption: Table 4: Classification results for different methods of comparing embeddings to detect risk of suicide. The higher the score (a.k.a. the greener), the better.
Claim: Table 2 shows that MPGD can lead to better test performance and lower accuracy gap when compared to the baseline (full batch GD) and Gaussian noise-perturbed GD.
Caption: Table 2: ResNet-18 trained on CIFAR-10 for 1000 epochs. Here, accuracy gap = training accuracy - validation accuracy. The results in parenthesis are achieved with the variant of MPGD ( 11 ). All the results are averaged over 5 models trained with different seed values.
Claim: Equally, the specialized projection of SpeechSim for the emotion facet, SpeechSim/Emotion performs best across all representations.
Context: In Table 3 we report average Spearman correlations for the tested representation models, when tested on EmoV and Expresso separately. From these results, we see that, generally, SpeechSim performs better than other general-purpose representations across all configurations.
Caption: Table 3 : Emotion diversity. Average Spearman correlations between the classes entropy in EmoV and Expresso and diversity scores induced by the speech representations. SpeechSim/Emotion-Expresso (resp. Speech/Emotion-EmoV) refers to SpeechSim emotion head trained on Expresso (resp. EmoV).