RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Among SNVs, ETS2 showed the highest mutation frequency (primarily mutation type: missense) ( Figure 4B ).
Context: The above pathways comprised 253 genes, of which 186 were present in TCGA dataset. Subsequently, a univariate analysis of these 186 genes yielded seven prognosis-related genes, including ETS2, SERPINE1, FOS, SHISA5, IL1A, TP53AIP1, and IGFBP7 ( Figure 4A ). Mutations in these seven genes in TNBC were further examined.
Caption: Figure 4: Analysis of cellular senescence-related prognostic genes.
(A) Forest plot of prognosis-related genes; (B) waterfall plot of SNVs in the seven prognosis-related genes; (C) Percentage plot of CNVs in the seven prognosis-related genes; (D) The expression of the seven prognosis-related genes expressed in tumor and normal tissues; (E) CDF curve of TCGA cohort; (F) CDF-delta area curves for TCGA cohort. The Delta area curve for consensus clustering indicates the relative change in the area under the CDF curve for each category number, k, relative to k-1. The horizontal axis represents the category number, k, and the vertical axis represents the relative change in area under the CDF curve; (G) Heat map of sample clustering at consensus k = 3; (H) KM curve of the relationship between the prognoses of the three subtypes in TCGA; (I) Heat map of the expression of seven prognosis-related genes among three subtypes in TCGA dataset.
Claim: However, we also find that it remains robust and maintains competitive accuracy even with limited access to calibration data, such as when using as few as 32 data points.
Context: In this section, we first compare the influence of different calibration sizes on FPQ . We vary the calibration size in \{32,64,128,256\} and test on MNLI, QQP, and CoLA. Table 4 shows that the evaluation on MNLI and QQP is more robust to different settings, and the variance is more significant on CoLA. We observe that FPQ performs well with a calibration set size of 128 data points.
Caption: Table 4: Ablation studies of different calibration sizes.
Claim: Specifically, Figure 5 highlights the model’s performance in various aspects, particularly LISA’s superiority over all methods in aspects of Writing, Roleplay, and STEM.
Context: As shown in Table 5 , LISA consistently produces better or on-par performance when compared with LoRA. Furthermore, in instruction-tuning tasks, LISA again surpasses full-parameter training, rendering it a competitive method for this setting.
Caption: Figure 5: Different aspects of LLaMA-2-70B model on MT-Bench.
Claim: Also here we can see that our proposed model outperforms the baseline models showing BLEU-4 score of 5.76 in test set.
Context: Table 4 presents the results of including facial landmarks as well as facial AUs with body and hands skeleton joints as input.
Caption: Table 4: Back translation results obtained from the generative models when using manual features and facial landmarks and AUs. Our proposed model has the highest scores in all metrics compared to the models using only gloss or text.
Claim: We observe that the teacher model is leading in terms of the adequacy of the explanations and preference rate, as expected from a larger system equipped for higher quality reasoning and generation capabilities.
Context: In Table 6 , we show adequacy and preference rates for explanations from the 3 systems, where an explanation is deemed adequate if both annotators agreed it is, and inadequate if both agreed it is not. The preference percentage is also taken among instances where the annotators agreed that the model’s explanation is preferred among all the adequate explanations. The average IAA using Cohen’s \kappa is 0.47, indicating moderate agreement Cohen ( 1960 ) .
Caption: Table 6: Adequacy and Preference rates for generated explanations.
Claim: C1 exhibited negative mean values for the knee and lower leg displacements during SD, significantly different from other clusters ( p < 0.001).
Context: To validate the quality and distinctiveness of the Louvain clustering solution, we calculated widely-used cluster validity indices. The Davies–Bouldin index was 2.09, and the Calinski–Harabasz index was 59.26. While the Davies–Bouldin index was somewhat high (lower values indicate better separation), the Calinski–Harabasz index supported the presence of distinct groupings in our data. The Louvain clustering algorithm identified four distinct clusters (C1–C4) with significant differences in all features (all p < 0.001, except for AHD-SU ( p = 0.017) and portion of EOA ( p = 0.019)) ( Table 2 ). Tukey’s post-hoc analysis revealed that those in C2 had the youngest age (44.80 ± 11.30 years, p < 0.001 vs. all other clusters) and highest BMI (25.59 ± 2.42, p < 0.001 vs. all other clusters). C2 included exclusively males, whereas the other clusters had a mixed sex composition. For SU kinematics, C4 exhibited the largest horizontal displacements for the pelvis, femur, knee, and lower leg, all significantly higher than other clusters ( p < 0.001). During SD, C3 showed the largest pelvis and femur displacements, while C4 had significantly higher ankle displacement ( p < 0.001 vs. all other clusters).
Caption: Table 2: Comparisons of features between the clusters classified by Louvain clustering unsupervised machine learning.
Claim: However, under anaerobic culture conditions, NEMM had greater alpha diversity indices of ACE and Chao1 than SHI ( Figure 5A ).
Context: To compare NEMM with SHI, which are suitable for cultivating specific bacterial types, we further analyzed 16S rRNA gene high-throughput sequencing data. Alpha diversity indices of ACE and Chao1 showed no significant difference between NEMM and SHI under the normoxia condition.
Caption: Figure 5: General bacterial community structure analysis between NEMM and SHI media during aerobic (normoxia) and anaerobic culture.
(A) Alpha diversity was measured with ACE index and Chao1 index. P < 0.05 was considered statistically significant. (B) NMDS was measured with Unweighted UniFrac. Each sample is represented by a dot. (C) Venn diagram displays the number of common and unique characteristics among the groups.
Claim: The true backpropagation gives the same curves for the ResNet and the HeunNet.
Context: We then apply a batch norm, a ReLU and iterate relation ( 1 ) where f is a pre-activation basic block (He et al., 2016b ) . We consider the zero residual initialisation: the last batch norm of each basic block is initialized to zero. We consider different values for the depth N and notice that in this setup, the deeper our model is, the better it performs in term of test accuracy. We then compare the performance of our model using a ResNet (forward rule ( 1 )) or a HeunNet (forward rule ( 8 )). We train our networks using either the classical backpropagation or our corresponding proxys using the adjoint method (formulas ( 6 ) and ( 9 )). We display the final test accuracy (median over 5 runs) for different values of the depth N in Figure 4 .
Caption: Figure 4: Comparison of the best test errors as a function of depth when using Euler or Heun’s discretization method with or without the adjoint method.
Claim: In the full training scenario, the increments are smaller, at only +1.4 and +1.1 .
Context: The improvements shown in Table 3 are particularly noticeable in few-shot training, with increases of +6.6 and +5.2 for binary and 3-way classification, respectively.
Caption: Table 3: Preliminary results, based on averaged macro-F1 scores across 34 languages. “EN Lex.” and “ML Lex.” indicate the English and multilingual NRC-VAD lexicons.
Claim: We can observe a clear push-back pattern: The higher the “coldness” or “incompetence” scores are for hateful stereotypes towards a target identity, the stronger the counter-stereotypes are in the opposite directions; consider, for illustration, the “warmth” dimension for gays and the “competence” dimension for women.
Context: The mean “warmth” and “competence” scores for each target identity are presented in Table 5 .
Caption: Table 5: The mean “warmth” and “competence” scores for hateful (H) and non-hateful (N/H) examples. We highlight the scores with the highest magnitude in bold .
Claim: The BERT multi-task classifier improves these numbers considerably (e.g., obtaining 59.69 and 59.60 micro F1 for CF and Skill, respectively), indicating the benefit of richer contextual representations.
Context: Table 1 reports the micro and macro F1 scores for various models on the CF, IC, and skill classification. We compare several baselines, including traditional TF-IDF and BERT multi-task classifiers, graph-based models without BERT, and our proposed CFiCS variants that integrate clinical BERT features. The TF-IDF multi-task Random Forest baseline achieves relatively modest performance, with micro F1 scores of 52.50, 74.59, and 53.02 for CF, IC, and Skill, respectively, and corresponding macro F1 scores of 20.43, 38.04, and 49.77.
Caption: Table 1: Three-fold cross-validation micro and macro F1 scores.
Claim: Additionally, we collected 1,034 answerable queries from the Spider dev dataset and augmented them with natural language explanations derived from their execution results.
Context: Table 3 shows the statistics of the dataset generated using the Spider dev set with Claude 3 sonnet. Note that the employed methodology can be seamlessly adapted to other text-to-SQL datasets like BIRD, WikiSQL, or any other synthetically generated answerable text-to-SQL corpora combined with any LLM (e.g., Llama3.1 or mixtral). The generated dataset consists of 1,802 ambiguous and unanswerable questions spanning various categories.
Caption: Table 3: Dataset statistics and human annotation accuracy on 20 samples per question type. "#Ex" column shows the number of examples generated for each category. "Acc" column shows average binary classification accuracy from human expert.
Claim: Our results indicate significant correlations between the peak angular momentums of the attack arm, non-attack arm, non-attack leg, forearm, and hand with the arm swing speed at BI ( Table 1 ).
Context: This study investigated the relationships between angular momentum variables and swing hand speed during jump serve aerial spiking, utilising correlation and regression analyses.
Caption: Table 1: Peak angular momentum variables and correlations with the arm swing speed at ball impact (BI).
Claim: It is also robust, with a standard deviation of 0.022 .
Context: The results comparing our proposed QuestMF ( ImbOLL ) framework with its ablations and current state-of-the-art methods are presented in Table 1 . Since the prior works only show their best results on a single run, we also pick our best results on CCC for a fair comparison. As we can see, our proposed QuestMF ( ImbOLL ) framework matches the performance of state-of-the-art models in CCC, the primary metric used to evaluate depression severity prediction tasks. In addition to comparable performance, QuestMF framework provides question-wise scores, improving interpretability over current methods, thus allowing clinicians to design personalised interventions. We also show the robustness of our frameworks over multiple runs, which is not done by the previous works. The mean and standard deviation of the performance over 3 runs are presented in Table 2 . We observe that Total and QuestMF (MSE) perform similarly. However, training with ordinal classification objective improves the performance, as we can see from the results of QuestMF (OLL). This shows the effectiveness of combining QuestMF with ordinal classification training. Training with our novel ImbOLL function further improves the results on CCC.
Caption: Table 2: Results of QuestMF ( ImbOLL ) framework over 3 different seed runs compared with ablation frameworks.
Claim: As shown in Figure 4(b) , increasing the value of \alpha can prevent the model from outputting more sensitive information, but it may also lead to the loss of necessary information.
Context: For unlearning, we found that adjusting the value of \alpha can serve as a balance between forgetting and retaining .
Caption: (a) Impact on instruction tuning; (b) Impact on unlearning; Impact of strength coefficient \alpha on performance
Claim: Comparing PECRS-small and -medium shows that Dist@K improvements can be achieved by scaling up the backbone model.
Context: Table 3 summarizes the results on conversation task, where PECRS achieves promising performance on both types of metrics. Both PECRS-small and -medium surpass all baselines over Dist@3 and Dist@4.
Caption: Table 3: Results of conversation task compared with the state-of-the-art on ReDial.
Claim: As shown in Table 3 , the performance on MiT and SSv2 suffers by a large margin, indicating that the projection design helps boost training by better utilizing multi-dataset information.
Context: How important is the projection loss? We then experiment with removing the projection heads (Section 3.2 ) during multi-dataset training. The model is trained with the original cross-entropy loss and the informative loss.
Caption: Table 3: Ablation experiments. We investigate the effectiveness of each component of our method as well as compare to vanilla multi-dataset training method.
“Vanilla” means using cross entropy (CE) loss in training.
“w/o informative los” means using CE and projection loss.
The numbers are top-1/top-5 accuracy, respectively.
Training data:
(e) Kinetics-400;
(f) SSv2;
(g) MiT;
(h) ActivityNet.
Claim: As shown in Table 1 , for CSQA2.0, higher scores are reported for retrieval only than for induction only, while the result is contrary for StrategyQA.
Context: Besides, the results on different setups of IAG-GPT suggest that, the relative contributions of the retrieval and the inductive knowledge can be different, depending on the tasks.
Caption: Table 1: Performance on two ODQA tasks. The first two columns report scores on CSQA2.0 dev set and StreategyQA test set respectively. The last two columns compare IAG with ChatGPT on a randomly held-out subset containing 50 examples for each task.
Claim: The improvements shown in Table 3 are particularly noticeable in few-shot training, with increases of +6.6 and +5.2 for binary and 3-way classification, respectively.
Caption: Table 3: Preliminary results, based on averaged macro-F1 scores across 34 languages. “EN Lex.” and “ML Lex.” indicate the English and multilingual NRC-VAD lexicons.
Claim: Surprisingly, the Medium task displays the least bias, likely because models perform best in this task.
Context: From Fig. 5 -right, extracting 4 categories in the Hard task shows the largest performance gap between mapping formats.
Caption: Figure 5: Average estimated true F1 scores ( Section 3.1 ) across models (left) and benchmarks (right) showing performance bias of LLMs across 2 widely used mapping formats.
Claim: Figure 8 shows that the T2DM group has lower contents of total SCFAs, acetic acid, propionic acid, butyric acid, isobutyric acid, valeric acid, and isovaleric acid, when compared with the HC group ( P < 0.05).
Caption: Figure 8: Total SCFAs (acetic acid, butyric acid, propionic acid, isobutyric acid, isovaleric acid, and valeric acid) after 24-h in-vitro fermentation with T2DM human fecal microflora.
Data were expressed as means ± SDs (n = 8).
Claim: We see that the full model outperforms all the variants, implying that all three types of components are essential for detection performance in this problem setting.
Context: We evaluate variants of the ConDA model, by removing one component at a time and compare these in Table 5 . ConDA \CEs removes the two cross-entropy losses, i.e. no supervision even for the source. ConDA \contrast removes the contrastive loss components for both source and target. ConDA \MMD removes the MMD loss between source and target. Hence the only component that makes use of the unlabeled target domain data is the target contrastive loss. Finally, ConDA is the full model.
Caption: Table 5: Comparison of different model variants; bold shows best performance. We randomly chose 3 target domains to show in this table due to space constraints.
Claim: Average reward model accuracy ("Avg") in Table 1 shows that English RMs surpass target language RMs in general.
Caption: Table 1: Multilingual RewardBench evaluation results on the target language ("Target") and English ("English") RMs. " \Delta " denotes the accuracy gain of English RMs compared to the target language RMs. English RMs show higher average scores in the lingual axis than target language RMs. Also, English RMs excel target language RMs in reasoning ("Reason") tasks with diverse evaluation sub-categories.
Claim: When looking at individual appraisal dimensions, we can see that the RMSE’s are in most cases lower in the random tie-breaking setting compared to using the confidence rating, and that applies to both the model and humans.
Context: The results of this experiment are shown in Table 2 columns GPT-4 {}_{\text{conf}} and Human {}_{\text{conf}} . We can see that using the confidence rating to break the ties did not improve the RSME neither for the model nor for the human reader-annotators compared to the random choice. The mean RMSE for GPT-4 from 1.12 to 1.15 (columns GPT-4 {}_{\text{maj}} and GPT-4 {}_{\text{conf}} in Table 2 ), and the mean RMSE for human reader-annotators increased from 0.99 to 1.06 (columns Human {}_{\text{maj}} and Human {}_{\text{conf}} in Table 2 ).
Caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Claim: Moreover, we observe that our pre-trained model exhibits a way lower variance between fine-tuning runs.
Context: We report labeled F1-score averaged on 100 fine-tuning runs in Table 2 .
Precision and recall are reported in Appendix B .
The Base model (using the full 12 layers) produces similar results on DOCILE no matter if pre-training on IIT-CDIP or DOCILE . However, on our internal Payslips datasets, our model pre-trained on DOCILE outperforms the original one.
Caption: Table 2: F1 scores for named-entity recognition using different pre-training and fine-tuning datasets. Results are averaged on 100 runs with different seeds.
Claim: For formality control, on Korean, the most distant language,
CG consistently achieves stronger control results than finetuning,
indicating more robustness when transferring to unfamiliar settings.
Context: While finetuning was consistently leading in supervised conditions (§ 5 ),
now under zero-shot conditions with unseen target languages, the gap shrinks.
Caption: Table 4: Zero-shot formality control results. Best and second best results under the same data condition are marked.
Claim: We find that all QA models suffer a large performance drop ( \sim 20% in EM) in MisinfoQA-noisy compared to MisinfoQA-clean , showing that the models are largely distracted by the fake contexts rather than by the presence of additional contexts.
Context: Table 4 reports the EM and F1 for both human and different QA models.
Caption: Table 4: QA performance under the reading comprehension settings with clean and noisy contexts.
Claim: Following these were premature birth ( p -value < 0.001), with 156 (82.5%) smokers and 176 (93.1%), miscarriages ( p -value = 0.048), with 149 (78.8%) smokers and 164 (86.8%) non-smokers, neck cancer ( p -value < 0.001) with 144 (76.2%) smokers and 165 (87.3%) non-smokers and lastly gangrene ( p -value < 0.001) with 115 (60.8%) smokers and 139 (73.5%) who answered ‘Yes’ for respective health effects.
Context: Figure 2 describes the known health effects of tobacco use among respondents. The two most recognised health effects of smoking by both groups of respondents are lung cancer ( p -value = 0.011) and mouth cancer ( p -value < 0.001), with 170 (89.9%) smokers and 184 (97.4%) non-smokers who answered ‘Yes’ for lung cancer while 152 (80.4%) smokers and 181 (95.8%) non-smokers answered ‘Yes’ for mouth cancer.
Caption: Figure 2: Percentage of known health effects of tobacco use among respondents (n = 378).
Claim: In Fig. 5(a) , the accuracy is poorer in range [1,10] , grows in range [10,20] , and saturates thereafter.
Context: We vary the number of MC samples M from 1 to 50.
Caption: (a) Accuracy-runtime tradeoff; (b) Parameter count comparison; Ablations on CIFAR100 showing: (a) performance trade-off with the number of MC samples M , (b) the number of trainable parameters in different finetuning methods.
Claim: Language coverage of the NRC-VAD multilingual lexicons remains limited in 109 languages.
Caption: Table 1: Languages used in this paper. “✓” (green) and “✗” (red) mean that the language has and has not been seen by the models or language resources. CW indicates code-switching text. The language coverage for GPT-3.5 is derived from GPT-3 Brown et al. ( 2020 ) .
Claim: Development sets are 5k instances samples from each set.
Context: We synthesize approximately 1.5 million multi-hop questions and 1.9 million claims. We use neucleus sampling (Holtzman et al., 2020 ) with a top- p probability of 0.9 for decoding when generating the data.
Caption: Table 1: Dataset statistics for synthetic data generated in this work. We omit the average length of answers for fact verification as it is a classification task. SQ=Single-Query. TQ=Two-Queries.
Claim: Also, longer training schedule does not lead to better performance on some datasets, including SSv2, suggesting vanilla multi-dataset training is unstable.
Context: Does our proposed robust loss help? We compare our model training with vanilla multi-dataset training, where multiple classification heads are attached to the same backbone and the model is trained simply with cross-entropy loss. The vanilla model is trained from a K400 checkpoint as ours. As shown in Table 3 , we try training the vanilla model with both the same training schedule as ours and a 4x longer schedule. As we see, there is a significant gap between the overall performance of the vanilla model and ours, validating the efficacy of our proposed method.
Caption: Table 3: Ablation experiments. We investigate the effectiveness of each component of our method as well as compare to vanilla multi-dataset training method.
“Vanilla” means using cross entropy (CE) loss in training.
“w/o informative los” means using CE and projection loss.
The numbers are top-1/top-5 accuracy, respectively.
Training data:
(e) Kinetics-400;
(f) SSv2;
(g) MiT;
(h) ActivityNet.
Claim: Fifty healthy students from University Europea de Madrid volunteered for the study Table 1 .
Caption: Table 1: Descriptive data.