RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Language-contrastive decoding brings additional gains of 0.4 (M2M-100) and 0.2 (SMaLL-100) points.
Context: We report results using source-contrastive decoding ( C_{src} ), and combining source-contrastive and language-contrastive decoding ( C_{src+lang} ) in Table 1 . 10 10 10 See Appendix A for full results. Across 57 translation directions, chrF2 improves by 1.3 (M2M-100) and 1.1 (SMaLL-100) points with source-contrastive decoding.
Caption: Table 1: Automatic evaluation results. Averages over different sets of translation directions.
Claim: GPT-4+DIN-SQL obtain EX score of 6.73% on Archer test set, while it is able to achieve 85.3% test-suite execution performance on Spider test set Pourreza and Rafiei ( 2024 ) .
Caption: Table 2: Baseline performance on Archer. GPT-4+DIN-SQL was tested only on the English set due to cost and its English-specific design. We only report the fine-tuned model’s performance on the test set.
Claim: The observation is that MQAG achieves a higher correlation than the best SpanQAG on 5 out of 6 tasks.
Context: The baseline and MQAG results are shown in Table 5 .
Caption: Table 5: Pearson Correlation Coefficient (PCC) between the scores of summary evaluation methods and human judgements. PCCs are computed at the summary level on QAG and XSum-H, and at the system level on Podcast and SummEval. PCCs on Podcast are computed on 15 abstractive systems. Our best performing MQAG configuration consists of (i) generation stage G generates questions from summary y (i.e. MQAG-Sum), (ii) statistical distance is total variation, (iii) the answerability threshold \mathcal{N}^{\tau}_{y} is set to 2.0. Underline denotes where MQAG outperforms the best SpanQAG system, which is 5 out of 6 tasks. When compared to all baselines, MQAG achieves the highest PCC on 4 out of 6 tasks. The results of all MQAG configurations are provided in Table 10 , and Spearman’s correlation results are provided in Table 11 in the appendix.
Claim: Notably, 81.6% of participants had erectile dysfunction, with the mean IIEF total score indicating a mild level of dysfunction (18.72 ± 3.60) ( Table 1 ).
Context: Table 1 presents the personal data of the participants along with the total score of IIEF. The participants had a mean age of 63.26 ± 7.62 years, with 97.3% being married. Additionally, 46.6% of participants were primary school graduates, 9.4% reported alcohol use, and 13% were smokers, with an average duration of smoking of 22.83 ± 6.94 years. The mean duration of hypertension was 7.92 ± 4.06 years, and the most commonly used medication among participants (27.8%) was ACE Inhibitors.
Caption: Table 1: Participants’ personal data, IIEF total score (n= 223).
Claim: A total of 1,896 chiastic structures were identified at the half-verse level, with an average length of 5.93 textual units ( \pm 1.34) and an average score of 0.32 ( \pm 0.1).
Context: Table 2 presents an overview of the system’s output for chiastic structures at the half-verse and verse levels.
Caption: Table 2: Summary of detected chiasmi. 2700+ chiasmi were detected at the verse and half-verse level. The highest number of chiasmi was found in the Book of Genesis and Book of Numbers. Both the precision and the inter-annotator agreement increase for the verse-level chiasmi.
Claim: Furthermore, selecting datapoints based solely on high predictive uncertainty without considering diversity ( HighLoss ) is an ineffective strategy, having the second lowest proportion of wins (Fig. 5 , right).
Caption: Figure 5: Summary of subset selection strategies performances from. Left: percentage of times each strategy gets the best performance out of 35 settings (across each of the 7 languages and 5 \hat{D}^{Syn}_{train} sizes). Right: bootstrapped confidence intervals for the percentages on the left.
Claim: Thereafter, Swin becomes comparatively much more robust as occlusion becomes severe.
Context: For (1), we provide evidence that this difference between ResNets and DeiTs is not due to the nature of transformer vs. conv layers. In order to do so, we re-run the random patch drop experiments in [ 47 ] on all pairs of ConvNext and Swin networks. We present results for different levels of information loss in Fig. 3 . For all networks we observe a drop in performance as information loss increases. We show that ConvNext networks do not suffer the same critical failure mode that DenseNet121, ResNet50, SqueezeNet and VGG19 exhibit in Naseer et al. [ 47 ] . The top accuracy of these four classic ConvNets collapses to half after about 10% of patches are occluded and to 0 after about 40% information loss. ConvNext shows that it is much more resistant, conserving a large part of its accuracy even after 50% information loss. Next, we observe that ConvNext networks are slightly less robust than Swin Transformers for low amounts of information loss (under 50%).
Caption: Figure 3: Random patch drop occlusion study on ConvNext and Swin networks on the ImageNet-1k validation set. Swin Transformers are slightly more robust to this type of artificial occlusion than ConvNext networks when the information loss is small, although they become comparatively much stronger as the information loss is increased.
Claim: As depicted in the figure, AgA with sign selection is aligned towards the fastest update direction, resulting in AgA progressing approximately 6 steps ahead of AgA without Sign at the end of the trajectory.
Context: In addition, Fig. 3 presents a critical comparison between the trajectories of AgA (in red) and AgA without sign alignment (AgA w/o Sign, depicted in purple), as outlined in Corollary 4.3 . This side-by-side comparison covers 40 steps and features highlighted points every tenth step. It provides a visual illustration of the efficiency introduced by sign alignment starting from the 14th step in the trajectories of AgA and AgA without sign alignment. Remarkably, norm gradients are represented by blue arrows, indicating the direction of the fastest updates.
Caption: (a) Cleanup: AgA with different \lambda and AgA/Sign; (b) Cleanup: comparison between AgA and baselines; (c) Harvest: AgA with different \lambda and AgA/Sign; (d) Cleanup: comparison between AgA and baselines; Comparison of Episodic Social Welfare: our proposed AgA methods along with baselines on Cleanup (figure a and b) and Harvest (figure c and d) tasks. Figure (a) and (c) provide an illustration of varying alignment parameter \lambda values in AgA, as well as AgA sans sign selection (AgA/Sign), as applied to Cleanup and Harvest tasks respectively. Subfigures (b) and (d) exhibit the social welfare obtained by AgA utilizing the optimal \lambda value, in comparison to those secured by Simul-Ind, Simal-Co, CGA, SGA and SVO. Bold lines represent the average of the social welfare across three seeds, with the encompassing shaded areas denoting the standard deviation.
Claim: Among CNVs, IGFBP7 had the highest frequency of acquired CNVs (CNV_gain), whereas SHISA5 had the highest frequency of deletion mutations (CNV_loss) ( Figure 4C ).
Context: The above pathways comprised 253 genes, of which 186 were present in TCGA dataset. Subsequently, a univariate analysis of these 186 genes yielded seven prognosis-related genes, including ETS2, SERPINE1, FOS, SHISA5, IL1A, TP53AIP1, and IGFBP7 ( Figure 4A ). Mutations in these seven genes in TNBC were further examined. Among SNVs, ETS2 showed the highest mutation frequency (primarily mutation type: missense) ( Figure 4B ).
Caption: Figure 4: Analysis of cellular senescence-related prognostic genes.
(A) Forest plot of prognosis-related genes; (B) waterfall plot of SNVs in the seven prognosis-related genes; (C) Percentage plot of CNVs in the seven prognosis-related genes; (D) The expression of the seven prognosis-related genes expressed in tumor and normal tissues; (E) CDF curve of TCGA cohort; (F) CDF-delta area curves for TCGA cohort. The Delta area curve for consensus clustering indicates the relative change in the area under the CDF curve for each category number, k, relative to k-1. The horizontal axis represents the category number, k, and the vertical axis represents the relative change in area under the CDF curve; (G) Heat map of sample clustering at consensus k = 3; (H) KM curve of the relationship between the prognoses of the three subtypes in TCGA; (I) Heat map of the expression of seven prognosis-related genes among three subtypes in TCGA dataset.
Claim: Meantime, Self-Taught-Llama-70B with extensive multilingual pre-training demonstrates the strongest cross-lingual transfer, achieving the best accuracies in all four non-English Multilingual RewardBench .
Context: Interestingly, we can observe strong cross-lingual transfer in the generative RMs in Table 2 , as in the classifier RMs. As discussed in Section 3.2 , fine-grained representation learning is a crucial component for having strong downstream generative abilities. While the extent of multilingual pre-training in GPT-4o is not verifiable, GPT-4o has the least decrement in non-English settings.
Caption: Table 2: Averaged Multilingual RewardBench results in two classifier RMs (top) and two generative RMs (bottom). Off-the-shelf RMs based on MLMs show strong cross-lingual transfer as in Table 1 .
Claim: All models can relatively accurately identify hateful posts with negative emotions and non-hateful posts with positive emotions.
Context: We further group the fine-grained emotions into positive (1), negative (-1), and ambiguous (0), based on Demszky et al. ( 2020 ) ’s taxonomy and present the models’ classification accuracy in the presence of emotions with different polarities in Table 4 . The result is revelatory:
Caption: Table 4: Classification accuracy of HateBERT, ToxDect-roberta, Perspective API, and Llama Guard 3 1/8B on GPT-HateCheck grouped by the hatefulness label (hate) and the polarity of the detected emotions (emo). We highlight the “positive” labels in green (“non-hateful” and positive emotions) and “negative” labels in red . The ambiguous emotions are highlighted in yellow .
Claim: However, there were no significant differences in maternal age, number of births, abortion history, history of fetal abnormalities, mode of conception, pregnancy complications, education level, employment status, or place of abode ( Table 1 ).
Context: Chi-square test analysis showed significant differences in the gestational age at diagnosis of fetal abnormalities, the number of fetal abnormalities, diagnostic gestational age, and treatment recommendations of doctors among pregnant women ( P < 0.05).
Caption: Table 1: Comparison of pregnancy outcomes in pregnant women with different characteristics of fetal abnormalities.
Claim: Tukey’s post-hoc analysis revealed that those in C2 had the youngest age (44.80 ± 11.30 years, p < 0.001 vs. all other clusters) and highest BMI (25.59 ± 2.42, p < 0.001 vs. all other clusters).
Context: To validate the quality and distinctiveness of the Louvain clustering solution, we calculated widely-used cluster validity indices. The Davies–Bouldin index was 2.09, and the Calinski–Harabasz index was 59.26. While the Davies–Bouldin index was somewhat high (lower values indicate better separation), the Calinski–Harabasz index supported the presence of distinct groupings in our data. The Louvain clustering algorithm identified four distinct clusters (C1–C4) with significant differences in all features (all p < 0.001, except for AHD-SU ( p = 0.017) and portion of EOA ( p = 0.019)) ( Table 2 ).
Caption: Table 2: Comparisons of features between the clusters classified by Louvain clustering unsupervised machine learning.
Claim: Lachnospiraceae were positively associated with 1, 3-propanediol and L-proline, and were negatively correlated with hydracrylic acid and CH 4 .
Context: To further explore correlations of metabolites in human fecal matter and intestinal flora after SNP treatment, thermography was used to analyze contents of metabolites showing significant differences with 30 most abundant genera. Figure 10 shows the results. Bifidobacterium was positively associated with glycerol, CO 2 , 4-aminobutanoic acid, L-cysteine, and D-gluconic acid, and could be negatively correlated with DL-phenylalanine. Faecalibacterium was positively associated with pentanoic acid and 4-aminobutanoic acid, while were negatively correlated with hydracrylic acid. Lactobacillus was positively correlated with xylitol and showed a negative correlation with 5-aminovaleric acid. Dorea was positively associated with phenylalanine, butyric acid, and isovaleric acid, while showed a negative correlation with hydracrylic acid, CH 4 , myo-inositol, L-threonine, glycerol, CO 2 , L-cysteine, and D-gluconic acid. Escherichia - Shigella could be positively associated with H 2 S, L-proline, cadaverine, DL-phenylalanine, benzeneacetic acid, 1,3-propanediol, and were negatively correlated with glycerol, CO 2, 4-aminobutanoic acid, L-cysteine, D-gluconic acid, propionic acid, and L-lysine. Phascolarctobacterium was positively associated with 1,3-propanediol, L-proline, cadaverine, and DL-phenylalanine, and was negatively correlated with glycerol, CO 2 , 4-aminobutanoic acid, L-cysteine, and myo-inositol.
Caption: Figure 10: Correlation analysis of characteristic metabolites and bacteria.
Corr represents correlation. Red and blue separately denote positive and negative correlations. The darker the color, the stronger the correlation. *P < 0.05, **P < 0.01, and ***P < 0.001.
Claim: The generated dataset consists of 1,802 ambiguous and unanswerable questions spanning various categories.
Context: Table 3 shows the statistics of the dataset generated using the Spider dev set with Claude 3 sonnet. Note that the employed methodology can be seamlessly adapted to other text-to-SQL datasets like BIRD, WikiSQL, or any other synthetically generated answerable text-to-SQL corpora combined with any LLM (e.g., Llama3.1 or mixtral).
Caption: Table 3: Dataset statistics and human annotation accuracy on 20 samples per question type. "#Ex" column shows the number of examples generated for each category. "Acc" column shows average binary classification accuracy from human expert.
Claim: These variables include age ( p = 0.002), having children ( p = 0.002), being related to the health sector ( p = 0.001), perception of the efficacy and protective effect of the booster dose (<0.001), as well as fear of adverse effects ( p < 0.001) were significantly associated with the IBV ( Table 2 ).
Context: Several variables were found to be significantly associated with the intention to be vaccinated (IBV), as detailed in Table 2 .
Caption: Table 2: Bivariate analysis of study characteristics according to intention to be vaccinated with the booster dose COVID-19 in Peru, July 2022 (n= 924).
Claim: From these results, we see that, generally, SpeechSim performs better than other general-purpose representations across all configurations.
Context: In Table 3 we report average Spearman correlations for the tested representation models, when tested on EmoV and Expresso separately.
Caption: Table 3 : Emotion diversity. Average Spearman correlations between the classes entropy in EmoV and Expresso and diversity scores induced by the speech representations. SpeechSim/Emotion-Expresso (resp. Speech/Emotion-EmoV) refers to SpeechSim emotion head trained on Expresso (resp. EmoV).
Claim: In this case, eleven different metrics are extracted.
Context: This study considers four different close-ended healthcare tasks, which include nine different datasets ( e.g. , MedQA). These are all assessed using the accuracy metric. At the same time, six open-ended tasks are studied, based on nine distinct datasets ( e.g. , MedText).
Caption: Table 1: This table presents the tasks implemented in this paper. The first column specifies the different tasks. The second details the metrics used (ROUGE includes ROUGE1, ROUGE2 and ROUGEL, and Perplexity includes Bits per Byte, Byte Perplexity, and Word Perplexity). The third column outlines the benchmarks used for each task.
Claim: In particular, LISA requires much less activation memory consumption than LoRA since it does not introduce additional parameters brought by the adaptor.
Context: In Figure 3 , it is worth noticing that the memory reduction in LISA allows LLaMA-2-7B to be trained on a single RTX4090 (24GB) GPU, which makes high-quality fine-tuning affordable even on a laptop computer.
Caption: Figure 3: GPU memory consumption of LLaMA-2-7B with different methods and batch size 1.
Claim: The results showed that there were decreases in Gram-positive bacteria, facultatively anaerobic bacteria and stress tolerance ability, but increases in Gram-negative bacteria, anaerobic respiration, biofilms forming and potential pathogenicity in clinical samples compared with cultured samples under aerobic and anaerobic conditions ( Figure 4 ).
Context: To assess the predicted function of the bacterial community in clinical samples and the cultured bacteria samples, microbial phenotype prediction was evaluated by BugBase followed by the Mann– Whitney U test.
Caption: Figure 4: BugBase analysis among different groups.
The three lines from the bottom up are the lower quartile, the mean and the upper quartile in BugBase analysis.
Claim: The reason is that the model struggles in distinguishing between refutes and NEI claims in these datasets, as reflected by Table 5 .
Context: Many works Jiang et al. ( 2020 ); Saakyan et al. ( 2021 ) do not consider NEI claims due to their ambiguity. To explore whether our previous observations also hold for the task of binary fact verification , we evaluate the generalization results for all 11 datasets using only the supports and refutes claims for training and evaluation, shown in Table 3 . In this setting, artificial claims also generalize well to natural claims in other domains. In 6 of the 7 datasets with natural claims, the best generalization score is from a model trained on artificial claims. This also holds for the evidence length: datasets with sentence-level evidence tend to generalize better than document-level datasets. Finally, compared with the three-class result in Table 2 , generalization improves a lot on Climate-FEVER, SciFact, and PubHealth.
Caption: Table 5: Class-wise F1 of 3-class fact verification for the zero-shot generalization setup (left) and the in-domain training setup (right). S: supports; R: refutes; N: NEI.
Claim: For targets GROVER_mega and GPT-2_xl, ConDA performs within 3 and 6 F1 points of the fully-supervised model.
Context: Next, we compare the performance of our model with a fully-supervised detector trained on the target domain in Table 3 . For ConDA, we show the test performance for all target-source pairs. For the supervised model, we use a pre-trained RoBERTa (roberta-base) fine-tuned on the target data. We then evaluate the model on the test set of the same target domain, and essentially this is our upper bound performance. ConDA achieves test performance comparable to fully-supervised models. In particular, for targets CTRL and XLM, ConDA (with GROVER_mega as source) achieves upper bound performance.
Caption: Table 3: Performance of our ConDA model on each of the target domains, with each of the other domains as source. Numbers in bold are the best performing ConDA models for each target domain, i.e. closest to fully supervised performance.
Claim: Overall, the CDD values reported in Table 2 are relatively low (all below 0.006), as indicated by prior studies Wachter et al. ( 2021 ); Koumeri et al. ( 2023 ); Wachter et al. ( 2020 ) , suggesting minimal gender-based disparity across the different classes.
Caption: Table 2: One-vs.-rest CDD values for each class across four conflicts.
Claim: Additionally, few-shot learning and larger LLMs demonstrate notable advantages over their smaller counterparts.
Context: (1) Fine-tuned language models consistently outperform generic LLMs, the performance gap can be narrowed through prompt design, few-shot learning, and model size. Table 1 demonstrates that fine-tuned language models surpass generic LLMs in zero-shot direct prompting. However, the performance of generic LLMs improves significantly with diverse zero-shot prompting styles, surpassing the prompt designs proposed by Shah et al., 2023 .
Caption: Table 1: Performance of different fine-tuned language models and LLMs under different prompts on FiNER-ORD task.
Claim: Results can be found in Table 3 . Unsurprisingly, more samples seems to improve calibration, though it seems on the unambiguous Natural Questions slice, sampling diversity with a small number of samples works relatively well for the cost.
Context: As sampling-based approaches require a linear increase in compute for the number of samples, we also examine how calibration scales with the number. In particular, we test three, five, and eight samples and compare that to the original results containing ten samples.
Caption: Table 3: Calibration by number of samples (N). EM is excluded, because it is solely based on the greedy answer and does not depend on the number of samples.
Claim: Additionally, those in NEMM displayed an increased relative abundance of protein functions linked to defense metabolism, inorganic ion transport and metabolism under anaerobic conditions.
Context: To assess the effects of the two media on the predicted gene categories (COGs), we compared the predicted COGs between aNEMM and aSHI, nNEMM and nSHI ( Figure 8A ). NEMM showed more ability in carbohydrate transport and metabolism than SHI medium under both anaerobic conditions and normoxia conditions.
Caption: Figure 8: Function prediction by PICRUSt2.
The differential analysis of COG function (A) and KEGG pathway (B) show the abundance ratio of different functions between groups, whose middle figures show the difference ratio of functional abundance within the 95% confidence interval.