RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: For SU kinematics, C4 exhibited the largest horizontal displacements for the pelvis, femur, knee, and lower leg, all significantly higher than other clusters ( p < 0.001).
Context: To validate the quality and distinctiveness of the Louvain clustering solution, we calculated widely-used cluster validity indices. The Davies–Bouldin index was 2.09, and the Calinski–Harabasz index was 59.26. While the Davies–Bouldin index was somewhat high (lower values indicate better separation), the Calinski–Harabasz index supported the presence of distinct groupings in our data. The Louvain clustering algorithm identified four distinct clusters (C1–C4) with significant differences in all features (all p < 0.001, except for AHD-SU ( p = 0.017) and portion of EOA ( p = 0.019)) ( Table 2 ). Tukey’s post-hoc analysis revealed that those in C2 had the youngest age (44.80 ± 11.30 years, p < 0.001 vs. all other clusters) and highest BMI (25.59 ± 2.42, p < 0.001 vs. all other clusters). C2 included exclusively males, whereas the other clusters had a mixed sex composition.
Caption: Table 2: Comparisons of features between the clusters classified by Louvain clustering unsupervised machine learning.
Claim: Of the 19 most predominant bacterial genera found in NEMM and SHI media, Ligilactobacillus , Limosilactobacillus , Lactobacillus , and Desemzia showed the highest abundance levels.
Context: Differential predominant bacteria between NEMM and SHI were analyzed in the relative abundance of bacteria at the genus level by ANOVA.
Caption: Figure 6: The relative abundance of differential predominant bacteria between NEMM and SHI analyzed at genus level by ANOVA.
Claim: Most non-smokers ( n = 161, 85.2%) never had the urge to smoke upon the sight of pictorial warnings on the cigarette packaging.
Context: Figure 3 shows the impact of pictorial warnings on the initiation of smoking by non-smokers.
Caption: Figure 3: Impact of pictorial warnings on the initiation to smoking by non-smokers (n = 189).
Claim: Across 57 translation directions, chrF2 improves by 1.3 (M2M-100) and 1.1 (SMaLL-100) points with source-contrastive decoding.
Context: We report results using source-contrastive decoding ( C_{src} ), and combining source-contrastive and language-contrastive decoding ( C_{src+lang} ) in Table 1 . 10 10 10 See Appendix A for full results.
Caption: Table 1: Automatic evaluation results. Averages over different sets of translation directions.
Claim: While Flan-T5 introduces more errors than GPT-3.5 overall, the trends are analogous.
Context: Figure 1(a) shows the average proportion of sentences marked as inconsistent (with respect to the corresponding input) in summaries generated by GPT-3.5 Brown et al. ( 2020 ) and Flan-T5 XL Chung et al. ( 2022 ) for three domains: News, medical, and legal. Perhaps surprisingly, we observe a higher prevalence of
inconsistencies for news articles, as compared to the specialized domains of medicine and law.
Caption: (a) Prevalence of factual errors in each of domains; (b) Distribution of error categories across domains; Distribution of errors and error categories across domains
Claim: (1) Fine-tuned language models consistently outperform generic LLMs, the performance gap can be narrowed through prompt design, few-shot learning, and model size.
Caption: Table 1: Performance of different fine-tuned language models and LLMs under different prompts on FiNER-ORD task.
Claim: As expected, p(\boldsymbol{x|y}) shows a higher correlation with accuracy than fluency, and p(\boldsymbol{y}) shows the opposite pattern.
Context: A similar conclusion is suggested by Figure 5 , which shows Pearson correlations of translation probability ( p(\boldsymbol{y|x}) ), \text{accuracy}_{M} ( p(\boldsymbol{x|y}) ) and \text{fluency}_{M} ( p(\boldsymbol{y}) ) with human ratings of accuracy and fluency for RLTC and MTMQM. 9 9 9 Values are ranked by percentile.
Caption: Figure 5: \text{accuracy}_{M} and \text{fluency}_{M} predict human accuracy and fluency ratings for RLTC and WMT submissions to the general translation task in 2022 and 2023. zhen and ende refer to Chinese-English and English-German language pairs. All correlations reported are significant ( p<.001 ).
Claim: These findings demonstrate that BiAMIL is a more accurate and reliable method for estimating the BRCA status compared to the traditional ResNet 34 model.
Context: The BiAMIL model was developed by first inputting normalized tiles into the ResNet 34 model to extract 1,000-dimensional features. These features were then aggregated using a Bi-directional self-attention mechanism approach. Subsequently, the aggregated feature matrix was fed into a multilayer perceptron (MLP) classifier to predict the probability of BRCA gene mutation status for each WSI ( Figure 3 ). BiAMIL achieved AUC values of 0.819 (95% CI [0.673–0.965]) in the internal test set ( Figure 4A ), and 0.817 (95% CI [0.712–0.923]) in the external test set ( Figure 4B ). The five-fold cross-validation results for the BiAMIL model on the training set are displayed in Table S1 . The model was compared with the ResNet 34 by Wang et al. (2021) . ResNet 34 achieved AUC values of 0.783 (95% CI [0.624–0.941]) the internal test set ( Figure 4C ), and 0.684 (95% CI [0.559–0.810]) in the external test sets ( Figure 4D ). The five-fold cross-validation results for the ResNet 34 model on the training set are shown in Table S2 . In the internal test and external test sets, BiAMIL outperformed the ResNet 34 model in accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 score ( Figs. 4E and 4F ). Due to hormone receptor-positive and triple-negative breast cancer accounting for 80% of breast cancer cases and the high occurrence of BRCA1 in triple-negative breast cancer, a subgroup analysis of BiAMIL’s performance in these subtypes was conducted, with results shown in Table 2 .
Caption: Table 2: The performance of the BiAMIL in predictingBRCAPV in hormone receptor + and triple-negative breast cancer.
Claim: No study demonstrated sufficient validity for TMS biomarker use ( Bielekova & Martin, 2004 ).
Context: See Table 6 for a condensed summary of biomarker assessment findings and Table S6 for more detailed results.
Caption: Table 6: Biomarker assessment.
Claim: Our findings indicate that regression performs better for binary classification, while classification leads to better results for 3-way classification.
Context: In Table 5 we present the average performance across the four language groups to compare the effectiveness of lexicon-based pretraining in regression and classification tasks for both binary and 3-way classification.
Caption: Table 5: Regression vs. classification in lexicon-based pretraining for zero-shot sentiment analysis.
Claim: ReDial and INSPIRED contain 6,637 and 1,546 unique items in total ( Table 1 ) and 1,872 and 264 items in the test set, respectively.
Context: We follow the common practices Yang et al. ( 2022 ); Wang et al. ( 2022c ) to evaluate PECRS on both recommendation performance and response generation quality. For recommendation subtask, we measure recall with Recall@K (R@K) metric, taking K\in\{1,10,50\} . In order to assess the recommendation coverage, we also report the number of different items predicted by the model over the test set, denoted as Unique .
Caption: Table 1: Statistics on ReDial and INSPIRED datasets, combined over train, dev and test sets.
Claim: In contrast, translations into Marathi show the lowest scores, ranging from 26.5 to 29.8, likely due to the complexity of translating between less common language pairs.
Context: Translation Quality : Table 3 shows translation scores from the Hebrew Old Testament, Greek Old Testament, and Greek New Testament into five target languages. English and Turkish consistently achieve the highest scores across all manuscripts, with English translations ranging from 61.2 to 72.6, and Turkish from 65.4 to 68.2, suggesting strong translation quality for these language pairs.
Caption: Table 3: COMET scores . Top-scoring translation for each source manuscript is in bold text. Second top-scoring translation is in italics.
Claim: The open-source model Llama-3.1 70B performs competitively on the answerable questions achieving 76.31% accuracy, only 2.9% lower than Claude 3.5 sonnet.
Context: Table 5 shows our baseline method’s (DIN-SQL) performance on SQL prediction of various LLMs given the interaction between the user and the assistant. Overall, Mixtral-large-v2 and Claude 3.5 Sonnet achieve the highest average accuracy of 71.95% and 72.15% on the ambiguous/unanswerable questions. Claude 3.5 sonnet achieves the highest performance of 79.21% on the answerable questions (original Spider dev set).
Caption: Table 5: Execution accuracy of SQLs predicted with DIN-SQL using different LLMs on various categories of ambiguous, unanswerable, and answerable questions. The "All" column shows the overall average accuracy across all categories, while the "Avg. Excluding Answerable" column shows the average accuracy excluding the answerable questions from the Spider dataset.
Claim: As shown in Figure 8 , the integrated model achieved an AUC of 0.838 compared with 0.654, 0.768, and 0.893 for clinicians D1, D2, and D3, respectively, suggesting differences between the model and the expert evaluation.
Context: The performance of the integrated model was compared with that of clinical experts. Three clinicians with varying levels of experience in oral diseases, i.e ., less than 5 years (D1), less than 10 years but longer than D1 (D2), and more than 15 years (D3), were asked to interpret the imaging and clinical data. The decision of the three clinicians was compared with that of the integrated model by plotting ROC curves.
Caption: Figure 8: The ROC curves of the integrated model and diagnostic results of each radiologist in the test cohort.
The integrated model reached a high AUCs (AUC: 0.838), doctor1 acquired 0.654 of AUC, the AUC of doctor2 was 0.768 and doctor3 demonstrated the highest with 0.893.
Claim: When focusing on oscillatory hallucinations according to TNG in Table 4 , the improvement is even more pronounced,
with a reduction from 9.3% to 0.7% (-92%) for M2M-100, and from 5.9% to 1.5% (-75%) for SMaLL-100.
Context: The proportion of translations with chrF2 below 10 is shown in Table 3 . We observe large reductions in the number of defect translations, with a reduction from 7.3% to 1.2% (-83%) for M2M-100, and from 5.6% to 1.8% (-67%) for SMaLL-100.
Caption: Table 4: Proportion of translations with oscillatory hallucinations according to TNG.
Claim: In the experiments of LiPopt, we only used LiPopt-2.
Caption: Table 1: \ell_{\infty} -FGL estimation of various methods: DGeoLIP and NGeoLIP induce the same values on two layer networks. DGeoLIP always produces tighter estimations than LiPopt and MP do.
Claim: However, SpeechSim finetuned on EmoV gets a Spearman correlation equal to 0.321.
Context: To make sure that our models can measure diversity of emotions beyond that is covered by the (limited) label sets of EmoV and Expresso, we run an experiment where the projection model is trained on one dataset and tested on another (the only common label is “neutral”). Table 3 shows that SpeechSim finetuned on Expresso does transfer to EmoV classes as it obtains a high Spearman correlation.
Caption: Table 3 : Emotion diversity. Average Spearman correlations between the classes entropy in EmoV and Expresso and diversity scores induced by the speech representations. SpeechSim/Emotion-Expresso (resp. Speech/Emotion-EmoV) refers to SpeechSim emotion head trained on Expresso (resp. EmoV).
Claim: NEMM showed more ability in carbohydrate transport and metabolism than SHI medium under both anaerobic conditions and normoxia conditions.
Context: To assess the effects of the two media on the predicted gene categories (COGs), we compared the predicted COGs between aNEMM and aSHI, nNEMM and nSHI ( Figure 8A ).
Caption: Figure 8: Function prediction by PICRUSt2.
The differential analysis of COG function (A) and KEGG pathway (B) show the abundance ratio of different functions between groups, whose middle figures show the difference ratio of functional abundance within the 95% confidence interval.
Claim: Both PECRS-small and -medium surpass all baselines over Dist@3 and Dist@4.
Context: Table 3 summarizes the results on conversation task, where PECRS achieves promising performance on both types of metrics.
Caption: Table 3: Results of conversation task compared with the state-of-the-art on ReDial.
Claim: In the case of the asymmetries, reliability was ‘excellent’ for the active range of motion in flexion (ICC: 0.96); ‘good’ for the myotonometry parameters (ICC: 0.72–0.78), the Nine-Hole Peg test scoring (ICC: 0.72), the total and extension active range of motion (ICC: 0.73–0.89), and the maximal isometric strength (ICC: 0.73); and ‘moderate’ for the pressure pain thresholds on both points (ICC: 0.53–0.58).
Context: Table 3 presents ICC 3,1 , SEM, MDC 90 , and MDC 95 for each variable. For the muscle mechanical properties of the dominant and non-dominant sides, reliability was ‘excellent’ for the frequency and stiffness parameters (ICC: 0.91–0.96), and ‘good’ to ‘excellent’ for the decrement (ICC: 0.86–0.91). For the manual dexterity, reliability was ‘good’ on both sides (ICC: 0.79–0.82). For the pressure pain thresholds, reliability was ‘excellent’ for the extensor carpi radialis brevis point (ICC: 0.90–0.93), and ‘good’ for the deltoideus point (ICC: 0.84–0.89). For the active range of motion, reliability was ‘excellent’ for both extension, flexion, and total range (ICC: 0.94–0.97). Similarly, reliability was ‘excellent’ for the maximal isometric strength on both sides (ICC: 0.97).
Caption: Table 3: Reliability indicators of myotonometry, manual dexterity, pressure pain thresholds, active range of motion, and maximal isometric strength for the wrist extensor muscles.
Claim: The first source “segment” \boldsymbol{x} (leftmost panel) has relatively high probability p(\boldsymbol{x}) , and no tradeoff is observed in this case.
Context: We initially assume that both \boldsymbol{x} and \boldsymbol{y} are one-dimensional vectors. Figure 2(a) shows the relationship between p(\boldsymbol{y}) and p(\boldsymbol{x|y}) for 3 samples \boldsymbol{x} . Each point in each panel corresponds to a candidate translation \boldsymbol{y} , and candidates with highest p(\boldsymbol{y|x}) are shown in yellow. The correlation above each panel results from applying Equation 1 to all translations with p(\boldsymbol{y|x}) above the 90th percentile (i.e. all points in the brightest part of each plot).
Caption: (a) Simulation with one-dimensional \boldsymbol{x} and \boldsymbol{y} . The three panels correspond to three different source “segments” \boldsymbol{x} of decreasing probability p(\boldsymbol{x}) , and the points in each panel are candidate translations \boldsymbol{y} . Brighter colors indicate translations with larger p(\boldsymbol{y|x}) . Pearson correlations across translations ranked in the top 10% based on p(\boldsymbol{y|x}) are shown at the top of each panel.; (b) Kernel density plots of tradeoffs across the top 10% (left) and across all translation choices (right). The tradeoff persists in higher dimensional space, and is stronger when selecting only \boldsymbol{y} with the highest values of p(\boldsymbol{y|x}) .; Tradeoffs between p(\boldsymbol{x|y}) and p(\boldsymbol{y}) in synthetic data.
Claim: Pregnant women who were advised to terminate their pregnancies or carefully consider whether to continue gestation were 41.113 times more likely to undergo labor induction compared to those who were advised to continue the pregnancy (OR = 41.113, 95% CI [11.028–153.267]), ( Table 2 ).
Context: Pregnant women with multiple fetal abnormalities were 3.774 times more likely to undergo labor induction than those with a single fetal abnormality (OR = 3.774, 95% CI [1.640–8.683]).
Caption: Table 2: Analysis of influencing factors of pregnancy outcome in pregnant women with fetal abnormalities.
Claim: Under normoxic culture conditions, there were no significant differences in Gram-positive bacteria among the four groups (aNEMM, aSHI, nNEMM and nSHI).
Context: Bugbase was used to predict the function of bacterial community in NEMM and SHI under anaerobic and normoxia culture conditions. The results showed that the microbiota developed in NEMM were predicted stronger anaerobic ability, whether under anaerobic conditions or normoxic conditions.
Caption: Figure 7: BugBase analysis among different groups.
Note: X-axis, group name; Y-axis, Relative abundance in percentage. The three lines from bottom to top are lines indicating lower quartile, average and upper quartile.
Claim: Finally, the original SpeechSim embeddings have negative correlation with the gender diversity.
Context: We report results in Table 6 . From this results we see that SpeechSim/Gender is positively correlated with the gender diversity. In contrast, the SpeechSim/Voice metric tracks the speaker diversity and hence is negatively correlated with the gender diversity.
Caption: Table 6 : Changing voice and gender diversity in the opposite directions. Average Spearman correlation between the gender diversity and the diversity scores induced by speech representations.
Claim: With task-specific encoders (right), the cross-task class centroids are further apart than those with one global encoder (left).
Context: where \{z^{t}_{i}\}_{m=1}^{M} are the M task-specific MC samples. Task-specific encoders help us sample latent variables that are more discriminative across tasks. This is depicted in Fig. 3 using the cosine distance between the embeddings of class-specific samples.
Caption: Figure 3 : Need for task-specific distribution encoders: Cosine distance between the centroids of class-specific latent variables produced without ( left ) and with ( right ) the use of task-specific encoders on CIFAR100 (10 tasks, 10 classes per task). Centroids of class-specific latent variables are more separable on the right.
Claim: Overall, we observe that multilingual models fine-tuned on SST tend to perform the best in high/medium-resource languages, with mBERT {}_{\text{Base}} being the exception.
Context: Table 4 presents the averaged zero-shot performance of all models categorized by four language groups. The reported results use regression and classification in lexicon-based pretraining for binary and 3-way classification, respectively. In the case of high/medium-resource languages (HM-R), English is excluded to ensure a fair comparison with models fine-tuned on the English SST dataset.
Caption: Table 4: Full zero-shot results. The underlined score indicates the highest performance within the respective group, while scores in bold indicate the best global performance. “HM-R” = high/medium-resource languages, excluding English, “CS” = code-switched text, and “ML Lex.” indicates the multilingual NRC-VAD lexicon. “SST (sentence-level data)” is cross-lingual zero-shot transfer that is trained on English sentence-level sentiment data.
Claim: Table 4 shows that the evaluation on MNLI and QQP is more robust to different settings, and the variance is more significant on CoLA.
Context: In this section, we first compare the influence of different calibration sizes on FPQ . We vary the calibration size in \{32,64,128,256\} and test on MNLI, QQP, and CoLA.
Caption: Table 4: Ablation studies of different calibration sizes.
Claim: Overall, the cognitive abilities of OPT, Llama-2-chat-70B, GPT-3.5-Turbo, and GPT-4 models successively increase, and the performance of each model gradually declines with the increase of stage, consistent with humans.
Context: As shown in Table 3 , We run the model with the largest number of parameters in each series on CogLM , and report the adult human performance for comparison.
Caption: Table 3: Calibrated accuracy (%) of largest model in evaluating series. Acc and Age refer to calibrated accuracy and the age of equivalent human performance. The value of Age is calculated according to Equation 2 . Bold indicates the best performance.
Claim: Moreover, reflection does not always yield perfect outcomes; the model can repeat decisions from previous cycles despite being instructed not to.
Context: Table 2 shows common decision paths during reflection. Interestingly, reflection can lead to abrupt decision changes, such as a direct jump from Buy to Strong Sell, instead of gradual shifts (e.g., Buy \rightarrow Hold \rightarrow Sell).
Caption: Table 2: The most common decision paths during reflection and their percentages in the training data.
Claim: It can be seen that in both configurations, KL-divergence yields lower correlations than other distances, and on average total variation slightly outperforms Hellinger and one-best distances.
Context: In Table 3 , our results compare statistical distances.
Caption: Table 3: Comparison of Statistical Distances using MQAG-Sum without answerability.
Claim: We note that while DISTANCE and LABEL show similar performance, LabDist in general is the most performant and consistent classifier.
Context: We perform extensive experiments over the different LaGoNN configurations.
Caption: Figure 2: First row: performance for all LaGoNN configurations and balance regimes for the Hate Speech Offensive dataset. Second row: LaGoNN performance for one to five neighbors for all balance regimes on a collapsed version of the LIAR dataset. We use the LaGoNN lite fine-tuning strategy (see Section 5.1 ).
Claim: Functional enrichment analysis of the upregulated DEGs revealed that the top five biological processes were skeletal system development, sister chromatid cohesion, ossification, mitotic nuclear division, and extracellular matrix organization ( Figure 3A ).
Context: GO annotation and KEGG pathway enrichment analyses were performed to better understand the functional significance of the DEGs.
Caption: Figure 3: GO and KEGG pathway enrichment analyses of upregulated genes.
GO functional classification of DEGs. The x-axis represents the number of DEGs, with individual GO terms plotted on the y-axis. The graph displays only significantly enriched GO terms (P < 0.05). All GO terms were grouped into three categories: (A) BP analysis, (B) CC analysis, and (C) MF analysis. (D) KEGG pathway analysis. The x-axis represents the number of DEGs annotated in a pathway, with individual KEGG terms shown on the y-axis. The graph displays only significantly enriched KEGG terms (P < 0.05).
Claim: Clinical isolates presented the higher resistance (100%) to the three antibiotics: imipenem, meropenem, and piperacillin/tazobactam.
Caption: Figure 3: Clinical and environmental isolates susceptibility profile.
The percentages of clinical isolates that were resistant, intermediate, or sensitive to each antibiotic. (A) Clinical isolates susceptibility profile, (B) river isolates susceptibility profile. Acronyms: CIP, ciprofloxacin; GEN, gentamycin; SXT, trimethoprim/sulfamethoxazole; AMK, amikacin; FEP, cefepime; MEN, meropenem; IMP, imipenem; TOB, tobramycin; SAM, ampicillin/sulbactam; CAZ, ceftazidime; TZP, piperacillin/tazobactam; R, resistant; I, intermediate; S, sensitive.
Claim: Notably, AgA with \lambda=0.1 underperforms, even when contrasted with AgA’s ablation study.
Context: Results. Fig. 5(a) compares average win rates for various \lambda parameters in AgA, also encompassing AgA without sign selection, as stated in Corollary 4.3 . The results suggest comparable performance for \lambda=1 and 100 , with a lower value of 0.1 trailing noticeably.
Caption: (a) MMM2: AgA with various \lambda and AgA/Sign; (b) MMM2: comparison between AgA and baselines; The comparison of win rates on the mixed-motive MMM2 map of SMAC.
The left figure shows results of varying alignment parameter \lambda values in AgA, as well as AgA sans sign selection (AgA/Sign),
The right figure compare the win rate of AgA (\lambda=100) with baselines.
The bold lines illustrate the mean of the win rate over 3 seeds, while the encompassing shaded regions depict the standard deviation.