RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: There are more extrinsic errors introduced in the news domain compared to the niche domain datasets.
Context: We next characterize the distribution of error categories in factually inconsistent summaries generated by models across the domains considererd. Figure 1(b) reports the distribution of error categories for both models. 3 3 3 Model-specific distributions are in Appendix A.6
Caption: (a) Prevalence of factual errors in each of domains; (b) Distribution of error categories across domains; Distribution of errors and error categories across domains
Claim: The model has a slight edge in performance on the sarcasm and visual metaphor subsets of the task, perhaps due to difficulty of these subsets and any potential spurious correlations during fine-tuning.
Context: To find out how humans perform on the task, we hire two expert annotators with formal education in linguistics. We present them with 10 example instances and then ask them to complete 99 randomly sampled test set instances. We also evaluate our best model (see Table 3 ) on the same set. Results are shown in Table 4 . Human performance is quite strong, almost reaching 90 F1@0 score overall. Human performance is better than our strongest fine-tuned model (LLaVA-7B-eVil+VF) performance with p<0.05 for Annotator 1 and p<0.07 for Annotator 2. Humans excel at interpreting memes, with both annotators reaching a 100% F1 score. Humans also perform noticeably better on the NYCartoons dataset and on the idiom subset of the task.
Caption: Table 4: Human baseline results (F1@0) by phenomenon and source dataset.
Claim: Kaplan-Meier analysis implicated that patients in the cluster two group had worse PFS outcomes compared with those in the cluster 1 group, in which the log-rank test P value was 0.013 ( Figure 5K ).
Context: Utilizing the unsupervised clustering algorithm, we conducted consensus clustering analysis to distinguish PRAD patients in the training cohort into subgroups based on the expression of ARGs. The K = 2 was identified with the optimal clustering stability ( Figs. 5G – 5I ). We further conducted the PCA analysis and could classify the patients into two distinct groups with individual features (cluster 1 & cluster 2) in Figure 5J .
Caption: Figure 5: External validation of ARGs signature and clustering analysis across PRAD samples.
 (A) The ROC analysis of ARGs exhibiting its predictive efficiency of 3-, 5- and 7-year in predicting PFS. (B) Kaplan-Meier curves showing the differential survival outcomes in AGRs-high and ARGs-low samples in GSE116918. (C and D) The ROC analysis and Kaplan-Meier curves of ARGs in GSE70769. (E and F) The ROC analysis and Kaplan-Meier curves of ARGs in MSKCC-PRAD cohort. (G) Color‐coded heatmap corresponding to the consensus matrix for k = 2 obtained by consensus clustering. The color gradients from 0 to 1 represent the degree of consensus, with white corresponding to 0 and dark blue corresponding to 1. (H) Consensus among clusters for each category number k. (I) Delta area curve of consensus clustering indicating the relative change in area under the cumulative distribution function (CDF) curve for each category number k compared to k − 1. (J) PCA analysis indicated the identified two groups based on ARGs signature. (K) Kaplan-Meier analysis indicated the differential survival outcomes of two groups. (L) Gene set enrichment analysis (GSEA) exhibiting the enriched crosstalk between Cluster1 and Cluster2.
Claim: Table 1 shows that our probabilistic inference module consistently outperforms its deterministic counterpart in terms of Avg and Last accuracy.
Context: To understand our probabilistic inference modules further, we examine their performance against the deterministic variant of ours (Ours w/o VI).
Caption: Table 1 : Performance comparison of different methods averaged over three runs. Best scores are in bold . Second best scores are in blue . The results for L2P, DualPrompt, and PROOF are taken from [ 92 ] . See App. Table 14 for std. dev. scores.
Claim: The peak angular momentum of the non-attack leg (−0.026 ± 0.012 s –1 ) exhibited negative correlations with the arm swing speed at BI ( Table 1 ).
Context: In the later part of the arm-cocking phase, the angular momentums of the attack and non-attack legs simultaneously rotated clockwise (−).
Caption: Table 1: Peak angular momentum variables and correlations with the arm swing speed at ball impact (BI).
Claim: From Fig. 5 -right, extracting 4 categories in the Hard task shows the largest performance gap between mapping formats.
Caption: Figure 5: Average estimated true F1 scores ( Section 3.1 ) across models (left) and benchmarks (right) showing performance bias of LLMs across 2 widely used mapping formats.
Claim: Among tumor region samples, the model correctly identified 86% as tumor, while misclassifying 14% as non-tumor.
Context: A segmentation network was developed to accurately distinguish tumor regions from WSIs. This model demonstrated high efficacy, achieving an AUC of 0.960 (95% CI [0.959–0.961]). The accuracy, sensitivity, and specificity were 0.888 (95% CI [0.887–0.890]), 0.859 (95% CI [0.856–0.861]), and 0.908 (95% CI [0.906–0.909]), respectively. Compared with three pathologists of varying experience (junior, medium, and senior), the model’s performance was almost equivalent to that of a medium-level pathologist ( Figure 2A ). The confusion matrix shows specific performance metrics of the segmentation model.
Caption: Figure 2: Performance of the segmentation network.
(A) Receiver operating characteristic (ROC) curve comparing the performance of the segmentation network with three pathologists (senior, medium, and junior) in identifying tumor areas. (B) Confusion matrix for the segmentation network. (C) Original WSIs and annotation maps identified by the segmentation network. Left: original WSI images; right: corresponding annotation maps.
Claim: The remaining prompts types ( Discourse, Mask ) are comparable to Corpus .
Context: To further analyze the outputs, we calculate the perplexity (PPL) of the generated predictions to determine their plausibility (Wilcox et al., 2020 ) . Here, we choose the model with the best \textit{WHR}_{5} score on the MKR-NQ benchmark, and calculate the mean perplexity over all queries for each prompt type (5 completions for each query). PPL is calculated as the exponentiated average negative log-likelihood of a sequence, with exponent base e . As a point of reference, we calculated the average perplexity of the provided completion of the original non-negated dataset (denoted Corpus ). From the reported perplexities ( Table 3 ), we can see that Default output are the most plausible (with PPL markedly lower than Corpus ), while Contrasting is the least natural.
Caption: Table 3: Mean perplexity (PPL) calculated using the GPT-J-6B model. Only the strings enclosed in square brackets are considered during calculation in order to provide a fair comparison with similar token length. For Corpus, PPL is calculated using the provided gold completion.
Claim: It follows that 12 out of 189 rarely and sometimes (6.3%) had the urge to smoke, respectively.
Context: Figure 3 shows the impact of pictorial warnings on the initiation of smoking by non-smokers. Most non-smokers ( n = 161, 85.2%) never had the urge to smoke upon the sight of pictorial warnings on the cigarette packaging.
Caption: Figure 3: Impact of pictorial warnings on the initiation to smoking by non-smokers (n = 189).
Claim: Phascolarctobacterium was positively associated with 1,3-propanediol, L-proline, cadaverine, and DL-phenylalanine, and was negatively correlated with glycerol, CO 2 , 4-aminobutanoic acid, L-cysteine, and myo-inositol.
Context: To further explore correlations of metabolites in human fecal matter and intestinal flora after SNP treatment, thermography was used to analyze contents of metabolites showing significant differences with 30 most abundant genera. Figure 10 shows the results. Bifidobacterium was positively associated with glycerol, CO 2 , 4-aminobutanoic acid, L-cysteine, and D-gluconic acid, and could be negatively correlated with DL-phenylalanine. Faecalibacterium was positively associated with pentanoic acid and 4-aminobutanoic acid, while were negatively correlated with hydracrylic acid. Lactobacillus was positively correlated with xylitol and showed a negative correlation with 5-aminovaleric acid. Dorea was positively associated with phenylalanine, butyric acid, and isovaleric acid, while showed a negative correlation with hydracrylic acid, CH 4 , myo-inositol, L-threonine, glycerol, CO 2 , L-cysteine, and D-gluconic acid. Escherichia - Shigella could be positively associated with H 2 S, L-proline, cadaverine, DL-phenylalanine, benzeneacetic acid, 1,3-propanediol, and were negatively correlated with glycerol, CO 2, 4-aminobutanoic acid, L-cysteine, D-gluconic acid, propionic acid, and L-lysine.
Caption: Figure 10: Correlation analysis of characteristic metabolites and bacteria.
Corr represents correlation. Red and blue separately denote positive and negative correlations. The darker the color, the stronger the correlation. *P < 0.05, **P < 0.01, and ***P < 0.001.
Claim: Finally, on comparing average cosine vs. Vendi Score aggregation, we observe that often Vendi Score shows higher correlations.
Context: From Table 1 we firstly notice that, across all columns, SpeechSim shows higher correlation scores than off-the-shelf embedding models and the models trained from scratch. This supports our decision to rely on SpeechSim as a basis for a per-facet models. In turn, further voice specialization of SpeechSim (SpeechSim/Voice) results in similar scores.
Caption: Table 1 : Voice diversity. Average Spearman correlations ( \pm standard error) between number of distinct speakers and diversity scores induced by speech representations. Male (Female) refer to samples with male-only (female-only) voices. Best results are in bold, second best results are underlined.
Claim: We report labeled F1-score averaged on 100 fine-tuning runs in Table 2 .
Precision and recall are reported in Appendix B .
The Base model (using the full 12 layers) produces similar results on DOCILE no matter if pre-training on IIT-CDIP or DOCILE .
Caption: Table 2: F1 scores for named-entity recognition using different pre-training and fine-tuning datasets. Results are averaged on 100 runs with different seeds.
Claim: We can observe that classification errors rise with increasing noise levels and region sizes.
Context: To investigate the trade-off between the shift scope (dense vs. sparse) and severity, we simulate different levels of corruption severity and corrupted region sizes and evaluate a classical TTA method TENT [ 15 ] on these configurations. Following [ 45 ] , we inject impulse noise to the CIFAR10 dataset, with noise levels ranging from 1 to 10 to simulate various severity levels. To control the shift’s scope, we crop regions of various sizes and introduce corruption only to this region. Figure 3 displays classification error curves under various shift severity levels and region sizes.
Caption: Figure 3 : TTA classification errors under different levels of shift severity levels and scopes.
Claim: As shown in Table 6 , in most cognitive abilities, COC can significantly improve the performance of Llama-2-chat-7B.
Context: Although there is still room for improvement, the cognitive abilities of advanced LLMs have approached levels close to that of adult humans as discussed in Section 3.2 . A natural question is, what are the potential applications for advanced LLMs’ cognitive abilities? When humans address cognitive questions, they deduce and provide answers based on their cognitive abilities. While we have demonstrated in Section 3.3.2 that the cognitive chain-of-thought (Chain-of-Cognition, COC) generated by LLMs barely help self-address cognitive questions, we are curious whether COC can assist early-aged LLMs in improving cognitive performance. On this basis, we use the question together with the COCs generated by advanced LLM (GPT-3.5-Turbo) as input to test the performance of early-aged LLM (Llama-2-chat-7B) on CogLM .
Caption: Table 6: Calibrated accuracy of Llama-2-chat-7B on CogLM with and without Chain-of-Cognition from GPT-3.5-Turbo as input.
Claim: Even for languages unseen in the pre-training corpus of XLM-R, it outperforms Glot-500 in most cases as long as the written scripts are seen.
Context: Figure 2 compares MLP, XLM-R and Glot-500 models based on language and script coverage in pre-training based on four groups: (1) language seen, script seen in XLM-R (2) language unseen, script seen in XLM-R (3) script unseen in XLM-R, language seen in Glot-500 (4) script unseen by both models. The results in each group are sorted by their performance on fine-tuned XLM-R model. Overall, XLM-R performs the best on all languages seen in its pre-training corpus without any exception .
Caption: Figure 2: Fully supervised Model Performance. We group languages by whether they and their scripts are seen in the pre-training corpus of XLM-R. Languages are ordered by the XLM-R performance in every group.
Claim: Evaluation_NS is unique in being poor, receiving an average of 2.47 .
Context: The average annotator rating of relation-fit for generation with each of the relations is presented in Table 2 . The overall average, 3.49 , is well within the positive range.
Caption: Table 2: The human-evaluation statistics for each relation, where None is generation with the language model alone. The metrics are ( Rel[ation-fit]), ( Flu[ency]), and ( Rea[sonableness]).