alabnii/sciclaimeval-shared-task huggingface dataset:
Claim: From Fig. 5 -right, extracting 4 categories in the Hard task shows the largest performance gap between mapping formats.
Evidence type: figure
Evidence caption: Figure 5: Average estimated true F1 scores ( Section 3.1 ) across models (left) and benchmarks (right) showing performance bias of LLMs across 2 widely used mapping formats.
Verdict: Supported
Claim: In contrast, BERT exhibits no preference for label orders and consistently demonstrates a uniform distribution in its predicted label indices.
Evidence type: figure
Evidence caption: (a) TACRED; (b) TACREV; (c) Re-TACRED; (d) Banking77; The distribution of predicted indices of the test instances with label shuffling before every prediction.
Context: The empirical results in Section 3.2 indicate that ChatGPT’s predictions are affected by label order. To deeper delve into the effects of label orders on ChatGPT, we analyze the distribution of predicted label indices (e.g., if the prediction is the first label, the label index is 1 ), as introduced in Section 2.2 . We visualize the distributions in Fig. 4 .
Notably, the distribution of ChatGPT’s predictions consistently deviates from the uniform distribution, displaying a consistent bias towards smaller indices across different datasets.
Verdict: Supported
Claim: We note that while DISTANCE and LABEL show similar performance, LabDist in general is the most performant and consistent classifier.
Evidence type: figure
Evidence caption: Figure 2: First row: performance for all LaGoNN configurations and balance regimes for the Hate Speech Offensive dataset. Second row: LaGoNN performance for one to five neighbors for all balance regimes on a collapsed version of the LIAR dataset. We use the LaGoNN lite fine-tuning strategy (see Section 5.1 ).
Context: We perform extensive experiments over the different LaGoNN configurations.
Verdict: Supported
Claim: While the performance of the Decider model almost overlaps with the Q Drawer asking questions in random dialogue turns, using model uncertainty significantly improves accuracy, even when a few questions are asked.
Evidence type: figure
Evidence caption: Figure 4: Effect of the average number of questions per dialogue (considering all dialogues in the test set) on the size accuracy. We compare the uncertainty-guided Q Drawer with a version that asks questions in random turns and one that asks questions in the turns selected by an external Decider model.
Context: We compare the uncertainty-driven Q Drawer against (1) a version that asks questions in random dialogue turns with different average number of questions per dialogue and (2) the Q Drawer paired with the Decider module as described in Section 6.2 . For the latter, the maximum possible number of questions per dialogue (average of 1.5) is defined by the Decider model output. Figure 4 shows the effect of the number of questions on size accuracy for different model configurations.
Verdict: Supported
Claim: While Flan-T5 introduces more errors than GPT-3.5 overall, the trends are analogous.
Evidence type: figure
Evidence caption: (a) Prevalence of factual errors in each of domains; (b) Distribution of error categories across domains; Distribution of errors and error categories across domains
Context: Figure 1(a) shows the average proportion of sentences marked as inconsistent (with respect to the corresponding input) in summaries generated by GPT-3.5 Brown et al. ( 2020 ) and Flan-T5 XL Chung et al. ( 2022 ) for three domains: News, medical, and legal. Perhaps surprisingly, we observe a higher prevalence of
inconsistencies for news articles, as compared to the specialized domains of medicine and law.
Verdict: Supported
Claim: The news domain has a higher frequency of such cases.
Evidence type: figure
Evidence caption: (a) Prevalence of factual errors in each of domains; (b) Distribution of error categories across domains; Distribution of errors and error categories across domains
Context: We next characterize the distribution of error categories in factually inconsistent summaries generated by models across the domains considererd. Figure 1(b) reports the distribution of error categories for both models. 3 3 3 Model-specific distributions are in Appendix A.6 There are more extrinsic errors introduced in the news domain compared to the niche domain datasets. We include “mixed” errors for cases where errors were classified as different types (intrinsic/extrinsic) by annotators.
Verdict: Supported
Claim: The first source “segment” \boldsymbol{x} (leftmost panel) has relatively high probability p(\boldsymbol{x}) , and no tradeoff is observed in this case.
Evidence type: figure
Evidence caption: (a) Simulation with one-dimensional \boldsymbol{x} and \boldsymbol{y} . The three panels correspond to three different source “segments” \boldsymbol{x} of decreasing probability p(\boldsymbol{x}) , and the points in each panel are candidate translations \boldsymbol{y} . Brighter colors indicate translations with larger p(\boldsymbol{y|x}) . Pearson correlations across translations ranked in the top 10% based on p(\boldsymbol{y|x}) are shown at the top of each panel.; (b) Kernel density plots of tradeoffs across the top 10% (left) and across all translation choices (right). The tradeoff persists in higher dimensional space, and is stronger when selecting only \boldsymbol{y} with the highest values of p(\boldsymbol{y|x}) .; Tradeoffs between p(\boldsymbol{x|y}) and p(\boldsymbol{y}) in synthetic data.
Context: We initially assume that both \boldsymbol{x} and \boldsymbol{y} are one-dimensional vectors. Figure 2(a) shows the relationship between p(\boldsymbol{y}) and p(\boldsymbol{x|y}) for 3 samples \boldsymbol{x} . Each point in each panel corresponds to a candidate translation \boldsymbol{y} , and candidates with highest p(\boldsymbol{y|x}) are shown in yellow. The correlation above each panel results from applying Equation 1 to all translations with p(\boldsymbol{y|x}) above the 90th percentile (i.e. all points in the brightest part of each plot).
Verdict: Supported
Claim: The tradeoff emerges, however, and becomes increasingly strong as \boldsymbol{x} moves away from the mode of the distribution p(\boldsymbol{x}) .
Evidence type: figure
Evidence caption: (a) Simulation with one-dimensional \boldsymbol{x} and \boldsymbol{y} . The three panels correspond to three different source “segments” \boldsymbol{x} of decreasing probability p(\boldsymbol{x}) , and the points in each panel are candidate translations \boldsymbol{y} . Brighter colors indicate translations with larger p(\boldsymbol{y|x}) . Pearson correlations across translations ranked in the top 10% based on p(\boldsymbol{y|x}) are shown at the top of each panel.; (b) Kernel density plots of tradeoffs across the top 10% (left) and across all translation choices (right). The tradeoff persists in higher dimensional space, and is stronger when selecting only \boldsymbol{y} with the highest values of p(\boldsymbol{y|x}) .; Tradeoffs between p(\boldsymbol{x|y}) and p(\boldsymbol{y}) in synthetic data.
Context: We initially assume that both \boldsymbol{x} and \boldsymbol{y} are one-dimensional vectors. Figure 2(a) shows the relationship between p(\boldsymbol{y}) and p(\boldsymbol{x|y}) for 3 samples \boldsymbol{x} . Each point in each panel corresponds to a candidate translation \boldsymbol{y} , and candidates with highest p(\boldsymbol{y|x}) are shown in yellow. The correlation above each panel results from applying Equation 1 to all translations with p(\boldsymbol{y|x}) above the 90th percentile (i.e. all points in the brightest part of each plot). The first source “segment” \boldsymbol{x} (leftmost panel) has relatively high probability p(\boldsymbol{x}) , and no tradeoff is observed in this case.
Verdict: Supported
Claim: Larger distances appear less frequently and hence the range of distance is smaller in the first bars, while the last bars have larger ranges.
Evidence type: figure
Evidence caption: Figure 3: Barplot where each bar represents a range of distances between events in the gold pairs. The y axis shows the F1 score of the predictions for the pairs in each bar.
Context: Since clinical notes contain long texts (see Section 4.1 ), we perform an analysis based on the distance of event pairs for the best-performing LLM (Llama CoT). First, we calculate the distance in terms of characters between the events for all the gold pairs. Then, we sort the pairs by their distances and split them to 10 bins, so that each bin contains roughly the same amount of pairs. Finally, the F1 score is calculated for the prediction of the pairs contained in each bin. Figure 3 depicts the barplot with the bars representing the pairs in the specific distance range and the their corresponding F1 scores. We observe that 37.5% of the pairs have a distance of 0 to 100 characters.
Verdict: Supported
Claim: The high bias in the SciDocsRR task is because Mistral and Gemma mostly failed to perform this task following the “Bullet” and “Special character” list formats while excelling in solving it following the other formats.
Evidence type: figure
Evidence caption: (a); (b); Average EstTrueF1 (SemEval2017) and EstTrueMAP (SciDocsRR) ( Section 3.1 ) across models (left) and benchmarks (right) showing performance difference of LLMs across 4 widely used list formats.
Context: The performance bias is regardless of the task as plotted in Fig. 4 -right, with the highest BiasF_{o} value of 67.07\%^{2} on the order list generation task SciDocsRR, and significantly lower ( 27.58\%^{2} ) on SemEval2017 task.
Verdict: Supported
Claim: Surprisingly, the Medium task displays the least bias, likely because models perform best in this task.
Evidence type: figure
Evidence caption: Figure 5: Average estimated true F1 scores ( Section 3.1 ) across models (left) and benchmarks (right) showing performance bias of LLMs across 2 widely used mapping formats.
Context: From Fig. 5 -right, extracting 4 categories in the Hard task shows the largest performance gap between mapping formats.
Verdict: Supported
Claim: By repeating the wrapping instructions of ChatGPT thrice, we observed a decrease in the BiasF_{o} (ii) score presented in Fig. 6 . Combining (i) and (ii) suggests that this strategy is an effective mitigation.
Evidence type: figure
Evidence caption: Figure 6: More demonstrations and repeating format instructions mitigate format bias. Finetuning mostly eliminates the format bias. The performance is reported using ChatGPT on MMLU (Appx.- Tab. 18 for num. results).
Context: We found that repeating instructions generally increases FI scores (i) across most formats except “Placeholder”, which can consequently lessen the mode’s token bias towards format instructions ( Section 5.1 ). Using our two proposed criteria for effective format bias mitigation in Section 3.2 , it is worth examining if this approach reduces BiasF_{o} , thereby being an effective mitigation. Our answer is yes.
Verdict: Supported
Claim: In numerical domains (Number, Number+Text), LLMs are much more accurate in QA versus RQA , revealing a clear abduction weakness.
Evidence type: figure
Evidence caption: Figure 2: LLM RQA ( blue ) and QA ( red ) accuracy with 95% CIs for metric error rate. LLMs are much weaker in abductive RQA in numerical settings (Number/Number+Text), but in text settings (Easy/Hard Entity), deductive QA is slightly weaker.
Context: We first see if RQA ( red , no stripe) or QA ( blue , striped) is consistently harder for LLMs (Figure 2 ).
Verdict: Supported
Claim: Number+Text a have lower Dolma token counts when RQA fails (Fig 4 ), so LLMs struggle to recall long-tail numerical facts Kandpal et al. ( 2023 ) .
Evidence type: figure
Evidence caption: Figure 4: Answer token count and question difficulty of when RQA succeeds/fails, averaged over all LLMs.
Verdict: Supported
Claim: Our findings show that: (1) As expected, using the most closely related language as the auxiliary example bank yields the best performance, as it provides relevant guidance to the LLM, and (2) Selecting a random or unrelated language results in little to no improvement, with performance remaining close to the relevance-based EPR baseline Rubin et al. ( 2021 ) that only uses target-language specific data.
Evidence type: figure
Evidence caption: Figure 1: Token-F1 evaluation on the cross-lingual QA task in Manipuri, with retrievers trained using different auxiliary high-resource example banks: (1) Closely related language, (2) Random language, and (3) Unrelated language.
Context: We evaluate three setups: (1) Our method (Alg. 2 ) of selecting the most closely related language as the auxiliary example bank, (2) selecting a random language, and (3) selecting the most unrelated language. We show the few-shot performance on cross-lingual QA task in Figure 1 , with retrievers trained under each setup.
Verdict: Supported
Claim: ChatGPT is affected by the primacy effect : ChatGPT tends to select labels in earlier positions in the prompt (see Fig. 1 ), which present clear bias with respect to the label order.
Evidence type: figure
Evidence caption: Figure 1: Primacy Effect of ChatGPT : ChatGPT tends to return labels in earlier positions as the answer. This plot shows the distribution of ChatGPT’s predicted label indices in TACRED (42 classes), where we randomly shuffle labels before every prediction (see Sec. 2.2 ).
Verdict: Supported
Claim: We visualize the distributions in Fig. 4 .
Notably, the distribution of ChatGPT’s predictions consistently deviates from the uniform distribution, displaying a consistent bias towards smaller indices across different datasets.
Evidence type: figure
Evidence caption: (a) TACRED; (b) TACREV; (c) Re-TACRED; (d) Banking77; The distribution of predicted indices of the test instances with label shuffling before every prediction.
Context: The empirical results in Section 3.2 indicate that ChatGPT’s predictions are affected by label order. To deeper delve into the effects of label orders on ChatGPT, we analyze the distribution of predicted label indices (e.g., if the prediction is the first label, the label index is 1 ), as introduced in Section 2.2 .
Verdict: Supported
Claim: Loss method outperformed all others in approximately one-third of the 35 settings examined, as indicated in Figure 5 (left).
Evidence type: figure
Evidence caption: Figure 5: Summary of subset selection strategies performances from. Left: percentage of times each strategy gets the best performance out of 35 settings (across each of the 7 languages and 5 \hat{D}^{Syn}_{train} sizes). Right: bootstrapped confidence intervals for the percentages on the left.
Context: Effective subsets have high diversity and predictive uncertainty . Our analysis reveals statistically significant differences between the subset selection strategies, highlighting the effectiveness of the hybrid approaches ( UMT/EMT+ Loss ) that consider both diversity and predictive uncertainty. Among the strategies tested, the UMT+
Verdict: Supported
Claim: Furthermore, selecting datapoints based solely on high predictive uncertainty without considering diversity ( HighLoss ) is an ineffective strategy, having the second lowest proportion of wins (Fig. 5 , right).
Evidence type: figure
Evidence caption: Figure 5: Summary of subset selection strategies performances from. Left: percentage of times each strategy gets the best performance out of 35 settings (across each of the 7 languages and 5 \hat{D}^{Syn}_{train} sizes). Right: bootstrapped confidence intervals for the percentages on the left.
Verdict: Supported
Claim: As shown in Figure 3 , \mathcal{Q}_{\text{All}} hardly outperforms the RAG baseline ( 64.3\rightarrow 64.5 ).
Evidence type: figure
Evidence caption: Figure 3: Comparison of different warmup strategies. Scores are reported on the StrategyQA dev set.
Verdict: Supported
Claim: As shown in Figure 4 , IAG- Student achieves the best performance with the statement number between 5 and 7.
Evidence type: figure
Evidence caption: Figure 4: Scores of IAG- Student on StrategyQA dev set with different numbers of knowledge statements.
Context: Our implementation of IAG samples 5 knowledge statements to feed into the generator. To justify this design choice, we evaluate the performance of IAG- Student with varying statement numbers.
Verdict: Supported
Claim: There are more extrinsic errors introduced in the news domain compared to the niche domain datasets.
Evidence type: figure
Evidence caption: (a) Prevalence of factual errors in each of domains; (b) Distribution of error categories across domains; Distribution of errors and error categories across domains
Context: We next characterize the distribution of error categories in factually inconsistent summaries generated by models across the domains considererd. Figure 1(b) reports the distribution of error categories for both models. 3 3 3 Model-specific distributions are in Appendix A.6
Verdict: Supported
Claim: Even for languages unseen in the pre-training corpus of XLM-R, it outperforms Glot-500 in most cases as long as the written scripts are seen.
Evidence type: figure
Evidence caption: Figure 2: Fully supervised Model Performance. We group languages by whether they and their scripts are seen in the pre-training corpus of XLM-R. Languages are ordered by the XLM-R performance in every group.
Context: Figure 2 compares MLP, XLM-R and Glot-500 models based on language and script coverage in pre-training based on four groups: (1) language seen, script seen in XLM-R (2) language unseen, script seen in XLM-R (3) script unseen in XLM-R, language seen in Glot-500 (4) script unseen by both models. The results in each group are sorted by their performance on fine-tuned XLM-R model. Overall, XLM-R performs the best on all languages seen in its pre-training corpus without any exception .
Verdict: Supported
Claim: We observe that all languages that are included in the pre-training corpus of XLM-R, the cross-lingual transfer performs similarly to fully supervised methods.
Evidence type: figure
Evidence caption: Figure 5: Comparison of Various Scenarios. We group languages by whether they and their scripts are seen in the pre-training corpus of XLM-R. Languages are ordered by the XLM-R fully-supervised performance in every group.
Context: Here, we compare the performance between cross-lingual transfer and fully-supervised methods.
Verdict: Supported
Claim: The advantage of fully supervised methods over cross-lingual transfer becomes prominent mainly when the target language is not included in the pre-training corpus of XLM-R and its script is included.
Evidence type: figure
Evidence caption: Figure 5: Comparison of Various Scenarios. We group languages by whether they and their scripts are seen in the pre-training corpus of XLM-R. Languages are ordered by the XLM-R fully-supervised performance in every group.
Context: Here, we compare the performance between cross-lingual transfer and fully-supervised methods. We observe that all languages that are included in the pre-training corpus of XLM-R, the cross-lingual transfer performs similarly to fully supervised methods. The best source language for cross-lingual transfer is, surprisingly, French, rather than English, which has the largest amount of pre-training corpus, though the difference among various source languages is tiny. This suggests languages included in the XLM-R pre-training corpus are pretty well aligned with all the four chosen high-resource languages .
Verdict: Supported
Claim: As we can see in Fig. 2 , the model with r=4 , yields poorer performance, highlighting the need for high rank for the frozen tensors.
Evidence type: figure
Evidence caption: Figure 2: Performance of ELoRA with two different ranks of the frozen projection matrices.
Context: To understand the high rank requirement for the frozen projection metrices in ELoRA, we conduct two sets of fine-tuning on SST-2 and MRPC, with ELoRA having rank ( r ) of 1024 and 4, respectively.
Verdict: Supported
Claim: As expected, p(\boldsymbol{x|y}) shows a higher correlation with accuracy than fluency, and p(\boldsymbol{y}) shows the opposite pattern.
Evidence type: figure
Evidence caption: Figure 5: \text{accuracy}_{M} and \text{fluency}_{M} predict human accuracy and fluency ratings for RLTC and WMT submissions to the general translation task in 2022 and 2023. zhen and ende refer to Chinese-English and English-German language pairs. All correlations reported are significant ( p<.001 ).
Context: A similar conclusion is suggested by Figure 5 , which shows Pearson correlations of translation probability ( p(\boldsymbol{y|x}) ), \text{accuracy}_{M} ( p(\boldsymbol{x|y}) ) and \text{fluency}_{M} ( p(\boldsymbol{y}) ) with human ratings of accuracy and fluency for RLTC and MTMQM. 9 9 9 Values are ranked by percentile.
Verdict: Supported