RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Table 4 shows that the evaluation on MNLI and QQP is more robust to different settings, and the variance is more significant on CoLA.
Evidence type: table
Caption: Table 4: Ablation studies of different calibration sizes.
Context: In this section, we first compare the influence of different calibration sizes on FPQ . We vary the calibration size in \{32,64,128,256\} and test on MNLI, QQP, and CoLA.
Verdict: Supported
Claim: However, we also find that it remains robust and maintains competitive accuracy even with limited access to calibration data, such as when using as few as 32 data points.
Evidence type: table
Caption: Table 4: Ablation studies of different calibration sizes.
Context: In this section, we first compare the influence of different calibration sizes on FPQ . We vary the calibration size in \{32,64,128,256\} and test on MNLI, QQP, and CoLA. Table 4 shows that the evaluation on MNLI and QQP is more robust to different settings, and the variance is more significant on CoLA. We observe that FPQ performs well with a calibration set size of 128 data points.
Verdict: Supported
Claim: We can see in the Table 6 that in all of the cases we get strong agreement ( >0.70 ) among the annotators for the style strength and appropriateness evaluation.
Evidence type: table
Caption: Table 6: Inter-annotator agreement scores for the three human evaluation tasks. Standard deviations over all data points are shown in brackets for the style strength and appropriateness evaluation tasks. The detailed procedure for calculating the agreement scores can be found in Appendix D .
Context: The inter-annotator agreement in all of the tasks are shown in Table 6 . Note that, for calculating agreement in the semantic correctness evaluation task, all of the data points are aggregated to measure the agreement score as they represent categorical evaluation measures. On the other hand, that is not possible in case of ranking based evaluations for style strength and appropriateness. So, we measure the agreement for each data point and take the average agreement over all data points.
Verdict: Supported
Claim: As evident from the results in Table 5 , GPT-NeoX with a history length of 100 shows comparable performance to supervised learning approaches, without any fine-tuning on any TKG training dataset.
Evidence type: table
Caption: Table 5: Performance (Hits@K) comparison between supervised models and ICL for single-step (top) and multi-step (bottom) prediction.
The first group in each table consists of supervised models, whereas the second group consists of ICL models, i.e., GPT-NeoX with a history length of 100.
The best model for each dataset in the first group is shown in bold , and the second best is underlined .
Context: We present a comparative analysis of the top-performing ICL model against established supervised learning methodologies for TKG reasoning, which are mostly based on graph representation learning.
Verdict: Supported
Claim: The experimental results presented in Table 6 demonstrate that ICL exhibits superior performance to rule-based baselines.
Evidence type: table
Caption: Table 6: Performance (Hits@K) with rule-based predictions. The best model for each dataset is shown in bold .
Context: To determine the extent to which LLMs engage in pattern analysis, beyond simply relying on frequency and recency biases, we run a comparative analysis between GPT-NeoX and heuristic-rules ( i.e., frequency & recency ) on the ICEWS14 dataset, with history length set to 100. frequency identifies the target that appears most frequently in the provided history while recency selects the target associated with the most recent fact in the provided history. The reason for our focus on ICEWS is that each quadruple represents a single distinct event in time. In contrast, the process of constructing YAGO and WIKI involves converting durations to two timestamps to display events across the timeline. This step has resulted in recency heuristics outperforming all of the existing models, showcasing the shortcoming of existing TKG benchmarks (See Appendix A.5 ).
Verdict: Supported
Claim: It can be seen that in both configurations, KL-divergence yields lower correlations than other distances, and on average total variation slightly outperforms Hellinger and one-best distances.
Evidence type: table
Caption: Table 3: Comparison of Statistical Distances using MQAG-Sum without answerability.
Context: In Table 3 , our results compare statistical distances.
Verdict: Supported
Claim: The observation is that MQAG achieves a higher correlation than the best SpanQAG on 5 out of 6 tasks.
Evidence type: table
Caption: Table 5: Pearson Correlation Coefficient (PCC) between the scores of summary evaluation methods and human judgements. PCCs are computed at the summary level on QAG and XSum-H, and at the system level on Podcast and SummEval. PCCs on Podcast are computed on 15 abstractive systems. Our best performing MQAG configuration consists of (i) generation stage G generates questions from summary y (i.e. MQAG-Sum), (ii) statistical distance is total variation, (iii) the answerability threshold \mathcal{N}^{\tau}_{y} is set to 2.0. Underline denotes where MQAG outperforms the best SpanQAG system, which is 5 out of 6 tasks. When compared to all baselines, MQAG achieves the highest PCC on 4 out of 6 tasks. The results of all MQAG configurations are provided in Table 10 , and Spearman’s correlation results are provided in Table 11 in the appendix.
Context: The baseline and MQAG results are shown in Table 5 .
Verdict: Supported
Claim: Results can be found in Table 3 . Unsurprisingly, more samples seems to improve calibration, though it seems on the unambiguous Natural Questions slice, sampling diversity with a small number of samples works relatively well for the cost.
Evidence type: table
Caption: Table 3: Calibration by number of samples (N). EM is excluded, because it is solely based on the greedy answer and does not depend on the number of samples.
Context: As sampling-based approaches require a linear increase in compute for the number of samples, we also examine how calibration scales with the number. In particular, we test three, five, and eight samples and compare that to the original results containing ten samples.
Verdict: Supported
Claim: Overall, the cognitive abilities of OPT, Llama-2-chat-70B, GPT-3.5-Turbo, and GPT-4 models successively increase, and the performance of each model gradually declines with the increase of stage, consistent with humans.
Evidence type: table
Caption: Table 3: Calibrated accuracy (%) of largest model in evaluating series. Acc and Age refer to calibrated accuracy and the age of equivalent human performance. The value of Age is calculated according to Equation 2 . Bold indicates the best performance.
Context: As shown in Table 3 , We run the model with the largest number of parameters in each series on CogLM , and report the adult human performance for comparison.
Verdict: Supported
Claim: Specifically, the latest state-of-the-art model, GPT-4, has demonstrated remarkable cognitive capabilities, achieving a level comparable to that of a 20-year-old human.
Evidence type: table
Caption: Table 3: Calibrated accuracy (%) of largest model in evaluating series. Acc and Age refer to calibrated accuracy and the age of equivalent human performance. The value of Age is calculated according to Equation 2 . Bold indicates the best performance.
Context: As shown in Table 3 , We run the model with the largest number of parameters in each series on CogLM , and report the adult human performance for comparison. Overall, the cognitive abilities of OPT, Llama-2-chat-70B, GPT-3.5-Turbo, and GPT-4 models successively increase, and the performance of each model gradually declines with the increase of stage, consistent with humans.
Verdict: Supported
Claim: Despite its superior performance, GPT-4’s performance on plan ability (59.4) is still barely satisfactory, far behind that of humans (95.6), which is consistent with the conclusion of Valmeekam et al. ( 2022 ) .
Evidence type: table
Caption: Table 3: Calibrated accuracy (%) of largest model in evaluating series. Acc and Age refer to calibrated accuracy and the age of equivalent human performance. The value of Age is calculated according to Equation 2 . Bold indicates the best performance.
Context: As shown in Table 3 , We run the model with the largest number of parameters in each series on CogLM , and report the adult human performance for comparison. Overall, the cognitive abilities of OPT, Llama-2-chat-70B, GPT-3.5-Turbo, and GPT-4 models successively increase, and the performance of each model gradually declines with the increase of stage, consistent with humans. Specifically, the latest state-of-the-art model, GPT-4, has demonstrated remarkable cognitive capabilities, achieving a level comparable to that of a 20-year-old human. It is also worth noting that both GPT-3.5-Turbo and GPT-4 surpass humans in empathy ability at stage 2, which is natural, as humans tend to have some degree of selfishness.
Verdict: Supported
Claim: As shown in Table 6 , in most cognitive abilities, COC can significantly improve the performance of Llama-2-chat-7B.
Evidence type: table
Caption: Table 6: Calibrated accuracy of Llama-2-chat-7B on CogLM with and without Chain-of-Cognition from GPT-3.5-Turbo as input.
Context: Although there is still room for improvement, the cognitive abilities of advanced LLMs have approached levels close to that of adult humans as discussed in Section 3.2 . A natural question is, what are the potential applications for advanced LLMs’ cognitive abilities? When humans address cognitive questions, they deduce and provide answers based on their cognitive abilities. While we have demonstrated in Section 3.3.2 that the cognitive chain-of-thought (Chain-of-Cognition, COC) generated by LLMs barely help self-address cognitive questions, we are curious whether COC can assist early-aged LLMs in improving cognitive performance. On this basis, we use the question together with the COCs generated by advanced LLM (GPT-3.5-Turbo) as input to test the performance of early-aged LLM (Llama-2-chat-7B) on CogLM .
Verdict: Supported
Claim: Notably, ALTER demonstrates the best performance in single-round reasoning among all other methods that utilize result ensemble techniques in the LLM era.
Evidence type: table
Caption: Table 1: Results of different methods on WikiTQ and TabFact. 1 1 1 For the Dater method, we report the results of using the LLM-based method as backbone (We use underline to denote the second-best performance, bold to denote the best performance for each region: Pre-LLM era, LLM era with result ensemble and without ensemble)
Context: We present the results on the WikiTQ and TabFact datasets. The experimental outcomes are summarized in Table 1 . From the results, we observe that our ALTER method achieves comparatively outstanding outcomes. Specifically, on the WikiTQ dataset, while the Mix SC method do marginally outperforms our results by aggregating multiple reasoning paths (with 10 sampling times), ALTER still managed to exceed the performance of all other methods under comparison.
Verdict: Supported
Claim: Furthermore, on the TabFact datasets, both augmentation methods have a much larger impact on hard questions than on simple questions.
Evidence type: table
Caption: Table 2: Ablation results of query augmentor on the test sets of WikiTQ and TabFact.
Context: To analyze the impact of two query augmentation methods in the query augmentor. We conducted experiments on the WikiTQ and TabFact datasets by discarding the step-back augmentation module (denoted as w/o step-back) and the sub-query augmentation module (denoted as w/o sub-query). For each dataset, we further categorized the questions based on the difficulty level, following Ye et al. ( 2023 ) . This stratification facilitates a more comprehensive evaluation of each module’s impact across different types of questions. The ablation test results are reported in Table 2 . From the results in the table, it is anticipated that employing both augmentation methods simultaneously yields the best performance under all experimental settings. For WikiTQ datasets, the accuracy of ALTER without step-back/sub-query augmentation drops by 2.9\%/2.0\% , demonstrating the necessity of augmented information from multi-queries.
Verdict: Supported
Claim: The performance improvement is particularly noteworthy when dealing with large tables.
Evidence type: table
Caption: Table 4: Comparison of methods in the LLM era with tables divided by token count on WikiTQ. ( underline denotes the second-best performance; bold denotes the best performance)
Context: Figure 3 shows the comparison results of ALTER and methods following the pre-LLM era, including CABINET and OMNITAB, partitioning tables in the WikiTQ dataset by the number of cells. In Table 4 , we present the results based on different table sizes divided by the token count in the WikiTQ dataset, comparing our method with Dater, Chain-of-TABLE, and Binder unique to the LLM era. Table 4 shows that ALTER significantly outperforms all three methods in the LLM era across different table sizes.
Verdict: Supported
Claim: We report labeled F1-score averaged on 100 fine-tuning runs in Table 2 .
Precision and recall are reported in Appendix B .
The Base model (using the full 12 layers) produces similar results on DOCILE no matter if pre-training on IIT-CDIP or DOCILE .
Evidence type: table
Caption: Table 2: F1 scores for named-entity recognition using different pre-training and fine-tuning datasets. Results are averaged on 100 runs with different seeds.
Verdict: Supported
Claim: However, on our internal Payslips datasets, our model pre-trained on DOCILE outperforms the original one.
Evidence type: table
Caption: Table 2: F1 scores for named-entity recognition using different pre-training and fine-tuning datasets. Results are averaged on 100 runs with different seeds.
Context: We report labeled F1-score averaged on 100 fine-tuning runs in Table 2 .
Precision and recall are reported in Appendix B .
The Base model (using the full 12 layers) produces similar results on DOCILE no matter if pre-training on IIT-CDIP or DOCILE .
Verdict: Supported
Claim: Moreover, we observe that our pre-trained model exhibits a way lower variance between fine-tuning runs.
Evidence type: table
Caption: Table 2: F1 scores for named-entity recognition using different pre-training and fine-tuning datasets. Results are averaged on 100 runs with different seeds.
Context: We report labeled F1-score averaged on 100 fine-tuning runs in Table 2 .
Precision and recall are reported in Appendix B .
The Base model (using the full 12 layers) produces similar results on DOCILE no matter if pre-training on IIT-CDIP or DOCILE . However, on our internal Payslips datasets, our model pre-trained on DOCILE outperforms the original one.
Verdict: Supported
Claim: Results are shown in Table 4 . Human performance is quite strong, almost reaching 90 F1@0 score overall.
Evidence type: table
Caption: Table 4: Human baseline results (F1@0) by phenomenon and source dataset.
Context: To find out how humans perform on the task, we hire two expert annotators with formal education in linguistics. We present them with 10 example instances and then ask them to complete 99 randomly sampled test set instances. We also evaluate our best model (see Table 3 ) on the same set.
Verdict: Supported
Claim: Humans also perform noticeably better on the NYCartoons dataset and on the idiom subset of the task.
Evidence type: table
Caption: Table 4: Human baseline results (F1@0) by phenomenon and source dataset.
Context: To find out how humans perform on the task, we hire two expert annotators with formal education in linguistics. We present them with 10 example instances and then ask them to complete 99 randomly sampled test set instances. We also evaluate our best model (see Table 3 ) on the same set. Results are shown in Table 4 . Human performance is quite strong, almost reaching 90 F1@0 score overall. Human performance is better than our strongest fine-tuned model (LLaVA-7B-eVil+VF) performance with p<0.05 for Annotator 1 and p<0.07 for Annotator 2. Humans excel at interpreting memes, with both annotators reaching a 100% F1 score.
Verdict: Supported
Claim: The model has a slight edge in performance on the sarcasm and visual metaphor subsets of the task, perhaps due to difficulty of these subsets and any potential spurious correlations during fine-tuning.
Evidence type: table
Caption: Table 4: Human baseline results (F1@0) by phenomenon and source dataset.
Context: To find out how humans perform on the task, we hire two expert annotators with formal education in linguistics. We present them with 10 example instances and then ask them to complete 99 randomly sampled test set instances. We also evaluate our best model (see Table 3 ) on the same set. Results are shown in Table 4 . Human performance is quite strong, almost reaching 90 F1@0 score overall. Human performance is better than our strongest fine-tuned model (LLaVA-7B-eVil+VF) performance with p<0.05 for Annotator 1 and p<0.07 for Annotator 2. Humans excel at interpreting memes, with both annotators reaching a 100% F1 score. Humans also perform noticeably better on the NYCartoons dataset and on the idiom subset of the task.
Verdict: Supported
Claim: (1) Fine-tuned language models consistently outperform generic LLMs, the performance gap can be narrowed through prompt design, few-shot learning, and model size.
Evidence type: table
Caption: Table 1: Performance of different fine-tuned language models and LLMs under different prompts on FiNER-ORD task.
Verdict: Supported
Claim: Additionally, few-shot learning and larger LLMs demonstrate notable advantages over their smaller counterparts.
Evidence type: table
Caption: Table 1: Performance of different fine-tuned language models and LLMs under different prompts on FiNER-ORD task.
Context: (1) Fine-tuned language models consistently outperform generic LLMs, the performance gap can be narrowed through prompt design, few-shot learning, and model size. Table 1 demonstrates that fine-tuned language models surpass generic LLMs in zero-shot direct prompting. However, the performance of generic LLMs improves significantly with diverse zero-shot prompting styles, surpassing the prompt designs proposed by Shah et al., 2023 .
Verdict: Supported
Claim: CoT prompting only improves the performance of the GPT-4o-mini model, whereas it significantly degrades the performance of the LLaMA 3.1 series.
Evidence type: table
Caption: Table 1: Performance of different fine-tuned language models and LLMs under different prompts on FiNER-ORD task.
Context: (2) Chain-of-Thought prompting has limited effect on LLMs performance and can sometimes reduce effectiveness. While few-shot learning generally enhances generic LLMs’ performance, Table 1 shows that the difference between prompting styles is marginal.
Verdict: Supported
Claim: In contrast, translations into Marathi show the lowest scores, ranging from 26.5 to 29.8, likely due to the complexity of translating between less common language pairs.
Evidence type: table
Caption: Table 3: COMET scores . Top-scoring translation for each source manuscript is in bold text. Second top-scoring translation is in italics.
Context: Translation Quality : Table 3 shows translation scores from the Hebrew Old Testament, Greek Old Testament, and Greek New Testament into five target languages. English and Turkish consistently achieve the highest scores across all manuscripts, with English translations ranging from 61.2 to 72.6, and Turkish from 65.4 to 68.2, suggesting strong translation quality for these language pairs.
Verdict: Supported
Claim: As suggested by McGovern et al. ( 2024 ) , we indeed see that human translations over or underemphasize the intertextuality present in the text, whereas machine translations provide a neutral baseline, based on these results.
Evidence type: table
Caption: Table 5: Overemphasized Intertextuality by Human Translation. The intertextuality from Hebrews 8:12 to Isaiah 43:25 is amplified by the human translator’s decision to render different words as "sin". The machine translation abstains from this and restores the original distance, but loses coherence.
Verdict: Supported
Claim: When the LLM’s answers contain calculations, the accuracy drops significantly for almost all models except for PaLM.
Evidence type: table
Caption: Table 1: The redundancy ( Red. ) and accuracy of LLMs’ responses.
We report the average accuracy ( Avg. ) on all questions (second column), the accuracy for answers without calculation (Cal. ✗, third column) and with calculation (Cal. ✓, fourth column). \dagger : The accuracy of GPT-4 is 100% by construction since we use GPT-4 to filter samples when constructing GSM8K-Zero.
Context: After discussing redundancy and accuracy independently, we want to know if redundant calculation co-occurs more often with wrong answers. We separate the model outputs into two groups: one that contains calculations and another that does not have calculations, and we calculate the accuracy for the two groups. The results are shown in the two rightmost columns in Table 1 .
Verdict: Supported
Claim: The average pairwise RMSE (APRMSE) values, macro-averaged over all appraisal dimensions, were very close to each other, ranging from 0.6 to 0.63 over the five runs, with a mean of 0.61 (see Table 1 .
Evidence type: table
Caption: Table 1: Average Pairwise RMSE (APRMSE) and Spearman correlation coefficients ( \rho ) of 108 random samples of five GPT-4 runs. All Spearman correlations were statistically significant at the level of p<0.001 .
Verdict: Supported
Claim: The application of the majority voting algorithm improved the RMSE on average by about 30% from 1.61 to 1.12 (columns GPT-4 {}_{\text{avg}} and GPT-4 {}_{\text{maj}} in Table 2 ).
Evidence type: table
Caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Verdict: Supported
Claim: The mean RMSE for GPT-4 from 1.12 to 1.15 (columns GPT-4 {}_{\text{maj}} and GPT-4 {}_{\text{conf}} in Table 2 ), and the mean RMSE for human reader-annotators increased from 0.99 to 1.06 (columns Human {}_{\text{maj}} and Human {}_{\text{conf}} in Table 2 ).
Evidence type: table
Caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Context: The results of this experiment are shown in Table 2 columns GPT-4 {}_{\text{conf}} and Human {}_{\text{conf}} . We can see that using the confidence rating to break the ties did not improve the RSME neither for the model nor for the human reader-annotators compared to the random choice.
Verdict: Supported