alabnii/sciclaimeval-shared-task huggingface dataset:
Claim: When looking at individual appraisal dimensions, we can see that the RMSE’s are in most cases lower in the random tie-breaking setting compared to using the confidence rating, and that applies to both the model and humans.
Evidence type: table
Evidence caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Context: The results of this experiment are shown in Table 2 columns GPT-4 {}_{\text{conf}} and Human {}_{\text{conf}} . We can see that using the confidence rating to break the ties did not improve the RSME neither for the model nor for the human reader-annotators compared to the random choice. The mean RMSE for GPT-4 from 1.12 to 1.15 (columns GPT-4 {}_{\text{maj}} and GPT-4 {}_{\text{conf}} in Table 2 ), and the mean RMSE for human reader-annotators increased from 0.99 to 1.06 (columns Human {}_{\text{maj}} and Human {}_{\text{conf}} in Table 2 ).
Verdict: Supported
Claim: The average RMSE scores for five completions applying the majority voting algorithm was 1.22 (see column GPT-4 {}_{\text{emo}} ), which is worse than the same task without the emotion label prediction studied in relation to research question Q3.
Evidence type: table
Evidence caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Context: The results of adding emotion to a prompt are shown in the last column of Table 2 .
Verdict: Supported
Claim: Appraisals consistently showing better accuracy across both GPT-4 models and human annotators include Pleasantness, Unpleasantness, Goal Support, and External Norms (marked as bold in Table 2 ), suggesting that these are the easiest to infer based on text.
Evidence type: table
Evidence caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Context: In analyzing appraisal dimensions across all experiments, we looked for patterns by comparing the RMSE of individual appraisal dimensions to the macro-averaged RMSE.
Verdict: Supported
Claim: Own Responsibility, Own Control, Not Consider, and Effort also generally performed well, although Own Responsibility is predicted remarkably worse in the GPT-4 {}_{\text{emo}} setting.
Evidence type: table
Evidence caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Context: In analyzing appraisal dimensions across all experiments, we looked for patterns by comparing the RMSE of individual appraisal dimensions to the macro-averaged RMSE. Appraisals consistently showing better accuracy across both GPT-4 models and human annotators include Pleasantness, Unpleasantness, Goal Support, and External Norms (marked as bold in Table 2 ), suggesting that these are the easiest to infer based on text.
Verdict: Supported
Claim: Overall, the CDD values reported in Table 2 are relatively low (all below 0.006), as indicated by prior studies Wachter et al. ( 2021 ); Koumeri et al. ( 2023 ); Wachter et al. ( 2020 ) , suggesting minimal gender-based disparity across the different classes.
Evidence type: table
Evidence caption: Table 2: One-vs.-rest CDD values for each class across four conflicts.
Verdict: Supported
Claim: As seen in Table 1 , five of the seven relations are parsed in accordance with each’s desired relation more than 82\% of the time, four greater than or equal to 95\% of the time, and one is parsed to the desired relation for all tested prompts.
Evidence type: table
Evidence caption: Table 1: The automatic-evaluation statistics for each relation, where None is generation with the language model alone.
Context: The input text alongside its completion is automatically parsed using the DMRST parser.
Verdict: Supported
Claim: Evaluation_NS is unique in being poor, receiving an average of 2.47 .
Evidence type: table
Evidence caption: Table 2: The human-evaluation statistics for each relation, where None is generation with the language model alone. The metrics are ( Rel[ation-fit]), ( Flu[ency]), and ( Rea[sonableness]).
Context: The average annotator rating of relation-fit for generation with each of the relations is presented in Table 2 . The overall average, 3.49 , is well within the positive range.
Verdict: Supported
Claim: Conversely, RoBERTa leads in precision at 0.72, but with lower recall at 0.84.
Evidence type: table
Evidence caption: Table 4: Classification results for different methods of comparing embeddings to detect risk of suicide. The higher the score (a.k.a. the greener), the better.
Context: The results show RACLETTE’s Combined method achieving the highest recall of 0.95, indicating superior ability to identify relevant cases, though this comes with a trade-off in precision at 0.63.
Verdict: Supported
Claim: Both JS Divergence and Cosine Similarity methods show similar patterns, with high recall (0.93) but lower precision.
Evidence type: table
Evidence caption: Table 4: Classification results for different methods of comparing embeddings to detect risk of suicide. The higher the score (a.k.a. the greener), the better.
Context: The results show RACLETTE’s Combined method achieving the highest recall of 0.95, indicating superior ability to identify relevant cases, though this comes with a trade-off in precision at 0.63. Conversely, RoBERTa leads in precision at 0.72, but with lower recall at 0.84. The KL Divergence variant of RACLETTE stands out for its balanced performance, maintaining strong scores across all metrics (precision: 0.71, recall: 0.90, F1: 0.79, accuracy: 0.77).
Verdict: Supported
Claim: The color intensity (green shading) in Table 4 indicates better performance, visually highlighting that RACLETTE’s approaches generally outperform the benchmark models.
Evidence type: table
Evidence caption: Table 4: Classification results for different methods of comparing embeddings to detect risk of suicide. The higher the score (a.k.a. the greener), the better.
Context: The results show RACLETTE’s Combined method achieving the highest recall of 0.95, indicating superior ability to identify relevant cases, though this comes with a trade-off in precision at 0.63. Conversely, RoBERTa leads in precision at 0.72, but with lower recall at 0.84. The KL Divergence variant of RACLETTE stands out for its balanced performance, maintaining strong scores across all metrics (precision: 0.71, recall: 0.90, F1: 0.79, accuracy: 0.77). Both JS Divergence and Cosine Similarity methods show similar patterns, with high recall (0.93) but lower precision.
Verdict: Supported
Claim: In 6 of the 7 datasets with natural claims, the best generalization score is from a model trained on artificial claims.
Evidence type: table
Evidence caption: Table 3: F1 of binary fact verification on the evaluation set for all datasets in a zero-shot generalization setup. Rows correspond to the training dataset and columns to the evaluated dataset.
Context: Many works Jiang et al. ( 2020 ); Saakyan et al. ( 2021 ) do not consider NEI claims due to their ambiguity. To explore whether our previous observations also hold for the task of binary fact verification , we evaluate the generalization results for all 11 datasets using only the supports and refutes claims for training and evaluation, shown in Table 3 . In this setting, artificial claims also generalize well to natural claims in other domains.
Verdict: Supported
Claim: From the results, we notice that the correlations showed by the non-specialized embeddings are extremely weak.
Evidence type: table
Evidence caption: Table 2 : Gender diversity. Average Spearman correlations between proportion of female voices and diversity scores induced by speech representations.
Context: In Table 2 we report correlation scores for the gender diversity. Again, we split in two groups: male-voice and female voice-dominant.
Verdict: Supported
Claim: Here, again, we notice that SpeechSim has higher correlations than other non-specialized embedding models.
Evidence type: table
Evidence caption: Table 4 : Accent diversity. Average Spearman correlations between the accent class entropy and diversity scores induced by speech representations.
Context: We report average correlations in Table 4 .
Verdict: Supported
Claim: We observe that the teacher model is leading in terms of the adequacy of the explanations and preference rate, as expected from a larger system equipped for higher quality reasoning and generation capabilities.
Evidence type: table
Evidence caption: Table 6: Adequacy and Preference rates for generated explanations.
Context: In Table 6 , we show adequacy and preference rates for explanations from the 3 systems, where an explanation is deemed adequate if both annotators agreed it is, and inadequate if both agreed it is not. The preference percentage is also taken among instances where the annotators agreed that the model’s explanation is preferred among all the adequate explanations. The average IAA using Cohen’s \kappa is 0.47, indicating moderate agreement Cohen ( 1960 ) .
Verdict: Supported