RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: These are all assessed using the accuracy metric.
Evidence type: table
Caption: Table 1: This table presents the tasks implemented in this paper. The first column specifies the different tasks. The second details the metrics used (ROUGE includes ROUGE1, ROUGE2 and ROUGEL, and Perplexity includes Bits per Byte, Byte Perplexity, and Word Perplexity). The third column outlines the benchmarks used for each task.
Context: This study considers four different close-ended healthcare tasks, which include nine different datasets ( e.g. , MedQA).
Verdict: Supported
Claim: At the same time, six open-ended tasks are studied, based on nine distinct datasets ( e.g. , MedText).
Evidence type: table
Caption: Table 1: This table presents the tasks implemented in this paper. The first column specifies the different tasks. The second details the metrics used (ROUGE includes ROUGE1, ROUGE2 and ROUGEL, and Perplexity includes Bits per Byte, Byte Perplexity, and Word Perplexity). The third column outlines the benchmarks used for each task.
Context: This study considers four different close-ended healthcare tasks, which include nine different datasets ( e.g. , MedQA). These are all assessed using the accuracy metric.
Verdict: Supported
Claim: In this case, eleven different metrics are extracted.
Evidence type: table
Caption: Table 1: This table presents the tasks implemented in this paper. The first column specifies the different tasks. The second details the metrics used (ROUGE includes ROUGE1, ROUGE2 and ROUGEL, and Perplexity includes Bits per Byte, Byte Perplexity, and Word Perplexity). The third column outlines the benchmarks used for each task.
Context: This study considers four different close-ended healthcare tasks, which include nine different datasets ( e.g. , MedQA). These are all assessed using the accuracy metric. At the same time, six open-ended tasks are studied, based on nine distinct datasets ( e.g. , MedText).
Verdict: Supported
Claim: Our findings indicate that adding reinforcement learning (STRUX+RL) leads to stronger performance compared to using the SFT method alone.
Evidence type: table
Caption: Table 1: Our STRUX system outperforms strong benchmarks in making stock investment decisions. We present macro-averaged precision, recall, F-scores, accuracy for the test set. LLMs evaluated are: Llama3-8b-Instruct and gpt-4o-mini-2024-07-18 .
Context: System Comparisons. Table 1 shows the macro-averaged precision, recall, F-scores, and accuracy for the test set. STRUX outperforms strong baselines in accuracy and F-scores for stock investment decisions.
Verdict: Supported
Claim: The selection is predominantly positive, with 8 positive and 1 negative fact; about half of the negative fact has an impact strength of 2–3.
Evidence type: table
Caption: Table 3: Statistics of supporting facts.
Context: Supporting Facts. We analyzed the supporting facts identified by the model in cases of correct decisions after reflections. Statistics are presented in Table 3 . Each transcript is distilled into a table of about 40 facts, from which the model selects 9.
Verdict: Supported
Claim: Moreover, reflection does not always yield perfect outcomes; the model can repeat decisions from previous cycles despite being instructed not to.
Evidence type: table
Caption: Table 2: The most common decision paths during reflection and their percentages in the training data.
Context: Table 2 shows common decision paths during reflection. Interestingly, reflection can lead to abrupt decision changes, such as a direct jump from Buy to Strong Sell, instead of gradual shifts (e.g., Buy \rightarrow Hold \rightarrow Sell).
Verdict: Supported
Claim: All in all, gender-specific projector gets the higher correlation score.
Evidence type: table
Caption: Table 2 : Gender diversity. Average Spearman correlations between proportion of female voices and diversity scores induced by speech representations.
Context: In Table 2 we report correlation scores for the gender diversity. Again, we split in two groups: male-voice and female voice-dominant. From the results, we notice that the correlations showed by the non-specialized embeddings are extremely weak. However, both specialized models reach very high correlations (e.g., up to 0.946 for SpeechSim/Gender).
Verdict: Supported
Claim: From these results, we see that, generally, SpeechSim performs better than other general-purpose representations across all configurations.
Evidence type: table
Caption: Table 3 : Emotion diversity. Average Spearman correlations between the classes entropy in EmoV and Expresso and diversity scores induced by the speech representations. SpeechSim/Emotion-Expresso (resp. Speech/Emotion-EmoV) refers to SpeechSim emotion head trained on Expresso (resp. EmoV).
Context: In Table 3 we report average Spearman correlations for the tested representation models, when tested on EmoV and Expresso separately.
Verdict: Supported
Claim: Equally, the specialized projection of SpeechSim for the emotion facet, SpeechSim/Emotion performs best across all representations.
Evidence type: table
Caption: Table 3 : Emotion diversity. Average Spearman correlations between the classes entropy in EmoV and Expresso and diversity scores induced by the speech representations. SpeechSim/Emotion-Expresso (resp. Speech/Emotion-EmoV) refers to SpeechSim emotion head trained on Expresso (resp. EmoV).
Context: In Table 3 we report average Spearman correlations for the tested representation models, when tested on EmoV and Expresso separately. From these results, we see that, generally, SpeechSim performs better than other general-purpose representations across all configurations.
Verdict: Supported
Claim: Finally, the original SpeechSim embeddings have negative correlation with the gender diversity.
Evidence type: table
Caption: Table 6 : Changing voice and gender diversity in the opposite directions. Average Spearman correlation between the gender diversity and the diversity scores induced by speech representations.
Context: We report results in Table 6 . From this results we see that SpeechSim/Gender is positively correlated with the gender diversity. In contrast, the SpeechSim/Voice metric tracks the speaker diversity and hence is negatively correlated with the gender diversity.
Verdict: Supported
Claim: Moreover, its correlation coefficient is smaller than that of SpeechSim/Voice.
Evidence type: table
Caption: Table 6 : Changing voice and gender diversity in the opposite directions. Average Spearman correlation between the gender diversity and the diversity scores induced by speech representations.
Context: We report results in Table 6 . From this results we see that SpeechSim/Gender is positively correlated with the gender diversity. In contrast, the SpeechSim/Voice metric tracks the speaker diversity and hence is negatively correlated with the gender diversity. Finally, the original SpeechSim embeddings have negative correlation with the gender diversity.
Verdict: Supported
Claim: All models can relatively accurately identify hateful posts with negative emotions and non-hateful posts with positive emotions.
Evidence type: table
Caption: Table 4: Classification accuracy of HateBERT, ToxDect-roberta, Perspective API, and Llama Guard 3 1/8B on GPT-HateCheck grouped by the hatefulness label (hate) and the polarity of the detected emotions (emo). We highlight the “positive” labels in green (“non-hateful” and positive emotions) and “negative” labels in red . The ambiguous emotions are highlighted in yellow .
Context: We further group the fine-grained emotions into positive (1), negative (-1), and ambiguous (0), based on Demszky et al. ( 2020 ) ’s taxonomy and present the models’ classification accuracy in the presence of emotions with different polarities in Table 4 . The result is revelatory:
Verdict: Supported
Claim: This result is alarming since it suggests that HS detectors are entangled with emotion polarity, and some safe posts with negative emotions, such as counter-speech expressing disapproval or sadness, are likely marked as hateful, potentially silencing the voices of vulnerable groups.
Evidence type: table
Caption: Table 4: Classification accuracy of HateBERT, ToxDect-roberta, Perspective API, and Llama Guard 3 1/8B on GPT-HateCheck grouped by the hatefulness label (hate) and the polarity of the detected emotions (emo). We highlight the “positive” labels in green (“non-hateful” and positive emotions) and “negative” labels in red . The ambiguous emotions are highlighted in yellow .
Context: We further group the fine-grained emotions into positive (1), negative (-1), and ambiguous (0), based on Demszky et al. ( 2020 ) ’s taxonomy and present the models’ classification accuracy in the presence of emotions with different polarities in Table 4 . The result is revelatory: All models can relatively accurately identify hateful posts with negative emotions and non-hateful posts with positive emotions. However, the accuracy degrades drastically for non-hateful posts with negative emotions, especially for HateBERT and ToxDect-roberta. 11 11 11 We observe a similar trend for hateful posts with positive or ambiguous emotions, although the number of such cases is much smaller.
Verdict: Supported
Claim: Additional synthetic datasets and high-quality retrieval datasets further improve the embedding model quality despite the tasks these datasets solve already being well represented in the basic datasets.
Evidence type: table
Caption: Table 2: Different data sources impact. Model performance is measured on ruMTEB. Avg. stands for the average score and is computed as the mean of the category scores. The best score is put in bold, the second best is underlined.
Context: Results presented in Table 2 indicate that the embedding model gets better results when trained on data in Russian and English simultaneously.
Verdict: Supported
Claim: Merging the encoder layers after language adaptation with the original model improves the model quality while using RetroMAE leads to decreased results.
Evidence type: table
Caption: Table 3: Results of the model, method, and data variation. † The reference results for the training objective and training examples sections is model based on ru-en-RoBERTa. Each experiment changes a single component (e.g., use AnglE similarity instead of cosine). Model performance is evaluated on ruMTEB. The best score across all experiments is bold, the second best is underlined.
Context: We perform contrastive fine-tuning for each model and then evaluate them on ruMTEB. Results (see Table 3 ) show that ru-en-RoBERTa outperforms both baselines by a significant margin. Additionally, the fact that XLM-R slightly outperforms ruRoBERTa may indicate that XLM-R copes better with knowledge transfer from basic English datasets, which provide diverse examples of high quality.
Verdict: Supported
Claim: The book of Genesis contains the highest number of half-verse chiasmi, while Numbers contains the most verse-level chiasmi.
Evidence type: table
Caption: Table 2: Summary of detected chiasmi. 2700+ chiasmi were detected at the verse and half-verse level. The highest number of chiasmi was found in the Book of Genesis and Book of Numbers. Both the precision and the inter-annotator agreement increase for the verse-level chiasmi.
Context: Table 2 presents an overview of the system’s output for chiastic structures at the half-verse and verse levels. A total of 1,896 chiastic structures were identified at the half-verse level, with an average length of 5.93 textual units ( \pm 1.34) and an average score of 0.32 ( \pm 0.1). For verse-level groupings, 879 chiastic structures were found, with an average length of 6.01 lines ( \pm 1.38) and an average score of 0.29 ( \pm 0.08).
Verdict: Supported
Claim: Meantime, Self-Taught-Llama-70B with extensive multilingual pre-training demonstrates the strongest cross-lingual transfer, achieving the best accuracies in all four non-English Multilingual RewardBench .
Evidence type: table
Caption: Table 2: Averaged Multilingual RewardBench results in two classifier RMs (top) and two generative RMs (bottom). Off-the-shelf RMs based on MLMs show strong cross-lingual transfer as in Table 1 .
Context: Interestingly, we can observe strong cross-lingual transfer in the generative RMs in Table 2 , as in the classifier RMs. As discussed in Section 3.2 , fine-grained representation learning is a crucial component for having strong downstream generative abilities. While the extent of multilingual pre-training in GPT-4o is not verifiable, GPT-4o has the least decrement in non-English settings.
Verdict: Supported
Claim: The results show that UTs are able to generalize better for the A and B splits, outperforming the LSTM and VT.
Evidence type: table
Caption: Table 5: Test accuracy of the models, trained on operation lengths of \leq 6 , with their out-of-distribution results shown here (lengths 7-12).
LSTM baseline from Bowman et al. ( 2015 ) , and Transformer baseline from Shen et al. ( 2019 )
Context: We train a 12 layer model with Attention (E=12,k=4,H=2,D=32,W=1) and FFD (E=12,K=4,D=128) and halting. Refer to Appendix A for further details. Training a 12-layer Vanilla Transformer achieves approximately the same results as in Shen et al. ( 2019 ) , so we report their results. Our results in Table 5 confirm the findings of Tran et al. ( 2018 ) , showing that with recurrence in SUTs, we are able to generalize to longer sequences of the task. While there are other models that induce a tree structure that performs exceedingly well on the task, we wanted to evaluate our model against other popular architectures. The LSTM is a strong baseline, and we find that UT outperforms it in generalization. We also evaluate UTs on the compositional generalization splits as proposed in Shen et al. ( 2019 ) , where the splits A, B, and C are in increasing difficulty.
Verdict: Supported
Claim: As shown in Table 1 , for CSQA2.0, higher scores are reported for retrieval only than for induction only, while the result is contrary for StrategyQA.
Evidence type: table
Caption: Table 1: Performance on two ODQA tasks. The first two columns report scores on CSQA2.0 dev set and StreategyQA test set respectively. The last two columns compare IAG with ChatGPT on a randomly held-out subset containing 50 examples for each task.
Context: Besides, the results on different setups of IAG-GPT suggest that, the relative contributions of the retrieval and the inductive knowledge can be different, depending on the tasks.
Verdict: Supported
Claim: The only exception is the case of style strength evaluation task in the direction of H_{1} \rightarrow H_{2} , using the GPT-NeoX model.
Evidence type: table
Caption: Table 6: Inter-annotator agreement scores for the three human evaluation tasks. Standard deviations over all data points are shown in brackets for the style strength and appropriateness evaluation tasks. The detailed procedure for calculating the agreement scores can be found in Appendix D .
Context: The inter-annotator agreement in all of the tasks are shown in Table 6 . Note that, for calculating agreement in the semantic correctness evaluation task, all of the data points are aggregated to measure the agreement score as they represent categorical evaluation measures. On the other hand, that is not possible in case of ranking based evaluations for style strength and appropriateness. So, we measure the agreement for each data point and take the average agreement over all data points. We can see in the Table 6 that in all of the cases we get strong agreement ( >0.70 ) among the annotators for the style strength and appropriateness evaluation.
Verdict: Supported
Claim: Yet still only half of its explanations are considered adequate.
Evidence type: table
Caption: Table 6: Adequacy and Preference rates for generated explanations.
Context: In Table 6 , we show adequacy and preference rates for explanations from the 3 systems, where an explanation is deemed adequate if both annotators agreed it is, and inadequate if both agreed it is not. The preference percentage is also taken among instances where the annotators agreed that the model’s explanation is preferred among all the adequate explanations. The average IAA using Cohen’s \kappa is 0.47, indicating moderate agreement Cohen ( 1960 ) . We observe that the teacher model is leading in terms of the adequacy of the explanations and preference rate, as expected from a larger system equipped for higher quality reasoning and generation capabilities.
Verdict: Supported
Claim: It is also robust, with a standard deviation of 0.022 .
Evidence type: table
Caption: Table 2: Results of QuestMF ( ImbOLL ) framework over 3 different seed runs compared with ablation frameworks.
Context: The results comparing our proposed QuestMF ( ImbOLL ) framework with its ablations and current state-of-the-art methods are presented in Table 1 . Since the prior works only show their best results on a single run, we also pick our best results on CCC for a fair comparison. As we can see, our proposed QuestMF ( ImbOLL ) framework matches the performance of state-of-the-art models in CCC, the primary metric used to evaluate depression severity prediction tasks. In addition to comparable performance, QuestMF framework provides question-wise scores, improving interpretability over current methods, thus allowing clinicians to design personalised interventions. We also show the robustness of our frameworks over multiple runs, which is not done by the previous works. The mean and standard deviation of the performance over 3 runs are presented in Table 2 . We observe that Total and QuestMF (MSE) perform similarly. However, training with ordinal classification objective improves the performance, as we can see from the results of QuestMF (OLL). This shows the effectiveness of combining QuestMF with ordinal classification training. Training with our novel ImbOLL function further improves the results on CCC.
Verdict: Supported
Claim: However, by some simple maths, the readers can easily verify that even considering this, the accuracy of answers containing calculations is still much lower than that of answers that do not include calculations.
Evidence type: table
Caption: Table 1: The redundancy ( Red. ) and accuracy of LLMs’ responses.
We report the average accuracy ( Avg. ) on all questions (second column), the accuracy for answers without calculation (Cal. ✗, third column) and with calculation (Cal. ✓, fourth column). \dagger : The accuracy of GPT-4 is 100% by construction since we use GPT-4 to filter samples when constructing GSM8K-Zero.
Context: Next, for accuracy, even if LLMs are wrong for all the invalid questions, their accuracy should be around 85% if they get all the valid questions correct. However, this is clearly not the case for all LLMs except Claude-2.
Next, for the rightmost column in Table 1 , if we assume that all the invalid samples happen to be the samples that LLMs include calculations in the answer, the accuracy in this column should increase.
Verdict: Supported
Claim: At the same time, the RMSE obtained using the majority voting is also considerably lower than the results shown in the previous subsection from just one run (1.12 vs 1.45; columns GPT-4 {}_{\text{maj}} and GPT-4 in Table 2 ).
Evidence type: table
Caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Context: The application of the majority voting algorithm improved the RMSE on average by about 30% from 1.61 to 1.12 (columns GPT-4 {}_{\text{avg}} and GPT-4 {}_{\text{maj}} in Table 2 ). However, compared to the results shown in previous Section 4.2 (also shown in Table 2 ), generating five runs and taking the average increased the RMSE considerably, which is now in the same range to the average over the human reader-annotator ratings (1.61 vs 1.57; columns GPT-4 {}_{\text{avg}} and Human {}_{\text{avg}} in Table 2 ).
Verdict: Supported
Claim: Also in this case, it can be observed that the choice of fine-tuning a generative model, leveraging its autoregressive characteristics for classification, leads to the best results.
Evidence type: table
Caption: Table 2: Emotional accuracy comparison between RACLETTE and other benchmarks (best results highlighted in green).
Context: Table 2 shows how RACLETTE outperforms the benchmarks considered.
Verdict: Supported
Claim: However, for NusaX and the African languages, a considerable number of them are not covered.
Evidence type: table
Caption: Table 1: Languages used in this paper. “✓” (green) and “✗” (red) mean that the language has and has not been seen by the models or language resources. CW indicates code-switching text. The language coverage for GPT-3.5 is derived from GPT-3 Brown et al. ( 2020 ) .
Context: In Table 1 , we provide an overview of the languages and datasets used in this paper, categorized into: (1) high/medium-resource languages; (2) NusaX, covering local Indonesian languages (low resource); (3) African languages from SemEval 2023 (low resource); and (4) code-switching texts. The high/medium-resource languages and individual languages present in the code-switching texts are covered by all pretrained models and the NRC-VAD multilingual lexicon.
Verdict: Supported
Claim: Language coverage of the NRC-VAD multilingual lexicons remains limited in 109 languages.
Evidence type: table
Caption: Table 1: Languages used in this paper. “✓” (green) and “✗” (red) mean that the language has and has not been seen by the models or language resources. CW indicates code-switching text. The language coverage for GPT-3.5 is derived from GPT-3 Brown et al. ( 2020 ) .
Verdict: Supported
Claim: The improvements shown in Table 3 are particularly noticeable in few-shot training, with increases of +6.6 and +5.2 for binary and 3-way classification, respectively.
Evidence type: table
Caption: Table 3: Preliminary results, based on averaged macro-F1 scores across 34 languages. “EN Lex.” and “ML Lex.” indicate the English and multilingual NRC-VAD lexicons.
Verdict: Supported
Claim: In the full training scenario, the increments are smaller, at only +1.4 and +1.1 .
Evidence type: table
Caption: Table 3: Preliminary results, based on averaged macro-F1 scores across 34 languages. “EN Lex.” and “ML Lex.” indicate the English and multilingual NRC-VAD lexicons.
Context: The improvements shown in Table 3 are particularly noticeable in few-shot training, with increases of +6.6 and +5.2 for binary and 3-way classification, respectively.
Verdict: Supported
Claim: Overall, we observe that multilingual models fine-tuned on SST tend to perform the best in high/medium-resource languages, with mBERT {}_{\text{Base}} being the exception.
Evidence type: table
Caption: Table 4: Full zero-shot results. The underlined score indicates the highest performance within the respective group, while scores in bold indicate the best global performance. “HM-R” = high/medium-resource languages, excluding English, “CS” = code-switched text, and “ML Lex.” indicates the multilingual NRC-VAD lexicon. “SST (sentence-level data)” is cross-lingual zero-shot transfer that is trained on English sentence-level sentiment data.
Context: Table 4 presents the averaged zero-shot performance of all models categorized by four language groups. The reported results use regression and classification in lexicon-based pretraining for binary and 3-way classification, respectively. In the case of high/medium-resource languages (HM-R), English is excluded to ensure a fair comparison with models fine-tuned on the English SST dataset.
Verdict: Supported