alabnii/sciclaimeval-shared-task huggingface dataset:
Claim: Finally, on comparing average cosine vs. Vendi Score aggregation, we observe that often Vendi Score shows higher correlations.
Evidence type: table
Evidence caption: Table 1 : Voice diversity. Average Spearman correlations ( \pm standard error) between number of distinct speakers and diversity scores induced by speech representations. Male (Female) refer to samples with male-only (female-only) voices. Best results are in bold, second best results are underlined.
Context: From Table 1 we firstly notice that, across all columns, SpeechSim shows higher correlation scores than off-the-shelf embedding models and the models trained from scratch. This supports our decision to rely on SpeechSim as a basis for a per-facet models. In turn, further voice specialization of SpeechSim (SpeechSim/Voice) results in similar scores.
Verdict: Supported
Claim: In this setup, Vendi Score underperforms w.r.t. the average cosine dissimilarity.
Evidence type: table
Evidence caption: Table 2 : Gender diversity. Average Spearman correlations between proportion of female voices and diversity scores induced by speech representations.
Context: In Table 2 we report correlation scores for the gender diversity. Again, we split in two groups: male-voice and female voice-dominant. From the results, we notice that the correlations showed by the non-specialized embeddings are extremely weak. However, both specialized models reach very high correlations (e.g., up to 0.946 for SpeechSim/Gender). All in all, gender-specific projector gets the higher correlation score.
Verdict: Supported
Claim: However, SpeechSim finetuned on EmoV gets a Spearman correlation equal to 0.321.
Evidence type: table
Evidence caption: Table 3 : Emotion diversity. Average Spearman correlations between the classes entropy in EmoV and Expresso and diversity scores induced by the speech representations. SpeechSim/Emotion-Expresso (resp. Speech/Emotion-EmoV) refers to SpeechSim emotion head trained on Expresso (resp. EmoV).
Context: To make sure that our models can measure diversity of emotions beyond that is covered by the (limited) label sets of EmoV and Expresso, we run an experiment where the projection model is trained on one dataset and tested on another (the only common label is “neutral”). Table 3 shows that SpeechSim finetuned on Expresso does transfer to EmoV classes as it obtains a high Spearman correlation.
Verdict: Supported
Claim: SpeechSim/Noise combined with Vendi Score obtains the second highest correlation, following HuBERT.
Evidence type: table
Evidence caption: Table 5 : Background noise diversity. Average Spearman correlations between the number of different classes of noise and diversity scores induced by speech representations.
Context: We report our results in Table 5 . We see that generally all representations have somewhat low levels of correlation.
Verdict: Supported
Claim: Table 1 displays the number of documents for each target identity in both datasets, and Appendix A shows examples from the two datasets for each functionality.
Evidence type: table
Evidence caption: Table 1: Number of examples for each target identity in HateCheck and GPT-HateCheck . We omit the functionalities without targeting identity, such as abusing objects or non-protected groups.
Context: We use the HateCheck (Röttger et al., 2021 ) and GPT-HateCheck (Jin et al., 2024 ) datasets to conduct our analyses, as these datasets provide additional diagnostic insights. Both datasets cover the same seven target identities and 24 functionalities ( GPT-HateCheck omitted the five functionalities related to spelling variations in HateCheck ).
Verdict: Supported
Claim: We can observe a clear push-back pattern: The higher the “coldness” or “incompetence” scores are for hateful stereotypes towards a target identity, the stronger the counter-stereotypes are in the opposite directions; consider, for illustration, the “warmth” dimension for gays and the “competence” dimension for women.
Evidence type: table
Evidence caption: Table 5: The mean “warmth” and “competence” scores for hateful (H) and non-hateful (N/H) examples. We highlight the scores with the highest magnitude in bold .
Context: The mean “warmth” and “competence” scores for each target identity are presented in Table 5 .
Verdict: Supported
Claim: A total of 1,896 chiastic structures were identified at the half-verse level, with an average length of 5.93 textual units ( \pm 1.34) and an average score of 0.32 ( \pm 0.1).
Evidence type: table
Evidence caption: Table 2: Summary of detected chiasmi. 2700+ chiasmi were detected at the verse and half-verse level. The highest number of chiasmi was found in the Book of Genesis and Book of Numbers. Both the precision and the inter-annotator agreement increase for the verse-level chiasmi.
Context: Table 2 presents an overview of the system’s output for chiastic structures at the half-verse and verse levels.
Verdict: Supported
Claim: Average reward model accuracy ("Avg") in Table 1 shows that English RMs surpass target language RMs in general.
Evidence type: table
Evidence caption: Table 1: Multilingual RewardBench evaluation results on the target language ("Target") and English ("English") RMs. " \Delta " denotes the accuracy gain of English RMs compared to the target language RMs. English RMs show higher average scores in the lingual axis than target language RMs. Also, English RMs excel target language RMs in reasoning ("Reason") tasks with diverse evaluation sub-categories.
Verdict: Supported
Claim: Observation of the results show that social causal reasoning tasks scored higher on BERTScore while physical causal reasoning tasks scored higher on SentenceBERT .
Evidence type: table
Evidence caption: Table 5: GPT-4 performance for videos partitioned by the physical (changes due to outside, real forces) and social (changes related to social cues) on the Infilling-1 task with structured, FAMOuS descriptions. Significant differences between partitions are underlined .
Context: We examine the difference in performance between physical and social causal reasoning ( Table 5 ). A task necessitates physical causal reasoning when the video changes stem from external, tangible forces, like a wave crashing on a beach. Conversely, a task involves social causal reasoning when video changes result from social cues, such as a character becoming joyful during a surprise party.
Verdict: Supported
Claim: In the most challenging second-order ToM inference tasks, where agents estimate others’ beliefs about their own mental states, GPT-4 + Belief agents correctly respond in nearly 70% of cases.
Evidence type: table
Evidence caption: Table 2: LLM-based agents’ performance in ToM inference tasks. Natural language answers are annotated by experimenters and compared with the ground truth based on global interaction history. Percentages represent the inference accuracy.
Context: A critical aspect of teamwork is inferring teammates’ mental states, including beliefs, desires, and intentions. We assess LLM-based agents by asking them to conduct Theory of Mind inferences during the mission. As seen in Table 2 , LLM-based agents can estimate their own and their teammates’ mental states.
Verdict: Supported
Claim: The evaluation results are shown in Table 6 . InstructBLIP performs the best under all settings, highlighting the importance of instruction-tuning on large visual instruction corpora.
Evidence type: table
Evidence caption: Table 6: Evaluation results of LVLMs on POPE and VQA. For VQA tasks, we report the VQA score on A-OKVQA and Accuracy on GQA. For POPE, we copy the result under the random setting from Table 11 .
Verdict: Supported
Claim: The evaluation results are presented in Table 1 . While the state-of-the-art models demonstrate competitive performance, SocialSense outperforms all other models across all evaluation metrics consistently.
Evidence type: table
Evidence caption: Table 1: Response forecasting results. We report the Spearman and Pearson correlations for the forecasting of sentiment intensity, as well as Micro F1 and Macro F1 scores for the sentiment polarity prediction. The best overall performance is in bold. Our framework outperforms the baselines consistently.
Context: We conduct an evaluation of the proposed SocialSense model and the baseline models introduced in Section 4.2 for the supervised response forecasting task.
Verdict: Supported
Claim: Furthermore, our model, SocialSense {}_{\text{Zero}} , achieves the highest scores consistently across all metrics.
Evidence type: table
Evidence caption: Table 2: The above Zero-Shot Response forecasting results highlight that the Social Prompt from Section 3.4 consistently offers an advantage.
Context: In addition to supervised response forecasting, we also evaluate our framework under the zero-shot setting (Section 3.4 ). The results are presented in Table 2 . Based on the higher scores attained by ChatGPT L , it is evident that the inclusion of latent structured persona information indeed aids the model in comprehending the user more effectively.
Verdict: Supported
Claim: The number of the direct and contextual rationales is the largest among the other types, which further increases when we look at the error cases of InstructGPT.
Evidence type: table
Evidence caption: Table 4: Annotation results of rationale types on 100 examples randomly sampled from all subquestions (left) and from the error examples by InstructGPT (right).
Context: We report our annotation results in Table 4 .
Verdict: Supported
Claim: We observe an improvement when the selective rationale is added; however, degradation occurs when we add the eliminative rationale, even if it is provided with the selective rationale.
Evidence type: table
Evidence caption: Table 5: MainQ accuracy of InstructGPT that uses the selective or eliminative rationales in the input.
Context: Because the collected rationales are expected to support the decision of selecting and eliminating answer options, we investigate whether adding the rationales to the main questions improves the performance in the five-shot InstructGPT. We append the rationale to the context, main question, and four options with the Rationale: label. The results are shown in Table 5 .
Verdict: Supported