alabnii/sciclaimeval-shared-task huggingface dataset:
Claim: In the full training scenario, the increments are smaller, at only +1.4 and +1.1 .
Evidence type: table
Evidence caption: Table 3: Preliminary results, based on averaged macro-F1 scores across 34 languages. “EN Lex.” and “ML Lex.” indicate the English and multilingual NRC-VAD lexicons.
Context: The improvements shown in Table 3 are particularly noticeable in few-shot training, with increases of +6.6 and +5.2 for binary and 3-way classification, respectively.
Verdict: Supported
Claim: Overall, we observe that multilingual models fine-tuned on SST tend to perform the best in high/medium-resource languages, with mBERT {}_{\text{Base}} being the exception.
Evidence type: table
Evidence caption: Table 4: Full zero-shot results. The underlined score indicates the highest performance within the respective group, while scores in bold indicate the best global performance. “HM-R” = high/medium-resource languages, excluding English, “CS” = code-switched text, and “ML Lex.” indicates the multilingual NRC-VAD lexicon. “SST (sentence-level data)” is cross-lingual zero-shot transfer that is trained on English sentence-level sentiment data.
Context: Table 4 presents the averaged zero-shot performance of all models categorized by four language groups. The reported results use regression and classification in lexicon-based pretraining for binary and 3-way classification, respectively. In the case of high/medium-resource languages (HM-R), English is excluded to ensure a fair comparison with models fine-tuned on the English SST dataset.
Verdict: Supported
Claim: Our findings indicate that regression performs better for binary classification, while classification leads to better results for 3-way classification.
Evidence type: table
Evidence caption: Table 5: Regression vs. classification in lexicon-based pretraining for zero-shot sentiment analysis.
Context: In Table 5 we present the average performance across the four language groups to compare the effectiveness of lexicon-based pretraining in regression and classification tasks for both binary and 3-way classification.
Verdict: Supported
Claim: Although this setting works reasonably well for XLM-R, it yields poor performance for mT5.
Evidence type: table
Evidence caption: Table 5: Regression vs. classification in lexicon-based pretraining for zero-shot sentiment analysis.
Context: In Table 5 we present the average performance across the four language groups to compare the effectiveness of lexicon-based pretraining in regression and classification tasks for both binary and 3-way classification. Our findings indicate that regression performs better for binary classification, while classification leads to better results for 3-way classification. However, regression in 3-way classification presents a challenge when determining the neutral class boundary during inference. In the zero-shot setting, we lack specific data for hyper-parameter tuning, leading us to arbitrarily set the neutral class boundaries to - 1 and + 1.
Verdict: Supported
Claim: Table 6 shows the average F1 scores on each task, demonstrating that lexicon-based pretraining boosts the performance of vanilla mBERT {}_{\text{Base}} .
Evidence type: table
Evidence caption: Table 6: Lexicon-based pretraining performance (macro-F1) over stance detection, hate speech detection, and emotion classification. The results are based on the limited training data scenario.
Context: For each task, we take two datasets and perform experiments in few-shot training using mBERT {}_{\text{Base}} , following the setup described in Section 4.2 . Instead of using the multilingual lexicon, we use the English NRC-VAD lexicon since all the data is in English. For detailed information about the datasets and results, see the Appendix.
Verdict: Supported
Claim: As reported in Table 2 , in Q Drawer the value of the entropy threshold \theta has, by design, a direct impact on the number of questions asked: i.e., a model that only asks for clarification in the face of high uncertainty ( \theta=1.1 ) gives rise to fewer dialogues with questions and to a lower average number of questions per dialogue.
Evidence type: table
Evidence caption: Table 2: The effect of different \theta values on the number of questions asked by Q Drawer. We report the percentage of dialogues with at least one clarification question and, within this subset, the average number of clarification questions per dialogue.
Context: Asking clarification questions carries a cost, both from the perspective of the agent asking the question and the agent processing it (Clark, 1996 ; Purver, 2004 ) . We take this partially into account by adopting a simple approach: we evaluate Q Drawer by using different entropy threshold values \theta to control for the number of questions generated by the model.
Verdict: Supported
Claim: Here, while finetuning achieves higher accuracy for gendered words, it also degrades translation quality by 0.9 COMET.
Evidence type: table
Evidence caption: Table 3: Grammatical gender control results in supervised condition (cross-domain: controller trained on gender-annotated data from a different domain).
Context: As motivated in § 4.2 ,
the gender control results in Table 3 allow us to assess the impact of domain mismatch between the controller training data and the test data, a very realistic scenario in practice.
Verdict: Supported
Claim: For formality control, on Korean, the most distant language,
CG consistently achieves stronger control results than finetuning,
indicating more robustness when transferring to unfamiliar settings.
Evidence type: table
Evidence caption: Table 4: Zero-shot formality control results. Best and second best results under the same data condition are marked.
Context: While finetuning was consistently leading in supervised conditions (§ 5 ),
now under zero-shot conditions with unseen target languages, the gap shrinks.
Verdict: Supported
Claim: In contrast,
for the pretrained NLLB,
there is no clear distinction between the multilingual systems and rest.
Evidence type: table
Evidence caption: Table 4: Zero-shot formality control results. Best and second best results under the same data condition are marked.
Context: In Table 4 , controllers trained on multiple translation directions ( multi ) are compared to those trained on single directions ( en \rightarrow es or de ). On Transformer-base, multi consistently outperforms its single-direction counterparts, regardless whether the controller is finetuning- or CG-based.
Verdict: Supported
Claim: Second, pairwise comparisons with the baseline show both CG and finetuning are effective in formality control, where CG has slightly higher win ratio than FT against the baseline.
Evidence type: table
Evidence caption: Table 6: Human evaluation on Bengali,
with quality on a 5-point scale \uparrow and formality on a 3-point scale ( \uparrow : formal) with standard deviations .
Last two columns show pairwise comparison of formality scores to baseline NLLB-200 given the same source sentences (winning: scoring more in the direction of the desired formality).
Context: The results are in Table 6 .
First, adding attribute control does not appear to impact translation quality.
Verdict: Supported
Claim: Third, the impact on formality scores is more prominent when steering towards informal translation.
Evidence type: table
Evidence caption: Table 6: Human evaluation on Bengali,
with quality on a 5-point scale \uparrow and formality on a 3-point scale ( \uparrow : formal) with standard deviations .
Last two columns show pairwise comparison of formality scores to baseline NLLB-200 given the same source sentences (winning: scoring more in the direction of the desired formality).
Context: The results are in Table 6 .
First, adding attribute control does not appear to impact translation quality. Second, pairwise comparisons with the baseline show both CG and finetuning are effective in formality control, where CG has slightly higher win ratio than FT against the baseline.
Verdict: Supported
Claim: When focusing on oscillatory hallucinations according to TNG in Table 4 , the improvement is even more pronounced,
with a reduction from 9.3% to 0.7% (-92%) for M2M-100, and from 5.9% to 1.5% (-75%) for SMaLL-100.
Evidence type: table
Evidence caption: Table 4: Proportion of translations with oscillatory hallucinations according to TNG.
Context: The proportion of translations with chrF2 below 10 is shown in Table 3 . We observe large reductions in the number of defect translations, with a reduction from 7.3% to 1.2% (-83%) for M2M-100, and from 5.6% to 1.8% (-67%) for SMaLL-100.
Verdict: Supported
Claim: We find that all QA models suffer a large performance drop ( \sim 20% in EM) in MisinfoQA-noisy compared to MisinfoQA-clean , showing that the models are largely distracted by the fake contexts rather than by the presence of additional contexts.
Evidence type: table
Evidence caption: Table 4: QA performance under the reading comprehension settings with clean and noisy contexts.
Context: Table 4 reports the EM and F1 for both human and different QA models.
Verdict: Supported
Claim: Humans obtained an EM of 69.13 in MisinfoQA-noisy , which, though higher than most QA models’ performance, also shows a significant drop when compared to the MisinfoQA-clean setting (86.57 EM).
Evidence type: table
Evidence caption: Table 4: QA performance under the reading comprehension settings with clean and noisy contexts.
Context: Table 4 reports the EM and F1 for both human and different QA models. We find that all QA models suffer a large performance drop ( \sim 20% in EM) in MisinfoQA-noisy compared to MisinfoQA-clean , showing that the models are largely distracted by the fake contexts rather than by the presence of additional contexts.
Verdict: Supported
Claim: Also here we can see that our proposed model outperforms the baseline models showing BLEU-4 score of 5.76 in test set.
Evidence type: table
Evidence caption: Table 4: Back translation results obtained from the generative models when using manual features and facial landmarks and AUs. Our proposed model has the highest scores in all metrics compared to the models using only gloss or text.
Context: Table 4 presents the results of including facial landmarks as well as facial AUs with body and hands skeleton joints as input.
Verdict: Supported
Claim: For gender+ethnicity, QAGNN has an equivalent amount or more than BioLinkBert, though for gender+sexual orientation, BioLinkBert has more than double the amount for homosexuals, with a massive percentage of 23.
Evidence type: table
Evidence caption: Table 2: Percentage of questions with changed answers as compared to a question with no demographic information about the patient. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation; Random=Random change as described in Section 3.5 .
Context: The first column of Table 2 , “Random”, shows the result of our random change (Sec. 3.5 ). While the other values in the table are larger, and while the words “patient” and “person” may have different connotations for each model based on its training data, this suggests that, to some extent, random noise plays a role in the amount of change each model exhibits. Notably for gender, ethnicity, and sexual orientation, both models change around the same number of answers, except that BioLinkBert has a much higher number for Asian. Additionally, both models have almost double the amount of changed answers for homosexual than bisexual or heterosexual.
Verdict: Supported
Claim: While the reported accuracy on the original test dataset is 38% for QAGNN and 40% for BioLinkBert, the accuracy on our 100 randomly selected demographic-independent questions use to construct the vignettes is 40% for QAGNN and 39% for BioLinkBert.
Evidence type: table
Evidence caption: Table 4: Accuracy (in percentages) of the two models on our demographically enhanced datasets. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation; O*=original test dataset; O=the original, unmodified 100 vignettes; D=No demographic information; Gen=Gender; 1=QAGNN; 2=BioLinkBERT.
Verdict: Supported
Claim: From Table 8 it is clear that BioLinkBert significantly outperforms its generic LM variation.
Evidence type: table
Evidence caption: Table 8: Accuracy (in percentages) of the biomedical and generic models on our demographically enhanced datasets. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation; O*=original test dataset; O=the original, unmodified 100 questions; D=No demographic information; 1=Generic; 2=Biomedical.
Context: Similar to our analysis between QAGNN and BioLinkBert above, our analysis between the biomedical and generic models can be split into the amount of answers and accuracy that changes when the dimensions change. From Table 7 it is visible that the generic transformer has more than double the amount of answers change for each gender. It also has an equivalent amount or more for almost any ethnicity, except for Asians. Notably, for sexual orientation, the generic transformer has almost double the amount of answers change for bisexuals, while the biomedical transformer has more for homosexuals. The generic transformer has significantly larger values than the biomedical transformer in any gender+ethnicity combination, while for gender+sexual orientation, the biomedical system has significantly larger values for homosexuals.
Verdict: Supported
Claim: Compared with the in-domain performance (training and testing on the same dataset), the best zero-shot generalization performance shows a large drop of 20.80% on average.
Evidence type: table
Evidence caption: Table 2: Macro-F1 of 3-class fact verification on the evaluation set for all datasets in a zero-shot generalization setup. Rows correspond to the training dataset and columns to the evaluated dataset. The row SELF corresponds to the in-domain performance (training and testing on the same target dataset).
Context: Table 2 shows the zero-shot generalization results in macro-averaged F1 for the 3-class fact verification task on all the datasets that have supports/refutes/NEI labels, where we partition by dataset group: Group I, top (datasets with artificial claims); Group II, bottom (datasets with natural claims). In general, the RoBERTa model generalizes poorly in this zero-shot setup.
Verdict: Supported
Claim: The bottom left of Table 2 shows that the model trained on natural claims generalizes badly to datasets with artificial claims, with an average F1 drop of 72% relative to the in-domain performance on the three artificial datasets.
Evidence type: table
Evidence caption: Table 2: Macro-F1 of 3-class fact verification on the evaluation set for all datasets in a zero-shot generalization setup. Rows correspond to the training dataset and columns to the evaluated dataset. The row SELF corresponds to the in-domain performance (training and testing on the same target dataset).
Verdict: Supported
Claim: The in-domain results are in line with the empirical observation that Jiang et al. ( 2020 ) it is often ambiguous to differentiate between refutes and NEI claims even for trained human annotators.
Evidence type: table
Evidence caption: Table 5: Class-wise F1 of 3-class fact verification for the zero-shot generalization setup (left) and the in-domain training setup (right). S: supports; R: refutes; N: NEI.
Context: Table 5 shows the breakdown of the class-wise F1 score. For each dataset, we show the average class-wise F1 when training the model on other datasets (zero-shot) and the class-wise F1 for training on the same dataset (in-domain). The results show that the refutes claim has the worst prediction score (in bold) almost for all datasets, in both the zero-shot and the in-domain setting.
Verdict: Supported
Claim: From Table 2 , we find that fact verification in a dataset with document-level evidence is more difficult than in the same dataset with sentence-level evidence (an average of 13.29% drop of in-domain F1).
Evidence type: table
Evidence caption: Table 2: Macro-F1 of 3-class fact verification on the evaluation set for all datasets in a zero-shot generalization setup. Rows correspond to the training dataset and columns to the evaluated dataset. The row SELF corresponds to the in-domain performance (training and testing on the same target dataset).
Verdict: Supported