RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: In 6 of the 7 datasets with natural claims, the best generalization score is from a model trained on artificial claims.
Context: Many works Jiang et al. ( 2020 ); Saakyan et al. ( 2021 ) do not consider NEI claims due to their ambiguity. To explore whether our previous observations also hold for the task of binary fact verification , we evaluate the generalization results for all 11 datasets using only the supports and refutes claims for training and evaluation, shown in Table 3 . In this setting, artificial claims also generalize well to natural claims in other domains.
Caption: Table 3: F1 of binary fact verification on the evaluation set for all datasets in a zero-shot generalization setup. Rows correspond to the training dataset and columns to the evaluated dataset.
Claim: There were nine male and nine female participants in each group ( Table 1 ).
Context: Fifty-four participants were recruited by advertisement from universities in Taipei. This sample size was determined based on an a priori power analysis (alpha = 0.05, power = 0.80), and the effect size, f = 0.44 ( Cohen, 1992 ), and similar to that seen in ERP studies investigating visuospatial attention and working memory in athletes ( Chueh et al., 2017 ). They were divided into three groups based on their sport level: eighteen ETT players, eighteen ATT players, and eighteen nonathletes. ETT athletes (age 20.2 ± 2.4) were those who competed at the national division 1 level (top 3), and international competition experience ( n = 10). ATT athletes (age 22.2 ± 2.2) had no experience competing at national division 1 level, but with national division 2 level experience (top 8). Nonathletes (age 20.7 ± 1.2) were those who had not engaged in regular physical activity as measured by the International Physical Activity Questionnaire (IPAQ) ( Liou et al., 2008 ). Both levels of table tennis players maintained regular training for at least 6 months preceding their participation in the experiment.
Caption: Table 1: Demographics of the participants in each group (mean ± SD).
Claim: ConFrag achieves the lowest ERR while maintaining above-average selection rates, resulting in the best MRAE.
Context: Selection/ERR/MRAE comparison. Fig. 5 compares ConFrag to five selection and refurbishment baselines of CNLCU-H, BMM, DY-S, AUX, Selfie on IMDB-Clean-B using the selection rate, ERR, and MRAE. Ideally, a model should attain a high selection rate and a low ERR. It is worth noting that the relative importance of ERR and selection rate may vary depending on the dataset and the task.
Caption: Figure 5: Selection/ERR/MRAE comparison between ConFrag and strong baselines of CNLCU-H, BMM, DY-S, AUX and Selfie on IMDB-Clean-B. We exclude the performance during the warm-up.
Claim: The results show that UTs are able to generalize better for the A and B splits, outperforming the LSTM and VT.
Context: We train a 12 layer model with Attention (E=12,k=4,H=2,D=32,W=1) and FFD (E=12,K=4,D=128) and halting. Refer to Appendix A for further details. Training a 12-layer Vanilla Transformer achieves approximately the same results as in Shen et al. ( 2019 ) , so we report their results. Our results in Table 5 confirm the findings of Tran et al. ( 2018 ) , showing that with recurrence in SUTs, we are able to generalize to longer sequences of the task. While there are other models that induce a tree structure that performs exceedingly well on the task, we wanted to evaluate our model against other popular architectures. The LSTM is a strong baseline, and we find that UT outperforms it in generalization. We also evaluate UTs on the compositional generalization splits as proposed in Shen et al. ( 2019 ) , where the splits A, B, and C are in increasing difficulty.
Caption: Table 5: Test accuracy of the models, trained on operation lengths of \leq 6 , with their out-of-distribution results shown here (lengths 7-12).
LSTM baseline from Bowman et al. ( 2015 ) , and Transformer baseline from Shen et al. ( 2019 )
Claim: While the performance of the Decider model almost overlaps with the Q Drawer asking questions in random dialogue turns, using model uncertainty significantly improves accuracy, even when a few questions are asked.
Context: We compare the uncertainty-driven Q Drawer against (1) a version that asks questions in random dialogue turns with different average number of questions per dialogue and (2) the Q Drawer paired with the Decider module as described in Section 6.2 . For the latter, the maximum possible number of questions per dialogue (average of 1.5) is defined by the Decider model output. Figure 4 shows the effect of the number of questions on size accuracy for different model configurations.
Caption: Figure 4: Effect of the average number of questions per dialogue (considering all dialogues in the test set) on the size accuracy. We compare the uncertainty-guided Q Drawer with a version that asks questions in random turns and one that asks questions in the turns selected by an external Decider model.
Claim: It is also intriguing to observe from Figure 5 that Vanilla LLaMA-2-70B excelled in Writing, but full-parameter fine-tuning led to a decline in these areas, a phenomenon known as the “Alignment Tax” (Ouyang et al., 2022 ) .
Caption: Figure 5: Different aspects of LLaMA-2-70B model on MT-Bench.
Claim: STRUX outperforms strong baselines in accuracy and F-scores for stock investment decisions.
Context: System Comparisons. Table 1 shows the macro-averaged precision, recall, F-scores, and accuracy for the test set.
Caption: Table 1: Our STRUX system outperforms strong benchmarks in making stock investment decisions. We present macro-averaged precision, recall, F-scores, accuracy for the test set. LLMs evaluated are: Llama3-8b-Instruct and gpt-4o-mini-2024-07-18 .
Claim: However, on our internal Payslips datasets, our model pre-trained on DOCILE outperforms the original one.
Context: We report labeled F1-score averaged on 100 fine-tuning runs in Table 2 .
Precision and recall are reported in Appendix B .
The Base model (using the full 12 layers) produces similar results on DOCILE no matter if pre-training on IIT-CDIP or DOCILE .
Caption: Table 2: F1 scores for named-entity recognition using different pre-training and fine-tuning datasets. Results are averaged on 100 runs with different seeds.
Claim: For Factuality, the annotators demonstrate moderate agreement (Krippendorff’s Alpha = 0.68), suggesting that the conversations are consistently viewed as highly factual, which implies that the SQL queries in our dataset are of high quality.
Context: Table 4 shows the mean, standard deviation, and Krippendorff’s Alpha for inter-annotator agreement. The high mean scores close to 1 (1.15-1.5) and substantial agreement (Alpha 0.68-0.82) indicate high-quality, natural conversations with factual and helpful responses. For Naturalness, we observe that annotators have a substantial agreement (Krippendorff’s Alpha = 0.82), indicating that the conversations are generally perceived as natural and fluent.
Caption: Table 4: Summary of Human Annotation Scores for Naturalness, Factuality, and Helpfulness.
Claim: It also has an equivalent amount or more for almost any ethnicity, except for Asians.
Context: Similar to our analysis between QAGNN and BioLinkBert above, our analysis between the biomedical and generic models can be split into the amount of answers and accuracy that changes when the dimensions change. From Table 7 it is visible that the generic transformer has more than double the amount of answers change for each gender.
Caption: Table 7: Percentage of questions with changed answers between the biomedical and generic model as compared to a question with no demographic information about the patient. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation.
Claim: Parabacteroides had positive correlations with L-leucine, L-valine, and L-isoleucine, while negatively correlated with glycerol, CO 2 , 4-aminobutanoic acid, L-cysteine, D-gluconic acid, and propionic acid.
Context: To further explore correlations of metabolites in human fecal matter and intestinal flora after SNP treatment, thermography was used to analyze contents of metabolites showing significant differences with 30 most abundant genera. Figure 10 shows the results. Bifidobacterium was positively associated with glycerol, CO 2 , 4-aminobutanoic acid, L-cysteine, and D-gluconic acid, and could be negatively correlated with DL-phenylalanine. Faecalibacterium was positively associated with pentanoic acid and 4-aminobutanoic acid, while were negatively correlated with hydracrylic acid. Lactobacillus was positively correlated with xylitol and showed a negative correlation with 5-aminovaleric acid. Dorea was positively associated with phenylalanine, butyric acid, and isovaleric acid, while showed a negative correlation with hydracrylic acid, CH 4 , myo-inositol, L-threonine, glycerol, CO 2 , L-cysteine, and D-gluconic acid. Escherichia - Shigella could be positively associated with H 2 S, L-proline, cadaverine, DL-phenylalanine, benzeneacetic acid, 1,3-propanediol, and were negatively correlated with glycerol, CO 2, 4-aminobutanoic acid, L-cysteine, D-gluconic acid, propionic acid, and L-lysine. Phascolarctobacterium was positively associated with 1,3-propanediol, L-proline, cadaverine, and DL-phenylalanine, and was negatively correlated with glycerol, CO 2 , 4-aminobutanoic acid, L-cysteine, and myo-inositol. Lachnospiraceae were positively associated with 1, 3-propanediol and L-proline, and were negatively correlated with hydracrylic acid and CH 4 .
Caption: Figure 10: Correlation analysis of characteristic metabolites and bacteria.
Corr represents correlation. Red and blue separately denote positive and negative correlations. The darker the color, the stronger the correlation. *P < 0.05, **P < 0.01, and ***P < 0.001.
Claim: A first observation is that one can transfer these weights to deeper ResNets without significantly affecting the test accuracy of the model: it remains above 94.5\% on CIFAR-10 and 72\% on ImageNet.
Context: In addition, we also want our pretrained model to verify assumption 2 so we consider the following setup. On CIFAR (resp. ImageNet) we train a ResNet with 4 (resp. 8) blocks in each layer, where weights are tied within each layer.
Caption: Table 2: Test accuracy (ResNet)