RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Pregnant women with multiple fetal abnormalities were 3.774 times more likely to undergo labor induction than those with a single fetal abnormality (OR = 3.774, 95% CI [1.640–8.683]).
Caption: Table 2: Analysis of influencing factors of pregnancy outcome in pregnant women with fetal abnormalities.
Claim: The greatest mean maximum load was obtained for the medially positioned plate ( Table 2 ), but the difference between this group and the dorsal LP lag screw and the medial LP lag screw groups was not significant ( p = 0.70 and p = 0.52, respectively).
Context: The addition of an interfragmentary lag screw to the medially positioned plate increased the initial flexural stiffness by 153% and the shear stiffness by 352%, see Table 2 . The dorsal plate with the lag screw was the stiffest construct ( Figs. 5A , 5B ). Significant differences in stiffness were observed ( p < 0.05, p -values are given in the Supplemental File ) between all pairs of fixation techniques used in the MTP-1 joint arthrodesis. A greater initial flexural stiffness than the flexural stiffness in the range of 5–200 N was observed especially in the constructs with the interfragmentary lag screw.
Caption: Table 2: Comparison of mean values of flexural stiffness, initial flexural stiffness, shear stiffness and maximum load, and their standard deviations (SDs) for different fixation methods.
Claim: Results are shown in Table 4 . Human performance is quite strong, almost reaching 90 F1@0 score overall.
Context: To find out how humans perform on the task, we hire two expert annotators with formal education in linguistics. We present them with 10 example instances and then ask them to complete 99 randomly sampled test set instances. We also evaluate our best model (see Table 3 ) on the same set.
Caption: Table 4: Human baseline results (F1@0) by phenomenon and source dataset.
Claim: The contrastive fragment pairing demonstrates superior performance to other pairing methods.
Context: Fragment pairing. Fig. 6 (a) compares contrastive pairing to alternative pairings
using MRAE as a metric.
Caption: Figure 6: Analysis with 40% symmetric noise. (a) Comparison between the proposed contrastive pairing and other pairings on IMDB-Clean-B.
(b) Comparison between fragment numbers on SHIFT15M-B and IMDB-Clean-B.
Claim: Furthermore, pathways of membrane transport and cellular community-prokaryotes were associated with aNEMM group ( Figure 8B ).
Context: To assess the effects of the two media on the predicted gene categories (COGs), we compared the predicted COGs between aNEMM and aSHI, nNEMM and nSHI ( Figure 8A ). NEMM showed more ability in carbohydrate transport and metabolism than SHI medium under both anaerobic conditions and normoxia conditions. Additionally, those in NEMM displayed an increased relative abundance of protein functions linked to defense metabolism, inorganic ion transport and metabolism under anaerobic conditions.
Caption: Figure 8: Function prediction by PICRUSt2.
The differential analysis of COG function (A) and KEGG pathway (B) show the abundance ratio of different functions between groups, whose middle figures show the difference ratio of functional abundance within the 95% confidence interval.
Claim: For gender+ethnicity, QAGNN has an equivalent amount or more than BioLinkBert, though for gender+sexual orientation, BioLinkBert has more than double the amount for homosexuals, with a massive percentage of 23.
Context: The first column of Table 2 , “Random”, shows the result of our random change (Sec. 3.5 ). While the other values in the table are larger, and while the words “patient” and “person” may have different connotations for each model based on its training data, this suggests that, to some extent, random noise plays a role in the amount of change each model exhibits. Notably for gender, ethnicity, and sexual orientation, both models change around the same number of answers, except that BioLinkBert has a much higher number for Asian. Additionally, both models have almost double the amount of changed answers for homosexual than bisexual or heterosexual.
Caption: Table 2: Percentage of questions with changed answers as compared to a question with no demographic information about the patient. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation; Random=Random change as described in Section 3.5 .
Claim: In this setup, Vendi Score underperforms w.r.t. the average cosine dissimilarity.
Context: In Table 2 we report correlation scores for the gender diversity. Again, we split in two groups: male-voice and female voice-dominant. From the results, we notice that the correlations showed by the non-specialized embeddings are extremely weak. However, both specialized models reach very high correlations (e.g., up to 0.946 for SpeechSim/Gender). All in all, gender-specific projector gets the higher correlation score.
Caption: Table 2 : Gender diversity. Average Spearman correlations between proportion of female voices and diversity scores induced by speech representations.
Claim: We note that sampling time of FuncMol is an order of magnitude better than baselines.
Context: Table 1 report the metrics described in Section ˜ 5.1 . We see that FuncMol slightly improves VoxMol and both models perform worse compared to the equivariant point-cloud based baselines.
Caption: Table 1: QM9 results w.r.t. test set for 10000 samples per model. \uparrow / \downarrow indicate that higher/lower numbers are better. The row data are randomly sampled molecules from the validation set. We report 1-sigma error bars over 3 sampling runs.
Claim: 36 pregnant women were advised to carefully consider whether to continue with the termination, and 229 pregnant women were advised to continue with pregnancy follow-up ( Table 1 ).
Context: A total of 398 pregnant women visited our hospital between May 2022 and March 2023. Among them, 273 had a birth outcome (delivery or termination of pregnancy). However, three participants did not answer the phone during the telephone follow-up and five refused to participate in the study. Ultimately, 265 pregnant women were included in this study. A total of 190 pregnant women gave birth either through natural labor ( n = 91) or cesarean section ( n = 99); however, six newborns died within 1 month of birth. Of the remaining 75 pregnant women with fetal abnormalities, 67 chose to induce labor, and eight pregnant women with twin pregnancies opted for fetal reduction. In addition, the average age of the pregnant women was 30.05 ± 4.60 years (range, 20–47 years), with 257 singleton pregnancies and eight twin pregnancies. The gestational age at diagnosis of fetal abnormalities was ≤13 +6 weeks in 5 case (1.9%), 14–27 +6 weeks in 183 case (69.1%), and ≥28 weeks in 77 cases (29%).
Caption: Table 1: Comparison of pregnancy outcomes in pregnant women with different characteristics of fetal abnormalities.
Claim: Compared with the in-domain performance (training and testing on the same dataset), the best zero-shot generalization performance shows a large drop of 20.80% on average.
Context: Table 2 shows the zero-shot generalization results in macro-averaged F1 for the 3-class fact verification task on all the datasets that have supports/refutes/NEI labels, where we partition by dataset group: Group I, top (datasets with artificial claims); Group II, bottom (datasets with natural claims). In general, the RoBERTa model generalizes poorly in this zero-shot setup.
Caption: Table 2: Macro-F1 of 3-class fact verification on the evaluation set for all datasets in a zero-shot generalization setup. Rows correspond to the training dataset and columns to the evaluated dataset. The row SELF corresponds to the in-domain performance (training and testing on the same target dataset).
Claim: It lowers the memory requirement of Hydra [ 20 ] by 7% while improving the accuracy by 0.1 pp .
Context: We conduct experiments evaluating the performance of VeLoRA for full-tuning and how it complements other PEFT methods. In Tab. 1 we reproduce a large set of results for LoRA [ 17 ] , SSF [ 26 ] , and Hydra [ 20 ] on a subset of the VTAB-1K benchmark, where the sub-token size for each experiment is given in the Appendix. Unlike what is more common in the PEFT literature [ 19 , 20 ] , we do not perform any task-specific hyperparameter tuning that would change the memory, such as batch size and rank, and to also avoid any potential overfitting to the specific task. For all experiments we used the authors provided implementations for the adapters and integrated them into the same training framework for a fair comparison. We observe that VeLoRA improves the performance compared to full-tuning by 1.5 percentage points ( pp ), while lowering the memory requirements. We also observe that when combined with PEFT methods, VeLoRA can come with improvement in memory and performance. VeLoRA lowers the memory requirement of SSF [ 26 ] by 16% with only a minor degradation (0.1 pp ) in accuracy.
Caption: Results on a subset of the VTAB-1k benchmark. All methods use a ViT-Base-224/16 model pre-trained on ImageNet-21k. The batch sizes and ranks are the same across all tasks.
Claim: VeLoRA lowers the memory requirement of SSF [ 26 ] by 16% with only a minor degradation (0.1 pp ) in accuracy.
Context: We conduct experiments evaluating the performance of VeLoRA for full-tuning and how it complements other PEFT methods. In Tab. 1 we reproduce a large set of results for LoRA [ 17 ] , SSF [ 26 ] , and Hydra [ 20 ] on a subset of the VTAB-1K benchmark, where the sub-token size for each experiment is given in the Appendix. Unlike what is more common in the PEFT literature [ 19 , 20 ] , we do not perform any task-specific hyperparameter tuning that would change the memory, such as batch size and rank, and to also avoid any potential overfitting to the specific task. For all experiments we used the authors provided implementations for the adapters and integrated them into the same training framework for a fair comparison. We observe that VeLoRA improves the performance compared to full-tuning by 1.5 percentage points ( pp ), while lowering the memory requirements. We also observe that when combined with PEFT methods, VeLoRA can come with improvement in memory and performance.
Caption: Results on a subset of the VTAB-1k benchmark. All methods use a ViT-Base-224/16 model pre-trained on ImageNet-21k. The batch sizes and ranks are the same across all tasks.
Claim: We observe accuracies between 60-80% across all tasks and observe that debiasing can substantially increase accuracy.
Context: One can also measure the accuracy of the comparative system at a comparison level. Table 7 shows the pairwise comparison accuracy for Summeval, over all candidate pairs where the true score of the candidate response varies.
Caption: Table 7: Accuracy of the comparative systems, at a comparison level, for SummEval.
Claim: Dorea was positively associated with phenylalanine, butyric acid, and isovaleric acid, while showed a negative correlation with hydracrylic acid, CH 4 , myo-inositol, L-threonine, glycerol, CO 2 , L-cysteine, and D-gluconic acid.
Context: To further explore correlations of metabolites in human fecal matter and intestinal flora after SNP treatment, thermography was used to analyze contents of metabolites showing significant differences with 30 most abundant genera. Figure 10 shows the results. Bifidobacterium was positively associated with glycerol, CO 2 , 4-aminobutanoic acid, L-cysteine, and D-gluconic acid, and could be negatively correlated with DL-phenylalanine. Faecalibacterium was positively associated with pentanoic acid and 4-aminobutanoic acid, while were negatively correlated with hydracrylic acid. Lactobacillus was positively correlated with xylitol and showed a negative correlation with 5-aminovaleric acid.
Caption: Figure 10: Correlation analysis of characteristic metabolites and bacteria.
Corr represents correlation. Red and blue separately denote positive and negative correlations. The darker the color, the stronger the correlation. *P < 0.05, **P < 0.01, and ***P < 0.001.
Claim: First, this results shows that the architecture needs to be deep enough, because it scales in \frac{1}{N} : for instance, we fail to train a ResNet-101 (He et al., 2016a ) on the ImageNet dataset using the adjoint method on its third layer (depth 23 ), as shown in Figure 3 (b).
Context: In Prop. 3 we showed under assumption 2 , that is if the residuals are bounded and Lipschitz continuous with constant independent of the depth N , then the error for computing the activations backward would scale in \frac{1}{N} as well as the error for the gradients (Prop. 4 ).
Caption: Figure 3: (a) Test accuracy on CIFAR-10 as a function of the number of blocks in each layer of the ResNet. Within each layer, weights are tied ( 3 runs ). (b) Failure of the adjoint method with a ResNet-101 on ImageNet (the approximated gradients are only used in the third layer of the network, that contains 23 blocks). (c) Relative error between the approximated gradients using adjoint method and the true gradient, whether using a ResNet or a HeunNet. Each point corresponds to one parameter.
Claim: These are all assessed using the accuracy metric.
Context: This study considers four different close-ended healthcare tasks, which include nine different datasets ( e.g. , MedQA).
Caption: Table 1: This table presents the tasks implemented in this paper. The first column specifies the different tasks. The second details the metrics used (ROUGE includes ROUGE1, ROUGE2 and ROUGEL, and Perplexity includes Bits per Byte, Byte Perplexity, and Word Perplexity). The third column outlines the benchmarks used for each task.
Claim: In numerical domains (Number, Number+Text), LLMs are much more accurate in QA versus RQA , revealing a clear abduction weakness.
Context: We first see if RQA ( red , no stripe) or QA ( blue , striped) is consistently harder for LLMs (Figure 2 ).
Caption: Figure 2: LLM RQA ( blue ) and QA ( red ) accuracy with 95% CIs for metric error rate. LLMs are much weaker in abductive RQA in numerical settings (Number/Number+Text), but in text settings (Easy/Hard Entity), deductive QA is slightly weaker.
Claim: When the LLM’s answers contain calculations, the accuracy drops significantly for almost all models except for PaLM.
Context: After discussing redundancy and accuracy independently, we want to know if redundant calculation co-occurs more often with wrong answers. We separate the model outputs into two groups: one that contains calculations and another that does not have calculations, and we calculate the accuracy for the two groups. The results are shown in the two rightmost columns in Table 1 .
Caption: Table 1: The redundancy ( Red. ) and accuracy of LLMs’ responses.
We report the average accuracy ( Avg. ) on all questions (second column), the accuracy for answers without calculation (Cal. ✗, third column) and with calculation (Cal. ✓, fourth column). \dagger : The accuracy of GPT-4 is 100% by construction since we use GPT-4 to filter samples when constructing GSM8K-Zero.
Claim: Lastly, we note that using larger values of \gamma naively does not guarantee better test performance: one has to fine tune the parameters \beta , \mu , \sigma , \eta appropriately to achieve favorable trade-off between training stability and test performance.
Context: For the training, we use a fully connected shallow neural network of width 16 with ReLU activation and train for 3000 epochs with the learning rate \eta=0.1 , using mean square error (MSE) as the loss and choosing \beta=0.5 . Table 1 reports the average root MSE (RMSE) and the RMSE gap (defined as test RMSE - train RMSE) evaluated for models that are trained with 5 different seed values for this task. We can see that MPGD leads to both lower test RMSE and RMSE gap when compared with vanilla GD (baseline) and GD with uncorrelated Gaussian perturbations (see the results not in parenthesis in Table 1 ; here \mu=0.01 , \sigma=0.02 ). Using the form of the perturbations in ( 11 ) instead can also give lower RMSE gap (see the results in parenthesis in Table 1 ; here \sigma=\mu=0.01 ). Overall these results support our generalization theory for MPGD.
Caption: Table 1: Shallow neural nets trained on the airfoil data set. The results in parenthesis are achieved with the variant ( 11 ).
All the results are averaged over 5 models trained with different seed values.
Claim: IS was negatively correlated with SS (Index = −0.179, P < 0.001).
Context: A multiple mediation analysis was conducted to explore the mediation effects of IS and SS in a college student population. Control variables included gender, age, BMI, GPA, being an only child or not, and home address. AA and depression were entered as independent and dependent variables respectively. The proposed mediators were IS and SS. Results of the analysis ( Table 3 ) showed that AA was positively correlated with depression (Index = 0.168, P < 0.001). Secondly, AA was positively correlated with IS (Index = 0.575, P < 0.001) and negatively correlated with SS (Index = −0.234, P < 0.001).
Caption: Table 3: Regression analysis of appearance anxiety, interpersonal sensitivity, social support, and depression.
Claim: The selection is predominantly positive, with 8 positive and 1 negative fact; about half of the negative fact has an impact strength of 2–3.
Context: Supporting Facts. We analyzed the supporting facts identified by the model in cases of correct decisions after reflections. Statistics are presented in Table 3 . Each transcript is distilled into a table of about 40 facts, from which the model selects 9.
Caption: Table 3: Statistics of supporting facts.
Claim: Specifically, the T5-3B model trained on the augmented training set achieved an EX score of 4.81% on the English test (matching the performance of GPT-3.5+CT-3+COT) set and 1.92% on the Chinese test set (matching the performance of GPT-3.5+CT-3).
Context: From Table 2 . we observe that T5 from scale base to 3B (XL) trained on Archer training set achieve 0.00% EX scores. This outcome could be attributed to the small-scale nature of Archer combined with its high complexity. However, when Archer training set was augmented with the Spider/CSpider training set, the VA scores of T5 models exhibited a substantial improvement.
Caption: Table 2: Baseline performance on Archer. GPT-4+DIN-SQL was tested only on the English set due to cost and its English-specific design. We only report the fine-tuned model’s performance on the test set.