RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: 10 . From Fig. 4 -left, we notice that Mistral exhibits the most bias, with the BiasF_{o} value ( Eq. 8 ) of 353.80\%^{2} .
Evidence type: figure
Caption: (a); (b); Average EstTrueF1 (SemEval2017) and EstTrueMAP (SciDocsRR) ( Section 3.1 ) across models (left) and benchmarks (right) showing performance difference of LLMs across 4 widely used list formats.
Context: Fig. 4 displays the key findings of our evaluation across models and datasets with numerical results in Appx.- Tab.
Verdict: Supported
Claim: In contrast, ChatGPT and Gemma show much lower bias, with values of 7.08\%^{2} and 1.32\%^{2} , respectively.
Evidence type: figure
Caption: (a); (b); Average EstTrueF1 (SemEval2017) and EstTrueMAP (SciDocsRR) ( Section 3.1 ) across models (left) and benchmarks (right) showing performance difference of LLMs across 4 widely used list formats.
Context: Fig. 4 displays the key findings of our evaluation across models and datasets with numerical results in Appx.- Tab. 10 . From Fig. 4 -left, we notice that Mistral exhibits the most bias, with the BiasF_{o} value ( Eq. 8 ) of 353.80\%^{2} .
Verdict: Supported
Claim: Compared to ELoRA we can yield up to 1.86\times and 2.96\times runtime and FLOPs improvement while remain comparable with LoRA in these two metrices.
Evidence type: figure
Caption: Figure 3: A comparison of various system performance between LoRA, ELoRA, and AFLoRA.
Context: Runtime & FLOPs Comparison. Fig. 3 shows the comparison of the normalized average training runtime, normalized FLOPs, and normalized trainable parameters. For AFLoRA, we average the training time, FLOPs, and trainable parameters over six GLUE datasets (except the MNLI and QQP datasets). Note, for LoRA and ELoRA, the trainable parameters and FLOPs remain fixed for all the dataset. We compute their average runtime same way as ours.
Verdict: Supported
Claim: Compared to LoRA we yield 9.5\times parameter reduction, while remain comparable with ELoRA.
Evidence type: figure
Caption: Figure 3: A comparison of various system performance between LoRA, ELoRA, and AFLoRA.
Context: Runtime & FLOPs Comparison. Fig. 3 shows the comparison of the normalized average training runtime, normalized FLOPs, and normalized trainable parameters. For AFLoRA, we average the training time, FLOPs, and trainable parameters over six GLUE datasets (except the MNLI and QQP datasets). Note, for LoRA and ELoRA, the trainable parameters and FLOPs remain fixed for all the dataset. We compute their average runtime same way as ours. Compared to ELoRA we can yield up to 1.86\times and 2.96\times runtime and FLOPs improvement while remain comparable with LoRA in these two metrices.
Verdict: Supported
Claim: In specific, we illustrate the specific number of iterations required before freezing each component in Fig. 5 . Interestingly, as can be seen from the figure, analysis reveals that the down-projection matrix parallel the intermediate linear layer require longer training duration prior to being frozen, as compared to the other PMs.
Evidence type: figure
Caption: Figure 5: Visualization of freezing iterations for each layer. ‘out’ and ‘inter’ refer to the second and the first MLP layer of the FFN, respectively. ‘A’ and ‘B’ represent the down-projection and up-projection matrix, respectively. The darker the color, the more iterations the matrix has to go through before freezing.
Context: Discussion on Freezing Trend. We use the RTE dataset as a case study, to understand the freezing trend of the PMs across different layers.
Verdict: Supported
Claim: These variables include age ( p = 0.002), having children ( p = 0.002), being related to the health sector ( p = 0.001), perception of the efficacy and protective effect of the booster dose (<0.001), as well as fear of adverse effects ( p < 0.001) were significantly associated with the IBV ( Table 2 ).
Evidence type: table
Caption: Table 2: Bivariate analysis of study characteristics according to intention to be vaccinated with the booster dose COVID-19 in Peru, July 2022 (n= 924).
Context: Several variables were found to be significantly associated with the intention to be vaccinated (IBV), as detailed in Table 2 .
Verdict: Supported
Claim: These findings demonstrate that BiAMIL is a more accurate and reliable method for estimating the BRCA status compared to the traditional ResNet 34 model.
Evidence type: table
Caption: Table 2: The performance of the BiAMIL in predictingBRCAPV in hormone receptor + and triple-negative breast cancer.
Context: The BiAMIL model was developed by first inputting normalized tiles into the ResNet 34 model to extract 1,000-dimensional features. These features were then aggregated using a Bi-directional self-attention mechanism approach. Subsequently, the aggregated feature matrix was fed into a multilayer perceptron (MLP) classifier to predict the probability of BRCA gene mutation status for each WSI ( Figure 3 ). BiAMIL achieved AUC values of 0.819 (95% CI [0.673–0.965]) in the internal test set ( Figure 4A ), and 0.817 (95% CI [0.712–0.923]) in the external test set ( Figure 4B ). The five-fold cross-validation results for the BiAMIL model on the training set are displayed in Table S1 . The model was compared with the ResNet 34 by Wang et al. (2021) . ResNet 34 achieved AUC values of 0.783 (95% CI [0.624–0.941]) the internal test set ( Figure 4C ), and 0.684 (95% CI [0.559–0.810]) in the external test sets ( Figure 4D ). The five-fold cross-validation results for the ResNet 34 model on the training set are shown in Table S2 . In the internal test and external test sets, BiAMIL outperformed the ResNet 34 model in accuracy, sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F1 score ( Figs. 4E and 4F ). Due to hormone receptor-positive and triple-negative breast cancer accounting for 80% of breast cancer cases and the high occurrence of BRCA1 in triple-negative breast cancer, a subgroup analysis of BiAMIL’s performance in these subtypes was conducted, with results shown in Table 2 .
Verdict: Supported
Claim: As shown in Table 2 , there was a statistically significant difference in the frequency of colds experienced by middle-aged and elderly individuals with varying levels of weekly PA ( P < 0.001).
Evidence type: table
Caption: Table 2: Cold in middle-aged and elderly with different characteristics.
Verdict: Supported
Claim: However, no significant differences were found in cold frequency between participants of different sexes or age groups ( P > 0.05).
Evidence type: table
Caption: Table 2: Cold in middle-aged and elderly with different characteristics.
Context: As shown in Table 2 , there was a statistically significant difference in the frequency of colds experienced by middle-aged and elderly individuals with varying levels of weekly PA ( P < 0.001).
Verdict: Supported
Claim: Fifty healthy students from University Europea de Madrid volunteered for the study Table 1 .
Evidence type: table
Caption: Table 1: Descriptive data.
Verdict: Supported
Claim: No study demonstrated sufficient validity for TMS biomarker use ( Bielekova & Martin, 2004 ).
Evidence type: table
Caption: Table 6: Biomarker assessment.
Context: See Table 6 for a condensed summary of biomarker assessment findings and Table S6 for more detailed results.
Verdict: Supported
Claim: In addition, significant differences were observed in all SU and SD kinematics except for KHD during SD.
Evidence type: table
Caption: Table 1: Participants characteristics.
Context: A total of 43 recreational table tennis players were included in this study, with 44 and 42 legs in the non-EOA and EOA groups, respectively ( Table 1 ). There were no significant differences between the groups in terms of sex distribution ( p = 0.22), age ( p = 0.46), height ( p = 0.07), weight ( p = 0.78), BMI ( p = 0.29), or KHD during SD ( p = 0.05).
Verdict: Supported
Claim: Tukey’s post-hoc analysis revealed that those in C2 had the youngest age (44.80 ± 11.30 years, p < 0.001 vs. all other clusters) and highest BMI (25.59 ± 2.42, p < 0.001 vs. all other clusters).
Evidence type: table
Caption: Table 2: Comparisons of features between the clusters classified by Louvain clustering unsupervised machine learning.
Context: To validate the quality and distinctiveness of the Louvain clustering solution, we calculated widely-used cluster validity indices. The Davies–Bouldin index was 2.09, and the Calinski–Harabasz index was 59.26. While the Davies–Bouldin index was somewhat high (lower values indicate better separation), the Calinski–Harabasz index supported the presence of distinct groupings in our data. The Louvain clustering algorithm identified four distinct clusters (C1–C4) with significant differences in all features (all p < 0.001, except for AHD-SU ( p = 0.017) and portion of EOA ( p = 0.019)) ( Table 2 ).
Verdict: Supported
Claim: C2 included exclusively males, whereas the other clusters had a mixed sex composition.
Evidence type: table
Caption: Table 2: Comparisons of features between the clusters classified by Louvain clustering unsupervised machine learning.
Context: To validate the quality and distinctiveness of the Louvain clustering solution, we calculated widely-used cluster validity indices. The Davies–Bouldin index was 2.09, and the Calinski–Harabasz index was 59.26. While the Davies–Bouldin index was somewhat high (lower values indicate better separation), the Calinski–Harabasz index supported the presence of distinct groupings in our data. The Louvain clustering algorithm identified four distinct clusters (C1–C4) with significant differences in all features (all p < 0.001, except for AHD-SU ( p = 0.017) and portion of EOA ( p = 0.019)) ( Table 2 ). Tukey’s post-hoc analysis revealed that those in C2 had the youngest age (44.80 ± 11.30 years, p < 0.001 vs. all other clusters) and highest BMI (25.59 ± 2.42, p < 0.001 vs. all other clusters).
Verdict: Supported
Claim: As shown in Table 1 , there are significant differences in the mean of AA scores across grade levels and BMI index (F = 3.402 P = 0.009; F = 5.752 P = 0.003).
Evidence type: table
Caption: Table 1: Differences in appearance anxiety, interpersonal sensitivity, social support, and depression under different subgroups of basic demographic characteristics.
Context: All participants were asked to complete a questionnaire that included gender, age, grade, height, weight, whether they were an only child, whether their home address was town or village, monthly household income, GPA for the previous school year, and disposable income. Specific issues and analysis results are shown in Table 1 .
Verdict: Supported
Claim: Results of the analysis ( Table 3 ) showed that AA was positively correlated with depression (Index = 0.168, P < 0.001).
Evidence type: table
Caption: Table 3: Regression analysis of appearance anxiety, interpersonal sensitivity, social support, and depression.
Context: A multiple mediation analysis was conducted to explore the mediation effects of IS and SS in a college student population. Control variables included gender, age, BMI, GPA, being an only child or not, and home address. AA and depression were entered as independent and dependent variables respectively. The proposed mediators were IS and SS.
Verdict: Supported
Claim: Secondly, AA was positively correlated with IS (Index = 0.575, P < 0.001) and negatively correlated with SS (Index = −0.234, P < 0.001).
Evidence type: table
Caption: Table 3: Regression analysis of appearance anxiety, interpersonal sensitivity, social support, and depression.
Context: A multiple mediation analysis was conducted to explore the mediation effects of IS and SS in a college student population. Control variables included gender, age, BMI, GPA, being an only child or not, and home address. AA and depression were entered as independent and dependent variables respectively. The proposed mediators were IS and SS. Results of the analysis ( Table 3 ) showed that AA was positively correlated with depression (Index = 0.168, P < 0.001).
Verdict: Supported
Claim: IS was negatively correlated with SS (Index = −0.179, P < 0.001).
Evidence type: table
Caption: Table 3: Regression analysis of appearance anxiety, interpersonal sensitivity, social support, and depression.
Context: A multiple mediation analysis was conducted to explore the mediation effects of IS and SS in a college student population. Control variables included gender, age, BMI, GPA, being an only child or not, and home address. AA and depression were entered as independent and dependent variables respectively. The proposed mediators were IS and SS. Results of the analysis ( Table 3 ) showed that AA was positively correlated with depression (Index = 0.168, P < 0.001). Secondly, AA was positively correlated with IS (Index = 0.575, P < 0.001) and negatively correlated with SS (Index = −0.234, P < 0.001).
Verdict: Supported
Claim: There were nine male and nine female participants in each group ( Table 1 ).
Evidence type: table
Caption: Table 1: Demographics of the participants in each group (mean ± SD).
Context: Fifty-four participants were recruited by advertisement from universities in Taipei. This sample size was determined based on an a priori power analysis (alpha = 0.05, power = 0.80), and the effect size, f = 0.44 ( Cohen, 1992 ), and similar to that seen in ERP studies investigating visuospatial attention and working memory in athletes ( Chueh et al., 2017 ). They were divided into three groups based on their sport level: eighteen ETT players, eighteen ATT players, and eighteen nonathletes. ETT athletes (age 20.2 ± 2.4) were those who competed at the national division 1 level (top 3), and international competition experience ( n = 10). ATT athletes (age 22.2 ± 2.2) had no experience competing at national division 1 level, but with national division 2 level experience (top 8). Nonathletes (age 20.7 ± 1.2) were those who had not engaged in regular physical activity as measured by the International Physical Activity Questionnaire (IPAQ) ( Liou et al., 2008 ). Both levels of table tennis players maintained regular training for at least 6 months preceding their participation in the experiment.
Verdict: Supported
Claim: There was a significant difference in IQ ( F (2, 51) = 9.061, p < 0.01, η 2 = 0.262), and a post hoc comparison revealed that ETT players had a lower IQ than ATT players ( p < 0.003) and nonathletes ( p < 0.001), but no significant difference between the ATT and nonathletes ( p < 0.368) was observed.
Evidence type: table
Caption: Table 1: Demographics of the participants in each group (mean ± SD).
Context: Table 1 presents the participants’ characteristics. No significant differences were observed in age ( F (2, 51) = 2.863, p = 0.066), height ( F (2, 51) = 0.420, p = 0.659), weight ( F (2, 51) = 0.556, p = 0.577), handedness scores ( F (2, 51) = 0.536, p = 0.589), video game experience ( F (2, 51) = 1.387, p = 0.259), and socioeconomic status of the family ( F (2, 51) = 1.579, p = 0.216) among groups.
Verdict: Supported
Claim: A total of 54 COPD patients (25 in stages I–II and 29 in stages III–IV) and 24 healthy individuals participated in the study from September 9, 2021, to May 1, 2022.
Evidence type: table
Caption: Table 1: Participants’ clinical characteristics and demographics.
Verdict: Supported
Claim: 36 pregnant women were advised to carefully consider whether to continue with the termination, and 229 pregnant women were advised to continue with pregnancy follow-up ( Table 1 ).
Evidence type: table
Caption: Table 1: Comparison of pregnancy outcomes in pregnant women with different characteristics of fetal abnormalities.
Context: A total of 398 pregnant women visited our hospital between May 2022 and March 2023. Among them, 273 had a birth outcome (delivery or termination of pregnancy). However, three participants did not answer the phone during the telephone follow-up and five refused to participate in the study. Ultimately, 265 pregnant women were included in this study. A total of 190 pregnant women gave birth either through natural labor ( n = 91) or cesarean section ( n = 99); however, six newborns died within 1 month of birth. Of the remaining 75 pregnant women with fetal abnormalities, 67 chose to induce labor, and eight pregnant women with twin pregnancies opted for fetal reduction. In addition, the average age of the pregnant women was 30.05 ± 4.60 years (range, 20–47 years), with 257 singleton pregnancies and eight twin pregnancies. The gestational age at diagnosis of fetal abnormalities was ≤13 +6 weeks in 5 case (1.9%), 14–27 +6 weeks in 183 case (69.1%), and ≥28 weeks in 77 cases (29%).
Verdict: Supported
Claim: Chi-square test analysis showed significant differences in the gestational age at diagnosis of fetal abnormalities, the number of fetal abnormalities, diagnostic gestational age, and treatment recommendations of doctors among pregnant women ( P < 0.05).
Evidence type: table
Caption: Table 1: Comparison of pregnancy outcomes in pregnant women with different characteristics of fetal abnormalities.
Verdict: Supported
Claim: However, there were no significant differences in maternal age, number of births, abortion history, history of fetal abnormalities, mode of conception, pregnancy complications, education level, employment status, or place of abode ( Table 1 ).
Evidence type: table
Caption: Table 1: Comparison of pregnancy outcomes in pregnant women with different characteristics of fetal abnormalities.
Context: Chi-square test analysis showed significant differences in the gestational age at diagnosis of fetal abnormalities, the number of fetal abnormalities, diagnostic gestational age, and treatment recommendations of doctors among pregnant women ( P < 0.05).
Verdict: Supported
Claim: Pregnant women with multiple fetal abnormalities were 3.774 times more likely to undergo labor induction than those with a single fetal abnormality (OR = 3.774, 95% CI [1.640–8.683]).
Evidence type: table
Caption: Table 2: Analysis of influencing factors of pregnancy outcome in pregnant women with fetal abnormalities.
Verdict: Supported
Claim: Pregnant women who were advised to terminate their pregnancies or carefully consider whether to continue gestation were 41.113 times more likely to undergo labor induction compared to those who were advised to continue the pregnancy (OR = 41.113, 95% CI [11.028–153.267]), ( Table 2 ).
Evidence type: table
Caption: Table 2: Analysis of influencing factors of pregnancy outcome in pregnant women with fetal abnormalities.
Context: Pregnant women with multiple fetal abnormalities were 3.774 times more likely to undergo labor induction than those with a single fetal abnormality (OR = 3.774, 95% CI [1.640–8.683]).
Verdict: Supported
Claim: A day effect was only found for the Nine Hole Peg test, revealing that the time for completing the task during the assessment-2 was less than during the assessment-1 (MD: 0.39 s; 95% CI [0.19–0.60]; P < 0.001).
Evidence type: table
Caption: Table 2: Clinical characteristics of participants in the different test for the wrist extensor muscles.
Context: No side * day interaction was found in the RM-ANOVA for any of the variables ( P > 0.114), indicating a similar pattern across assessments of the dominant and non-dominant sides.
Verdict: Supported
Claim: For the active range of motion, reliability was ‘excellent’ for both extension, flexion, and total range (ICC: 0.94–0.97).
Evidence type: table
Caption: Table 3: Reliability indicators of myotonometry, manual dexterity, pressure pain thresholds, active range of motion, and maximal isometric strength for the wrist extensor muscles.
Context: Table 3 presents ICC 3,1 , SEM, MDC 90 , and MDC 95 for each variable. For the muscle mechanical properties of the dominant and non-dominant sides, reliability was ‘excellent’ for the frequency and stiffness parameters (ICC: 0.91–0.96), and ‘good’ to ‘excellent’ for the decrement (ICC: 0.86–0.91). For the manual dexterity, reliability was ‘good’ on both sides (ICC: 0.79–0.82). For the pressure pain thresholds, reliability was ‘excellent’ for the extensor carpi radialis brevis point (ICC: 0.90–0.93), and ‘good’ for the deltoideus point (ICC: 0.84–0.89).
Verdict: Supported
Claim: Similarly, reliability was ‘excellent’ for the maximal isometric strength on both sides (ICC: 0.97).
Evidence type: table
Caption: Table 3: Reliability indicators of myotonometry, manual dexterity, pressure pain thresholds, active range of motion, and maximal isometric strength for the wrist extensor muscles.
Context: Table 3 presents ICC 3,1 , SEM, MDC 90 , and MDC 95 for each variable. For the muscle mechanical properties of the dominant and non-dominant sides, reliability was ‘excellent’ for the frequency and stiffness parameters (ICC: 0.91–0.96), and ‘good’ to ‘excellent’ for the decrement (ICC: 0.86–0.91). For the manual dexterity, reliability was ‘good’ on both sides (ICC: 0.79–0.82). For the pressure pain thresholds, reliability was ‘excellent’ for the extensor carpi radialis brevis point (ICC: 0.90–0.93), and ‘good’ for the deltoideus point (ICC: 0.84–0.89). For the active range of motion, reliability was ‘excellent’ for both extension, flexion, and total range (ICC: 0.94–0.97).
Verdict: Supported
Claim: In the case of the asymmetries, reliability was ‘excellent’ for the active range of motion in flexion (ICC: 0.96); ‘good’ for the myotonometry parameters (ICC: 0.72–0.78), the Nine-Hole Peg test scoring (ICC: 0.72), the total and extension active range of motion (ICC: 0.73–0.89), and the maximal isometric strength (ICC: 0.73); and ‘moderate’ for the pressure pain thresholds on both points (ICC: 0.53–0.58).
Evidence type: table
Caption: Table 3: Reliability indicators of myotonometry, manual dexterity, pressure pain thresholds, active range of motion, and maximal isometric strength for the wrist extensor muscles.
Context: Table 3 presents ICC 3,1 , SEM, MDC 90 , and MDC 95 for each variable. For the muscle mechanical properties of the dominant and non-dominant sides, reliability was ‘excellent’ for the frequency and stiffness parameters (ICC: 0.91–0.96), and ‘good’ to ‘excellent’ for the decrement (ICC: 0.86–0.91). For the manual dexterity, reliability was ‘good’ on both sides (ICC: 0.79–0.82). For the pressure pain thresholds, reliability was ‘excellent’ for the extensor carpi radialis brevis point (ICC: 0.90–0.93), and ‘good’ for the deltoideus point (ICC: 0.84–0.89). For the active range of motion, reliability was ‘excellent’ for both extension, flexion, and total range (ICC: 0.94–0.97). Similarly, reliability was ‘excellent’ for the maximal isometric strength on both sides (ICC: 0.97).
Verdict: Supported
Claim: In addition to plastic, various other types of anthropogenic materials found in 23 out of 1,447 pellets were identified and classified as non-plastic debris.
Evidence type: table
Caption: Table 2: Composition of plastic and non-plastic debris found in 1,447 analyzed pellets of the neotropic cormorant,N. brasilianus, on the Circuito de Playas Costa Verde (CPCV), Lima, Perú, during both the pre-pandemic and pandemic phases.
Verdict: Supported
Claim: The most encountered category was fibers, followed by metal, glass, wood, paint chips and balloons ( Table 2 ).
Evidence type: table
Caption: Table 2: Composition of plastic and non-plastic debris found in 1,447 analyzed pellets of the neotropic cormorant,N. brasilianus, on the Circuito de Playas Costa Verde (CPCV), Lima, Perú, during both the pre-pandemic and pandemic phases.
Context: In addition to plastic, various other types of anthropogenic materials found in 23 out of 1,447 pellets were identified and classified as non-plastic debris.
Verdict: Supported
Claim: Among the nine independent variables, five correlated with arm swing speed at BI ( Table 1 ).
Evidence type: table
Caption: Table 1: Peak angular momentum variables and correlations with the arm swing speed at ball impact (BI).
Verdict: Supported