RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: The only exception is the case of style strength evaluation task in the direction of H_{1} \rightarrow H_{2} , using the GPT-NeoX model.
Context: The inter-annotator agreement in all of the tasks are shown in Table 6 . Note that, for calculating agreement in the semantic correctness evaluation task, all of the data points are aggregated to measure the agreement score as they represent categorical evaluation measures. On the other hand, that is not possible in case of ranking based evaluations for style strength and appropriateness. So, we measure the agreement for each data point and take the average agreement over all data points. We can see in the Table 6 that in all of the cases we get strong agreement ( >0.70 ) among the annotators for the style strength and appropriateness evaluation.
Caption: Table 6: Inter-annotator agreement scores for the three human evaluation tasks. Standard deviations over all data points are shown in brackets for the style strength and appropriateness evaluation tasks. The detailed procedure for calculating the agreement scores can be found in Appendix D .
Claim: Additionally, 46.6% of participants were primary school graduates, 9.4% reported alcohol use, and 13% were smokers, with an average duration of smoking of 22.83 ± 6.94 years.
Context: Table 1 presents the personal data of the participants along with the total score of IIEF. The participants had a mean age of 63.26 ± 7.62 years, with 97.3% being married.
Caption: Table 1: Participants’ personal data, IIEF total score (n= 223).
Claim: In addition, significant differences were observed in all SU and SD kinematics except for KHD during SD.
Context: A total of 43 recreational table tennis players were included in this study, with 44 and 42 legs in the non-EOA and EOA groups, respectively ( Table 1 ). There were no significant differences between the groups in terms of sex distribution ( p = 0.22), age ( p = 0.46), height ( p = 0.07), weight ( p = 0.78), BMI ( p = 0.29), or KHD during SD ( p = 0.05).
Caption: Table 1: Participants characteristics.
Claim: Among the three kinds of prompts, CT-3 achieves the highest EX scores on both English data (EX: 13.34%) and Chinese data (EX: 12.86%).
Context: GPT-4+DIN-SQL obtain EX score of 6.73% on Archer test set, while it is able to achieve 85.3% test-suite execution performance on Spider test set Pourreza and Rafiei ( 2024 ) . To evaluate the overall difficulty of Archer, we test the zero-shot performance of GPT-3.5 with API Doc, CT-3, CT-3+COT prompts on the full Archer data.
Caption: Table 2: Baseline performance on Archer. GPT-4+DIN-SQL was tested only on the English set due to cost and its English-specific design. We only report the fine-tuned model’s performance on the test set.
Claim: In specific, we illustrate the specific number of iterations required before freezing each component in Fig. 5 . Interestingly, as can be seen from the figure, analysis reveals that the down-projection matrix parallel the intermediate linear layer require longer training duration prior to being frozen, as compared to the other PMs.
Context: Discussion on Freezing Trend. We use the RTE dataset as a case study, to understand the freezing trend of the PMs across different layers.
Caption: Figure 5: Visualization of freezing iterations for each layer. ‘out’ and ‘inter’ refer to the second and the first MLP layer of the FFN, respectively. ‘A’ and ‘B’ represent the down-projection and up-projection matrix, respectively. The darker the color, the more iterations the matrix has to go through before freezing.
Claim: Lastly, we calculated the ARGs scores for each patient in TCGA-PRAD cohort and divided the samples into high-risk and low-risk groups, and Kaplan-Meier analysis revealed that patients in high-risk groups suffered from worse survival outcomes relative to those from the low-risk groups with log-rank test P < 0.001 ( Figure 3E ).
Context: To identify the essential ARGs with prognostic significance in PRAD, we selected the top 53 DEGs and performed the LASSO analysis, where 10 hub genes were found ( Figs. 3A and 3B ). Accordingly, the significant differential expression of 10 ARGs in two clusters were shown in dot plot ( Figure 3C ). The ARGs signature was thus constructed via multi-variate Cox regression analysis, where 3-, 5-, and 7-year of the areas under the curve (AUCs) of the ROC curves were 0.789, 0.799 and 0.837, respectively ( Figure 3D ).
Caption: Figure 3: Identification and construction of ARGs in PCa samples.
(A and B) We conducted the LASSO algorithmto identify the 10 prognostic genes in TCGA training cohort, where the optimal cutoff value was −4 and the minimum account of genes was 18. (C) The bubble plot exhibited the significantly differential expressions of 10 prognostic genes. (D) The receiver operating characteristic curve (ROC) analysis showing the predictive efficiency of 3-, 5- and 7-year in predicting PFS of patients. (E) Kaplan-Meier analysis showing the differential survival outcomes between ARGs-high and ARGs-low samples.
Claim: Compared to ELoRA we can yield up to 1.86\times and 2.96\times runtime and FLOPs improvement while remain comparable with LoRA in these two metrices.
Context: Runtime & FLOPs Comparison. Fig. 3 shows the comparison of the normalized average training runtime, normalized FLOPs, and normalized trainable parameters. For AFLoRA, we average the training time, FLOPs, and trainable parameters over six GLUE datasets (except the MNLI and QQP datasets). Note, for LoRA and ELoRA, the trainable parameters and FLOPs remain fixed for all the dataset. We compute their average runtime same way as ours.
Caption: Figure 3: A comparison of various system performance between LoRA, ELoRA, and AFLoRA.
Claim: Among the nine independent variables, five correlated with arm swing speed at BI ( Table 1 ).
Caption: Table 1: Peak angular momentum variables and correlations with the arm swing speed at ball impact (BI).
Claim: Claude 3.5 sonnet achieves the highest performance of 79.21% on the answerable questions (original Spider dev set).
Context: Table 5 shows our baseline method’s (DIN-SQL) performance on SQL prediction of various LLMs given the interaction between the user and the assistant. Overall, Mixtral-large-v2 and Claude 3.5 Sonnet achieve the highest average accuracy of 71.95% and 72.15% on the ambiguous/unanswerable questions.
Caption: Table 5: Execution accuracy of SQLs predicted with DIN-SQL using different LLMs on various categories of ambiguous, unanswerable, and answerable questions. The "All" column shows the overall average accuracy across all categories, while the "Avg. Excluding Answerable" column shows the average accuracy excluding the answerable questions from the Spider dataset.
Claim: We also see that acquiring based on the lookahead posteriors incurs only a minimal increase in error compared to the sampling-based method, despite being far more efficient.
Context: Figure 7 presents the root-mean-square error, averaged across the test instances, when imputing \mathbf{x}_{u} with different numbers of acquired features. We see that our models are able to achieve lower error than EDDI.
Caption: Figure 7: Average RMSE of reconstructions for greedy active feature acquisition.
Claim: C2 included exclusively males, whereas the other clusters had a mixed sex composition.
Context: To validate the quality and distinctiveness of the Louvain clustering solution, we calculated widely-used cluster validity indices. The Davies–Bouldin index was 2.09, and the Calinski–Harabasz index was 59.26. While the Davies–Bouldin index was somewhat high (lower values indicate better separation), the Calinski–Harabasz index supported the presence of distinct groupings in our data. The Louvain clustering algorithm identified four distinct clusters (C1–C4) with significant differences in all features (all p < 0.001, except for AHD-SU ( p = 0.017) and portion of EOA ( p = 0.019)) ( Table 2 ). Tukey’s post-hoc analysis revealed that those in C2 had the youngest age (44.80 ± 11.30 years, p < 0.001 vs. all other clusters) and highest BMI (25.59 ± 2.42, p < 0.001 vs. all other clusters).
Caption: Table 2: Comparisons of features between the clusters classified by Louvain clustering unsupervised machine learning.
Claim: The color intensity (green shading) in Table 4 indicates better performance, visually highlighting that RACLETTE’s approaches generally outperform the benchmark models.
Context: The results show RACLETTE’s Combined method achieving the highest recall of 0.95, indicating superior ability to identify relevant cases, though this comes with a trade-off in precision at 0.63. Conversely, RoBERTa leads in precision at 0.72, but with lower recall at 0.84. The KL Divergence variant of RACLETTE stands out for its balanced performance, maintaining strong scores across all metrics (precision: 0.71, recall: 0.90, F1: 0.79, accuracy: 0.77). Both JS Divergence and Cosine Similarity methods show similar patterns, with high recall (0.93) but lower precision.
Caption: Table 4: Classification results for different methods of comparing embeddings to detect risk of suicide. The higher the score (a.k.a. the greener), the better.
Claim: Total protein foods consumed more only during the 6 months after intervention period ( P < 0.001).
Context: Figure 1 shows the comparison of component scores of HEI-2015 recorded during the pre-intervention, 6 weeks after intervention and 6 months after intervention. Accordingly, participants consumed significantly more total fruit ( P < 0.001) and whole fruit ( P < 0.001) after the intervention period. Although scores were higher at 6 months after intervention, the difference between the scores of 6 weeks and 6 months after intervention did not reach to a statistically significant difference. Similarly, scores of total vegetables significantly increased both in 6 weeks ( P < 0.001) and 6 months ( P < 0.001) after intervention period. The consumption of greens & beans only increased during the period of 6 weeks after intervention significantly ( P < 0.05) and then decreased at the end of the 6 months after intervention phase. The strongest impact of the intervention was observed on the scores of whole grains and dairy since the consumption of these groups continued to increase significantly both in 6 weeks and 6 months after intervention phase.
Caption: Figure 1: Comparison of component scores of HEI-2015 measured during pre-intervention, 6 weeks after intervention and 6 months after intervention.
In HEI-2015, a higher score indicates a higher diet quality. The Friedman test indicated significant differences in the component scores of HEI-2015 between pre-intervention, 6 weeks after intervention and 6 months after intervention values. *P < 0.05, **P < 0.01, ***P < 0.001.
Claim: A comparative analysis of hematology parameters indicated that children exposed to lead had significantly lower values of monocytes, basophil, hemoglobin, red blood cell count (RBC), hematocrit (HCT), MCH, MCHC, mean platelet volume (MPV), and platelet volume distribution width (PDW), and higher white blood cell count (WBC), neutrophils, eosinophil, red cell distribution width coefficient of variation (RDW-CV), RDW-SD, platelet, and plateletcrit (PCT) levels compared to healthy children ( Table 1 ).
Caption: Table 1: Hematologic parameters of the lead-poisoned children and the healthy controls.
Claim: The mean duration of hypertension was 7.92 ± 4.06 years, and the most commonly used medication among participants (27.8%) was ACE Inhibitors.
Context: Table 1 presents the personal data of the participants along with the total score of IIEF. The participants had a mean age of 63.26 ± 7.62 years, with 97.3% being married. Additionally, 46.6% of participants were primary school graduates, 9.4% reported alcohol use, and 13% were smokers, with an average duration of smoking of 22.83 ± 6.94 years.
Caption: Table 1: Participants’ personal data, IIEF total score (n= 223).
Claim: The in-domain results are in line with the empirical observation that Jiang et al. ( 2020 ) it is often ambiguous to differentiate between refutes and NEI claims even for trained human annotators.
Context: Table 5 shows the breakdown of the class-wise F1 score. For each dataset, we show the average class-wise F1 when training the model on other datasets (zero-shot) and the class-wise F1 for training on the same dataset (in-domain). The results show that the refutes claim has the worst prediction score (in bold) almost for all datasets, in both the zero-shot and the in-domain setting.
Caption: Table 5: Class-wise F1 of 3-class fact verification for the zero-shot generalization setup (left) and the in-domain training setup (right). S: supports; R: refutes; N: NEI.
Claim: Although a few of the source-only models have satisfactory performance on the target, using our ConDA framework, we achieve performance gains over the source-only model in almost all tasks (rows with positive \Delta F1 values).
Context: To evaluate the performance of ConDA on each of the target domains, i.e. generators, we first look at how our model improves over a source-only model. Table 2 shows the results for this experiment, grouped by target domain. We report F1 scores for the ConDA framework and a source-only model, along with scores averaged over sources, for each target. The source-only model is a pre-trained RoBERTa (roberta-base) fine-tuned only on the source domain S. The source-only scores provide an estimate of how well a model trained just on the source transfers to the target domain.
Caption: Table 2: Performance of ConDA on unlabeled target domains. Source-only model for each task S \rightarrow T refers to zero-shot evaluation of a model trained on S and evaluated on test set of T. \Delta F1 is increase (or decrease, in a few cases) in F1 scores of the ConDA model over the source-only model. Avg. scores in bold indicate where ConDA out-performs the source-only model.
Claim: There was no difference in the prevalence of osteoporosis and low bone mass rate between the two groups, indicating that there was no significant difference in BMD between the two groups ( Table 4 ).
Context: There was no significant difference in BMD between the two groups ( P > 0.05).
Caption: Table 4: Comparison of two groups of BMD and abnormal rates.
Claim: For instance, on the Cifar-10 benchmark it achieves diversity scores >6.9\% and up to 9.2\% (depending on ensemble size), against 6.8\% for a naive ensemble, see Fig. 1 .
Context: Inspired by that line of work, we propose a new, efficient ensemble method, FiLM-Ensemble. Our method adapts
feature-wise linear modulation as an alternative way to construct an ensemble for (epistemic) uncertainty estimation. FiLM-Ensemble greatly reduces the computational overhead compared to the naïve ensemble approach, while performing almost on par with it, sometimes even better. In a nutshell, FiLM-Ensemble can be described as an implicit model ensemble in which each individual member is defined via its own set of linear modulation parameters for the activations, whereas all other network parameters are shared among ensemble members – and therefore only need to be stored and trained once. Thanks to this design, our method requires only a very small number of additional parameters on top of the base network, e.g., converting a single ResNet-18 model to an ensemble of 16 models increases the parameter count by 1.3%, compared to an increase by 1500% when setting up a naïve ensemble, see Table 1 . We further show that FiLM-Ensemble results in more diverse ensemble members compared to other efficient ensemble methods.
Caption: Figure 1: Diversity analysis: CIFAR-10/VGG-11 experiment. With increasing number of members, FiLM-Ensemble achieves more diverse representations. See Section 3.1 .
Claim: Merging the encoder layers after language adaptation with the original model improves the model quality while using RetroMAE leads to decreased results.
Context: We perform contrastive fine-tuning for each model and then evaluate them on ruMTEB. Results (see Table 3 ) show that ru-en-RoBERTa outperforms both baselines by a significant margin. Additionally, the fact that XLM-R slightly outperforms ruRoBERTa may indicate that XLM-R copes better with knowledge transfer from basic English datasets, which provide diverse examples of high quality.
Caption: Table 3: Results of the model, method, and data variation. † The reference results for the training objective and training examples sections is model based on ru-en-RoBERTa. Each experiment changes a single component (e.g., use AnglE similarity instead of cosine). Model performance is evaluated on ruMTEB. The best score across all experiments is bold, the second best is underlined.
Claim: Also in this case, it can be observed that the choice of fine-tuning a generative model, leveraging its autoregressive characteristics for classification, leads to the best results.
Context: Table 2 shows how RACLETTE outperforms the benchmarks considered.
Caption: Table 2: Emotional accuracy comparison between RACLETTE and other benchmarks (best results highlighted in green).
Claim: As shown in Figure 3 , \mathcal{Q}_{\text{All}} hardly outperforms the RAG baseline ( 64.3\rightarrow 64.5 ).
Caption: Figure 3: Comparison of different warmup strategies. Scores are reported on the StrategyQA dev set.
Claim: Faecalibacterium was positively associated with pentanoic acid and 4-aminobutanoic acid, while were negatively correlated with hydracrylic acid.
Context: To further explore correlations of metabolites in human fecal matter and intestinal flora after SNP treatment, thermography was used to analyze contents of metabolites showing significant differences with 30 most abundant genera. Figure 10 shows the results. Bifidobacterium was positively associated with glycerol, CO 2 , 4-aminobutanoic acid, L-cysteine, and D-gluconic acid, and could be negatively correlated with DL-phenylalanine.
Caption: Figure 10: Correlation analysis of characteristic metabolites and bacteria.
Corr represents correlation. Red and blue separately denote positive and negative correlations. The darker the color, the stronger the correlation. *P < 0.05, **P < 0.01, and ***P < 0.001.
Claim: Although this setting works reasonably well for XLM-R, it yields poor performance for mT5.
Context: In Table 5 we present the average performance across the four language groups to compare the effectiveness of lexicon-based pretraining in regression and classification tasks for both binary and 3-way classification. Our findings indicate that regression performs better for binary classification, while classification leads to better results for 3-way classification. However, regression in 3-way classification presents a challenge when determining the neutral class boundary during inference. In the zero-shot setting, we lack specific data for hyper-parameter tuning, leading us to arbitrarily set the neutral class boundaries to - 1 and + 1.
Caption: Table 5: Regression vs. classification in lexicon-based pretraining for zero-shot sentiment analysis.
Claim: Despite its superior performance, GPT-4’s performance on plan ability (59.4) is still barely satisfactory, far behind that of humans (95.6), which is consistent with the conclusion of Valmeekam et al. ( 2022 ) .
Context: As shown in Table 3 , We run the model with the largest number of parameters in each series on CogLM , and report the adult human performance for comparison. Overall, the cognitive abilities of OPT, Llama-2-chat-70B, GPT-3.5-Turbo, and GPT-4 models successively increase, and the performance of each model gradually declines with the increase of stage, consistent with humans. Specifically, the latest state-of-the-art model, GPT-4, has demonstrated remarkable cognitive capabilities, achieving a level comparable to that of a 20-year-old human. It is also worth noting that both GPT-3.5-Turbo and GPT-4 surpass humans in empathy ability at stage 2, which is natural, as humans tend to have some degree of selfishness.
Caption: Table 3: Calibrated accuracy (%) of largest model in evaluating series. Acc and Age refer to calibrated accuracy and the age of equivalent human performance. The value of Age is calculated according to Equation 2 . Bold indicates the best performance.
Claim: Ant-dir : We trained models for performing the task on an episode of 200 timesteps and found that MetODS can adapt in a few time-steps, similar to memory-based models such as RL^{2} , (Figure 3 -f) thanks to its continual adaptation mechanism.
Context: \diamond MuJoCo
Caption: Figure 3 : a-b) Schemas of the Harlow and Mujoco Ant-directional locomotion task. c-d) Evolution of accumulated reward over training. In the Harlow task, we conduct an ablation study by either reducing the number of recursive iterations (S=1) or removing the trainable plasticity weights \bm{\alpha} resulting in sub-optimal policy. In Ant-dir we compare our agent training profile against MAML and RL 2 . e) We can interpret the learned policy in terms of a Hopfield energy adapting with experience. We show horizontally two reward profiles of different episodes and the energy E_{\bm{W}_{t}}(v_{1},v_{2})=-v^{T}_{1}\bm{W}_{t}v_{2} along two principal components of the vector trajectory \bm{v_{t}}. In the first episode, the error in the first presentation (red square) transforms the energy landscape which changes the agent policy, while on the other episode, the model belief does not change over time. Note the two modes for every energy map, which allows the model to handle the potential position permutation of the presented values. f) Average rewards per timestep during a single episode of the Ant-dir task.
Claim: However, no significant differences were found in cold frequency between participants of different sexes or age groups ( P > 0.05).
Context: As shown in Table 2 , there was a statistically significant difference in the frequency of colds experienced by middle-aged and elderly individuals with varying levels of weekly PA ( P < 0.001).
Caption: Table 2: Cold in middle-aged and elderly with different characteristics.
Claim: Table 2 shows that comparative assessment yields highly impressive performance for long-spoken summarization, with comparative assessment out-competing all other baselines.
Context: Podcast Assessment : When considering podcast summarization with long inputs of over 5k tokens on average, only Llama2 models (which have a limit of 4k tokens) were used (as FlanT5 has a limit of 1k tokens).
Caption: Table 2: Spearman correlation coefficient for Podcast .
Claim: Specifically, the latest state-of-the-art model, GPT-4, has demonstrated remarkable cognitive capabilities, achieving a level comparable to that of a 20-year-old human.
Context: As shown in Table 3 , We run the model with the largest number of parameters in each series on CogLM , and report the adult human performance for comparison. Overall, the cognitive abilities of OPT, Llama-2-chat-70B, GPT-3.5-Turbo, and GPT-4 models successively increase, and the performance of each model gradually declines with the increase of stage, consistent with humans.
Caption: Table 3: Calibrated accuracy (%) of largest model in evaluating series. Acc and Age refer to calibrated accuracy and the age of equivalent human performance. The value of Age is calculated according to Equation 2 . Bold indicates the best performance.