RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Here, while finetuning achieves higher accuracy for gendered words, it also degrades translation quality by 0.9 COMET.
Context: As motivated in § 4.2 ,
the gender control results in Table 3 allow us to assess the impact of domain mismatch between the controller training data and the test data, a very realistic scenario in practice.
Caption: Table 3: Grammatical gender control results in supervised condition (cross-domain: controller trained on gender-annotated data from a different domain).
Claim: Own Responsibility, Own Control, Not Consider, and Effort also generally performed well, although Own Responsibility is predicted remarkably worse in the GPT-4 {}_{\text{emo}} setting.
Context: In analyzing appraisal dimensions across all experiments, we looked for patterns by comparing the RMSE of individual appraisal dimensions to the macro-averaged RMSE. Appraisals consistently showing better accuracy across both GPT-4 models and human annotators include Pleasantness, Unpleasantness, Goal Support, and External Norms (marked as bold in Table 2 ), suggesting that these are the easiest to infer based on text.
Caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Claim: Alpha diversity indices of ACE and Chao1 showed no significant difference between NEMM and SHI under the normoxia condition.
Context: To compare NEMM with SHI, which are suitable for cultivating specific bacterial types, we further analyzed 16S rRNA gene high-throughput sequencing data.
Caption: Figure 5: General bacterial community structure analysis between NEMM and SHI media during aerobic (normoxia) and anaerobic culture.
(A) Alpha diversity was measured with ACE index and Chao1 index. P < 0.05 was considered statistically significant. (B) NMDS was measured with Unweighted UniFrac. Each sample is represented by a dot. (C) Venn diagram displays the number of common and unique characteristics among the groups.
Claim: At the same time, the RMSE obtained using the majority voting is also considerably lower than the results shown in the previous subsection from just one run (1.12 vs 1.45; columns GPT-4 {}_{\text{maj}} and GPT-4 in Table 2 ).
Context: The application of the majority voting algorithm improved the RMSE on average by about 30% from 1.61 to 1.12 (columns GPT-4 {}_{\text{avg}} and GPT-4 {}_{\text{maj}} in Table 2 ). However, compared to the results shown in previous Section 4.2 (also shown in Table 2 ), generating five runs and taking the average increased the RMSE considerably, which is now in the same range to the average over the human reader-annotator ratings (1.61 vs 1.57; columns GPT-4 {}_{\text{avg}} and Human {}_{\text{avg}} in Table 2 ).
Caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Claim: For the active range of motion, reliability was ‘excellent’ for both extension, flexion, and total range (ICC: 0.94–0.97).
Context: Table 3 presents ICC 3,1 , SEM, MDC 90 , and MDC 95 for each variable. For the muscle mechanical properties of the dominant and non-dominant sides, reliability was ‘excellent’ for the frequency and stiffness parameters (ICC: 0.91–0.96), and ‘good’ to ‘excellent’ for the decrement (ICC: 0.86–0.91). For the manual dexterity, reliability was ‘good’ on both sides (ICC: 0.79–0.82). For the pressure pain thresholds, reliability was ‘excellent’ for the extensor carpi radialis brevis point (ICC: 0.90–0.93), and ‘good’ for the deltoideus point (ICC: 0.84–0.89).
Caption: Table 3: Reliability indicators of myotonometry, manual dexterity, pressure pain thresholds, active range of motion, and maximal isometric strength for the wrist extensor muscles.
Claim: Among young women, the HRT group had higher clinical pregnancy and live birth rates and lower biochemical pregnancy and miscarriage rates than the GnRHa-HRT group, although none of the differences were statistically significant.
Context: Table 2 shows the clinical outcomes of FET. Women aged <35 years had higher clinical pregnancy and live birth rates than women aged ≥35 years.
Caption: Table 2: Comparison of reproductive outcomes.
Claim: In contrast, ChatGPT and Gemma show much lower bias, with values of 7.08\%^{2} and 1.32\%^{2} , respectively.
Context: Fig. 4 displays the key findings of our evaluation across models and datasets with numerical results in Appx.- Tab. 10 . From Fig. 4 -left, we notice that Mistral exhibits the most bias, with the BiasF_{o} value ( Eq. 8 ) of 353.80\%^{2} .
Caption: (a); (b); Average EstTrueF1 (SemEval2017) and EstTrueMAP (SciDocsRR) ( Section 3.1 ) across models (left) and benchmarks (right) showing performance difference of LLMs across 4 widely used list formats.
Claim: Humans obtained an EM of 69.13 in MisinfoQA-noisy , which, though higher than most QA models’ performance, also shows a significant drop when compared to the MisinfoQA-clean setting (86.57 EM).
Context: Table 4 reports the EM and F1 for both human and different QA models. We find that all QA models suffer a large performance drop ( \sim 20% in EM) in MisinfoQA-noisy compared to MisinfoQA-clean , showing that the models are largely distracted by the fake contexts rather than by the presence of additional contexts.
Caption: Table 4: QA performance under the reading comprehension settings with clean and noisy contexts.
Claim: As reported in Table 2 , in Q Drawer the value of the entropy threshold \theta has, by design, a direct impact on the number of questions asked: i.e., a model that only asks for clarification in the face of high uncertainty ( \theta=1.1 ) gives rise to fewer dialogues with questions and to a lower average number of questions per dialogue.
Context: Asking clarification questions carries a cost, both from the perspective of the agent asking the question and the agent processing it (Clark, 1996 ; Purver, 2004 ) . We take this partially into account by adopting a simple approach: we evaluate Q Drawer by using different entropy threshold values \theta to control for the number of questions generated by the model.
Caption: Table 2: The effect of different \theta values on the number of questions asked by Q Drawer. We report the percentage of dialogues with at least one clarification question and, within this subset, the average number of clarification questions per dialogue.
Claim: As shown in Figure 4 , LISA provides almost 2.9\times speedup when compared with full-parameter tuning, and \sim 1.5\times speedup against LoRA, partially due to the removal of adaptor structures.
Context: On top of that, a reduction in memory footprint from LISA also leads to an acceleration in speed.
Caption: Figure 4: Single-iteration time cost of LLaMA-2-7B with different methods and batch size 1.
Claim: As seen in Table 1 , five of the seven relations are parsed in accordance with each’s desired relation more than 82\% of the time, four greater than or equal to 95\% of the time, and one is parsed to the desired relation for all tested prompts.
Context: The input text alongside its completion is automatically parsed using the DMRST parser.
Caption: Table 1: The automatic-evaluation statistics for each relation, where None is generation with the language model alone.
Claim: However, percentage of population that were married had statistically significant low diabetes-related ED visit rate ( Table 5 ).
Context: Based on the final multivariable negative binomial model, the following independent variables had significant high county-level non-gestational diabetes-related ED visit rates: percentages of population who were non-Hispanic Black, current smokers, had diabetes, and had no insurance coverage ( Table 5 ).
Caption: Table 5: Results of a multivariable negative binomial regression model examining the associations between county characteristics and the rate of diabetes-related emergency department visits in Florida, 2019.
Claim: An intriguing observation is that the decodability of both attributes decreases as the depth of the neural network increases, which is also observed in [ 10 ] .
Context: We employ the concept of feature decodability to assess the extent to which the specific features of a given dataset can be reliably decoded from the models with varying depths. Hermann et al . [ 10 ] demonstrated that the visual features can be decoded from the higher layers of untrained models. Additionally, they observed that the feature decodability from an untrained model has a significant impact in determining which features are emphasized and suppressed during the model training. Following their approach, we specifically focus on assessing the decodability of bias and core attributes from the penultimate layer of untrained models. In order to evaluate the decodability of an attribute in a dataset, we train a decoder to map the activations from the penultimate layer of a frozen, untrained model to attribute labels. The decoder comprises a single linear layer followed by a softmax activation function. The decoder is trained using an unbiased validation set associated with the dataset, where each instance is labeled according to the attribute under consideration. Subsequently, the linear decodability of the attribute, measured in accuracy, is reported on the unbiased test set. We investigate the decodability of digit and color attributes in the CMNIST dataset from MLP models with varying depths, including 3, 4, and 5 layers, and the results are depicted in Fig. 2(a) .
Caption: (a); (b); Exploring the Effect of depth modulation: (a) illustrates how the linear decodability of features decreases as neural network depth increases, while (b) dives into the training dynamics of MLPs with varying depths under ERM.
Claim: There was a significant difference in IQ ( F (2, 51) = 9.061, p < 0.01, η 2 = 0.262), and a post hoc comparison revealed that ETT players had a lower IQ than ATT players ( p < 0.003) and nonathletes ( p < 0.001), but no significant difference between the ATT and nonathletes ( p < 0.368) was observed.
Context: Table 1 presents the participants’ characteristics. No significant differences were observed in age ( F (2, 51) = 2.863, p = 0.066), height ( F (2, 51) = 0.420, p = 0.659), weight ( F (2, 51) = 0.556, p = 0.577), handedness scores ( F (2, 51) = 0.536, p = 0.589), video game experience ( F (2, 51) = 1.387, p = 0.259), and socioeconomic status of the family ( F (2, 51) = 1.579, p = 0.216) among groups.
Caption: Table 1: Demographics of the participants in each group (mean ± SD).
Claim: However, in our exploratory experiment on Mixtral 8x7B [ 17 ] , we find simply raising the number of activated experts (blue lines in Figure 1 ) does not lead to stable improvements and may even hurt performance on different tasks.
Context: In this paper, we investigate the impact of unchosen experts 1 1 1 Unchosen experts refer to the experts not selected by default routing ( e.g. , top-2 routing in Mixtral 8x7B). on the performance of MoE models and explore their suitable usage. A direct hypothesis is that incorporating more experts improves MoE models and helps solve more difficult problems [ 3 ; 36 ; 14 ] .
Caption: Figure 1 : Performance comparison between increasing the value of top- k ( i.e. , ensemble routing) and SCMoE. SCMoE surpasses the performance of ensemble routing across various benchmarks.
Claim: With the increase of data size, the performance increases.
Context: We show the performance of models with different settings in the validation set in Figure 6 . With the same data size, the model using DAgger performs better.
Caption: Figure 6: The Cumulative Reward of Anonymous in Validation Dataset. The performance of DAgger models is related to the number of iteration rounds.
Claim: In contrast,
for the pretrained NLLB,
there is no clear distinction between the multilingual systems and rest.
Context: In Table 4 , controllers trained on multiple translation directions ( multi ) are compared to those trained on single directions ( en \rightarrow es or de ). On Transformer-base, multi consistently outperforms its single-direction counterparts, regardless whether the controller is finetuning- or CG-based.
Caption: Table 4: Zero-shot formality control results. Best and second best results under the same data condition are marked.
Claim: The bottom left of Table 2 shows that the model trained on natural claims generalizes badly to datasets with artificial claims, with an average F1 drop of 72% relative to the in-domain performance on the three artificial datasets.
Caption: Table 2: Macro-F1 of 3-class fact verification on the evaluation set for all datasets in a zero-shot generalization setup. Rows correspond to the training dataset and columns to the evaluated dataset. The row SELF corresponds to the in-domain performance (training and testing on the same target dataset).
Claim: Moreover, PECRS without training for generation can achieve 11.907 on Dist@2 metric (see w/o Generation loss in Table 5 ), but merely 7.76 on RG-1 metric.
Context: We first study the effects of different LM’s decoding strategies on conversational performance over Dist@K metric. Specifically, we analyze the greedy decoding, beam search, diverse beam search Vijayakumar et al. ( 2018 ) , top-k sampling Fan et al. ( 2018 ) and nucleus sampling Holtzman et al. ( 2020 ) strategies on PECRS-small. Reported in Table 8 , reference-based metrics (RG-K) show much less variance on different decoding strategies compared to the reference-free metrics (Dist@K). Meanwhile, the correlation between reference-based and reference-free metrics is weak under different decoding strategies.
Caption: Table 5: Models comparison with different modules and optimization strategies on ReDial with PECRS-small.
Claim: Also, note that with the increasing number of members, improvements in diversity metrics for the naïve explicit deep ensemble are negligible.
Context: For both metrics, higher numbers correspond to higher diversity. See Fig. 1 .
We observe that the average diversity increases with the number of ensemble members. In both metrics, FiLM-Ensemble achieves higher scores than the naïve, explicit deep ensemble. Meaning that the predictions of FiLM-Ensemble are less correlated than those of an equivalent number of networks trained independently with different random seeds.
Caption: Figure 1: Diversity analysis: CIFAR-10/VGG-11 experiment. With increasing number of members, FiLM-Ensemble achieves more diverse representations. See Section 3.1 .
Claim: Lactobacillus was positively correlated with xylitol and showed a negative correlation with 5-aminovaleric acid.
Context: To further explore correlations of metabolites in human fecal matter and intestinal flora after SNP treatment, thermography was used to analyze contents of metabolites showing significant differences with 30 most abundant genera. Figure 10 shows the results. Bifidobacterium was positively associated with glycerol, CO 2 , 4-aminobutanoic acid, L-cysteine, and D-gluconic acid, and could be negatively correlated with DL-phenylalanine. Faecalibacterium was positively associated with pentanoic acid and 4-aminobutanoic acid, while were negatively correlated with hydracrylic acid.
Caption: Figure 10: Correlation analysis of characteristic metabolites and bacteria.
Corr represents correlation. Red and blue separately denote positive and negative correlations. The darker the color, the stronger the correlation. *P < 0.05, **P < 0.01, and ***P < 0.001.