RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Secondly, AA was positively correlated with IS (Index = 0.575, P < 0.001) and negatively correlated with SS (Index = −0.234, P < 0.001).
Context: A multiple mediation analysis was conducted to explore the mediation effects of IS and SS in a college student population. Control variables included gender, age, BMI, GPA, being an only child or not, and home address. AA and depression were entered as independent and dependent variables respectively. The proposed mediators were IS and SS. Results of the analysis ( Table 3 ) showed that AA was positively correlated with depression (Index = 0.168, P < 0.001).
Caption: Table 3: Regression analysis of appearance anxiety, interpersonal sensitivity, social support, and depression.
Claim: We observe that all languages that are included in the pre-training corpus of XLM-R, the cross-lingual transfer performs similarly to fully supervised methods.
Context: Here, we compare the performance between cross-lingual transfer and fully-supervised methods.
Caption: Figure 5: Comparison of Various Scenarios. We group languages by whether they and their scripts are seen in the pre-training corpus of XLM-R. Languages are ordered by the XLM-R fully-supervised performance in every group.
Claim: We also show that our performance boost does not come from the additional training dataset of ActivityNet in Table 3 .
Context: We then compare our method with state-of-the-art on these datasets. We train a higher resolution model with larger spatial inputs (312p) and achieves better performance compared to recent multi-dataset training methods, CoVER [ 59 ] and PolyVit [ 38 ] , on Kinetics-400, and significantly better on MiT and SSv2, as shown in Table 1 . Note that our model does not use any image training datasets, and our model computation cost is only a fraction of the baselines.
Caption: Table 3: Ablation experiments. We investigate the effectiveness of each component of our method as well as compare to vanilla multi-dataset training method.
“Vanilla” means using cross entropy (CE) loss in training.
“w/o informative los” means using CE and projection loss.
The numbers are top-1/top-5 accuracy, respectively.
Training data:
(e) Kinetics-400;
(f) SSv2;
(g) MiT;
(h) ActivityNet.
Claim: The book of Genesis contains the highest number of half-verse chiasmi, while Numbers contains the most verse-level chiasmi.
Context: Table 2 presents an overview of the system’s output for chiastic structures at the half-verse and verse levels. A total of 1,896 chiastic structures were identified at the half-verse level, with an average length of 5.93 textual units ( \pm 1.34) and an average score of 0.32 ( \pm 0.1). For verse-level groupings, 879 chiastic structures were found, with an average length of 6.01 lines ( \pm 1.38) and an average score of 0.29 ( \pm 0.08).
Caption: Table 2: Summary of detected chiasmi. 2700+ chiasmi were detected at the verse and half-verse level. The highest number of chiasmi was found in the Book of Genesis and Book of Numbers. Both the precision and the inter-annotator agreement increase for the verse-level chiasmi.
Claim: This is illustrated in Figure 1 where we show that a deep ResNet can easily break the topology of the input space, which is impossible for a Neural ODE.
Context: Neural ODEs also provide a theoretical framework to study deep learning models from the continuous viewpoint, using the arsenal of ODE theory (Teh et al., 2019 ; Li et al., 2019 ; Teshima et al., 2020 ) . Importantly, they can also be seen as the continuous analog of ResNets. Indeed, consider for N an integer, the Euler scheme for solving Eq. ( 2 ) with time step \frac{1}{N} starting from x_{0} and iterating x_{n+1}=x_{n}+\frac{1}{N}\varphi_{\Theta}(x_{n},\frac{n}{N}) . Under mild assumptions on \varphi_{\Theta} , this scheme is known to converge to the true solution of Eq. ( 2 ) as N goes to +\infty . Also, if \Theta=(\theta_{n}^{N})_{i\in[N-1]} and \varphi_{\Theta}(.,\frac{n}{N})=f(.,\theta_{n}^{N}) , then the ResNet equation Eq. ( 1 ) corresponds to a Euler discretization with time step \frac{1}{N} of Eq. ( 2 ). However, for a given ResNet with fixed depth N and weights, the activations in Eq. ( 1 ) can be far from the solution of Eq. ( 2 ).
Caption: Figure 1: Trajectory of ResNets with 300 layers. Left: we learn x\to\frac{x^{2}}{2} , trajectories are smooth and do not intersect. Right: we learn x\to\frac{-x^{2}}{2} , trajectories are not smooth and intersect.
Claim: The results showed that the microbiota developed in NEMM were predicted stronger anaerobic ability, whether under anaerobic conditions or normoxic conditions.
Context: Bugbase was used to predict the function of bacterial community in NEMM and SHI under anaerobic and normoxia culture conditions.
Caption: Figure 7: BugBase analysis among different groups.
Note: X-axis, group name; Y-axis, Relative abundance in percentage. The three lines from bottom to top are lines indicating lower quartile, average and upper quartile.
Claim: From Table 2 , we find that fact verification in a dataset with document-level evidence is more difficult than in the same dataset with sentence-level evidence (an average of 13.29% drop of in-domain F1).
Caption: Table 2: Macro-F1 of 3-class fact verification on the evaluation set for all datasets in a zero-shot generalization setup. Rows correspond to the training dataset and columns to the evaluated dataset. The row SELF corresponds to the in-domain performance (training and testing on the same target dataset).
Claim: The mean RMSE for GPT-4 from 1.12 to 1.15 (columns GPT-4 {}_{\text{maj}} and GPT-4 {}_{\text{conf}} in Table 2 ), and the mean RMSE for human reader-annotators increased from 0.99 to 1.06 (columns Human {}_{\text{maj}} and Human {}_{\text{conf}} in Table 2 ).
Context: The results of this experiment are shown in Table 2 columns GPT-4 {}_{\text{conf}} and Human {}_{\text{conf}} . We can see that using the confidence rating to break the ties did not improve the RSME neither for the model nor for the human reader-annotators compared to the random choice.
Caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Claim: We see that LUI is the only method capable of converging to optimal policies both before and after the change in environment.
Context: Finally, we investigate the behaviour of the learning methods when after a fixed number of trajectories the probabilities of the true MDP change, as introduced in Section 5.4 . Figure 5 shows the performance of the robust policy for each learning method on the Chain environment, results for the same experiment on the Betting Game can be found in Appendix F .
After {\dagger}\in\{10^{2},\dots,10^{5}\} trajectories, we change the environment by swapping the transition probabilities, for three different bounds on the strength intervals. We similarly bound the MAP-estimation priors for MAP and UCRL. PAC is omitted from this experiment as PAC guarantees lose all meaning when changing the underlying distribution. After the change in environment, the new optimal policy has to use the opposite action from the previously optimal policy.
Caption: Figure 5 : Environment change on the Chain Problem at different points {\dagger}\in\{10^{2},\dots,10^{5}\} using randomization parameter \xi=0.8 in the exploration.
Claim: Our findings show that: (1) As expected, using the most closely related language as the auxiliary example bank yields the best performance, as it provides relevant guidance to the LLM, and (2) Selecting a random or unrelated language results in little to no improvement, with performance remaining close to the relevance-based EPR baseline Rubin et al. ( 2021 ) that only uses target-language specific data.
Context: We evaluate three setups: (1) Our method (Alg. 2 ) of selecting the most closely related language as the auxiliary example bank, (2) selecting a random language, and (3) selecting the most unrelated language. We show the few-shot performance on cross-lingual QA task in Figure 1 , with retrievers trained under each setup.
Caption: Figure 1: Token-F1 evaluation on the cross-lingual QA task in Manipuri, with retrievers trained using different auxiliary high-resource example banks: (1) Closely related language, (2) Random language, and (3) Unrelated language.
Claim: While the reported accuracy on the original test dataset is 38% for QAGNN and 40% for BioLinkBert, the accuracy on our 100 randomly selected demographic-independent questions use to construct the vignettes is 40% for QAGNN and 39% for BioLinkBert.
Caption: Table 4: Accuracy (in percentages) of the two models on our demographically enhanced datasets. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation; O*=original test dataset; O=the original, unmodified 100 vignettes; D=No demographic information; Gen=Gender; 1=QAGNN; 2=BioLinkBERT.
Claim: Results of the analysis ( Table 3 ) showed that AA was positively correlated with depression (Index = 0.168, P < 0.001).
Context: A multiple mediation analysis was conducted to explore the mediation effects of IS and SS in a college student population. Control variables included gender, age, BMI, GPA, being an only child or not, and home address. AA and depression were entered as independent and dependent variables respectively. The proposed mediators were IS and SS.
Caption: Table 3: Regression analysis of appearance anxiety, interpersonal sensitivity, social support, and depression.
Claim: The experimental results, as shown in Table 2 , reveal that enhancing the strong activation of SCMoE can further boost MoE models’ performance.
Context: As revealed by Figure 1 , using default top-2 routing is not optimal for all tasks. For instance, top-3 routing yields best results on GSM8K, while top-4 routing achieves the highest accuracy on HumanEval and StrategyQA. This leads us to consider whether enhancing the strong activation in SCMoE can further unlock MoE models’ potential. To this end, we adjust the strong activation of Mixtral 8x7B to top-3 for GSM8K, and to top-4 for StrategyQA, MBPP, and HumanEval, while keeping the weak activation with rank-2 routing as before.
Caption: Table 2: Experimental results of different strong activations.
We set the weak activation with rank- 2 routing.
For each benchmark,
we select the top- k routing yielding the best performance in Figure 1 as the ideal strong activation.
The specific hyperparameter settings can be found in Table 9 .
Claim: Nevertheless, the main resistance was to piperacillin/tazobactam (100%).
Context: Clinical isolates presented the higher resistance (100%) to the three antibiotics: imipenem, meropenem, and piperacillin/tazobactam. Forty-eight percent were susceptible to tobramycin ( Figure 3A ). The river isolates displayed an antimicrobial profile that varied between resistant and intermediate to eight antibiotics tested.
Caption: Figure 3: Clinical and environmental isolates susceptibility profile.
The percentages of clinical isolates that were resistant, intermediate, or sensitive to each antibiotic. (A) Clinical isolates susceptibility profile, (B) river isolates susceptibility profile. Acronyms: CIP, ciprofloxacin; GEN, gentamycin; SXT, trimethoprim/sulfamethoxazole; AMK, amikacin; FEP, cefepime; MEN, meropenem; IMP, imipenem; TOB, tobramycin; SAM, ampicillin/sulbactam; CAZ, ceftazidime; TZP, piperacillin/tazobactam; R, resistant; I, intermediate; S, sensitive.
Claim: However, for NusaX and the African languages, a considerable number of them are not covered.
Context: In Table 1 , we provide an overview of the languages and datasets used in this paper, categorized into: (1) high/medium-resource languages; (2) NusaX, covering local Indonesian languages (low resource); (3) African languages from SemEval 2023 (low resource); and (4) code-switching texts. The high/medium-resource languages and individual languages present in the code-switching texts are covered by all pretrained models and the NRC-VAD multilingual lexicon.
Caption: Table 1: Languages used in this paper. “✓” (green) and “✗” (red) mean that the language has and has not been seen by the models or language resources. CW indicates code-switching text. The language coverage for GPT-3.5 is derived from GPT-3 Brown et al. ( 2020 ) .
Claim: From Table 8 it is clear that BioLinkBert significantly outperforms its generic LM variation.
Context: Similar to our analysis between QAGNN and BioLinkBert above, our analysis between the biomedical and generic models can be split into the amount of answers and accuracy that changes when the dimensions change. From Table 7 it is visible that the generic transformer has more than double the amount of answers change for each gender. It also has an equivalent amount or more for almost any ethnicity, except for Asians. Notably, for sexual orientation, the generic transformer has almost double the amount of answers change for bisexuals, while the biomedical transformer has more for homosexuals. The generic transformer has significantly larger values than the biomedical transformer in any gender+ethnicity combination, while for gender+sexual orientation, the biomedical system has significantly larger values for homosexuals.
Caption: Table 8: Accuracy (in percentages) of the biomedical and generic models on our demographically enhanced datasets. M =male; F =female; W =White; B =Black; A-A =African-American; H =Hispanic; As =Asian; SOr =sexual orientation; O*=original test dataset; O=the original, unmodified 100 questions; D=No demographic information; 1=Generic; 2=Biomedical.
Claim: However, by some simple maths, the readers can easily verify that even considering this, the accuracy of answers containing calculations is still much lower than that of answers that do not include calculations.
Context: Next, for accuracy, even if LLMs are wrong for all the invalid questions, their accuracy should be around 85% if they get all the valid questions correct. However, this is clearly not the case for all LLMs except Claude-2.
Next, for the rightmost column in Table 1 , if we assume that all the invalid samples happen to be the samples that LLMs include calculations in the answer, the accuracy in this column should increase.
Caption: Table 1: The redundancy ( Red. ) and accuracy of LLMs’ responses.
We report the average accuracy ( Avg. ) on all questions (second column), the accuracy for answers without calculation (Cal. ✗, third column) and with calculation (Cal. ✓, fourth column). \dagger : The accuracy of GPT-4 is 100% by construction since we use GPT-4 to filter samples when constructing GSM8K-Zero.
Claim: We see that FuncMol slightly improves VoxMol and both models perform worse compared to the equivariant point-cloud based baselines.
Context: Table 1 report the metrics described in Section ˜ 5.1 .
Caption: Table 1: QM9 results w.r.t. test set for 10000 samples per model. \uparrow / \downarrow indicate that higher/lower numbers are better. The row data are randomly sampled molecules from the validation set. We report 1-sigma error bars over 3 sampling runs.
Claim: The high bias in the SciDocsRR task is because Mistral and Gemma mostly failed to perform this task following the “Bullet” and “Special character” list formats while excelling in solving it following the other formats.
Context: The performance bias is regardless of the task as plotted in Fig. 4 -right, with the highest BiasF_{o} value of 67.07\%^{2} on the order list generation task SciDocsRR, and significantly lower ( 27.58\%^{2} ) on SemEval2017 task.
Caption: (a); (b); Average EstTrueF1 (SemEval2017) and EstTrueMAP (SciDocsRR) ( Section 3.1 ) across models (left) and benchmarks (right) showing performance difference of LLMs across 4 widely used list formats.
Claim: As shown in Figure 4 , IAG- Student achieves the best performance with the statement number between 5 and 7.
Context: Our implementation of IAG samples 5 knowledge statements to feed into the generator. To justify this design choice, we evaluate the performance of IAG- Student with varying statement numbers.
Caption: Figure 4: Scores of IAG- Student on StrategyQA dev set with different numbers of knowledge statements.
Claim: 10 . From Fig. 4 -left, we notice that Mistral exhibits the most bias, with the BiasF_{o} value ( Eq. 8 ) of 353.80\%^{2} .
Context: Fig. 4 displays the key findings of our evaluation across models and datasets with numerical results in Appx.- Tab.
Caption: (a); (b); Average EstTrueF1 (SemEval2017) and EstTrueMAP (SciDocsRR) ( Section 3.1 ) across models (left) and benchmarks (right) showing performance difference of LLMs across 4 widely used list formats.
Claim: Significant negative correlations were observed between age and the IIEF total score, as well as between the duration of hypertension and the IIEF total score ( p < 0.05).
Context: Table 3 displays the correlation between age, duration of hypertension, duration of smoking, and the IIEF total score.
Caption: Table 3: Correlation between age, duration of hypertension, duration of smoking and IIEF total.
Claim: From the results, we notice that the correlations showed by the non-specialized embeddings are extremely weak.
Context: In Table 2 we report correlation scores for the gender diversity. Again, we split in two groups: male-voice and female voice-dominant.
Caption: Table 2 : Gender diversity. Average Spearman correlations between proportion of female voices and diversity scores induced by speech representations.
Claim: The addition of an interfragmentary lag screw to the medially positioned plate increased the initial flexural stiffness by 153% and the shear stiffness by 352%, see Table 2 .
Caption: Table 2: Comparison of mean values of flexural stiffness, initial flexural stiffness, shear stiffness and maximum load, and their standard deviations (SDs) for different fixation methods.
Claim: However, we observe considerably high bias, with some set-ups even selecting the first option 80% of the time.
Context: We investigate whether the comparative prompts have any implicit positional bias, and whether systems prefer the first/second position. Table 5 shows the fraction of comparisons that selected the candidate in the first position for SummEval. Since all comparisons in both permutations are considered, this fraction should be 0.50 for an unbiased system.
Caption: Table 5: Positional bias P(A) for both prompt templates, for various systems in the comparative setup on SummEval.
Claim: We plot the proportion of conserved correct predictions for the panoramic camera rotation in Fig. 7 . First, we observe that both networks struggle to a higher extent with images taken on opposite to the position, with the light source behind the camera.
Caption: Figure 7: Counterfactual study of all sizes of ConvNext and Swin networks for panoramic 360 ∘ camera rotation.
Claim: Statistics are presented in Table 3 . Each transcript is distilled into a table of about 40 facts, from which the model selects 9.
Context: Supporting Facts. We analyzed the supporting facts identified by the model in cases of correct decisions after reflections.
Caption: Table 3: Statistics of supporting facts.
Claim: Table 1 displays the number of documents for each target identity in both datasets, and Appendix A shows examples from the two datasets for each functionality.
Context: We use the HateCheck (Röttger et al., 2021 ) and GPT-HateCheck (Jin et al., 2024 ) datasets to conduct our analyses, as these datasets provide additional diagnostic insights. Both datasets cover the same seven target identities and 24 functionalities ( GPT-HateCheck omitted the five functionalities related to spelling variations in HateCheck ).
Caption: Table 1: Number of examples for each target identity in HateCheck and GPT-HateCheck . We omit the functionalities without targeting identity, such as abusing objects or non-protected groups.
Claim: Compared to LoRA we yield 9.5\times parameter reduction, while remain comparable with ELoRA.
Context: Runtime & FLOPs Comparison. Fig. 3 shows the comparison of the normalized average training runtime, normalized FLOPs, and normalized trainable parameters. For AFLoRA, we average the training time, FLOPs, and trainable parameters over six GLUE datasets (except the MNLI and QQP datasets). Note, for LoRA and ELoRA, the trainable parameters and FLOPs remain fixed for all the dataset. We compute their average runtime same way as ours. Compared to ELoRA we can yield up to 1.86\times and 2.96\times runtime and FLOPs improvement while remain comparable with LoRA in these two metrices.
Caption: Figure 3: A comparison of various system performance between LoRA, ELoRA, and AFLoRA.
Claim: {N} makes the network behave badly at large depth, while it still works well with our scaling \frac{1}{N} , as shown in Figure 3 (a).
Context: We first train a ResNet-101 (He et al., 2016a ) on CIFAR-10 and ImageNet using the same hyper-parameters. Experimental details are in appendix B and results are summarized in table 1 , showing that the explicit addition of the step size \frac{1} {N} does not affect accuracy. In strike contrast, the classical ResNet rule without the scaling \frac{1}
Caption: Figure 3: (a) Test accuracy on CIFAR-10 as a function of the number of blocks in each layer of the ResNet. Within each layer, weights are tied ( 3 runs ). (b) Failure of the adjoint method with a ResNet-101 on ImageNet (the approximated gradients are only used in the third layer of the network, that contains 23 blocks). (c) Relative error between the approximated gradients using adjoint method and the true gradient, whether using a ResNet or a HeunNet. Each point corresponds to one parameter.
Claim: Appraisals consistently showing better accuracy across both GPT-4 models and human annotators include Pleasantness, Unpleasantness, Goal Support, and External Norms (marked as bold in Table 2 ), suggesting that these are the easiest to infer based on text.
Context: In analyzing appraisal dimensions across all experiments, we looked for patterns by comparing the RMSE of individual appraisal dimensions to the macro-averaged RMSE.
Caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Claim: All in all, gender-specific projector gets the higher correlation score.
Context: In Table 2 we report correlation scores for the gender diversity. Again, we split in two groups: male-voice and female voice-dominant. From the results, we notice that the correlations showed by the non-specialized embeddings are extremely weak. However, both specialized models reach very high correlations (e.g., up to 0.946 for SpeechSim/Gender).
Caption: Table 2 : Gender diversity. Average Spearman correlations between proportion of female voices and diversity scores induced by speech representations.