RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Second, pairwise comparisons with the baseline show both CG and finetuning are effective in formality control, where CG has slightly higher win ratio than FT against the baseline.
Context: The results are in Table 6 .
First, adding attribute control does not appear to impact translation quality.
Caption: Table 6: Human evaluation on Bengali,
with quality on a 5-point scale \uparrow and formality on a 3-point scale ( \uparrow : formal) with standard deviations .
Last two columns show pairwise comparison of formality scores to baseline NLLB-200 given the same source sentences (winning: scoring more in the direction of the desired formality).
Claim: Surprisingly, the OpenAI GPT-2 Detector performs poorly on the GPT-2_xl data from TuringBench, although it can be considered supervised for this particular target.
Context: We compare our ConDA framework with relevant unsupervised baselines and report results in Table 4 . Out of the four GLTR measures ( \log p(x) , Rank, Log Rank, and Entropy), the first three fare quite well for detecting CTRL-generated text, but performance on other generators is quite poor. DetectGPT, which is the most recent method we evaluate, performs poorly on almost all generators, with some satisfactory performance on CTRL and XLM.
Caption: Table 4: Performance of ConDA in comparison to unsupervised baselines, as AUROC. For ConDA, we report the average AUROC over all sources (for each target) and also the maximum AUROC (across all sources), along with the corresponding source in parentheses. Bold shows superior performance across each target.
Claim: A day effect was only found for the Nine Hole Peg test, revealing that the time for completing the task during the assessment-2 was less than during the assessment-1 (MD: 0.39 s; 95% CI [0.19–0.60]; P < 0.001).
Context: No side * day interaction was found in the RM-ANOVA for any of the variables ( P > 0.114), indicating a similar pattern across assessments of the dominant and non-dominant sides.
Caption: Table 2: Clinical characteristics of participants in the different test for the wrist extensor muscles.
Claim: Number+Text a have lower Dolma token counts when RQA fails (Fig 4 ), so LLMs struggle to recall long-tail numerical facts Kandpal et al. ( 2023 ) .
Caption: Figure 4: Answer token count and question difficulty of when RQA succeeds/fails, averaged over all LLMs.
Claim: S2 . Figure 1A displays the inhibitory effect of 23 tyrosine kinase inhibitors on tramadol metabolism, with crizotinib, sorafenib, and regorafenib exhibiting the highest inhibitory rates of 97.22%, 96.66%, and 83.65%, respectively.
Context: The Michaelis–Menten curve for tramadol in RLM and HLM, along with the corresponding K m value, is shown in Fig.
Caption: Figure 1: Determine the interaction between representative tyrocine kinase inhibitors and tramadol.
(A) The inhibitory effect of 23 types of tyrosine kinase inhibitors on the production of O-desmethyl tramadol in RLM compared with the control group. (B) Evaluate the half-maximal inhibitory concentration (IC50) of crizotinib with various concentrations (0.01, 0.1, 1, 10, 25, 50 and 100 µM) on tramadol metabolism in RLM and HLM. Data are presented as the mean ± SD, n = 3.
Claim: In addition to plastic, various other types of anthropogenic materials found in 23 out of 1,447 pellets were identified and classified as non-plastic debris.
Caption: Table 2: Composition of plastic and non-plastic debris found in 1,447 analyzed pellets of the neotropic cormorant,N. brasilianus, on the Circuito de Playas Costa Verde (CPCV), Lima, Perú, during both the pre-pandemic and pandemic phases.
Claim: Humans also perform noticeably better on the NYCartoons dataset and on the idiom subset of the task.
Context: To find out how humans perform on the task, we hire two expert annotators with formal education in linguistics. We present them with 10 example instances and then ask them to complete 99 randomly sampled test set instances. We also evaluate our best model (see Table 3 ) on the same set. Results are shown in Table 4 . Human performance is quite strong, almost reaching 90 F1@0 score overall. Human performance is better than our strongest fine-tuned model (LLaVA-7B-eVil+VF) performance with p<0.05 for Annotator 1 and p<0.07 for Annotator 2. Humans excel at interpreting memes, with both annotators reaching a 100% F1 score.
Caption: Table 4: Human baseline results (F1@0) by phenomenon and source dataset.
Claim: Furthermore, SCMoE can enhance the major@20 accuracy from 75.59 to 78.31 (+2.72) on GSM8K.
Context: Using self-consistency [ 35 ] for multiple sampling and taking a majority vote to determine the final answer is a common method to improve LLMs’ performance. Therefore, we explore whether SCMoE can combined with self-consistency. For vanilla self-consistency, we use temperature sampling with temperature \tau=0.7 to reach the best baseline performance. For self-consistency with SCMoE, we simply employ \beta=0.5 , rank-3 routing as weak activation, according to the best hyperparameters setting from Table 8 . It is worth noting that since SCMoE already has a mask \alpha=0.1 to limit the sampling range of the vocabulary, we do not perform any additional temperature processing on the final logits. As shown in Figure 5 ,
SCMoE (67.94) yields comparable results with major@5 (66.87).
Caption: Figure 5 : Experimental results on combining SCMoE with self-consistency on GSM8K using Mixtral 8x7B.