RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: At the same time, six open-ended tasks are studied, based on nine distinct datasets ( e.g. , MedText).
Context: This study considers four different close-ended healthcare tasks, which include nine different datasets ( e.g. , MedQA). These are all assessed using the accuracy metric.
Caption: Table 1: This table presents the tasks implemented in this paper. The first column specifies the different tasks. The second details the metrics used (ROUGE includes ROUGE1, ROUGE2 and ROUGEL, and Perplexity includes Bits per Byte, Byte Perplexity, and Word Perplexity). The third column outlines the benchmarks used for each task.
Claim: Conversely, low-frequency adaptation resulted in a 4% increase in numerosity (linear regression slope of 1.03, compared to the baseline slope of 0.99), the line shifted towards the upper side compared to the baseline (no adaptation), indicating an overestimation of perceived numerosity ( Figure 2B ).
Context: The adaptation effect is measured by adaptation index. We calculated the adaptation index separately for the grouping and no-grouping conditions. The results are presented in Figure 2A , which illustrates the adaptation index under both grouping and no-grouping conditions. Notably, the grouping condition exhibits a more pronounced adaptation effect compared to the no-grouping condition ( t = 3.233, p = 0.23, Cohen’s d = 1.572). Figure 2B illustrates the mean perceived numerosity (averaged across all subjects) plotted against the physical number of pulses. The mean estimates without adaptation, represented by the gray dotted line, closely aligned with the veridical values. A zero-anchored linear regression model ( R 2 = 0.99) effectively described the data, exhibiting a best-fitting slope of 0.99. The introduction of high-frequency adaptation consistently led to a 9% reduction in apparent numerosity across all tested numerosities (linear regression slope of 0.90), as can be seen from Figure 2B , the line shifted towards the upper side compared to the baseline (no adaptation), indicating an overestimation of perceived numerosity.
Caption: Figure 2: Results.
(A) Adaptation index under grouping and no-grouping conditions; (B) The perceived numerosity, averaged across trials and subjects, varies as a function of physical numerosity in the three adaptation conditions. The analysis includes best-fitting linear regressions (R2 >0.98 in all conditions). The regression slopes are as follows: no adaptation (dotted line) = 0.99; high-frequency adaptation (square) = 0.90; low-frequency adaptation (triangle) = 1.03 (p < 0.001 in both conditions).
Claim: The evaluation results are shown in Table 6 . InstructBLIP performs the best under all settings, highlighting the importance of instruction-tuning on large visual instruction corpora.
Caption: Table 6: Evaluation results of LVLMs on POPE and VQA. For VQA tasks, we report the VQA score on A-OKVQA and Accuracy on GQA. For POPE, we copy the result under the random setting from Table 11 .
Claim: In clinical samples, except Veillonella , Leptotrichia and Streptococcus , other bacteria occupied the most of bacterial community.
Context: To display the composition of dominant bacteria, the stacked column plot showed the percentage of each high-abundance bacteria.
Caption: Figure 3: Bacterial community structure analysis.
(A) The distributions of the predominant bacteria at genus level. (B) The relative abundance of differential predominant bacteria among aerobic culture, anaerobic culture and clinical groups analyzed at genus level by ANOVA.
Claim: Our findings indicate that adding reinforcement learning (STRUX+RL) leads to stronger performance compared to using the SFT method alone.
Context: System Comparisons. Table 1 shows the macro-averaged precision, recall, F-scores, and accuracy for the test set. STRUX outperforms strong baselines in accuracy and F-scores for stock investment decisions.
Caption: Table 1: Our STRUX system outperforms strong benchmarks in making stock investment decisions. We present macro-averaged precision, recall, F-scores, accuracy for the test set. LLMs evaluated are: Llama3-8b-Instruct and gpt-4o-mini-2024-07-18 .
Claim: We also show that adversarial training learns a useful generative model with meaningful latent space by interpolating between sentences (Table 3 ).
Context: We obtain further insight into what the adversary learns by analyzing the word dropout scores for different sentences. Table 2 shows that the adversary applies lower dropout probabilities to less informative words such as ‘unknown’ tokens that replace all out-of-dictionary words, and so offer little information about the next word. Depending on the data semantics, the adversary selects different types of words: for SNLI, verbs tend to be picked that explain the activity, e.g. working and participating ; for Yahoo, words are identified that carry question semantics, e.g. what , how , if , and when . Figure 4 (a) shows a quantitative analysis of dropout saliency map across different part-of-speech (POS) tags. Verb (verb), interjections (intj), and nouns (noun) have higher saliency scores (higher chances of being dropped) compared to punctuation (punc), determiners (det), and the start (sos) and end tokens (eos) which are relatively easier to predict given previous words.
Caption: Table 3 : Sentence interpolation (Yelp dataset). Representations of two sentences (top, bottom) are obtained by feeding them through an adversarially trained VAE encoder. Three linearly interpolated representations are passed to the VAE decoder and sentences generated by greedy sampling (middle) .
Claim: This figure is inverted
for Raven which obtains a 4-fold improvement over the base model inferring the correct answer more than 85% of the time.
Context: The results are summarised in Table 1 .
Compared to the base model, Raven significantly improves the results on the PhraseBank dataset by an absolute 25.9%. On the Wiki-SQL dataset the base model is unable to infer the correct answer almost 80% of the time.
Caption: Table 1: The data statistics and experimental results (Exact Match) of various benchmarks and models. The best results are in bold . GPT-3.5 results are based on 5-shots. Sota is based on previously published results.
Claim: Regarding methodological quality assessment tools, 12 studies utilized the Cochrane Collaboration Network RoB risk assessment tool, four employed the PEDro scale, one used the Cochrane Collaboration Network RoB and PEDro, and one applied the EPHPP for risk assessment.
Context: The included meta-analyses were published between 2018 and 2023, and all the original studies were randomized and controlled. The largest study included 2,526 participants and the smallest had 182 participants, involving a total of 18,461 individuals. The interventions in the trial groups consisted of aerobic, physical, mental, resistance, and multipart exercises. The control group in the randomized trials maintained their regular daily activities without any specific exercise interventions.
Caption: Table 1: Basic characteristics of the literature included in the study.