RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Finally, we see ConDA outperforms all the baselines in terms of maximum AUROC, and all but one in terms of average AUROC.
Context: We compare our ConDA framework with relevant unsupervised baselines and report results in Table 4 . Out of the four GLTR measures ( \log p(x) , Rank, Log Rank, and Entropy), the first three fare quite well for detecting CTRL-generated text, but performance on other generators is quite poor. DetectGPT, which is the most recent method we evaluate, performs poorly on almost all generators, with some satisfactory performance on CTRL and XLM. Surprisingly, the OpenAI GPT-2 Detector performs poorly on the GPT-2_xl data from TuringBench, although it can be considered supervised for this particular target.
Caption: Table 4: Performance of ConDA in comparison to unsupervised baselines, as AUROC. For ConDA, we report the average AUROC over all sources (for each target) and also the maximum AUROC (across all sources), along with the corresponding source in parentheses. Bold shows superior performance across each target.
Claim: The baseline characteristics were described in Table 1 . For women <35 years old, the median age for the HRT group and GnRH-a group were 30.6 and 31.3 years, respectively.
Caption: Table 1: Demographic characteristics of patients between different age groups and endometrial preparation groups.
Claim: CoT prompting only improves the performance of the GPT-4o-mini model, whereas it significantly degrades the performance of the LLaMA 3.1 series.
Context: (2) Chain-of-Thought prompting has limited effect on LLMs performance and can sometimes reduce effectiveness. While few-shot learning generally enhances generic LLMs’ performance, Table 1 shows that the difference between prompting styles is marginal.
Caption: Table 1: Performance of different fine-tuned language models and LLMs under different prompts on FiNER-ORD task.
Claim: We find that the BIC value for RR-RL 2 is substantially lower compared to that of the hybrid model ( 5562.63 against 6158.91 ) when aggregated across all participants; all other models provide a less adequate fit to human choices.
Context: Having established that different styles of exploration emerge in RR-RL 2 depending on its description length, we next set out to test how well it explains human choices. In order to do so, we conducted a Bayesian model comparison (Bishop, 2006 ) . A detailed summary of our comparison procedure is provided in Appendix B . We used the Bayesian information criterion (BIC, Schwarz, 1978 ) as an approximation to the model evidence. The resulting BIC values for each candidate model are illustrated in Figure 2 (a).
Caption: Figure 2: Model comparison results on the two-armed bandit data from Gershman ( 2018 ). (a) Bayesian information criterion (BIC) values for the aggregated data of all participants. Lower values correspond to a better fit to human behavior. (b) Posterior probabilities for each model and participant. Higher values correspond to a better fit to human behavior.
Claim: Moreover, its correlation coefficient is smaller than that of SpeechSim/Voice.
Context: We report results in Table 6 . From this results we see that SpeechSim/Gender is positively correlated with the gender diversity. In contrast, the SpeechSim/Voice metric tracks the speaker diversity and hence is negatively correlated with the gender diversity. Finally, the original SpeechSim embeddings have negative correlation with the gender diversity.
Caption: Table 6 : Changing voice and gender diversity in the opposite directions. Average Spearman correlation between the gender diversity and the diversity scores induced by speech representations.
Claim: As suggested by McGovern et al. ( 2024 ) , we indeed see that human translations over or underemphasize the intertextuality present in the text, whereas machine translations provide a neutral baseline, based on these results.
Caption: Table 5: Overemphasized Intertextuality by Human Translation. The intertextuality from Hebrews 8:12 to Isaiah 43:25 is amplified by the human translator’s decision to render different words as "sin". The machine translation abstains from this and restores the original distance, but loses coherence.
Claim: A total of 54 COPD patients (25 in stages I–II and 29 in stages III–IV) and 24 healthy individuals participated in the study from September 9, 2021, to May 1, 2022.
Caption: Table 1: Participants’ clinical characteristics and demographics.
Claim: The evaluation results are presented in Table 1 . While the state-of-the-art models demonstrate competitive performance, SocialSense outperforms all other models across all evaluation metrics consistently.
Context: We conduct an evaluation of the proposed SocialSense model and the baseline models introduced in Section 4.2 for the supervised response forecasting task.
Caption: Table 1: Response forecasting results. We report the Spearman and Pearson correlations for the forecasting of sentiment intensity, as well as Micro F1 and Macro F1 scores for the sentiment polarity prediction. The best overall performance is in bold. Our framework outperforms the baselines consistently.
Claim: As we can see in Fig. 2 , the model with r=4 , yields poorer performance, highlighting the need for high rank for the frozen tensors.
Context: To understand the high rank requirement for the frozen projection metrices in ELoRA, we conduct two sets of fine-tuning on SST-2 and MRPC, with ELoRA having rank ( r ) of 1024 and 4, respectively.
Caption: Figure 2: Performance of ELoRA with two different ranks of the frozen projection matrices.
Claim: Here, again, we notice that SpeechSim has higher correlations than other non-specialized embedding models.
Context: We report average correlations in Table 4 .
Caption: Table 4 : Accent diversity. Average Spearman correlations between the accent class entropy and diversity scores induced by speech representations.
Claim: Sonnet achieve the highest average accuracy of 71.95% and 72.15% on the ambiguous/unanswerable questions.
Context: Table 5 shows our baseline method’s (DIN-SQL) performance on SQL prediction of various LLMs given the interaction between the user and the assistant. Overall, Mixtral-large-v2 and Claude 3.5
Caption: Table 5: Execution accuracy of SQLs predicted with DIN-SQL using different LLMs on various categories of ambiguous, unanswerable, and answerable questions. The "All" column shows the overall average accuracy across all categories, while the "Avg. Excluding Answerable" column shows the average accuracy excluding the answerable questions from the Spider dataset.
Claim: Women aged <35 years had higher clinical pregnancy and live birth rates than women aged ≥35 years.
Context: Table 2 shows the clinical outcomes of FET.
Caption: Table 2: Comparison of reproductive outcomes.
Claim: We observe an improvement when the selective rationale is added; however, degradation occurs when we add the eliminative rationale, even if it is provided with the selective rationale.
Context: Because the collected rationales are expected to support the decision of selecting and eliminating answer options, we investigate whether adding the rationales to the main questions improves the performance in the five-shot InstructGPT. We append the rationale to the context, main question, and four options with the Rationale: label. The results are shown in Table 5 .
Caption: Table 5: MainQ accuracy of InstructGPT that uses the selective or eliminative rationales in the input.
Claim: In the most challenging second-order ToM inference tasks, where agents estimate others’ beliefs about their own mental states, GPT-4 + Belief agents correctly respond in nearly 70% of cases.
Context: A critical aspect of teamwork is inferring teammates’ mental states, including beliefs, desires, and intentions. We assess LLM-based agents by asking them to conduct Theory of Mind inferences during the mission. As seen in Table 2 , LLM-based agents can estimate their own and their teammates’ mental states.
Caption: Table 2: LLM-based agents’ performance in ToM inference tasks. Natural language answers are annotated by experimenters and compared with the ground truth based on global interaction history. Percentages represent the inference accuracy.
Claim: This result is alarming since it suggests that HS detectors are entangled with emotion polarity, and some safe posts with negative emotions, such as counter-speech expressing disapproval or sadness, are likely marked as hateful, potentially silencing the voices of vulnerable groups.
Context: We further group the fine-grained emotions into positive (1), negative (-1), and ambiguous (0), based on Demszky et al. ( 2020 ) ’s taxonomy and present the models’ classification accuracy in the presence of emotions with different polarities in Table 4 . The result is revelatory: All models can relatively accurately identify hateful posts with negative emotions and non-hateful posts with positive emotions. However, the accuracy degrades drastically for non-hateful posts with negative emotions, especially for HateBERT and ToxDect-roberta. 11 11 11 We observe a similar trend for hateful posts with positive or ambiguous emotions, although the number of such cases is much smaller.
Caption: Table 4: Classification accuracy of HateBERT, ToxDect-roberta, Perspective API, and Llama Guard 3 1/8B on GPT-HateCheck grouped by the hatefulness label (hate) and the polarity of the detected emotions (emo). We highlight the “positive” labels in green (“non-hateful” and positive emotions) and “negative” labels in red . The ambiguous emotions are highlighted in yellow .
Claim: For instance, top-3 routing yields best results on GSM8K, while top-4 routing achieves the highest accuracy on HumanEval and StrategyQA.
Context: As revealed by Figure 1 , using default top-2 routing is not optimal for all tasks.
Caption: Figure 1 : Performance comparison between increasing the value of top- k ( i.e. , ensemble routing) and SCMoE. SCMoE surpasses the performance of ensemble routing across various benchmarks.
Claim: The average RMSE scores for five completions applying the majority voting algorithm was 1.22 (see column GPT-4 {}_{\text{emo}} ), which is worse than the same task without the emotion label prediction studied in relation to research question Q3.
Context: The results of adding emotion to a prompt are shown in the last column of Table 2 .
Caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.
Claim: Additional synthetic datasets and high-quality retrieval datasets further improve the embedding model quality despite the tasks these datasets solve already being well represented in the basic datasets.
Context: Results presented in Table 2 indicate that the embedding model gets better results when trained on data in Russian and English simultaneously.
Caption: Table 2: Different data sources impact. Model performance is measured on ruMTEB. Avg. stands for the average score and is computed as the mean of the category scores. The best score is put in bold, the second best is underlined.
Claim: The average pairwise RMSE (APRMSE) values, macro-averaged over all appraisal dimensions, were very close to each other, ranging from 0.6 to 0.63 over the five runs, with a mean of 0.61 (see Table 1 .
Caption: Table 1: Average Pairwise RMSE (APRMSE) and Spearman correlation coefficients ( \rho ) of 108 random samples of five GPT-4 runs. All Spearman correlations were statistically significant at the level of p<0.001 .
Claim: We also see that the representations we learn lead to competitive results with respect to BERT and RoBERTa despite Sentence-BERT not being suited to classification.
Context: Similar to the clustering setting, we compare results from a Logistic Regression (LR) applied on the original sentence embeddings with and without the filtering operation we introduced. We also use fine-tuned BERT and RoBERTa (2 epochs) as baselines; we use the base versions due to computational restrictions. We report the results in Table 4 . We see that this operation leads to better performances on the classification task on the majority of the datasets with respect to the filterless Sentence-BERT but this performance increase is not as pronounced as for the clustering task.
Caption: Table 4: Classification results in terms of F1 score on the eight data sets.
Claim: Forty-eight percent were susceptible to tobramycin ( Figure 3A ).
Context: Clinical isolates presented the higher resistance (100%) to the three antibiotics: imipenem, meropenem, and piperacillin/tazobactam.
Caption: Figure 3: Clinical and environmental isolates susceptibility profile.
The percentages of clinical isolates that were resistant, intermediate, or sensitive to each antibiotic. (A) Clinical isolates susceptibility profile, (B) river isolates susceptibility profile. Acronyms: CIP, ciprofloxacin; GEN, gentamycin; SXT, trimethoprim/sulfamethoxazole; AMK, amikacin; FEP, cefepime; MEN, meropenem; IMP, imipenem; TOB, tobramycin; SAM, ampicillin/sulbactam; CAZ, ceftazidime; TZP, piperacillin/tazobactam; R, resistant; I, intermediate; S, sensitive.
Claim: Matching the main result of the experimental study, we find that strategic directed exploration increases with description length, while strategic random exploration remains unaffected.
Context: Results: We trained RR-RL 2 with a targeted description length of \{100,200,\ldots,10000\} nats on the same distribution used in the original experimental study. Figure 4 (b) visualizes how strategic directed and random exploration change as the description length of RR-RL 2 increases.
Caption: Figure 4: Illustration of strategic directed and random exploration in the horizon task. (a) Human data from Somerville et al. ( 2017 ). During adolescence, people start to engage more in strategic directed exploration, whereas strategic random exploration remains constant over time. (b) Data simulated from RR-RL 2 with varying description lengths. Like in the human data, we observe an increase in strategic directed exploration, but no change in strategic random exploration.
Claim: The number of the direct and contextual rationales is the largest among the other types, which further increases when we look at the error cases of InstructGPT.
Context: We report our annotation results in Table 4 .
Caption: Table 4: Annotation results of rationale types on 100 examples randomly sampled from all subquestions (left) and from the error examples by InstructGPT (right).
Claim: Moreover, we check the effect of attacking the generated images, which leads to a lower AUROC score.
Context: Ablation study: ATD uses HAT as the feature extractor and performs adversarial training on open and closed sets to robustify the discriminator. As an ablation study using CIFAR-10 as the in-distribution data, we replace HAT with a standard trained model to check its effectiveness. Also, in another experiment, we replace the discriminator adversarial training with the standard training. The results in Table 3 demonstrate that these settings are not as effective as ATD in achieving a robust detection model. As another ablation, we consider removing the feature extractor and generating images instead of features. Based on the results, the trained discriminator with this method is also not as robust as the ATD.
Caption: Table 3: Ablation study on our method. Other choices for the feature extractor, training method, and generated data attacking are tested on CIFAR-10, but none of them is as effective as the setting used in the ATD.
Claim: Conversely, RoBERTa leads in precision at 0.72, but with lower recall at 0.84.
Context: The results show RACLETTE’s Combined method achieving the highest recall of 0.95, indicating superior ability to identify relevant cases, though this comes with a trade-off in precision at 0.63.
Caption: Table 4: Classification results for different methods of comparing embeddings to detect risk of suicide. The higher the score (a.k.a. the greener), the better.
Claim: Table 8 illustrates the self-consistency measured by the accuracy when comparing pairs, and demonstrates that even when using few outputs, the model is very consistent to the final rankings that would be achieved by using many more examples.
Context: SummEval has 16 summaries per context which leads to 240 possible comparisons. If one were to instead randomly sample N outputs and consider all N\!\cdot\!(N\!-\!1) comparisons, how consistent would the rankings with the subset of systems be with respect to the final predicted rankings?
Caption: Table 8: Accuracy when using fewer systems with respect to final rankings (using all 16 systems) and the ground truth labels. Results shown for Summeval COH using FlanT5-xl.
Claim: Positive correlations between AR and ORC1 were confirmed in the TCGA-PRAD samples ( Figure 7D ).
Context: To elucidate the function of ORC1 in PRAD, we established stable ORC1-overexpressing PRAD cells (22RV-1 and C4-2B). The up-regulation of ORC1 significantly enhanced PRAD cell soft agar colony formation efficiency ( Figure 7A ). Besides, the self-renewal potentiality of PRAD cells was also increased when cells were transfected with ORC1, indicating that ORC1 could also maintain stemness features of PRAD ( Figure 7B ). Next, we intended to figure out the underlying regulations between AR and ORC1. Firstly, we found AR overexpression could notably elevate ORC1 expression levels, whereas AR knockdown decreased its levels, along with another known AR downstream target of PSA ( Figure 7C ).
Caption: Figure 7: AR activates ORC1 to sustain tumor progression and enzalutamide resistance in PRAD. (A) The anchorage-independent growth of C4-2B and 22RV-1 cells in soft agar (scale bars = 200 µm, left). Quantification of the soft agar colony formation assay results (right). (B) Sphere formation assays revealing the self-renewal ability of ctrl and ORC1-overexpressing cells (scale bars = 200 µm, left). (C) Western-blotting assay indicating the regulations between AR and ORC1. (D) The scatter diagram indicating the positive correlations between AR and ORC1. (E) ChIP assays were utilized to determine the level of AR binding (left) and the enrichment of H3K27ac (right) at the promoter of ORC1 in AR deficiency or control PCa cells. (F) Cell proliferation determined in indicated cells with enzalumitide treatment. C4-2B-Parental and C4-2B-ENZR cells were transiently transfected as indicated. (G) Mice bearing C4-2B-ENZR xenografts were treated with contro DMSO and enzalutamide (10 mg/kg p.o.) for 4 weeks (n = 5/group). (H) The tumor volumes in indicated groups were detected the curves were generated. (I) The tumor weight in indicated groups were recorded and compared. ***p < 0.001.
Claim: Similarly, reliability was ‘excellent’ for the maximal isometric strength on both sides (ICC: 0.97).
Context: Table 3 presents ICC 3,1 , SEM, MDC 90 , and MDC 95 for each variable. For the muscle mechanical properties of the dominant and non-dominant sides, reliability was ‘excellent’ for the frequency and stiffness parameters (ICC: 0.91–0.96), and ‘good’ to ‘excellent’ for the decrement (ICC: 0.86–0.91). For the manual dexterity, reliability was ‘good’ on both sides (ICC: 0.79–0.82). For the pressure pain thresholds, reliability was ‘excellent’ for the extensor carpi radialis brevis point (ICC: 0.90–0.93), and ‘good’ for the deltoideus point (ICC: 0.84–0.89). For the active range of motion, reliability was ‘excellent’ for both extension, flexion, and total range (ICC: 0.94–0.97).
Caption: Table 3: Reliability indicators of myotonometry, manual dexterity, pressure pain thresholds, active range of motion, and maximal isometric strength for the wrist extensor muscles.
Claim: The underfitting delta model had a win rate of 54 when \alpha was 1.0, which increased to 64 when \alpha was 2.0.
Context: For instruction tuning, we tested values within the range of [0.5, 2] and evaluated them on the first 50 data points of AlpacaEval. As shown in Figure 4(a) , we found that the performance was optimal when the \alpha value was 1.0, and increasing or decreasing \alpha resulted in decreased performance. This is similar to causing the delta model to overfit or underfit, so we explored whether adjusting the \alpha value during inference could counteract the overfitting and underfitting during training. Specifically, the delta model performed best after training for four epochs, and we selected the checkpoint after training for two epochs as the underfitting scenario and the checkpoint after training for eight epochs as the overfitting scenario. We found that adjusting the \alpha value could counteract the overfitting or underfitting during training.
Caption: (a) Impact on instruction tuning; (b) Impact on unlearning; Impact of strength coefficient \alpha on performance
Claim: Finally, we show that ViTCA does not dramatically suffer when no explicit positioning is used—in contrast to typical Transformer-based models—as cells are still able to localize themselves by relying on their stored hidden information.
Context: As shown in Tab. 2 , ViTCA benefits from an increase to most CA and Transformer-centric parameters, at the cost of computational complexity and/or an increase in parameter count. A noticeable decrease in performance is observed when embed size d\!=\!512 , most likely due to the vast increase in parameter count necessitating more training. In the original ViT, multiple encoding blocks were needed before the model could exhibit performance equivalent to their baseline CNN [ 13 ] , as verified in our ablation with our ViT. However, for ViTCA we notice an inverse relationship of the effect of Transformer depth, causing a divergence in cell state. It is not clear why this is the case, as we have observed that the LN layers and overflow losses otherwise encourage a contractive F_{\theta} . This is an investigation we leave for future work. Despite the benefits of increasing h , we use h\!=\!4 for our baseline to optimize runtime performance.
Caption: Quantitative ablation for denoising autoencoding with ViTCA (unless otherwise stated via prefix) on CelebA [ 47 ] . Boldface and underlining denote best and second best results. Italicized items denote baseline configuration settings. † Trained with gradient checkpointing [ 44 ] , which slightly alters round-off error during backpropagation, resulting in slight variations of results compared to training without checkpointing. See Appendix A.2 .
Claim: As shown in Table 2 , there was a statistically significant difference in the frequency of colds experienced by middle-aged and elderly individuals with varying levels of weekly PA ( P < 0.001).
Caption: Table 2: Cold in middle-aged and elderly with different characteristics.
Claim: The application of the majority voting algorithm improved the RMSE on average by about 30% from 1.61 to 1.12 (columns GPT-4 {}_{\text{avg}} and GPT-4 {}_{\text{maj}} in Table 2 ).
Caption: Table 2: RMSE results for research questions Q2, Q3, and Q4. GPT-4 stands for the GPT-4 annotator, Human for the human reader-annotator, avg : average of five GPT-4 completions/human guesses, maj : majority vote of five GPT-4 completions/human guesses, conf : majority vote of five GPT-4 completions/human guesses with confidence rating as a tiebreaker, emo : majority vote of five GPT-4 completions with the emotion prediction task in a prompt. Bold/underline marks the appraisal dimensions that are consistently predicted better/worse than the macro average.