RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Compared with three pathologists of varying experience (junior, medium, and senior), the model’s performance was almost equivalent to that of a medium-level pathologist ( Figure 2A ).
Context: A segmentation network was developed to accurately distinguish tumor regions from WSIs. This model demonstrated high efficacy, achieving an AUC of 0.960 (95% CI [0.959–0.961]). The accuracy, sensitivity, and specificity were 0.888 (95% CI [0.887–0.890]), 0.859 (95% CI [0.856–0.861]), and 0.908 (95% CI [0.906–0.909]), respectively.
Caption: Figure 2: Performance of the segmentation network.
(A) Receiver operating characteristic (ROC) curve comparing the performance of the segmentation network with three pathologists (senior, medium, and junior) in identifying tumor areas. (B) Confusion matrix for the segmentation network. (C) Original WSIs and annotation maps identified by the segmentation network. Left: original WSI images; right: corresponding annotation maps.
Claim: The results indicated that there were more bacterial categories in clinical samples than cultures by media.
Context: To display the composition of dominant bacteria, the stacked column plot showed the percentage of each high-abundance bacteria. In clinical samples, except Veillonella , Leptotrichia and Streptococcus , other bacteria occupied the most of bacterial community. No matter under anaerobic and normoxia conditions, the percentages of other bacteria, Veillonella and Leptotrichia were clearly decreased. Meanwhile, there were observed increases in Desemzia , Actietobacter , Lactobacillus , Limosilactobacillus , Ligilactobacillus , Vagococcus and Streptococcus ( Figure 3A ).
Caption: Figure 3: Bacterial community structure analysis.
(A) The distributions of the predominant bacteria at genus level. (B) The relative abundance of differential predominant bacteria among aerobic culture, anaerobic culture and clinical groups analyzed at genus level by ANOVA.
Claim: Third, the impact on formality scores is more prominent when steering towards informal translation.
Context: The results are in Table 6 .
First, adding attribute control does not appear to impact translation quality. Second, pairwise comparisons with the baseline show both CG and finetuning are effective in formality control, where CG has slightly higher win ratio than FT against the baseline.
Caption: Table 6: Human evaluation on Bengali,
with quality on a 5-point scale \uparrow and formality on a 3-point scale ( \uparrow : formal) with standard deviations .
Last two columns show pairwise comparison of formality scores to baseline NLLB-200 given the same source sentences (winning: scoring more in the direction of the desired formality).
Claim: Our method further reduces the memory needed for training to 2.23GB, an improvement of 18% compared to LoRA, and 45% compared to GaLore, while still reaching higher results than either of them.
Context: We now evaluate our method with M=16 on various language tasks, using RoBERTa-Base [ 28 ] in the GLUE benchmark, and compare it with full fine-tuning, LoRA [ 17 ] and GaLore [ 53 ] , presenting the results in Tab. 2 .
We observe that both GaLore and LoRA lower the memory requirements compared to fine-tuning from 4.64GB to 4.04GB, respectively to 2.71GB, at a cost of accuracy degradation with GaLore performance lowered by 0.39 pp , while LoRA accuracy drops by 0.67 pp .
Caption: Comparison of our method with full fine-tuning, GaLore and LORA on GLUE benchmark using pre-trained RoBERTa-Base. Our method reaches the best overall results while showing significant memory improvements, especially compared to GaLore. We bold the best results from the considered PEFT methods. The GPU memory is measured on-device.
Claim: For Base-22k and Large, the difference becomes large after the scale multiplier is under 0.6.
Context: In order to compare the robustness of ConvNext and Swin networks to scale, we first generate images of all main object models with unit object scale. Although objects have different volume in this state (e.g. microwaves are larger than spatulas), they take a fair amount of space in the frame and don’t present much of a challenge to large architectures. Specifically, top-5 accuracies for Swin-L and ConvNext-L are 90.5% and 93.8% respectively. We then perform counterfactual simulation testing by plotting the stability of correct predictions when the object scale multiplier is reduced from 1. We show these results in Fig. 5 . We observe that for Small, Base-22k and Large-22k networks, ConvNext vastly outperforms Swin for smaller objects.
Caption: Figure 5: Counterfactual study of all sizes of ConvNext and Swin networks for object scale.
Claim: Yet still only half of its explanations are considered adequate.
Context: In Table 6 , we show adequacy and preference rates for explanations from the 3 systems, where an explanation is deemed adequate if both annotators agreed it is, and inadequate if both agreed it is not. The preference percentage is also taken among instances where the annotators agreed that the model’s explanation is preferred among all the adequate explanations. The average IAA using Cohen’s \kappa is 0.47, indicating moderate agreement Cohen ( 1960 ) . We observe that the teacher model is leading in terms of the adequacy of the explanations and preference rate, as expected from a larger system equipped for higher quality reasoning and generation capabilities.
Caption: Table 6: Adequacy and Preference rates for generated explanations.
Claim: There was no significant difference in BMD between the two groups ( P > 0.05).
Caption: Table 4: Comparison of two groups of BMD and abnormal rates.
Claim: The experimental results presented in Table 6 demonstrate that ICL exhibits superior performance to rule-based baselines.
Context: To determine the extent to which LLMs engage in pattern analysis, beyond simply relying on frequency and recency biases, we run a comparative analysis between GPT-NeoX and heuristic-rules ( i.e., frequency & recency ) on the ICEWS14 dataset, with history length set to 100. frequency identifies the target that appears most frequently in the provided history while recency selects the target associated with the most recent fact in the provided history. The reason for our focus on ICEWS is that each quadruple represents a single distinct event in time. In contrast, the process of constructing YAGO and WIKI involves converting durations to two timestamps to display events across the timeline. This step has resulted in recency heuristics outperforming all of the existing models, showcasing the shortcoming of existing TKG benchmarks (See Appendix A.5 ).
Caption: Table 6: Performance (Hits@K) with rule-based predictions. The best model for each dataset is shown in bold .
Claim: Chi-square test analysis showed significant differences in the gestational age at diagnosis of fetal abnormalities, the number of fetal abnormalities, diagnostic gestational age, and treatment recommendations of doctors among pregnant women ( P < 0.05).
Caption: Table 1: Comparison of pregnancy outcomes in pregnant women with different characteristics of fetal abnormalities.
Claim: Better sample selection due to jittering subsequently leads to significantly better performance in regression (Fig. 4 (c)).
Context: Fig. 4 (a) shows that with jittering, the feature extractor exhibits higher accuracy on the clean test data due to its regularization effect. In the sample selection stage (Fig. 4 (b)),
the feature extractor trained without jittering easily overfits the noise, resulting in over-selection and higher ERR (§ 4.2 ). In contrast, the jittered feature extractor achieves a relatively low selection rate with halved ERR, indicating that the noisier samples are filtered out.
Caption: Figure 4: Jittering analysis. (a) When trained without jittering, feature extractors easily overfit the noisy training data (yellow-shaded region), while jittering-regularized feature extractors robustly learn from the noisy training data. (b) Overfitted feature extractors (yellow-shaded region) on noisy samples increase their likelihood, leading to a higher selection rate and ERR.
It exhibits nearly twice higher ERR (a lower value is better).
(c) Most importantly, jittering regularization improves performance in regression.
Claim: Notably, ALTER demonstrates the best performance in single-round reasoning among all other methods that utilize result ensemble techniques in the LLM era.
Context: We present the results on the WikiTQ and TabFact datasets. The experimental outcomes are summarized in Table 1 . From the results, we observe that our ALTER method achieves comparatively outstanding outcomes. Specifically, on the WikiTQ dataset, while the Mix SC method do marginally outperforms our results by aggregating multiple reasoning paths (with 10 sampling times), ALTER still managed to exceed the performance of all other methods under comparison.
Caption: Table 1: Results of different methods on WikiTQ and TabFact. 1 1 1 For the Dater method, we report the results of using the LLM-based method as backbone (We use underline to denote the second-best performance, bold to denote the best performance for each region: Pre-LLM era, LLM era with result ensemble and without ensemble)
Claim: Despite that they outperform GCN, SGC and GAT, they still perform worse than implicit models with infinitely deep layers (i.e., EIGNN and IGNN) since finite layers for aggregations cannot effectively capture underlying dependencies along extremely long chains (e.g., with length > 30).
Context: We first consider binary classification ( c=2 ) with 20 chains of each class as in Gu et al. [ 10 ] . We conduct experiments on graphs with chains of different lengths (10 to 200 with the interval 10). The averaged accuracies with respect to the length of chains are illustrated in Figure 1(a) . In general, EIGNN and IGNN always outperform the other baselines. GCN, SGC and GAT provide similarly poor performances, which indicates that these three classic GNN models cannot effectively capture long-range dependencies. H2GCN has an even worse performance than GAT, SGC, GCN, which is caused by its ego- and neighbor-embedding separation focusing more on ego-embedding rather than distant information. APPNP, JKNet, GCNII are either designed to consider different range of neighbors or to mitigate oversmoothing [ 16 , 33 ] .
Caption: (a) Results of binary classification; (b) Results of multiclass classification; Averaged accuracies with respect to the length of chains.
Claim: As shown in Table 1 , there are significant differences in the mean of AA scores across grade levels and BMI index (F = 3.402 P = 0.009; F = 5.752 P = 0.003).
Context: All participants were asked to complete a questionnaire that included gender, age, grade, height, weight, whether they were an only child, whether their home address was town or village, monthly household income, GPA for the previous school year, and disposable income. Specific issues and analysis results are shown in Table 1 .
Caption: Table 1: Differences in appearance anxiety, interpersonal sensitivity, social support, and depression under different subgroups of basic demographic characteristics.
Claim: The news domain has a higher frequency of such cases.
Context: We next characterize the distribution of error categories in factually inconsistent summaries generated by models across the domains considererd. Figure 1(b) reports the distribution of error categories for both models. 3 3 3 Model-specific distributions are in Appendix A.6 There are more extrinsic errors introduced in the news domain compared to the niche domain datasets. We include “mixed” errors for cases where errors were classified as different types (intrinsic/extrinsic) by annotators.
Caption: (a) Prevalence of factual errors in each of domains; (b) Distribution of error categories across domains; Distribution of errors and error categories across domains
Claim: Finally, it lowers the memory requirements of LoRA [ 17 ] by 4% while improving the accuracy by 0.6 pp
Context: We conduct experiments evaluating the performance of VeLoRA for full-tuning and how it complements other PEFT methods. In Tab. 1 we reproduce a large set of results for LoRA [ 17 ] , SSF [ 26 ] , and Hydra [ 20 ] on a subset of the VTAB-1K benchmark, where the sub-token size for each experiment is given in the Appendix. Unlike what is more common in the PEFT literature [ 19 , 20 ] , we do not perform any task-specific hyperparameter tuning that would change the memory, such as batch size and rank, and to also avoid any potential overfitting to the specific task. For all experiments we used the authors provided implementations for the adapters and integrated them into the same training framework for a fair comparison. We observe that VeLoRA improves the performance compared to full-tuning by 1.5 percentage points ( pp ), while lowering the memory requirements. We also observe that when combined with PEFT methods, VeLoRA can come with improvement in memory and performance. VeLoRA lowers the memory requirement of SSF [ 26 ] by 16% with only a minor degradation (0.1 pp ) in accuracy. It lowers the memory requirement of Hydra [ 20 ] by 7% while improving the accuracy by 0.1 pp .
Caption: Results on a subset of the VTAB-1k benchmark. All methods use a ViT-Base-224/16 model pre-trained on ImageNet-21k. The batch sizes and ranks are the same across all tasks.
Claim: Although we see satisfactory performance across most methods, our ConDA framework with source as FAIR_wmt19 and GPT2_xl has the best and the second best performance, respectively.
Context: Given recent concerns surrounding OpenAI’s ChatGPT and GPT-4 OpenAI ( 2023 ) , it is important to create detectors for text generated by these conversational language models. With the incredible fluency and writing quality these language models possess, not only can such text easily fool humans Else ( 2023 ) but can also be extremely difficult for detectors to identify. Even OpenAI’s detector struggles to detect AI-generated text reliably 6 6 6 https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text . Hence in this case study, we are interested in evaluating our ConDA framework on ChatGPT-generated news articles, in an unsupervised manner. Since there is no existing dataset of ChatGPT-generated vs. human-written text or news, we create our own dataset as explained in Section 4.1 . We assign ChatGPT as the unlabeled target domain and assume that we have labeled data from the 6 other generators (Table 1 ). Therefore we emulate a real-world scenario where labeled data from older generators may be available, but it might be hard to find labeled samples for newer LLMs. We sample 4k articles from our ChatGPT News dataset and evaluate the same 3 unsupervised models as in Section 4.2 (upper row block in Table 6 ), and our ConDA framework over 6 source generators (lower row block in Table 6 ) on this data. For GLTR, we report the average over the 4 statistical measures.
Caption: Table 6: Results on our ChatGPT News dataset using unsupervised baselines (upper row) and ConDA (lower row). Scores are AUROC. Bold shows best and underline shows second best performance.
Claim: In the adjusted regression, being male (aPR = 1.05; 95% CI [1.01–1.10]; p = 0.024), recommendation by family and friends (aPR = 1.08; 95% CI [1.04–1.12]; p < 0.001), and recommendation by physicians (aPR = 1.40; 95% CI [1.27–1.55]; p < 0.001) were associated with a higher prevalence of IBV with the booster dose, while age 45 years and older was associated with lower IBV (aPR = 0.89; 95% CI [0.81–0.98]; p = 0.018) ( Table 3 ).
Context: In the simple regression, being related to health (PR = 1.10; 95% CI [1.04–1.16]; p < 0.001), having a good perception of its efficacy and protective effect (PR = 3.69; 95% CI [2.57–5.30]; p < 0.001), the recommendation of family and friends (PR = 1.28; 95% CI [1.22–1.34]; p < 0.001), physicians (PR = 1.52; 95% CI [1.38–1.66]; p < 0.001), government health authorities (MINSA) (PR = 1.32; 95% CI [1.24–1.40]; p < 0.001) and politicians (PR = 1.17; 95% CI [1.13–1.21]; p < 0.001) were associated with a higher IBV. Conversely, being aged 45 or older (PR = 0.89; 95% CI [0.81–0.97]; p = 0.011), having children (PR = 0.92; 95% CI [0.87–0.98]; p = 0.006) and being afraid of adverse effects (PR = 0.91; 95% CI [0.87–0.96]; p < 0.001), were associated with a lower prevalence of IBV with the booster dose.
Caption: Table 3: Factors associated with the intention to be vaccinated with the COVID-19 booster dose in Peru, July 2022.
Claim: Furthermore, our model, SocialSense {}_{\text{Zero}} , achieves the highest scores consistently across all metrics.
Context: In addition to supervised response forecasting, we also evaluate our framework under the zero-shot setting (Section 3.4 ). The results are presented in Table 2 . Based on the higher scores attained by ChatGPT L , it is evident that the inclusion of latent structured persona information indeed aids the model in comprehending the user more effectively.
Caption: Table 2: The above Zero-Shot Response forecasting results highlight that the Social Prompt from Section 3.4 consistently offers an advantage.
Claim: We can see in the Table 6 that in all of the cases we get strong agreement ( >0.70 ) among the annotators for the style strength and appropriateness evaluation.
Context: The inter-annotator agreement in all of the tasks are shown in Table 6 . Note that, for calculating agreement in the semantic correctness evaluation task, all of the data points are aggregated to measure the agreement score as they represent categorical evaluation measures. On the other hand, that is not possible in case of ranking based evaluations for style strength and appropriateness. So, we measure the agreement for each data point and take the average agreement over all data points.
Caption: Table 6: Inter-annotator agreement scores for the three human evaluation tasks. Standard deviations over all data points are shown in brackets for the style strength and appropriateness evaluation tasks. The detailed procedure for calculating the agreement scores can be found in Appendix D .
Claim: On top of that, LISA displays consistently higher performance than LoRA on all subtasks, underscoring LISA’s effectiveness across diverse tasks.
Context: As shown in Table 5 , LISA consistently produces better or on-par performance when compared with LoRA. Furthermore, in instruction-tuning tasks, LISA again surpasses full-parameter training, rendering it a competitive method for this setting. Specifically, Figure 5 highlights the model’s performance in various aspects, particularly LISA’s superiority over all methods in aspects of Writing, Roleplay, and STEM.
Caption: Figure 5: Different aspects of LLaMA-2-70B model on MT-Bench.
Claim: Furthermore, on the TabFact datasets, both augmentation methods have a much larger impact on hard questions than on simple questions.
Context: To analyze the impact of two query augmentation methods in the query augmentor. We conducted experiments on the WikiTQ and TabFact datasets by discarding the step-back augmentation module (denoted as w/o step-back) and the sub-query augmentation module (denoted as w/o sub-query). For each dataset, we further categorized the questions based on the difficulty level, following Ye et al. ( 2023 ) . This stratification facilitates a more comprehensive evaluation of each module’s impact across different types of questions. The ablation test results are reported in Table 2 . From the results in the table, it is anticipated that employing both augmentation methods simultaneously yields the best performance under all experimental settings. For WikiTQ datasets, the accuracy of ALTER without step-back/sub-query augmentation drops by 2.9\%/2.0\% , demonstrating the necessity of augmented information from multi-queries.
Caption: Table 2: Ablation results of query augmentor on the test sets of WikiTQ and TabFact.
Claim: Among non-tumor region samples, 91% were correctly identified as non-tumor, with 9% misclassified as tumor ( Figure 2B ).
Context: A segmentation network was developed to accurately distinguish tumor regions from WSIs. This model demonstrated high efficacy, achieving an AUC of 0.960 (95% CI [0.959–0.961]). The accuracy, sensitivity, and specificity were 0.888 (95% CI [0.887–0.890]), 0.859 (95% CI [0.856–0.861]), and 0.908 (95% CI [0.906–0.909]), respectively. Compared with three pathologists of varying experience (junior, medium, and senior), the model’s performance was almost equivalent to that of a medium-level pathologist ( Figure 2A ). The confusion matrix shows specific performance metrics of the segmentation model. Among tumor region samples, the model correctly identified 86% as tumor, while misclassifying 14% as non-tumor.
Caption: Figure 2: Performance of the segmentation network.
(A) Receiver operating characteristic (ROC) curve comparing the performance of the segmentation network with three pathologists (senior, medium, and junior) in identifying tumor areas. (B) Confusion matrix for the segmentation network. (C) Original WSIs and annotation maps identified by the segmentation network. Left: original WSI images; right: corresponding annotation maps.
Claim: Venn diagram showed that the OTU numbers of NEMM cultured bacteria under anaerobic and normoxia conditions were higher than that in SHI cultured bacteria under the same conditions ( Figure 5C ).
Context: To compare NEMM with SHI, which are suitable for cultivating specific bacterial types, we further analyzed 16S rRNA gene high-throughput sequencing data. Alpha diversity indices of ACE and Chao1 showed no significant difference between NEMM and SHI under the normoxia condition. However, under anaerobic culture conditions, NEMM had greater alpha diversity indices of ACE and Chao1 than SHI ( Figure 5A ). NMDS analysis showed that regardless of whether anaerobic or normoxia, cultured bacteria samples had similar features ( Figure 5B ).
Caption: Figure 5: General bacterial community structure analysis between NEMM and SHI media during aerobic (normoxia) and anaerobic culture.
(A) Alpha diversity was measured with ACE index and Chao1 index. P < 0.05 was considered statistically significant. (B) NMDS was measured with Unweighted UniFrac. Each sample is represented by a dot. (C) Venn diagram displays the number of common and unique characteristics among the groups.
Claim: Observation of the results show that social causal reasoning tasks scored higher on BERTScore while physical causal reasoning tasks scored higher on SentenceBERT .
Context: We examine the difference in performance between physical and social causal reasoning ( Table 5 ). A task necessitates physical causal reasoning when the video changes stem from external, tangible forces, like a wave crashing on a beach. Conversely, a task involves social causal reasoning when video changes result from social cues, such as a character becoming joyful during a surprise party.
Caption: Table 5: GPT-4 performance for videos partitioned by the physical (changes due to outside, real forces) and social (changes related to social cues) on the Infilling-1 task with structured, FAMOuS descriptions. Significant differences between partitions are underlined .
Claim: Table 6 shows the average F1 scores on each task, demonstrating that lexicon-based pretraining boosts the performance of vanilla mBERT {}_{\text{Base}} .
Context: For each task, we take two datasets and perform experiments in few-shot training using mBERT {}_{\text{Base}} , following the setup described in Section 4.2 . Instead of using the multilingual lexicon, we use the English NRC-VAD lexicon since all the data is in English. For detailed information about the datasets and results, see the Appendix.
Caption: Table 6: Lexicon-based pretraining performance (macro-F1) over stance detection, hate speech detection, and emotion classification. The results are based on the limited training data scenario.