RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Although a few of the source-only models have satisfactory performance on the target, using our ConDA framework, we achieve performance gains over the source-only model in almost all tasks (rows with positive \Delta F1 values).
Evidence type: table
Caption: Table 2: Performance of ConDA on unlabeled target domains. Source-only model for each task S \rightarrow T refers to zero-shot evaluation of a model trained on S and evaluated on test set of T. \Delta F1 is increase (or decrease, in a few cases) in F1 scores of the ConDA model over the source-only model. Avg. scores in bold indicate where ConDA out-performs the source-only model.
Context: To evaluate the performance of ConDA on each of the target domains, i.e. generators, we first look at how our model improves over a source-only model. Table 2 shows the results for this experiment, grouped by target domain. We report F1 scores for the ConDA framework and a source-only model, along with scores averaged over sources, for each target. The source-only model is a pre-trained RoBERTa (roberta-base) fine-tuned only on the source domain S. The source-only scores provide an estimate of how well a model trained just on the source transfers to the target domain.
Verdict: Supported
Claim: For targets GROVER_mega and GPT-2_xl, ConDA performs within 3 and 6 F1 points of the fully-supervised model.
Evidence type: table
Caption: Table 3: Performance of our ConDA model on each of the target domains, with each of the other domains as source. Numbers in bold are the best performing ConDA models for each target domain, i.e. closest to fully supervised performance.
Context: Next, we compare the performance of our model with a fully-supervised detector trained on the target domain in Table 3 . For ConDA, we show the test performance for all target-source pairs. For the supervised model, we use a pre-trained RoBERTa (roberta-base) fine-tuned on the target data. We then evaluate the model on the test set of the same target domain, and essentially this is our upper bound performance. ConDA achieves test performance comparable to fully-supervised models. In particular, for targets CTRL and XLM, ConDA (with GROVER_mega as source) achieves upper bound performance.
Verdict: Supported
Claim: Surprisingly, the OpenAI GPT-2 Detector performs poorly on the GPT-2_xl data from TuringBench, although it can be considered supervised for this particular target.
Evidence type: table
Caption: Table 4: Performance of ConDA in comparison to unsupervised baselines, as AUROC. For ConDA, we report the average AUROC over all sources (for each target) and also the maximum AUROC (across all sources), along with the corresponding source in parentheses. Bold shows superior performance across each target.
Context: We compare our ConDA framework with relevant unsupervised baselines and report results in Table 4 . Out of the four GLTR measures ( \log p(x) , Rank, Log Rank, and Entropy), the first three fare quite well for detecting CTRL-generated text, but performance on other generators is quite poor. DetectGPT, which is the most recent method we evaluate, performs poorly on almost all generators, with some satisfactory performance on CTRL and XLM.
Verdict: Supported
Claim: Finally, we see ConDA outperforms all the baselines in terms of maximum AUROC, and all but one in terms of average AUROC.
Evidence type: table
Caption: Table 4: Performance of ConDA in comparison to unsupervised baselines, as AUROC. For ConDA, we report the average AUROC over all sources (for each target) and also the maximum AUROC (across all sources), along with the corresponding source in parentheses. Bold shows superior performance across each target.
Context: We compare our ConDA framework with relevant unsupervised baselines and report results in Table 4 . Out of the four GLTR measures ( \log p(x) , Rank, Log Rank, and Entropy), the first three fare quite well for detecting CTRL-generated text, but performance on other generators is quite poor. DetectGPT, which is the most recent method we evaluate, performs poorly on almost all generators, with some satisfactory performance on CTRL and XLM. Surprisingly, the OpenAI GPT-2 Detector performs poorly on the GPT-2_xl data from TuringBench, although it can be considered supervised for this particular target.
Verdict: Supported
Claim: We see that the full model outperforms all the variants, implying that all three types of components are essential for detection performance in this problem setting.
Evidence type: table
Caption: Table 5: Comparison of different model variants; bold shows best performance. We randomly chose 3 target domains to show in this table due to space constraints.
Context: We evaluate variants of the ConDA model, by removing one component at a time and compare these in Table 5 . ConDA \CEs removes the two cross-entropy losses, i.e. no supervision even for the source. ConDA \contrast removes the contrastive loss components for both source and target. ConDA \MMD removes the MMD loss between source and target. Hence the only component that makes use of the unlabeled target domain data is the target contrastive loss. Finally, ConDA is the full model.
Verdict: Supported
Claim: Although we see satisfactory performance across most methods, our ConDA framework with source as FAIR_wmt19 and GPT2_xl has the best and the second best performance, respectively.
Evidence type: table
Caption: Table 6: Results on our ChatGPT News dataset using unsupervised baselines (upper row) and ConDA (lower row). Scores are AUROC. Bold shows best and underline shows second best performance.
Context: Given recent concerns surrounding OpenAI’s ChatGPT and GPT-4 OpenAI ( 2023 ) , it is important to create detectors for text generated by these conversational language models. With the incredible fluency and writing quality these language models possess, not only can such text easily fool humans Else ( 2023 ) but can also be extremely difficult for detectors to identify. Even OpenAI’s detector struggles to detect AI-generated text reliably 6 6 6 https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text . Hence in this case study, we are interested in evaluating our ConDA framework on ChatGPT-generated news articles, in an unsupervised manner. Since there is no existing dataset of ChatGPT-generated vs. human-written text or news, we create our own dataset as explained in Section 4.1 . We assign ChatGPT as the unlabeled target domain and assume that we have labeled data from the 6 other generators (Table 1 ). Therefore we emulate a real-world scenario where labeled data from older generators may be available, but it might be hard to find labeled samples for newer LLMs. We sample 4k articles from our ChatGPT News dataset and evaluate the same 3 unsupervised models as in Section 4.2 (upper row block in Table 6 ), and our ConDA framework over 6 source generators (lower row block in Table 6 ) on this data. For GLTR, we report the average over the 4 statistical measures.
Verdict: Supported
Claim: ReDial and INSPIRED contain 6,637 and 1,546 unique items in total ( Table 1 ) and 1,872 and 264 items in the test set, respectively.
Evidence type: table
Caption: Table 1: Statistics on ReDial and INSPIRED datasets, combined over train, dev and test sets.
Context: We follow the common practices Yang et al. ( 2022 ); Wang et al. ( 2022c ) to evaluate PECRS on both recommendation performance and response generation quality. For recommendation subtask, we measure recall with Recall@K (R@K) metric, taking K\in\{1,10,50\} . In order to assess the recommendation coverage, we also report the number of different items predicted by the model over the test set, denoted as Unique .
Verdict: Supported
Claim: Both PECRS-small and -medium surpass all baselines over Dist@3 and Dist@4.
Evidence type: table
Caption: Table 3: Results of conversation task compared with the state-of-the-art on ReDial.
Context: Table 3 summarizes the results on conversation task, where PECRS achieves promising performance on both types of metrics.
Verdict: Supported
Claim: Comparing PECRS-small and -medium shows that Dist@K improvements can be achieved by scaling up the backbone model.
Evidence type: table
Caption: Table 3: Results of conversation task compared with the state-of-the-art on ReDial.
Context: Table 3 summarizes the results on conversation task, where PECRS achieves promising performance on both types of metrics. Both PECRS-small and -medium surpass all baselines over Dist@3 and Dist@4.
Verdict: Supported
Claim: Moreover, PECRS without training for generation can achieve 11.907 on Dist@2 metric (see w/o Generation loss in Table 5 ), but merely 7.76 on RG-1 metric.
Evidence type: table
Caption: Table 5: Models comparison with different modules and optimization strategies on ReDial with PECRS-small.
Context: We first study the effects of different LM’s decoding strategies on conversational performance over Dist@K metric. Specifically, we analyze the greedy decoding, beam search, diverse beam search Vijayakumar et al. ( 2018 ) , top-k sampling Fan et al. ( 2018 ) and nucleus sampling Holtzman et al. ( 2020 ) strategies on PECRS-small. Reported in Table 8 , reference-based metrics (RG-K) show much less variance on different decoding strategies compared to the reference-free metrics (Dist@K). Meanwhile, the correlation between reference-based and reference-free metrics is weak under different decoding strategies.
Verdict: Supported
Claim: On the Wiki-SQL dataset the base model is unable to infer the correct answer almost 80% of the time.
Evidence type: table
Caption: Table 1: The data statistics and experimental results (Exact Match) of various benchmarks and models. The best results are in bold . GPT-3.5 results are based on 5-shots. Sota is based on previously published results.
Context: The results are summarised in Table 1 .
Compared to the base model, Raven significantly improves the results on the PhraseBank dataset by an absolute 25.9%.
Verdict: Supported
Claim: This figure is inverted
for Raven which obtains a 4-fold improvement over the base model inferring the correct answer more than 85% of the time.
Evidence type: table
Caption: Table 1: The data statistics and experimental results (Exact Match) of various benchmarks and models. The best results are in bold . GPT-3.5 results are based on 5-shots. Sota is based on previously published results.
Context: The results are summarised in Table 1 .
Compared to the base model, Raven significantly improves the results on the PhraseBank dataset by an absolute 25.9%. On the Wiki-SQL dataset the base model is unable to infer the correct answer almost 80% of the time.
Verdict: Supported
Claim: Table 2 shows that comparative assessment yields highly impressive performance for long-spoken summarization, with comparative assessment out-competing all other baselines.
Evidence type: table
Caption: Table 2: Spearman correlation coefficient for Podcast .
Context: Podcast Assessment : When considering podcast summarization with long inputs of over 5k tokens on average, only Llama2 models (which have a limit of 4k tokens) were used (as FlanT5 has a limit of 1k tokens).
Verdict: Supported
Claim: However, we observe considerably high bias, with some set-ups even selecting the first option 80% of the time.
Evidence type: table
Caption: Table 5: Positional bias P(A) for both prompt templates, for various systems in the comparative setup on SummEval.
Context: We investigate whether the comparative prompts have any implicit positional bias, and whether systems prefer the first/second position. Table 5 shows the fraction of comparisons that selected the candidate in the first position for SummEval. Since all comparisons in both permutations are considered, this fraction should be 0.50 for an unbiased system.
Verdict: Supported
Claim: We observe accuracies between 60-80% across all tasks and observe that debiasing can substantially increase accuracy.
Evidence type: table
Caption: Table 7: Accuracy of the comparative systems, at a comparison level, for SummEval.
Context: One can also measure the accuracy of the comparative system at a comparison level. Table 7 shows the pairwise comparison accuracy for Summeval, over all candidate pairs where the true score of the candidate response varies.
Verdict: Supported
Claim: Table 8 illustrates the self-consistency measured by the accuracy when comparing pairs, and demonstrates that even when using few outputs, the model is very consistent to the final rankings that would be achieved by using many more examples.
Evidence type: table
Caption: Table 8: Accuracy when using fewer systems with respect to final rankings (using all 16 systems) and the ground truth labels. Results shown for Summeval COH using FlanT5-xl.
Context: SummEval has 16 summaries per context which leads to 240 possible comparisons. If one were to instead randomly sample N outputs and consider all N\!\cdot\!(N\!-\!1) comparisons, how consistent would the rankings with the subset of systems be with respect to the final predicted rankings?
Verdict: Supported
Claim: The BERT multi-task classifier improves these numbers considerably (e.g., obtaining 59.69 and 59.60 micro F1 for CF and Skill, respectively), indicating the benefit of richer contextual representations.
Evidence type: table
Caption: Table 1: Three-fold cross-validation micro and macro F1 scores.
Context: Table 1 reports the micro and macro F1 scores for various models on the CF, IC, and skill classification. We compare several baselines, including traditional TF-IDF and BERT multi-task classifiers, graph-based models without BERT, and our proposed CFiCS variants that integrate clinical BERT features. The TF-IDF multi-task Random Forest baseline achieves relatively modest performance, with micro F1 scores of 52.50, 74.59, and 53.02 for CF, IC, and Skill, respectively, and corresponding macro F1 scores of 20.43, 38.04, and 49.77.
Verdict: Supported
Claim: GPT-4+DIN-SQL obtain EX score of 6.73% on Archer test set, while it is able to achieve 85.3% test-suite execution performance on Spider test set Pourreza and Rafiei ( 2024 ) .
Evidence type: table
Caption: Table 2: Baseline performance on Archer. GPT-4+DIN-SQL was tested only on the English set due to cost and its English-specific design. We only report the fine-tuned model’s performance on the test set.
Verdict: Supported
Claim: Among the three kinds of prompts, CT-3 achieves the highest EX scores on both English data (EX: 13.34%) and Chinese data (EX: 12.86%).
Evidence type: table
Caption: Table 2: Baseline performance on Archer. GPT-4+DIN-SQL was tested only on the English set due to cost and its English-specific design. We only report the fine-tuned model’s performance on the test set.
Context: GPT-4+DIN-SQL obtain EX score of 6.73% on Archer test set, while it is able to achieve 85.3% test-suite execution performance on Spider test set Pourreza and Rafiei ( 2024 ) . To evaluate the overall difficulty of Archer, we test the zero-shot performance of GPT-3.5 with API Doc, CT-3, CT-3+COT prompts on the full Archer data.
Verdict: Supported
Claim: Specifically, the T5-3B model trained on the augmented training set achieved an EX score of 4.81% on the English test (matching the performance of GPT-3.5+CT-3+COT) set and 1.92% on the Chinese test set (matching the performance of GPT-3.5+CT-3).
Evidence type: table
Caption: Table 2: Baseline performance on Archer. GPT-4+DIN-SQL was tested only on the English set due to cost and its English-specific design. We only report the fine-tuned model’s performance on the test set.
Context: From Table 2 . we observe that T5 from scale base to 3B (XL) trained on Archer training set achieve 0.00% EX scores. This outcome could be attributed to the small-scale nature of Archer combined with its high complexity. However, when Archer training set was augmented with the Spider/CSpider training set, the VA scores of T5 models exhibited a substantial improvement.
Verdict: Supported
Claim: Development sets are 5k instances samples from each set.
Evidence type: table
Caption: Table 1: Dataset statistics for synthetic data generated in this work. We omit the average length of answers for fact verification as it is a classification task. SQ=Single-Query. TQ=Two-Queries.
Context: We synthesize approximately 1.5 million multi-hop questions and 1.9 million claims. We use neucleus sampling (Holtzman et al., 2020 ) with a top- p probability of 0.9 for decoding when generating the data.
Verdict: Supported
Claim: STRUX outperforms strong baselines in accuracy and F-scores for stock investment decisions.
Evidence type: table
Caption: Table 1: Our STRUX system outperforms strong benchmarks in making stock investment decisions. We present macro-averaged precision, recall, F-scores, accuracy for the test set. LLMs evaluated are: Llama3-8b-Instruct and gpt-4o-mini-2024-07-18 .
Context: System Comparisons. Table 1 shows the macro-averaged precision, recall, F-scores, and accuracy for the test set.
Verdict: Supported
Claim: Statistics are presented in Table 3 . Each transcript is distilled into a table of about 40 facts, from which the model selects 9.
Evidence type: table
Caption: Table 3: Statistics of supporting facts.
Context: Supporting Facts. We analyzed the supporting facts identified by the model in cases of correct decisions after reflections.
Verdict: Supported
Claim: The generated dataset consists of 1,802 ambiguous and unanswerable questions spanning various categories.
Evidence type: table
Caption: Table 3: Dataset statistics and human annotation accuracy on 20 samples per question type. "#Ex" column shows the number of examples generated for each category. "Acc" column shows average binary classification accuracy from human expert.
Context: Table 3 shows the statistics of the dataset generated using the Spider dev set with Claude 3 sonnet. Note that the employed methodology can be seamlessly adapted to other text-to-SQL datasets like BIRD, WikiSQL, or any other synthetically generated answerable text-to-SQL corpora combined with any LLM (e.g., Llama3.1 or mixtral).
Verdict: Supported
Claim: Additionally, we collected 1,034 answerable queries from the Spider dev dataset and augmented them with natural language explanations derived from their execution results.
Evidence type: table
Caption: Table 3: Dataset statistics and human annotation accuracy on 20 samples per question type. "#Ex" column shows the number of examples generated for each category. "Acc" column shows average binary classification accuracy from human expert.
Context: Table 3 shows the statistics of the dataset generated using the Spider dev set with Claude 3 sonnet. Note that the employed methodology can be seamlessly adapted to other text-to-SQL datasets like BIRD, WikiSQL, or any other synthetically generated answerable text-to-SQL corpora combined with any LLM (e.g., Llama3.1 or mixtral). The generated dataset consists of 1,802 ambiguous and unanswerable questions spanning various categories.
Verdict: Supported
Claim: Sonnet achieve the highest average accuracy of 71.95% and 72.15% on the ambiguous/unanswerable questions.
Evidence type: table
Caption: Table 5: Execution accuracy of SQLs predicted with DIN-SQL using different LLMs on various categories of ambiguous, unanswerable, and answerable questions. The "All" column shows the overall average accuracy across all categories, while the "Avg. Excluding Answerable" column shows the average accuracy excluding the answerable questions from the Spider dataset.
Context: Table 5 shows our baseline method’s (DIN-SQL) performance on SQL prediction of various LLMs given the interaction between the user and the assistant. Overall, Mixtral-large-v2 and Claude 3.5
Verdict: Supported
Claim: Claude 3.5 sonnet achieves the highest performance of 79.21% on the answerable questions (original Spider dev set).
Evidence type: table
Caption: Table 5: Execution accuracy of SQLs predicted with DIN-SQL using different LLMs on various categories of ambiguous, unanswerable, and answerable questions. The "All" column shows the overall average accuracy across all categories, while the "Avg. Excluding Answerable" column shows the average accuracy excluding the answerable questions from the Spider dataset.
Context: Table 5 shows our baseline method’s (DIN-SQL) performance on SQL prediction of various LLMs given the interaction between the user and the assistant. Overall, Mixtral-large-v2 and Claude 3.5 Sonnet achieve the highest average accuracy of 71.95% and 72.15% on the ambiguous/unanswerable questions.
Verdict: Supported
Claim: The open-source model Llama-3.1 70B performs competitively on the answerable questions achieving 76.31% accuracy, only 2.9% lower than Claude 3.5 sonnet.
Evidence type: table
Caption: Table 5: Execution accuracy of SQLs predicted with DIN-SQL using different LLMs on various categories of ambiguous, unanswerable, and answerable questions. The "All" column shows the overall average accuracy across all categories, while the "Avg. Excluding Answerable" column shows the average accuracy excluding the answerable questions from the Spider dataset.
Context: Table 5 shows our baseline method’s (DIN-SQL) performance on SQL prediction of various LLMs given the interaction between the user and the assistant. Overall, Mixtral-large-v2 and Claude 3.5 Sonnet achieve the highest average accuracy of 71.95% and 72.15% on the ambiguous/unanswerable questions. Claude 3.5 sonnet achieves the highest performance of 79.21% on the answerable questions (original Spider dev set).
Verdict: Supported
Claim: However, it performs only at 67.58% accuracy on ambiguous/unanswerable questions, lagging 3.7% behind Claude 3.5 sonnet.
Evidence type: table
Caption: Table 5: Execution accuracy of SQLs predicted with DIN-SQL using different LLMs on various categories of ambiguous, unanswerable, and answerable questions. The "All" column shows the overall average accuracy across all categories, while the "Avg. Excluding Answerable" column shows the average accuracy excluding the answerable questions from the Spider dataset.
Context: Table 5 shows our baseline method’s (DIN-SQL) performance on SQL prediction of various LLMs given the interaction between the user and the assistant. Overall, Mixtral-large-v2 and Claude 3.5 Sonnet achieve the highest average accuracy of 71.95% and 72.15% on the ambiguous/unanswerable questions. Claude 3.5 sonnet achieves the highest performance of 79.21% on the answerable questions (original Spider dev set). The open-source model Llama-3.1 70B performs competitively on the answerable questions achieving 76.31% accuracy, only 2.9% lower than Claude 3.5 sonnet.
Verdict: Supported
Claim: For Factuality, the annotators demonstrate moderate agreement (Krippendorff’s Alpha = 0.68), suggesting that the conversations are consistently viewed as highly factual, which implies that the SQL queries in our dataset are of high quality.
Evidence type: table
Caption: Table 4: Summary of Human Annotation Scores for Naturalness, Factuality, and Helpfulness.
Context: Table 4 shows the mean, standard deviation, and Krippendorff’s Alpha for inter-annotator agreement. The high mean scores close to 1 (1.15-1.5) and substantial agreement (Alpha 0.68-0.82) indicate high-quality, natural conversations with factual and helpful responses. For Naturalness, we observe that annotators have a substantial agreement (Krippendorff’s Alpha = 0.82), indicating that the conversations are generally perceived as natural and fluent.
Verdict: Supported