RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: As evident from the results in Table 5 , GPT-NeoX with a history length of 100 shows comparable performance to supervised learning approaches, without any fine-tuning on any TKG training dataset.
Context: We present a comparative analysis of the top-performing ICL model against established supervised learning methodologies for TKG reasoning, which are mostly based on graph representation learning.
Caption: Table 5: Performance (Hits@K) comparison between supervised models and ICL for single-step (top) and multi-step (bottom) prediction.
The first group in each table consists of supervised models, whereas the second group consists of ICL models, i.e., GPT-NeoX with a history length of 100.
The best model for each dataset in the first group is shown in bold , and the second best is underlined .
Claim: Loss method outperformed all others in approximately one-third of the 35 settings examined, as indicated in Figure 5 (left).
Context: Effective subsets have high diversity and predictive uncertainty . Our analysis reveals statistically significant differences between the subset selection strategies, highlighting the effectiveness of the hybrid approaches ( UMT/EMT+ Loss ) that consider both diversity and predictive uncertainty. Among the strategies tested, the UMT+
Caption: Figure 5: Summary of subset selection strategies performances from. Left: percentage of times each strategy gets the best performance out of 35 settings (across each of the 7 languages and 5 \hat{D}^{Syn}_{train} sizes). Right: bootstrapped confidence intervals for the percentages on the left.
Claim: Table 2 shows that the adversary applies lower dropout probabilities to less informative words such as ‘unknown’ tokens that replace all out-of-dictionary words, and so offer little information about the next word.
Context: We obtain further insight into what the adversary learns by analyzing the word dropout scores for different sentences.
Caption: Table 2 : Analysis of the Adversary. For selected SNLI (upper) and Yahoo (lower) sentences, word dropout scores are from a trained adversary and normalized per sentence. Darker colouring indicates a higher dropout probability. Boxed words are those selected to be dropped. <sos> = start of the sequence token; <eos> = end of the sequence token; _unk = unknown token.
Claim: On the Wiki-SQL dataset the base model is unable to infer the correct answer almost 80% of the time.
Context: The results are summarised in Table 1 .
Compared to the base model, Raven significantly improves the results on the PhraseBank dataset by an absolute 25.9%.
Caption: Table 1: The data statistics and experimental results (Exact Match) of various benchmarks and models. The best results are in bold . GPT-3.5 results are based on 5-shots. Sota is based on previously published results.
Claim: Figure 3 shows that across different noise levels, EIGNN consistently outperforms IGNN.
Context: On synthetic chain datasets, we add random perturbations to node features as in Wu et al. [ 31 ] .
Specifically, we add uniform noise \epsilon\sim\mathcal{U}(-\alpha,\alpha) to each node’s features for constructing noisy datasets.
Caption: Figure 3 : Accuracy on datasets with different feature noise.