alabnii/sciclaimeval-shared-task huggingface dataset:
Claim: Both results are shown in Figure 6 . we observe that Kendall-Tau ( \tau ) correlations increase linearly with queries and pairwise judgments.
Evidence type: figure
Evidence caption: Figure 6: Sampling experiments to reduce computation cost. (left) reduces the number of queries whereas (right) reduces the pairwise judgments.
Context: Non-exhaustive pairwise comparisons lead to performance degradation. Exhaustive pairwise comparisons across a subset of 19 models in \mirage using GPT-4o as a teacher for “all” queries is quite expensive. To avoid this, we investigate whether really (all) pairwise exhaustive comparisons are necessary. We utilize two sampling techniques: (i) full pairwise judgments on a subsample of 100 queries, e.g., 20 or 50 queries; (ii) partially judge a non-exhaustive random sample of the pairwise judgments across 100 queries, e.g., only 50% of all the exhaustive pairwise combinations.
Verdict: Supported
Claim: In summary, an exhaustive pairwise comparison and a sufficient amount of queries, such as 100, is necessary without impacting the leaderboard rankings.
Evidence type: figure
Evidence caption: Figure 6: Sampling experiments to reduce computation cost. (left) reduces the number of queries whereas (right) reduces the pairwise judgments.
Context: Non-exhaustive pairwise comparisons lead to performance degradation. Exhaustive pairwise comparisons across a subset of 19 models in \mirage using GPT-4o as a teacher for “all” queries is quite expensive. To avoid this, we investigate whether really (all) pairwise exhaustive comparisons are necessary. We utilize two sampling techniques: (i) full pairwise judgments on a subsample of 100 queries, e.g., 20 or 50 queries; (ii) partially judge a non-exhaustive random sample of the pairwise judgments across 100 queries, e.g., only 50% of all the exhaustive pairwise combinations. Both results are shown in Figure 6 . we observe that Kendall-Tau ( \tau ) correlations increase linearly with queries and pairwise judgments.
Verdict: Supported