RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: In contrast, models such as Qwen-2 or Gemma-1.1 tend to under-cite in their response.
Evidence type: figure
Caption: Figure 3: Lollipop plots denoting the average heuristic-based feature scores achieved by baselines in \mirage . x -axis denotes the languages in \mirage. whereas y -axis plots every heuristic feature value. models in the same family are represented as the same color in a lollipop (as multiple circles). Figure 9 provides lollipop plots for all eleven heuristic-based features.
Context: Figure 3 shows lollipop plots indicating the average heuristic-feature value ( y -axis) distribution across all languages ( x -axis). In English detection, smaller LLMs such as Gemma-1.1 (2B) do not generate output in the required target language, but rather rely on English. Next, for citation quality and support evaluation, OpenAI models achieve better Recall @10 and Entailment scores (except Llama-3 (70B) for a few languages), indicating baseline responses include grounded citations from relevant passages.
Verdict: Supported
Claim: We observe that proprietary models such as GPT-4o and GPT-4, and larger models such as LLAMA-3 (70B) and Mixtral (8x22B), are slightly better than other baselines on \mirage .
Evidence type: figure
Caption: Figure 4: \mirage arena-based leaderboards: (left) Bradley-Terry model coefficients with rankings with GPT-4o as a judge pairwise judgments on a subset of 100 sampled queries. (right) Synthetic rankings using heuristic-based features and a learning to rank model. Each highlighted value is the rank of the model on the language (the lower the better). Models have been sorted by lowest to highest average rank across all 18 languages.
Context: Figure 4 (left) shows the arena-based leaderboard using bootstrapping and Bradley-Terry modeling after conducting 200 tournaments and sampling 100 matches per tournament on a subset of 100 queries using GPT-4o pairwise comparisons.
Verdict: Supported
Claim: Baseline rankings across languages are usually stable; with a few notable exceptions such as Gemma-1.1 (7B) which achieves a rank of 4 in Telugu.
Evidence type: figure
Caption: Figure 4: \mirage arena-based leaderboards: (left) Bradley-Terry model coefficients with rankings with GPT-4o as a judge pairwise judgments on a subset of 100 sampled queries. (right) Synthetic rankings using heuristic-based features and a learning to rank model. Each highlighted value is the rank of the model on the language (the lower the better). Models have been sorted by lowest to highest average rank across all 18 languages.
Context: Figure 4 (left) shows the arena-based leaderboard using bootstrapping and Bradley-Terry modeling after conducting 200 tournaments and sampling 100 matches per tournament on a subset of 100 queries using GPT-4o pairwise comparisons. We observe that proprietary models such as GPT-4o and GPT-4, and larger models such as LLAMA-3 (70B) and Mixtral (8x22B), are slightly better than other baselines on \mirage .
Verdict: Supported
Claim: Command R (35B) performs poorly in low-resource languages such as Bengali (rank 13) or Swahili (rank 14).
Evidence type: figure
Caption: Figure 4: \mirage arena-based leaderboards: (left) Bradley-Terry model coefficients with rankings with GPT-4o as a judge pairwise judgments on a subset of 100 sampled queries. (right) Synthetic rankings using heuristic-based features and a learning to rank model. Each highlighted value is the rank of the model on the language (the lower the better). Models have been sorted by lowest to highest average rank across all 18 languages.
Context: Figure 4 (left) shows the arena-based leaderboard using bootstrapping and Bradley-Terry modeling after conducting 200 tournaments and sampling 100 matches per tournament on a subset of 100 queries using GPT-4o pairwise comparisons. We observe that proprietary models such as GPT-4o and GPT-4, and larger models such as LLAMA-3 (70B) and Mixtral (8x22B), are slightly better than other baselines on \mirage . Baseline rankings across languages are usually stable; with a few notable exceptions such as Gemma-1.1 (7B) which achieves a rank of 4 in Telugu.
Verdict: Supported
Claim: From Figure 7 , we observe that GPT-4o is a strong teacher, Mistral-v0.2 (7B) fine-tuned on GPT-4o distilled training data achieves the rank 2 outperforming the Llama-3 (70B) model.
Evidence type: figure
Caption: Figure 7: Approximate rankings using heuristic features after fine-tuning Llama-3 (8B) and Mistral-v0.2 (7B) on \mirage dataset across four configurations.
Context: Does fine-tuning on \mirage training data help? We evaluate three variants of the \mirage training dataset using two backbones: Mistral-v0.2 (7B) and Llama-3 (8B). We fine-tune the \mirage training datasets using (i) both on GPT-4o, (ii) Llama-3 (8B) on Llama-3 (70B), and (iii) Mistral-v0.2 (7B) on Mixtral (8x22B).
Verdict: Supported