RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Claim: Command R (35B) performs poorly in low-resource languages such as Bengali (rank 13) or Swahili (rank 14).
Context: Figure 4 (left) shows the arena-based leaderboard using bootstrapping and Bradley-Terry modeling after conducting 200 tournaments and sampling 100 matches per tournament on a subset of 100 queries using GPT-4o pairwise comparisons. We observe that proprietary models such as GPT-4o and GPT-4, and larger models such as LLAMA-3 (70B) and Mixtral (8x22B), are slightly better than other baselines on \mirage . Baseline rankings across languages are usually stable; with a few notable exceptions such as Gemma-1.1 (7B) which achieves a rank of 4 in Telugu.
Caption: Figure 4: \mirage arena-based leaderboards: (left) Bradley-Terry model coefficients with rankings with GPT-4o as a judge pairwise judgments on a subset of 100 sampled queries. (right) Synthetic rankings using heuristic-based features and a learning to rank model. Each highlighted value is the rank of the model on the language (the lower the better). Models have been sorted by lowest to highest average rank across all 18 languages.