Ablations & Discussion
To better understand the gaps observed during training of the regression model, we conduct further ablations on a subset of seven languages including Arabic (ar), Bengali (bn), Finnish (fi), Japanese (ja), Korean (ko), Russian (ru) and Telugu (te):

Regression model choice. We compare several learning to rank models as choices for learning the Bradley-Terry model coefficients. We conduct our experiments on the train set, where models contain pairwise judgments, and on a randomly sampled holdout set, a realistic scenario, with no available training data. We evaluate Random Forest, Linear Regression, MLP Regressor, XGB Regressor, and SVR. All learning to rank models are implemented via scikit-learn . 9 9 9 https://scikit-learn.org/stable/supervised_learning.html The results are depicted in Table 3 . Random forest achieves the best R^{2} value on the holdout set for 4/7 languages. SVR also achieves a R^{2} value on the holdout subset, however, underperforms random forest on the training dataset. Other baselines, such as XGB Regressor and MLP Regressor show signs of overfitting on the training subset, thereby underperforming random forest on the holdout subset in \mirage .

Non-exhaustive pairwise comparisons lead to performance degradation. Exhaustive pairwise comparisons across a subset of 19 models in \mirage using GPT-4o as a teacher for “all” queries is quite expensive. To avoid this, we investigate whether really (all) pairwise exhaustive comparisons are necessary. We utilize two sampling techniques: (i) full pairwise judgments on a subsample of 100 queries, e.g., 20 or 50 queries; (ii) partially judge a non-exhaustive random sample of the pairwise judgments across 100 queries, e.g., only 50% of all the exhaustive pairwise combinations. Both results are shown in Figure 6 . we observe that Kendall-Tau ( \tau ) correlations increase linearly with queries and pairwise judgments. In summary, an exhaustive pairwise comparison and a sufficient amount of queries, such as 100, is necessary without impacting the leaderboard rankings.

Are all heuristic features necessary? We experiment with the set of features used for learning to rank model training as a surrogate judge. We evaluate four training configurations: (i) all features (ii) without LLM-measured features (iii) without language detection and support, i.e., the low-correlation features observed in Figure 5 , and (iv) including only LLM-measured features. From Table 4 , we observe that removing low-correlated heuristic RAG features, actually helps the learning to rank model to learn better leading to a conclusion that not necessarily all features are important.
Next, removing the LLM-measured features completely or only keeping them decreases the Kendall-Tau ( \tau ) correlation score in \mirage .

Does fine-tuning on \mirage training data help? We evaluate three variants of the \mirage training dataset using two backbones: Mistral-v0.2 (7B) and Llama-3 (8B). We fine-tune the \mirage training datasets using (i) both on GPT-4o, (ii) Llama-3 (8B) on Llama-3 (70B), and (iii) Mistral-v0.2 (7B) on Mixtral (8x22B). From Figure 7 , we observe that GPT-4o is a strong teacher, Mistral-v0.2 (7B) fine-tuned on GPT-4o distilled training data achieves the rank 2 outperforming the Llama-3 (70B) model. This shows \mirage training data is useful to improve the generation task in \mirage .