RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Figure 5: Boxplot with the feature importance value (averaged across 18 languages in \mirage ) observed by the learning to rank (random forest) model.; Boxplot with the feature importance value (averaged across 18 languages in \mirage ) observed by the learning to rank (random forest) model.
Figure 6: Sampling experiments to reduce computation cost. (left) reduces the number of queries whereas (right) reduces the pairwise judgments.; Sampling experiments to reduce computation cost. (left) reduces the number of queries whereas (right) reduces the pairwise judgments.
Figure 7: Approximate rankings using heuristic features after fine-tuning Llama-3 (8B) and Mistral-v0.2 (7B) on \mirage dataset across four configurations.; Approximate rankings using heuristic features after fine-tuning Llama-3 (8B) and Mistral-v0.2 (7B) on \mirage dataset across four configurations.
Figure 8: Approximate rankings using heuristic features including the newer models, Llama-3.1 Dubey et al. ( 2024 ) and Gemma-2 Team et al. ( 2024 ) on \mirage dataset across all 18 languages. Gemma-2 (27B) and Llama-3.1 (70B) achieve a strong rank of 4 and 5 respectively in the \mirage evaluation dataset.; Approximate rankings using heuristic features including the newer models, Llama-3.1 Dubey et al. ( 2024 ) and Gemma-2 Team et al. ( 2024 ) on \mirage dataset across all 18 languages. Gemma-2 (27B) and Llama-3.1 (70B) achieve a strong rank of 4 and 5 respectively in the \mirage evaluation dataset.
Figure 9: Lollipop plots denoting the average heuristic-based feature scores achieved by baselines in \mirage for all eleven heurisitc-based features. x -axis denotes the languages in \mirage ; whereas y -axis plots every heuristic feature value. Multiple LLMs in the same family are represented as a single color lollipop (multiple circles).; Lollipop plots denoting the average heuristic-based feature scores achieved by baselines in \mirage for all eleven heurisitc-based features. x -axis denotes the languages in \mirage ; whereas y -axis plots every heuristic feature value. Multiple LLMs in the same family are represented as a single color lollipop (multiple circles).
Figure 10: Prompt template for all baseline models for multilingual RAG generation for queries across all languages in \mirage . We include the language-specific query in \mirage under “Question:”. Next, we concatenate both relevant and non-relevant passages (randomly shuffled and truncated at maximum length) and place them under “Contexts:”. Lastly, we provide our instruction in English asking the model to generate a response in the required language under the placeholder “{{language}}”. The example above is shown for a query in English (en) from \mirage , where contexts are truncated ( … ) for demonstration purposes.; Prompt template for all baseline models for multilingual RAG generation for queries across all languages in \mirage . We include the language-specific query in \mirage under “Question:”. Next, we concatenate both relevant and non-relevant passages (randomly shuffled and truncated at maximum length) and place them under “Contexts:”. Lastly, we provide our instruction in English asking the model to generate a response in the required language under the placeholder “{{language}}”. The example above is shown for a query in English (en) from \mirage , where contexts are truncated ( … ) for demonstration purposes.
Figure 11: Prompt template used by Llama-3 (8B) model as a judge to evaluate the answer overlap heuristic feature. We include a grading rubric within the prompt template.{{Label}} is a placeholder for the gold truth answer provided using the GPT-4; {{language}} is a placeholder for the target language; {{Question}} is a placeholder for the \mirage query; {{Documents}} is a placeholder for both \mirage relevant and non-relevant passages concatenated together; {{Response}} is a placeholder for RAG model output.; Prompt template used by Llama-3 (8B) model as a judge to evaluate the answer overlap heuristic feature. We include a grading rubric within the prompt template.{{Label}} is a placeholder for the gold truth answer provided using the GPT-4; {{language}} is a placeholder for the target language; {{Question}} is a placeholder for the \mirage query; {{Documents}} is a placeholder for both \mirage relevant and non-relevant passages concatenated together; {{Response}} is a placeholder for RAG model output.
Figure 12: Prompt template used by Llama-3 (8B) model as a judge to evaluate the fluency of a RAG response. We first explain the criteria for evaluation and the model outputs an explanation and score between [1,5] indicating the fluency of the output. {{language}} is a placeholder for the target language; {{Question}} is a placeholder for the \mirage query; {{Documents}} is a placeholder for both \mirage relevant and non-relevant passages concatenated together; {{Summary}} is a placeholder for RAG model output.; Prompt template used by Llama-3 (8B) model as a judge to evaluate the fluency of a RAG response. We first explain the criteria for evaluation and the model outputs an explanation and score between [1,5] indicating the fluency of the output. {{language}} is a placeholder for the target language; {{Question}} is a placeholder for the \mirage query; {{Documents}} is a placeholder for both \mirage relevant and non-relevant passages concatenated together; {{Summary}} is a placeholder for RAG model output.
Figure 13: Prompt template used by LLM as a judge to evaluate the RAG response in a pairwise evaluation involving a head-to-head battle. The template is taken and modified from RAGEval Rackauckas et al. ( 2024 ) . We explain the evaluation criteria and ask the judge to evaluate two RAG responses based on multiple factors, including correctness, helpfulness, completeness, accuracy, depth, and level of detail. The Judge provides a justification for their model choice and at the end of the response indicates as either “[[A]]”, “[[B]]”, or “[[C]]” denoting a tie. {{query}} is a placeholder for the input \mirage query; {{documents}} is a placeholder for both \mirage relevant and non-relevant passages concatenated together; {{answer_a}} is a placeholder for the output response of model A; {{answer_b}} is a placeholder for the output response of model B.; Prompt template used by LLM as a judge to evaluate the RAG response in a pairwise evaluation involving a head-to-head battle. The template is taken and modified from RAGEval Rackauckas et al. ( 2024 ) . We explain the evaluation criteria and ask the judge to evaluate two RAG responses based on multiple factors, including correctness, helpfulness, completeness, accuracy, depth, and level of detail. The Judge provides a justification for their model choice and at the end of the response indicates as either “[[A]]”, “[[B]]”, or “[[C]]” denoting a tie. {{query}} is a placeholder for the input \mirage query; {{documents}} is a placeholder for both \mirage relevant and non-relevant passages concatenated together; {{answer_a}} is a placeholder for the output response of model A; {{answer_b}} is a placeholder for the output response of model B.
Dataset statistics in \mirage ; All tokens are calculated using the GPT-4o tokenizer OpenAI ( 2023 ) ; (Rel.) denotes relevancy; (# Avg GPT-4o) counts the tokens in the GPT-4o generated context and answer.
Kendall \tau rank correlation between GPT-4o as a judge pairwise Bradley-Terry model and arena-based ranking leaderboard using heuristics and trained regression model for all languages in \mirage .
Train and Holdout R^{2} scores using different regression models. Each experiment has been repeated 50 times with four held-out models.
Kendall Tau ( \tau ) scores using different features for training the random forest regression model.
\bar{R}^{2} mean scores with 95% confidence interval with bootstrapping across all languages in \mirage .
Hyperparameter settings set during supervised fine-tuning of Mistral-v0.2 (7B) and Llama-3 (8B) on the \mirage training dataset.
Bradley-Terry logits using GPT-4o as a judge for all languages in \mirage . Scores are computed using the Bradley-Terry model with 200 tournaments using a maximum of 100 randomly sampled queries. Mean scores and 95% confidence intervals are reported (repeated 200 times). A higher logit score indicates a better performance, therefore achieving a higher rank on \mirage . Models are sorted in descending order of mean score for each language in \mirage .
Bradley-Terry logits using GPT-4o as a judge for all languages in \mirage . Scores are computed using the Bradley-Terry model with 200 tournaments using a maximum of 100 randomly sampled queries. Mean scores and 95% confidence intervals are reported (repeated 200 times). A higher logit score indicates a better performance, therefore achieving a higher rank on \mirage . Models are sorted in descending order of mean score for every language in \mirage .