Venue: NAACL 2025

# MILU: A Multi-task Indic Language Understanding Benchmark

Sshubam Verma\({}^{1}\) Mohammad Safi Ur Rahman Khan\({}^{1,2}\)

Vishwajeet Kumar\({}^{3}\) Rudra Murthy\({}^{3}\) Jaydeep Sen\({}^{3}\)

\({}^{1}\)Nilekani Centre at AI4Bharat \({}^{2}\)Indian Institute of Technology, Madras

\({}^{3}\)IBM Research, India

###### Abstract

Evaluating Large Language Models (LLMs) in low-resource and linguistically diverse languages remains a significant challenge in NLP, particularly for languages using non-Latin scripts like those spoken in India. Existing benchmarks predominantly focus on English, leaving substantial gaps in assessing LLM capabilities in these languages. We introduce MILU--**MU**ulti-task **I**ndic **L**anguage **O**nderstanding Benchmark--a comprehensive evaluation benchmark designed to address this gap. MILU spans 8 domains and 41 subjects across 11 Indic languages, reflecting both general and culturally specific knowledge. With an India-centric design, MILU incorporates material from regional and state-level examinations, covering topics such as local history, arts, festivals, and laws, alongside standard subjects like science. We evaluate over 42 LLMs, and find that current LLMs struggle with MILU, with GPT-4o achieving the highest average accuracy at 74%. Open multilingual models outperform language-specific fine-tuned models, which perform only slightly better than random baselines. Models also perform better in high-resource languages as compared to low-resource ones. Domain-wise analysis indicates that models perform poorly in culturally relevant areas like Arts & Humanities and Law & Governance compared to general fields like STEM. To the best of our knowledge, MILU is the first of its kind benchmark focused on Indic languages, serving as a crucial step towards comprehensive cultural evaluation. All code, benchmarks, and artifacts will be made publicly available to foster open research.

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 10076–10132

April 29 - May 4, 2025 ©2025 Association for Computational Linguistics

## 1 Introduction

Recent advancements in Large Language Models (LLMs) have reshaped the field of NLP, by enabling these models to perform a variety of tasks across diverse domains Doddapaneni et al. (2023); OpnAI et al. (2023); Team et al. (2024); Anthropic). While many LLMs now claim to support multiple languages, there is still a huge discrepancy in their performance in English and other languages Liu et al. (2024). Particularly languages using non-Latin scripts, such as those in India, are affected the most by this discrepancy Ahuja et al. (2023). One key reason for this is the absence of high-quality benchmarks for these languages. Well-designed benchmarks are crucial in driving model development by revealing limitations and guiding improvements McIntosh et al. (2024). However, most existing benchmarks focus primarily on English, _leaving significant gaps in evaluating LLMs capability in low-resource and linguistically diverse languages_.

India's diversity presents a unique challenge for these models. With over 1.4 billion people speaking more than 120 languages and around 19,500 dialects across 28 states Javed et al. (2024), many

Figure 1: Average performance of all the evaluated models on MILU. The closed models are shown in Orange, the open models are shown in Green, and the language-specific models are shown in Blue.



of which are underrepresented in NLP research, the need for not just linguistic but culturally appropriate benchmarks becomes urgent (Doddapaneni et al., 2023; Singh et al., 2024). Standard benchmarks like MMLU Hendrycks et al. (2021) and AGIEval Zhong et al. (2023), while useful for evaluating general world knowledge, falls short in capturing these intricacies of India. Each state in India has its own history, traditions, festivals, and art forms, forming a rich cultural mosaic. Translating existing English benchmarks into Indian languages fails to capture this knowledge, which is required for real-world applications. Given the increasing deployment of LLMs in tasks that directly impact local populations, _the need for a benchmark that evaluates both linguistic competence and cultural understanding has become more pressing than ever._

In this work, we introduce MILU-**M**ulti-task **I**ndic **L**anguage **C**nderstanding Benchmark, a comprehensive evaluation dataset designed to address these gaps. MILU spans 8 domains and 41 subjects across 11 Indic languages, reflecting both general and culturally specific knowledge. We designed MILU with an India-first perspective by collecting questions from various national, state, and regional exams. These questions include culturally relevant subjects such as local history, arts, festivals, and laws, alongside traditional academic subjects like science. Following previous efforts Hendrycks et al. (2021); Zhong et al. (2023), we create this benchmark by collecting questions from over 1500 competitive exams from India. We focus on region-specific exams to authentically capture local knowledge in the respective language.

We evaluate 45 different LLMs - a mix of closed proprietary, open-source, and language-specific models- on MILU. Our findings suggest that models struggle with MILU, with GPT-4o achieving the highest average accuracy at 74%. Interestingly, open multilingual models outperform language-specific models, which only achieve slightly better than random scores. Our analysis of in-context learning reveals that adding more examples improves performance in base models, but the effect on instruct models remains inconclusive. We also explore how performance scales with the number of parameters, finding significant improvements as model size increases. Our domain-wise analysis reveals that models perform poorly in culturally relevant areas, such as Arts & Humanities and Social Sciences, compared to more general fields like STEM. All the artifacts will be released publicly.

## 2 Related Works

Large Language Models (LLMs):Recent LLMs, both proprietary-GPT-4o, GPT-4o-mini, Claude-3, and Gemini Team et al. (2024a)-and open-source-the Llama-3 Dubey et al. (2024) and Gemma Team et al. (2024b) series-have demonstrated significant improvements across various tasks and benchmarks Chiang et al. (2024); Wang et al. (2024). While these models are primarily trained in English, many claim to include a reasonable amount of multilingual data in their pretraining corpus Team et al. (2024); Dubey et al. (2024); Aryabumi et al. (2024). Additionally, significant progress has been made in developing language-specific models Ustun et al. (2024); Nguyen et al. (2023), including for Indic languages Gala et al. (2024); Balachandran (2023); Kohli et al. (2023). Most of these models are built on top of stronger English base models by (optionally) either continually pretraining on smaller, language-specific datasets Zhao et al. (2024), or by language-specific instruction fine-tuning Ghosh et al. (2024). In this work, we conduct a comprehensive evaluation of the performance of these models on MILU.

LLM Evaluation Benchmarks:Over the years, various benchmarks have been developed to evaluate the performance of large language models (LLMs). Recent benchmarks such as MMLU Hendrycks et al. (2021), MMLUProWang et al. (2024), AGIEvalZhong et al. (2023), BigGenBenchKim et al. (2024), and HellaSwagZellers et al. (2019) assess these models across a wide range of tasks. However, these benchmarks primarily focus on English, and progress on multilingual benchmarks has been comparatively slower Doddapaneni et al. (2023); Kumar et al. (2022). Some popular multilingual benchmarks include OMGEVal Liu et al. (2024), XTREME Hu et al. (2020), and XQuAD Artetxe et al. (2020), though these are largely limited to simple natural language understanding (NLU) tasks. As a result, previous studies often rely on translations of popular English benchmarks to evaluate performance in other languages Dubey et al. (2024); Gala et al. (2024). This approach is suboptimal, as it fails to account for cultural nuances and concepts unique to specific languages.

Indic LLM Evaluation Benchmarks:The first major Indic language benchmarks, IndicGLUE Kakwani et al. (2020) and IndicNLIG 

Benchmark Kumar et al. (2022), cover 11 languages, focusing on various language understanding and generation tasks, respectively. IndicX-TREME Doddapaneni et al. (2023) extends these efforts to include all 22 scheduled Indian languages for NLU evaluation. More recently, IndicGenBench Singh et al. (2024b) offers a more comprehensive evaluation of multilingual language generation by consolidating multiple generation tasks. In addition, the Indic-QA BenchmarkSingh et al. (2024a) assesses context-based question-answering (QA) performance in 11 Indian languages, while L3Cube-IndicQuest Rohera et al. (2024) explores regional knowledge through a translation-based approach to data creation. More recently, Pariksha Watts et al. (2024) evaluated more than 25 Indic models and created dynamic leaderboards by collecting more than 90,000 human preferences. Other initiatives, such as Airavata Gala et al. (2024) and the Indic-LLM-Leaderboard Kolaivi (2024), focus on translating existing English benchmarks into Indian languages. In contrast, MILU addresses this limitation by centering on India-specific topics and questions, providing a more culturally relevant and context-aware evaluation.

## 3 MILU: The IndicMMLU Benchmark

In this section, we describe the collection process (SS3.1), the cleaning and filtering process (SS3.2), and present the analysis (SS3.3).

### Questions Curation

MILU is a large, multi-domain test set containing multiple-choice based questions (MCQs) taken from over 41 subjects with an emphasis on Indiaspecific knowledge. This benchmark covers many domains, including Science, Social Sciences, Humanities, Arts, Business Studies, and Law, among others. MILU is designed as a culturally relevant benchmark to assess general problem-solving abilities and India-specific knowledge. These questions were sourced following an approach similar to AGIEval Zhong et al. (2023), collecting the questions from various public exams taken by individuals intending to either pursue higher studies or seek career advancements, such as qualification tests and national and state-level civil services exams, among others.

We gathered exam-specific questions by scraping various online exam portals that offer previously released question papers from various exams in multiple different languages. These portals typically tag questions manually with topic names and language details, and subject experts ensure the accuracy of the answers. Our benchmark includes questions from over 40 different types of exams conducted both at the national and state levels over recent years. Regional state exams are particularly valuable as they cover various state-level topics and emphasize the official language of each state. Further details about the different exams included in our collection are provided in the Appendix A.

In total, we collected more than 150K questions across 11 Indian Languages- Bengali (_bn_), Gujarati (_gu_), Hindi (_hi_), Kannada (_kn_), Malayalam (_ml_), Marathi (_mr_), Odia (_or_), Punjabi (_pn_), Tamil (_ta_), Telugu (_te_), and English (_en_)-spanning 41 diverse subjects. English questions are also included as these often address Indian culture-specific content, which is notably missing from existing popular benchmarks. Table 1 presents a brief comparison of the type of questions present in MMLU Hendrycks et al. (2021) and MILU.

\begin{table}
\begin{tabular}{p{86.7pt} p{142.3pt}} \hline \multicolumn{1}{c}{**MMLU**} & \multicolumn{1}{c}{**MILU**} \\ \hline Which principle was established by the Supreme Court’s decision in My Mahrabarria government on 2dth Marburg v. Madison? & Which of the decisions was taken by Mahrabarria government on 2dth November 2001 but had had a farreaching impact on the education field? \\ \hline At breakfast, lunch, and dinner, Joe randomly chooses with equal probabilities either an apple, an orange, or a banana to eat on a given day, what is the probability that Joe will eat at least two different kinds of fruit? & Shyam’s monthly income is Rs, I200. He says Re. 1200. Find the percent of his savings and his expenditure. \\ \hline The Space Shuttle orbits 300 km above Earth’s surface; Earth’s radius is 6,400 km near Ohr. What is the gravitational acceleration experienced by the Space Shuttle? & FASTag at the Toll Gates uses which waves? \\ \hline The primary goal of the Grammar Radman Acts of 1985 and 1987 was to Leaderboard Kolavi (2024). & Under the Pradhan Mantri Kasan Sammin Nothi Scheme, the financial help provided to the small and marginal farmers is \\ \hline A guitar string creates a sound wave of known frequency. Which of the following describes a correct and practical method of measuring the wavelength of the sound wave with a metersick? \\ \end{tabular}
\end{table}
Table 1: Comparison of some question from MMLU Hendrycks et al. (2021) with similar questions from MILU. The Red highlights the global concepts covered in MMLU, while Green highlights the Indiancentric concepts covered in MILU.

 



### Data Cleaning and Filtering

Despite our best efforts to maintain the quality of questions collected, some amount of noise or errors may still be present. To address potential noise in the questions, we employ multiple layers of manual and automated cleaning filters. Initially, we manually review a large sample of questions to detect and eliminate potential sources of noise. During the collection process, we exclude any reading-comprehension-style questions, images-based questions, and those with more than four answer options to ensure uniformity and consistency. To remove incorrect language entries, we utilize a combination of IndicLID (Madhani et al., 2023) and Unicode-based filtering Khan et al. (2024), ensuring that the questions are in the correct language. To further refine the dataset, we remove any duplicate questions to retain only the unique ones. As a final step, we manually verify a sample of questions from each language to ensure accuracy and correct any remaining errors.

Upon examination, we found that approximately 45% of questions were accurately labeled with a topic name, while the remaining questions lacked this information. To address this issue, we first translate the untagged questions into English using IndicTrans2 (Gala et al., 2023) and then prompt GPT-4o-mini model to assign an appropriate topic name to the question. Finally, in total, we get around 20K tags. However, these tags are highly fine-grained, often having a heavy overlap. To organize them, we embed the tags using the NV-Embed-v2 (Lee et al., 2024) model and apply K-means clustering to group tags into 50 clusters. We manually review these clusters and assign appropriate subject labels. Following the manual merging of related clusters, we determine 41 distinct subject names, which fall into eight main domains: Arts and Humanities, Social Sciences, Environmental Sciences, Law and Governance, Health and Medicine, Science, Engineering and Technology, and Business Studies. The final distribution of domains within MILU is shown in Figure 2. Detailed analysis of question distribution by topic & language is provided in Appendix B.

Finally, we observed that some topics in certain languages had less than 100 questions. To ensure thorough evaluation across all subjects and languages, we aimed to have at least 100 questions per subject in each language. For subjects with insufficient questions, we sampled questions from the English set from that subject and translated them into the required language using GPT-4o. We chose GPT-4o over specialized translation models for their ability to remain task-aware during translation Ahuja et al. (2024), ensuring the translated content aligns with the intent of the question.

In total, we release around 79K questions across 41 subjects across 8 domains in 11 languages, capping each subject-language pair at 500 questions for feasible evaluations.

### Data Analysis

Table 2 shows the overall statistics of MILU. Of the total 79K questions, only 25% of questions are translated from English, with the remainder

\begin{table}
\begin{tabular}{l|c c c} \hline \hline _lang_ & **Total Qs** & \multicolumn{2}{c}{**Total Translated Qs**} & \multicolumn{1}{r}{**Avg Words Per Q**} \\ \hline _bn_ & 6638 & 1601 & 15.12 \\ _gu_ & 4827 & 2755 & 16.12 \\ _hi_ & 14837 & 115 & 20.61 \\ _kn_ & 6234 & 1522 & 12.42 \\ _ml_ & 4321 & 3354 & 12.39 \\ _mr_ & 6924 & 1235 & 18.76 \\ _or_ & 4525 & 3100 & 14.96 \\ _pa_ & 4099 & 3411 & 19.26 \\ _ta_ & 6372 & 1524 & 13.14 \\ _te_ & 7304 & 1298 & 15.71 \\ _en_ & 13536 & - & 22.07 \\ \hline _total_ & **79617** & **19915** & **16.41** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Overall statistics of MILU. Refer to Section (§3.3) for more details.

Figure 2: Distribution of the number of questions across different domains, averaged across all languages. Refer to Section (§3.3) for more details.



preserved in their original source languages. The average question length varies across languages, with languages such as _kn_, _ml_, and _ta_ having shorter word counts due to their agglutinative nature.

As shown in Figure 2, MILU consists of eight domains.

**Arts & Humanities:** This domain covers topics such as Indian art, literature, dance, festivals, and architecture. Its content varies significantly by language, as it is highly dependent on regional culture.

**Science:** Includes topics in physics, chemistry, biology, with references to Ayurveda, and ancient as well as modern scientific achievements.

**Health & Medicine:** Covers public health policies, modern healthcare, traditional Ayurveda and Unani medicine, and health-related government initiatives.

**Business Studies:** Focuses on entrepreneurship, trade, and economic policies such as Make in India. It also covers topics like taxation, MSMEs, and global trade.

**Law & Governance:** Includes topics on the constitution, governance, judiciary, public administration, fundamental rights, and various governmental schemes, including local governance policies.

**Environmental Sciences:** Covers topics related to biodiversity, environmental policies, and both local and national environmental initiatives.

**Social Sciences:** Explores topics such as history, geography, and politics. The content in this domain is region-specific and varies by language.

**Engineering & Technology:** Includes discussions on modern developments in India, such as IT, telecommunications, infrastructure, and space technology, with a focus on government policies.

Figure 2 shows the domain-wise statistics of MILU. We see a fairly large number of questions in the culturally relevant domains of Arts & Humanities and Social Sciences. Detailed statistics per language can be found in the Appendix B.

## 4 Experimental Setup

We evaluate 42 different models on MILU, including large proprietary models, open-source multilingual models, and popular fine-tuned models specific to Indic languages. Both the base versions and instruction fine-tuned variants of these models, wherever applicable, are evaluated to measure the improvements gained from fine-tuning. All models, except for proprietary models and Llama-3.1-405B, are tested under 0-shot, 1-shot, and 5-shot setups. We maintain a separate validation set of approximately 9,000 questions to serve as examples for few-shot evaluations.

For non-API-based models, we use the LM-Evaluation-Harness Gao et al. (2024); Biderman et al. (2024) to ensure clean and reproducible evaluations. We use the log-likelihood method, where the probability of a given output string is computed by conditioning it on some provided input Brown et al. (2020). Specifically, the log-likelihood of an answer \((a)\) given the question \((x)\), i.e., \(logP(a|x)\), is calculated by concatenating the answer \((a)\) with question \((x)\), and then summing up the log probabilities, of each target token. For multiple choice questions, given \(k\) possible answer strings, we select the answer string \((a_{i})\) with the highest conditional log probability, i.e., \(argmax(logP(a_{1}|x),...,logP(a_{k}|x))\).

The API-based models are evaluated using the generative approach due to the lack of support for prompt log probabilities. We explicitly prompt these models to generate the correct response in a structured JSON format to simplify response parsing. Due to the high costs involved, these models are evaluated only in the zero-shot setup.

## 5 Results and Discussions

In this section, we discuss the results and our findings across all the experiments conducted.

### How do models supporting all languages perform?

Table 3 shows the performance of various models supporting all the languages in the 5-shot setup. Among these, GPT-4o emerges as the strongest model, consistently outperforming all competitors across languages, with an average accuracy of 74.74%. Closed proprietary models, generally achieve higher performance than the open-source alternatives. Among the open-source models, Llama-3.1-70B shows the best performance.

Performance is notably higher in high-resource languages- _en_, _hi_, _bn_ -as compared to the low-resource ones. Interestingly, the models also show low accuracies in agglutinative languages, such as _ta_, _te_. Notably, models specifically trained for Indian languages such as Krutrim-spectre-v2 and sarvan-2b-v0.5, perform slightly below their English counterparts of comparable scale. Additionally, while several models achieve better results in English, their performance in other lan 

guages is often nearly at the random baseline.

### How do language-specific fine-tuned models perform?

We evaluate around 16 Indic language LLMs on MILU. These models are primarily built by adapting English LLMs, such as Llama-2-7B, by first continually pretraining on small amount Indic language data, followed by optionally instruction finetuning them. As seen from Table 4, across languages, the models exhibit average performance comparable to random baselines, with minimal variations among them.

Among the evaluated models, the AryAhatta-GemmaGenZ-Vikas model stands out, performing better than its counterparts. When analyzed across domains, the models generally perform worse in Arts, Humanities, and Social Sciences than in STEM subjects.

### How do the models respond to In-Context examples?

We compare the performance of different Base and Instruct models across zero, one, and five-shot setups. As shown in Figure 3, the performance of base models consistently improves with an increasing number of in-context examples, with the 5-shot setup yielding the best results. In contrast, Instruct models exhibit more varied behavior, where models either stagnate or even degrade in performance. This also aligns with expectations, as Instruct models are specifically fine-tuned as conversation assistants and may not respond well to the few-shot in-context examples format. This also correlates with results in Table 3 where most instruction finetuned variants performs worse on English itself. This essentially implies the inferior performance is less about the language shift but due to the change of tasks during instruction finetuning and inference

\begin{table}
\begin{tabular}{l|c c c c c c c c c c|c} \hline _Model_ & _bn_ & _en_ & _gu_ & _hi_ & _kn_ & _ml_ & _mr_ & _or_ & _pa_ & _tu_ & _te_ & _org_ \\ \hline  & \multicolumn{8}{c}{**Closed Models**} \\ \hline GPT-4o & 77.59 & 81.75 & 74.64 & 78.62 & 76.93 & 72.57 & 73.44 & 70.07 & 73.84 & 70.15 & 72.53 & 74.74 \\ GPT-4o-mini & 60.69 & 70.52 & 57.84 & 63.14 & 56.92 & 54.52 & 55.76 & 47.31 & 57.89 & 50.84 & 53.31 & 57.16 \\ Gemini-1.5-Pro & 68.8 & 76.91 & 69.68 & 70.63 & 69.68 & 65.5 & 65.63 & 62.21 & 65.37 & 66.03 & 64.47 & 67.72 \\ Gemini-1.5-Flash & 61.93 & 69.42 & 61.52 & 64.81 & 64.22 & 59.56 & 59.44 & 56.88 & 62.23 & 58.89 & 58.13 & 61.55 \\ Krutzin-spectre-v2 & 44.48 & 46.59 & 43.43 & 45.39 & 46.55 & 41.89 & 40.89 & 36.8 & 40.57 & 40.48 & 39.22 & 42.34 \\ \hline  & \multicolumn{8}{c}{**Open Multilingual Models**} \\ \hline meta-llama/Llama-3.2-IB & 25.37 & 34.35 & 25.46 & 27.01 & 25.51 & 25.27 & 26.5 & 25.02 & 24.52 & 25.56 & 25.79 & 26.4 \\ meta-llama/Llama-3.2-IB-Instruct & 25.17 & 25.84 & 25.19 & 25.24 & 25.91 & 25.02 & 25.82 & 24.4 & 24.74 & 26.8 & 26.11 & 25.48 \\ sarvanai/sarvan-2b-v0.5 & 29.83 & 34.73 & 29.5 & 31.37 & 29.03 & 25.94 & 28.96 & 27.09 & 29.06 & 27.68 & 28.57 & 29.25 \\ servamani/servam-1 & 33.31 & 30.3 & 29.83 & 33.05 & 33.29 & 30.8 & 27.9 & 29.33 & 28.76 & 27.64 & 28.5 & 30.25 \\ google/genma-2b & 29.72 & 50.86 & 28.82 & 32.32 & 29.77 & 28.72 & 30.99 & 25.28 & 30.64 & 30.1 & 28.46 & 31.43 \\google/genmna-2.2-it & 30.6 & 52.4 & 29.25 & 34.33 & 28.52 & 28.81 & 32.18 & 26.32 & 29.01 & 30.46 & 29.22 & 31.92 \\ meta-llama/Llama-3.2-3B & 32.75 & 50.25 & 31.32 & 34.63 & 31.75 & 29.07 & 32.41 & 28.07 & 30.42 & 30.93 & 29.75 & 32.85 \\ meta-llama/Llama-3.2-3B-Instrct & 32.9 & 31.74 & 32.82 & 30.55 & 31.83 & 30.36 & 28.54 & 26.48 & 32.35 & 31.32 & 31.19 & 30.92 \\ nvidia/Nemotron-4-Mini-Hindel-4B-Base & 31.41 & 51.37 & 25.85 & 51.42 & 27.4 & 27.01 & 39.37 & 26.9 & 26.47 & 27.89 & 28.25 & 33.03 \\ \hline  & \multicolumn{8}{c}{**Pho 27R Models**} \\ \hline Telugu-LLM-Labs/Navarasa-2.0 & 36.43 & 51.69 & 37.44 & 39.86 & 40.01 & 36.40 & 37.00 & 35.36 & 36.86 & 37.18 & 36.86 & 38.64 \\ neula/Page-7B & 42.06 & 56.41 & 36.81 & 42.21 & 33.83 & 33.86 & 36.73 & 32.18 & 36.45 & 32.72 & 36.43 & 38.15 \\ CohererFordAv/Java-23-8B & 27.85 & 45.76 & 26.93 & 36.11 & 27.53 & 28.4 & 30.96 & 27.51 & 26.98 & 28.88 & 26.34 & 30.31 \\ meta-llama/Llama-3.1-SB & 37.89 & 59.66 & 35.26 & 42.69 & 37.09 & 34.71 & 37.65 & 32.86 & 36.33 & 36.25 & 34.94 & 38.67 \\ meta-llama/Llama-3.1-SB-Instruct & 34.53 & 46.22 & 33.77 & 35.73 & 36.04 & 34.37 & 33.28 & 30.08 & 33.33 & 34.15 & 33.6 & 35.01 \\ googgle/genma-2-9b & 53.12 & 69.8 & 51.19 & 56.24 & 53.99 & 49.55 & 49.81 & 40.42 & 49.7 & 50.05 & 48.78 & 52.06 \\ google/genma-2-9b+ it & 46.04 & 63.61 & 38.24 & 48.23 & 43.47 & 40.8 & 37.62 & 34.19 & 40.42 & 38.83 & 39.44 & 42.55 \\ google/genma-2-2rb & 63.47 & 74.82 & 59.29 & 64.33 & 63.43 & 59.62 & 59.66 & 44.86 & 59.94 & 58.6 & 57.94 & 60.54 \\ google/genma-2-27b-it & 59.82 & 73.7 & 55.17 & 61.25 & 58.23 & 53.37 & 54.78 & 44.99 & 55.55 & 55.6 & 54.29 & 56.98 \\ \hline  & \multicolumn{8}{c}{\(>278\) Models} \\ \hline CohereForAI/Jaya-23.35B & 32.9 & 53.09 & 30.35 & 43.53 & 31.39 & 32.42 & 37.22 & 31.27 & 27.27 & 35 & 29.34 & 34.89 \\ meta-llama/Llama-3.1-70B & 67.37 & 74.59 & 62.42 & 66.97 & 68.37 & 62.69 & 62.2 & 57.48 & 64.6 & 60.7 & 60.64 & 64.37 \\ meta-llama/Llama-3.1-70B-Instruct & 62.77 & 54.00 & 60.26 & 58.90 & 63.02 & 58.94 & 53.71 & 52.81 & 60.67 & 58.41 & 57.66 & 58.29 \\ meta-llama/Llama-3.1-405B & 65.15 & 61.42 & 61.11 & 50.98 & 63.69 & 63.64 & 50.21 & 55 

task for MILUevaluation. Understandably, the instruction finetuned variants performs poor on the overall average across languages as well.

### How does the performance vary with scale?

We evaluate the Llama and Gemma family of models, ranging from 1B to 405B parameters, to analyze how performance scales with model size. Figure 5 shows that the model performance improves significantly with increasing scale. Notably, instruction-tuned models in the Llama family show more substantial improvements as compared to those in the Gemma family.

### How do the models perform on different domains?

We analyze the performance of various base and instruct models across multiple domains and languages. Similar trends to those in Section (SS5.2) are observed where the open models perform poorly in domains specific to Indian culture--such as Arts & Humanities, Social Sciences, and Law & Governance--but demonstrate higher performance in STEM fields. This suggests that the training corpora for these models lack sufficient culturally specific data. Bridging this gap requires a more inclusive data distribution that ensures equitable representation of all cultures and languages.

### Does language adaptation on models help?

As most Indic LLMs are built on English base models like Llama-2-7b, we assess the impact of language adaptation on their performance. Table 5 compares language-specific models with the original Llama-2-7b, and instruction-tuned models with Llama-2-7b-chat. Our findings show minimal gains, with some models even underperforming post-adaptation.

Given the varied training data, this is not a di

\begin{table}
\begin{tabular}{l|l|c c c c c c c c c} \hline _lang_ & **Model** & \multicolumn{3}{c}{**Business**} & \multicolumn{1}{c}{**Engg.**} & \multicolumn{1}{c}{**Social**} & \multicolumn{1}{c}{**Env.**} & \multicolumn{1}{c}{**Law \& Science**} & \multicolumn{1}{c}{**Arts \& \& Health \& Avg**} \\  & & **Studies** & \& **Tech** & **Sciences** & **Sciences** & **Governance** & \multicolumn{1}{c}{**Hummalities**} & \multicolumn{1}{c}{**Medicine**} & \multicolumn{1}{c}{**Avg**} \\ \hline \multirow{8}{*}{_hi_} & OpenHathi-7B-Hi & 29.72 & 28.59 & 28.81 & 27.74 & 30.96 & 28.53 & 28.28 & 30.76 & 29.17 \\  & Airvata & 28.03 & 28.45 & 28.92 & 27.14 & 27.59 & 28.95 & 27.54 & 32.77 & 28.67 \\  & ProjectIndus & 26.21 & 25.83 & 26.95 & 25.03 & 22.93 & 24.91 & 25.13 & 24.41 & 25.18 \\  & Grajendra-v0.1 & 29.30 & 28.36 & 25.54 & 29.30 & 27.75 & 29.18 & 28.38 & 30.26 & 28.51 \\  & open-aditi-v6-llama3 & 29.30 & 31.65 & 29.99 & 30.15 & 34.78 & 32.57 & 32.43 & 37.12 & 32.25 \\  & AryaBlatta-GermanaGEmZ-Merged & **39.42** & **37.88** & **37.59** & **41.14** & **39.60** & **41.25** & **35.71** & **48.49** & **40.14** \\ \hline \multirow{4}{*}{_te_} & telugu-lamaa-7b-instruct-v0.1 & **30.19** & **27.20** & **26.27** & 24.03 & 27.16 & 26.41 & 26.81 & 24.63 & **26.59** \\  & Telugu-Llamaa-7B-voBase & 22.96 & 27.10 & 24.80 & **27.01** & **26.14** & **27.13** & 25.42 & 27.53 & 26.01 \\  & Telugu-Llamaa-7B-voInstruct & 26.03 & 24.00 & 24.45 & 23.68 & 24.48 & 26.82 & **27.17** & **28.01** & 25.58 \\ \hline \multirow{4}{*}{_ka_} & Kan-LLAA-7B-base & 29.00 & 27.42 & **30.37** & 27.42 & **28.47** & 26.92 & **29.28** & 30.50 & 28.67 \\  & Ambari-7B-base-v0.1 & **31.49** & 27.52 & 29.08 & **28.70** & 27.15 & **27.82** & 27.93 & **31.00** & **28.84** \\  & Ambari-7B-Instruct-v0.1 & 28.17 & **28.02** & 24.83 & 27.78 & 23.84 & 27.56 & 27.42 & 27.00 & 26.83 \\ \hline \multirow{4}{*}{_mr_} & Misal-7B-base-v0.1 & **26.96** & **26.65** & **25.02** & **27.19** & **27.28** & **26.11** & **26.82** & **24.05** & **26.26** \\  & Misal-7B-instruct-v0.1 & 25.05 & 21.44 & 21.55 & 22.89 & 21.82 & 24.59 & 23.36 & 22.16 & 22.86 \\ \hline \multirow{4}{*}{_ta_} & tamil-lama-7b-instruct-v0.2 & 22.22 & 24.12 & 25.32 & 24.89 & 28.69 & 24.98 & 25.07 & 22.79 & 24.76 \\ \cline{1-1} \cline{2-11}  & malayalam-limaa-7b-instruct-v0.1 & 30.00 & 27.92 & 23.42 & 27.88 & 25.00 & 27.57 & 23.33 & 20.00 & 25.64 \\ \hline \end{tabular}
\end{table}
Table 4: Evaluation results of all the language-specific fine-tuned models on MILU. We report domain level 5-shot accuracies for all the models on the language supported by the model. Higher values indicate better model performance for the given domain. Refer to Section (§5.2) for more details.

Figure 3: Comparison of Base and Instruct models averaged across all languages for varying number of context examples. We plot the average accuracies of the Gemma and Llama series of models, highlighting the performance trend as the number of in-context examples increases. Refer to Section (§5.3) for more details.

 

rect "apples-to-apples" comparison but highlights key challenges. We conjecture that limited performance gains may result from small language-specific datasets and reliance on parameter-efficient methods like LoRA Hu et al. (2022). Another contributing factor could be the lack of diversity in instruction fine-tuning datasets. Models like AiraVata, which utilize more diverse data Gala et al. (2024), exhibit noticeably better performance. Further investigation is required to fully understand the limitations and opportunities in this area.

## 6 Conclusion

In this paper, we introduced MILU--**M**ultilingual **I**ndic **L**anguage Understanding Benchmark-a comprehensive benchmark specifically designed to evaluate LLMs across 11 Indic languages, spanning diverse domains and culturally relevant subjects. We evaluate 45 different LLMs and find that the majority of LLMs struggle on MILU, with GPT-4o achieving the highest average accuracy. The analysis also shows that models perform significantly better in high-resource languages than low-resource ones, highlighting the need for more robust multilingual strategies. Additionally, the domain-specific analysis indicates that models perform better in general fields such as STEM while facing challenges in culturally relevant subjects like Arts, Humanities, and Law, highlighting the lack of this knowledge in the current models and datasets. To foster open research, we release MILU along

Figure 4: Evaluation results of Base models (3(b)) and Instruct models (3(c)) on the different domains supported in MILU. The plot shows the average 5-shot accuracies across all languages for various models. Refer to Section (§5.5)

\begin{table}
\begin{tabular}{l l c r r r r} \hline \hline _lang_ & **Model** & **CPT** & **IPT** & **0** & **1** & **5** \\ \hline  & OpenInitial & Y & N & -0.28\% & -0.25\% & 0.58\% \\ _hi_ & Aravata & Y & Y & **1.70\%** & **2.40\%** & 3.11\% \\  & Gajenda\textgreater{}0.1 & Y & Y & 0.73\% & 0.92\% & **3.22\%** \\ \hline  & Ambari-base & Y & N & **0.72\%** & 1.17\% & 2.46\% \\ _kn_ & Ambari-Instruct & Y & V & 0.60\% & 0.49\% & 1.87\% \\  & Kan-7B-Base & Y & N & **1.13\%** & **0.84\%** & **1.30\%** \\ \hline  & eluegu-lama-instruct & Y & Y & **1.13\%** & **0.84\%** & **1.30\%** \\ _te_ & Telugu-7B-v0-Base & Y & N & 0.32\% & -0.52\% & 0.88\% \\  & Telugu-7B-v0-Instruct & Y & Y & -0.98\% & -0.12\% & 0.29\% \\ \hline _mr_ & Misial-7B-base-v0.1 & Y & N & **0.16\%** & **–0.60\%** & **-0.93\%** \\  & Misal-7B-abstract-v0.1 & Y & Y & -2.55\% & -2.17\% & -2.38\% \\ \hline _la_ & Tamil-lama-instruct & Y & Y & -0.06\% & -0.04\% & -0.19\% \\ \hline _ml_ & malayalam-lima-instruct & Y & Y & 0.77\% & 1.09\% & 0.33\% \\ \hline \end{tabular}
\end{table}
Table 5: Performance differences of Indian language models compared to the Llama-2-7B baseline at different in-context example settings. Each value shows the relative improvement or decline (% difference) over the baseline, with a +ve value indicating improvement and a -ve value indicating a drop in the performance. Refer to Section (§5.6) for more details.

Figure 5: Comparison of performance of Gamma and Llama family of models across different parameter scales. We plot the zero-shot average accuracies of all models across languages. Refer to Section (§5.4) for more details.

 

side all code, and other artifacts. As LLMs continue to become pivotal in modern applications, we hope that MILU offers a foundational benchmark for developing more inclusive, culturally aware models that perform well across both general and culturally relevant domains.

## Limitations

This work has a few limitations. First, we restricted our study to the top 11 languages due to the lack of readily available questions in low-resource languages, which we aim to address in future work. Second, limited computational resources prevented a thorough evaluation of larger models, such as Llama-3.1-70B-Instruct and Llama-3.1-405B. Third, the scarcity of questions necessitated translating a portion of the dataset. Finally, our evaluation primarily relies on the log-likelihood approach, which may yield different results compared to other established methods, such as generation-based evaluation and chain-of-thought (CoT) prompting.

## Acknowledgements

We would like to thank EkStep Foundation and Nilekani Philanthropies for their generous grant towards building datasets, models, tools and other resources for Indian languages. We are also immensely grateful to the volunteers from the AI4Bharat team for their motivation and meticulous efforts in conducting manual audits.

## Ethics

All data described in this work was scraped from publicly available resources. The datasets used in this paper will be made available under permissible licenses. Additionally, the code used for our evaluations will be made publicly available under the MIT License. We only used ChatGPT for assistance purely with the language of the paper, e.g., paraphrasing, spell-checking, or polishing the author's original content, without suggesting new content.

## References

* Ahuja et al. [2023] Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshy Nambi, Tanuya Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023. Mega: Multilingual evaluation of generative ai. _arXiv preprint arXiv: 2303.12528_.
* Ahuja et al. [2024] Sanchit Ahuja, Kumar Tanmay, Hardik Hansrajbhai Chauhan, Barun Patra, Kriti Aggarwal, Luciano Del Corro, Arindam Mitra, Tejas Indulah Dhamecha, Ahmed Awadallah, Monojit Choudhary, et al. 2024. sphinx: Sample efficient multilingual instruction fine-tuning through n-shot guided prompting. _arXiv preprint arXiv:2407.09879_.
* Anthropic [2024] Anthropic. Introducing the next generation of claude. [https://www.antropic.com/news/claude-3-f](https://www.antropic.com/news/claude-3-f) amily. Accessed: 2024-10-14.
* Artetxe et al. [2020] Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. _ACL_.
* Aryabumi et al. [2024] Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Ustuin, and Sara Hooker. 2024. Aya 23: Open weight releases to further multilingual progress. _arXiv preprint arXiv: 2405.15032_.
* Balachandran [2023] Abhinand Balachandran. 2023. Tamil-llama: A new tamil language model based on llama 2. _arXiv preprint arXiv: 2311.05845_.
* Biderman et al. [2024] Stella Biderman, Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baben Abbasi, Alham Fiki Aji, Pawan Sasanka Amamanachi, Sid Black, Jordan Clive, Anthony DiForfi, Julien Ekrainz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Mimnas Jaiswal, Wilson Y. Lee, Haonan Li, Charles Loverning, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Ariya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, Francocis Yvon, and Andy Zou. 2024. Lessons from the trenches on reproducible evaluation of language models. _ArXiv_, abs/2405.14782.
* Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girishi Sastry, Amanda Askell, Sandhini Agarwal,riel Herbert-Voss, Grecthen Krueger, Tom Henigian, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Chiang et al. [2024] Wei-Lin Chiang, Tianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot arena: An open platform for evaluating llms by human preference 

Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar 2033. Towards leaving no Indic language behind: Building monolingual corpora, benchmark and models for Indic languages. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12402-12426, Toronto, Canada. Association for Computational Linguistics.
* Dubey et al. [2015] Abhimanyu Dubey, Abhinav Jauhiri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethamby Brion, Binn Tang, Bobibe Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esioub, Dhruv Chouldary, Dhruw Mahajan, Diego Garcia-Olano, Diogo Perino, Dieuwke Hupless, Egor Lakomkin, Ehab AlbAlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeeve, Gabriele Lee, Georgia Lewis Anderson, Graeme Mail, Gregoire Milalon, Guan Pang, Guillem Cucurrell, Hailey Nguyen, Hannah Korcevar, Hu Xu, Hugo Touvorn, Iliyan Zarov, Imanol Arrieta barra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spiskak, Jongsoo Pavlov, Joseph Rocca, Joshua Johnston, Joshax Saxe, Junting Jia, Kalyan Vasudeh Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kishitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Munzzi, Mahesh Pasupulet, Mannat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladr Chatterji, Olivier Duchenne, Onur Celebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Purxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Roman Sauveste, Ronnie Poltorodro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennbaspappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shengye Wan, Shruti Bhasale, Shunn Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gurwangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihalyov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonquet, Virginic Do, Vispien Vogeli, Vladan Petrovric, Weiwei Chu, Wenhan Xiong, Wenvin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaqiang Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhenging Chen, Zoe Papakiops, Aadivis Singh, Aaron Grattaforatiori, Abha Jain, Adam Kelsey, Adam Shaighfeld, Adity Gangidi, Adolfo Victoria, Ahura Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Beavski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Sarraf, Arkrabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Brit Montlayc, Carl Parker, Carly Burton, Catalina Meija, Changhan Wang, Chandsyu Kim, Chao Zhou, Chester Hu, China-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Arcuate, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzzman, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Bader, Georgia Swee, Gil Happern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zhao, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Goldman, Ibrahim Damlabj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gata, Jake Weissman, James Geboski, James Kohli, Jarbet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang, Joe Cummings, Jon Carrill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junje Wang, Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartiky Kandelwal, Katayoung Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelea, Keqian Li, Kun Huang, Kunal Chawla, Kwald Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng 

Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lemnie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Kenelally, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihar Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Muqiet Jubert Hermoso, Mo Metanet, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navaya Bauva, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Partb Parekh, Paul Saab, Pavan Balaji, Pedro Ritmer, Philip Pontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratnamchandi, Pritish Yurraj, Qian Liang, Rachachad Alao, Rachel Rodriguez, Rafii Ayub, Raghotham Murthy, Raghu Naguni, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruuty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Sheng Feng, Shengaho Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprascal, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Syoudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tezook Shaked, Varun VontimittA, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Aliiberto, Vlad Ionescu, Vlad Poenaru, Vlad Tabierti Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Constable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xie, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosbnrick, Zhadoue Wen, Zhenyu Yang, and Zhiwei Zhao. 2024. The llama 3 herd of models. _arXiv preprint arXiv: 2407.21783_.
* Gala et al. (2024) Jay Gala, Thanmay Jayakumar, Jaavid Aktar Husain J, Aswanth Kumar Md, Mohammed Safi Ur Rahman Khan, Diptesh Kanojita, Ratish Puduppully, Mitesh M Khapra, Raj Dabre, Rudra Murphy, and Anoop Kunchukuttan. 2024. Airavata: Introducing hindi instruction-tuned LLM.
* Gala et al. (2023) Jay P. Gala, Pranjal A. Chatale, AK Raghavan, Varun Gumma, Sumanth Doddapaneni, M. AswanthKumar, J. Nawake, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages. _Trans. Mach. Learn. Res._
* Gao et al. (2024) Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Goldingle, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Coiepa, Jason Phanga, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. A framework for few-shot language model evaluation.
* Ghosh et al. (2024)Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Ramanesaran S., Deepali Aneja, Zeyu Jin, Ramani Duraiswami, and Dinesh Manocha. 2024. A closer look at the limitations of instruction tuning. In _Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024_. OpenReview.net.
* Hendrycks et al. (2021)Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
* Hu et al. (2022)Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
* Hu et al. (2020)Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. _arXiv preprint arXiv: 2003.11080_.
* Javed et al. (2024) Tahir Javed, Janki Nawale, Eldho George, Sakshi Joshi, Kaushal Bhogale, Devrat Mehendale, Ishwindet Sethi, Aparnna Annathanarayanan, Hafsah Faquih, Pratti Palit, Sneka Ravishankar, Saranya Sukumaran, Tripuran Panchangula, Sanjay Murali, Kunal Gandhi, Ambujaravalli R, Manickam M, C Vaijayanthi, Krishnan Karunganni, Pratyush Kumar, and Mitesh Khapra. 2024. IndicVoices: Towards building an inclusive multilingual speech dataset for Indian languages. In _Findings of the Association for Computational Linguistics ACL 2024_, pages 10740-10782, Bangokk, Thailand and virtual meeting. Association for Computational Linguistics.
* Kakwani et al. (2020)Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Goklul N.C.,vik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSute: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In _Findings of the Association for Computational Linguistics: EMNLP 2020_, pages 4948-4961, Online. Association for Computational Linguistics.
* Khan et al. (2021)Mahammed Safi Ur Rahman Khan, Priyam Mehta, Ananth Sankar, Umashankar Kumaravalem, Sumanth Doddapaneni, Suriyaprasaad G, Varun Balan G, 

Sparsh Jain, Anoop Kunchukuttan, Pratyush Kumar, Raj Dabre, and Mitesh M. Khapra. 2024. Indicllmsuite: A blueprint for creating pre-training and fine-tuning datasets for indian languages. _arXiv preprint arXiv: 2403.06350_.
* Kim et al. (2024) Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyunk, Hyeonbin Hwang, Jinkving Jo, Hywoon Cho, Haebin Shin, Seongyun Lee, Hanscoi Oh, Noah Lee, Namygu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo Chae, Jamin Shin, Joel Jang, Seonghyeon Ye, Bill Yuchen Lin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. The biggest bench: A principled benchmark for fine-grained evaluation of language models with language models. _arXiv preprint arXiv: 2406.05761_.
* Kohli et al. (2023) Guneet Singh Kohli, Shantipriya Parida, Sambit Sekhar, Samirit Saha, Nipun B Nair, Parul Agarwal, Somal Khosla, Kusumlata Patiyal, and Debasish Dhal. 2023. Building a llama2-finetuned llm for odia language utilizing domain knowledge instruction set. _arXiv preprint arXiv: 2312.12624_.
* Kolaiv (2024) Adithya S Kolaiv. 2024. Indic llvm leaderboard. [https://huggingface.co/spaces/CognitiveLab/indic_llm_leaderboard](https://huggingface.co/spaces/CognitiveLab/indic_llm_leaderboard).
* Kumar et al. (2022)Aman Kumar, Himani Shrotriya, Prachi Sahu, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Amogh Mishra, Mitesh M. Khapra, and Pratyush Kumar. 2022. Induclng benchmark: Multilingual datasets for diverse nlg tasks in indic langauges. _arXiv preprint arXiv: 2203.05437_.
* Lee et al. (2024)Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. Nv-embed: Improved techniques for training llms as generalist embedding models. _arXiv preprint arXiv: 2405.17428_.
* Liu et al. (2024) Yang Liu, Meng Xu, Shuo Wang, Liner Yang, Haoyu Wang, Zhenghao Liu, Cunliang Kong, Yun Chen, Yang Liu, Maosong Sun, and Erhong Yang. 2024. Omgeval: An open multilingual generative evaluation benchmark for large language models. _arXiv preprint arXiv: 2402.13524_.
* Madhani et al. (2023)Yash Madhani, Mitesh M. Khapra, and Anoop Kunchukutan. 2023. Bhasa-abhijnanaam: NativeScript and romanized language identification for 22 Indic languages. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 816-826, Toronto, Canada. Association for Computational Linguistics.
* McIntosh et al. (2024)Timothy R. McIntosh, Teo Susnjak, Nalin Arachchilage, Tong Liu, Paul Watters, and Malka N. Halgalamuge. 2024. Inadequacies of large language model benchmarks in the era of generative artificial intelligence. _arXiv preprint arXiv: 2402.09880_.
* large language models for southeast asia. _arXiv preprint arXiv: 2312.00738_.
* OpenAI et al. (2023)OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Abraman, Diogo Almeida, Janko Atlenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haining Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lennyg Bogdonoff, Oleg Boiko, Madealine Boyd, Anna-Luisa Brakman, Greg Brockmann, Tim Brooks, Miles Brundagex, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chatzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jererinal Currier, Yunzing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simon Posada Fishman, Juston Forte, Isabella Fullford, Leo Gao, Elei Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Golb, Rapha Gonitojolopes, Jonathan Gordon, Morgan Garstein, Scott Gray, Ryan Greene, Joshua Gross, Shixing Shang Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidelberg, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Greten Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinnney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Melly, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Heyonvoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascando, Joel Parish, Emy Paparrita, Alex 

Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Penes, Michael Petrov, Henrique Pande de Oliveira Pinto, Michael Pokerny, Michelle Polkenny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Seslam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Syzmon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeeling B Thompson, Phil Tillet, Amin Tootochanin, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Ceron Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinada, Peter Welinder, Jiayi Weng, Lilian Wong, Matt Weithoff, Dave Wilmer, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023. Gpt-4 technical report. _PREPRINT_.
* Rohera et al. (2024) Pritika Rohera, Chaitrali Gnilmay, Akanksha Salunke, Gayatri Sawant, and Raviraj Joshi. 2024. L3cube-indicquest: A benchmark question answering dataset for evaluating knowledge of l1ms in indicte context. _arXiv preprint arXiv: 2409.08706_.
* Singh et al. (2024a) Abhishek Kumar Singh, Rudra Murthy, Vishwajeet kumar, Jaydeep Sen, and Ganesh Ramakrishnan. 2024a. Indic qa benchmark: A multilingual benchmark to evaluate question answering capability of llms for indic languages. _arXiv preprint arXiv: 2407.13522_.
* Singh et al. (2024b) Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dianesh Tewari, and Partha Talukdar. 2024b. Indicgenbench: A multilingual benchmark to evaluate generation capabilities of lfms on indic langauges. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangokok, Thailand, August 11-16, 2024_, pages 11047-11073. Association for Computational Linguistics.
* Team et al. (2020) Gemini Team, Rohan Anil, Sebastian Borgeaud, JeanBaptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schlakwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Melvin Johnson, Ioannis Antonoglou, Julian Schrittweiser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Mollooy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Eric Moreira, Kareem Ayoub, Megha Goel, Jack Krawczyk, Cosmo Du, Ed Chi, Heng-Tze Cheng, Eric Ni, Purvi Shah, Patrick Kane, Betty Chan, Manal Farauqui, Aliaskei Severyn, Hanzhao Lin, YaGuang Li, Yong Cheng, Abe Ittychebrahi, Mahdis Mahdieh, Mia Chen, Pei Sun, Dustin Tran, Sumit Bagri, Balaji Lakshminarayanan, Jeremiah Liu, Andras Orbban, Fabian Gura, Hao Zhou, Xinying Song, Aurelien Boffy, Harish Ganapathy, Steven Zheng, Hyun Jeong Choe, Agoston Weisz, Tao Zhu, Yifeng Lu, Siddharth Gopal, Jarrod Kahn, Maciej Kula, Jeff Pitman, Rushin Shah, Emanuel Taropa, Majd AI Meery, Martin Baeuml, Zhifeng Chen, Laurent El Shafev, Yujing Zhang, Olcan Sercinoglu, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anais White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khaliman, Jakub Sygnowski, Alexandre Fredente, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Frustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Siffe, Marcello Maggioni, Fred Alcboer, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahjuja, Gaurav Singh Tomar, Evan Senter, Martin Chadwick, Ilya Kornakov, Nithya Attaluri, Iizaki Iturrate, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grismald, Ale Jeske Hartman, Xavier Garcia, Thanmalayaan Sanchararanaryana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adria Puigdomeench Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergeye Brin, Sharid Iqbal, Gabriela Surita, Jane Labamowski, Abiba Rao, Stephanie Winkler, Emilio Parisotto, Timing Gu, Kate Olszewskiska, Ravi Addanki, Antoine Miech, Annise Louis, Denis Peplyashin, Geoff Brown, Elliot Catt, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Stjerrgren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejjman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godobole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sebastien M. R. Arnold, Vijay Vasudevan, Subham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanhpurn, Srivastani Srinivasan, Hycontane Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Rachel Reid, Elizabeth Cole, Aakarksha Chowdhrey, Dipjanjan Das, Dominika Rogozinska, Vitaliy Nikolaev, Pablo Sprechennam, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newman, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu 

Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Gimenez, Legg Yeung, James Keeling, Petako Georgiev, Diana Micu, Bovic Wu, Salem Haykal, Rachel Saputto, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yilin Chang, Paul Komarek, Ross McIlroy, Mario Lucic, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakereri, Christina Butterfield, Justin Chung, Paul Kishenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Aarel Lenc, Timothy Chung, Acadn Pope, Loren Maggiore, Jacke Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Majia Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko Ilinuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphael Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Adivis Siddhant, Nand Tomasev, Jinwei Xing, Christina Greet, Helen Miller, Shereen Ashraf, Aurko Roy, Zixhao Zhang, Ada Ma, Angelos Filios, Milos Besta, Rory Brevlins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasa,mit Marathe, Steven Hansen, Shotlo Douglas, Rajkumar Samuel, Mingqui Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjosund, Sebastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyam Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Leonard Hussteen, Livido Baldiani Soares, Kate Baumli, Michael B. Chang, Adria Recascens, Ben Caine, Alexander Pritzel, Filip Pavietic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajjit Roy, Ethan Dyer, Victor Campos Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindald, Sharak Vikram, Zhita Gong, Sergi Caelles, Ross Hemsky, Gregory Thornton, Fangxiaoy Feng, Wojiecich Stokowice, Ce Zheng, Phoee Thacker, Caglar Unlu, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piiyush Pattil, Ankesh Anand, Roman Ring, Katerina Tsillas, Argu Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kuwaitowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerre, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganimi, Jean-Baptiste Lespiau, Alexandre Mourfex, Samer Hassan, Kaushik Shrikavlamar, Joost van Amersfoort, Amol Mandahane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakicevic, Mostafa Dehghani, Fanyju Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Mathew Lamm, Nicola De Cao, Charlie Chen, Sidharth Mudgal, Romina Stella, Kevin Brooks, Gaunt Vanudevan, Chennai Liu, Maincha Ninvedita Melinkeri, Aaron Cohen, Venus Wang, Kristic Seymore, Sergey Zubov, Rahul Goel, Summer Yue, Saix Krishnakumaran, Brian Albert, Nate Hurley, Motoki Sano, Anhad Mohananey, Jonah Joughin, Egor Filonov, Tomasz Keps, Yomna Eldawy, Jianvern Lim, Rahul Rishi, Shirin Badizcagedgen, Taylor Bos, Jerry Chang, Sanil Jain, Sri Gayatri Sundara Padmanabhan, Suba Puttagunta, Kalpesh Krishna, Leslie Baker, Norbert Kalb, Vamsi Bedapudi, Adam Kurrzrobk, Shunting Lei, Anthony Yu, Oren Litvin, Xiang Zhou, Zhichun Wu, Sam Sobell, Andrea Siciliano, Alan Papir, Robby Neale, Jonas Bragaagnolo, Tejo Tor, Tina Chen, Valentin Anklin, Ferian Wang, Richie Feng, Mild Gholami, Kevin Ling, Lijun Liu, Jules Walter, Hamid Moghaddam, Arnu Kishore, Jakub Adamek, Tyler Mercado, Jonathan Mallinson, Siddhinita Wandekar, Stephen Cagle, Eran Ofek, Guillermo Garrido, Clemens Lombriser, Maksim Mukha, Botu Sun, Hafezezahl Rahman Mohammad, Josip Mutak, Yadid Qian, Vikas Peswani, Pawel Janus, Quan Yuan, Leif Schelin, Osma David, Ankur Garg, Yifan He, Oleksii Uuzbiy, Anton Algmyr, Timothee Lottaz, Qi Li, Vikas Yadav, Luyao Xu, Alex Chinien, Rakesh Shivanna, Aleksandr Chuklin, Josie Li, Carrie Spadine, Travis Wolfe, Kareem Mohamed, Subhbrata Das, Zahang Dai, Kyle He, Daniel von Dincklage, Shyam Upadhyay, Akanksha Manurya, Luyan Ci, Sebastian Krause, Khalil Salama, Pam G Rabinovitch, Pavan Kumar Reddy M, Aarush Selvan, Mikhail Dekitarev, Golnaz Ghiasi, Erdem Guven, Himanshu Gupta, Boyi Liu, Deepak Sharma, Idan Heimlich Shtacher, Shachi Paul, Oscar Akrelund, Francois-Xavier Aubett, Terry Huang, Chen Zhu, Eric Zhu, Elico Teixeira, Matthew Fritze, Francesco Bertolini, Liana-Eleonora Marinescu, Martin Bolle, Dominik Paulus, Khyatti Gupta, Tejasi Latkar, Max Chang, Jason Sanders, Roopa Wilson, Xuewei Wu, Yi-Xuan Tan, Lam Nguyen Thiet, Tulsee Doshi, Sid Lall, Swaroop Mishra, Wanming Chen, Thang Luong, Seth Benjamin, Jasmine Lee, Ewa Andrejczuk, Dominik Rabiej, Viupil Ranjan, Krzygart Story, Pengcheng Yin, Jon Simon, Malcolm Rose Harriott, Mudit Bansal, Alexei Robsky, Geoff Bacon, David Greene, Daniil Miryllenka, Chen Zhou, Obaid Sarvana, Abhimanya Goyal, Samuel Andermatt, Patrick Siegel, Ben Horn, Assaf Is 

rael, Francesco Pongetiti, Chih-Wei "Louis" Chen, Marco Selvatiic, Pedro Silva, Kathie Wang, Jackson Tollins, Kelvin Guu, Reoy Yogev, Xiaochen Cai, Alessandro Agostini, Mauilsh Shah, Hung Nguyen, Noah O Donnaille, Sebastien Pereira, Linda Friso, Adam Stambler, Adam Kurzok, Chenkai Kuang, Yan Romanikhin, Mark Geller, ZJ Yan, Kane Jang, Cheng-Chun Lee, Wojciech Fica, Eric Malmi, Qijun Tan, Dan Banica, Daniel Balle, Ryan Pham, Yanping Huang, Diana Avram, Hongzh Shi, Jasiot Singh, Chris Hidey, Nihraka Ahuja, Panah Saxena, Dan Dooley, Srividya Pranaraju, Eileen O'Neill, Anand Gokhulchandran, Ryan Foley, Kai Zhao, Mike Disenberry, Yuan Liu, Pulkit Mehta, Ragha Kotikalapudi, Chalence Safraneek-Shrader, Andrew Goodman, Joshua Kessinger, Eran Globen, Praetek Kolhar, Chris Gorgolevski, Ali Ibrahim, Yang Song, Ali Eichenbaum, Thomas Brovelli, Sahitya Potluri, Preteith Lahiot, Cip Baetu, Ali Ghorbani, Charles Chen, Andy Crawford, Shaliani Pal, Mukund Sridhar, Petru Gurita, Asier Mujika, Igor Petrovski, Pierre-Louis Cedoz, Chenmei Li, Shiyuan Chen, Niccolo Dal Santo, Siddarthath Goyal, Jitesh Punjabi, Karthik Kapagnuthu, Chester Kwak, Pallavi LV, Sarmishta Velury, Himadi Choudhary, Jamie Hall, Premad Shah, Ricardo Figueira, Matt Thomas, Minjie Lu, Ting Zhou, Chinut Kumar, Thomas Jurdi, Sharat Chhikerur, Yennai Ma, Adams Yu, Soo Kwak, Victor Abdel, Sujeevan Rajayogam, Travis Choma, Fei Liu, Aditya Barua, Colin Ji, Ji Ho Park, Vincent Hellendoorn, Alex Bailey, Taylan Bilal, Huanig Zhu, Mehrdad Khatir, Charles Sutton, Vojciciech Razdkowski, Fiona Macintosh, Konstantin Shagin, Paul Medina, Chen Liang, Jinjing Zhou, Pararth Shah, Yingying Bi, Attila Dankovics, Shipra Banga, Sabine Lehmann, Marissa Bredesen, Zifan Lin, John Eric Hoffmann, Jonathan Lai, Raynald Chung, Kai Yang, Nihal Balani, Arthur Brazinaskas, Andrei Sozanisch, Matthew Hayes, Hector Fernandez Alcalde, Peter Makarov, Will Chen, Antonio Stella, Liselotte Snijders, Michael Mandl, Ante Karrman, Pawel Nowak, Xinyi Wu, Alex Dyck, Krishnan Vaidyanathan, Raghavender R, Jessica Mallet, Mitch Rudominer, Eric Johnston, Sushl Mittal, Akhill Uddhu, Janara Christensen, Vishal Verma, Zach Irving, Andreas Santucci, Gamaldelin Elsayed, Elhaz Davoodi, Marin Georgiev, Ian Tenney, Nan Hua, Geoffrey Cleron, Edouard Leurent, Mahmoud Alnahawli, Ionut Georgescu, Nan Wei, Ivy Zheng, Phylan Scandinaro, Heinrich Jiang, Jasper Snoeck, Mukund Sundararajan, Xuezhi Wang, Zack Ontieveros, Itay Karo, Jeremy Cole, Vinu Rajasekhar, Lara Tumeh, Eyal Ben-David, Ribtab Jain, Jonathan USato, Romina Datta, Oskar Bunyan, Shimu Wu, John Zhang, Piotor Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jia Miao, Andrew Lee, Nino Vieillard, Jane Park, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abbijit Kammarkar, Zhe Dong, Jong Lee, Aviral Kumar, Louwei Zhou, Jonathan Evens, William Isaac, Geoffrey Irving, Edward Loper, Michael Fink, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Ivan Petrychenko, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Peter Grabowski, Yu Mao, Alberto Magni, Kaising Yao, Javier Snaidern, Norman Casagrande, Evan Palmer, Paul Suganthan, Alfonso Castano, Irene Giannoumis, Wooyeol Kim, Mikolaj Rybinski, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckeneyer, Willi Gierek, Mohsen Jafari, Meenti Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Weir, Harash Vasishisht, Yaa Kuliziskaya, Jay Hoover, Maigo Leo, Lu Li, Chimeize lwananywu, Lu Liu, Kevin Ramirez, Andrey Knorlin, Albert Cui, Tian LIN, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Ginger Peng, Elena Allica Abellan, Mingyang Zhang, Ishita Dasgupta, Nate Kushman, Ivo Penchev, Alena Repina, Xihui Wu, Tom van der Weide, Priyu Ponnampalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ic, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Daniel Andor, Pedro Valenzuela, Minnie Lui, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somerc Greene, Duc Jung Nguyen, Paula Kurylowicz, Cassidy Hardin, Lucas Dixon, Lilj Janzer, Kiam Cho, Ziqiang Feng, Biao Zhang, Achinitya Singhal, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolubasi, Orpad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Ken Franko, Anna Bulnova, Remi Leibdon, Shirley Chung, Harry Askhann, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Mark Omernick, Colton Bishop, Rachel Sterneck, Rohan Jain, Jiawei Xia, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseen Banzali, Daniel J. Mankowitz, Alex Polozov, Victoria Krakkova, Sasha Brown, MohammadHossein Bateni, Dennis Duan, Vlad Fireiu, Meghana Thtokaturi, Tom Natan, Matthieu Geist, Ser tan Girgin, Hui Li, Jiayi Ye, Ofir Royal, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Cathy Wu, David Miller, Nicolas Sonnerat, Denis Venukov, Roy Greje, Jenifer Beatite, Emily Cavenes, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomm Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Nejc Trdin, Daniel Toyama, Evan Rosen, Susan Tavakkol, Limting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakerius, Rupert Kemp, Sushant Kafle, Tanya Gruning, Rishika Sinha, Alice Talbert, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Saaber Fatehi, John Wieting, Omar Ajmerri, Bengino Unia, Yeongil Ko, Laura Knight, Amelie Heliou, Ning Niu, Shane Gu, Chenxi Pang, Yeqing Li, Nur Levine, Ariel Stolovich, Rebeca Santamaria-Fernandez, Sonam Goenka, Wenny Yusstalim, Robin Strudel, Ali Elqursh, Charlie Deck, Hyo Lee, Zonglin Li, Kyle Levin, Raphaael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Sho 

Arora, Christy Koh, Soheil Hassas Yeganeh, Siim Poder, Mukarram Tario, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Zhiyu Liu, Anand Gulati, Jasmin Liu, Xinyu Ye, Bart Chrzaszcz, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Vinod Koverkathu, Christopher A. Choquette-Choo, Yunje Li, TfJ Lu, Abe Ittycheriah, Prakash Shroff, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Guillaume Desjardins, Marco Cornero, Bronen Roebenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Riviere, Alanna Walton, Clement Crepy, Aliicia Parrish, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidigeland, Salvatore Secallato, Eri Latorne-Chimoto, Hanna Kimczak-Plucirska, Davaid Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Lynette Webb, Sahli Dua, Dong Li, Surya Bhupatiraju, Dan Hut, Haroon Qureshi, Ananth Agarwal, Tumer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyfer Lichetly, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Mantish Reddy Vuryuru, John Aslanides, Nichi Vyas, Martin Wicke, Xiao Ma, Evgenii Eltyshev, Nina Martin, Hadrie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nielsh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polyouts, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjill Ozair, Christof Angermueller, Xiaowei Li, Anooip Sinha, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurunmurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennarinag Palomaki, Chrisantha Fernandez0, Ken Durden, Harsh Mehta, Nikolao Momchev, Clahae Rahimovigh, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jindhyuk Lee, Denny Zhou, Komal Jalan, Dinghua Li, Blake Hedtman, Parker Schuh, Milad Nasr, Kieran Milan, Vladimir Miklukila, Juliana Franco, Tim Green, Nam Nguyen, Joe Kelley, Arona Mahendru, Andrea Hu, Joshau Howland, Ben Vargas, Jeffrey Hui, Kshitij Bansal, Vikram Rao, Rakesh Ghiya, Emma Wang, Ke Ye, Jean Michel Sarr, Melanie Moranski Preston, Madeleine Elish, Steve Li, Aakash Kak, Jiger Gupta, Ice Pasupat, Da-Cheng Juan, Milan Someswar, Teiyi M., Xinpu Chen, Akida Amini, Alex Fabrikant, Eric Chu, Xuanyi Dong, Ampratta Muthal, Senaka Butthipatiya, Sarthak Jauhari, Nan Hua, Urvashi Khandelwal, Ayal Hitron, Jie Ren, Larissa Rinaldi, Shahar Drath, Avvigail Dabush, Nan-Jiang Jiang, Harshal Godhis, Uli Sachs, Anthony Chen, Yicheng Fan, Hagai Taiitelbaum, Hila Noga, Zhuyun Dai, James Wang, Chen Liang, Jenny Hamer, Chun-Sung Ferng, Chenel Elkind, Aviel Atias, Paulina Lee, Vit Liststhatis Carlen, Jan van de Kerkhof, Marcin Pikus, Kunroslav Zahef, Paul Muller, Sasha Zykova, Richard Stefanetc, Vitaly Gatsko, Christoph Hirnschall, Ashwin Sethi, Xingyu Federico Xu, Chetan Ahuja, Beth Tsai, Anca Stefanoiu, Bo Feng, Keshav Chandhanania, Manish Katyal, Akshay Gupta, Athara Parulekar, Divya Pitta, Jing Zhao, Vivana Bhatia, Yashodha Bhavanni, Ourz Alhadlaq, Xiaolin Li, Peter Dancenberg, Dennis Tu, Alex Pine, Vera Filippova, Abhipos Ghosh, Ben Limonchik, Bharagava Urala, Chaitanya Krishna Lanka, Derik Clive, Yi Sun, Edward Li, Hao Wu, Kevin Hongtongsak, Inna Li, Kalind Thakkar, Kuanysh Omarov, Kushal Majumdar, Michael Alverson, Michael Kubacharski, Mohak Patel, Mudit Jain, Maksim Zhebeliin, Paolo Pelagatti, Rohan Kohli, Saurab Kumar, Joseph Kim, Swetha Sankar, Vineet Shah, Lakshmi Ramachandran, Xiangkai Zeng, Ben Bairach, Laura Weidanger, Tu Vu, Alek Andreev, Antoine He, Kevin Hui, Shepleem Kashem, Amar Subramanya, Sissie Hsiao, Demis Hasabis, Koray Kavukcuoglu, Adam Sadovsky, Quoc Le, Trevor Strohman, Yonghui Wu, Slav Petrov, Jeffrey Dean, and Oriol Vinyals. 2024a. Gemini: A family of highly capable multimodal models.
* [1]Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatriaju, Leonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, Nino Vieillard, Piotr Stanczyk, Sertan Girgin, Nikola Momchev, Matt Hoffman, Shantau Thakoor, Jean-Bastian Grill, Behnam Neyshabur, Olivier Bachem, Alamana Wallaton, Aliaksei Severyn, Alicia Parrish, Aliya Ahmad, Allen Hutchison, Alvin Abdagic, Amanda Carl, Amy Shen, Andy Brock, Andy Coonen, Anthony Laforge, Antonia Paterson, Ben Bastian, Bilal Piot, Bo Wu, Brandon Royal, Charlie Chen, Chintu Kumar, Chris Perry, Chris Welty, Christopher A. Choquette-Choo, Dahlio Singopalnikov, David Weinberger, Dimpliv Vijakumar, Dominica Rogozifaska, Dustin Herbison, Elisa Bandy, Emma Wang, Eric Noland, Eric Moreira, Evan Senter, Evgenii Eltyshev, Francesco Visin, Gabriel Rasskin, Gary Wei, Glenn Cameron, Guy Martins, Hadi Hashemi, Hanna Klimczak-Plucirska, Hafeen Batra, Harsh Dhand, Ivan Nardin, Jacinda Mein, Jack Zhou, James Svenson, Jeff Stanway, Jetha Chan, Jim Peng Zhou, Joana Carrascqueira, Joana Iljazi, Jocelyn Becker, Joe Fernandez, Joost van Amersoort, Josh Gordon, Josh Lipschitz, Josh Newlan, Ju yeong Ji, Kareem Mohamed, Kartikaeg Badola, Kat Black, Katie Millican, Keelin McDonell, Kelvin Nguyen, Kirranbir Sodinia, Kish Greene, Lars Lowe Sjoesund, Lauren Usui, Laurent Siffre, Lena Heuremann, Leica Lago, Lilly McNealus, Livio Baldini Soares, Logan Kilpatrick, Lucas Dixon, Luciano Martins, Machel Reid, Manwinder Singh, Mark Iverson, Mar 

tin Gorner, Mat Velloso, Mateo Wirth, Matt Davidson, Matt Miller, Matthew Rahtz, Matthew Watson, Meg Risdal, Mehran Kazemi, Michael Moynthan, Ming Zhang, Minusuk Kaing, Minwoo Park, Mofi Rahman, Mohit Khatwani, Natalie Dao, Nenshad Bardoliwalla, Nesh Devanathan, Neta Dumai, Nilay Chauhan, Oscar Wahlinez, Pankil Botarda, Parker Barnes, Paul Barham, Paul Michel, Pengchong Jin, Petko Georgiev, Phil Culliton, Pradeep Kuppala, Ramona Comanescu, Ramona Merhej, Reena Jana, Reza Ardeshik Rozhi, Riishab Agarwal, Ryan Mullins, Samaneh Saadat, Sara Mc Carthy, Sarah Perrin, Sebastien M. R. Arnold, Sebastian Krause, Shengyang Dai, Shruti Garg, Shruti Sheth, Sue Ronstrom, Susan Chan, Timothy Jordan, Ting Yu, Tom Eccles, Tom Hennigan, Tomas Kocisky, Tulse Doshi, Vilhan Jain, Vikas Yadav, Vilobu Meshram, Vishal Dharmadhikari, Warren Barkley, Wei Wei, Wenming Ye, Woohyun Han, Woosuk Kwon, Xiang Xu, Zhe Shen, Zhitao Gong, Zichuan Wei, Victor Cotruta, Phoebe Kirk, Anand Rao, Minh Giang, Ludovic Peran, Tris Warketnin, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, D. Sculley, jeanine Banks, Anca Dragan, Slav Petrov, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Sebastian Borgeaud, Noah Fiedel, Armand Joulin, Kathleen Kenealy, Robert Dadashi, and Alek Andreev. 2024b. Germma 2: Improving open language models at a practical size. _arXiv preprint arXiv: 2408.00118_.
* Ustun et al. (2024) A. Ustun, Virataryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction fine-tuned open-access multilingual language model. _Annual Meeting of the Association for Computational Linguistics_.
* Wang et al. (2024) Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguyang Guo, Weiming Ren, Azaran Antraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wanhu Chen. 2024. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark (published at neurips 2024 track datasets and benchmarks). _arXiv preprint arXiv: 2406.01574_.
* Watts et al. (2024) Ishaan Watts, Varun Gunmna, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, and Sunayama Sitaram. 2024. PARIKSHA: A large-scale investigation of human-LLM evaluator agreement on multilingual and multi-cultural data. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 7900-7932, Miami, Florida, USA. Association for Computational Linguistics.
* Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? _ACL_.
* Zhao et al. (2024) Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llama beyond english: An empirical study on language capability transfer. _arXiv preprint arXiv: 2401.01055_.
* Zhong et al. (2023) Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2023. Agieval: A human-centric benchmark for evaluating foundation models. _arXiv preprint arXiv: 2304.06364_.

## Appendix A Details about different Exams

We collected our questions from over 40 exam types ranging from various National and state level civil service examinations to examinations conducted by various government and private organizations. Tables 6, 7 and 8 



\begin{table}
\begin{tabular}{l|c} \hline
**Organization** & **Years Covered** \\ \hline Railway Recruitment Board & 2018-2019 \\ Railway Recruitment Cell & 2018 \\ Dedicated Freight Corridor Corporation of India Limited & 2016-2021 \\ Intelligence Bureau & 2012-2021 \\ Union Public Service Commission & 2010-2023 \\ \hline \end{tabular}
\end{table}
Table 6: Overview of various national-level exams and the corresponding years of coverage considered in MILU.

\begin{table}
\begin{tabular}{l|r} \hline
**Organization** & **Years Covered** \\ \hline Punjab Police & 2016-2022 \\ Punjab State Power Corporation Limited & 2018 \\ Chandigarh Police & 2018 \\ Telangana Police & 2015-2022 \\ Andhra Pradesh Police & 2016-2018 \\ Tamil Nadu Public Service Commission & 2010-2021 \\ Tamil Nadu Uniformed Services Recruitment Board & 2010-2022 \\ Odisha Police & 2022 \\ Karnataka State Police & 2019-2023 \\ Madhya Pradesh Police & 2016-2023 \\ Delhi Police & 2014-2022 \\ Haryana Police & 2021 \\ Lucknow Metro Rail Corporation & 2018 \\ Delhi Metro Rail Corporation & 2017-2020 \\ Punjab Subordinate Service Selection Board & 2022 \\ Punjab State Transmission Corporation Limited & 2016 \\ Gujarat Metro Rail Corporation & 2022 \\ Kolkata Police & 2023 \\ Maharashtra Police & 2021 \\ Jharkhand Police & 2015-2021 \\ \hline \end{tabular}
\end{table}
Table 7: Overview of various government and private organization exams and the corresponding years of coverage considered in MILU.

\begin{table}
\begin{tabular}{l|r} \hline
**Organization** & **Years Covered** \\ \hline Staff Selection Commission & 2017-2023 \\ West Bengal Civil Service & 2015-2021 \\ Bihar Public Service Commission & 2015-2020 \\ Maharashtra Public Service Commission & 2013-2023 \\ Rajasthan Subordinate and Ministerial Services Selection Board & 2016-2022 \\ Odisha Public Service Commission & 2016-2022 \\ Uttar Pradesh Public Service Commission & 2012-2023 \\ Haryana Public Service Commission & 2014-2022 \\ Andhra Pradesh Public Service Commission & 2017-2019 \\ Chhattisgarth Public Service Commission & 2014-2022 \\ Jammu \& Kashmir Public Service Commission & 2021 \\ Himachal Pradesh Public Service Commission & 2015-2022 \\ Jharkhand Public Service Commission & 2021 \\ Delhi Subordinate Services Selection Board & 2019-2022 \\ \hline \end{tabular}
\end{table}
Table 8: Overview of various State-level civil services exams and the corresponding years of coverage considered in MILU.



* [10] M. C. Bender, J. L. Friedman, and R. G. Tibshirani, _The Visual Classification of Image Processing Systems_, vol. 34, no. 2, pp. 57-68, 1996.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l|c c c c c c c c c c c c|c} \hline

**TOTAL** & 6638 & 13536 & 4827 & 14837 & 6234 & 4321 & 6924 & 4525 & 4099 & 6372 & 7304 & **79617** \\ \hline  & & & & & & & & & & & & \\ \end{tabular}
\end{table}
Table 9: Detailed subject level statistics of MILU across different languages.





\begin{table}
\begin{tabular}{l|c c|c} \hline
* & Link \\ GPT-4o-mini & Instruct &
- & Link \\ Gemini-1.5-Pro & Instruct &
- & Link \\ Gemini-1.5-Flash & Instruct &
- & Link \\ Krutrim-spectre-v2 & Instruct &
- & Link \\ \hline meta-llama/Llama-3.2-1B & Base & 1B & Link \\ meta-llama/Llama-3.2-1B-Instruct & Instruct & 1B & Link \\ sarvamai/sarvam-2b-v0.5 & Base & 2B & Link \\ google/gemma-2-2b & Base & 2B & Link \\ google/gemma-2-2b-it & Instruct & 2B & Link \\ meta-llama/Llama-3.2-3B & Base & 3B & Link \\ meta-llama/Llama-3.2-3B-Instruct & Instruct & 3B & Link \\ Telugu-LLM-Labs/Indic-gamma-7b-finetuned-sft-Navarasa-2.0 & Instruct & 7B & Link \\ CoherenceForAI/aya-23-8B & Base & 8B & Link \\ meta-llama/Llama-3.1-8B-Instruct & Instruct & 8B & Link \\ google/gemma-2-9b & Base & 9B & Link \\ google/gemma-2-9b-it & Instruct & 9B & Link \\ google/gemma-2-27b & Base & 27B & Link \\ Google/gemma-2-27b-it & Instruct & 27B & Link \\ CohereForAI/aya-23-35B & Base & 35B & Link \\ meta-llama/Llama-3.1-70B & Base & 70B & Link \\ meta-llama/Llama-3.1-70B-Instruct & Instruct & 70B & Link \\ meta-llama/Llama-3.1-405B & Base & 405B & Link \\ \hline sarvamai/OpenHathi-7B-Hi-v0.1-Base & Base & 7B & Link \\ ai4bharat/Airavata & Instruct & 7B & Link \\ BhabhaAl/Gajendra-v0.1 & Instruct & 7B & Link \\ manishitig/open-aditi-v6-llama3 & Instruct & 8B & Link \\ GenVRadmin/Aryabhatta-GammaGenZ-Vikas-Merged & Instruct & 7B & Link \\ nickmalhotra/ProjectIndus & Base & 1B & Link \\ abhinand/telugu-llama-7b-instruct-v0.1 & Instruct & 7B & Link \\ Telugu-LLM-Labs/Telugu-Lambda2-7B-v0-Base & Base & 7B & Link \\ Telugu-LLM-Labs/Telugu-Libama2-7B-v0-Instruct & Instruct & 7B & Link \\ Tensoic/Kan-LLaMA-7B-base & Base & 7B & Link \\ Cognitive-Lab/Ambari-7B-base-v0.1 & Base & 7B & Link \\ Cognitive-Lab/Ambari-7B-Insruct-v0.1 & Instruct & 7B & Link \\ smallstepai/Misal-7B-base-v0.1 & Base & 7B & Link \\ smallstepai/Misal-7B-instruct-v0.1 & Instruct & 7B & Link \\ abhinantand/tamil-llama-7b-instruct-v0.2 & Instruct & 7B & Link \\ abhinand/malayalam-llama-7b-instruct-v0.1 & Instruct & 7B & Link \\ \hline \end{tabular}
\end{table}
Table 10: Details about the different models evaluated on MILU.




\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **ta** \\ \hline Agriculture & 29.00/ 28.00/ 26.00 Anthropology & 21.00/ 23.00/ 26.00 Architecture and Design & 19.00/ 24.00/ 20.00 Arts and Culture & 31.52/ 32.73/ 29.70 Astronomy and Astrophysics & 22.00/ 32.00/ 32.00 Biology & 24.20/ 19.43/ 24.52 Business and Management & 36.00/ 28.00/ 24.00 Chemistry & 23.74/ 24.04/ 23.44 Computer Science & 28.49/ 25.00/ 26.74 Defense and Security & 29.00/ 24.00/ 32.00 Earth Sciences & 28.00/ 25.00/ 28.00 Economics & 24.60/ 20.63/ 19.05 Education & 28.00/ 24.00/ 26.00 Energy and Power & 29.00/ 20.00/ 17.00 Engineering & 26.60/ 26.00/ 25.20 Environmental Science & 23.00/ 19.00/ 20.00 Ethics and Human Rights & 31.00/ 27.00/ 29.00 Finance and Investment & 27.93/ 27.93/ 23.46 Food Science & 27.00/ 21.00/ 21.00 Geography & 22.86/ 26.29/ 24.57 Health and Medicine & 22.41/ 24.14/ 24.14 History & 26.18/ 24.73/ 24.00 Information Technology & 28.00/ 23.00/ 25.00 International Relations & 26.00/ 28.00/ 28.00 Language Studies & 24.72/ 24.34/ 25.47 Law and Ethics & 21.26/ 26.77/ 22.83 Literature and Linguistics & 26.32/ 25.51/ 23.48 Logical Reasoning & 20.72/ 21.12/ 25.10 Materials Science & 25.00/ 19.00/ 18.00 Media and Communication & 18.00/ 19.00/ 19.00 Music and Performing Arts & 25.00/ 27.00/ 27.00 Physics & 22.60/ 24.40/ 24.20 Politics and Governance & 29.05/ 31.12/ 30.29 Psychology & 32.00/ 28.00/ 27.00 Public Administration & 26.00/ 21.00/ 19.00 Religion and Spirituality & 29.00/ 31.00/ 33.00 Social Welfare and Development & 21.00/ 23.00/ 21.00 Sociology & 19.82/ 22.52/ 23.42 Sports and Recreation & 32.05/ 30.13/ 30.77 Technology and Innovation & 21.00/ 24.00/ 29.00 Transportation and Logistics & 23.00/ 29.00/ 27.00 \\ \hline \hline \end{tabular}
\end{table}
Table 11: Detailed subject-wise evaluation for abhinand/tamil-llama-7b-instruct-v0.2 on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **hi** \\ \hline Agriculture & 26.56/ 26.96/ 29.98 \\ Anthropology & 30.30/ 26.26/ 30.30 \\ Architecture and Design & 28.00/ 31.00/ 26.00 \\ Arts and Culture & 25.45/ 24.65/ 27.45 \\ Astronomy and Astrophysics & 24.22/ 29.69/ 29.69 \\ Biology & 25.05/ 32.06/ 32.06 \\ Business and Management & 27.87/ 29.51/ 26.70 \\ Chemistry & 29.66/ 31.46/ 31.26 \\ Computer Science & 28.60/ 30.60/ 30.60 \\ Defense and Security & 28.91/ 22.27/ 24.64 \\ Earth Sciences & 26.05/ 27.45/ 26.25 \\ Economics & 26.81/ 25.20/ 27.02 \\ Education & 27.22/ 25.15/ 26.04 \\ Energy and Power & 31.53/ 29.83/ 31.53 \\ Engineering & 24.00/ 25.60/ 26.40 \\ Environmental Science & 23.65/ 25.85/ 25.65 \\ Ethics and Human Rights & 33.00/ 25.00/ 29.00 \\ Finance and Investment & 26.60/ 28.20/ 30.20 \\ Food Science & 24.24/ 31.31/ 33.33 \\ Geography & 26.31/ 27.11/ 26.71 \\ Health and Medicine & 29.06/ 32.87/ 32.67 \\ History & 27.02/ 28.63/ 27.62 \\ Information Technology & 32.96/ 34.64/ 32.96 \\ International Relations & 31.91/ 31.91/ 31.52 \\ Language Studies & 25.05/ 28.86/ 25.25 \\ Law and Ethics & 29.20/ 29.40/ 29.20 \\ Literature and Linguistics & 32.87/ 31.86/ 30.06 \\ Logical Reasoning & 26.45/ 27.45/ 27.25 \\ Materials Science & 24.67/ 28.60/ 25.55 \\ Media and Communication & 23.81/ 25.85/ 24.49 \\ Music and Performing Arts & 25.17/ 24.50/ 25.83 \\ Physics & 26.20/ 25.20/ 23.40 \\ Politics and Governance & 28.71/ 27.71/ 26.91 \\ Psychology & 19.38/ 23.26/ 23.26 \\ Public Administration & 39.39/ 41.41/ 40.40 \\ Religion and Spirituality & 27.82/ 29.03/ 29.44 \\ Social Welfare and Development & 22.28/ 23.32/ 27.98 \\ Sociology & 23.40/ 23.20/ 24.20 \\ Sports and Recreation & 29.40/ 30.20/ 31.60 \\ Technology and Innovation & 26.20/ 28.60/ 30.60 \\ Transportation and Logistics & 25.87/ 26.50/ 27.13 \\ \hline \hline \end{tabular}
\end{table}
Table 12: Detailed subject-wise evaluation for AI4Bharat/Airavata on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **hi** \\ \hline Agriculture & 27.97/ 27.36/ 28.97 \\ Anthropology & 27.27/ 26.26/ 20.20 \\ Architecture and Design & 23.00/ 20.00/ 25.00 \\ Arts and Culture & 28.06/ 28.26/ 24.25 \\ Astronomy and Astrophysics & 27.34/ 30.47/ 35.94 \\ Biology & 31.06/ 30.06/ 30.46 \\ Business and Management & 26.23/ 24.12/ 29.27 \\ Chemistry & 28.26/ 25.65/ 30.06 \\ Computer Science & 25.80/ 28.80/ 29.80 \\ Defense and Security & 23.22/ 30.81/ 28.91 \\ Earth Sciences & 23.65/ 26.45/ 29.46 \\ Economics & 27.02/ 27.82/ 29.44 \\ Education & 26.33/ 28.99/ 31.07 \\ Energy and Power & 27.80/ 26.44/ 29.83 \\ Engineering & 20.40/ 20.20/ 24.00 \\ Environmental Science & 25.45/ 26.25/ 30.06 \\ Ethics and Human Rights & 27.00/ 24.00/ 24.00 \\ Finance and Investment & 23.60/ 25.00/ 29.20 \\ Food Science & 32.32/ 37.37/ 28.28 \\ Geography & 26.71/ 25.90/ 28.71 \\ Health and Medicine & 30.46/ 30.46/ 30.66 \\ History & 26.01/ 28.02/ 29.03 \\ Information Technology & 27.93/ 31.28/ 36.87 \\ International Relations & 25.68/ 25.29/ 24.12 \\ Language Studies & 23.05/ 24.85/ 26.65 \\ Law and Ethics & 26.60/ 26.60/ 28.80 \\ Literature and Linguistics & 27.45/ 27.25/ 29.26 \\ Logical Reasoning & 22.24/ 27.05/ 26.05 \\ Materials Science & 26.86/ 29.26/ 29.48 \\ Media and Communication & 23.81/ 26.53/ 32.65 \\ Music and Performing Arts & 25.17/ 22.52/ 28.48 \\ Physics & 24.20/ 25.40/ 27.80 \\ Politics and Governance & 23.09/ 24.70/ 27.11 \\ Psychology & 26.36/ 25.58/ 25.58 \\ Public Administration & 36.36/ 34.34/ 32.32 \\ Religion and Spirituality & 25.81/ 22.98/ 29.03 \\ Social Welfare and Development & 25.91/ 24.87/ 25.91 \\ Sociology & 26.20/ 25.60/ 24.80 \\ Sports and Recreation & 26.00/ 26.40/ 26.60 \\ Technology and Innovation & 28.80/ 28.20/ 30.40 \\ Transportation and Logistics & 25.24/ 23.97/ 24.29 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Detailed subject-wise evaluation for BhabhaAI/Gajandra-v0.1 on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **kn** \\ \hline Agriculture & 25.00/ 28.00/ 23.00 Anthropology & 25.00/ 26.00/ 26.00 \\ Architecture and Design & 25.00/ 31.00/ 28.00 Arts and Culture & 26.99/ 29.45/ 24.54 Astronomy and Astrophysics & 28.00/ 30.00/ 30.00 Biology & 29.10/ 26.23/ 28.28 Business and Management & 36.00/ 25.00/ 26.00 Chemistry & 31.37/ 25.82/ 28.43 Computer Science & 28.40/ 33.14/ 33.14 Defense and Security & 27.00/ 31.00/ 33.00 Earth Sciences & 29.00/ 31.00/ 33.00 Economics & 29.00/ 32.00/ 37.00 Education & 28.00/ 30.00/ 29.00 Energy and Power & 26.00/ 22.00/ 19.00 Engineering & 24.60/ 27.20/ 28.20 Environmental Science & 21.88/ 23.44/ 29.69 Ethics and Human Rights & 29.00/ 19.00/ 26.00 Finance and Investment & 29.63/ 28.40/ 31.48 Food Science & 23.00/ 30.00/ 38.00 Geography & 27.85/ 27.40/ 28.77 Health and Medicine & 23.00/ 29.00/ 24.00 History & 24.54/ 28.57/ 29.67 Information Technology & 26.00/ 35.00/ 31.00 International Relations & 32.00/ 36.00/ 26.00 Language Studies & 28.68/ 19.38/ 27.91 Law and Ethics & 25.48/ 22.29/ 26.75 Literature and Linguistics & 18.33/ 23.33/ 23.33 Logical Reasoning & 21.99/ 21.58/ 25.73 Materials Science & 28.00/ 24.00/ 23.00 Media and Communication & 32.00/ 22.00/ 28.00 Music and Performing Arts & 32.00/ 29.00/ 30.00 Physics & 26.40/ 26.60/ 26.00 Politics and Governance & 27.53/ 23.89/ 25.51 Psychology & 23.00/ 34.00/ 31.00 Public Administration & 36.36/ 30.30/ 32.32 Religion and Spirituality & 27.00/ 27.00/ 31.00 Social Welfare and Development & 28.00/ 26.00/ 32.00 Sociology & 24.00/ 33.00/ 30.00 Sports and Recreation & 22.47/ 29.78/ 27.53 Technology and Innovation & 25.00/ 29.00/ 32.00 Transportation and Logistics & 25.25/ 26.26/ 29.29 \\ \hline \hline \end{tabular}
\end{table}
Table 14: Detailed subject-wise evaluation for Cognitive-Lab/Ambari-7B-base-v0.1 on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **kn** \\ \hline Agriculture & 31.00/ 26.00/ 23.00 Anthropology & 29.00/ 23.00/ 22.00 Architecture and Design & 29.00/ 21.00/ 23.00 Arts and Culture & 25.15/ 28.83/ 30.06 Astronomy and Astrophysics & 29.00/ 24.00/ 33.00 Biology & 27.87/ 25.82/ 29.10 Business and Management & 38.00/ 31.00/ 28.00 Chemistry & 31.37/ 30.07/ 31.05 Computer Science & 24.26/ 25.44/ 30.77 Defense and Security & 27.00/ 22.00/ 19.00 Earth Sciences & 26.00/ 28.00/ 27.00 Economics & 31.00/ 32.00/ 29.00 Education & 32.00/ 30.00/ 32.00 Energy and Power & 26.00/ 21.00/ 26.00 Engineering & 23.20/ 26.20/ 28.20 Environmental Science & 25.78/ 28.12/ 26.56 Ethics and Human Rights & 29.00/ 25.00/ 28.00 Finance and Investment & 23.46/ 25.31/ 27.78 Food Science & 22.00/ 21.00/ 33.00 Geography & 31.51/ 24.20/ 31.05 Health and Medicine & 17.00/ 22.00/ 21.00 History & 27.11/ 28.57/ 30.04 Information Technology & 33.00/ 32.00/ 35.00 International Relations & 33.00/ 32.00/ 35.00 International Relations & 35.00/ 26.00/ 26.00 Language Studies & 22.00/ 27.00/ 26.00 Language Studies & 25.58/ 30.23/ 23.26 Law and Ethics & 28.03/ 23.57/ 21.66 Literature and Linguistics & 23.33/ 22.50/ 24.17 Logical Reasoning & 21.16/ 21.16/ 24.07 Materials Science & 22.00/ 29.00/ 26.00 Media and Communication & 30.00/ 25.00/ 26.00 Music and Performing Arts & 29.00/ 27.00/ 25.00 Physics & 26.40/ 23.60/ 24.20 Politics and Governance & 22.67/ 29.15/ 25.51 Psychology & 17.00/ 18.00/ 25.00 Public Administration & 24.24/ 23.23/ 28.28 Religion and Spirituality & 27.00/ 26.00/ 29.00 Social Welfare and Development & 29.00/ 25.00/ 27.00 Sociology & 18.00/ 19.00/ 22.00 Sports and Recreation & 19.10/ 24.16/ 24.16 Technology and Innovation & 32.00/ 23.00/ 25.00 Transportation and Logistics & 32.32/ 31.31/ 27.27 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Detailed subject-wise evaluation for CognitiveLab/Ambari-7B-Instruct-v0.1 on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **hi** \\ \hline Agriculture & 29.38/ 30.18/ 36.62 \\ Anthropology & 37.37/ 32.32/ 40.40 \\ Architecture and Design & 26.00/ 28.00/ 35.00 \\ Arts and Culture & 26.25/ 29.46/ 40.48 \\ Astronomy and Astrophysics & 37.50/ 42.19/ 46.88 \\ Biology & 35.27/ 35.07/ 50.30 \\ Business and Management & 31.15/ 32.32/ 38.64 \\ Chemistry & 28.86/ 38.68/ 48.30 \\ Computer Science & 32.60/ 34.00/ 36.60 \\ Defense and Security & 28.91/ 36.02/ 35.55 \\ Earth Sciences & 32.06/ 28.66/ 40.48 \\ Economics & 27.42/ 32.06/ 41.53 \\ Education & 23.96/ 27.51/ 30.77 \\ Energy and Power & 26.10/ 31.19/ 43.39 \\ Engineering & 25.80/ 26.60/ 33.20 \\ Environmental Science & 29.66/ 30.26/ 44.89 \\ Ethics and Human Rights & 30.00/ 31.00/ 41.00 \\ Finance and Investment & 27.40/ 34.20/ 38.00 \\ Food Science & 24.24/ 33.33/ 47.47 \\ Geography & 31.33/ 30.72/ 42.57 \\ Health and Medicine & 34.27/ 34.67/ 48.70 \\ History & 27.02/ 32.86/ 36.90 \\ Information Technology & 34.64/ 37.99/ 50.84 \\ International Relations & 31.52/ 31.13/ 42.41 \\ Language Studies & 22.65/ 27.05/ 29.26 \\ Law and Ethics & 28.00/ 29.80/ 38.60 \\ Literature and Linguistics & 24.85/ 28.86/ 33.47 \\ Logical Reasoning & 25.45/ 27.25/ 29.66 \\ Materials Science & 31.88/ 30.35/ 35.59 \\ Media and Communication & 25.85/ 26.53/ 40.82 \\ Music and Performing Arts & 21.19/ 30.46/ 33.77 \\ Physics & 32.60/ 32.80/ 40.00 \\ Politics and Governance & 25.50/ 27.71/ 41.97 \\ Psychology & 23.26/ 22.48/ 42.64 \\ Public Administration & 30.30/ 27.27/ 36.36 \\ Religion and Spirituality & 28.23/ 33.87/ 42.34 \\ Social Welfare and Development & 26.94/ 27.98/ 35.75 \\ Sociology & 23.80/ 26.60/ 28.00 \\ Sports and Recreation & 29.00/ 34.80/ 43.80 \\ Technology and Innovation & 31.80/ 30.80/ 41.60 \\ Transportation and Logistics & 23.97/ 23.97/ 30.28 \\ \hline \hline \end{tabular}
\end{table}
Table 16: Detailed subject-wise evaluation for GenVRadmin/AryaBhatta-GemmaGenZ-Vikas-Merged on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **hi** \\ \hline Agriculture & 29.38/ 28.97/ 27.16 Anthropology & 36.36/ 36.36/ 28.28 Architecture and Design & 34.00/ 31.00/ 33.00 Arts and Culture & 34.07/ 33.67/ 34.87 Astronomy and Astrophysics & 42.97/ 41.41/ 37.50 Biology & 36.67/ 38.08/ 36.07 Business and Management & 32.55/ 31.15/ 29.04 Chemistry & 37.47/ 34.67/ 31.66 Computer Science & 37.40/ 34.00/ 35.80 Defense and Security & 30.33/ 28.44/ 27.49 Earth Sciences & 31.86/ 31.46/ 33.07 Economics & 34.68/ 33.06/ 30.44 Education & 31.36/ 29.88/ 30.77 Energy and Power & 34.58/ 32.20/ 32.54 Engineering & 28.60/ 27.60/ 27.80 Environmental Science & 27.60/ 27.80 Environmental Science & 31.86/ 31.26/ 30.26 Ethics and Human Rights & 36.00/ 35.00/ 36.00 Finance and Investment & 29.80/ 30.20/ 28.40 Food Science & 36.36/ 36.36/ 28.28 Geography & 33.33/ 32.73/ 30.12 Health and Medicine & 44.09/ 40.28/ 38.88 History & 39.92/ 38.10/ 36.49 Information Technology & 47.49/ 47.49/ 43.02 International Relations & 40.08/ 40.47/ 34.63 Language Studies & 29.86/ 29.06/ 27.45 Law and Ethics & 41.60/ 39.20/ 40.00 Literature and Linguistics & 29.66/ 30.26/ 29.86 Logical Reasoning & 30.46/ 28.46/ 29.06 Materials Science & 29.26/ 31.00/ 30.35 Media and Communication & 36.73/ 34.01/ 31.29 Music and Performing Arts & 36.42/ 32.45/ 34.44 Physics & 33.60/ 29.80/ 29.00 Politics and Governance & 36.95/ 36.75/ 32.33 Psychology & 41.86/ 37.21/ 33.33 Public Administration & 32.32/ 33.33/ 34.34 Religion and Spirituality & 29.84/ 31.45/ 29.44 Social Welfare and Development & 30.05/ 30.57/ 30.57 Sociology & 30.00/ 30.00/ 27.40 Sports and Recreation & 32.00/ 30.80/ 28.60 Technology and Innovation & 36.60/ 36.80/ 34.00 Transportation and Logistics & 30.60/ 29.65/ 28.71 \\ \hline \hline \end{tabular}
\end{table}
Table 17: Detailed subject-wise evaluation for manishinity/open-aditi-v6-llama3 on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **hi** \\ \hline Agriculture & 22.74/ 22.74/ 21.93 \\ Anthropology & 25.25/ 24.24/ 24.24 \\ Architecture and Design & 32.00/ 28.00/ 29.00 \\ Arts and Culture & 24.85/ 26.05/ 26.25 \\ Astronomy and Astrophysics & 35.94/ 35.94/ 32.03 \\ Biology & 24.85/ 24.45/ 24.05 \\ Business and Management & 25.29/ 25.06/ 25.76 \\ Chemistry & 25.85/ 25.25/ 24.45 \\ Computer Science & 26.40/ 27.00/ 25.20 \\ Defense and Security & 27.01/ 24.64/ 25.12 \\ Earth Sciences & 25.45/ 24.05/ 23.85 \\ Economics & 28.02/ 27.42/ 27.42 \\ Education & 26.04/ 24.85/ 24.85 \\ Energy and Power & 23.73/ 24.07/ 25.08 \\ Engineering & 23.60/ 23.40/ 24.00 \\ Environmental Science & 28.86/ 28.26/ 27.86 \\ Ethics and Human Rights & 30.00/ 31.00/ 22.00 \\ Finance and Investment & 24.00/ 25.00/ 25.40 \\ Food Science & 27.27/ 29.29/ 27.27 \\ Geography & 27.11/ 25.90/ 26.51 \\ Health and Medicine & 25.45/ 26.05/ 23.85 \\ History & 24.80/ 25.81/ 26.21 \\ Information Technology & 26.26/ 27.37/ 25.14 \\ International Relations & 26.85/ 28.40/ 29.96 \\ Language Studies & 21.44/ 21.44/ 22.24 \\ Law and Ethics & 24.80/ 24.80/ 22.80 \\ Literature and Linguistics & 23.65/ 23.85/ 22.24 \\ Logical Reasoning & 27.05/ 27.86/ 27.45 \\ Materials Science & 28.17/ 29.04/ 28.60 \\ Media and Communication & 28.57/ 31.29/ 26.53 \\ Music and Performing Arts & 21.19/ 22.52/ 23.84 \\ Physics & 24.40/ 24.00/ 21.60 \\ Politics and Governance & 26.71/ 25.30/ 22.29 \\ Psychology & 27.91/ 27.91/ 26.36 \\ Public Administration & 26.26/ 27.27/ 28.28 \\ Religion and Spirituality & 26.21/ 29.44/ 27.82 \\ Social Welfare and Development & 26.94/ 27.98/ 26.42 \\ Sociology & 27.00/ 28.00/ 26.40 \\ Sports and Recreation & 27.80/ 27.40/ 26.60 \\ Technology and Innovation & 23.80/ 25.20/ 26.40 \\ Transportation and Logistics & 23.97/ 26.81/ 24.92 \\ \hline \hline \end{tabular}
\end{table}
Table 18: Detailed subject-wise evaluation for nickmalhotra/projectindus on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **hi** \\ \hline Agriculture & 24.35/ 30.18/ 29.98 \\ Anthropology & 20.20/ 25.25/ 26.26 \\ Architecture and Design & 23.00/ 32.00/ 27.00 \\ Arts and Culture & 27.66/ 27.05/ 28.06 \\ Astronomy and Astrophysics & 24.22/ 31.25/ 33.59 \\ Biology & 27.45/ 26.65/ 28.86 \\ Business and Management & 26.70/ 25.53/ 29.74 \\ Chemistry & 26.65/ 26.25/ 28.86 \\ Computer Science & 26.80/ 29.20/ 25.80 \\ Defense and Security & 27.96/ 24.64/ 27.96 \\ Earth Sciences & 22.04/ 24.65/ 26.85 \\ Economics & 26.21/ 30.85/ 33.27 \\ Education & 24.26/ 26.92/ 25.15 \\ Energy and Power & 30.17/ 31.19/ 32.54 \\ Engineering & 22.20/ 21.40/ 25.40 \\ Environmental Science & 25.05/ 25.65/ 27.05 \\ Ethics and Human Rights & 31.00/ 33.00/ 36.00 \\ Finance and Investment & 27.00/ 27.80/ 26.20 \\ Food Science & 20.20/ 24.24/ 30.30 \\ Geography & 24.30/ 27.91/ 27.11 \\ Health and Medicine & 33.07/ 28.66/ 30.86 \\ History & 29.44/ 30.24/ 32.66 \\ Information Technology & 25.70/ 30.17/ 35.20 \\ International Relations & 26.46/ 28.40/ 32.30 \\ Language Studies & 25.45/ 24.85/ 28.46 \\ Law and Ethics & 28.60/ 28.80/ 34.20 \\ Literature and Linguistics & 26.25/ 25.85/ 27.45 \\ Logical Reasoning & 28.86/ 23.85/ 28.26 \\ Materials Science & 22.93/ 24.45/ 25.76 \\ Media and Communication & 29.93/ 27.89/ 24.49 \\ Music and Performing Arts & 27.15/ 24.50/ 24.50 \\ Physics & 26.00/ 27.20/ 29.60 \\ Politics and Governance & 23.49/ 24.90/ 27.91 \\ Psychology & 24.81/ 23.26/ 26.36 \\ Public Administration & 29.29/ 33.33/ 32.32 \\ Religion and Spirituality & 28.63/ 26.21/ 30.24 \\ Social Welfare and Development & 27.46/ 25.91/ 26.42 \\ Sociology & 28.80/ 28.40/ 25.80 \\ Sports and Recreation & 28.80/ 29.00/ 31.40 \\ Technology and Innovation & 29.00/ 29.60/ 30.40 \\ Transportation and Logistics & 29.02/ 24.29/ 27.44 \\ \hline \hline \end{tabular}
\end{table}
Table 19: Detailed subject-wise evaluation for sarvamai/OpenHathi-7B-Hi-vo.1-Base on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **kn** \\ \hline Agriculture & 23.00/ 29.00/ 28.00 \\ Anthropology & 30.00/ 41.00/ 38.00 \\ Architecture and Design & 26.00/ 27.00/ 25.00 \\ Arts and Culture & 26.38/ 26.99/ 29.45 \\ Astronomy and Astrophysics & 26.00/ 30.00/ 21.00 \\ Biology & 23.77/ 29.51/ 28.69 \\ Business and Management & 30.00/ 28.00/ 28.00 \\ Chemistry & 28.10/ 27.12/ 25.82 \\ Computer Science & 29.59/ 36.09/ 36.69 \\ Defense and Security & 25.00/ 26.00/ 28.00 \\ Earth Sciences & 24.00/ 24.00/ 24.00 \\ Economics & 30.00/ 32.00/ 29.00 \\ Education & 28.00/ 38.00/ 36.00 \\ Energy and Power & 28.00/ 22.00/ 25.00 \\ Engineering & 23.00/ 23.60/ 24.80 \\ Environmental Science & 23.44/ 32.81/ 30.47 \\ Ethics and Human Rights & 24.00/ 28.00/ 30.00 \\ Finance and Investment & 26.54/ 31.48/ 29.63 \\ Food Science & 27.00/ 29.00/ 31.00 \\ Geography & 32.88/ 31.96/ 26.94 \\ Health and Medicine & 17.00/ 27.00/ 30.00 \\ History & 27.11/ 26.01/ 31.14 \\ Information Technology & 33.00/ 37.00/ 35.00 \\ International Relations & 25.00/ 25.00/ 29.00 \\ Language Studies & 26.36/ 19.38/ 24.03 \\ Law and Ethics & 26.11/ 26.75/ 24.84 \\ Literature and Linguistics & 22.50/ 25.00/ 21.67 \\ Logical Reasoning & 22.41/ 23.24/ 22.82 \\ Materials Science & 25.00/ 26.00/ 24.00 \\ Media and Communication & 28.00/ 32.00/ 32.00 \\ Music and Performing Arts & 30.00/ 28.00/ 29.00 \\ Physics & 26.20/ 28.20/ 26.60 \\ Politics and Governance & 26.72/ 29.55/ 30.36 \\ Psychology & 29.00/ 29.00/ 25.00 \\ Public Administration & 31.31/ 29.29/ 31.31 \\ Religion and Spirituality & 26.00/ 27.00/ 35.00 \\ Social Welfare and Development & 32.00/ 34.00/ 31.00 \\ Sociology & 29.00/ 28.00/ 30.00 \\ Sports and Recreation & 26.40/ 30.90/ 29.21 \\ Technology and Innovation & 29.00/ 30.00/ 34.00 \\ Transportation and Logistics & 36.36/ 31.31/ 32.32 \\ \hline \hline \end{tabular}
\end{table}
Table 20: Detailed subject-wise evaluation for Tensoic/Kan-LLAMA-7B-base on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **ml** \\ \hline Agriculture & 22.22/ 18.18/ 25.25 Anthropology & 24.00/ 20.00/ 21.00 Architecture and Design & 22.00/ 21.00/ 23.00 Arts and Culture & 25.00/ 20.00/ 23.00 Astronomy and Astrophysics & 27.00/ 28.00/ 26.00 Biology & 24.24/ 27.27/ 26.26 Business and Management & 26.00/ 24.00/ 28.00 Chemistry & 25.00/ 26.00/ 26.00 Computer Science & 24.00/ 28.00/ 27.00 Defense and Security & 18.00/ 18.00/ 18.00 Earth Sciences & 30.00/ 31.00/ 34.00 Economics & 23.00/ 26.00/ 27.00 Education & 23.00/ 21.00/ 22.00 Energy and Power & 32.00/ 28.00/ 30.00 Engineering & 27.35/ 27.78/ 25.64 Environmental Science & 30.00/ 31.00/ 28.00 Ethics and Human Rights & 29.00/ 19.00/ 24.00 Finance and Investment & 35.00/ 31.00/ 35.00 Food Science & 27.00/ 22.00/ 20.00 Geography & 30.30/ 27.27/ 24.24 Health and Medicine & 23.00/ 23.00/ 20.00 History & 30.00/ 22.00/ 25.00 Information Technology & 35.00/ 37.00/ 37.00 International Relations & 23.00/ 25.00/ 24.00 Language Studies & 21.00/ 17.00/ 20.00 Law and Ethics & 27.00/ 30.00/ 27.00 Literature and Linguistics & 28.00/ 24.00/ 25.00 Logical Reasoning & 29.00/ 26.00/ 27.00 Materials Science & 28.00/ 25.00/ 20.00 Media and Communication & 24.00/ 24.00/ 22.00 Music and Performing Arts & 25.00/ 21.00/ 24.00 Physics & 25.79/ 27.89/ 30.53 Politics and Governance & 30.00/ 28.00/ 31.00 Psychology & 26.00/ 25.00/ 24.00 Public Administration & 23.00/ 25.00/ 20.00 Religion and Spirituality & 27.00/ 24.00/ 26.00 Social Welfare and Development & 34.00/ 33.00/ 30.00 Sociology & 20.00/ 21.00/ 20.00 Sports and Recreation & 31.00/ 27.00/ 25.00 Technology and Innovation & 25.00/ 26.00/ 26.00 Transportation and Logistics & 24.00/ 25.00/ 32.00 \\ \hline \hline \end{tabular}
\end{table}
Table 21: Detailed subject-wise evaluation for abhinand/malayalam-llama-7b-instruct-v0.1 on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **te** \\ \hline Agriculture & 25.00/ 23.00/ 21.00 Anthropology & 33.00/ 30.00/ 33.00 Architecture and Design & 24.00/ 20.00/ 20.00 Arts and Culture & 28.64/ 29.65/ 28.14 Astronomy and Astrophysics & 25.00/ 25.00/ 25.00 Biology & 24.53/ 23.27/ 22.01 Business and Management & 36.79/ 36.79/ 39.62 Chemistry & 25.69/ 27.43/ 27.93 Computer Science & 25.56/ 30.04/ 28.70 Defense and Security & 26.00/ 25.00/ 25.00 Earth Sciences & 27.00/ 29.00/ 29.00 Economics & 24.55/ 28.74/ 29.94 Education & 22.00/ 22.00/ 28.00 Energy and Power & 26.00/ 26.00/ 24.00 Engineering & 30.40/ 29.20/ 28.00 Environmental Science & 21.23/ 23.46/ 23.46 Ethics and Human Rights & 21.00/ 22.00/ 20.00 Finance and Investment & 26.79/ 24.64/ 26.79 Food Science & 28.00/ 25.00/ 21.00 Geography & 28.65/ 28.12/ 23.44 Health and Medicine & 28.04/ 30.84/ 28.04 History & 31.71/ 32.48/ 29.67 Information Technology & 30.00/ 38.00/ 33.00 International Relations & 30.00/ 38.00/ 33.00 International Relations & 27.00/ 29.00/ 31.00 Language Studies & 27.01/ 28.16/ 27.59 Law and Ethics & 27.50/ 24.37/ 26.87 Literature and Linguistics & 25.35/ 21.83/ 18.31 Logical Reasoning & 24.75/ 24.50/ 25.25 Materials Science & 26.00/ 24.00/ 27.00 Media and Communication & 23.00/ 23.00/ 25.00 Music and Performing Arts & 28.00/ 23.00/ 30.00 Physics & 22.80/ 25.20/ 28.20 Politics and Governance & 27.59/ 27.59/ 29.48 Psychology & 28.00/ 22.00/ 22.00 Public Administration & 35.00/ 31.00/ 28.00 Religion and Spirituality & 22.00/ 24.00/ 24.00 Social Welfare and Development & 30.00/ 27.00/ 29.00 Sociology & 21.18/ 20.59/ 18.82 Sports and Recreation & 24.76/ 25.71/ 26.67 Technology and Innovation & 29.00/ 24.00/ 22.00 Transportation and Logistics & 26.00/ 22.00/ 26.00 \\ \hline \hline \end{tabular}
\end{table}
Table 22: Detailed subject-wise evaluation for abhinand/telugu-llama-7b-instruct-v0.1 on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **mr** \\ \hline Agriculture & 20.00/ 20.00/ 17.00 Anthropology & 15.00/ 20.00/ 20.00 Architecture and Design & 18.00/ 19.00/ 19.00 Arts and Culture & 20.22/ 20.77/ 22.95 Astronomy and Astrophysics & 20.00/ 19.00/ 19.00 Biology & 24.18/ 23.58/ 23.28 Business and Management & 20.00/ 21.00/ 22.00 Chemistry & 23.75/ 25.33/ 25.59 Computer Science & 26.63/ 25.44/ 27.22 Defense and Security & 24.00/ 22.00/ 19.00 Earth Sciences & 23.00/ 29.00/ 28.00 Economics & 28.17/ 26.76/ 28.17 Education & 24.20/ 23.57/ 22.29 Energy and Power & 27.00/ 25.00/ 22.00 Engineering & 21.60/ 21.60/ 22.20 Environmental Science & 23.16/ 23.16/ 26.84 Ethics and Human Rights & 20.00/ 18.00/ 17.00 Finance and Investment & 23.16/ 24.29/ 24.29 Food Science & 25.00/ 27.00/ 21.00 Geography & 21.30/ 20.71/ 18.93 Health and Medicine & 20.54/ 27.68/ 23.21 History & 24.91/ 24.54/ 25.65 Information Technology & 25.00/ 24.00/ 23.00 International Relations & 24.00/ 27.00/ 23.00 Language Studies & 21.48/ 23.49/ 23.49 Law and Ethics & 21.12/ 21.55/ 23.71 Literature and Linguistics & 20.35/ 23.01/ 22.12 Logical Reasoning & 26.18/ 24.79/ 26.74 Materials Science & 16.00/ 19.00/ 16.00 Media and Communication & 20.00/ 17.00/ 17.00 Music and Performing Arts & 27.00/ 27.00/ 31.00 Physics & 25.00/ 24.40/ 23.40 Politics and Governance & 21.07/ 20.22/ 22.75 Psychology & 24.00/ 25.00/ 24.00 Public Administration & 24.24/ 25.25/ 18.18 Religion and Spirituality & 19.00/ 22.00/ 24.00 Social Welfare and Development & 18.00/ 19.00/ 22.00 Sociology & 18.24/ 19.50/ 21.38 Sports and Recreation & 25.99/ 22.03/ 22.03 Technology and Innovation & 18.00/ 20.00/ 18.00 Transportation and Logistics & 21.43/ 26.53/ 24.49 \\ \hline \hline \end{tabular}
\end{table}
Table 23: Detailed subject-wise evaluation for smallestpi/Misal-7B-instruct-v0.1 on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **mr** \\ \hline Agriculture & 34.00/ 28.00/ 26.00 Anthropology & 27.00/ 35.00/ 37.00 Architecture and Design & 21.00/ 18.00/ 19.00 Arts and Culture & 25.68/ 25.14/ 26.23 Astronomy and Astrophysics & 25.00/ 23.00/ 24.00 Biology & 23.88/ 25.97/ 25.07 Business and Management & 19.00/ 27.00/ 26.00 Chemistry & 29.82/ 27.70/ 29.02 Computer Science & 20.12/ 24.26/ 23.08 Defense and Security & 23.00/ 26.00/ 22.00 Earth Sciences & 30.00/ 28.00/ 25.00 Economics & 27.46/ 29.58/ 30.99 Education & \\ Energy and Power & 25.00/ 24.00/ 28.00 Engineering & 21.60/ 28.00/ 26.80 Environmental Science & 26.32/ 30.53/ 32.11 Ethics and Human Rights & 20.00/ 22.00/ 25.00 Finance and Investment & 25.99/ 22.03/ 24.29 Food Science & 19.00/ 17.00/ 21.00 Geography & 24.85/ 21.30/ 23.67 Health and Medicine & 29.46/ 33.93/ 26.79 History & 26.77/ 27.14/ 28.25 Information Technology & 19.00/ 17.00/ 21.00 International Relations & 30.00/ 30.00/ 25.00 Language Studies & 22.15/ 22.15/ 20.81 Law and Ethics & 25.00/ 29.31/ 32.33 Literature and Linguistics & 22.12/ 27.43/ 29.20 Logical Reasoning & 30.64/ 30.64/ 30.36 Materials Science & 37.00/ 27.00/ 24.00 Media and Communication & 24.00/ 31.00/ 31.00 Music and Performing Arts & 23.00/ 23.00/ 30.00 Physics & 26.00/ 24.00/ 23.00 Politics and Governance & 30.62/ 25.84/ 26.12 Psychology & 30.00/ 22.00/ 21.00 Public Administration & 29.29/ 25.25/ 25.25 Religion and Spirituality & 18.00/ 26.00/ 28.00 Social Welfare and Development & 25.00/ 18.00/ 23.00 Sociology & 25.79/ 19.50/ 16.35 Sports and Recreation & 26.55/ 24.29/ 29.38 Technology and Innovation & 28.00/ 30.00/ 31.00 Transportation and Logistics & 27.55/ 25.51/ 28.57 \\ \hline \hline \end{tabular}
\end{table}
Table 24: Detailed subject-wise evaluation for smallstepai/Misal–7B-base-v0.1 on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c} \hline \hline
**Subject** & **te** \\ \hline Agriculture & 25.00/ 21.00/ 29.00 Anthropology & 22.00/ 19.00/ 21.00 Architecture and Design & 19.00/ 14.00/ 17.00 Arts and Culture & 24.62/ 21.11/ 22.11 Astronomy and Astrophysics & 31.00/ 27.00/ 28.00 Biology & 19.81/ 19.50/ 22.33 Business and Management & 17.92/ 16.98/ 18.87 Chemistry & 27.93/ 27.43/ 29.43 Computer Science & 27.80/ 27.35/ 30.49 Defense and Security & 35.00/ 29.00/ 32.00 Earth Sciences & 31.00/ 32.00/ 31.00 Economics & 26.95/ 23.35/ 23.35 Education & 30.00/ 24.00/ 23.00 Energy and Power & 24.00/ 20.00/ 26.00 Engineering & 26.80/ 27.00/ 26.60 Environmental Science & 26.82/ 23.46/ 24.02 Ethics and Human Rights & 17.00/ 18.00/ 19.00 Finance and Investment & 25.00/ 22.50/ 24.29 Food Science & 22.00/ 24.00/ 25.00 Geography & 26.56/ 26.56/ 27.08 Health and Medicine & 25.23/ 28.04/ 29.91 History & 29.16/ 26.85/ 26.09 Information Technology & 32.00/ 30.00/ 33.00 International Relations & 16.00/ 21.00/ 21.00 Language Studies & 28.74/ 27.59/ 29.31 Law and Ethics & 27.50/ 20.62/ 23.75 Literature and Linguistics & 28.17/ 23.94/ 25.35 Logical Reasoning & 26.49/ 26.73/ 26.24 Materials Science & 24.00/ 27.00/ 31.00 Media and Communication & 33.00/ 36.00/ 34.00 Music and Performing Arts & 23.00/ 17.00/ 25.00 Physics & 26.80/ 25.40/ 27.40 Politics and Governance & 29.25/ 28.77/ 27.36 Psychology & 26.00/ 26.00/ 23.00 Public Administration & 27.00/ 32.00/ 34.00 Religion and Spirituality & 20.00/ 27.00/ 26.00 Social Welfare and Development & 32.00/ 27.00/ 30.00 Sociology & 25.29/ 24.71/ 25.88 Sports and Recreation & 21.43/ 22.38/ 21.90 Technology and Innovation & 22.00/ 21.00/ 21.00 Transportation and Logistics & 22.00/ 24.00/ 27.00 \\ \hline \hline \end{tabular}
\end{table}
Table 25: Detailed subject-wise evaluation for Telugu-LLM-Labs/Telugu-Llama2-7B-v0-Base on MILU across different languages. The results reported are X / Y / Z where X denote the 0-shot, Y denotes 1-shot and Z denotes 5-shot performance respectively.





\begin{table}
\begin{tabular}{l c r r r r r r r r r r}





\begin{table}
\begin{tabular}{l c r r r r r r r r r r}





\begin{table}
\begin{tabular}{l c r r r r r r r r r r r}





\begin{table}
\begin{tabular}{l c r r r r r r r r r r r}





\begin{table}
\begin{tabular}{l c c c c c c c c c c c}





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c}





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c}
**Topi** & **a** & **b** & **c** & **d** & **e** & **f** & **g** & **e** & **f** & **g** & **f** \\ \hline Avignature & 30.948/126 & 30.75/126 & 30.75/126 & 20.98/126 & 20.98/126 & 30.75/126 & 30.75/126 & 20.98/126 & 20.98/126 & 20.98/126 & 10.98/126 & 20.98/126 & 20.98/126 & 20.98/126 \\ Avignature & 20.98/126 & 30.75/126 & 20.98/126 & 20.98/126 & 20.98/126 & 20.98/126 & 20.98/126 & 20.98/126 & 20.98/126 & 20.98/126 & 20.98/126 & 20.98/126 \\ Avignature and Bottom & 30.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature and Bottom & 30.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature and Bottom & 30.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature and Bottom & 30.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature & 20.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature & 20.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature & 20.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature & 20.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature & 20.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature & 20.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature and Bottom & 30.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature & 20.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature & 20.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature and Bottom & 30.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature & 20.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature and Bottom & 30.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature & 30.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature & 30.98/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 & 30.75/126 \\ Avignature and Bottom & 30.98/126 & 30.75/126 & 30.75/126 & 30.





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c}
**Topi** & **a** & **b** & **c** & **d** & **e** & **f** & **g** & **e** & **f** & **g** & **h** \\ \hline Avgnance & 3619.7820 & 45.04/46.03 & 23.98/32.71 & 23.78/20.24 & 16.70/49.24 & 23.98/32.75 & 23.98/32.96 & 24.81/32.96 & 24.81/32.96 & 24.80/32.96 & 24.80/32.96 \\ Avgnancy & 20.92/32.9820 & 45.04/46.03 & 23.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Avnolence and Rupp & 23.78/32.8128 & 45.04/46.03 & 24.98/32.96 & 23.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ An All-Clave & 36.78/32.8128 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Annes and Asplines & 36.78/32.8128 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Annes and Asplines & 36.78/32.8128 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Annes and Asplines & 36.78/32.8128 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Babys & 36.89/32.8127 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Bennot Marguerti & 20.92/32.8897 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Consorites & 20.92/32.8893 & 25.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Computer Based & 20.92/32.8893 & 25.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Border Ends & 36.93/32.8128 & 36.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Real Source & 36.93/32.8128 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Exomits & 36.93/32.8128 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Exomits & 36.93/32.8128 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Exomits & 36.93/32.8128 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Exomit & 36.93/32.8128 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Empirical Results & 20.92/32.8893 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Ermes and Income & 25.32/32.8893 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Eval Home & 25.32/32.8893 & 45.04/46.03 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Emotions & 25.32/32.8893 & 45.04/46.04 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 & 24.98/32.96 \\ Goupds & 25.32/32.8893 & 45.04/45.02 & 24.98/32.9



\begin{table}
\begin{tabular}{l c c c c c c c c c c c c}
**Topi** & **a** & **b** & **c** & **d** & **e** & **f** & **g** & **u** & **e** & **f** & **g** & **u** \\ \hline Avgnance & 120/38/96.7569, 279/83/92.42 & 279/83/92.43 & 279/83/93.31 & 279/83/93.31 & 139/83/93.





\begin{table}
\begin{tabular}{l c r r r r r r r r r r r r r}
**TopTop** & **In** & **m** & **m** & **g** & **N** & **m** & **m** & **m** & **m** & **m** & **m** \\ \hline Agtenetene & 27.912.823 & 40.534.736 & 27.16 & 23.339.223 & 27.333.47 & 27.426 & 23.202.320 & 27.320.333 & 33.980.000 & 27.982.033 & 31.192.827 & 31.426.330 & 32.064.820 \\ Ardehappierey & 31.089.830 & 32.04.543 & 30.30 & 33.930.30 & 35.437.934 & 29.007.049 & 24.077.030 & 31.042.510 & 31.192.820 & 34.913.703 & 34.913.703 & 34.913.640 & 25.264.900 \\ Ascoliment and Design & 31.083.840 & 40.03.762 & 32.00 & 30.03.762 & 32.00 & 32.00 & 34.00/34.020 & 30.047.040 & 31.00 & 30.037.030 & 31.00 25.260 & 32.00 & 32.00 & 32.00 & 33.00 & 33.00 & 33.00 & 33.00 & 33.00 \\ Arai-Calfare & 27.983.128 & 27.92.568 & 27.092.09 & 27.033.725 & 27.092.327 & 27.320.327 & 27.320.324 & 26.317.232 & 24.462.032 & 27.0913.820 & 33.032.452 & 27.982.072 & 27.983.127 & 27.983.127 & 27.983.127 & 27.983.127 \\ Antoinership & 36.329.800 & 57.920.560 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 \\ Ruiggy & 32.475.952 & 40.584.873 & 37.30 & 40.592.76 & 36.500 & 37.313.90 & 39.002.380 & 30.743.922 & 25.982.573 & 24.950.230 & 36.500.360 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 & 36.500 \\ Rutomass Management & 25.040.400 & 78.733.733 & 34.405.940 & 28.400 & 27.000 & 28.500 & 28.





\begin{table}
\begin{tabular}{l c c c c c c c c c c c}





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c}
**Topi** & **a** & **a** & **b** & **c** & **d** & **e** & **f** & **g** & **g** & **h** \\ \hline Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.17/2.21 & 23.7/2.20 & 24.2/2.04 & 24.0/2.16 & 35.4/2.16 & 35.4/2.16 & 35.4/2.16 & 35.4/2.16 & 35.4/2.16 & 35.4/2.16 & 35.4/2.16 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.17/2.20 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature and Bottom & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature and Bottom & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature and Bottom & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature and Bottom & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 & 23.0/2.19 \\ Avignature & 20.16/20.18/20.19 & 35.48/4.65 & 23.0/2.19 & 23.





\begin{table}
\begin{tabular}{l c c c c c c c c c c c}





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c}
**Topi** & **a** & **a** & **b** & **c** & **d** & **e** & **f** & **g** & **v** & **g** & **g** & **h** \\ \hline Avignature & 32.649/81.026 & 40.85/81.752 & 32.34/81.933 & 23.64/81.946 & 33.09/81.946 & 33.17/81.946 & 33.09/81.976 & 33.09/81.976 & 20.36/81.966 & 20.36/81.966 \\ Avignature & 32.79/81.026 & 40.85/81.646 & 32.09/81.946 & 33.09/81.946 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 20.36/81.936 & 20.36/81.966 & 20.36/81.966 \\ Avignature and Bottom & 32.81/81.936 & 40.85/81.936 & 32.09/81.946 & 32.09/81.936 & 33.17/81.946 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 20.36/81.936 & 20.36/81.936 \\ Avignature and Bottom & 32.81/81.936 & 40.85/81.936 & 33.09/81.946 & 33.09/81.946 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 20.36/81.936 \\ Avignature and Bottom & 32.81/81.936 & 40.85/81.936 & 33.09/81.946 & 33.09/81.946 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 \\ Avignature and Bottom & 32.81/81.936 & 40.85/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 \\ Avignature & 32.81/81.936 & 40.85/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 \\ Avignature & 32.81/81.936 & 40.85/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 \\ Avignature & 32.81/81.936 & 40.85/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 \\ Avignature & 32.81/81.936 & 40.85/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 & 33.09/81.936 \\ Avignature & 32.81/81.936 & 40.85/81.936 & 33.09/81.936 & 33.09/81.936 & 33.





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c}
**Topi** & **b** & **c** & **a** & **p** & **b** & **b** & **c** & **d** & **e** & **f** & **g** & **g** & **g** \\ \hline Applegate & 40.94/60.8319 & 50.74/60.865 & 23.78/60.85 & 23.88/60.851 & 42.00/60.866 & 31.88/60.866 & 24.18/60.866 & 34.20/60.866 & 34.20/60.866 & 34.20/60.866 \\ Applegate & 40.94/60.830 & 50.94/60.866 & 30.40/60.866 & 40.94/60.866 & 40.94/60.866 & 40.94/60.866 & 40.94/60.866 & 40.94/60.866 & 40.94/60.866 & 40.94/60.866 \\ Anoffee & 40.94/60.830 & 50.94/60.866 & 30.94/60.866 & 30.94/60.866 & 30.94/60.866 & 40.94/60.866 & 40.94/60.866 & 34.20/60.866 & 34.20/60.866 & 34.20/60.866 \\ Amon and Applegate & 50.94/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 40.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Bailey & 51.95/60.844 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 40.94/60.866 & 40.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Bonneau Magenet & 43.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Canesley & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Comosseosse & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ General Assembly & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Benneau Magenet & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ German Assembly & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Kenne Sanson & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Emonsons & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Kinetics & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Energy of Brown & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Engrautti & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Engrautti & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Engrautti & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Engrautti & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Engrautti-Escourte & 33.02/60.830 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 & 50.94/60.866 \\ Esposito & 33.02/60.830 & 50.94/60.866 & 50.





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c c}
**Topi** & **a** & **b** & **c** & **d** & **e** & **f** & **g** & **f** & **g** & **h** \\ \hline Avgnance & 20.95/87.616 & 3.48/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avgenuity & 20.95/87.698 & 3.49/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 \\ Avnolomev and Hopping & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.4/88.988 & 30.





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c}
**Topi** & **a** & **b** & **c** & **d** & **e** & **f** & **g** & **h** & **e** & **f** & **g** & **g** \\ \hline Avignature & 40.953/6.810 & 42.94/6.053 & 43.78/6.93 & 44.04/6.02 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature & 30.792/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 & 43.78/6.93 \\ Avignature and Hopping & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 40.953/6.93 & 43.78/6.93 & 43.78/6.93 & 4



## Appendix A

\begin{table}
\begin{tabular}{l c r r r r r r r r r r r r r r} \hline
**Topist** & **k** & **m** & **r** & **q** & **z** & **d** & **u** & **v** & **w** & **v** & **v** & **w** & **v** \\ \hline Agreements & 51.69/83.790 & 80.73/81.72 & 72.42 & 52.95/44.95 & 80.76/84.84 & 40.00/80.649 & 85.87/84.62 & 40.00/80.649 & 85.87/84.62 & 40.00/80.510 & 40.94/80.510 & 40.94/80.510 & 40.94/80.510 & 40.94/80.510 & 40.94/80.510 \\ Ardership & 53.67/85.69 & 55.67/85.10 & 71.70/80.49 & 40.94/80.62 & 40.95/80.67 & 57.00/80.75 & 47.00/80.620 & 56.70/80.710 & 56.59/85.610 & 55.65/85.69 & 55.69/85.60 & 55.69/85.60 & 55.69/85.60 & 55.69/85.60 \\ Ackenhaus and Boeing & 51.63/81.710 & 52.72/80.710 & 40.94/80.60 & 40.95/80.60 & 52.00/80.630 & 40.95/80.630 & 40.95/80.600 & 40.40/80.790 & 40.45/80.790 & 54.00/80.500 & 52.00/80.570 & 40.95/80.40 \\ Arai and Calame & 48.78/16.73 & 50.41/27.76 & 40.93/80.74 & 40.93/80.74 & 54.07/80.74 & 54.07/80.744 & 50.07/80.600 & 44.33/80.742 & 50.07/80.744 & 40.45/80.744 & 40.45/80.790 & 40.73/80.510 & 40.74/80.510 & 40.74/80.510 \\ Abstorey and Appleby & 66.45/85.76 & 69.45/85.66 & 52.00/80.72 & 69.02/80.73 & 50.73/75.83 & 50.00/80.720 & 69.03/80.731 & 60.00/80.731 & 60.00/80.731 & 60.00/80.736 & 60.59/85.62 & 60.57/80.731 \\ Bailey & 48.67/85.76 & 69.45/85.66 & 52.00/80.72 & 69.03/75.78 & 51.67/85.783 & 50.00/80.646 & 65.73/80.613 & 40.95/80.751 & 40.94/80.750 & 40.94/80.750 & 40.94/80.750 & 40.94/80.750 & 40.94/80.750 \\ Bilateral Management & 48.78/85.79 & 51.45/85.66 & 53.00/80.74 & 52.00/80.75 & 53.00/80.75 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 \\ Clearings & 54.87/85.79 & 51.45/85.68 & 53.00/80.74 & 52.00/80.75 & 53.00/80.75 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 \\ Corporate Science & 39.14/85.49 & 45.43/85.93 & 54.00/80.52 & 46.00/80.52 & 40.35/85.54 & 50.00/80.700 & 44.40/80.790 & 36.30/80.740 & 53.00/80.740 & 41.34/80.540 & 36.30/80.740 & 41.34/80.540 & 36.30/80.740 \\ Infima and Security & 50.87/88.94 & 50.79/80.70 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 \\ Belt Sensors & 44.66/85.69 & 50.81/85.80 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 \\ Expresses & 51.21/72.75 & 55.69/78.75 & 47.90/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 \\ Extreme & 48.67/85.73 & 50.47/85.76 & 48.00/80.50 & 48.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 \\ Energy and Power & 54.67/85.77 & 50.87/85.77 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 \\ Erpingping & 39.84/85.74 & 50.66/85.66 & 54.01/85.45 & 54.32/85.94 & 53.78/85.74 & 53.27/85.94 & 53.65/85.74 & 53.31/85.94 & 53.41/85.94 & 53.41/85.94 & 53.41/85.94 & 53.41/85.94 & 53.41/85.94 \\ Excessed Average & 48.65/85.73 & 50.79/80.60 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 \\ Erice and Inbase Bi(s) & 52.69/85.73 & 50.75/85.77 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 & 53.00/80.700 \\ Fresse and Inbase Bi(s) & 52.69/85.





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c}





\begin{table}
\begin{tabular}{l c c c c c c c c c c c c}





\begin{table}
\begin{tabular}{l c r r r r r r r r r r r}