we instruction tuned models 011 such as Qwen-2.5-14B-Instruct and Phi-4 on a mixed- 063 language dataset
| Model | ARC-C | ARC-E | BoolQ | CMCQ | MMLU | Average* | MMLU-Pro | GPQA | MuSR | BBH | MATH |
|---|---|---|---|---|---|---|---|---|---|---|---|
| AryaBhatta-GemmaUltra-8.5B | 22.70 | 25.04 | 22.95 | 62.23 | 23.70 | 31.32 | 22.66 | 25.34 | 42.72 | 41.12 | 2.95 |
| Airavata-7B | 25.09 | 30.47 | 25.31 | 62.17 | 33.20 | 35.25 | 16.35 | 27.43 | 37.57 | 36.00 | 13.60 |
| sarvam-1-2B | 30.03 | 33.25 | 62.17 | 42.80 | 27.90 | 39.23 | - | - | - | - | - |
| Nemotron-4-Mini-Hindi-Instruct | 55.80 | 71.63 | 62.11 | 68.10 | 43.20 | 60.17 | 25.95 | 30.87 | 41.53 | 40.11 | 2.04 |
| Llama-3-Nanda-10B-Chat | 65.36 | 80.64 | 82.29 | 67.60 | 50.61 | 69.30 | - | - | - | - | - |
| Krutrim-2-12b-instruct | 67.32 | 81.10 | 84.74 | 76.30 | 56.10 | 73.11 | - | - | - | - | - |
| aya-expanse-8b | 74.06 | 87.08 | 86.45 | 83.30 | 56.89 | 77.56 | 30.04 | 30.29 | 37.17 | 49.42 | 7.02 |
| aya-expanse-32B | 85.41 | 95.08 | 90.43 | 89.80 | 69.71 | 86.08 | 41.30 | 32.55 | 38.62 | 56.29 | 13.37 |
| Our Qwen Model (14b) | 90.61 | 94.82 | 88.53 | 90.70 | 75.00 | 87.93 | 52.63 | 36.24 | 44.84 | 64.97 | 25.08 |
| Our Phi Model (14b) | 97.39 | 92.24 | 87.65 | 87.40 | 75.59 | 88.05 | 52.39 | 39.77 | 49.07 | 66.97 | 23.11 |
Table 1: Metrics (.2f) of our Qwen-2.5-14B and other LLMs over several English benchmarks
| Model | ARC-C | ARC-E | BoolQ | CMCQ | MMLU | Average |
|---|---|---|---|---|---|---|
| AryaBhatta-GemmaUltra-8.5B | 22.70 | 25.08 | 22.95 | 62.17 | 23.80 | 31.34 |
| Airavata-7B | 22.87 | 25.13 | 23.28 | 62.17 | 33.20 | 33.33 |
| sarvam-1-2B | 32.76 | 35.06 | 62.16 | 47.10 | 24.22 | 40.26 |
| Llama-3-Nanda-10B-Chat | 45.99 | 60.56 | 71.96 | 54.70 | 36.35 | 53.91 |
| Nemotron-4-Mini-Hindi-4B-Instruct | 50.68 | 63.72 | 68.74 | 51.30 | 37.18 | 54.32 |
| Krutrim-2-12b-instruct | 56.83 | 70.66 | 78.86 | 64.10 | 46.51 | 63.39 |
| aya-expanse-8b | 57.42 | 72.90 | 80.42 | 69.00 | 43.39 | 64.63 |
| aya-expanse-32B | 73.29 | 85.48 | 87.73 | 79.70 | 56.96 | 76.63 |
| Our Qwen Model (14b) | 74.06 | 81.23 | 84.07 | 78.20 | 53.85 | 74.82 |
| Our Phi Model (14b) | 81.74 | 89.06 | 86.02 | 78.70 | 56.39 | 78.38 |
Table 2: Metrics (.2f) of our Qwen-2.5-14B and other LLMs over several Hindi benchmarks
| Benchmark | Lang | Qwen-2.5-14B-Instruct | Our Qwen | Change | Phi-4 | Our Phi | Change |
|---|---|---|---|---|---|---|---|
| ARC-Easy | En | 95.45 | 94.82 | 🔻 0.63 | 97.31 | 97.39 | 🔼 0.08 |
| Hi | 78.49 | 81.23 | 🔼 2.74 | 86.87 | 89.06 | 🔼 2.19 | |
| ARC-Challenge | En | 90.87 | 90.61 | 🔻 0.26 | 92.41 | 92.24 | 🔻 0.17 |
| Hi | 69.62 | 74.06 | 🔼 4.44 | 79.18 | 81.74 | 🔼 2.56 | |
| BoolQ | En | 86.09 | 88.53 | 🔼 2.44 | 86.30 | 87.65 | 🔼 1.35 |
| Hi | 78.89 | 84.07 | 🔼 5.18 | 82.72 | 86.02 | 🔼 3.30 | |
| Context-MCQ | En | 91.20 | 90.70 | 🔻 0.50 | 86.30 | 87.40 | 🔼 1.10 |
| Hi | 77.40 | 78.20 | 🔼 0.80 | 75.70 | 78.70 | 🔼 3.00 | |
| MMLU | En | 74.37 | 75.00 | 🔼 0.63 | 74.67 | 75.59 | 🔼 0.92 |
| Hi | 52.16 | 53.85 | 🔼 1.69 | 53.24 | 56.39 | 🔼 3.15 | |
| Average | En | 87.60 | 87.93 | 🔼 0.33 | 87.40 | 88.05 | 🔼 0.65 |
| Hi | 71.31 | 74.82 | 🔼 3.51 | 75.54 | 78.38 | 🔼 2.84 | |
| Overall | 79.46 | 81.38 | 🔼 1.92 | 81.47 | 83.22 | 🔼 1.75 |
Table 3: Performance of our Qwen-2.5-14B model compared to originals over each benchmark : evals through log likelihoods
| Benchmark | Lang | Qwen-2.5-14B-Instruct | Our Qwen | Change | Phi-4 | Our Phi | Change |
|---|---|---|---|---|---|---|---|
| MMLU-Pro | En | 49.04 | 52.63 | 🔼 3.59 | 53.78 | 52.39 | 🔻 1.39 |
| MATH hard | En | 00.00 | 25.08 | 🔷 N/A | 12.31 | 23.11 | 🔼 10.80 |
| GPQA | En | 32.21 | 36.24 | 🔼 4.03 | 33.72 | 39.77 | 🔼 6.05 |
| MuSR | En | 40.87 | 44.84 | 🔼 3.97 | 41.01 | 49.07 | 🔼 8.06 |
| BigBench-Hard | En | 63.74 | 64.97 | 🔼 1.23 | 68.60 | 66.97 | 🔻 1.63 |
| Average | 37.17 | 44.75 | 🔼 7.58 | 41.88 | 46.26 | 🔼 4.38 |
Table 4: Performance of our Qwen-2.5-14B model compared to originals over each benchmark : evals through eval-harness