Update README.md

README.md

@@ -354,378 +354,6 @@ Our training dataset [Darija-SFT-Mixture](https://huggingface.co/datasets/MBZUAI
 Atlas-Chat models are based on Gemma 2 models. The Atlas-Chat models were trained using 8 Nvidia's A100 80 GB GPUs in parallel using FSDP on AWS Sagemaker. The model is trained using HuggingFace transformers and parameter-efficient fine-tuning with LoRA rank of 256.
 
 
-<!--
-## Evaluation
-The Atlas-Chat models were evaluated on a comprehensive suite of tasks using various datasets and benchmarks to assess their performance across multiple dimensions. These included tasks such as:
-
-* **DarijaMMLU:** A Darija version of ArabicMMLU and MMLU benchmarks translated from MSA and English respectively.
-* **DarijaHellaSwag:** A Darija version of HellaSwag.
-* **Belebele Ary_Arab:** Belebele is a multiple-choice machine reading comprehension dataset published by Facebook spanning 122 language variants. The Evaluation is done on the Ary_Arab part of Belebele that refers to Darija.
-* **Sentiment Analysis.**
-* **Translation:** Including six directions and four languages: Darija, MSA, English and French.
-* **Transliteration:** Transforming a sentence from Darija (written in Arabic characters) to Arabizi (Written in Latin characters) and vice-versa.
-* **Summarization.**
-
-The models were compared against a collection of existing open-source Arabic models to gauge their effectiveness, with a particular focus on performance in Darija. All scores are based on zero-shot performance. The prompts are written mainly in Darija. The metric used for DarijaMMLU, DarijaHellaSwag, Belebele Ary and Sentiment Analysis is the normalized accuracy. We used [Language Model Evaluation Harness](https://github.com/MBZUAI-Paris/lm-evaluation-harness-atlas-chat) to conduct these evaluations.
-
-<table>
-    <tr>
-        <td rowspan="2">Model</td>
-        <td rowspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaMMLU" target="_blank">DarijaMMLU</a></td>
-        <td rowspan="2"><a href="MBZUAI-Paris/DarijaHellaSwag" target="_blank">DarijaHellaSwag</a></td>
-        <td rowspan="2"><a href="https://huggingface.co/datasets/facebook/belebele/viewer/ary_Arab" target="_blank">Belebele Ary</a></td>
-        <td rowspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">Sentiment Analysis</a></td>
-        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">DODa-10k (Translation)</a></td>
-        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">MADAR (Translation)</a></td>
-        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">FLORES+ (Translation)</a></td>
-        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">NLLB-Seed (Translation)</a></td>
-        <td colspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">DODa-10k (Transliteration)</a></td>
-        <td rowspan="2"><a href="https://huggingface.co/datasets/MBZUAI-Paris/DarijaBench" target="_blank">MArSum (Summarization)</a><br/>(LLM as a judge)</td>
-    </tr>
-    <tr>
-        <td>BLEU</td>
-        <td>chrF</td>
-        <td>BLEU</td>
-        <td>chrF</td>
-        <td>BLEU</td>
-        <td>chrF</td>
-        <td>BLEU</td>
-        <td>chrF</td>
-        <td>BLEU</td>
-        <td>chrF</td>
-    </tr>
-    <tr>
-        <td><a href="https://huggingface.co/inceptionai/jais-family-1p3b-chat" target="_blank">jais-family-1p3b-chat</a></td>
-        <td>35.39</td>
-        <td>32.51</td>
-        <td>38.33</td>
-        <td>45.29</td>
-        <td>00.13</td>
-        <td>06.18</td>
-        <td>00.50</td>
-        <td>15.43</td>
-        <td>02.44</td>
-        <td>19.14</td>
-        <td>01.99</td>
-        <td>12.60</td>
-        <td>00.01</td>
-        <td>03.01</td>
-        <td>00.50</td>
-    </tr>
-    <tr>
-        <td><a href="https://huggingface.co/inceptionai/jais-family-2p7b-chat" target="_blank">jais-family-2p7b-chat</a></td>
-        <td>37.44</td>
-        <td>34.49</td>
-        <td>44.11</td>
-        <td>51.56</td>
-        <td>00.25</td>
-        <td>07.46</td>
-        <td>00.62</td>
-        <td>16.36</td>
-        <td>04.25</td>
-        <td>18.22</td>
-        <td>03.10</td>
-        <td>08.19</td>
-        <td>00.01</td>
-        <td>03.27</td>
-        <td>00.90</td>
-    </tr>
-    <tr>
-        <td><a href="https://huggingface.co/google/gemma-2-2b-it" target="_blank">gemma-2-2b-it</a></td>
-        <td>28.58</td>
-        <td>32.42</td>
-        <td>25.22</td>
-        <td>53.36</td>
-        <td>00.10</td>
-        <td>04.96</td>
-        <td>00.12</td>
-        <td>06.66</td>
-        <td>01.55</td>
-        <td>18.59</td>
-        <td>02.78</td>
-        <td>23.69</td>
-        <td>00.01</td>
-        <td>02.08</td>
-        <td>06.80</td>
-    </tr>
-    <tr>
-        <td><a href="meta-llama/Llama-3.2-1B-Instruct" target="_blank">Llama-3.2-1B-Instruct</a></td>
-        <td>27.66</td>
-        <td>26.88</td>
-        <td>28.89</td>
-        <td>46.27</td>
-        <td>00.07</td>
-        <td>05.95</td>
-        <td>00.80</td>
-        <td>18.71</td>
-        <td>04.53</td>
-        <td>18.39</td>
-        <td>04.52</td>
-        <td>17.06</td>
-        <td>00.02</td>
-        <td>03.74</td>
-        <td>08.23</td>
-    </tr>
-    <tr>
-        <td><a href="meta-llama/Llama-3.2-3B-Instruct" target="_blank">Llama-3.2-3B-Instruct</a></td>
-        <td>32.60</td>
-        <td>28.33</td>
-        <td>38.00</td>
-        <td>49.20</td>
-        <td>00.62</td>
-        <td>13.67</td>
-        <td>01.18</td>
-        <td>22.12</td>
-        <td>08.59</td>
-        <td>35.21</td>
-        <td>13.75</td>
-        <td>43.63</td>
-        <td>00.21</td>
-        <td>09.68</td>
-        <td>08.23</td>
-    </tr>
-    <tr>
-        <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-2B" target="_blank">Atlas-Chat-2B</a></strong></td>
-        <td><b>44.97</td>
-        <td><b>41.48</td>
-        <td><b>53.89</td>
-        <td><b>73.99</td>
-        <td><b>22.76</td>
-        <td><b>44.86</td>
-        <td><b>16.67</td>
-        <td><b>41.64</td>
-        <td><b>14.92</td>
-        <td><b>43.03</td>
-        <td><b>23.88</td>
-        <td><b>52.19</td>
-        <td><b>08.18</td>
-        <td><b>21.54</td>
-        <td><b>55.22</td>
-    </tr>
-    <tr style="border-top: 4px solid;"></tr>
-    <tr>
-        <td><a href="https://huggingface.co/inceptionai/jais-family-6p7b-chat" target="_blank">jais-family-6p7b-chat</a></td>
-        <td>39.96</td>
-        <td>41.57</td>
-        <td>51.22</td>
-        <td>56.78</td>
-        <td>00.73</td>
-        <td>11.85</td>
-        <td>01.88</td>
-        <td>23.22</td>
-        <td>04.25</td>
-        <td>18.22</td>
-        <td>04.62</td>
-        <td>20.22</td>
-        <td>00.02</td>
-        <td>03.79</td>
-        <td>03.02</td>
-    </tr>
-    <tr>
-        <td><a href="https://huggingface.co/inceptionai/jais-adapted-7b-chat" target="_blank">jais-adapted-7b-chat</a></td>
-        <td>39.30</td>
-        <td>35.19</td>
-        <td>43.67</td>
-        <td>52.72</td>
-        <td>00.60</td>
-        <td>09.43</td>
-        <td>03.45</td>
-        <td>25.88</td>
-        <td>07.25</td>
-        <td>23.21</td>
-        <td>01.25</td>
-        <td>02.22</td>
-        <td>00.04</td>
-        <td>03.24</td>
-        <td>02.82</td>
-    </tr>
-    <tr>
-        <td><a href="https://huggingface.co/inceptionai/jais-family-13b-chat" target="_blank">jais-family-13b-chat</a></td>
-        <td>45.11</td>
-        <td>43.90</td>
-        <td>58.67</td>
-        <td>41.73</td>
-        <td>00.92</td>
-        <td>11.71</td>
-        <td>04.01</td>
-        <td>28.48</td>
-        <td>05.70</td>
-        <td>27.24</td>
-        <td>04.50</td>
-        <td>22.56</td>
-        <td>00.03</td>
-        <td>03.57</td>
-        <td>01.77</td>
-    </tr>
-    <tr>
-        <td><a href="https://huggingface.co/inceptionai/jais-adapted-13b-chat" target="_blank">jais-adapted-13b-chat</a></td>
-        <td>45.20</td>
-        <td>40.65</td>
-        <td>49.67</td>
-        <td>66.68</td>
-        <td>00.87</td>
-        <td>10.52</td>
-        <td>04.02</td>
-        <td>25.29</td>
-        <td>06.66</td>
-        <td>23.46</td>
-        <td>20.14</td>
-        <td>47.87</td>
-        <td>0.04</td>
-        <td>04.77</td>
-        <td>01.92</td>
-    </tr>
-    <tr>
-        <td><a href="https://huggingface.co/FreedomIntelligence/AceGPT-7B-chat" target="_blank">AceGPT-7b-chat</a></td>
-        <td>35.98</td>
-        <td>36.57</td>
-        <td>30.11</td>
-        <td>40.23</td>
-        <td>00.44</td>
-        <td>11.33</td>
-        <td>01.05</td>
-        <td>19.24</td>
-        <td>06.92</td>
-        <td>36.03</td>
-        <td>11.05</td>
-        <td>44.55</td>
-        <td>00.06</td>
-        <td>04.74</td>
-        <td>02.28</td>
-    </tr>
-    <tr>
-        <td><a href="https://huggingface.co/FreedomIntelligence/AceGPT-13B-chat" target="_blank">AceGPT-13b-chat</a></td>
-        <td>41.09</td>
-        <td>38.35</td>
-        <td>33.11</td>
-        <td>59.58</td>
-        <td>00.98</td>
-        <td>16.70</td>
-        <td>00.81</td>
-        <td>20.23</td>
-        <td>08.73</td>
-        <td>40.76</td>
-        <td>14.02</td>
-        <td>48.28</td>
-        <td>00.12</td>
-        <td>06.32</td>
-        <td>02.80</td>
-    </tr>
-    <tr>
-        <td><a href="https://huggingface.co/google/gemma-2-9b-it" target="_blank">gemma-2-9b-it</a></td>
-        <td>35.91</td>
-        <td>42.43</td>
-        <td>31.00</td>
-        <td>59.87</td>
-        <td>03.10</td>
-        <td>19.16</td>
-        <td>01.72</td>
-        <td>24.35</td>
-        <td>05.18</td>
-        <td>36.96</td>
-        <td>08.23</td>
-        <td>43.57</td>
-        <td>00.17</td>
-        <td>09.14</td>
-        <td>13.81</td>
-    </tr>
-    <tr>
-        <td><a href="meta-llama/Meta-Llama-3.1-8B-Instruct" target="_blank">Llama-3.1-8B-Instruct</a></td>
-        <td>44.13</td>
-        <td>38.24</td>
-        <td>47.00</td>
-        <td>44.08</td>
-        <td>00.92</td>
-        <td>14.19</td>
-        <td>01.46</td>
-        <td>23.82</td>
-        <td>08.89</td>
-        <td>33.08</td>
-        <td>11.85</td>
-        <td>35.51</td>
-        <td>00.11</td>
-        <td>06.02</td>
-        <td>01.28</td>
-    </tr>
-    <tr>
-        <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-9B" target="_blank">Atlas-Chat-9B</a></strong></td>
-        <td><b>58.23</td>
-        <td><b>57.75</td>
-        <td><b>74.56</td>
-        <td><b>81.89</td>
-        <td><b>28.08</td>
-        <td><b>50.48</td>
-        <td><b>18.16</td>
-        <td><b>43.91</td>
-        <td><b>18.63</td>
-        <td><b>47.53</td>
-        <td><b>29.98</td>
-        <td><b>58.26</td>
-        <td><b>22.08</td>
-        <td><b>34.17</td>
-        <td><b>59.76</td>
-    </tr>
-    <tr style="border-top: 4px solid;"></tr>
-    <tr>
-        <td><a href="https://huggingface.co/inceptionai/jais-family-30b-8k-chat" target="_blank">jais-family-30b-8k-chat</a></td>
-        <td>51.88</td>
-        <td>35.61</td>
-        <td>65.67</td>
-        <td>56.73</td>
-        <td>01.10</td>
-        <td>14.40</td>
-        <td>01.67</td>
-        <td>23.37</td>
-        <td>08.52</td>
-        <td>35.41</td>
-        <td>13.71</td>
-        <td>41.33</td>
-        <td>00.05</td>
-        <td>04.48</td>
-        <td>00.46</td>
-    </tr>
-    <tr>
-        <td><a href="https://huggingface.co/google/gemma-2-27b-it" target="_blank">gemma-2-27b-it</a></td>
-        <td>36.47</td>
-        <td>37.04</td>
-        <td>35.78</td>
-        <td>57.59</td>
-        <td>00.67</td>
-        <td>13.04</td>
-        <td>01.74</td>
-        <td>24.63</td>
-        <td>05.17</td>
-        <td>37.08</td>
-        <td>07.36</td>
-        <td>42.49</td>
-        <td>00.03</td>
-        <td>04.94</td>
-        <td>11.10</td>
-    </tr>
-    <tr>
-        <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-27B" target="_blank">Atlas-Chat-27B</a></strong></td>
-        <td><b>61.95</td>
-        <td><b>48.37</td>
-        <td><b>75.67</td>
-        <td>73.00</td>
-        <td><b>29.55</td>
-        <td><b>51.74</td>
-        <td><b>19.66</td>
-        <td><b>45.65</td>
-        <td><b>20.34</td>
-        <td><b>49.19</td>
-        <td><b>31.61</td>
-        <td><b>59.37</td>
-        <td><b>33.03</td>
-        <td><b>40.95</td>
-        <td><b>60.70</td>
-    </tr>
-
-
-    
-</table>
--->
-
 ## Evaluation
 The Atlas-Chat models were evaluated on a comprehensive suite of tasks using various datasets and benchmarks to assess their performance across multiple dimensions. These included tasks such as:
 
@@ -752,14 +380,14 @@ The models were compared against a collection of existing open-source Arabic mod
     <tr>
         <td><a href="https://huggingface.co/inceptionai/jais-family-1p3b-chat" target="_blank">jais-family-1p3b-chat</a></td>
         <td>35.39</td>
-        <td>32.51</td>
+        <td>27.71</td>
         <td>38.33</td>
         <td>35.56</td>
     </tr>
     <tr>
         <td><a href="https://huggingface.co/inceptionai/jais-family-2p7b-chat" target="_blank">jais-family-2p7b-chat</a></td>
         <td>37.44</td>
-        <td>34.49</td>
+        <td>29.10</td>
         <td>44.11</td>
         <td>52.97</td>
     </tr>
@@ -787,7 +415,7 @@ The models were compared against a collection of existing open-source Arabic mod
     <tr>
         <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-2B" target="_blank">Atlas-Chat-2B</a></strong></td>
         <td><b>44.97</b></td>
-        <td><b>41.48</b></td>
+        <td><b>35.08</b></td>
         <td><b>53.89</b></td>
         <td><b>92.31</b></td>
     </tr>
@@ -795,35 +423,35 @@ The models were compared against a collection of existing open-source Arabic mod
     <tr>
         <td><a href="https://huggingface.co/inceptionai/jais-family-6p7b-chat" target="_blank">jais-family-6p7b-chat</a></td>
         <td>39.96</td>
-        <td>41.57</td>
+        <td>32.64</td>
         <td>51.22</td>
         <td>65.18</td>
     </tr>
     <tr>
         <td><a href="https://huggingface.co/inceptionai/jais-adapted-7b-chat" target="_blank">jais-adapted-7b-chat</a></td>
         <td>39.30</td>
-        <td>35.19</td>
+        <td>29.55</td>
         <td>43.67</td>
         <td>61.84</td>
     </tr>
     <tr>
         <td><a href="https://huggingface.co/inceptionai/jais-family-13b-chat" target="_blank">jais-family-13b-chat</a></td>
         <td>45.11</td>
-        <td>43.90</td>
+        <td>33.98</td>
         <td>58.67</td>
         <td>69.93</td>
     </tr>
     <tr>
         <td><a href="https://huggingface.co/inceptionai/jais-adapted-13b-chat" target="_blank">jais-adapted-13b-chat</a></td>
         <td>45.20</td>
-        <td>40.65</td>
+        <td>32.84</td>
         <td>49.67</td>
         <td>77.52</td>
     </tr>
     <tr>
         <td><a href="https://huggingface.co/FreedomIntelligence/AceGPT-7B-chat" target="_blank">AceGPT-7b-chat</a></td>
         <td>35.98</td>
-        <td>36.57</td>
+        <td>30.33</td>
         <td>30.11</td>
         <td>47.31</td>
     </tr>
@@ -837,21 +465,21 @@ The models were compared against a collection of existing open-source Arabic mod
     <tr>
         <td><a href="https://huggingface.co/google/gemma-2-9b-it" target="_blank">gemma-2-9b-it</a></td>
         <td>35.91</td>
-        <td>42.43</td>
+        <td>32.19</td>
         <td>31.00</td>
         <td>90.86</td>
     </tr>
     <tr>
         <td><a href="meta-llama/Meta-Llama-3.1-8B-Instruct" target="_blank">Llama-3.1-8B-Instruct</a></td>
         <td>44.13</td>
-        <td>38.24</td>
+        <td>31.40</td>
         <td>47.00</td>
         <td>78.08</td>
     </tr>
     <tr>
         <td><strong><a href="https://huggingface.co/MBZUAI-Paris/Atlas-Chat-9B" target="_blank">Atlas-Chat-9B</a></strong></td>
         <td><b>58.23</b></td>
-        <td><b>57.75</b></td>
+        <td><b>43.65</b></td>
         <td><b>74.56</b></td>
         <td><b>95.62</b></td>
     </tr>