Add Model Card

.gitattributes

@@ -34,3 +34,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
 *.json filter=lfs diff=lfs merge=lfs -text
+aya-fig1.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,312 @@
 ---
 license: apache-2.0
+datasets:
+  - CohereForAI/xP3x
+  - CohereForAI/aya_dataset
+  - CohereForAI/aya_collection
+  - DataProvenanceInitiative/Commercially-Verified-Licenses
+  - CohereForAI/aya_evaluation_suite
+language:
+  - afr
+  - amh
+  - ara
+  - aze
+  - bel
+  - ben
+  - bul
+  - cat
+  - ceb
+  - ces
+  - cym
+  - dan
+  - deu
+  - ell
+  - eng
+  - epo
+  - est
+  - eus
+  - fin
+  - fil
+  - fra
+  - fry
+  - gla
+  - gle
+  - glg
+  - guj
+  - hat
+  - hau
+  - heb
+  - hin
+  - hun
+  - hye
+  - ibo
+  - ind
+  - isl
+  - ita
+  - jav
+  - jpn
+  - kan
+  - kat
+  - kaz
+  - khm
+  - kir
+  - kor
+  - kur
+  - lao
+  - lav
+  - lat
+  - lit
+  - ltz
+  - mal
+  - mar
+  - mkd
+  - mlg
+  - mlt
+  - mon
+  - mri
+  - msa
+  - mya
+  - nep
+  - nld
+  - nor
+  - nso
+  - nya
+  - ory
+  - pan
+  - pes
+  - pol
+  - por
+  - pus
+  - ron
+  - rus
+  - sin
+  - slk
+  - slv
+  - smo
+  - sna
+  - snd
+  - som
+  - sot
+  - spa
+  - sqi
+  - srp
+  - sun
+  - swa
+  - swe
+  - tam
+  - tel
+  - tgk
+  - tha
+  - tur
+  - twi
+  - ukr
+  - urd
+  - uzb
+  - vie
+  - xho
+  - yid
+  - yor
+  - zho
+  - zul
+metrics:
+  - accuracy
+  - bleu
 ---
+
+<img src="aya-fig1.png" alt="Aya model summary image" width="800" style="margin-left:'auto' margin-right:'auto' display:'block'"/>
+
+# Model Card for Aya Model
+
+## Model Summary
+
+> The Aya model is a massively multilingual generative language model that follows instructions in 101 languages.
+> Aya outperforms [mT0](https://huggingface.co/bigscience/mt0-xxl) and [BLOOMZ](https://huggingface.co/bigscience/bloomz) a wide variety of automatic and human evaluations despite covering double the number of languages.
+> The Aya model is trained using [xP3x](https://huggingface.co/datasets/CohereForAI/xP3x), [Aya Dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset), [Aya Collection](https://huggingface.co/datasets/CohereForAI/aya_collection), a subset of [DataProvenance collection](https://huggingface.co/datasets/DataProvenanceInitiative/Commercially-Verified-Licenses) and ShareGPT-Command.
+> We release the checkpoints under a Apache-2.0 license to further our mission of multilingual technologies empowering a
+> multilingual world.
+
+- **Developed by:** Cohere For AI
+- **Model type:** a Transformer style autoregressive massively multilingual language model.
+- **Paper**: [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](arxiv.com)
+- **Point of Contact**: [Ahmet Ustun](mailto:ahmet@cohere.com)
+- **Languages**: Refer to the list of languages in the `language` section of this model card.
+- **License**: Apache-2.0
+- **Model**: [Aya](https://huggingface.co/CohereForAI/aya)
+- **Model Size**: 13 billion parameters
+- **Datasets**: [xP3x](https://huggingface.co/datasets/CohereForAI/xP3x), [Aya Dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset), [Aya Collection](https://huggingface.co/datasets/CohereForAI/aya_collection), [DataProvenance collection](https://huggingface.co/datasets/DataProvenanceInitiative/Commercially-Verified-Licenses), ShareGPT-Command.
+
+## Use
+
+```bash
+# pip install -q transformers
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+
+checkpoint = "CohereForAI/aya_model"
+
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+aya_model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint)
+
+inputs = tokenizer.encode("Translate to English: Je t’aime.", return_tensors="pt")
+outputs = aya_model.generate(inputs)
+print(tokenizer.decode(outputs[0]))
+```
+
+## Model Details
+
+### Training
+
+- Architecture: Same as [mt5-xxl](https://huggingface.co/google/mt5-xxl)
+- Number of Finetuning Samples: 25M
+- Batch size: 256
+- Hardware: TPUv4-128
+- Software: T5X, Jax
+
+### Data Sources
+
+The Aya model is trained on the following datasets:
+
+- [xP3x](https://huggingface.co/datasets/CohereForAI/xP3x)
+- [Aya Dataset](https://huggingface.co/datasets/CohereForAI/aya_dataset)
+- [Aya Collection](https://huggingface.co/datasets/CohereForAI/aya_collection)
+- [DataProvenance collection](https://huggingface.co/datasets/DataProvenanceInitiative/Commercially-Verified-Licenses)
+- ShareGPT-Command
+
+All datasets are subset to the 101 languages supported by [mT5]. See the [paper](arxiv.com) for details about filtering and pruning.
+
+## Evaluation
+
+We refer to Section 5 from our paper for multilingual eval across 99 languages – including discriminative, generative tasks, human evaluation and simulated win rates that cover both held-out tasks and in-distribution performance.
+
+## Bias, Risks, and Limitations
+
+
+For a detailed overview of our effort at safety mitigation and benchmarking toxicity and bias across multiple languages, we refer Sections 6 and 7 of our paper: [Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model](arxiv.com).
+
+We hope that the release of the Aya model will make community-based redteaming efforts possible, by exposing an open-source massively-multilingual model for community research.
+
+## Citation
+
+**BibTeX:**
+
+```
+@article{,
+  title={},
+  author={},
+  journal={Preprint},
+  year={2024}
+}
+```
+
+**APA:**
+
+## Languages Covered
+
+Below is the list of languages used in finetuning the Aya Model. We group languages into higher-, mid-, and lower-resourcedness based on a language classification by [Joshi et. al, 2020](https://microsoft.github.io/linguisticdiversity/). For further details, refer to our [paper]()
+
+| ISO Code | Language Name   |    Script    |     Family      |    Subgrouping    | Resourcedness |
+| :------- | :-------------- | :----------: | :-------------: | :---------------: | :-----------: |
+| afr      | Afrikaans       |    Latin     |  Indo-European  |     Germanic      |      Mid      |
+| amh      | Amharic         |    Ge'ez     |  Afro-Asiatic   |      Semitic      |      Low      |
+| ara      | Arabic          |    Arabic    |  Afro-Asiatic   |      Semitic      |     High      |
+| aze      | Azerbaijani     | Arabic/Latin |     Turkic      |   Common Turkic   |      Low      |
+| bel      | Belarusian      |   Cyrillic   |  Indo-European  |   Balto-Slavic    |      Mid      |
+| ben      | Bengali         |   Bengali    |  Indo-European  |    Indo-Aryan     |      Mid      |
+| bul      | Bulgarian       |   Cyrillic   |  Indo-European  |   Balto-Slavic    |      Mid      |
+| cat      | Catalan         |    Latin     |  Indo-European  |      Italic       |     High      |
+| ceb      | Cebuano         |    Latin     |  Austronesian   | Malayo-Polynesian |      Mid      |
+| ces      | Czech           |    Latin     |  Indo-European  |   Balto-Slavic    |     High      |
+| cym      | Welsh           |    Latin     |  Indo-European  |      Celtic       |      Low      |
+| dan      | Danish          |    Latin     |  Indo-European  |     Germanic      |      Mid      |
+| deu      | German          |    Latin     |  Indo-European  |     Germanic      |     High      |
+| ell      | Greek           |    Greek     |  Indo-European  |  Graeco-Phrygian  |      Mid      |
+| eng      | English         |    Latin     |  Indo-European  |     Germanic      |     High      |
+| epo      | Esperanto       |    Latin     |   Constructed   |    Esperantic     |      Low      |
+| est      | Estonian        |    Latin     |     Uralic      |      Finnic       |      Mid      |
+| eus      | Basque          |    Latin     |     Basque      |         -         |     High      |
+| fin      | Finnish         |    Latin     |     Uralic      |      Finnic       |     High      |
+| fil      | Tagalog         |    Latin     |  Austronesian   | Malayo-Polynesian |      Mid      |
+| fra      | French          |    Latin     |  Indo-European  |      Italic       |     High      |
+| fry      | Western Frisian |    Latin     |  Indo-European  |     Germanic      |      Low      |
+| gla      | Scottish Gaelic |    Latin     |  Indo-European  |      Celtic       |      Low      |
+| gle      | Irish           |    Latin     |  Indo-European  |      Celtic       |      Low      |
+| glg      | Galician        |    Latin     |  Indo-European  |      Italic       |      Mid      |
+| guj      | Gujarati        |   Gujarati   |  Indo-European  |    Indo-Aryan     |      Low      |
+| hat      | Haitian Creole  |    Latin     |  Indo-European  |      Italic       |      Low      |
+| hau      | Hausa           |    Latin     |  Afro-Asiatic   |      Chadic       |      Low      |
+| heb      | Hebrew          |    Hebrew    |  Afro-Asiatic   |      Semitic      |      Mid      |
+| hin      | Hindi           |  Devanagari  |  Indo-European  |    Indo-Aryan     |     High      |
+| hun      | Hungarian       |    Latin     |     Uralic      |         -         |     High      |
+| hye      | Armenian        |   Armenian   |  Indo-European  |      Armenic      |      Low      |
+| ibo      | Igbo            |    Latin     | Atlantic-Congo  |    Benue-Congo    |      Low      |
+| ind      | Indonesian      |    Latin     |  Austronesian   | Malayo-Polynesian |      Mid      |
+| isl      | Icelandic       |    Latin     |  Indo-European  |     Germanic      |      Low      |
+| ita      | Italian         |    Latin     |  Indo-European  |      Italic       |     High      |
+| jav      | Javanese        |    Latin     |  Austronesian   | Malayo-Polynesian |      Low      |
+| jpn      | Japanese        |   Japanese   |     Japonic     |     Japanesic     |     High      |
+| kan      | Kannada         |   Kannada    |    Dravidian    |  South Dravidian  |      Low      |
+| kat      | Georgian        |   Georgian   |   Kartvelian    |   Georgian-Zan    |      Mid      |
+| kaz      | Kazakh          |   Cyrillic   |     Turkic      |   Common Turkic   |      Mid      |
+| khm      | Khmer           |    Khmer     |  Austroasiatic  |      Khmeric      |      Low      |
+| kir      | Kyrgyz          |   Cyrillic   |     Turkic      |   Common Turkic   |      Low      |
+| kor      | Korean          |    Hangul    |    Koreanic     |      Korean       |     High      |
+| kur      | Kurdish         |    Latin     |  Indo-European  |      Iranian      |      Low      |
+| lao      | Lao             |     Lao      |    Tai-Kadai    |      Kam-Tai      |      Low      |
+| lav      | Latvian         |    Latin     |  Indo-European  |   Balto-Slavic    |      Mid      |
+| lat      | Latin           |    Latin     |  Indo-European  |      Italic       |      Mid      |
+| lit      | Lithuanian      |    Latin     |  Indo-European  |   Balto-Slavic    |      Mid      |
+| ltz      | Luxembourgish   |    Latin     |  Indo-European  |     Germanic      |      Low      |
+| mal      | Malayalam       |  Malayalam   |    Dravidian    |  South Dravidian  |      Low      |
+| mar      | Marathi         |  Devanagari  |  Indo-European  |    Indo-Aryan     |      Low      |
+| mkd      | Macedonian      |   Cyrillic   |  Indo-European  |   Balto-Slavic    |      Low      |
+| mlg      | Malagasy        |    Latin     |  Austronesian   | Malayo-Polynesian |      Low      |
+| mlt      | Maltese         |    Latin     |  Afro-Asiatic   |      Semitic      |      Low      |
+| mon      | Mongolian       |   Cyrillic   | Mongolic-Khitan |     Mongolic      |      Low      |
+| mri      | Maori           |    Latin     |  Austronesian   | Malayo-Polynesian |      Low      |
+| msa      | Malay           |    Latin     |  Austronesian   | Malayo-Polynesian |      Mid      |
+| mya      | Burmese         |   Myanmar    |  Sino-Tibetan   |   Burmo-Qiangic   |      Low      |
+| nep      | Nepali          |  Devanagari  |  Indo-European  |    Indo-Aryan     |      Low      |
+| nld      | Dutch           |    Latin     |  Indo-European  |     Germanic      |     High      |
+| nor      | Norwegian       |    Latin     |  Indo-European  |     Germanic      |      Low      |
+| nso      | Northern Sotho  |    Latin     | Atlantic-Congo  |    Benue-Congo    |      Low      |
+| nya      | Chichewa        |    Latin     | Atlantic-Congo  |    Benue-Congo    |      Low      |
+| ory      | Oriya           |    Oriya     |  Indo-European  |    Indo-Aryan     |      Low      |
+| pan      | Punjabi         |   Gurmukhi   |  Indo-European  |    Indo-Aryan     |      Low      |
+| pes      | Persian         |    Arabic    |  Indo-European  |      Iranian      |     High      |
+| pol      | Polish          |    Latin     |  Indo-European  |   Balto-Slavic    |     High      |
+| por      | Portuguese      |    Latin     |  Indo-European  |      Italic       |     High      |
+| pus      | Pashto          |    Arabic    |  Indo-European  |      Iranian      |      Low      |
+| ron      | Romanian        |    Latin     |  Indo-European  |      Italic       |      Mid      |
+| rus      | Russian         |   Cyrillic   |  Indo-European  |   Balto-Slavic    |     High      |
+| sin      | Sinhala         |   Sinhala    |  Indo-European  |    Indo-Aryan     |      Low      |
+| slk      | Slovak          |    Latin     |  Indo-European  |   Balto-Slavic    |      Mid      |
+| slv      | Slovenian       |    Latin     |  Indo-European  |   Balto-Slavic    |      Mid      |
+| smo      | Samoan          |    Latin     |  Austronesian   | Malayo-Polynesian |      Low      |
+| sna      | Shona           |    Latin     |  Indo-European  |    Indo-Aryan     |      Low      |
+| snd      | Sindhi          |    Arabic    |  Indo-European  |    Indo-Aryan     |      Low      |
+| som      | Somali          |    Latin     |  Afro-Asiatic   |     Cushitic      |      Low      |
+| sot      | Southern Sotho  |    Latin     | Atlantic-Congo  |    Benue-Congo    |      Low      |
+| spa      | Spanish         |    Latin     |  Indo-European  |      Italic       |     High      |
+| sqi      | Albanian        |    Latin     |  Indo-European  |     Albanian      |      Low      |
+| srp      | Serbian         |   Cyrillic   |  Indo-European  |   Balto-Slavic    |     High      |
+| sun      | Sundanese       |    Latin     |  Austronesian   | Malayo-Polynesian |      Low      |
+| swa      | Swahili         |    Latin     | Atlantic-Congo  |    Benue-Congo    |      Low      |
+| swe      | Swedish         |    Latin     |  Indo-European  |     Germanic      |     High      |
+| tam      | Tamil           |    Tamil     |    Dravidian    |  South Dravidian  |      Mid      |
+| tel      | Telugu          |    Telugu    |    Dravidian    |  South Dravidian  |      Low      |
+| tgk      | Tajik           |   Cyrillic   |  Indo-European  |      Iranian      |      Low      |
+| tha      | Thai            |     Thai     |    Tai-Kadai    |      Kam-Tai      |      Mid      |
+| tur      | Turkish         |    Latin     |     Turkic      |   Common Turkic   |     High      |
+| twi      | Twi             |    Latin     | Atlantic-Congo  |    Niger-Congo    |      Low      |
+| ukr      | Ukrainian       |   Cyrillic   |  Indo-European  |   Balto-Slavic    |      Mid      |
+| urd      | Urdu            |    Arabic    |  Indo-European  |    Indo-Aryan     |      Mid      |
+| uzb      | Uzbek           |    Latin     |     Turkic      |   Common Turkic   |      Mid      |
+| vie      | Vietnamese      |    Latin     |  Austroasiatic  |      Vietic       |     High      |
+| xho      | Xhosa           |    Latin     | Atlantic-Congo  |    Benue-Congo    |      Low      |
+| yid      | Yiddish         |    Hebrew    |  Indo-European  |     Germanic      |      Low      |
+| yor      | Yoruba          |    Latin     | Atlantic-Congo  |    Benue-Congo    |      Low      |
+| zho      | Chinese         |     Han      |  Sino-Tibetan   |      Sinitic      |     High      |
+| zul      | Zulu            |    Latin     | Atlantic-Congo  |    Benue-Congo    |      Low      |
+
+## Model Card Contact
+
+For errors in this model card, contact Ahmet or Viraat, `{ahmet, viraat} at cohere dot com`.