Hugging Face's logo

bigscience
/
mt0-xxl-mt

Text2Text Generation
Transformers
PyTorch
Safetensors
mt5
Eval Results
Inference Endpoints
Model card Files Community
5
mt0-xxl-mt / README.md
TimeRobber's picture
TimeRobber
Update README.md
d6061c1
preview code
|
raw
history blame
14.1 kB
metadata
license: apache-2.0
language:
  - multilingual
  - af
  - am
  - ar
  - az
  - be
  - bg
  - bn
  - ca
  - ceb
  - co
  - cs
  - cy
  - da
  - de
  - el
  - en
  - eo
  - es
  - et
  - eu
  - fa
  - fi
  - fil
  - fr
  - fy
  - ga
  - gd
  - gl
  - gu
  - ha
  - haw
  - hi
  - hmn
  - ht
  - hu
  - hy
  - ig
  - is
  - it
  - iw
  - ja
  - jv
  - ka
  - kk
  - km
  - kn
  - ko
  - ku
  - ky
  - la
  - lb
  - lo
  - lt
  - lv
  - mg
  - mi
  - mk
  - ml
  - mn
  - mr
  - ms
  - mt
  - my
  - ne
  - nl
  - 'no'
  - ny
  - pa
  - pl
  - ps
  - pt
  - ro
  - ru
  - sd
  - si
  - sk
  - sl
  - sm
  - sn
  - so
  - sq
  - sr
  - st
  - su
  - sv
  - sw
  - ta
  - te
  - tg
  - th
  - tr
  - uk
  - und
  - ur
  - uz
  - vi
  - xh
  - yi
  - yo
  - zh
  - zu
datasets:
  - mc4
  - bigscience/xP3

Multilingual Text-to-Text Transfer Transformer Zero (MT0) Version 1. / 28 October 2022

Models

mT5 is pretrained on the mC4 corpus, covering 101 languages:

Afrikaans, Albanian, Amharic, Arabic, Armenian, Azerbaijani, Basque, Belarusian, Bengali, Bulgarian, Burmese, Catalan, Cebuano, Chichewa, Chinese, Corsican, Czech, Danish, Dutch, English, Esperanto, Estonian, Filipino, Finnish, French, Galician, Georgian, German, Greek, Gujarati, Haitian Creole, Hausa, Hawaiian, Hebrew, Hindi, Hmong, Hungarian, Icelandic, Igbo, Indonesian, Irish, Italian, Japanese, Javanese, Kannada, Kazakh, Khmer, Korean, Kurdish, Kyrgyz, Lao, Latin, Latvian, Lithuanian, Luxembourgish, Macedonian, Malagasy, Malay, Malayalam, Maltese, Maori, Marathi, Mongolian, Nepali, Norwegian, Pashto, Persian, Polish, Portuguese, Punjabi, Romanian, Russian, Samoan, Scottish Gaelic, Serbian, Shona, Sindhi, Sinhala, Slovak, Slovenian, Somali, Sotho, Spanish, Sundanese, Swahili, Swedish, Tajik, Tamil, Telugu, Thai, Turkish, Ukrainian, Urdu, Uzbek, Vietnamese, Welsh, West Frisian, Xhosa, Yiddish, Yoruba, Zulu.

mt5 was then finetuned on:

Model Flavors

Multilingual model capable of following user instructions in a variety of languages. Together with our paper [TODO: LINK], we release the following models:

Basics

This section provides information about the model type, version, license, funders, release date, developers, and contact information. It is useful for anyone who wants to reference the model.

Click to expand

All collaborators are either volunteers or have an agreement with their employer. (Further breakdown of participants forthcoming.)

Model Type: Transformer-based Language Model

Checkpoints format: transformers

Version: 1.0.0

Languages: Multiple; see training data

License: Apache 2.0

Release Date Estimate: Friday, 28.October.2022

Send Questions to: niklas@huggingface.co

Funded by:

  • The French government.
  • Hugging Face (website).

Technical Specifications

This section includes details about the model objective and architecture, and the compute infrastructure. It is useful for people interested in model development.

Click to expand

Model Architecture and Objective

  • Same architecture as mt5

  • Encoder-decoder architecture

Objective Function: Cross Entropy with mean reduction on target tokens (see API documentation).

Compute infrastructure

Models were finetuned on TPUv4:

  • mt0-small was finetuned on TPUv4-64
  • mt0-base was finetuned on TPUv4-64
  • mt0-large was finetuned on TPUv4-64
  • mt0-xl was finetuned on TPUv4-128
  • mt0-xxl was finetuned on TPUv4-256
  • mt0-mt-xxl was finetuned on TPUv4-256
  • mt0-p3-xxl was finetuned on TPUv4-256

Software

Training

This section provides information about the training data, the speed and size of training elements, and the environmental impact of training. It is useful for people who want to learn more about the model inputs and training footprint.

Click to expand

Training Data

This section provides a high-level overview of the training data. It is relevant for anyone who wants to know the basics of what the model is learning.

It was pretrained on mC4 and then finetuned on xP3, P3 or xP3mt.

Languages

// TODO @thomasw21: Copy list from mt5

Speeds, Sizes, Times

// TODO @adarob: Maybe we can push tensorboard on this repo as well

  • Training logs:

  • Checkpoint size: 51.7GB (Bf16 weights)

  • Number of epochs: 1

  • Precision: bfloat16

Environmental Impact

The evaluation supercomputer, Jean Zay, uses mostly nuclear energy. The heat generated by it is reused for heating campus housing.

Uses

This section addresses questions around how the model is intended to be used, discusses the foreseeable users of the model (including those affected by the model), and describes uses that are considered out of scope or misuse of the model. It is useful for anyone considering using the model or who is affected by the model.

Click to expand

How to use

This model can be easily used and deployed using HuggingFace's ecosystem. This needs transformers and accelerate installed. The model can be downloaded as follows:

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

checkpoint = "..." # "checkpoint_1006000" for example
model_name = "bigscience/mt0-xxl"
model = AutoModelForSeq2SeqLM.from_pretrained(model_name, revision=checkpoint, torch_dtype="auto", device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name, revision=checkpoint)

inputs = tokenizer.encode("Commentaire: C'est la meilleure crêpière que j'ai jamais eu. Je l'adore.\nCe commentaire est-il positif ou négatif?", return_tensors="pt")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))

Intended Use

This model is being created in order to enable public research on large language models (LLMs). LLMs are intended to be used for language generation or as a pretrained base model that can be further finetuned for specific tasks. Use cases below are not exhaustive.

Direct Use

  • Text generation

  • Exploring characteristics of language generated by a language model

    • Examples: Cloze tests, counterfactuals, generations with reframings

Downstream Use

  • Tasks that leverage language models include: Information Extraction, Question Answering, Summarization

Risks and Limitations

This section identifies foreseeable harms and misunderstandings.

Click to expand

Model may:

  • Overrepresent some viewpoints and underrepresent others

  • Contain stereotypes

  • Contain personal information

  • Generate:

    • Hateful, abusive, or violent language

    • Discriminatory or prejudicial language

    • Content that may not be appropriate for all settings, including sexual content

  • Make errors, including producing incorrect information as if it were factual

  • Generate irrelevant or repetitive outputs

  • Induce users into attributing human traits to it, such as sentience or consciousness

Evaluation

This section describes the evaluation protocols and provides the results.

Click to expand

Results

Results are based on the Metrics.

Train-time Evaluation:

// TODO @adarob: Pending if we can get access to tensorboard

Zero-shot evaluations:

// TODO @niklas

Recommendations

This section provides information on warnings and potential mitigations.

Click to expand

  • Indirect users should be made aware when the content they're working with is created by the LLM.

  • Users should be aware of Risks and Limitations, and include an appropriate age disclaimer or blocking interface as necessary.

  • Models trained or finetuned downstream of MT0 should include an updated Model Card.

  • Users of the model should provide mechanisms for those affected to provide feedback, such as an email address for comments.

Glossary and Calculations

This section defines common terms and how metrics are calculated.

Click to expand

More Information

This section provides links to writing on dataset creation, technical specifications, lessons learned, and initial results.

Click to expand

Intermediate checkpoints

For academic (or any) usage, we published the intermediate checkpoints, corresponding to the model state at each 1000 steps. There are available as branches in this repository. You can use them using transformers:

from transformers import AutoModel

checkpoint = "..." # "checkpoint_1006000" for example
model = AutoModel.from_pretrained("bigscience/mt0-xxl", revision=checkpoint, torch_dtype="auto", device_map="auto")

Dataset Creation

// TODO @niklas: Point to the arxiv paper

Original checkpoints

The checkpoints in this repo correspond to the HuggingFace Transformers format. We'll provide T5X checkpoints as well.

Citing MT0

Please use the following bibtex entry to cite T0:

TODO @niklas