|
--- |
|
language: |
|
- af |
|
- am |
|
- ar |
|
- az |
|
- be |
|
- bg |
|
- bn |
|
- ca |
|
- ceb |
|
- co |
|
- cs |
|
- cy |
|
- da |
|
- de |
|
- el |
|
- en |
|
- eo |
|
- es |
|
- et |
|
- eu |
|
- fa |
|
- fi |
|
- fil |
|
- fr |
|
- fy |
|
- ga |
|
- gd |
|
- gl |
|
- gu |
|
- ha |
|
- haw |
|
- he |
|
- hi |
|
- hmn |
|
- ht |
|
- hu |
|
- hy |
|
- id |
|
- ig |
|
- is |
|
- it |
|
- iw |
|
- ja |
|
- jv |
|
- ka |
|
- kk |
|
- km |
|
- kn |
|
- ko |
|
- ku |
|
- ky |
|
- la |
|
- lb |
|
- lo |
|
- lt |
|
- lv |
|
- mg |
|
- mi |
|
- mk |
|
- ml |
|
- mn |
|
- mr |
|
- ms |
|
- mt |
|
- my |
|
- ne |
|
- nl |
|
- 'no' |
|
- ny |
|
- pa |
|
- pl |
|
- ps |
|
- pt |
|
- ro |
|
- ru |
|
- sd |
|
- si |
|
- sk |
|
- sl |
|
- sm |
|
- sn |
|
- so |
|
- sq |
|
- sr |
|
- st |
|
- su |
|
- sv |
|
- sw |
|
- ta |
|
- te |
|
- tg |
|
- th |
|
- tr |
|
- uk |
|
- und |
|
- ur |
|
- uz |
|
- vi |
|
- xh |
|
- yi |
|
- yo |
|
- zh |
|
- zu |
|
license: mit |
|
datasets: |
|
- mc4 |
|
--- |
|
|
|
# MyT5 |
|
|
|
|
|
|
|
## Model Details |
|
|
|
MyT5 (**My**te **T5**) is a multilingual language model based on T5 architecture. |
|
The model uses a **m**orphologically-driven **byte** (**MYTE**) representation described in our paper [Limisiewicz et al., 2024](https://arxiv.org/pdf/2403.10691.pdf). |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
- **Developed by:** Tomasz Limisiewicz, Terra Blevins, Hila Gonen, Orevaoghene Ahia, Luke Zettlemoyer |
|
- **Funded by:** University of Washington Fellowship, Charles University Grant Agency |
|
- **Model type:** T5 |
|
- **Language(s) (NLP):** Multilingual |
|
- **License:** MIT |
|
|
|
### Model Sizes |
|
|
|
- **[Small](https://huggingface.co/Tomlim/myt5-small)**: 300M parameters |
|
- **[Base](https://huggingface.co/Tomlim/myt5-base)**: 582M parameters |
|
- **[Large](https://huggingface.co/Tomlim/myt5-large)**: 1.2B parameters |
|
|
|
### Model Sources |
|
|
|
<!-- Provide the basic links for the model. --> |
|
|
|
- **[Repository](https://github.com/tomlimi/MYTE)** |
|
- **[Paper](https://arxiv.org/pdf/2403.10691.pdf)** |
|
|
|
## How to Get Started with the Model |
|
|
|
The snippet below shows the basic usage of the model for multilingual language modeling. |
|
Custom Tokenizer is available in [GitHub](https://github.com/tomlimi/MYTE])repository, in `src/myt5/myt5_tokenizer.py`. |
|
We also plan to release it on HuggingFace in the future. |
|
|
|
```python |
|
from transformers import T5ForConditionalGeneration |
|
from src.myt5.myt5_tokenizer import MyT5Tokenizer |
|
import torch |
|
|
|
MODEL_SIZE = "large" # small, base, or large |
|
|
|
model = T5ForConditionalGeneration.from_pretrained(f"Tomlim/MyT5_{MODEL_SIZE}", use_safetensors=True) |
|
tokenizer = MyT5Tokenizer() |
|
|
|
pre_texts = ['"We now have', |
|
'„Mamy teraz myszy w wieku', |
|
'"""எங்களிடம் இப்போது'] |
|
post_texts = ['4-month-old mice that are non-diabetic that used to be diabetic," he added.', |
|
'4 miesięcy, które miały cukrzycę, ale zostały z niej wyleczone” – dodał.', |
|
'4-மாத-வயதுடைய எலி ஒன்று உள்ளது, முன்னர் அதற்கு நீரிழிவு இருந்தது தற்போது இல்லை"" என்று அவர் மேலும் கூறினார்."'] |
|
|
|
inputs = tokenizer(pre_texts, padding="longest", return_tensors="pt") |
|
targets = tokenizer(post_texts, padding="longest", return_tensors="pt") |
|
|
|
|
|
outputs = model(**inputs, labels=targets.input_ids) |
|
probs = torch.nn.functional.softmax(outputs.logits, dim=-1) |
|
``` |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
The model was trained on the standard T5 task of restoring corrupted spans in the multilingual MC4 dataset. |
|
|
|
### Preprocessing |
|
|
|
Instead of UTF-8 bytes, we used morphologically-driven byte representation. |
|
See the description in our [paper](https://arxiv.org/pdf/2403.10691.pdf) for more details. |
|
|
|
|
|
### Training Hyperparameters |
|
|
|
We used the same hyperparameters as in the original ByT5 paper. |
|
The only difference is that we decreased the number of training steps to 250,000 to avoid overfiting. |
|
|
|
### Computational Infrastructure |
|
|
|
Models were trained on TPUs available through TPU Research Cloud (TRC). |
|
We used v3-8 TPU for training small and base models and v3-32 for a large model. |
|
The training for each instance took: |
|
|
|
- **Small**: 90h |
|
- **Base**: 230h |
|
- **Large**: 190h |
|
|
|
# Evaluation |
|
|
|
<!-- This section describes the evaluation protocols and provides the results. --> |
|
|
|
MyT5 models are compared with reimplementation of [ByT5](https://huggingface.co/docs/transformers/model_doc/byt5) models trained for 250,000 steps. |
|
|
|
## Language Modeling |
|
|
|
We have evaluated LM performance on multi-parallel [FLORES 200](https://arxiv.org/pdf/2207.04672v3.pdf) corpus. |
|
To compare the scores across languages and models, we used a normalized metric, i.e., Bit-per-English-Byte (BPEB). |
|
|
|
### Results |
|
|
|
| | | ByT5 | | MyT5 | | |
|
|-------|-----------|------|--------|------|--------| |
|
| | | BPEB | T (ms) | BPEB | T (ms) | |
|
| small | All | 10.1 | 7.0 | 4.6 | 6.7 | |
|
| | Latin | 4.6 | 5.9 | 4.2 | 6.6 | |
|
| | Non Latin | 18.1 | 8.5 | 5.1 | 6.8 | |
|
| base | All | 8.2 | 11.5 | 5.8 | 8.9 | |
|
| | Latin | 4.9 | 9.4 | 5.0 | 8.7 | |
|
| | Non Latin | 13.0 | 14.6 | 6.9 | 9.1 | |
|
| large | All | 13.4 | 31.8 | 4.6 | 26.7 | |
|
| | Latin | 10.1 | 28.1 | 4.0 | 26.6 | |
|
| | Non Latin | 18.2 | 37.3 | 5.4 | 27.0 | |
|
|
|
Byte-per-English-Bits and Inference times (average per Flores 200 sentence) averaged for three language groupings. |
|
The inference was run on an A40 GPU core. |
|
|
|
## Downstream Tasks |
|
|
|
We tested the large model in four end-tasks: question answering, NER, semantic parsing, and machine translation. |
|
The test data come from XTREME-UP benchmark ([Ruder, Clark et al., 2023](https://arxiv.org/pdf/2305.11938.pdf)), which covers mainly low-resource languages |
|
|
|
### Fine-tuning |
|
|
|
In each task, we fine-tuned for all languages jointly. |
|
We used 1e-3 learning rate with square root decay and dropout of 0.1. |
|
The batch size and training varied across tasks: |
|
|
|
- **NER**: 128 examples per batch, 6000 steps |
|
- **QA**: 64 examples per batch, 6500 steps |
|
- **Semantic Parsing**: 64 examples per batch, 1000 steps |
|
- **MT**: 64 examples per batch, 10000 steps |
|
|
|
|
|
### Results |
|
|
|
Task | QA (F1) | NER (F1) | Semantic Parsing (EM)| MT (chrF) |
|
------------|------|------|------------------|------ |
|
Flan-PaLM* | 22.9 | 12.0 | 0.1 | --- |
|
mT5* | 59.7 | 74.0 | 21.8 | --- |
|
ByT5 | 73.2 | 81.5 | 25.1 | 20.1 |
|
MyT5 | 75.3 | 80.8 | 19.6 | 20.4 |
|
Inference Times per example (ms) |
|
ByT5 | 36.2 | 13.8 | 13.2 | 15.9 |
|
MyT5 | 35.6 | 12.6 | 12.4 | 12.6 |
|
|
|
The average result of XTREME-UP tasks across low-resource languages. |
|
The baseline results of mT5 and Flan-PaLM (in-context-learning evaluation) are reported in [Ruder, Clark et al., 2023](https://arxiv.org/pdf/2305.11938.pdf). |
|
The reported inference time is an average across evaluation examples; the inference was run on an A40 GPU core. |
|
|
|
## Citation |
|
|
|
```bibtex |
|
@misc{limisiewicz2024myte, |
|
title={MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling}, |
|
author={Tomasz Limisiewicz and Terra Blevins and Hila Gonen and Orevaoghene Ahia and Luke Zettlemoyer}, |
|
year={2024}, |
|
eprint={2403.10691}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
|
|
|
|
## Model Card Author |
|
|
|
[Tomasz Limisiewicz](mailto:limisewicz@ufal.mff.cuni.cz) |
