---
language: en
license: apache-2.0
---
# Model Description
T0* is a series of encoder-decoder models trained on a large set of different tasks specified in natural language prompts. We convert numerous English supervised datasets into prompts, each with multiple templates using varying formulations. These prompted datasets allow for benchmarking the ability of a model to perform completely unseen tasks specified in natural language. To obtain T0*, we fine-tune a pretrained language model on this multitask mixture covering many different NLP tasks.
# Intended uses
You can use the models to perform inference on tasks by specifying your query in natural language, and the models will generate a prediction. For instance, you can ask *"Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy"*, and the model will hopefully generate *"Positive"*.
# How to use
We make available the models presented in our [paper](TODO) along with the ablation models. We recommend using the [T0pp_11B](https://huggingface.co/bigscience/T0pp_11B) checkpoint as it leads (on average) to the best performances on a variety of NLP tasks.
|Model|Number of parameters|
|-|-|
|[T0_11B](https://huggingface.co/bigscience/T0_11B)|11 billion|
|[T0p_11B](https://huggingface.co/bigscience/T0p_11B)|11 billion|
|[T0pp_11B](https://huggingface.co/bigscience/T0pp_11B)|11 billion|
|[T0_11B_single_prompt](https://huggingface.co/bigscience/T0_11B_single_prompt)|11 billion|
|[T0_11B_original_task_only](https://huggingface.co/bigscience/T0_11B_original_task_only)|11 billion|
|[T0_3B](https://huggingface.co/bigscience/T0_3B)|3 billion|
Here is how to use the model in PyTorch:
```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("bigscience/T0pp_11B")
model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp_11B")
inputs = tokenizer.encode("Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy", return_tensors="pt")
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
```
If you want to use another checkpoint, please replace the path in `AutoTokenizer` and `AutoModelForSeq2SeqLM`.
# Training procedure
T0* models are based on [T5](https://huggingface.co/google/t5-v1_1-large), a Transformer-based encoder-decoder language model pre-trained with a masked language modeling-style objective on 34B tokens from [C4](https://huggingface.co/datasets/c4). We use the publicly available [language model-adapated T5 checkpoints](https://github.com/google-research/text-to-text-transfer-transformer/blob/main/released_checkpoints.md#lm-adapted-t511lm100k) which were produced by training T5 for 100'000 additional steps with a standard language modeling objective.
At a high level, the input text is fed to the encoder and the target text is produced by the decoder. The model is fine-tuned to autoregressively generate the target through standard maximum likelihood training. It is never trained to generate the input. We detail our training data in the next section.
Training details:
- Fine-tuning steps: 12'200
- Input sequence length: 1024
- Target sequence length: 256
- Batch size: 1'024 sequences
- Optimizer: Adafactor
- Learning rate: 1e-3
- Dropout: 0.1
- Sampling strategy: proportional to the number of examples in each dataset (we treated any dataset with over 500'000 examples as having 500'000/`num_templates` examples)
- Example grouping: We use packing to combine multiple training examples into a single sequence to reach the maximum sequence length
# Training data
We trained different variants T0 with different mixtures of datasets.
|Model|Training datasets|
|--|--|
|T0_11B|- Multiple-Choice QA: CommonsenseQA, DREAM, QUAIL, QuaRTz, Social IQA, WiQA, Cosmos, QASC, Quarel, SciQ, Wiki Hop
- Extractive QA: Adversarial QA, Quoref, TyDiQA, DuoRC, ROPES
- Closed-Book QA: Hotpot QA, Wiki QA
- Structure-To-Text: Common Gen, Wiki Bio
- Sentiment: Amazon, App Reviews, IMDB, Rotten Tomatoes, Yelp
- Summarization: CNN Daily Mail, Gigaword, MultiNews, SamSum, XSum
- Topic Classification: AG News, DBPedia, TREC
- Paraphrase Identification: MRPC, PAWS, QQP|
|T0p_11B|Same as T0_11B with a few additional datasets:
- Multiple-Choice QA: ARC, Circa, MC-TACO, Open Book QA, PiQA, RACE
- Extractive QA: CoQA, DROP, QA SRL,QuAC, ReCoRD, SQuAD v2
- Closed-Book QA: NQ Open, Trivia QA, Web Questions|
|T0pp_11B|Same as T0p_11B with a few additional datasets from SuperGLUE:
- BoolQ
- COPA
- MultiRC
- ReCoRD
- WiC
- WSC|
|T0_11B_single_prompt|Same as T0_11B but only one prompt per training dataset|
|T0_11B_original_task_only|Same as T0_11B but only original tasks templates|
|T0_3B|Same as T0_11B but starting from a T5-LM XL (3B parameters) pre-trained model|
For reproducibility, we release the data we used for training (and evaluation) in the [P3 dataset](TODO).
# Evaluation data
We systematically evaluate our models on a suite of held-out tasks:
|Task category|Datasets|
|-|-|
|Natural language inference|ANLI, CB, HANS, RTE|
|Coreference resolution|WSC, Winogrande|
|Word sense disambiguation|WiC|
|Sentence completion|COPA, HellaSwag, Story Cloze|
We also evaluate T0_11B, T0p_11B and T0pp_11B on the a subset of the [BIG-bench benchmark](https://github.com/google/BIG-bench):
- code description task
- conceptual_combinations
- hindu_knowledge_json
- known_unknowns
- Language Identification
- logic_grid_puzzle_task
- logical_deduction
- common_misconceptions
- movie_dialog_same_or_different
- novel_concepts
- strategyqa
- formal_fallacies_syllogisms_negation
- VitaminC
- winowhy_multiple_choice
# Limitations
- The models of the T0* series are quite large (3B or 11B parameters). Loading them and performing inference requires non-trivial computational ressources. When using multiple GPUs, it is possible to use [.parallelize()](https://huggingface.co/transformers/parallelism.html).
- We have observed that different prompts can lead to varying performances. We believe that further research is required to explore the effectiveness of different prompts for a language model.
- Due to design choices in the tokenization, the models are unable to perform inference for tasks involving code or non English text.
# Bias and fairness
Even if we took conservative decisions to exclude datasets with potentially harmful content, this model can have biased predictions. To evaluate our models on bias and fairness, we consider a series of datasets: [WinoGender Schemas](https://huggingface.co/datasets/super_glue) (also called AXG under SuperGLUE), [CrowS-Pairs](https://huggingface.co/datasets/crows_pairs), [WinoBias](https://huggingface.co/datasets/wino_bias), WIP.
WinoGender Schemas are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias. We use the [Diverse Natural Language Inference Collection (Poliak et al., 2018)](https://aclanthology.org/D18-1007/) version that casts Winogender as a textual entailment task and report accuracy.
CrowS-Pairs is a challenge dataset for measuring the degree to which U.S. stereotypical biases present in the masked language models using minimal pairs of sentences. We re-formulate the task by predicting which of two sentences is stereotipycal (or anti-stereotipycal) and report accuracy. For each dataset, we evaluate between 5 and 10 prompts.
|Set|Model|Metric|Average|Median|
|-|-|-|-|-|
|CrowS-Pairs|T0_11B|Acc.|59.2|83.8|
|CrowS-Pairs|T0p_11B|Acc.|57.6|83.8|
|CrowS-Pairs|T0pp_11B|Acc.|62.7|64.4|
|CrowS-Pairs|T0_11B_single_prompt|Acc.|57.6|69.5|
|CrowS-Pairs|T0_11B_original_task_only|Acc.|47.1|37.8|
|CrowS-Pairs|T0_3B|Acc.|56.9|82.6|
|WinoGender|T0_11B|Acc.|84.2|84.3|
|WinoGender|T0p_11B|Acc.|80.1|80.6|
|WinoGender|T0pp_11B|Acc.|89.2|90.0|
|WinoGender|T0_11B_single_prompt|Acc.|81.6|84.6|
|WinoGender|T0_11B_original_task_only|Acc.|83.7|83.8|
|WinoGender|T0_3B|Acc.|69.7|69.4|
# BibTeX entry and citation info
```bibtex
```