VictorSanh commited on
Commit
b031650
1 Parent(s): bad4725

Add numbers for wino bias and rename models

Browse files
Files changed (1) hide show
  1. README.md +134 -32
README.md CHANGED
@@ -13,23 +13,23 @@ You can use the models to perform inference on tasks by specifying your query in
13
 
14
  # How to use
15
 
16
- We make available the models presented in our [paper](TODO) along with the ablation models. We recommend using the [T0pp_11B](https://huggingface.co/bigscience/T0pp_11B) (pronounce "T zero plus plus") checkpoint as it leads (on average) to the best performances on a variety of NLP tasks.
17
 
18
  |Model|Number of parameters|
19
  |-|-|
20
- |[T0_11B](https://huggingface.co/bigscience/T0_11B)|11 billion|
21
- |[T0p_11B](https://huggingface.co/bigscience/T0p_11B)|11 billion|
22
- |[T0pp_11B](https://huggingface.co/bigscience/T0pp_11B)|11 billion|
23
- |[T0_11B_single_prompt](https://huggingface.co/bigscience/T0_11B_single_prompt)|11 billion|
24
- |[T0_11B_original_task_only](https://huggingface.co/bigscience/T0_11B_original_task_only)|11 billion|
25
  |[T0_3B](https://huggingface.co/bigscience/T0_3B)|3 billion|
26
 
27
  Here is how to use the model in PyTorch:
28
  ```python
29
  from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
30
 
31
- tokenizer = AutoTokenizer.from_pretrained("bigscience/T0pp_11B")
32
- model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp_11B")
33
 
34
  inputs = tokenizer.encode("Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy", return_tensors="pt")
35
  outputs = model.generate(inputs)
@@ -61,14 +61,14 @@ We trained different variants T0 with different mixtures of datasets.
61
 
62
  |Model|Training datasets|
63
  |--|--|
64
- |T0_11B|- Multiple-Choice QA: CommonsenseQA, DREAM, QUAIL, QuaRTz, Social IQA, WiQA, Cosmos, QASC, Quarel, SciQ, Wiki Hop<br>- Extractive QA: Adversarial QA, Quoref, TyDiQA, DuoRC, ROPES<br>- Closed-Book QA: Hotpot QA*, Wiki QA<br>- Structure-To-Text: Common Gen, Wiki Bio<br>- Sentiment: Amazon, App Reviews, IMDB, Rotten Tomatoes, Yelp<br>- Summarization: CNN Daily Mail, Gigaword, MultiNews, SamSum, XSum<br>- Topic Classification: AG News, DBPedia, TREC<br>- Paraphrase Identification: MRPC, PAWS, QQP|
65
- |T0p_11B|Same as T0_11B with additional datasets from GPT-3's evaluation suite:<br>- Multiple-Choice QA: ARC, OpenBook QA, PiQA, RACE, HellaSwag<br>- Extractive QA: SQuAD v2<br>- Closed-Book QA: Trivia QA, Web Questions|
66
- |T0pp_11B|Same as T0p_11B with a few additional datasets from SuperGLUE (excluding NLI sets):<br>- BoolQ<br>- COPA<br>- MultiRC<br>- ReCoRD<br>- WiC<br>- WSC|
67
- |T0_11B_single_prompt|Same as T0_11B but only one prompt per training dataset|
68
- |T0_11B_original_task_only|Same as T0_11B but only original tasks templates|
69
- |T0_3B|Same as T0_11B but starting from a T5-LM XL (3B parameters) pre-trained model|
70
 
71
- For reproducibility, we release the data we used for training (and evaluation) in the [P3 dataset](TODO). Prompts examples can be found on the dataset page.
72
 
73
  *: We recast Hotpot QA as closed-book QA due to long input sequence length.
74
 
@@ -83,7 +83,7 @@ We evaluate our models on a suite of held-out tasks:
83
  |Word sense disambiguation|WiC|
84
  |Sentence completion|COPA, HellaSwag, Story Cloze|
85
 
86
- We also evaluate T0_11B, T0p_11B and T0pp_11B on the a subset of the [BIG-bench benchmark](https://github.com/google/BIG-bench):
87
  - Code description task
88
  - Conceptual combinations
89
  - Hindu knowledge json
@@ -117,24 +117,126 @@ Since language models are trained via token prediction over a large (and typical
117
 
118
  To measure the ability of our model to recognize gender biases, we evaluate our models using the WinoGender Schemas (also called AXG under SuperGLUE) and CrowS-Pairs. WinoGender Schemas are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias. We use the *Diverse Natural Language Inference Collection* ([Poliak et al., 2018](https://aclanthology.org/D18-1007/)) version that casts WinoGender as a textual entailment task and report accuracy. CrowS-Pairs is a challenge dataset for measuring the degree to which U.S. stereotypical biases present in the masked language models using minimal pairs of sentences. We re-formulate the task by predicting which of two sentences is stereotipycal (or anti-stereotipycal) and report accuracy. For each dataset, we evaluate between 5 and 10 prompts.
119
 
120
- |Set|Model|Metric|Average|Median|
121
- |-|-|-|-|-|
122
- |CrowS-Pairs|T0_11B|Acc.|59.2|83.8|
123
- |CrowS-Pairs|T0p_11B|Acc.|57.6|83.8|
124
- |CrowS-Pairs|T0pp_11B|Acc.|62.7|64.4|
125
- |CrowS-Pairs|T0_11B_single_prompt|Acc.|57.6|69.5|
126
- |CrowS-Pairs|T0_11B_original_task_only|Acc.|47.1|37.8|
127
- |CrowS-Pairs|T0_3B|Acc.|56.9|82.6|
128
- |WinoGender|T0_11B|Acc.|84.2|84.3|
129
- |WinoGender|T0p_11B|Acc.|80.1|80.6|
130
- |WinoGender|T0pp_11B|Acc.|89.2|90.0|
131
- |WinoGender|T0_11B_single_prompt|Acc.|81.6|84.6|
132
- |WinoGender|T0_11B_original_task_only|Acc.|83.7|83.8|
133
- |WinoGender|T0_3B|Acc.|69.7|69.4|
134
-
135
- To measure the extent to which our model reproduces gender biases, we evaluate our models using the WinoBias Schemas. WinoBias Schemas are pronoun coreference resolution tasks that have the potential to be influenced by gender bias. WinoBias Schemas has two schemas (type1 and type2) which are partitioned into pro-stereotype and anti-stereotype subsets. An "anti-stereotype" example is one where the correct answer conforms to stereotypes, while an "anti-stereotype" example is one where it opposes stereotypes. All examples have an unambiguously correct answer, and so the difference in scores between the "pro-" and "anti-" subset measures the extent to which stereotypes can lead the model astray. We report F1 scores, as recommended by the WinoBias paper. We evaluate on 6 prompts.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
136
 
137
  # BibTeX entry and citation info
138
 
139
  ```bibtex
 
140
  ```
 
13
 
14
  # How to use
15
 
16
+ We make available the models presented in our [paper](TODO) along with the ablation models. We recommend using the [T0pp](https://huggingface.co/bigscience/T0pp) (pronounce "T zero plus plus") checkpoint as it leads (on average) to the best performances on a variety of NLP tasks.
17
 
18
  |Model|Number of parameters|
19
  |-|-|
20
+ |[T0](https://huggingface.co/bigscience/T0)|11 billion|
21
+ |[T0p](https://huggingface.co/bigscience/T0p)|11 billion|
22
+ |[T0pp](https://huggingface.co/bigscience/T0pp)|11 billion|
23
+ |[T0_single_prompt](https://huggingface.co/bigscience/T0_single_prompt)|11 billion|
24
+ |[T0_original_task_only](https://huggingface.co/bigscience/T0_original_task_only)|11 billion|
25
  |[T0_3B](https://huggingface.co/bigscience/T0_3B)|3 billion|
26
 
27
  Here is how to use the model in PyTorch:
28
  ```python
29
  from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
30
 
31
+ tokenizer = AutoTokenizer.from_pretrained("bigscience/T0pp")
32
+ model = AutoModelForSeq2SeqLM.from_pretrained("bigscience/T0pp")
33
 
34
  inputs = tokenizer.encode("Is this review positive or negative? Review: this is the best cast iron skillet you will ever buy", return_tensors="pt")
35
  outputs = model.generate(inputs)
 
61
 
62
  |Model|Training datasets|
63
  |--|--|
64
+ |T0|- Multiple-Choice QA: CommonsenseQA, DREAM, QUAIL, QuaRTz, Social IQA, WiQA, Cosmos, QASC, Quarel, SciQ, Wiki Hop<br>- Extractive QA: Adversarial QA, Quoref, TyDiQA, DuoRC, ROPES<br>- Closed-Book QA: Hotpot QA*, Wiki QA<br>- Structure-To-Text: Common Gen, Wiki Bio<br>- Sentiment: Amazon, App Reviews, IMDB, Rotten Tomatoes, Yelp<br>- Summarization: CNN Daily Mail, Gigaword, MultiNews, SamSum, XSum<br>- Topic Classification: AG News, DBPedia, TREC<br>- Paraphrase Identification: MRPC, PAWS, QQP|
65
+ |T0p|Same as T0 with additional datasets from GPT-3's evaluation suite:<br>- Multiple-Choice QA: ARC, OpenBook QA, PiQA, RACE, HellaSwag<br>- Extractive QA: SQuAD v2<br>- Closed-Book QA: Trivia QA, Web Questions|
66
+ |T0pp|Same as T0p with a few additional datasets from SuperGLUE (excluding NLI sets):<br>- BoolQ<br>- COPA<br>- MultiRC<br>- ReCoRD<br>- WiC<br>- WSC|
67
+ |T0_single_prompt|Same as T0 but only one prompt per training dataset|
68
+ |T0_original_task_only|Same as T0 but only original tasks templates|
69
+ |T0_3B|Same as T0 but starting from a T5-LM XL (3B parameters) pre-trained model|
70
 
71
+ For reproducibility, we release the data we used for training (and evaluation) in the [P3 dataset](https://huggingface.co/datasets/bigscience/P3). Prompts examples can be found on the dataset page.
72
 
73
  *: We recast Hotpot QA as closed-book QA due to long input sequence length.
74
 
 
83
  |Word sense disambiguation|WiC|
84
  |Sentence completion|COPA, HellaSwag, Story Cloze|
85
 
86
+ We also evaluate T0, T0p and T0pp on the a subset of the [BIG-bench benchmark](https://github.com/google/BIG-bench):
87
  - Code description task
88
  - Conceptual combinations
89
  - Hindu knowledge json
 
117
 
118
  To measure the ability of our model to recognize gender biases, we evaluate our models using the WinoGender Schemas (also called AXG under SuperGLUE) and CrowS-Pairs. WinoGender Schemas are minimal pairs of sentences that differ only by the gender of one pronoun in the sentence, designed to test for the presence of gender bias. We use the *Diverse Natural Language Inference Collection* ([Poliak et al., 2018](https://aclanthology.org/D18-1007/)) version that casts WinoGender as a textual entailment task and report accuracy. CrowS-Pairs is a challenge dataset for measuring the degree to which U.S. stereotypical biases present in the masked language models using minimal pairs of sentences. We re-formulate the task by predicting which of two sentences is stereotipycal (or anti-stereotipycal) and report accuracy. For each dataset, we evaluate between 5 and 10 prompts.
119
 
120
+ <table>
121
+ <tr>
122
+ <td>Dataset</td>
123
+ <td>Model</td>
124
+ <td>Average</td>
125
+ <td>Median</td>
126
+ </tr>
127
+ <tr>
128
+ <td rowspan="10">CrowS-Pairs</td><td>T0</td><td>59.2</td><td>83.8</td>
129
+ </tr>
130
+ <td>T0p</td><td>57.6</td><td>83.8</td>
131
+ <tr>
132
+ </tr>
133
+ <td>T0pp</td><td>62.7</td><td>64.4</td>
134
+ <tr>
135
+ </tr>
136
+ <td>T0_single_prompt</td><td>57.6</td><td>69.5</td>
137
+ <tr>
138
+ </tr>
139
+ <td>T0_original_task_only</td><td>47.1</td><td>37.8</td>
140
+ <tr>
141
+ </tr>
142
+ <td>T0_3B</td><td>56.9</td><td>82.6</td>
143
+ </tr>
144
+ <tr>
145
+ <td rowspan="10">WinoGender</td><td>T0</td><td>84.2</td><td>84.3</td>
146
+ </tr>
147
+ <td>T0p</td><td>80.1</td><td>80.6</td>
148
+ <tr>
149
+ </tr>
150
+ <td>T0pp</td><td>89.2</td><td>90.0</td>
151
+ <tr>
152
+ </tr>
153
+ <td>T0_single_prompt</td><td>81.6</td><td>84.6</td>
154
+ <tr>
155
+ </tr>
156
+ <td>T0_original_task_only</td><td>83.7</td><td>83.8</td>
157
+ <tr>
158
+ </tr>
159
+ <td>T0_3B</td><td>69.7</td><td>69.4</td>
160
+ </tr>
161
+ </table>
162
+
163
+ To measure the extent to which our model reproduces gender biases, we evaluate our models using the WinoBias Schemas. WinoBias Schemas are pronoun coreference resolution tasks that have the potential to be influenced by gender bias. WinoBias Schemas has two schemas (type1 and type2) which are partitioned into pro-stereotype and anti-stereotype subsets. A "pro-stereotype" example is one where the correct answer conforms to stereotypes, while an "anti-stereotype" example is one where it opposes stereotypes. All examples have an unambiguously correct answer, and so the difference in scores between the "pro-" and "anti-" subset measures the extent to which stereotypes can lead the model astray. We report accuracies by considering a prediction correct if the noun predicted by the model is present in the target. We evaluate on 6 prompts.
164
+
165
+ <table>
166
+ <tr>
167
+ <td rowspan="2">Model</td>
168
+ <td rowspan="2">Subset</td>
169
+ <td colspan="3">Average</td>
170
+ <td colspan="3">Median</td>
171
+ </tr>
172
+ <tr>
173
+ <td>Pro</td>
174
+ <td>Anti</td>
175
+ <td>Pro - Anti</td>
176
+ <td>Pro</td>
177
+ <td>Anti</td>
178
+ <td>Pro - Anti</td>
179
+ </tr>
180
+
181
+ <tr>
182
+ <td rowspan="2">T0</td>
183
+ <td>Type 1</td>
184
+ <td>68.0</td><td>61.9</td><td>6.0</td><td>71.7</td><td>61.9</td><td>9.8</td>
185
+ </tr>
186
+ <td>Type 2</td>
187
+ <td>79.3</td><td>76.4</td><td>2.8</td><td>79.3</td><td>75.0</td><td>4.3</td>
188
+ </tr>
189
+ </tr>
190
+ <td rowspan="2">T0p</td>
191
+ <td>Type 1</td>
192
+ <td>66.6</td><td>57.2</td><td>9.4</td><td>71.5</td><td>62.6</td><td>8.8</td>
193
+ </tr>
194
+ </tr>
195
+ <td>Type 2</td>
196
+ <td>77.7</td><td>73.4</td><td>4.3</td><td>86.1</td><td>81.3</td><td>4.8</td>
197
+ </tr>
198
+ </tr>
199
+ <td rowspan="2">T0pp</td>
200
+ <td>Type 1</td>
201
+ <td>63.8</td><td>55.9</td><td>7.9</td><td>72.7</td><td>63.4</td><td>9.3</td>
202
+ </tr>
203
+ </tr>
204
+ <td>Type 2</td>
205
+ <td>66.8</td><td>63.0</td><td>3.9</td><td>79.3</td><td>74.0</td><td>5.3</td>
206
+ </tr>
207
+ </tr>
208
+ <td rowspan="2">T0_single_prompt</td>
209
+ <td>Type 1</td>
210
+ <td>82.3</td><td>70.1</td><td>12.2</td><td>83.6</td><td>62.9</td><td>20.7</td>
211
+ </tr>
212
+ </tr>
213
+ <td>Type 2</td>
214
+ <td>83.8</td><td>76.5</td><td>7.3</td><td>85.9</td><td>75.0</td><td>10.9</td>
215
+ </tr>
216
+
217
+ </tr>
218
+ <td rowspan="2">T0_original_task_only</td>
219
+ <td>Type 1</td>
220
+ <td>73.7</td><td>60.5</td><td>13.2</td><td>79.3</td><td>60.6</td><td>18.7</td>
221
+ </tr>
222
+ </tr>
223
+ <td>Type 2</td>
224
+ <td>77.7</td><td>69.6</td><td>8.0</td><td>80.8</td><td>69.7</td><td>11.1</td>
225
+ </tr>
226
+
227
+ </tr>
228
+ <td rowspan="2">T0_3B</td>
229
+ <td>Type 1</td>
230
+ <td>82.3</td><td>70.1</td><td>12.2</td><td>83.6</td><td>62.9</td><td>20.7</td>
231
+ </tr>
232
+ </tr>
233
+ <td>Type 2</td>
234
+ <td>83.8</td><td>76.5</td><td>7.3</td><td>85.9</td><td>75</td><td>10.9</td>
235
+ </tr>
236
+ </table>
237
 
238
  # BibTeX entry and citation info
239
 
240
  ```bibtex
241
+ TODO
242
  ```