avi-skowron commited on
Commit
d8d8a47
1 Parent(s): 2026d54

Create README.md

Browse files
Files changed (1) hide show
  1. README.md +222 -0
README.md ADDED
@@ -0,0 +1,222 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - pytorch
6
+ - causal-lm
7
+ - pythia
8
+ license: apache-2.0
9
+ datasets:
10
+ - the_pile
11
+ ---
12
+
13
+ The *Pythia Scaling Suite* is a collection of models developed to facilitate
14
+ interpretability research. It contains two sets of eight models of sizes
15
+ 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B. For each size, there are two
16
+ models: one trained on the Pile, and one trained on the Pile after the dataset
17
+ has been globally deduplicated. All 8 model sizes are trained on the exact
18
+ same data, in the exact same order. All Pythia models are available
19
+ [on Hugging Face](https://huggingface.co/EleutherAI).
20
+
21
+ The Pythia model suite was deliberately designed to promote scientific
22
+ research on large language models, especially interpretability research.
23
+ Despite not centering downstream performance as a design goal, we find the
24
+ models match or exceed the performance of similar and same-sized models,
25
+ such as those in the OPT and GPT-Neo suites.
26
+
27
+ Please note that all models in the *Pythia* suite were renamed in January
28
+ 2023. For clarity, a <a href="#naming-convention-and-parameter-count">table
29
+ comparing the old and new names</a> is provided in this model card, together
30
+ with exact model parameter counts.
31
+
32
+ ## Pythia-1.4B
33
+
34
+ ### Model Details
35
+
36
+ - Developed by: [EleutherAI](http://eleuther.ai)
37
+ - Model type: Transformer-based Language Model
38
+ - Language: English
39
+ - Learn more: [Pythia's GitHub repository](https://github.com/EleutherAI/pythia)
40
+ for training procedure, config files, and details on how to use.
41
+ - Library: [GPT-NeoX](https://github.com/EleutherAI/gpt-neox)
42
+ - License: Apache 2.0
43
+ - Contact: to ask questions about this model, join the [EleutherAI
44
+ Discord](https://discord.gg/zBGx3azzUn), and post them in `#release-discussion`.
45
+ Please read the existing *Pythia* documentation before asking about it in the
46
+ EleutherAI Discord. For general correspondence: [contact@eleuther.
47
+ ai](mailto:contact@eleuther.ai).
48
+
49
+ <figure>
50
+
51
+ | Pythia model | Non-Embedding Params | Layers | Model Dim | Heads | Batch Size | Learning Rate | Equivalent Models |
52
+ | -----------: | -------------------: | :----: | :-------: | :---: | :--------: | :-------------------: | :--------------------: |
53
+ | 70M | 18,915,328 | 6 | 512 | 8 | 2M | 1.0 x 10<sup>-3</sup> | — |
54
+ | 160M | 85,056,000 | 12 | 768 | 12 | 4M | 6.0 x 10<sup>-4</sup> | GPT-Neo 125M, OPT-125M |
55
+ | 410M | 302,311,424 | 24 | 1024 | 16 | 4M | 3.0 x 10<sup>-4</sup> | OPT-350M |
56
+ | 1.0B | 805,736,448 | 16 | 2048 | 8 | 2M | 3.0 x 10<sup>-4</sup> | — |
57
+ | 1.4B | 1,208,602,624 | 24 | 2048 | 16 | 4M | 2.0 x 10<sup>-4</sup> | GPT-Neo 1.3B, OPT-1.3B |
58
+ | 2.8B | 2,517,652,480 | 32 | 2560 | 32 | 2M | 1.6 x 10<sup>-4</sup> | GPT-Neo 2.7B, OPT-2.7B |
59
+ | 6.9B | 6,444,163,072 | 32 | 4096 | 32 | 2M | 1.2 x 10<sup>-4</sup> | OPT-6.7B |
60
+ | 12B | 11,327,027,200 | 36 | 5120 | 40 | 2M | 1.2 x 10<sup>-4</sup> | — |
61
+ <figcaption>Engineering details for the <i>Pythia Suite</i>. Deduped and
62
+ non-deduped models of a given size have the same hyperparameters. “Equivalent”
63
+ models have <b>exactly</b> the same architecture, and the same number of
64
+ non-embedding parameters.</figcaption>
65
+ </figure>
66
+
67
+ ### Uses and Limitations
68
+
69
+ #### Intended Use
70
+
71
+ The primary intended use of Pythia is research on the behavior, functionality,
72
+ and limitations of large language models. This suite is intended to provide
73
+ a controlled setting for performing scientific experiments. To enable the
74
+ study of how language models change over the course of training, we provide
75
+ 143 evenly spaced intermediate checkpoints per model. These checkpoints are
76
+ hosted on Hugging Face as branches. Note that branch `143000` corresponds
77
+ exactly to the model checkpoint on the `main` branch of each model.
78
+
79
+ You may also further fine-tune and adapt Pythia-1.4B for deployment,
80
+ as long as your use is in accordance with the Apache 2.0 license. Pythia
81
+ models work with the Hugging Face [Transformers
82
+ Library](https://huggingface.co/docs/transformers/index). If you decide to use
83
+ pre-trained Pythia-1.4B as a basis for your fine-tuned model, please
84
+ conduct your own risk and bias assessment.
85
+
86
+ #### Out-of-scope use
87
+
88
+ The Pythia Suite is **not** intended for deployment. It is not a in itself
89
+ a product and cannot be used for human-facing interactions.
90
+
91
+ Pythia models are English-language only, and are not suitable for translation
92
+ or generating text in other languages.
93
+
94
+ Pythia-1.4B has not been fine-tuned for downstream contexts in which
95
+ language models are commonly deployed, such as writing genre prose,
96
+ or commercial chatbots. This means Pythia-1.4B will **not**
97
+ respond to a given prompt the way a product like ChatGPT does. This is because,
98
+ unlike this model, ChatGPT was fine-tuned using methods such as Reinforcement
99
+ Learning from Human Feedback (RLHF) to better “understand” human instructions.
100
+
101
+ #### Limitations and biases
102
+
103
+ The core functionality of a large language model is to take a string of text
104
+ and predict the next token. The token deemed statistically most likely by the
105
+ model need not produce the most “accurate” text. Never rely on
106
+ Pythia-1.4B to produce factually accurate output.
107
+
108
+ This model was trained on [the Pile](https://pile.eleuther.ai/), a dataset
109
+ known to contain profanity and texts that are lewd or otherwise offensive.
110
+ See [Section 6 of the Pile paper](https://arxiv.org/abs/2101.00027) for a
111
+ discussion of documented biases with regards to gender, religion, and race.
112
+ Pythia-1.4B may produce socially unacceptable or undesirable text, *even if*
113
+ the prompt itself does not include anything explicitly offensive.
114
+
115
+ If you plan on using text generated through, for example, the Hosted Inference
116
+ API, we recommend having a human curate the outputs of this language model
117
+ before presenting it to other people. Please inform your audience that the
118
+ text was generated by Pythia-1.4B.
119
+
120
+ ### Quickstart
121
+
122
+ Pythia models can be loaded and used via the following code, demonstrated here
123
+ for the third `pythia-70m-deduped` checkpoint:
124
+
125
+ ```python
126
+ from transformers import GPTNeoXForCausalLM, AutoTokenizer
127
+
128
+ model = GPTNeoXForCausalLM.from_pretrained(
129
+ "EleutherAI/pythia-70m-deduped",
130
+ revision="step3000",
131
+ cache_dir="./pythia-70m-deduped/step3000",
132
+ )
133
+
134
+ tokenizer = AutoTokenizer.from_pretrained(
135
+ "EleutherAI/pythia-70m-deduped",
136
+ revision="step3000",
137
+ cache_dir="./pythia-70m-deduped/step3000",
138
+ )
139
+
140
+ inputs = tokenizer("Hello, I am", return_tensors="pt")
141
+ tokens = model.generate(**inputs)
142
+ tokenizer.decode(tokens[0])
143
+ ```
144
+
145
+ Revision/branch `step143000` corresponds exactly to the model checkpoint on
146
+ the `main` branch of each model.
147
+
148
+ For more information on how to use all Pythia models, see [documentation on
149
+ GitHub](https://github.com/EleutherAI/pythia).
150
+
151
+ ### Training
152
+
153
+ #### Training data
154
+
155
+ [The Pile](https://pile.eleuther.ai/) is a 825GiB general-purpose dataset in
156
+ English. It was created by EleutherAI specifically for training large language
157
+ models. It contains texts from 22 diverse sources, roughly broken down into
158
+ five categories: academic writing (e.g. arXiv), internet (e.g. CommonCrawl),
159
+ prose (e.g. Project Gutenberg), dialogue (e.g. YouTube subtitles), and
160
+ miscellaneous (e.g. GitHub, Enron Emails). See [the Pile
161
+ paper](https://arxiv.org/abs/2101.00027) for a breakdown of all data sources,
162
+ methodology, and a discussion of ethical implications. Consult [the
163
+ datasheet](https://arxiv.org/abs/2201.07311) for more detailed documentation
164
+ about the Pile and its component datasets. The Pile can be downloaded from
165
+ the [official website](https://pile.eleuther.ai/), or from a [community
166
+ mirror](https://the-eye.eu/public/AI/pile/).
167
+
168
+ The Pile was **not** deduplicated before being used to train Pythia-1.4B.
169
+
170
+ #### Training procedure
171
+
172
+ Pythia uses the same tokenizer as [GPT-NeoX-
173
+ 20B](https://huggingface.co/EleutherAI/gpt-neox-20b).
174
+
175
+ All models were trained on the exact same data, in the exact same order. Each
176
+ model saw 299,892,736,000 tokens during training, and 143 checkpoints for each
177
+ model are saved every 2,097,152,000 tokens, spaced evenly throughout training.
178
+ This corresponds to training for just under 1 epoch on the Pile for
179
+ non-deduplicated models, and about 1.5 epochs on the deduplicated Pile.
180
+
181
+ All *Pythia* models trained for the equivalent of 143000 steps at a batch size
182
+ of 2,097,152 tokens. Two batch sizes were used: 2M and 4M. Models with a batch
183
+ size of 4M tokens listed were originally trained for 71500 steps instead, with
184
+ checkpoints every 500 steps. The checkpoints on Hugging Face are renamed for
185
+ consistency with all 2M batch models, so `step1000` is the first checkpoint
186
+ for `pythia-1.4b` that was saved (corresponding to step 500 in training), and
187
+ `step1000` is likewise the first `pythia-6.9b` checkpoint that was saved
188
+ (corresponding to 1000 “actual” steps).
189
+
190
+ See [GitHub](https://github.com/EleutherAI/pythia) for more details on training
191
+ procedure, including [how to reproduce
192
+ it](https://github.com/EleutherAI/pythia/blob/main/README.md#reproducing-training).
193
+
194
+ ### Evaluations
195
+
196
+ All 16 *Pythia* models were evaluated using the [LM Evaluation
197
+ Harness](https://github.com/EleutherAI/lm-evaluation-harness). You can access
198
+ the results by model and step at `results/json/*` in the [GitHub
199
+ repository](https://github.com/EleutherAI/pythia/tree/main/results/json).
200
+
201
+ February 2023 note: select evaluations and comparison with OPT and BLOOM
202
+ models will be added here at a later date.
203
+
204
+ ### Naming convention and parameter count
205
+
206
+ *Pythia* models were renamed in January 2023. It is possible that the old
207
+ naming convention still persists in some documentation by accident. The
208
+ current naming convention (70M, 160M, etc.) is based on total parameter count.
209
+
210
+ <figure style="width:32em">
211
+
212
+ | current Pythia suffix | old suffix | total params | non-embedding params |
213
+ | --------------------: | ---------: | -------------: | -------------------: |
214
+ | 70M | 19M | 70,426,624 | 18,915,328 |
215
+ | 160M | 125M | 162,322,944 | 85,056,000 |
216
+ | 410M | 350M | 405,334,016 | 302,311,424 |
217
+ | 1B | 800M | 1,011,781,632 | 805,736,448 |
218
+ | 1.4B | 1.3B | 1,414,647,808 | 1,208,602,624 |
219
+ | 2.8B | 2.7B | 2,775,208,960 | 2,517,652,480 |
220
+ | 6.9B | 6.7B | 6,857,302,016 | 6,444,163,072 |
221
+ | 12B | 13B | 11,846,072,320 | 11,327,027,200 |
222
+ </figure>