stellaathena commited on
Commit
40fb054
1 Parent(s): f5e76ba

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +56 -13
README.md CHANGED
@@ -1,29 +1,35 @@
1
  ---
2
- language: en
 
 
 
 
 
 
 
3
  datasets:
4
- - The Pile
5
  ---
6
 
7
- # GPT Neo 2.7B
8
 
9
- The GPT Neo model is pretrained on the [The Pile](https://github.com/EleutherAI/the-pile) dataset: an 825GiB English text corpus targeted at training large-scale language models.
10
- It is an open source replication of OpenAI's GPT-3 model and is released in several checkpoints: the 1.3B and 2.7B variants.
11
 
12
- It was released on EleutherAI's [GitHub page](https://github.com/EleutherAI/gpt-neo) the 21st of March 2021.
13
 
14
- ## Model Description
15
 
16
- GPT Neo is a transformers model pretrained on a very large corpus of English data in a self-supervised fashion. This means it was pretrained on the raw texts only, with no humans labelling them in any way (which is why it can use lots of publicly available data) with an automatic process to generate inputs and labels from those texts. More precisely, it was trained to guess the next word in sentences.
17
 
18
- More precisely, inputs are sequences of continuous text of a certain length and the targets are the same sequence, shifted one token (word or piece of word) to the right. The model uses internally a mask-mechanism to make sure the predictions for the token i only uses the inputs from 1 to i but not the future tokens.
19
 
20
- This way, the model learns an inner representation of the English language that can then be used to extract features useful for downstream tasks. The model is best at what it was pretrained for however, which is generating texts from a prompt.
21
 
22
- It obtains a 5.646 perplexity on The Pile, and a 11.39 perplexity on Wikitext
23
 
24
- It uses a mix of global and local attention across its layers. It was trained for 400000 steps.
25
 
26
- ## How to use
27
 
28
  You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:
29
 
@@ -34,3 +40,40 @@ You can use this model directly with a pipeline for text generation. This exampl
34
 
35
  [{'generated_text': 'EleutherAI has made a commitment to create new software packages for each of its major clients and has'}]
36
  ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
+ language:
3
+ - en
4
+ tags:
5
+ - text generation
6
+ - pytorch
7
+ - the Pile
8
+ - causal-lm
9
+ license: apache-2.0
10
  datasets:
11
+ - the Pile
12
  ---
13
 
14
+ # GPT-Neo 2.7B
15
 
16
+ ## Model Description
 
17
 
18
+ GPT-Neo 2.7B is a transformer model designed using EleutherAI's replication of the GPT-3 architecture. GPT-Neo refers to the class of models, while 2.7B represents the number of parameters of this particular pre-trained model. It is the same size as OpenAI's "Ada" model.
19
 
20
+ ## Training data
21
 
22
+ GPT-Neo 2.7B was trained on the Pile, a large scale curated dataset created by EleutherAI for the purpose of training this model.
23
 
24
+ ## Training procedure
25
 
26
+ This model was trained for 400,000 steps on the Pile. It was trained as a masked autoregressive language model, using cross-entropy loss.
27
 
28
+ ## Intended Use and Limitations
29
 
30
+ This way, the model learns an inner representation of the English language that can then be used to extract features useful for downstream tasks. The model is best at what it was pretrained for however, which is generating texts from a prompt.
31
 
32
+ ### How to use
33
 
34
  You can use this model directly with a pipeline for text generation. This example generates a different sequence each time it's run:
35
 
40
 
41
  [{'generated_text': 'EleutherAI has made a commitment to create new software packages for each of its major clients and has'}]
42
  ```
43
+
44
+ ### Limitations and Biases
45
+
46
+ GPT-Neo was trained as an autoregressive language model. This means that its core functionality is taking a string of text and predicting the next token. While language models are widely used for tasks other than this, there are a lot of unknowns with this work.
47
+
48
+ GPT-Neo was trained on the Pile, a dataset known to contain profanity, lewd, and otherwise abrasive language. Depending on your usecase GPT-Neo may produce socially unacceptable text. See Sections 5 and 6 of the Pile paper for a more detailed analysis of the biases in the Pile.
49
+
50
+ As with all language models, it is hard to predict in advance how GPT-Neo will respond to particular prompts and offensive content may occur without warning. We recommend having a human curate or filter the outputs before releasing them, both to censor undesirable content and to improve the quality of the results.
51
+ ## Eval results
52
+
53
+ ### Language Modeling Baselines
54
+
55
+ EleutherAI is currently in the process of carrying out further evaluations of GPT-Neo. The following table should be considered a work-in-progress. If you would like to contribute evaluations you have done, please reach out on our Discord.
56
+
57
+ | Model and Size | Pile BPB | Pile PPL | Wikitext PPL. |
58
+ | ---------------- | ------------- | ------------- | -------------- |
59
+ | GPT-Neo 1.3B | 0.7527 | 6.159 | 13.10 |
60
+ | GPT-3 1.3B | ------ | ----- | ----- |
61
+ | GPT-2 1.5B | 1.0468 | ----- | 17.48 |
62
+ | **GPT-Neo 2.7B** | **0.7165** | **5.646** | **11.39** |
63
+ | GPT-3 Ada 2.7B | 0.9631 | ----- | ----- |
64
+ | GPT-3 175B | 0.7177 | ----- | ----- |
65
+
66
+ All GPT-2 and GPT-3 scores are from their respective papers, except for the Pile test results which are from the Pile paper.
67
+
68
+ ### Down-Stream Applications
69
+
70
+ ### BibTeX entry and citation info
71
+
72
+ ```bibtex
73
+ @article{gao2020pile,
74
+ title={The Pile: An 800GB Dataset of Diverse Text for Language Modeling},
75
+ author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and others},
76
+ journal={arXiv preprint arXiv:2101.00027},
77
+ year={2020}
78
+ }
79
+ ```