GPT

Generative Pre-trained Transformer, a left-to-right transformer-based text generation model developed by OpenAI, succeeded by GPT-2 and GPT-3https://en.wikipedia.org/wiki/GPT
The original paper on generative pre-training (GPT) of a language model was written by Alec Radford and colleagues, and published in preprint on OpenAI's website on June 11, 2018.https://en.wikipedia.org/wiki/OpenAI#GPT{ "Improving Language Understanding by Generative Pre-Training" (PDF). Retrieved June 9, 2020. } r. https://en.wikipedia.org/wiki/OpenAI#cite_note-50
GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset{ We created a new dataset which emphasizes diversity of content, by scraping content from the Internet. In order to preserve document quality, we used only pages which have been curated/filtered by humans—specifically, we used outbound links from Reddit which received at least 3 karma. This can be thought of as a heuristic indicator for whether other users found the link interesting (whether educational or funny), leading to higher data quality than other similar datasets, such as CommonCrawl. ↩︎ } Better Language Models and Their Implications (openai.com). https://openai.com/blog/better-language-models/ { February 14, 2019. 24 minute read }Better Language Models and Their Implications (openai.com). https://openai.com/blog/better-language-models/
Our model, called GPT-2 (a successor to GPT), was trained simply to predict the next word in 40GB of Internet text. Due to our concerns about malicious applications of the technology, we are not releasing the trained model.Ibid.
r
G; P; T
Media