GPT-3 Explained. Understanding Transformer-Based... | by Rohan Jagtap | Towards Data Science

In this article, we’ll be discussing the renowned GPT-3 model proposed in the paper “Language Models are Few-Shot Learners” by OpenAI. It is the successor of GPT-2, which has a very similar architecture to that of GPT-3.
If you’re unaware of GPT-2, consider giving my article on GPT-2 a read, as most of GPT-3 is based on it and would help in understanding the model better.
Going back to GPT-2, it is essentially an autoregressive model based on the Transformer architecture (Vaswani et al.). But the novelty of GPT-2 lies in its pre-training approach.
The pre-training leverages multi-task learning at a dataset-level. It basically means that the input tells the model to perform a specific NLP task.
For example, a translation example can be of the format, “translate to french, <english text>, <french text>” in the original document itself. Or a reading comprehension task sample could be of the format, “answer the given question using, <document>, <question>, <answer>.”
— GPT-2 Blog
They call this zero-shot task-transfer or meta-learning, or in-context learning. This way, the model need not be fine-tuned on downstream NLP tasks, which is a step towards the unification of models and general intelligence.
GPT-3 is based on the same principle of in-context learning, but with some improvements in the model and the overall approach. The paper also addresses the issues with this approach and tries to achieve state-of-the-art results. We will see this in the upcoming sections.
As said earlier, GPT-3 is also based on the idea of in-context learning. This is shown in the above figure. Obviously, there is no direct “inner loop” of training on the sequences. For example, a sequence can be:
5 + 8 = 13, 7 + 2 = 9, 1 + 0 = 1, and so on...
And the model will be trained on the autoregressive objective, i.e., given the previous tokens, maximize the probability of the next. So in this way, it can capture patterns from the input directly.
The thought is simple; humans do not train on large datasets for each and every task. Sometimes,
“please tell me if this sentence describes something happy or something sad”
is enough to give us a context of what is expected. Or for someone naive,
“this sounds happy, this sounds sad, how do you think this sounds?”
is sufficient to make them understand the task at a reasonable competency.
Although the concept is promising, previous approaches like GPT-2 have far more inferior results than the state-of-the-art, which is mostly based on the pretraining-finetuning approaches.
Recent progress in Transformer-based models shows that scaling the model higher has substantially improved the fine-tuning results. And it really makes sense to scale the model higher in meta-learning objectives since the model captures many skills and tasks simultaneously and within the same model capacity (Note that in a fine-tuning setting, a new model is trained for each downstream task).
Hence, the authors trained a 175 BILLION parameter model!
It has at least 10x more parameters than the previous biggest model.
The architecture is pretty much the same as GPT-2, just scaled up by a huge factor. It includes custom weights initialization, pre-normalization, and byte-pair encoding. I have covered this in my article on GPT-2. Consider giving it a read if you’re interested.
Apart from this, some ideas are taken from Sparse Transformer, which adds a few modifications to the attention computation to reduce the complexity and support longer sequences. Basically, in full (dense) attention, every token attends every other token in the sequence, which results in O(n2) space, i.e., it scales quadratically with sequence length. To overcome this, Sparse Transformer suggests that every token may attend only a subset of tokens in the sequence, such that:
And,
Next comes making efficient choices for the subsets (A). Further discussion on this model is out of this article’s scope. You can refer to the Sparse Transformer paper if you’re interested.
GPT-3 alternates between dense and sparse attention patterns. However, it is not clear how exactly this alternating is done, but presumably, it’s either between layers or between residual blocks.
Moreover, the authors have trained GPT-3 in 8 different sizes to study the dependence of model performance on model size. Following are the configurations of these 8 models:
Additionally, the model is shared in the depth as well as in the width dimension across GPUs to minimize data transfer between the nodes.
This is the part we are interested in the most. GPT-3 is evaluated on more than two dozen datasets. For each of these tasks, it is evaluated for 3 settings — zero-shot, one-shot, and few-shot. We will see what these are, and we’ll compare them with the fine-tuning approach in this section.
In this approach, we first pre-train a model (mostly on an autoregressive or a cloze objective), which helps the model capture general patterns in the language; then, we retrain it separately for specific downstream NLP tasks. In the figure above, the model is fine-tuned for a translation task.
The main disadvantage of this approach is the need for large datasets for individual tasks. Plus, fine-tuning on one dataset may not provide good generalization over others for the same task. Although fine-tuning results are strong, it proves to be an unfair comparison with human performance.
In this setting, after pre-training the model (with in-context learning), we directly provide the model with the input for the task without any special training for that task. Just tell the model “what to do” along with the input. This is the most challenging setting, and in some cases, it may be “unfairly hard.” For example, for inputs like “make a table of world records for the 200m dash,” the output format is ambiguous.
Nevertheless, for at least some settings zero-shot is closest to how humans perform tasks.
— GPT-3 Paper
For example, in the above translation example, the input is enough clarification for a human to understand what is expected. Also, this is pretty much what GPT-2 offers.
In this setting, we provide 1) “what to do,” 2) exactly one example (one-shot) of the task, and then 3) the input. The example is meant for conditioning only, i.e., it is meant to provide some context on the task. You can see this as providing some kind of analogy to the model. We saw the “unfairly difficult” example in the zero-shot setting. In tasks like those, the input becomes more plausible to answer if at least one demonstration of the task is provided.
Finally, in the few-shot setting, the input includes 1) “what to do,” 2) a few examples (few-shot), and then 3) the input. This setting provides better conditioning over the input for the model to predict the output. Typically, K examples are added to the input where K is between 10 and 100. The model supports a context length of 2048, so approximately, at max, K = 100 examples can fit in the context window. The few-shot setting greatly reduces the amount of data required than fine-tuning. But there is no denying that at least some amount of task-specific data is required. The main disadvantage of this setting is that so far, the results obtained in this setting were way worse than the state-of-the-art. However, GPT-3 has managed to achieve results very similar to the state-of-the-art on many tasks.
Note that, unlike fine-tuning, the model is never trained on the examples in all these settings. The examples are just meant for conditioning, i.e., to provide some context on the input. They are used directly in the input at inference.
Hence, Language Models are Few-Shot Learners!
Like most language models, GPT-3, too, is trained on the CommonCrawl dataset. 41 shards of monthly CommonCrawl covering 2016 to 2019 with 45TB of data is collected. However, unfiltered or lightly filtered data from CommonCrawl tend to have lower quality than filtered datasets. So the authors have taken 3 steps to filter it:
The authors took some high-quality corpora and based on similarity with these corpora, they have filtered CommonCrawl.Fuzzy de-duplication is used to remove redundancies within and across datasets. This also ensures integrity, i.e., the model doesn’t train on validation data.And finally, known high-quality reference data are added to increase the diversity of the dataset.
The authors took some high-quality corpora and based on similarity with these corpora, they have filtered CommonCrawl.
Fuzzy de-duplication is used to remove redundancies within and across datasets. This also ensures integrity, i.e., the model doesn’t train on validation data.
And finally, known high-quality reference data are added to increase the diversity of the dataset.
Leaves us with 570GB data ~400B byte-pair tokens.
A major concern about large language models trained on a huge amount of Internet data is that there are chances that the model may have already seen the test data for downstream tasks during pre-training. To mitigate this data contamination, the authors have made attempts to find and remove such overlapping data.
However, a bug in filtering caused some data to overlap in the pre-training, and training the model again was not feasible due to the training cost.
Despite the celebrated pros, the GPT-3 does have a few limitations:
Although the quality of text generation is great, there are still times when the model generates repetitive text. While generating long documents, it may lose coherence, contradict itself, and sometimes even completely lose the context. The authors have also noticed that GPT-3 has trouble with “common sense physics” questions; specifically, with questions like “If I put cheese into the fridge, will it melt?”.
We know that for some tasks like ‘fill in the blanks,’ bidirectional models have outperformed autoregressive models. So, it can be said that a large autoencoding model can outperform GPT-3 at such tasks. This also opens a research opportunity for training bidirectional models at scale or training them on zero-shot or few-shot settings.
Since the model is trained generically, it does not have any task-specific bias. The model weighs all the tokens equally. Therefore, some real-world applications like virtual assistants may benefit from the fine-tuning approaches as they provide a more goal-oriented outcome rather than just predictions.
A model of the scale of GPT-3 would obviously struggle at inference. It is both expensive and inconvenient to perform inference on. This raises questions about the practical applicability of the model. This also opens research branches for the development of distilled versions of the model.
This one is a limitation for most of the deep learning models — the decisions taken by a model are based on the data it is trained on. So ultimately, there is a strong chance for the model to have stereotyped and prejudiced opinions. This is a topic of concern from a societal perspective. For details on this, you can refer to the ‘Broader Impacts’ Section in the GPT-3 Paper.
In this section, I’ll paste a few pieces of text generated by GPT-3:
A generated news article that humans had the greatest difficulty distinguishing from a human-written article:
Title: United Methodists Agree to Historic SplitSubtitle: Those who oppose gay marriage will form their own denominationArticle: After two days of intense debate, the United Methodist Church has agreed to a historic split - one that is expected to end in the creation of a new denomination, one that will be "theologically and socially conservative," according to The Washington Post. The majority of delegates attending the church's annual General Conference in May voted to strengthen a ban on the ordination of LGBTQ clergy and to write new rules that will "discipline" clergy who officiate at same-sex weddings. But those who opposed these measures have a new plan: They say they will form a separate denomination by 2020, calling their church the Christian Methodist denomination.The Post notes that the denomination, which claims 12.5 million members, was in the early 20th century the "largest Protestant denomination in the U.S.,"but that it has been shrinking in recent decades. The new split will be the second in the church's history. The first occurred in 1968, when roughly10 percent of the denomination left to form the Evangelical United Brethren Church. The Post notes that the proposed split "comes at a critical time for the church, which has been losing members for years," which has been"pushed toward the brink of a schism over the role of LGBTQ people in the church." Gay marriage is not the only issue that has divided the church. In 2016, the denomination was split over ordination of transgender clergy, with the North Pacific regional conference voting to ban them from serving as clergy, and the South Pacific regional conference voting to allow them.
Text completion examples:
To do a "farduddle" means to jump up and down really fast. An example of a sentence that uses the word farduddle is:One day when I was playing tag with my little sister, she got really excited and she started doing these crazy farduddles.A "yalubalu" is a type of vegetable that looks like a big pumpkin. An example of a sentence that uses the word yalubalu is:I was on a trip to Africa and I tried this yalubalu vegetable that was grown in a garden there. It was delicious.
Generated Poem:
In this article, we’ve discussed the famous GPT-3 model and saw how it works. The model has produced astounding results that have even fooled humans in their own game. However, there is still a lot of room for improvement, which is a motivation for further research.