Is bigger also smarter? — Open AI releases GPT-3 language model
By: Andreas Stöckl
Tags: Deep Learning, Machine Learning, Data Science, NLP, Language Model

Is bigger also smarter? — Open AI releases GPT-3 language model

The race for larger language models is entering the next round.

Image: www.pexels.com

Progress in NLP applications is driven by larger language models consisting of neural networks using the Transformer Architecture. On the occasion of the recently published results of the currently largest model — GPT-3 of Open AI, I would like to take a closer look at these advances.

Are the models more than just huge “lookup tables” with intelligent interpolation methods?

On May 28, 2020, a paper (https://arxiv.org/abs/2005.14165) by OpenAI researchers was published on ArXiv about GPT-3, a language model that is capable of achieving good results in a number of benchmark language processing tasks ranging from language translation and news article writing to question answering. The special thing about it is that these results are achieved without fine-tuning for the benchmark under consideration, but are achieved by the language model without any further information (“zero-shot”) or with little additional information (“one-shot” or “few-shot”).

GPT-3 has no less than 175 billion parameters. For comparison, the largest version of GPT-2 had 1.5 billion parameters, and the world’s largest transformer-based language model — introduced by Microsoft earlier this month — has 17 billion parameters.

When Open AI released the previous generation GPT-2 a year ago with great media echo, this was also due to the announcement not to release the largest model, because it could generate harmful things like fake news. Meanwhile, the generation of news from a given context with such models is nothing special anymore. However, the generated articles are not just pieced together from parts of articles from the training data, but are really new, as experiments have shown.

At first glance, the contributions are often indistinguishable from real news. In the current GPT-3 paper, one result is that generated news articles can hardly be distinguished from real news by people. In one experiment only 52% were correctly recognized, which is close to 50% that would be achieved by dicing. This could also be due to the fact that some real news items are so absurd that the dividing line can no longer be drawn sharply.

As my own experiments with stories about Austria have shown,

the articles are largely syntactically correct and read pleasingly, and the facts sound quite logical, but are often recognizable as incorrect when one is familiar with the facts.

It seems as if the model filters matching articles from a kind of “memory” through the given context, and then “interpolates” them into a new story. The memory is formed by the high number of parameters that are optimized when training with text data. GPT-3 was trained on the CommonCrawl dataset of almost one trillion words collected between 2016 and 2019, as well as on datasets related to web texts, books, and Wikipedia.

This reminds me of the saying that DeepLearning is just glorified “curve fitting”, which I discussed in an earlier article:

But it is not really “understanding” texts or even drawing conclusions from texts. As the results in the new paper also show, GPT-3 models have a hard time, especially in tests that require real conclusions.

An interesting experiment in the paper is also the attempt to calculate examples given as text, for example:

“ What is 48 plus 76? ”

to be calculated by the model. It is shown here that the largest model (175 billion parameters) allows simple addition and subtraction of two or three numbers, but not yet longer calculations. And that again without special fine-tuning for this task!

Chart from the paper https://arxiv.org/abs/2005.14165

As if the size of the “lookup table” was large enough for the shorter calculations, but not for the longer ones. Where does the data for these “tables “come from?

In the texts from the internet, which were used for the training of GPT-3, there is enough material for this. An Internet search for the calculations provides tables like the following one:

Screenshot from https://www.dollartimes.com/calculate/percentage/

The question arises for me if further scaling up of models and data will lead to further substantial improvements for NLP applications, or if it is time for new ideas and approaches?