Text Generation With GPT-2 in Python | Towards Data Science

Language generation is one of those natural language tasks that can really produce an incredible feeling of awe at how far the fields of machine learning and artificial intelligence have come.
GPT-1, 2, and 3 are OpenAI’s top language models — well known for their ability to produce incredibly natural, coherent, and genuinely interesting language.
In this article, we will take a small snippet of text and learn how to feed that into a pre-trained GPT-2 model using PyTorch and Transformers to produce high-quality language generation in just eight lines of code. We cover:
> PyTorch and Transformers  - Data> Building the Model  - Initialization  - Tokenization  - Generation  - Decoding> Results
If you prefer video, I’ve covered everything in this short video tutorial here:
We need both PyTorch and Transformers libraries installed to build our text generation model.
The setup instructions for PyTorch vary depending on your system, CUDA version (if any), and Python release. Fortunately, PyTorch has made a very easy to use guide here.
Next up is HuggingFace’s Transformers library. All this takes is a simple pip install transformers.
We need to feed into our model some text that our model will read and then generate more text from. I’ve taken the snippet above from Winston Churchill’s Wikipedia page — but feel free to use anything you like!
Once we have both frameworks installed, we can import what we need and initialize our tokenizer and model like so:
The tokenizer is used to translate between human-readable text and numeric indices. These indices are then mapped to word embeddings (numerical representations of words) by an embedding layer within the model.
All we need to do to tokenizer our input text is call the tokenizer.encode method like so:
Because we are using PyTorch, we add return_tensor='pt', if using TensorFlow, we would use return_tensor='tf'.
Now that we have our tokenization input text, we can begin generating text with GPT-2! All we do is call the model.generate method:
Here we set the maximum number of tokens to generate as 200. We also add do_sample=True to stop the model from just picking the most likely word at every step, which ends up looking like this:
He began his premiership by forming a five-man war cabinet which included Chamerlain as Lord President of the Council, Labour leader Clement Attlee as Lord Privy Seal (later as Deputy Prime Minister), Halifax as Foreign Secretary and Labour's Arthur Greenwood as a minister without portfolio. In practice, he was a very conservative figure, but he was also a very conservative figure in the sense that he was a very conservative figure in the sense that he was a very conservative figure in the sense that he was a very conservative figure in the sense that he was a very conservative figure in the sense that he was a very conservative figure in the sense that he was a very conservative figure in the sense that he was a very conservative figure
The top_k and temperature arguments can be used to modify our outputs' coherence/randomness — we will cover these at the end.
Our generate step outputs an array of tokens rather than words. To convert these tokens into words, we need to .decode them. This is easy to do:
All we need to add is skip_special_tokens=True to avoid decoding special tokens that are used by the model, such as the end of sequence token 
.
At this point, all we need to do is print our output!
We can get some great results with very little code. Here are a few examples that should give you a better understanding of the impact of each argument in the .generate method.
outputs = model.generate(    inputs, max_length=200, do_sample=True)tokenizer.decode(outputs[0], skip_special_tokens=True)[Out]: "He began his premiership by forming a five-man war cabinet which included Chamerlain as Lord President of the Council, Labour leader Clement Attlee as Lord Privy Seal (later as Deputy Prime Minister), Halifax as Foreign Secretary and Labour's Arthur Greenwood as a minister without portfolio. In practice, these cabinet officers were part of the National Security Council, then headed by Lord Chamberlain's secretary Arthur Hargreaves. A number of them became key cabinet secretaries, including Lord Hargreaves in 1948, Lord Butler as Justice Minister in 1949 and Lord Codds as justice minister until his death in 1975. After being replaced by Lord Hargreaves in 1955, there was speculation that the next general election would see Chamberlain and Howe try and avoid a hard line policy..."
We can add more randomness with temperature — the default value is 1, a high value like 5 will produce a pretty nonsensical output:
outputs = model.generate(    inputs, max_length=200, do_sample=True, temperature=5)tokenizer.decode(outputs[0], skip_special_tokens=True)[Out]: "He began his premiership by forming a five-man war cabinet which included Chamerlain as Lord President of the Council, Labour leader Clement Attlee as Lord Privy Seal (later as Deputy Prime Minister), Halifax as Foreign Secretary and Labour's Arthur Greenwood as a minister without portfolio. In practice, his foreign secretaries generally assumed other duties during cabinet so his job fell less and smaller - sometimes twice his overall stature so long a day seemed manageable after he became Chief Arctic Advisor: Mr Wilson led one reshover where we've also done another three, despite taking responsibility over appointments including Prime (for both) his time here since 1901)[31],[38-4]. (These last had fewer staff as many than he is responsible..."
Turning the temperature down below 1 will produce more linear but less creative outputs.
We can also add the top_k parameter — which limits the sample tokens to a given number of the most probable tokens. This results in text that tends to stick to the same topic (or set of words) for a longer period of time.
outputs = model.generate(    inputs, max_length=200, do_sample=True, top_k=50)tokenizer.decode(outputs[0], skip_special_tokens=True)[Out]: "He began his premiership by forming a five-man war cabinet which included Chamerlain as Lord President of the Council, Labour leader Clement Attlee as Lord Privy Seal (later as Deputy Prime Minister), Halifax as Foreign Secretary and Labour's Arthur Greenwood as a minister without portfolio. In practice, however, the two men had to choose between their public business and politcal career, to the exclusion of all the other political options. To the shock of many, Chamberlain, who was not just a keen public servant but a man of his word, became the first Australian prime minister to be sacked by a Labour minister in a five-year term..."
That is all for this tutorial on language generation with GPT-2 in Python. We have put together a model that can produce reasonable, believable, and interesting text in hardly any time at all.
In just eight lines of code, we have:
Imported all the frameworks we need
Initialized a GPT-2 tokenizer and model
Defined our input text
Tokenized it
Generated new text from our original input
Decoded the generated outputs back into human-readable text
It really is incredible how easy this can be when using the PyTorch and Transformers frameworks.
I hope you enjoyed this article! If you have any questions, let me know via Twitter or in the comments below. If you’d like more content like this, I post on YouTube too.
Thanks for reading!
🤖 NLP With Transformers Course