How to Use GPT-2 in Google Colab
By: Aidan Curley
Tags: Machine Learning, Data Science, Innovation, Programming, Artificial Intelligence

Image from https://openai.com/blog/better-language-models/

For those who do not have access to the GPT-3 API it is important to remember we can still experiment with and use GPT-2!

For everyone who is not familiar with what GPT-2 and GPT-3 are here is a short explanation from OpenAI, “We’ve trained a large-scale unsupervised language model which generates coherent paragraphs of text, achieves state-of-the-art performance on many language modeling benchmarks, and performs rudimentary reading comprehension, machine translation, question answering, and summarization — all without task-specific training.” Pretty awesome right?

So let’s get started with this awesome technology!

Getting Google Colab Setup

If you have used Google Colab before you can skip these next steps. If you have never used Google Colab before here are the necessary steps to get started.

You need to have a Google account. After you have the Google account created go to Google Colab. I would highly recommend upgrading to the paid version which is Google Colab Pro. It is only $9.99/month and you get access to more powerful GPUs, longer run times, and more memory which means more data!

Implementing GPT-2

TL;DR this is the link to the complete notebook: Here

Now that you have your Google account all setup with Google Colab we can move into implementing GPT-2.

1.Create a new notebook within Google Colab. If you are unfamiliar with how to do that click the + New in Google Drive like you would to create a Google Doc. You might have to click more and you will see a dropdown that says “Google Colaboratory.” Click that and it will open a blank notebook.

2. Of course name your notebook.

3. If you have Colab Pro make sure to change your run time type to Hardware Accelerator: GPU and Runtime Shape: High-RAM. This will give you the best performance.

4. Next step is mounting your Google Drive which is basically connecting your Google Drive to the Google Colab notebook. Here is the code to do that.

from google.colab import drive drive.mount('/content/drive')

When you run this cell it will tell you “Go to this URL in a browser:” and click that link. This will open a new tab and make sure to click the proper Google account. Click allow at the bottom. Copy that authorization code and paste it back into where it asks for it in Google Colab.

5. Next we need to create a folder for GPT-2 and clone their repo into there.

# ONLY RUN ONCE %cd /content/drive/My\ Drive/ !mkdir gpt-2 %cd gpt-2/ !git clone https://github.com/openai/gpt-2.git %cd cd gpt-2

It is important to remember to only run this cell block once since you do not want to clone the GPT-2 repo multiple times.

6. After cloning their repo we need to go one folder deeper and here is the code to do that!

%cd /content/drive/My\ Drive/gpt-2/gpt-2

If you are typing this out make sure you add a “\” after “My”. If the “\” is not there it will throw an error.

7. GPT-2 uses a certain version of Tensorflow so we need to update that in Colab as well. Run the code below to change the Tensorflow version in Colab

%tensorflow_version 1.x

8. Now we need to install the necessary requirements provided by GPT-2.

!pip3 install -r requirements.txt

If you are typing this out make sure the “!” is there or it will throw an error. If you run this and get an output that has “WARNING: The following packages were previously imported in this runtime: [idna,requests] You must restart the runtime in order to use newly installed versions.” in it you do not have to restart the runtime you can continue onto the next step.

9. After that is done installing we can now download their model data.

!python3 download_model.py 124M !python3 download_model.py 355M !python3 download_model.py 774M !python3 download_model.py 1558M

These will take some time to download since these models are fairly large.

10. After these models are downloading we can now play around with and experiment with the awesomeness of GPT-2!

This next line will now allow us to input text and GPT-2 will generate text based off of what we input!

!python3 src/interactive_conditional_samples.py --top_k 40

A small box will pop up. You can input any text you’d like in there and see what it generates. If nothing generates the first time when you run it just add the same text in the next small box that says Model prompt >>>.

11. Google Colab will eventually time out so you will have to rerun all of these steps EXCEPT the cloning their repo step. Also, when you go to rerun these steps if you have Colab Pro make sure to change the run time type again.

Good luck I hope you find some interesting insights into what can be generated!

Here is a link to the complete code: Here

A GPT-2 Generated Thank You

“I think a lot of you guys would’ve like to know about this, so please, follow along to this post. Thanks for reading, and stay tuned. I hope you enjoyed this article. Happy business.” - GPT-2

Good enough! Thank you to everyone who made it this far and if you have any questions feel free to reach out to me on LinkedIn and I will respond as soon as possible.