Looking Back on GPT-2
By: Hilary Lawlor
Tags: Gpt 2, Naturallanguageprocessing

Photo by Cookie the Pom on Unsplash

Now that the natural language processing community is in the honeymoon phase of a new romance with OpenAI’s newest Generative Pre-Trained Transformer — aka GPT-3 — it’s time to look back on our previous love affair with… OpenAI’s GPT-2. Yes, without a doubt, OpenAI is on the forefront of the natural language processing world, and GPT-3, like GPT-2 before it, has captured our hearts. So, let’s look back on some amazing GPT-2 projects as we move forward into the NLP future.

One of my favorite projects was done by Janelle Shane, author of You Look Like a Thing and I Love You: How Artificial Intelligence Works and Why It’s Making the World a Weirder Place. Years ago, Janelle founded the blog AI Weirdness, on which she would talk about her amazing AI projects (one of which, the neural-network-produced paint names, is my all-time favorite for “Stummy Beige” alone), and when GPT-2 became available, she did one project that captured my candy heart: candy heart messages. Among the best, in my humble opinion:

Image: AI Weirdness, Janelle Shane

“AM GOOD”

“LOVE 2000 HOGS YEA”

“CHERT FACE”

Truly, GPT-2 knows how to talk to a lover.

Another excellent project was a set of GPT-2-generated poems created by Gwern Branwen in 2019. My favorite line was as follows: “The mother of the gods, that day, With mortal feet and sweet voice speaks, And smiles, and speaks to men: “My Sweet, I shall not weary of thy pain.” Branwen trained GPT-2 on a 117 MB “poetry corpus” scraped from Project Gutenberg, to excellent results. As always, the more data you can feed to GPT-2 while you fine-tune it, the better it will behave.

Now is, of course, when I plug my OWN GPT-2 project — still in the works, because even though GPT-2 is no longer “cutting edge,” it is still awesome and GPT-3 is in beta and I can’t get an API key. Anyway, I’m generating classic ghost stories! I used Max Woolf’s gpt-2-simple wrapper and trained GPT-2 on a bunch of classic ghost stories, and thus far, the results have been truly beautiful. Please, allow me:

“You see, as John Money once told his friend, ‘Everything is made of tin.’ That is why in every word he said, he meant to say, ‘Everything is made of tin.’”

Thank you for your insight, John Money.

I call this one When GPT-2 Got Super Into Sparkling Wine In Particular:

“The journey home was now over, and I felt sure that my servant was returning with two bottles of bitter wine and a jug of sparkling water. He had brought with him several curious bottles of wine — the standing-room-only bottles — that had been lost under the deluge two weeks ago in the torrent of molasses that had dumped them. The first specimen I saw of the kind produced in Germany was a bottle of sparkling wine (that has not been mentioned in this connection anguishing from the bottles), a bottle of silver, and a glass of fine sparkling wine. As the feeling for the valuable wine returned I felt a change in my companion’s attitude; his eagerness was much illumined by the unaccountable sweetness the water in the bottles. Even then it was surprising to see such a beautiful animal among the spoils of valor among people of exceptional learning and talent.”

People of exceptional learning and talent love sparkling wine!

First of all: did I write this blog post only to plug my own GPT-2 project? No, but it’s a convenient place to do so. Look, the bottom line is, GPT-2 is still awesome — but more importantly it is currently publicly available! It’s not on the cutting edge of NLP anymore, but it’s still better for large-format content generation than nearly any other model. Our love affair with GPT-2 doesn’t have to end yet! Bring on the sparkling wine, U HACK.