AI-Memer: Using Machine Learning to Create Funny Memes | Towards Data Science

In this article, I’ll show you how I built a system called AI-Memer that generates memes using the latest AI models. I start with a high-level description of the system components before getting into the background of memes and details of the components. I’ll then show you how to generate your own memes using the Google Colab, here. After a brief discussion of results and next steps, you can see some sample memes in the appendix. Oh, and I’ll show a newly generated meme at the head of each section 😄.
The main system components are shown in the diagram below.
The user starts by entering a search query to find a background image, like “apple pie”. The system then checks for matching images in Wikimedia Commons [1] and the OpenImages dataset [2]. Both datasets have corresponding text descriptions of the images. I use the CLIP [3] encoders from OpenAI to first perform a semantic search on the text descriptions. A semantic search looks for matching concepts, not just a word search. I then perform a semantic search on the images. The user checks out the top 10 images that match the query and selects their favorite. Either the GPT-3 model from OpenAI [4] or the GPT-Neo model from EleutherAI [5] is used to generate 10 possible captions. The user selects the best caption to create the new meme, which can be downloaded.
The Wiktionary defines the word meme as “any unit of cultural information, such as a practice or idea, that is transmitted verbally or by repeated action from one mind to another in a comparable way to the transmission of genes” [6]. The term originated in Richard Dawkins’ book “The Selfish Gene” [7]. In the age of the Internet, the term meme has been narrowed to mean a piece of content, typically an image with a funny caption, that’s spread online via social media [8].
Dylan Wenzlau created an automatic meme generator using a deep convolutional network [9]. He used 100M public meme captions by users of the Imgflip Meme Generator and trained the system to generate captions based on 48 commonly used background images. You can read about his system here, and run it online here. Here are three examples.
These are pretty good but the system is limited to use only common background images. I was looking for a way to inject a set of new images into the memesphere.
The AI-Memer system creates memes in three steps: finding background images, generating captions, and typesetting the meme captions.
The background images are pulled from two sources, the Wikimedia Commons and the OpenImages dataset. I use OpenAI’s CLIP to perform a semantic search. The CLIP system accomplishes two functions, encoding both text and images into “embeddings”, which are strings of numbers that represent the gist of the original data. The CLIP model was pretrained on 40 million pairs of images with text labels such that the embeddings encoded from the images will be similar to the embeddings encoded from the text labels. For more information about how CLIP works, check out my article, here.
The Wikimedia Commons has over 73 million JPEG files. Most of them are released with permissive rights, like the Creative Commons Attribution license. I use Goldsmith’s Wikipedia search API [10] to find the top 3 pages related to the text query and gather the image descriptions using the CommonsAPI on the Magnus Toolserver [11]. I use the shutil.copyfileobj() function in python to download the image files. There are typically 3 to 10 images on a Wikipedia page so there will be about 9 to 30 images in total coming down.
The OpenImages dataset from Google is comprised of 675,000 photos scraped from Flikr that were all released under the Creative Commons Attribution license. A dataset of image descriptions is available for download [12]. I ran each of the descriptions through OpenAI’s CLIP system and cached the embeddings for quick access. When the user types in a query, I run it through CLIP and compare it to the cached embeddings. I then download the top 20 matching images using the OpenImages download API [13].
For a final filtering pass, I run the images from the 3 Wikipedia pages and the 20 images from the OpenImages through the image encoder and compare the results to the embedding of the text query. I present the top 10 images to the user to choose their favorite.
For example, if you search for “apple pie”, you will be presented with the top 10 images sorted by closest match.
Picture number 8 looks pretty good. It came from a user named W.carter via Wikipedia Commons. The description is “Simple and easy apple pie served with vanilla ice cream, on a gingham tablecloth in Lysekil, Sweden”. Up next we’ll generate some captions for our new meme.
I use two different implementations of GPT to generate the captions. There’s the latest GPT-3 Da Vinci model [4] from OpenAI that does an excellent job, but you have to be enrolled in their beta program to use it. And there’s the open-source GPT-Neo model [5] from EleutherAI. The model is a lot smaller, but it’s free to use.
OpenAI’s GPT-3 Da Vinci is currently the largest AI model for Natural Language Processing [4]. I am using their latest “zero-shot” style of prompting with their new Da Vinci Instruct model. Instead of providing examples of what you are asking the model to do, you can just simply ask it what to do directly.
Here is the prompt that creates a caption for the apple pie picture.
Create a funny caption for a new meme about apple pie. The background picture is Simple and easy apple pie served with vanilla ice cream, on a gingham tablecloth in Lysekil, Sweden.
I pass the prompt into the call to OpenAI along with some additional parameters. Here’s the Python code.
import openairesponse = openai.Completion.create(  engine="davinci-instruct-beta",  prompt=prompt,  max_tokens=64,  temperature=0.7,  top_p=0.5,  frequency_penalty=0.5,  presence_penalty=0.5,  best_of=1)
The max_token parameter indicates how long the response should be. The temperature and top_p parameters are similar in that they indicate the amount of variety in the response. The frequency_penalty and presence_penalty are also similar in that they control how often there are new deviations and new topics in the response. If you want to know what all these parameters do, check out my article from last month, here.
Before I show examples of the output from GPT-3, here is the legal disclaimer that OpenAI suggests that I show, which is all true.
The author generated the following text in part with GPT-3, OpenAI’s large-scale language-generation model. Upon generating draft language, the author reviewed and revised the language to their own liking and takes ultimate responsibility for the content of this publication.
Running the code 10 times will yield the following results, at a total cost of $0.03. Note that I formatted the text to be in uppercase.
 1: THIS IS THE PERFECT WAY TO END A DAY OF APPLE PICKING 2: NO, IT'S NOT THAT EASY 3: I'LL TAKE THE ONE WITH THE VANILLA ICE CREAM, PLEASE 4: APPLE PIE IS THE BEST! 5: THIS APPLE PIE IS SO GOOD, I CAN'T EVEN! 6: YOU'RE NOT THE ONLY ONE WHO LOVES APPLE PIE 7: IF YOU CAN'T FIND THE RECIPE, JUST GOOGLE IT 8: THE PIE IS GOOD, BUT IT'S NOT AS GOOD AS MY MOM'S 9: I'LL HAVE A SLICE OF THAT APPLE PIE, PLEASE10: WE'RE GOING TO NEED A BIGGER PIE
OK, these are pretty good. One thing I learned is that GTP-3 Da Vinci can be funny! For example, caption number 2 seems to refer to the “easy as pie” idiom.
Note that GPT-3, like all AI models trained on a large corpus of text, will reflect societal biases. Occasionally the system will produce text that may be inappropriate or offensive. OpenAI has a feature to label generated text with one of three warning levels: 0 - the text is safe, 1 - this text is sensitive, or 2 - this text is unsafe. My code will show a warning for any of the generated captions that are flagged as sensitive or unsafe.
GPT-Neo is a transformer model created primarily by developers known as sdtblck and leogao2 on GitHub. The project is an implementation of “GPT-2 and GPT-3-style models using the mesh-tensorflow library” [5]. So far, their system is the size of OpenAI’s GPT-3 Ada, their smallest model. But GPT-Neo is available for free. I used the Huggingface Transformers [14] interface to access GPT-Neo from my Python code.
Since GPT-Neo doesn’t have “instruct” versions of their pre-trained models, I had to write a “few-shot” prompt in order to get the system to generate captions for memes using examples. Here’s the prompt I wrote using Disaster Girl and Grumpy Cat memes with example captions.
Create a funny caption for a meme.Theme: disaster girlImage description: A picture of a girl looking at us as her house burns downCaption: There was a spider. It's gone now.Theme: grumpy catImage description: A face of a cat who looks unhappyCaption: I don't like Mondays.Theme: apple pie.Image description: Simple and easy apple pie served with vanilla ice cream, on a gingham tablecloth in Lysekil, Sweden.Caption:
After setting the temperature parameter to 0.7 and the top_p to 1.0, I pass the prompt into GPT-Neo to generate new captions. Here’s the code to generate a caption.
from transformers import pipeline, AutoTokenizergenerator = pipeline('text-generation',  device=0,  model='EleutherAI/gpt-neo-2.7B')results = generator(prompt,  do_sample=True,  min_length=50,  max_length=150,  temperature=0.7,  top_p=1.0,  pad_token_id=gpt_neo_tokenizer.eos_token_id)
Here are the sample results.
 1: I LOVE APPLE PIE 2: I CAN'T. I'M NOT ALLOWED 3: I LOVE THE SIMPLICITY OF AN APPLE PIE 4: APPLE PIE. THE ONLY THING BETTER THAN THIS IS A HOT BATH 5: I'M A PIE. YOU'RE A PIE 6: I LOVE PIE, AND THIS IS A GOOD ONE 7: I LOVE APPLES, BUT I'M NOT VERY GOOD AT BAKING 8: THE PIE IS DELICIOUS, BUT THE ICE CREAM IS NOT 9: I LOVE APPLE PIE. IT'S THE BEST10: THE BEST FOOD IS WHEN YOU CAN TASTE THE DIFFERENCE BETWEEN THE FOOD AND THE TABLECLOTH
Hmmm. These are not as good as the GPT-3 captions. Most of them are quite simple and not very funny. Number 10 is just plain absurd. But number 4 seems to be OK. Let’s use this as our caption.
The final step is to compose the meme by writing the caption into the background image.
Adding the captions to memes is fairly straightforward. Most memes are composed using the Impact font designed by Geoffrey Lee in 1965 [15]. For AI-Memer, I used some code by Emmanuel Pire for positioning and rendering the caption into the background image [16]. I give the user the option to adjust the size of the font and place the caption at the top or bottom of the image.
Here are our two memes. The caption on the left was generated by GPT-3 and the one on the right was generated by GPT-Neo.
You can create your own memes using the Google Colab, here.
With this project, I learned that large-scale language-generation models can create good captions for memes given a description of the image. Although many of the generated captions are straightforward, occasionally they can be very clever and funny. The GPT-3 Da Vinci model, in particular, seems to create clever memes frequently, demonstrating both a command of the language with a seemingly deep understanding of cultural history.
Although the results are pretty good, there is definitely room for improvement. For example, the choices for background images seem somewhat limited, especially for pop culture. This may be due to the fact that I am restricting the search to use only freely licensed photos. I don’t know if a US court has weighed in yet on whether or not the background images in memes can be deemed to be fair use or not, so I’ll leave that question to the lawyers.
The developers behind GPT-Neo at EleutherAI are continuing to build and train bigger language models. Their next model is called GPT-NeoX. They say their “primary goal is to train an equivalent model to the full-sized GPT⁠-⁠3 and make it available to the public under an open licence.” [17]
Don’t forget to check out more generated memes in the appendix below.
All source code for this project is available on GitHub. You can experiment with the code using this Google Colab. I released the source code under the CC BY-SA license.
If you create any memes with AI-Memer and post them online, please mention the project and add a link to this article.
I would like to thank Jennifer Lim and Oliver Strimpel for their help with this project.
[1] Wikimedia Commons (2004-present)
[2] OpenImages (2020)
[3] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning Transferable Visual Models From Natural Language Supervision (2021)
[4] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A.Neelakantan, et al. Language Models Are Few-Shot Learners (2020)
[5] EleutherAI, GPT-Neo (2020)
[6] Wiktionary (2004-present)
[7] R. Dawkins, The Selfish Gene (2016), Oxford University Press
[8] L. K. Börzsei, Makes a Meme Instead (2013), The Selected Works of Linda Börzsei
[9] D. Wenzlau, Meme Text Generation with a Deep Convolutional Network in Keras & Tensorflow (2019), Towards Data Science
[10] Goldsmith, Wikipedia API (2014)
[11] M. Manske, Wikimedia Commons API (2020)
[12] J. Pont-Tuset, J. Uijlings, S. Changpinyo, R. Soricut, and V. Ferrari, Connecting Vision and Language with Localized Narratives (2020) ECCV (Spotlight)
[13] OpenImages Download API (2018)
[14] Huggingface Transformers (2019)
[15] P. McNeil, The Visual History of Type (2017), Laurence King, p. 372–373
[16] E. Pire, Captioning Memes in Python (2017)
[17] EleutherAI, GPT-NeoX (2021)
Here are some more memes generated by AI-Memer.
To get unlimited access to all articles on Medium, become a member for $5/month. Non-members can only read three locked stories each month.
Some rights reserved