ClipMe: Automated Meme-Clip Generation
By: Rishabh Bansal
Tags: Naturallanguageprocessing, Computer Vision, Deep Learning, Attention, Memes

Approach

A novel architecture designed to generate meme clips, ClipMe comprises of four modules: Image Caption Generation, Meme Template Selection, Meme Generation, and Audio Mapper.

Image Caption Generation

The image caption generator consists of the following modules:

Encoder-Decoder: The encoder takes a colored image as input and converts it into a summarized representation of the original image. We are using 101 layered Residual Network (Kaiming He [2015]) trained on the ImageNet (Tsung-Y Lin [2014]) classification task for decoding an image. The decoder takes encoded images as input and generates a caption word by word. Architectures of the encoder and decoder are shown in Figures 4 and 5.

Figure 4: High-level architecture of the encoder model

Figure 5: Decoder model which generates word by word from the input image representation

Attention: To estimate the importance of a certain part of an image, we need to be aware of the sequence that is generated so far, so that we look at the image and decide what needs describing next. The attention mechanism achieves the same. The attention used in the model is soft, where the weights of the pixels add up to 1. If there are P pixels in our encoded image and αₚ is the weight of that pixel, then at each time step t, αₚ,ₜ follows:

Beam Search: Beam search uses top k candidates to generate sequences from which the sequence having the highest overall score is chosen.

The final architecture of the image captioning module is demonstrated in Figure 6.

Figure 6: Architecture of the image captioning module

Picture Courtesy: https://github.com/sgrvinod/a-PyTorch-Tutorial-to-Image-Captioning

Meme Template Selection

Pre-trained language representation models have been popular for language-related tasks. To map an input caption to a meme template, we fine-tuned models like BERT, XLNet, and Roberta, followed by a linear neural network to get the template with the maximum probability. We also implemented the LSTM model to compare the results as mentioned in Table 1. For the LSTM model, the input image caption is passed through an embedding layer to extract an embedding followed by the LSTM for the classification task.

Meme Generation

Meme Generation module as visualized in Figure 7 consists of the following three sub-modules:

Encoder: The motivation behind the encoder is to provide a meaningful state of the input to the decoder to initiate the meme generation process. To capture the image embeddings, we utilize the ResNet model trained using the ImageNet dataset. We employ an embedding layer consisting of a Linear layer, BatchNorm layer, and a Dropout layer at the end. The last fully-connected layer of the ResNet model is connected to the embedding layer.

Figure 7: Captioning with Meme Template Label

Decoder: The decoder consists of a uni-directional LSTM network. Every LSTM cell reuses the variables in the model. The output from the encoder and image caption is used to create a word embedding. These word embeddings are passed to LSTM. LSTM is run over inputs to predict the next token at each time step. We then utilize the method of the

Inference and Beam Search: Beam search algorithm to create meaningful captions. Meme caption generation is initiated via meme template embedding when for a single cell of the LSTM network. Beam search with a temperature hyper-parameter is used to filter top k scores where k is the beam size. At each time step k outputs are kept in memory and it sequentially computes their ”descendants” resulting in k sentences with the overall highest score.

Audio Mapper

Semantic similarity is the task of determining how similar two sentences are, in terms of meaning. Transformer-based models allow us to achieve near SOTA results in semantic similarity with minimal task-based fine-tuning.

Here, we want to semantically match the generated meme caption and the audio labels from the labeled funny audio dataset to find the most suitable audio file for the meme clip. To accurately find the most semantically similar audio for a given meme caption, we’ll use a pretrained model like BERT to calculate the cosine similarity between the meme and the harvested audio dataset.