Multimodal deep learning to predict movie genres | by Dhruv Verma | Towards Data Science

“Batman raises the stakes in his war on crime. With the help of Lt. Jim Gordon and District Attorney Harvey Dent, Batman sets out to dismantle the remaining criminal organizations that plague the streets. The partnership proves to be effective, but they soon find themselves prey to a reign of chaos unleashed by a rising criminal mastermind known to the terrified citizens of Gotham as the Joker.” -TMDB
‘The Dark Knight’ — one of my favourite movies. TMDB classifies this movie into four genres — Drama, Action, Crime, Thriller. A multimodal deep learning model I have trained classifies this movie as — Action, Drama, Thriller. Let’s take a look at how this model manages to do this.
Modality is a particular way of doing or experiencing something. We live our daily lives in a multimodal environment. We see things, hear sounds, smell odours and feel textures. Analogous to this, multimodal deep learning involves multiple modalities used together to predict some output. In this project, I concatenated the features extracted from images and text sequences using a Convolutional Neural Network (CNN) and a Long Short-Term Memory (LSTM) network, respectively. These features were used to try and predict movie genres.
For this project, I have used Kaggle’s The Movies Dataset. It consists of over 40,000 movies with overviews, the poster URL link on TMDB and genres taken from the TMDB website. After splitting the dataset, the training set consists of 26864 examples; the test set includes of 7463 samples, and the validation set consists of 2986 cases.
Before preprocessing the data, it was essential to understand how the class distribution of each genre was. Some genres such as “Aniplex”, “BROSTA TV” seem to occur only once in the entire dataset. Hence, I removed those. In addition to this, some genres like “TV Movie” and “Foreign” are not easily perceived by posters and movie summaries. Hence, those were removed as well. The snippet below shows a list of genres which were removed from the dataset.
invalid_genres = ['Aniplex', 'BROSTA TV', 'Carousel Productions', 'GoHands', 'Mardock Scramble Production Committee', 'Odyssey Media', 'Pulser Productions', 'Rogue State', 'Sentai Filmworks', 'Telescene Film Group Productions', 'The Cartel', 'Vision View Entertainment', 'TV Movie', 'Foreign']
I also scanned the dataset for rows which did not have a genre specified anymore. Over 2471 rows did not have any values in the genre_list column. So these rows were dropped.
In addition to this, it was essential to see how balanced the dataset was in terms of genre labels. As clear from the graph below, there is a class imbalance in the dataset.
Data cleaning and preprocessing allow us to remove irrelevant information from the data such as URLs. These familiar words occur very frequently without adding much semantic meaning such as articles in the English language. The text processing involved the following steps:
Removing of all punctuations from the text.Converting the text to lower case, removing non-alphabetic inputs and all NLTK stopwords.Word lemmatization using NLTK. The goal of lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. For example, ‘Finds’ can be replaced with just ‘Find’.
Removing of all punctuations from the text.
Converting the text to lower case, removing non-alphabetic inputs and all NLTK stopwords.
Word lemmatization using NLTK. The goal of lemmatization is to reduce inflectional forms and sometimes derivationally related forms of a word to a common base form. For example, ‘Finds’ can be replaced with just ‘Find’.
The cleaned text is tokenized using the Keras Tokenizer API. This tokenizer is used to convert all texts to sequences to be input into the LSTM network. All sequences are made of the same length by padding them with zeros. This is done to ensure that all inputs have the same dimensions.
One of the inputs to the model is the movie poster. Hence, I had to download a single poster corresponding to each movie in the dataset. The original dataset contains a URL to the posters stored in TMDB as well. However, the majority of links seem to obsolete, so I had to improvise and make use of the TMDB API. I used python’s tmdbv3api, which wraps around the TMDB API. Using the movie IDs from the original dataset, I found a single poster corresponding to a movie. A point to be noted is that some movies may not have posters available on TMDB.
The ‘Movie’ object allows us to search for movies based on the TMDB ID. The posters are downloaded with a simple GET request and saved to my Google Drive. The path corresponding to a movie poster is saved in a list which is added to the dataset as a column.
In case there are any errors in downloading the posters or the poster_path for a movie does not exist, that movie is dropped from the dataset.
Here is a glimpse of how the final dataset looks.
The final preprocessing step is to resize the posters and append them to a NumPy array. This NumPy array is saved to make it easier to access the posters in a valid input format. This process takes a lot of time, and the output NumPy array is around 2–3 GB. A considerable amount of RAM is needed to save this array. I would recommend using Google colab for this entire process. The code snippet for converting an image to a NumPy array is below. In case any posters are not found at the specified path in the dataset, the respective rows are dropped.
So I have finally preprocessed the text inputs and image inputs. Now, on to training the model.
Each movie is associated with a certain number of genres in the dataset. However, this needs to be made into a one-hot encoded vector where each one would correspond to a genre label. To do this, I used the MultilabelBinarizer class of the scikit-learn library. It results in a sparse matrix where each row vector is one-hot encoded. This means that a row vector will have a 1 in indexes associated with the label it has been classified as and 0 in all other places. Hence, this is a multilabel classification problem. As seen below, there are a total of 18 classes present in the dataset.
The model comprises of two networks — a stacked LSTM and a CNN. The text sequences are input to an embedding layer. I used Google’s pre-trained Word2Vec embedding. However, to fine-tune the model, I allowed the embedding layer to be trainable. The embedding layer is followed by two stacked LSTMs whose outputs are fed into a fully connected layer.
The images are input to a CNN. The CNN consists of consecutive 2D convolutional and max-pooling layers. The output of the last max-pooling layer is flattened and fed into a fully connected layer.
The fully connected layers of the LSTM and CNN then concatenated and fed forward till the final fully connected layer. The last layer has an output size of 18, each correlating to one genre.
DataLoader object was initialized with a batch size of 64, which passes inputs to the model. The TensorDataset object is used to wrap the dataset inputs and labels. Each TensorDataset is then provided to the DataLoader object. This will allow us to send batches of the dataset to the model for training, validation and testing.
The loss I used was binary cross-entropy along with the Adam optimizer. Gradient clipping was also used to prevent exploding gradients. The model was trained for 20 epochs. During the training, I computed the accuracy and loss for every epoch as an average of the accuracy and loss for all batches.
The final epoch had the following accuracy and loss:
Epoch 20: train_loss: 0.1340 train_acc: 0.9200 | val_loss: 0.3431 val_acc: 0.8726
After training and saving the model, I evaluated the accuracy and loss on the test set:
acc: 0.8737 loss: 0.3406
The accuracy and loss seem to show that the model is performing reasonably well. However, some better evaluation metrics would be precision, recall, F1 score and the Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) score due to a class imbalance in the dataset. The output consisting of un-thresholded probabilities of a movie belonging to a genre is converted to a sparse matrix with one-hot encoded row vectors to compute precision, recall and F1 score. This was done using an arbitrary threshold value of 0.5. This value can be further improved by using Matthew’s correlation coefficient. It can also be enhanced by plotting the ROC curve for each class.
Precision on test set: 0.5732Recall on test set: 0.5210F1 score on test set: 0.5080
The ROC AUC score was calculated using the non-thresholded outputs of the model.
roc auc macro score: 0.8021
Another evaluation metric used was the Hamming Loss. The Hamming loss gives a fraction of wrong labels to the total numbers of labels.
Hamming loss on the test set: 0.1078
As evident from the evaluation metrics above, the model seems to be performing well on the test set. The precision, recall and f1-score can probably be boosted further by taking an optimized threshold value for each label rather than an arbitrary value of 0.5. Another way of improving the model would be to evaluate the CNN and LSTM individually to judge the scope of improvement for both.
This project explores a multimodal deep learning approach to tackle a multilabel classification problem of predicting movie genres from movie posters and overviews.The model accuracy and loss on the test set were good. However, there are better-suited evaluation metrics for this problem — precision, recall, f1 score and the ROC AUC score.Such a model might be able to distinguish between the features of a comedy movie’s poster and overview from a horror movie’s poster and summary.
This project explores a multimodal deep learning approach to tackle a multilabel classification problem of predicting movie genres from movie posters and overviews.
The model accuracy and loss on the test set were good. However, there are better-suited evaluation metrics for this problem — precision, recall, f1 score and the ROC AUC score.
Such a model might be able to distinguish between the features of a comedy movie’s poster and overview from a horror movie’s poster and summary.
Thanks for reading this article! The entire code for this project can is at https://github.com/dh1105/Multi-modal-movie-genre-prediction.