Title: GATED MULTIMODAL UNITS FOR INFORMATION FU

Abstract: This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.

1. INTRODUCTION:

Representation learning methods have received a lot of attention by researchers and practitioners because of its successful application to complex problems in areas such as computer vision, speech recognition and text processing (LeCun et al., 2015) . Most of these efforts have concentrated on data involving one type of information (images, text, speech, etc.) , despite data being naturally multimodal. Multimodality refers to the fact that the same real-world concept can be described by different views or data types. Collaborative encyclopedias (such as Wikipedia) describe a famous person through a mixture of text, images and, in some cases, audio. Users from social networks comment events like concerts or sport games with small phrases and multimedia attachments (images/videos/audios). Medical records are represented by a collection of images, sound, text and signals, among others. The increasing availability of multimodal databases from different sources has motivated the development of automatic analysis techniques to exploit the potential of these data as a source of knowledge in the form of patterns and structures that reveal complex relationships (Bhatt & Kankanhalli, 2011; Atrey et al., 2010) . In recent years, multimodal tasks have acquired attention by the representation learning community. Strategies for visual question answering (Antol et al., 2015) , or image captioning (Vinyals et al., 2015; Xu et al., 2015; Johnson et al., 2015) have developed interesting ways of combining different representation learning architectures.

Most of these models are focused on mapping from one modality to another or solving an auxiliary task to create a common representation with the information of all modalities. In this work, we design a novel module that combines multiple sources of information, which is optimized with respect to the end goal objective function. Our proposed module is based on the idea of gates for selecting which parts of the input are more likely to contribute for correctly generating the desired output. We use multiplicative gates that assign importance to various features simultaneously, creating a rich multimodal representation that does not require manual tuning, but instead it learns directly from the training data. Our gated model can be reused in different network architectures for solving different tasks, and can be optimized end-to-end with other modules in the architecture using standard gradient-based optimization algorithms.

As an application use case, we explore the task of identifying a movie genre based on its plot and its poster. Genre classification has several application areas like document categorization (Kanaris & Stamatatos, 2009) , recommendation systems (Makita & Lenskiy, 2016a) , and information retrieval systems, among others. Figure 1 depicts the challenging task of assigning genres to a particular movie based solely on the usage of one modality. Such predictions were done with MaxoutMLP w2v and VGG transfer approaches (See Section 3), both of them are models based on representation learning. It can be seen that even a human might be confused if both modalities are not available.

The main hypothesis of this work is that a model using gating units, in contrast to a hand-coded multimodal fusion architecture, will be able to learn an input-dependent gate-activation pattern that determines how each modality contribute to the output of hidden units.

The rest of the paper is organized as follows: Section 2 presents a literature review and some considerations of the previous work. Section 3 describes the methods used as baseline as well as our representation-leaning-based model proposed. Section 4 presents the experimental evaluation setup along with the details of the MM-IMDb dataset. Section 5 shows and discusses the results for movie genre classification. Finally, Section 6 draws the conclusions and future work.

2. RELATED WORK:



2.1. MULTIMODAL FUSION:

Different reviews (Atrey et al., 2010; Bhatt & Kankanhalli, 2011; Li Deng, 2014; Deng, 2014) have summarized strategies that addressed multimodal analysis. Most of the collected works claimed the superiority of multimodal over unimodal approaches for automatic analysis tasks. A conventional multimodal analysis system receives as input two or more modalities that describe a particular concept. The most common multimodal sources are video, audio, images and text. In recent years there has been a consensus with respect to the use of representation learning models to characterize the information of this kind of sources (LeCun et al., 2015) . However, the way that such extracted features are combined is still in exploration.

Multimodal combination seeks to generate a single representation that makes easier automatic analysis tasks when building classifiers or other predictors. A simple approach is to concatenate features to get a final representation (Kiela & Bottou, 2014; Pei et al., 2013; Suk & Shen, 2013) . Although it is a straightforward strategy, it ignores inherent correlations between different modalities.

More complex fusion strategies include Restricted Boltzmann Machines (RBMs) and autoencoders. Ngiam et al. (2011) concatenated higher level representations and train two RBMs to reconstruct the original audio and video representations respectively. Additionally, they trained a model to recon-struct both modalities given only one of them as input. In an interesting result, Ngiam et al. (2011) were able to mimic a perceptual phenomenon that demonstrates an interaction between hearing and vision in speech perception known as McGurk effect. A similar approach was proposed by Srivastava & Salakhutdinov (2012) . They modified feature learning and reconstruction phases with Deep Boltzmann Machines. Authors claimed that such strategy is able to exploit large amounts of unlabeled data by improving the performance in retrieval and annotation tasks. Other similar strategies propose to fusion modalities using neural network architectures (Andrew et al., 2013; Feng et al., 2013; Kang et al., 2012; Kiros et al., 2014a; Lu et al., 2014; Mao et al., 2014; Tu et al., 2014; Wu et al., 2013) with two input layers separately and including a final supervised layer such as softmax regression classifier.

An alternative approach involves an objective or loss function suited for the target task (Akata et al., 2014; Frome et al., 2013; Kiros et al., 2014b; Mao et al., 2014; Socher et al., 2013; 2014; Zheng et al., 2014) . These strategies usually assume that there exists a common latent space where modalities can express the same semantic concept through a set of transformations of the raw data. The semantic embedding representations are such that two concepts are similar if and only if their semantic embeddings are close (Norouzi et al., 2014) . In (Socher et al., 2013) a multimodal strategy to perform zero-shot classification was proposed. They trained a word-based neural network model (Huang et al., 2012) to represent textual information, whilst use unsupervised feature learning models proposed in (Coates & Ng, 2011) to get image representation. The fusion was done by learning an image linear mapping to project images into the semantic word space learned in the neural network model. Additionally a Bayesian framework was included to decide whether an image is of a seen or unseen class. Frome et al. (2013) learn the image representation using a CNN trained with the Imagenet dataset and a word-based neural language model (Mikolov et al., 2013b) to represent the textual modality. To perform the fusion they re-train CNN using text representation as targets. This work outperforms scalability with respect to (Socher et al., 2013) from 2 to 20,000 unknown classes in the zero-shot learning task. A modified strategy of Frome et al. (2013) was presented by Norouzi et al. (2014) . Instead of re-train the CNN network, they built a convex combination with probabilities estimated by the classifier and semantic embedding vector of the unseen label. This simple strategy outperforms state-of-the-art results. Because the cost function involves both multimodal combination and supervision, these family of models are tied to the task of interest. Thus, if the domain or task conditions changes, adaptations are required.

The proposed model is closely related to the mixture of experts (MoE) approach (Jacobs et al., 1991) . However, the common usage of MoE is focused on performing decision fusion, i.e. combining predictors to address a supervised learning problem (Yuksel et al., 2012) . Our model is devised as a new component in the representation learning scheme, making it independent from the final task (e.g. classification, regression, unsupervised learning, etc) provided that the defined cost function be differentiable.

2.2. MOVIE GENRE CLASSIFICATION:

With respect to movie genre classification, several strategies also have been proposed. These strategies have used different modalities to characterize each movie, such as textual features, image features and multimedia features (audio and/or video). Huang et al. (2007) were one of the first teams exploring this task. They classified movie previews into 3 genres by extracting handcrafted features from the video and training a decision tree classifier. They evaluated the model using 44 films. Using only textual modality, Shah et al. (2013) performed single-label genre classification of movie scripts using clustering algorithms with 260 movies. Later, combining two modalities, Pais et al. (2012) classified movies between drama and non-drama using visual and textual features with 107 samples. Hong & Hwang (2015) explored different PLSA models to combine 3 modalities: audio, image and text to predict genre of movie previews. It was single label classification with 4 genres for 140 movies taken from IMDb.

Recently, Fu et al. (2015) used a set of handcrafted visual features for poster characterization and bag-of-words for synopsis. Then, they trained one SVM per each modality to combine their predictions. The dataset contained 2, 400 movies with one genre (out of 4) each.

The previous mentioned works present this problem in a single label setup. However, a more realistic scenario would be multilabel, since most of the movies belong to more than one genre, (e.g.

Matrix( 2000) is a Sci-fi/Action movie). In this setup, Anand (2014) explores the efficiency of using keywords and users' tags to perform multilabeling using the movies from MovieLens 1M dataset which contains 1, 700 movies. Also Ivasic-Kos et al. (2014; 2015) performed multilabel classification using handcrafted features from posters, with 1, 500 samples for 6 genres. Makita & Lenskiy (2016a; b) use movie ratings matrix and genre correlation matrix to predict the genre. It used a smaller version of the Movielens dataset with 18 movie genres.

Most of the above works have used the publicly available MovieLens datasets. However, there is not a single experimental setup defined so that all methods can be systematically compared. Also, to the best of our knowledge, none of the previous works contain more than 10, 000 samples. With this work we will release a dataset created with the movies of the MovieLens 20M dataset. We include not only genre, poster and plot information used in this work, but also the poster of the movie as well as more than 50 characteristics taken from the IMDb website. We will also release the source code to automatically add more movies and genres.

3. METHODS:

This paper presents a neural-network-based strategy for multilabel classification of multimodal data.

The key component of the strategy is a novel type of hidden unit, the Gated Multimodal Unit (GMU), which learns to decide how modalities influence the activation of the unit using gates. The details of the GMU are presented in Subsection 3.1.

Statistical properties usually are not shared across modalities (Srivastava & Salakhutdinov, 2012) . And thus, they require different representation strategies according to the nature of data. This work explored several strategies to address text and visual representation. For text information we evaluated word2vec models, n-grams models and RNN models. The details are discussed in Subsection 3.2. On the other hand, two different convolutional neural networks were evaluated for processing visual data and are presented in Subsection 3.3.

3.1. GATED MULTIMODAL UNIT FOR MULTIMODAL FUSION:

Multimodal learning is closely related to data fusion. Data fusion looks for optimal ways of combining different information sources into an integrated representation that provides more information than the individual sources (Bhatt & Kankanhalli, 2011) . This fusion can be performed at different levels, that can be categorized into two broad categories: feature fusion and decision fusion. Feature fusion, also called early fusion, looks for a subset of features from different modalities, or combinations of them, that better represent the information needed to solve a particular problem. On the other hand, decision fusion, or late fusion, combines decisions from different systems, e.g. classifiers, to produce consensus. This consensus may be reached by a simple average, a voting system or a more complex Bayesian framework.

In this work we present a model, based on gated neural networks, for data fusion that combines ideas from both feature and decision fusion. The model, called Gated Multimodal Unit (GMU), is inspired by the flow control in recurrent architectures like GRU or LSTM. A GMU is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. Figure 2 .a depicts the structure of a GMU. Each x i corresponds to a feature vector associated with modality i. Each feature vector feeds a neuron with a tanh activation function, which is intended to encode an internal representation feature based on the particular modality. For each input modality, x i , there is a gate neuron (represented by σ nodes in the diagram), which controls the contribution of the feature calculated from x i to the overall output of the unit. When a new sample is fed to the network, a gate neuron associated to modality i receives as input the feature vectors from all the modalities and uses them to decide whether the modality i may contribute, or not, to the internal encoding of the particular input sample.

Figure 2 .b shows a simplified version of the GMU for two input modalities, x v (visual modality) and x t (textual modality), that will be used in the remaining of the paper. It should be noted that both models are not completely equivalent, since in the bimodal case the gates are tied. Such weight tying constraints the model, so that the units trade off between both modalities while they use less parameters than the multimodal case. The equations governing this GMU are as follows:

x 1 σ tanh x 2 σ tanh • • • x k σ tanh + h z 1 h 1 z 2 h 2 z k h k (a) x v x t σ tanh tanh 1- + h z h t h v (b)

h v = tanh (W v • x v ) h t = tanh (W t • x t ) z = σ (W z • [x v , x t ]) h = z * h v + (1 -z) * h t Θ = {W v , W t , W z }

with Θ the parameters to be learned and [•, •] the concatenation operator. Since all are differentiable operations, this model can be easily coupled with other neural network architectures and trained with stochastic gradient descent.

3.2. TEXT REPRESENTATION:

Text representation is a critical step when classification tasks are addressed using machine learning methods. Traditional approaches are based on counting frequencies of n-gram occurrences such as words or sequences of characters (e.g. bag-of-words models). The main drawback of such approaches is the difficulty to model relationships between words and their context. An alternative approach was initially proposed by Bengio et al. (2003) , by building a language model based on a neural network architecture (NNLM). The NNLM was able to learn distributed representations of words that capture contextual information. Later, this model was simplified to deal with large corpora by removing hidden layers in the neural network architecture (word2vec) (Mikolov et al., 2013a) . This is a fully unsupervised model that takes advantage of large sets of unlabeled documents. Herein, three text representations were evaluated:

n-gram Following the strategy proposed by Kanaris & Stamatatos (2009) , we used the n-gram strategy for representing text. Despite their simplicity, n-gram models have shown to be a competitive baseline. Word2Vec Word2vec is an unsupervised learning algorithm that finds a vector representation for each word based on its context (Mikolov et al., 2013a) . It has been shown that this model is able to find semantic and syntactic relationships using arithmetic operations between the vectors. Based on this property, we represent a movie as the average of the vectors of words in the plot outline. The main motivation to aggregate word2vec vectors is the property of additive compositionality that this representation has exposed over different set of tasks such as word analogies. The usual way to aggregate is to sum vectors. We instead take the average to avoid large input values to the neural network. Recurrent neural network Here we take the plot outline as a sequence of words and train a supervised recurrent neural network. We evaluated two variants. The first one (RNN w2v) is a transfer learning model that takes as input the word vectors of word2vec as representations.

The second one learns the word vectors from scratch (RNN end2end).

3.3. VISUAL REPRESENTATION:

In computer vision tasks, Convolutional neural networks have become the de facto standard. It has been shown that CNN models trained with a huge amount of data are able to learn common features shared across different domains. This characteristic is usually exploited by transfer learning approaches. For visual representation we explored 2 strategies: transfer learning and end-to-end training.

VGG Transfer In this approach, the VGG Network (Simonyan & Zisserman, 2014) trained with the ImageNet dataset is used as feature extractor by taking the last hidden activations as the visual representation.

End2End CNN Here, a CNN with 5 convolutional layers and an MLP (see Section 3.4) on top was trained from scratch.

3.4. CLASSIFICATION MODEL:

Based on the defined representation, we explored two methods to map from feature vectors to genre classification. In particular we explored a simple Logistic regression and a neural network architecture. This is a multilayer perceptron (MLP) with two fully connected layers and maxout activation function. In particular, the maxout activation function h i : R n → R is a defined as:

EQUATION

where s ∈ R n is the input vector, z i,j = s T W •••ij +b ij is the output of the j-th linear transformation of the i-th hidden unit, and W ∈ R d×m×k and b ∈ R m×k are learned parameters. It has been shown that maxout models with just 2 hidden units behave as universal approximators, while are less prone to saturate units (Goodfellow et al., 2013) . 

4. EXPERIMENTAL EVALUATION:



4.1. MULTIMODAL IMDB DATASET:

With this work we will make publicly available the Multimodal IMDb (MM-IMDb)1 dataset. MM-IMDb dataset is built with the IMDb id's provided by the Movielens 20M dataset2 that contains ratings of 27, 000 movies. Using the IMDbPY3 library, movies which do not contain their poster image were filtered out. As the final result, the MM-IMDb dataset comprises 25, 959 movies along with their plot, poster, genres and other 50 additional metadata fields such as year, language, writer, director, aspect ratio, etc.

Notice that one movie may belong to more than one genre. Figure 3 shows the co-occurrence matrix, where the color bar indicates the representative co-occurrence per row, while Figure 4 and Figure 5 depict the distribution of the movie poster sizes and length of movie plots respectively. Each plot contains on average 92.5 words, while the longest one contains 1, 431 words and the average of genres per movie is 2.48. In this work, we defined the task of movie genre prediction based on its plot and image poster. Nevertheless, the additional metadata information encourages other interesting tasks such as rating prediction and content-based retrieval, among others.

4.2. EXPERIMENTAL SETUP:

The MM-IMDb dataset has been split in three subsets. Train, development and test subsets contain 15552, 2608 and 7799 respectively. The distribution of samples is listed in Table 1 . The sample was stratified so that training, dev and test sets comprises 60%, 10%, 30% samples of each genre respectively.

In the multilabel classification the performance evaluation can be more complex than traditional multi-class classification and the differences can be significant among several measures (Madjarov et al., 2012) . Herein, four averages of the f-score (f 1 ) are reported: samples computes the f-score per sample and then averages the results, micro computes the f-score using all predictions at once, macro computes the f-score per genre and then averages the results. weighted is the same as macro with a weighted average based on the number of positive samples per genre. Concretely, we calculate them as follows (Madjarov et al., 2012) : 

f sample 1 = 1 N N i=1 2 × |ŷ i ∩ y i | |ŷ i | + |y i | f macro 1 = 1 Q Q j=1 2×p j ×rj p j +rj f weighted 1 = 1 Q 2 Q j=1 Q j 2 × p j × r j p j + r j

p micro = Q j=1 tp j Q j=1 tp j + Q j=1 f p j r micro = Q j=1 tpj Q j=1 tpj + Q j=1 f nj f micro 1 = 2 × p micro × r micro p micro + r micro

With N the number of examples; Q the number of labels; Q j the number of true instances for the j-th label; p the precision, r the recall; ŷi , y i ∈ (0, 1) Q the prediction and ground truth binary tuples respectively; tp j , f p j andf n j the number of true positives, false positives and false negatives for the j-th label respectively.

TEXTUAL REPRESENTATION:

The pretrained Google Word2vec4 embedding space was used. After intersecting the Google word2vec available words with the MM-IMDb plots, the final vocabulary contains 41,612 words. Other than lowercase, no text preprocessing was applied. Since it is our intention to measure how the network's depth affects the performance of the model, we also evaluate the architecture with a single fully connected layer. In order to compare the performance of this textual representation, we evaluate it using two publicly available datasets: 7genre dataset that comprises 1,400 web pages with 7 disjoint genres and ki-04 dataset that comprises 1,239 samples classified under 8 genres. We compare the model with the state of the art results (Kanaris & Stamatatos, 2009) which used character n-grams with structured information from the HTML tags to predict the genre of web pages.

Since the first approach was to use VGG as a feature extractor. This model is referred as VGG Transfer. The second approach takes as input the raw images to a CNN. Since all the images do not have the same size, all images were scaled, and cropped when required, to 160 × 256 pixels keeping the aspect ratio. This CNN comprises 5 CNN layers of 5, 3, 3, 3, 3 squared filters and 2 × 2 pool sizes. Each convolutional layer has 16 hidden units. The convolutional layers are connected with the MaxoutMLP on top.

MULTIMODAL REPRESENTATION:

We evaluate 4 different ways to combine both modalities as baselines.

Average probability This can be seen as a late-fusion strategy. The probabilities obtained by the best model of each modality are averaged and thresholded. concatenation Different works have found that a simple concatenation of representations of different modalities are good for combining the information (Suk & Shen, 2013; Pei et al., 2013; Kiela & Bottou, 2014) . Herein, we concatenated both representations to train the MaxoutMLP architecture. 2015) combine text and images representation into a single space, this model adds a linear transformation for each modality so that both outputs have the same size to be summed up and then followed by the MaxoutMLP architecture.

C M y v y t x v ŷv γv γ 0 v , γ 1 v γ 0 t , γ 1 t ŷt γt x t

MoE:

The mixture of experts (MoE) (Jacobs et al., 1991) model was adapted for multilabel classification. two gating strategies were explored: tied, where a single gate multiplies all the logistics outputs, and untied where every logistic output has its own gate. Logistic regression and MaxoutMLP were evaluated as experts.

NEURAL NETWORK TRAINING:

Neural network models were trained using using Batch Normalization scheme (Ioffe & Szegedy, 2015) . This strategy applies a normalization step across samples that belong to the same batch, so that each hidden unit in the network receive a zero-mean and unit variance. Stochastic gradient descent with ADAM optimization (Kingma & Ba, 2014) was used to learn the weights of the neural network. Dropout and max-norm regularization were used to control overfitting. Hidden size ({64, 128, 256, 512}), learning rate ( 10 -3 , 10 -1 ), dropout ([0.3, 0.7]), max-norm ([5, 20] ) and initialization ranges ( 10 -3 , 10 -1 ) parameters were explored by training 25 models with random (uniform) hyperparameter initializations and the best was chosen according to validation performance. It has been reported that this strategy is preferable over grid search when training deep models (Bergstra & Bengio, 2012) . All the implementation was carried on with the Blocks framework (Van Merriënboer et al., 2015) 5 .

During the training process, we noticed that batch normalization considerably helped in terms of training time and convergence, resulting in less sensitivity to hyperparameters such as initialization ranges or learning rate. Also, dropout and max-norm regularization strategies helped to increase the performance at test time.

5. RESULTS:



5.1. EVALUATION OVER SYNTHETIC DATA:

In order to evaluate if the model is able to identify which modality is contributing more information to classify a particular sample, we created a synthetic task based on a generative model, which is depicted in Figure 6 . In this model we define the random binary variable C as the target and x v , x t ∈ R 2 as the input features. M is a random binary variable that decides which modality will contain the relevant information that determines the class. The input features of each modality can

γ 0 v γ 1 v γv x v x t γ 0 t γ 1 t γt γ 0 v γ 1 v γv x v x t γ 0 t γ 1 t γt

Figure 7 : Activations of z (left) and prediction (right) for a synthetic experiment with x v , x t ∈ R 1 .

be generated by a random source, ŷv and ŷt , or by an informed source, y v and y t . The generative model is specified as follows:

C ∼ Bernoulli(p C ) M ∼ Bernoulli(p M ) y v ∼ N (γ C v ) ŷv ∼ N (γ v ) x v = M y v + (1 -M )ŷ v y t ∼ N (γ C t ) ŷt ∼ N (γ t ) x t = M ŷt + (1 -M )y t

We trained a model with a single GMU and applied a sigmoid function over h, then the binary cross entropy was used as loss function. Using the generative model, 200 samples per class were generated for each experiment. 1000 synthetic experiments with different random seeds were run and the GMU outperformed a logistic regression classifier in 370 of them, while obtaining equal results in the remainder ones. Our goal in these simulations was to show that the model was able to learn a latent variable that determines which modality carries the useful information for the classification. An interesting result is that between M and the activations of the gate z there is a correlation of 1. This means the model was capable of learning such latent variable by only observing the x v and x t input features.

We also wanted to project back the z activations to the feature space in order to visualize regions depending on the modality. Figure 7 shows the activations in a synthetic experiment generated by the setup of Figure 6 for x v , x t ∈ R 1 . Each axis represents a modality, red and blue dots are the samples generated for the two classes and black Gaussian curves represent the γv and γt noises. The contour of the left figure (gray) represents the activation of z. Notice that in white regions (z = 1), the model gives more importance to the x v modality while in gray regions (z = 0) the x t modality is more relevant; i.e. the z gate is isolating the noise. The contour of the right figure (blue-red) represents the model prediction. It is noteworthy that the boundary defined by the gates still holds when the model solves the task. This also encourages the inclusion of non-linearities to the z gate so that it is able to discriminate more complex interactions between modalities.

5.2. GENRE CLASSIFICATION RESULTS:

Before using our text representation in the multimodal task, we wanted to be sure such representation was good enough to address the genre classification task. Thus, we evaluated it on 2 public datasets. We found MaxoutMLP w2v achieves the state of the art results on the ki-04 dataset and increases the performance in the 7Genre dataset from 0.841 to 0.854 (Kanaris & Stamatatos, 2009) . Notice that the baseline uses additional information from the HTML structure from the web page, while this representation uses only the text data. Table 2 shows the results in the proposed dataset. For the textual modality, the best performance is obtained by the combination of word2vec representation with an MLP classifier. The behavior of all representation methods are consistent across the performance measures. Learning from scratch the RNN model performed the worst. We hypothesize this has to do with the lack of data to learn meaningful relations among words. It has been shown that millions of words are required to train a model such as word2vec that is able to exploit common regularities between word co-occurrences.

For the visual modality, the usage of pretrained models works better than training the model from scratch. It seems it is still a small dataset to learn all the complexities of the posters. Now, comparing the performance independently per genre, as in Table 3 , it is interesting to notice that in Animation the visual modality outperforms the textual one.

In the multimodal scenario, by adding the GMU as building block to learn the fusion we obtained the best performance, improving independent modalities in the averaged measures and in 16 of out 23 genres and outperforming all other evaluated fusion strategies. The concatenation or the linear combination approaches were not enough to model the correlation between the modalities and MoE models did not perform better than simpler approaches. This is an expected behavior for MoE in a relatively small dataset because the data is fractionated over different experts, and thus it doesn't make an efficient use of the training samples.

In order to evaluate which modality influences more the model when assigning a particular label, we averaged the activations of a subset of z gates of the test samples to which the model assigned them such label. We counted the number of samples that pays more attention to the textual modality (z <= 0.5) or to the visual modality (z > 0.5). The units were chosen taking into account the mutual information between the predictions and the z activations. The result of this analysis is depicted in Figure 8 . As expected, the model is generally more influenced by the textual modality. But, in particular cases such as Animation or Family genres, the visual modality affects more the model. This is also consistent with results of Table 3 which reports better performances for visual modality.

We wanted to qualitative explore test examples in which performance was improved by a relative large margin. Table 4 illustrates cases where the model takes advantage of the most accurate modality, and in some cases removes false positives. It is noteworthy that some of these examples can be confusing for a human if one modality is missing, or additional context information is not given.

6. CONCLUSIONS:

This work presented a strategy to learn fusion transformations from multimodal sources. Similarly to the way recurrent models control the information flow, the proposed model is based on multiplicative gates. The Gated Multimodal Unit (GMU) receives two or more input sources and learns to determine how much each input modality affects the unit activation. In synthetic experiments the GMU was able to learn hidden latent variables, and in a real scenario it outperformed the singlemodality approaches. An interesting property of GMU is that, being a differentiable operation, it is easily coupled in any other neural network architecture and trained with standard gradient-based optimization algorithms. With this work we will also release a new dataset that contains around 27, 000 movie plots, images and other metadata. To the best of our knowledge, this is the biggest dataset used to perform movie genre classification based on multimodal information and the first one to be publicly available. In our future work we expect to explore deep architectures of GMU layers as well as integration with attention mechanism over the input modalities. Also, It will be interesting to explore in more depth the interpretability of the learned features.

:

http://lisi1.unal.edu.co/mmimdb/

http://grouplens.org/datasets/movielens/

http://imdbpy.sourceforge.net/

https://code.google.com/archive/p/word2vec/

https://github.com/johnarevalo/gmu-mmimdb

Figures and Tables:

Figure 1: Predictions of genre labels depending on the input modality. Red and blue labels indicate false positives and true positives respectively.

Figure 2: Illustration of gated units. a) The proposed model to use with more than two modalities. b) A simplification for the bimodal approach.

Figure 3: Co-ocurrence matrix of genre tags

Figure 4: Size distribution of movie posters.

Figure 6: Generative model for the synthetic task. Grayed nodes represent visible variables, the other nodes represent hidden variables.

Figure 8: Percentage of gates activations (z > 0.5: Visual; z <= 0.5: textual) for test samples to which the model assigned them the label.

Genre distribution per subset

Table content: <table><tr><td>Genre</td><td colspan="2">Train Dev</td><td>Test</td><td>Genre</td><td colspan="3">Train Dev Test</td></tr><tr><td>Drama</td><td>8424</td><td colspan="2">1401 4142</td><td>Family</td><td>978</td><td colspan="2">172 518</td></tr><tr><td>Comedy</td><td>5108</td><td>873</td><td>2611</td><td colspan="2">Biography 788</td><td colspan="2">144 411</td></tr><tr><td>Romance</td><td>3226</td><td>548</td><td>1590</td><td>War</td><td>806</td><td colspan="2">128 401</td></tr><tr><td>Thriller</td><td>3113</td><td>512</td><td>1567</td><td>History</td><td>680</td><td colspan="2">118 345</td></tr><tr><td>Crime</td><td>2293</td><td>382</td><td>1163</td><td>Music</td><td>634</td><td colspan="2">100 311</td></tr><tr><td>Action</td><td>2155</td><td>351</td><td>1044</td><td colspan="2">Animation 586</td><td colspan="2">105 306</td></tr><tr><td>Adventure</td><td>1611</td><td>278</td><td>821</td><td>Musical</td><td>503</td><td>85</td><td>253</td></tr><tr><td>Horror</td><td>1603</td><td>275</td><td>825</td><td>Western</td><td>423</td><td>72</td><td>210</td></tr><tr><td colspan="2">Documentary 1234</td><td>219</td><td>629</td><td>Sport</td><td>379</td><td>64</td><td>191</td></tr><tr><td>Mystery</td><td>1231</td><td>209</td><td>617</td><td>Short</td><td>281</td><td>48</td><td>142</td></tr><tr><td>Sci-Fi</td><td>1212</td><td>193</td><td>586</td><td colspan="2">Film-Noir 202</td><td>34</td><td>102</td></tr><tr><td>Fantasy</td><td>1162</td><td>186</td><td>585</td><td/><td/><td/><td/></tr></table>

Summary of classification task on the MM-IMDb dataset

Table content: <table><tr><td>Modality</td><td>Representation</td><td colspan="3">F-Score weighted samples micro macro</td></tr><tr><td colspan="2">Multimodal GMU</td><td>0.617</td><td>0.630</td><td>0.630 0.541</td></tr><tr><td/><td>Linear sum</td><td>0.600</td><td>0.607</td><td>0.607 0.530</td></tr><tr><td/><td>Concatenate</td><td>0.597</td><td>0.605</td><td>0.606 0.521</td></tr><tr><td/><td>AVG probs</td><td>0.604</td><td>0.616</td><td>0.615 0.491</td></tr><tr><td/><td>MoE MaxoutMLP</td><td>0.592</td><td>0.593</td><td>0.601 0.516</td></tr><tr><td/><td>MoE MaxoutMLP (tied)</td><td>0.579</td><td>0.579</td><td>0.587 0.489</td></tr><tr><td/><td>MoE Logistic</td><td>0.541</td><td>0.557</td><td>0.565 0.456</td></tr><tr><td/><td>MoE Logistic (tied)</td><td>0.483</td><td>0.507</td><td>0.518 0.358</td></tr><tr><td/><td>MaxoutMLP w2v</td><td>0.588</td><td>0.592</td><td>0.595 0.488</td></tr><tr><td/><td>RNN transfer</td><td>0.570</td><td>0.580</td><td>0.580 0.480</td></tr><tr><td/><td colspan="2">MaxoutMLP w2v 1 hidden 0.540</td><td>0.540</td><td>0.550 0.440</td></tr><tr><td>Text</td><td>Logistic w2v</td><td>0.530</td><td>0.540</td><td>0.550 0.420</td></tr><tr><td/><td>MaxoutMLP 3grams</td><td>0.510</td><td>0.510</td><td>0.520 0.420</td></tr><tr><td/><td>Logistic 3grams</td><td>0.510</td><td>0.520</td><td>0.530 0.400</td></tr><tr><td/><td>RNN end2end</td><td>0.490</td><td>0.490</td><td>0.490 0.370</td></tr><tr><td>Visual</td><td>VGG Transfer CNN end2end</td><td>0.410 0.370</td><td>0.429 0.350</td><td>0.437 0.284 0.340 0.210</td></tr></table>

Macro F-Score reported per genre for single and multimodal approaches.

Table content: <table><tr><td>Genre</td><td colspan="4">Textual Visual GMU Genre</td><td colspan="3">Textual Visual GMU</td></tr><tr><td>Drama</td><td>0.74</td><td>0.67</td><td>0.77</td><td>Fantasy</td><td>0.42</td><td>0.25</td><td>0.46</td></tr><tr><td>Comedy</td><td>0.65</td><td>0.59</td><td>0.68</td><td>Family</td><td>0.5</td><td>0.46</td><td>0.58</td></tr><tr><td>Romance</td><td>0.53</td><td>0.33</td><td>0.51</td><td colspan="2">Biography 0.4</td><td>0.02</td><td>0.25</td></tr><tr><td>Thriller</td><td>0.57</td><td>0.39</td><td>0.62</td><td>War</td><td>0.57</td><td>0.19</td><td>0.64</td></tr><tr><td>Crime</td><td>0.61</td><td>0.25</td><td>0.59</td><td>History</td><td>0.35</td><td>0.06</td><td>0.29</td></tr><tr><td>Action</td><td>0.58</td><td>0.37</td><td>0.6</td><td colspan="2">Animation 0.43</td><td>0.61</td><td>0.68</td></tr><tr><td>Adventure</td><td>0.51</td><td>0.32</td><td>0.51</td><td>Musical</td><td>0.14</td><td>0.18</td><td>0.28</td></tr><tr><td>Horror</td><td>0.65</td><td>0.41</td><td>0.69</td><td>Western</td><td>0.52</td><td>0.37</td><td>0.65</td></tr><tr><td colspan="2">Documentary 0.67</td><td>0.18</td><td>0.76</td><td>Sport</td><td>0.64</td><td>0.11</td><td>0.7</td></tr><tr><td>Mystery</td><td>0.38</td><td>0.11</td><td>0.39</td><td>Short</td><td>0.2</td><td>0.24</td><td>0.27</td></tr><tr><td>Sci-Fi</td><td>0.63</td><td>0.3</td><td>0.66</td><td colspan="2">Film-Noir 0.02</td><td>0.11</td><td>0.37</td></tr><tr><td>Music</td><td>0.51</td><td>0.01</td><td>0.48</td><td/><td/><td/><td/></tr></table>

Peer review (iclr2017):
Paper proposes Gated Muiltimodal Unit, a building block for connectionist models capable of handling multiple modalities. (Figure 2) The bimodal case returns weighted activation by gains of gating units, do you do anything special to keep multi-modal case weighted as well? I.e. how the equation for h in section 3.1 would look like for multi-modal case. Also what’s the rationale for using tanh nonlinearity (over, say RELU), is it somehow experimentally optimised choice? I would find interesting a discussion on a possibility of handling missing data in case one or more modalities are unavailable at test time. Is this possible in the current model to back-off to fewer modalities? Synthetic example may suggest that’s in fact possible. Those numbers, perhaps, could be added to table 2. In the synthetic experiment, you should compare MGU with the fully-connected MLP model really, with similar complexity - that is - at least two hidden units (as GMU has two such for each modality) followed by logistic regression. At least in terms of capability of drawing decision boundary, those should be comparable. I think, broader discussion shall be written on the related work associated with mixture of experts models (which is fact are very similar conceptually) as well as multiplicative RNN models [1]. Also, gating unit in LSTM can, in principle, play very similar role when multiple modalities are spliced in the input. Overall, the paper is interesting, so is the associated (and to be released) dataset. Minor comments/typos: Sec. 3.3: layers and a MLP (see Section 3.4) -> layers and an MLP Apologies for unacceptably late review. [1] Multiplicative LSTM for sequence modelling B Krause, L Lu, I Murray, S Renals