Introduction
Significant progress has been made in the field of machine learning in the past years due to the rapid development of deep learning BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5. Dating back to the dramatic increase in the accuracy of large-scale automatic speech recognition (ASR) using fully connected deep neural networks (DNN) and deep auto-encoders around 2010 BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, and followed by a set of breakthroughs in computer vision (CV) using deep convolutional neural network (CNN) models BIBREF17 for large-scale image classification around 2012 BIBREF18, BIBREF19, BIBREF20, BIBREF21 and large-scale object detection BIBREF22, BIBREF23, BIBREF24 around 2014, a set of major milestones have been achieved in pattern recognition with single input modality. Subsequently, in natural language processing (NLP), recurrent neural network (RNN) based semantic slot filling methods BIBREF25 achieved new state-of-the-art in spoken language understanding, and RNN-encoder-decoder models with attention mechanism BIBREF26, also referred to as sequence to sequence models BIBREF27, produced superior performance in machine translation in an end-to-end fashion BIBREF28, BIBREF29. For other NLP tasks without much training data, such as question answering (QA) and machine reading comprehension, generative pre-training that transfers parameters from a language model (LM) pre-trained on a large out-of-domain data set using unsupervised or self learning then followed by fine-tuning on small in-domain data sets, achieved record-breaking results over a set of tasks BIBREF30, BIBREF31, BIBREF32.
Despite the advances in vision, speech, and language processing, many problems in artificial intelligence involve more than one modality, such as an intelligent personal assistant (IPA) that should understand human communicative intentions embedded not only in spoken language, but also in body and pictorial languages BIBREF33. Therefore, it is of broad interests to study the modeling and learning approaches across multiple modalities BIBREF34. Benefiting from the advances in image processing and language understanding BIBREF35, a set of tasks that combine both image and text have drawn much attention, which include visual grounding tasks like referring expression understanding and phrase localization BIBREF36, BIBREF37, BIBREF38, image captioning BIBREF39, BIBREF40, BIBREF41, visual QA (VQA) BIBREF42, BIBREF43, BIBREF44, text-to-image generation BIBREF45, BIBREF46, BIBREF47, and visual-language navigation BIBREF48 etc. In these tasks, natural language plays a key role in helping the machine to “understand” the content of the images, where “understand” means to capture the underlying correlations between the semantics embedded in language with the visual features obtained from the images. In addition to text, vision can be combined with speech as well. Such tasks include audio-visual speech recognition BIBREF49, BIBREF50, BIBREF51, speaker recognition BIBREF52, BIBREF53, BIBREF54, as well as speech diarisation BIBREF55, BIBREF56, separation BIBREF57, BIBREF58 and enhancement BIBREF59, which mostly focused on the use of visual features to improve the robustness of the audio-only methods.
In this paper, a technical review of the models and learning methods for multimodal intelligence is provided. The main focus is the combination of CV and NLP, which has become an important area for both research communities covering many different tasks and technologies. To provide a more structured perspective, we organize the methods selected in this technical review according to three key topics: representation, fusion, and applications.
Learning representations for the input data is a core problem for deep learning. For multimodal tasks, collecting paralleled data across all modalities can be quite difficult and leveraging pre-trained representations with desired properties, such as suitable for zero-shot or few-shot learning, is often an effective solution to the issue. Both supervised and unsupervised training based multimodal representation learning methods are reviewed.
The fusion of the features or representations of the single modalities is undoubtedly a centric problem of any multimodal task. Different from previous studies that often categorise the related work into early, middle and late stage methods based on the stage that fusion happens in the procedure, we classify them according to the actual operation used in the fusion, such as attention and bilinear pooling, since it becomes difficult to classify some recent complex approaches into stages.
Three types of applications are reviewed in this paper, namely image captioning, text-to-image synthesis and VQA. This is to give an idea how representation learning and fusion can be applied to specific tasks, and to provide a viewpoint of the situation of the current development of the multimodal applications, especially those integrating vision with natural languages. Visual reasoning methods for VQA are also discussed in the end.
This paper is organised as follows. Section SECREF2 reviews the recent progress on developing representations for single or multiple modalities. Section SECREF3 introduces the commonly used fusion methods, particularly attention and bilinear pooling. Applications including caption generation, text-to-image synthesis, VQA, and visual reasoning are introduced in Section SECREF4, followed by conclusions.
Representations
Deep learning, as a special area in representational learning, studies the use of artificial neural networks (ANNs) with many hidden layers to automatically discover the representations or features suitable for specific tasks from the raw data BIBREF60. In practice, it is often found that better representations can simplify the subsequent learning tasks and therefore has a great value. Over the past decade, it becomes feasible to learn effective and robust representations for single modalities, such as text BIBREF61, BIBREF62, BIBREF63, BIBREF30, BIBREF31, BIBREF32, BIBREF64, BIBREF65, BIBREF66, BIBREF67, BIBREF68 and image BIBREF18, BIBREF19, BIBREF20, BIBREF21, BIBREF22, BIBREF23, BIBREF24, due to the availability of large data and development of deep learning. For multimodal representations, though attracting more and more attentions, it still remains a challenging problem due to the complex cross-modal interactions and possible mismatch between the training and test data of each modal.
In this section, commonly used types of single modal representations, such as text and image, are first reviewed which often serve as cornerstones for learning multimodal representations. Afterwards, both supervised and unsupervised methods for learning a joint representation space for multiple modalities are introduced. To empower the model to handle data samples with some missing modality, zero-shot learning problem is studied to increase the similarity of the representational spaces across the involved modalities. At last, inspired by the great success of adapting pre-trained LMs for downstream tasks in NLP, methods that leverage large unimodal data sets to improve the learning of multimodal representations are also discussed.
Representations ::: Unimodal Embeddings
The representations obtained using ANN models are often distributed, which entails that elements composing the representations can be set separately to allow more concepts to be encoded efficiently in a relatively low-dimensional space BIBREF64. This can be compared with the symbolic representations, such as the one-hot encoding that uses an element with value one to indicate the presence of the associated symbol or category, and value zero for the rest elements. In deep learning, the term embedding often refers to a mapping from a one-hot encoding representing a word or an image category to a distributed representation as a vector of real-valued numbers.
Representations ::: Unimodal Embeddings ::: Visual representation
The image embeddings can be acquired as the output values from the final CNN layers from models that classify images into categories, such as AlexNet BIBREF18, VGG nets BIBREF19, and residual neural network (ResNet) BIBREF20. AlexNet is a model with five CNN layers with rectified linear unit (ReLU) activation functions whose kernel sizes are $11\times 11$, $5\times 5$, and $3\times 3$. A VGG net often has 16 or 34 CNN layers, with all of them using very small $3\times 3$ kernels. ResNet can have a depth of up to 152 layers, mostly with $3\times 3$ kernels, due to the invention of residual connections. Comparing to the aforementioned models, GoogLeNet has a more different structure formed by stacking multiple Inception structures BIBREF21. The naïve Inception structure is a concatenation of CNN layers with $1\times 1$, $3\times 3$, and $5\times 5$ sized kernels, and a max pooling layer with $3\times 3$ kernels, and can be viewed as a sparsely connected convolutional architecture to reduce overfitting and computational cost. Later versions of the Inception models improve the structures further by factorizing the kernels and adding residual connections. AlexNet, GoogLeNet, and ResNet are the winners of the 2012, 2014, and 2015 ImageNet Large Scale Visual Recognition Competition for image classification respectively BIBREF69, BIBREF70. Alternatively, features with more direct relationships with the semantics can be used as visual embeddings, such as convolutional features and the associated class labels from selected regions found by object detection models, such as region with CNN features (R-CNN) BIBREF22, Fast R-CNN BIBREF23, and Faster R-CNN BIBREF24 etc.
Representations ::: Unimodal Embeddings ::: Language representations
Text embeddings can be derived from a neural network language model (NNLM) BIBREF65, which estimates the probability of a text sequence by factorizing it into word probabilities based on the chain rule of probability. A feedforward neural network with a linear projection layer and a non-linear hidden layer is often used as an implementation of an $n$-gram LM, which takes the previous $n-1$ words as the input to predict the probability of the current word. Each word is presented in one-hot encoding based on the vocabulary and converted into a real-valued vector, the word embedding, using the projection layer. An improved NNLM is to replace the feedforward model with an RNN, such as a long short-term memory (LSTM) BIBREF71 or gated recurrent unit (GRU) BIBREF72 model, which allows the use of information from all past words stored in a fixed-length recurrent vector when predicting the current word. Apart from NNLMs, continuous bag-of-words model (CBOW) and skip-gram model are two simple feedforward structures that learn word embedding either by predicting the current word based on the past and future context words or vice versa BIBREF66. The method of global vectors (GloVe) shows that effective text-embedding can be learnt using a global log-bilinear regression model based on the co-occurrence counts of words BIBREF73. Meanwhile, a serious of deep structured semantic models (DSSM) were proposed since 2013 for sentence level embedding learning through optimizing semantic-similarity driven objectives, with various neural network structures in a pseudo-simense network setting BIBREF61, BIBREF62, BIBREF63, BIBREF74, BIBREF75, BIBREF76, BIBREF77.
More recently, in order to transfer to use in downstream natural language understanding tasks without much training data, studies focus on learning general text embeddings by predicting word probabilities using NNLMs with complex structures on a large text corpus. Embeddings from language models (ELMo) uses a combined embedding from multiple layers of bidirectional LSTMs for forward and backward directions BIBREF30. Generative pre-training (GPT) and bidirectional encoder representations for Transformers (BERT) use the decoder and encoder part of the Transformer model to estimate the probability of the current subword unit BIBREF31, BIBREF32. Other technologies, such as masked language model and multi-task training are used in these methods nowadays BIBREF32. Besides word and subword levels, text embedding can be learnt at phrase, sentence, and even paragraph levels, such as the skip-though vectors that extends the skip-gram method to the sequence-to-sequence framework BIBREF78, BIBREF27. It uses two decoders to predict the previous and next sentences given the embedding of the current sentence generated by the encoder.
Representations ::: Unimodal Embeddings ::: Vector arithmetic for word and image embeddings
It is well-known that word embeddings can learn not only syntactic but also the semantic regularities. A famous example showed vector(“King”)$-$vector(“Man”)$+$vector(“Woman”) results in a vector closest to vector(“Queen”) where vector($\cdot $) denotes the vector representation of a word learnt by a RNN LM BIBREF79. Similar phenomenon has been observed for vision embeddings. It was shown that when using a generative adversarial network (GAN) BIBREF80, there exists a similar vector arithmetic that the representation of an image with a man wearing glasses subtracted by that of a man without glasses and finally add the representation of a woman without glasses will lead to the representation of a woman wearing glasses BIBREF81. This reveals GAN can capture image representation that distangles the concept of gender from the concept of wearing glasses. Such encouraging progress in text and image representations encouraged further studies on the joint representations of these two modalities. More details about GAN based image generation can be found later in Section SECREF31.
Representations ::: Multimodal Representations
Although significant progress has been made in the learning of representations for vision or language, it is theoretically insufficient to model a complete set of human concepts using only unimodal data. For example, the concept of “beautiful music” is clearly grounded in the auditory perceptron and one can be struggled to describe this by natural language or other approaches. Therefore, it is important to learn a joint embedding to leverage the complementarity from multimodal data to represent the concepts better. Both supervised and unsupervised training approaches are of broad interest and can be applied to tasks with different data availability. Meanwhile, by assuming the corresponding representations to have similar neighbourhood structures across modalities, the representation of a concept with zero training sample in one modal can be found based on its representations grounded in other modalities which have training data. For instance, when using zero-shot training for image labelling, the closest word vectors can be retrieved as labels by projecting images of objects unseen in the training set onto the linguistic space. More recently, inspired by Transformer and BERT from NLP, it becomes increasingly popular to apply these models to develop better bimodal representations combining vision and language.
Representations ::: Multimodal Representations ::: Unsupervised training methods
Joint embeddings for multimodal data can be learnt by simply reconstructing the raw input using multiple streams of deep Boltzmann machines or autoencoders with a shared layer as the shared representation space BIBREF82, BIBREF83, BIBREF84. Alternatively, with the development of methods for single modal representations, the shared representation space can be constructed based on those of the involved single modalities. For example, in BIBREF85, Fang et al. propose a deep multimodal similarity model (DMSM), which extended the text-modal DSSM to learning embedding representations of text and image in an unified vector space. BIBREF84, BIBREF86 perform simple fusion of the word and image embeddings with addition or concatenation. BIBREF87 learns to increase the similarity between corresponding Skip-Gram word embedding and AlexNet derived image features. BIBREF88, BIBREF89 maximize the correlation and mutual information between embeddings of different modalities respectively. BIBREF90 modifies the distance between CBOW word embeddings according to the similarities between their visual instantiations, which are found by clustering abstract scenes in an unsupervised way.
Further studies found correlating image regions/fragments with sentence fragments or attribute words generates fine-grained multimodal embeddings BIBREF91, by finding the alignments of the image and sentence fragments automatically. BIBREF92 unifies the embeddings of concepts at different levels, including objects, attributes, relations and full scenes. BIBREF93 proposed a stacked cross attention network (SCAN) to learning fine-grained word and image-object aligned embedding for image-text matching. BIBREF47 employs a deep attentional multimodal similarity model (DAMSM) extending DMSM with attention models to measure the similarity between image sub-regions and words as an additional loss for text-to-image generation.
Representations ::: Multimodal Representations ::: Supervised training methods
Supervisions can be used to improve multimodal representation learning. BIBREF94 factorizes the representations into two sets of independent factors: multimodal discriminative factors for supervised training and intra-modality generative factors for unsupervised training. The discriminative factors are shared across all modalities and are useful for discriminative tasks, whereas the generative factors can be used to reconstruct missing modalities. With detailed text annotations, BIBREF95 proposed to learn word embeddings from their visual co-occurrences (ViCo) when applying to the same natural scene image or image region. ViCo is found to be complementary to the GloVe embedding by better representing similarities and differences between visual concepts that are difficult to obtain from text corpora alone. BIBREF96 applies multiple supervised training tasks to different layers of the vision-language encoder. The order of the training tasks is arranged following the idea of curriculum learning to increase the complexity of training objective step-by-step.
Representations ::: Multimodal Representations ::: Methods for zero-shot learning
Zero-shot learning often applies to vision related tasks due to the difficulty to acquire sufficient labelled images for training for all possible object categories. Not all types of multimodal representations are suitable for zero-shot learning since they may require pair-wised data from both modalities to present at the same time. Here we review methods that rely on extra language source to remedy this issue.
Deep learning based zero-shot learning started by training a linear mapping layer between different pre-trained embeddings BIBREF97, BIBREF98. The deep visual-semantic embedding (DeViSE) model is built upon Skip-Gram word embedding and AlexNet visual features and allows both pre-trained models to be jointly trained with the linear mapping layer BIBREF98. It achieved a large-scale test with 1000 seen classes and 2000 unseen classes. Better representations could be learnt when correlated autoencoders are used to reconstruct the representations for each modality, which improves one-shot and few-shot image retrieval performance comparing to DeViSE BIBREF99. Richer information source can be used for both modalities, including words selected from Wikipedia articles and features derived from multiple CNN layers BIBREF100. Rather than direct text attribute input, sentence embedding generated by recurrent models can be used as the text interface for zero-shot learning to achieve competitive results BIBREF101. Moving beyond empirical findings, recent study analyzed the properties of deep learning based cross-modal mapping using a similarity measure BIBREF102.
Representations ::: Multimodal Representations ::: Transformer based methods
Transformer is a prevalent sequence-based encoder-decoder model formed by stacking many blocks of feedforward layers with multi-head self-attention models BIBREF103. The parameters in all blocks shared across time similar to the time-delayed neural networks BIBREF104 and quasi-RNN BIBREF105 without an explicit temporal order. Compared with RNN based encoder-decoder models BIBREF26, it can have higher training efficiency due to the additional parallel degree across-time and superior performance on longer sequences benefited from the removing of first-order Markovian assumption imposed to the RNNs. BERT, the encoder part of Transformer pre-trained on a large text corpus as a masked LM, becomes a standard choice for word piece embeddings for downstream tasks, particularly since it utilizes both past and future information easily. It is natural to generalize the text-only BERT to cover images as well that can be used as the pre-trained multimodal embeddings.
A straightforward way to extend the unimodal BERT to bimodal, is to include new tokens to indicate the input of visual features, such as Unicoder-VL BIBREF106, VL-BERT BIBREF107, VisualBERT BIBREF108, VideoBERT BIBREF109, and B2T2 BIBREF110. LXMERT BIBREF111, ViLBERT BIBREF112, and OmniNet BIBREF113 modify the Transformer model by introducing an extra encoder or attention structures for visual features. More details about the modified structures can be found from Section SECREF18. Furthermore, recent NLP study found that multitask training can improve the generalization ability of the BERT representations BIBREF114. Most of the aforementioned bimodal BERT style models adopt multitask training to improve their performance on downstream tasks like VQA, image and video captioning etc. Although it would be useful to rigorously compare the performance of these models to understand the impact of different design choices, it is hard to do so since different amount of parameters and pre-training data are used across papers.
Fusion
Fusion is a key research problem in multimodal studies, which integrates information extracted from different unimodal data into one compact multimodal representation. There is a clear connection between fusion and multimodal representation. We classify an approach into the fusion category if its focus is the architectures for integrating unimodal representations for particular a task.
Traditionally, fusion methods are divided based on the stage it appears in the procedure. Early fusion, or feature-level fusion, directly combines the features extracted from each type of unimodal data to stress the intra-modality interactions and can cause the inter-modality interactions to be suppressed. Late fusion, on the other hand, refers to model-level fusion that builds a separate model for each modality and combines their output BIBREF115, BIBREF116, BIBREF117, BIBREF118, BIBREF119. The late fusion methods are strong in modelling intra-modality interactions with the modality-specific models but may suffer from the limited power of simple output value combination since the inter-modality interactions are rather complex. Recent studies focus on the intermediate or middle-level methods that allows fusion to happen at multiple layers of a deep model.
In this section, a review on intermediate fusion is focused – not only as it is more flexible, but also because the boundaries between stages are less clear due to the use of unimodal features derived from pre-trained backbone models. Three types of methods mostly used to fuse text with image features are included: simple operation-based, attention-based, as well as tensor-based methods.
Fusion ::: Simple Operation-based Fusion
In deep learning, vectorized features from different information sources can be integrated using a simple operation, such as concatenation or weighted sum, which often has only a few or even no parameter associated since the joint training of deep models can adapt the layers for high-level feature extractions to adjust for the required operation.
Concatenation can be used to combine either low-level input features BIBREF119, BIBREF120, BIBREF121 or high-level features extracted by the pre-trained models BIBREF121, BIBREF122, BIBREF123.
For weighted sum with scalar weights, an iterative method is proposed BIBREF124 that requires the pre-trained vector representations for each modality to have the same number of elements arranged in an order that is suitable for element-wise addition. This is often achieved by jointly training a fully connected layer for dimension control and reordering for each modality, together with the scalar weights for fusion.
A recent study BIBREF125 employs neural architecture search with progressive exploration BIBREF126, BIBREF127, BIBREF128 to find suitable settings for a number of fusion functions. Each fusion function is configured by which layers to fuse and whether to use concatenation or weighted sum as the fusion operation. Other weak functions can also be used to fuse multiple layers from each modality BIBREF129.
Fusion ::: Attention-based Fusion
Attention mechanism is widely used for fusion, which often refers to weighted sum a set of vectors using scalar weights dynamically generated by a small “attention” model at each time-step BIBREF129, BIBREF130. Multiple glimpses (output heads) are often used by the attention model to generate multiple sets of dynamic weights for the summation, whose resulted values can be concatenated to reserve more information. When applying attention mechanism to an image, image feature vectors relevant to different regions are weighted differently to produce an attended image vector.
Fusion ::: Attention-based Fusion ::: Image attention
BIBREF131 extends an LSTM model for text question processing with an image attention model conditioned on the previous LSTM hidden state, whose input is a concatenation of the current word embedding with the attended image feature. The final LSTM hidden state is regarded as the fused multimodal representation to predict the answer for pointing and grounded VQA. The attention model for sequence-based encoder-decoder model is used to attend to the image features for image captioning BIBREF132. Further for VQA, attention model conditioned on both image and query feature vectors is applied to pinpoint the image regions relevant to the answer BIBREF133. Similarly, stacked attention networks (SANs) are proposed to use multiple layers of attention models to query an image multiple times to infer the answer progressively by simulating a multi-step reasoning procedure BIBREF134. At each layer, a refined query vector is generated and send to the next layer by adding the previous query vector to the attended image vector produced using the current attention model. Spatial memory network (SMem) is a multi-hop method for VQA, which aligns words to image regions in the first hop and performs image attention based on the entire question in the second hop to derive the answer BIBREF135.
In BIBREF136, dynamic memory network (DMN) is augmented to use separate input modules to encode the question and image, which uses attention based GRUs to update episodic memory iteratively to retrieve the required information. Bottom-up and top-down attention method (Up-Down), as its name suggested, simulates human visual system using a combination of two visual attention mechanisms BIBREF137. The bottom-up attention mechanism proposes a set of salient image regions found by a Faster R-CNN, and the top-down attention mechanism uses a concatenation of visual and linguistic features to estimate the attention weights and produce the attended image feature vector for image captioning or VQA. The attended image feature vector can be fused with the linguistic feature again using an element-wise product. Complementary image features derived from different models, such as ResNet and Faster R-CNN, are used for multiple image attention mechanisms BIBREF138. Moreover, the reverse of image attention that generates attended text feature with image and text input is used for text-to-image synthesis in BIBREF47 and BIBREF139.
Fusion ::: Attention-based Fusion ::: Image and text co-attention
Different from the aforementioned image attention methods, co-attention mechanism has a symmetric attention structure to generate not only an attended image feature vector, but also an attended language vector BIBREF140. The parallel co-attention uses a joint representation to derive the image and language attention distributions simultaneously; alternating co-attention, on the other hand, has a cascade structure that first generates the attended image vector using the linguistic features, followed by the attended language vector generated using the attended image vector.
Similar to the parallel co-attention, dual attention network (DAN) estimates attention distributions for image and language simultaneously to derive their attended feature vectors BIBREF141. The attention models are conditioned on both feature and memory vectors of the relevant modality. This is a key difference to co-attention since the memory vectors can be iteratively updated at each reasoning step by repeating the DAN structure. The memory vectors can be either shared for VQA or modality-specific for image-text matching. Stacked latent attention (SLA) improves SAN by concatenating the original attended image vector with values from earlier layers of the attention model to retain the latent information from intermediate reasoning stages BIBREF142. A parallel co-attention like twin stream structure is also included to attend to both image and language features that also allows to reason iteratively using multiple SLA layers. Dual recurrent attention units (DRAU) implements the parallel co-attention structure with LSTM models for text and image to attend to each input location of the representations obtain by convolving image features with a stack of CNN layers BIBREF143. To model high-order interactions between modalities, high-order correlations between two data modalities can be computed as the inner product of two feature vectors and used to construct high-order attention models to derive the attended feature vectors for both modalities BIBREF144.
Fusion ::: Attention-based Fusion ::: Attention in bimodal Transformer
Recall Section SECREF14, the bimodal extensions to BERT rely on different tokens to indicate whether a vector is a word piece or an image, and the attention models fuse images with words in bimodal input sequences BIBREF106, BIBREF107, BIBREF108, BIBREF109, BIBREF110. OmniNet uses the gated multi-head attention model in each decoder block to fuse the vectors from the other modalities with that produced for the current modality by the previous layers in the the block BIBREF113. LXMERT uses independent encoders to learn the intra-modality features for each modality, and a cross-modality encoder sitting above them to learn the cross-modality features using extra cross-attention layers BIBREF111. ViLBERT extends BERT to include two encoder streams to process visual and textual inputs separately, which can interact through parallel co-attention layers BIBREF111.
Fusion ::: Attention-based Fusion ::: Other attention like mechanisms
Gated multimodal unit is a method that can be viewed as the attention of image and text based on gating BIBREF145. It performs weighted sum of visual and textual feature vectors based on dimension-specific scalar weights generated dynamically by the gating mechanism. Similarly, element-wise multiplication can be used to fuse visual and textual representations, which is used to create the building blocks of a multimodal residual network (MRN) based on deep residual learning BIBREF146. Dynamic parameter prediction network (DPPnet) uses a dynamic weight matrix to transform the visual feature vectors, whose parameters are dynamically generated by hashing the text feature vector BIBREF147.
Fusion ::: Bilinear Pooling-based Fusion
Bilinear pooling is a method often used to fuse a visual feature vector with a textual feature vector into a joint representation space by computing their outer product, which allows a multiplicative interaction between all elements in both vectors and is also termed as second order pooling BIBREF148. Comparing to simple vector combination operations (assuming each vector has $n$ elements), such as weighted sum, element-wise multiplication, or concatenation that result in $n$ or $2n$ dimensional representations, bilinear pooling leads to an $n^2$ dimensional representation by linearizing the outer product resulted matrix into a vector and is therefore more expressive. The bilinear representation is often linearly transformed into an output vector using a two-dimensional weight matrix, which is equivalent to use a three-dimensional tensor operator to fuse the two input feature vectors. Each feature vector can be extended with an extra value one to reserve input single modal features in the bilinear representation via outer product BIBREF149. However, given its high dimensionality, typically on the order of hundreds of thousands to a few million, bilinear pooling often requires decomposing the weight tensor to have the associated model to be trained properly and efficiently.
Fusion ::: Bilinear Pooling-based Fusion ::: Factorization for bilinear pooling
Since bilinear representations are found to be closely related to the polynomial kernels, different low-dimensional approximations can be used to acquire compact bilinear representations BIBREF150. Count Sketches and convolutions can be used to approximate the polynomial kernels BIBREF151, BIBREF152 that leads to the multimodal compact bilinear pooling (MCB) method BIBREF153. Alternatively, by enforcing a low rank to the weight tensor, multimodal low-rank bilinear pooling (MLB) factorizes the three-dimensional weight tensor for bilinear pooling into three two-dimensional weight matrices BIBREF154. More precisely, the visual and textual feature vectors are linearly projected to low-dimensional modality-specific factors by the two input factor matrices, which are then fused using element-wise multiplication followed by a linear projection with the third matrix, the output factor matrix. Multimodal factorized bilinear pooling (MFB) modifies MLB by using an extra operation to pool the element-wise multiplication results by summing the values within each non-overlapped one-dimensional window BIBREF155. Multiple MFB models can be cascaded to model high-order interactions between input features and is called multi-modal factorized high-order pooling (MFH) BIBREF156.
MUTAN, a multimodal tensor-based Tucker decomposition method, uses Tucker decomposition BIBREF157 to factorize the original three-dimensional weight tensor operator with a small-dimensional core tensor and the three two-dimensional weight matrices used by MLB BIBREF158. The core tensor models the interactions across modalities. Comparing to MUTAN, MCB can be seen as MUTAN with fixed diagonal input factor matrices and a sparse fixed core tensor, while MLB is MUTAN with the core tensor set to identity. Recently, BLOCK, a block superdiagonal fusion framework is proposed to use block-term decomposition BIBREF159 to compute bilinear pooling BIBREF160. BLOCK generalizes MUTAN as a summation of multiple MUTAN models to provide a richer modeling of interactions between modalities. The MUTAN core tensors can be arranged as a superdiagonal tensor, similar to the submatrices of a block diagonal matrix. Furthermore, bilinear pooling can be generalized to more than two modalities, such as BIBREF149 and BIBREF161 that use outer products to model the interactions among the representations for video, audio, and language.
Fusion ::: Bilinear Pooling-based Fusion ::: Bilinear pooling and attention mechanism
Bilinear pooling can be used along with attention mechanism. MCB/MLB fused bimodal representation can be used as the input feature of an attention model to derive the attended image feature vector, which is fused with the textual feature vector using MCB/MLB again to form the final joint representation BIBREF153, BIBREF154. MFB/MFH can be used for alternating co-attention to learn the join representation BIBREF155, BIBREF156. Bilinear attention network (BAN) uses MLB to fuse image and text to produce a bilinear attention map as the attention distributions, which is used as the weight tensor for bilinear pooling to fuse the image and text features again BIBREF162.
Applications
In this section, selected applications for multimodal intelligence that combine vision and language are discussed, which include image captioning, text-to-image generation, and VQA. It is worth noting that there are other applications, such as text-based image retrieval BIBREF93, BIBREF163, BIBREF164, and visual-and-language navigation (VLN) BIBREF165, BIBREF166, BIBREF167, BIBREF168, BIBREF169, BIBREF170, BIBREF171, BIBREF172, BIBREF173, that we have not included in this paper due to space limitation.
Caption generation is a task that aims to automatically generate a natural language description of an image. It requires a level of image understanding beyond normal image recognition and object detection.
A reverse of caption generation is text-to-image synthesis, which often generates image pixels according to a description sentence or some key words provided by human.
VQA is related to caption generation, which often takes an image as the input and a free-form, open-ended natural language question about the image, to output a classification result as the output of the answer. Natural language understanding is required as the questions are in free form. Other capabilities such as knowledge based reasoning and common-sense reasoning can be important since the questions are open-ended.
Visual reasoning can be included in all of the aforementioned tasks. Visual reasoning methods for VQA are reviewed in the end.
Detailed task specifications, data sets, and selected work for each task will be introduced in this section.
Applications ::: Caption Generation
Image captioning BIBREF174 requires to generate a description of an image and is one of the earliest task that studies multi-modal combination of image and text. We mainly review the deep learning based methods for caption generation. Image captioning, such as BIBREF85, BIBREF175, BIBREF39, divide the task into several sub-tasks and generate caption in a step-by-step manner. Authors in BIBREF85 first trained a deep CNN model to detect the words from images, then built a log-linear language model to compose the words into sentences. Similarly, BIBREF175 fed the image feature into a log-linear language model to generate sentences. In contrast, BIBREF39 tried to find the exact matching of objects in images and words in sentences to determine if an image and a sentence match with each other.
Similar to the RNN-based encoder-decoder methods for machine translation BIBREF26, BIBREF176, BIBREF177, BIBREF178 propose to generate captions from images in an end-to-end manner via the encoder-decoder architecture. In those models, a CNN, typically pre-trained on ImageNet BIBREF69, encoded the image into a continuous vector, which is then fed into a RNN/LSTM decoder to generate the caption directly. Those works all followed the same architecture, but varied slightly the choice of CNN architecture and how the image vector was fed into the decoder. Though powerful and convenient, the encoder-decoder architecture lacks to ability capture the fine grained relationship between objects in images and words in sentences. To overcome this, attention-based encoder-decoder model BIBREF179 was proposed and has become the standard benchmark for this task since then. In the attention encoder-decoder model, before generating the next word, the decoder first calculates the matching score (attention) with objects in the image, then conditions on the weighted image feature to generate the next token. There has been lots of work that tried to improve the attention model by incorporating more structures. For example, BIBREF180 added a gate at every decoding step to determine if the next word should be generated using image information; BIBREF181 combined detected words and image features as inputs to the decoder network. More recently, there has been a lot works that add more structure/knowledge from either image BIBREF137 or text side BIBREF182. Specifically, BIBREF137 used an object detector to localize the features for image object and then generates the caption based on the localized features. It improved the previous state of art model by a large margin in a variety of evaluation metrics.
Image captions with richer information could be generated when incorporated with external knowledge. For example, based on an model that can recognize celebrities BIBREF183, a CaptionBot app is developed which can not only describe the facts (such as activities) in a picture, but also describe who is doing that if the person in the picture is recognized BIBREF184. Further, beside simply generating a factual description of the image, other approaches were also proposed for explicitly controlling the style BIBREF185, semantic content BIBREF181, and diversity BIBREF186 of the generated caption.
Applications ::: Text-to-Image Synthesis
Text-to-image synthesis or generation that relies on natural language to control image generation, is a fundamental problem in computer vision. It is considered as a difficult problem since it least involves two tasks: high quality image generation and language understanding. The generated image is required to be both visually realistic and semantically consistent to the language description. Deep learning based text-to-image synthesis can perhaps be dated back to the use of LSTM for iterative hand-writting generation BIBREF187. This iterative image generation idea is later extended to form the deep recurrent attentive writer (DRAW) method that combines an LSTM based sequential variational auto-encoder (VAE) with a spatial attention mechanism BIBREF188. alignDRAW modifies DRAW to use natural language based descriptions to synthesis images with general content BIBREF189. An attention model is used to compute the alignment between the input words and the patches drawn iteratively.
Applications ::: Text-to-Image Synthesis ::: GAN based methods
Comparing to VAE, conditional-GAN (CGAN) is found to be able to synthesis highly compelling images of specific categories that a human might mistake for real BIBREF190, BIBREF191. A GAN model consists of a generator that synthesize candidates based on input noises and a discriminator that evaluates them. Adversarial training is employed to make the generator to capture the true data distribution so that the discriminator can no longer discriminate the synthesized data from the real ones BIBREF80. CGAN extends the standard GAN structure by conditioning on extra category labels for both generator and discriminator. GAN-INT-CLS allows to synthesize visually plausible 64$\times $64 images using the embeddings of natural language descriptions to replace the category labels in CGAN BIBREF192. The automatic evaluation of the quality of text conditioned images can be less straightforward. To find the discriminability of GAN generated images, inception score (IS) BIBREF193 and Fréchet inception distance BIBREF194 (FID) are often used. Multi-scale structural similarity (MS-SSIM) BIBREF195 is commonly used to evaluate the diversity of images. To evaluate whether a generated image is semantically consistent with the input text description, R-precision BIBREF47 and visual-semantic similarity BIBREF196 are used as the metrics.
Applications ::: Text-to-Image Synthesis ::: Generating high quality images
Though basically reflecting the meaning of the descriptions, it is found the images produced by GAN-INT-CLS do not have necessary details and vivid object parts, and therefore leads to the StackGAN method BIBREF197. StackGAN decomposes image synthesize into more manageable sub-problems through a sketch-refinement process by stacking two CGANs trained separately. The first GAN produces 64$\times $64 low-resolution images by sketching the primitive shape and colors of the object based on the text, and the second GAN is trained after to generate 256$\times $256 images by rectifying defects and adding compelling details in the low-resolution image. StackGAN$++$ improves this idea by adding an extra GAN to generate 128$\times $128 images in between and training all GANs jointlyBIBREF198. To ensure the generated image semantically match the text precisely, BIBREF47 proposed attentional GAN (AttnGAN), which also stacks three GANs for different image resolutions BIBREF47, and while the first GAN is conditioned on the sentence embedding, the next two GANs are conditioned on bimodal embeddings produced by attention models fusing word-level features with low-resolution images. It is shown attention mechanism can help GAN to focus on words that are most relevant to the sub-region drawn at each stage. Apart from stacking the generators, BIBREF199 shows that high resolution images can also be generated with a dynamic memory module.
Applications ::: Text-to-Image Synthesis ::: Generating semantically consistent images
To improve the semantic consistency between relevant image and text features, DAMSM is proposed for AttnGAN BIBREF47. BIBREF196 tackles the same problem by leveraging hierarchical representations with extra adversarial constraints to discriminate not only real/fake image pairs, but also real/fake image-text pairs at multiple image resolutions in the discriminator, and is named as hierarchically-nested discriminator GAN (HDGAN). Similarly, text conditioned auxiliary classifier GAN (TAC-GAN) introduces an extra image classification task to the discriminator BIBREF200, whereas text-conditioned semantic classifier GAN (Text-SeGAN) alternates the classifier with a regression task to estimate the semantic relevance between image and text BIBREF201. Analogous to cycle consistency BIBREF202, MirrorGAN is proposed to improve the semantic consistency between the two modalities using an extra image captioning module BIBREF203.
Applications ::: Text-to-Image Synthesis ::: Semantic layout control for complex scenes
With the success in the generation of realistic and semantically consistent images for single objects, such as birds BIBREF204 or flowers BIBREF205, state-of-the-art text-to-image synthesis methods still struggle to generate complex scenes with many objects and relationships, such as those in the Microsoft COCO data set BIBREF206. In the pioneering work BIBREF207, not only text descriptions but also locations of objects specified by keypoints or bounding boxes are used as the input. Later, detailed semantic layout, such as a scene graph, is used to replace the natural language sentence as a more direct description of objects and their relationships BIBREF208, BIBREF209, BIBREF210. Meanwhile, efforts are made to keep natural language input while incorporating the idea of semantic layout. BIBREF211 includes extra object pathway to both generator and discriminator to explicit control the object locations. BIBREF212 employs a two-stage procedure that first builds a semantic layout automatically from the input sentence with LSTM based box and shape generators, and then synthesizes the image using image generator and discriminators. Since fine-grained word/object level information is not explicitly used for generation, such synthesized images do not contain enough details to make them look realistic. The object-driven attentive GAN (Obj-GAN) improves the two-stage generation idea using a pair of object-driven attentive image generator and object-wise discriminator BIBREF139. At every generation step, the generator uses the text description as a semantic layout and synthesizes the image region within a bounding box by focusing on the words that are most relevant to the object in it. ObjGAN is found to be more robust and interpretable, and significantly improves the object generation quality for complex scenes.
Applications ::: Text-to-Image Synthesis ::: Other topics
In addition to the layout, other types of fine-grained control in image generation have also been studied in literature. Attribute2Image BIBREF213 studies the use of attributes in face generation, such as age and gender etc. BIBREF214 uses the same idea for face editing, such as to remove the beard or change the hair color. Text-adaptive GAN BIBREF215 allows semantic modification of input images for birds and flowers via natural language. BIBREF216 enforces to learn the representation content and style as two disentangled variables using a dual inference mechanism based on cycle-consistency for text-to-image synthesis. The success of these methods validate GAN is able to learn some semantic concepts as disentangled representations, as in Section SECREF9. Text2Scene is another interesting work that generates compositional scene representation from natural langauge step-by-step without using GANs BIBREF217. It is shown with minor modifications, Text2Scene can generate cartoon like, semantic layout, and real image like scenes. Dialogue based interaction is studied to control image synthesis, in order to improve complex scene generation progressively BIBREF218, BIBREF219, BIBREF220, BIBREF221, BIBREF222. Meanwhile, text-to-image synthesis is extended to multiple images or videos, where visual consistency is required among the generated images BIBREF223, BIBREF224, BIBREF225.
Applications ::: Visual Question Answering ::: Task definition
VQA extends text-based QA from NLP by asking questions related to the visual information presented in an image or a video clip. For image based VQA, it is often considered as a visual Turing test, in which the system is required to understand any form of natural language-based questions and to answer them in a natural way. However, it is often simplified as a classification task defined in different ways to focus on the core problem BIBREF131, BIBREF226, BIBREF227, BIBREF42, BIBREF43. Initial works generated the questions using templates or by converting from description sentences using syntax trees BIBREF226, BIBREF228. Later later studies focus on the use of free-form natural language questions authored either by human or powerful deep generative models, such as GAN and VAE BIBREF43, BIBREF228, BIBREF229, BIBREF230. Different from the open-ended questions presented in complete sentence form, possible answers are often presented as a large set of classes (e.g. 3000) related to yes/no, counts, object classes and instances etc. To focus on the core understanding and reasoning problems, VQA can be simplified as to classify visual and textual features into the answer related classes.
Alternatively, VQA can be defined as to select among multiple (e.g. 4) choices, and each choice is associated with each answer presented in the form of a natural language sentence BIBREF131. This setup can be implemented as a classification to the choices based on features of the image, question, and answer candidates BIBREF153. There exist other types of VQA task definitions, such as the Visual Madlibs dataset that requires to answer the questions by “fill-in-the-blanks” BIBREF44. Furthermore, visual dialogue can be viewed as to answer a sequence of questions grounded in an image BIBREF231, BIBREF232, which extends VQA by requiring to generate more human like responses and to infer the context from the dialogue history.
Applications ::: Visual Question Answering ::: Common data sets and approaches
The first VQA data set, DAQUAR, uses real-world images with both template-based and human annotated questions BIBREF226. COCO-QA has more QA pairs than DAQUAR by converting image descriptions from the MS COCO data set into questions BIBREF228. Such questions are in general easier since they allow the model to rely more on the rough image rather than logical reasoning. VQA v1 and v2 are the most popular data sets for VQA consisting of open-ended questions and both real and abstract scenes BIBREF43, BIBREF233. A VQA Challenge based on these data sets is held annually as a CVPR workshop since 2016. Visual7W is a part of the Visual Genome data set for VQA with multiple choices BIBREF131. It contains questions related to “what”, “who”, and “how” for spatial reasoning, and “where”, “when”, and “why” questions for high-level common-sense reasoning. The 7th type of the questions in Visual7W are the “which” questions, which are also termed as the pointing questions, whose answer choices are associated with bounding boxes of objects in the image. Approaches designed for these data sets often focus on fusing image and question vectors with the previously discussed attention- and bilinear pooling-based methods, such as SAN, co-attention, Up-Down, MCB, MLB, and BAN etc.
Applications ::: Visual Question Answering ::: Integrating external knowledge source
Since most of the VQA questions in these data sets are about simple counting, colors, and object detections that do not need any external knowledge to answer, a further development of the task is to include more difficult questions that require knowing more than what the questions entail or what information is contained in the images. Both knowledge-based reasoning for VQA (KB-VQA) and fact-based VQA (FVQA) data sets incorporate structured knowledge base, which often requires extra steps to query the knowledge base that makes the method no longer an end-to-end trainable approach BIBREF234, BIBREF235. Different from the structured knowledge bases, outside knowledge VQA (OK-VQA) uses external knowledge in the form of natural language sentences collected by retrieving Wikipedia articles with search queries extracted from the question, and an extra ArticleNet model is trained to find the answers from the retrieved articles BIBREF236.
Applications ::: Visual Question Answering ::: Discounting language priors
Though significant achievements have been made, recent studies point out that the common VQA benchmarks suffer from strong and prevalent priors – most bananas are yellow and mostly the sky is blue, which can often cause the VQA model to over-fit to these statistical biases and tendencies from the answer distributions, and largely circumvent the need to understand the visual scenes. Based on the objects, attributes, and relations provided through the scene graphs from Visual Genome, a new data set, GQA, was created to greatly reduce such biases by generating questions with a functional program that controls the reasoning steps BIBREF237. New splits for VQA v1 and VQA v2 are generated to have different answer distributions for every question of the training and test sets, which are referred to as VQA under challenging priors (VQA-CP v1 and VQA-CP v2) BIBREF238. Recent methods propose to handle the biased priors with adversarial training or additional train only structures BIBREF239, BIBREF240.
Applications ::: Visual Question Answering ::: Other issues
Another problem that current VQA methods suffers from is the low robustness against linguistic variations from the questions. A data set, VQA-Rephrasings, modified the VQA v2.0 validation set with human authored rephrasing of the questions BIBREF202. A cycle-consistency BIBREF241 based method that improves the linguistic robustness by enforcing consistencies between the original and rephrased questions, and between the true answer and the answers predicted based on the original and rephrased questions. BIBREF242 suggests that attention mechanism can cause VQA models to suffer from counting the object proposals, and an extra model component was proposed as a solution. Moreover, it is the fact that the current VQA methods cannot even read text from images. A method is proposed to address this problem by fusing not text extracted from the image using optical character recognition BIBREF243. VizWiz is a goal oriented VQA data set collected by blind people taking possibly low quality pictures and asking questions in spoken English, which also include many text related questions BIBREF244.
Applications ::: Visual Reasoning
This section focuses on the study of a very interesting problem – visual reasoning, which is about how to conduct accurate, explicit, and expressive understanding and reasoning. Visual reasoning can involved by many language and vision based bimodal tasks, such as caption generation and text-to-image synthesis. However, in this section we mostly focus on the related methods for VQA as visual reasoning is particularly important when answering complicated questions. SAN is often considered as a pioneering work related to implicit visual reasoning since to its stacked structure can be viewed as to perform multiple reasoning steps. Shortly afterwards, feature-wise linear modulation (FiLM) is proposed to refine visual features iteratively using feature-wise affine transforms based on the scaling factors and bias values generated dynamically from the textual features BIBREF245. Multimodal relational network (MuRel) also has a structure with multiple MuRel cells based on bilinear pooling, which can be used iteratively BIBREF246.
Applications ::: Visual Reasoning ::: Neural module network based methods
Neural module network (NMN) is a method which composes a collection of jointly trained neural “modules” into a deep model for answering the question BIBREF247. A dependency parser first helps to convert the natural language question into a fixed and rule-based network layout, and specify both the set of modules used to answer the question and the connections between them. Then a deep model is assembled based on the layout to produce the prediction of the answers. SHAPES, a synthetic dataset consists of complex questions about simple arranges of ordered shapes, was also proposed to focus on the compositional phenomena of questions BIBREF247. A later study learns the model layout predictor jointly with the parameters of the modules by re-ranking a list of layout candidates using reinforcement learning, which is termed as dynamic NMN (D-NMN) BIBREF248. Modules such as “find” or “relate” operation uses attention models to focus on one or two regions of the input image and makes the forwarding of the assembled deep model similar to running a functional program BIBREF248. An end-to-end version of NMN (N2NMN) used an RNN question encoder to convert the input question into a layout policy without requiring the aid of a parser BIBREF249. The work is based on a more recent data set called compositional language and elementary visual reasoning diagnostics (CLEVR). As its name suggests, CLEVR is a synthetic diagnostic data set testing a range of visual reasoning abilities of objections and relationships with minimal biases and detailed annotations describing the kind of reasoning each question requires BIBREF250. Other implementations of NMN include the program generator and execution engine method (PG+EE) that shares generic design among some operations BIBREF251, stack-NMN that improves the parser and incorporates question features into the modules BIBREF252, and transparency by design network (TbD-net) that redesigns some modules from PG+EE to maintain the transparency of the reasoning procedure BIBREF253.
Applications ::: Visual Reasoning ::: Other types of end-to-end reasoning methods
Another end-to-end approach is the memory, attention, and composition (MAC) network that decomposes the question into a series of attention-based reasoning steps and perform each of them using a recurrent MAC cell that maintains a separation between the control and memory hidden states. Each hidden state is generated by an ANN model constructed based on attention and gating mechanisms BIBREF254. More recently, both deterministic symbolic programs and probabilistic symbolic models have been used as the execution engine for the generated programs to improve the transparency and data efficiency, which result in the neural-symbolic VQA (NS-VQA) and probabilistic neural-symbolic models (prob-NMN) respectively BIBREF255, BIBREF256. As an extension of NS-VQA, the neuro-symbolic concept learner (NS-CL) uses a neuro-symbolic reasoning module to execute programs on the scene representation. NS-CL can have its program generator, reasoning module, and visual perception components jointly trained in an end-to-end fashion without requiring any component-level supervisions BIBREF257. Its perception module learns visual concepts based on the language descriptions of the objects and facilitates learning new words and parsing new sentences.
We finish this section by reviewing the relation networks (RN), which has a simple structure that uses an ANN as the function to model the relationship between any pair of visual and textual features, and the resulted output values are accumulated and transformed by another ANN BIBREF258. Though RN merely models the relationship without any form of induction reasoning, it achieves very high VQA accuracy on CLEVR. This inspires a re-thinking of the connection between correlation and induction.
Conclusion
This paper reviews the area of modeling and machine learning across multiple modalities based on deep learning, particularly the combination of vision and natural language. In particular, we propose to organize the many pieces of work in the language-vision multimodal intelligence field from three aspects, which include multimodal representations, the fusion of multimodal signals, and the applications of multimodal intelligence. In the section of representations, both single modal and multimodal representations are reviewed under the key concept of embedding. The multimodal representation unifies the involved signals of different modalities into the same vector space for general downstream tasks. On multimodal fusion, special architectures, such as attention mechanism and bilinear pooling, are discussed. In the application section, three selected areas of broad interest are presented, which include image caption generation, text-to-image synthesis, and visual question answering. A set of visual reasoning methods for VQA is also discussed. Our review covers task definition, data set specification, development of commonly used methods, as well as issues and trends, and therefore can facilitate future studies in this emerging field of multimodal intelligence for our community.
Chao Zhang is an advisor of JD.com speech team, and a research associate in speech and natural language processing at the University of Cambridge. He received his B.E. and M.S. degrees in 2009 and 2012 respectively, both from the Department of Computer Science & Technology, Tsinghua University, and a Ph.D. degree in 2017 from Cambridge University Engineering Department.
Zichao Yang is a quantitative researcher at Citadel. Prior to that, he received his Phd in computer science from Carnegie Mellon University. His research interests are in machine learning, deep learning and their applications in natural language processing and computer vision. He has published dozens of papers in NeurIPS, ICML, CVPR, ICCV, EMNLP, NAACL etc.
Xiaodong He (IEEE Member 2003, Senior member 2008, Fellow 2019) is the Deputy Managing Director of JD AI Research, and Head of the Deep learning, NLP and Speech Lab. He is also Affiliate Professor of ECE at the University of Washington (Seattle). His research interests are mainly in deep learning, natural language processing, speech recognition, computer vision, information retrieval, and multimodal intelligence. He has held editorial positions on multiple IEEE Journals and the Transactions of the ACL, and served in the organizing committee/program committee of major speech and language processing conferences. He is a member of the IEEE SLTC for the term of 2015-2017 and the Chair of the IEEE Seattle Section in 2016. He received the Bachelor degree from Tsinghua University in 1996, MS degree from Chinese Academy of Sciences in 1999, and the PhD degree from the University of Missouri – Columbia in 2003.
Li Deng has been the Chief Artificial Intelligence Officer of Citadel since May 2017. Prior to Citadel, he was the Chief Scientist of AI, the founder of the Deep Learning Technology Center, and Partner Research Manager at Microsoft and Microsoft Research , Redmond (2000-2017). Prior to Microsoft, he was an assistant professor (1989-1992), tenured associate (1992-1996), and full professor (1996-1999) at the University of Waterloo in Ontario, Canada. He also held faculty or research positions at Massachusetts Institute of Technology (Cambridge, 1992-1993), Advanced Telecommunications Research Institute (ATR, Kyoto, Japan, 1997-1998), and HK University of Science and Technology (Hong Kong, 1995). He is a Fellow of the Academy of Engineering of Canada, a Fellow of the Washington State Academy of Sciences, a Fellow of the IEEE, a Fellow of the Acoustical Society of America, and a Fellow of the International Speech Communication Association. He has also been an Affiliate Professor at the University of Washington, Seattle.
He was an elected member of the Board of Governors of the IEEE Signal Processing Society, and was Editors-in-Chief of IEEE Signal Processing Magazine and of IEEE/ACM Transactions on Audio, Speech, and Language Processing (2008-2014), for which he received the IEEE SPS Meritorious Service Award. In recognition of the pioneering work on disrupting speech recognition industry using large-scale deep learning, he received the 2015 IEEE SPS Technical Achievement Award for “Outstanding Contributions to Automatic Speech Recognition and to Deep Learning”. He also received dozens of best paper and patent awards for the contributions to artificial intelligence, machine learning, information retrieval, multimedia signal processing, speech processing and recognition, and human language technology. He is an author or co-author of six technical books on deep learning, speech processing, pattern recognition and machine learning, and, the latest, natural language processing (Springer, June 2018).

Question: What are special architectures this review focuses on that are related to multimodal fusion?
Answer: attention mechanism, bilinear pooling

attention mechanism, bilinear pooling