On multimodal fusion, this review focuses on special architectures for the integration of the representation of unimodal signals for a particular task. On applications, selected areas of a broad interest in current literature are covered, including image-to-text caption generation, text-to-image generation, and visual question answering. We believe this review can facilitate future studies in the emerging field of multimodal intelligence for the community.

Conclusion	This paper reviews the area of modeling and machine learning across multiple modalities based on deep learning, particularly the combination of vision and natural language. In particular, we propose to organize the many pieces of work in the language-vision multimodal intelligence field from three aspects, which include multimodal representations, the fusion of multimodal signals, and the applications of multimodal intelligence. In the section of representations, both single modal and multimodal representations are reviewed under the key concept of embedding. The multimodal representation unifies the involved signals of different modalities into the same vector space for general downstream tasks. On multimodal fusion, special architectures, such as attention mechanism and bilinear pooling, are discussed. In the application section, three selected areas of broad interest are presented, which include image caption generation, text-to-image synthesis, and visual question answering. A set of visual reasoning methods for VQA is also discussed. Our review covers task definition, data set specification, development of commonly used methods, as well as issues and trends, and therefore can facilitate future studies in this emerging field of multimodal intelligence for our community.

Fusion	Fusion is a key research problem in multimodal studies, which integrates information extracted from different unimodal data into one compact multimodal representation. There is a clear connection between fusion and multimodal representation. We classify an approach into the fusion category if its focus is the architectures for integrating unimodal representations for particular a task. Traditionally, fusion methods are divided based on the stage it appears in the procedure. Early fusion, or feature-level fusion, directly combines the features extracted from each type of unimodal data to stress the intra-modality interactions and can cause the inter-modality interactions to be suppressed. Late fusion, on the other hand, refers to model-level fusion that builds a separate model for each modality and combines their output , , , , . The late fusion methods are strong in modelling intra-modality interactions with the modality-specific models but may suffer from the limited power of simple output value combination since the inter-modality interactions are rather complex. Recent studies focus on the intermediate or middle-level methods that allows fusion to happen at multiple layers of a deep model.

Multimodal Intelligence: Representation Learning, Information Fusion, and Applications	Deep learning has revolutionized speech recognition, image recognition, and natural language processing since 2010, each involving a single modality in the input signal. However, many applications in artificial intelligence involve more than one modality. It is therefore of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, a technical review of the models and learning methods for multimodal intelligence is provided. The main focus is the combination of vision and natural language, which has become an important area in both computer vision and natural language processing research communities. This review provides a comprehensive analysis of recent work on multimodal deep learning from three angles --- learning multimodal representations, the fusion of multimodal signals at various levels, and multimodal applications. On multimodal representation learning, we review the key concept of embedding, which unifies the multimodal signals into the same vector space and thus enables cross-modality signal processing. We also review the properties of the many types of embedding constructed and learned for general downstream tasks.

For multimodal tasks, collecting paralleled data across all modalities can be quite difficult and leveraging pre-trained representations with desired properties, such as suitable for zero-shot or few-shot learning, is often an effective solution to the issue. Both supervised and unsupervised training based multimodal representation learning methods are reviewed. The fusion of the features or representations of the single modalities is undoubtedly a centric problem of any multimodal task. Different from previous studies that often categorise the related work into early, middle and late stage methods based on the stage that fusion happens in the procedure, we classify them according to the actual operation used in the fusion, such as attention and bilinear pooling, since it becomes difficult to classify some recent complex approaches into stages. Three types of applications are reviewed in this paper, namely image captioning, text-to-image synthesis and VQA. This is to give an idea how representation learning and fusion can be applied to specific tasks, and to provide a viewpoint of the situation of the current development of the multimodal applications, especially those integrating vision with natural languages. Visual reasoning methods for VQA are also discussed in the end. This paper is organised as follows.

In these tasks, natural language plays a key role in helping the machine to “understand” the content of the images, where “understand” means to capture the underlying correlations between the semantics embedded in language with the visual features obtained from the images. In addition to text, vision can be combined with speech as well. Such tasks include audio-visual speech recognition , , , speaker recognition , , , as well as speech diarisation , , separation ,  and enhancement , which mostly focused on the use of visual features to improve the robustness of the audio-only methods. In this paper, a technical review of the models and learning methods for multimodal intelligence is provided. The main focus is the combination of CV and NLP, which has become an important area for both research communities covering many different tasks and technologies. To provide a more structured perspective, we organize the methods selected in this technical review according to three key topics: representation, fusion, and applications. Learning representations for the input data is a core problem for deep learning.

In this section, a review on intermediate fusion is focused – not only as it is more flexible, but also because the boundaries between stages are less clear due to the use of unimodal features derived from pre-trained backbone models. Three types of methods mostly used to fuse text with image features are included: simple operation-based, attention-based, as well as tensor-based methods.

Applications ::: Visual Reasoning	This section focuses on the study of a very interesting problem – visual reasoning, which is about how to conduct accurate, explicit, and expressive understanding and reasoning. Visual reasoning can involved by many language and vision based bimodal tasks, such as caption generation and text-to-image synthesis. However, in this section we mostly focus on the related methods for VQA as visual reasoning is particularly important when answering complicated questions. SAN is often considered as a pioneering work related to implicit visual reasoning since to its stacked structure can be viewed as to perform multiple reasoning steps. Shortly afterwards, feature-wise linear modulation (FiLM) is proposed to refine visual features iteratively using feature-wise affine transforms based on the scaling factors and bias values generated dynamically from the textual features . Multimodal relational network (MuRel) also has a structure with multiple MuRel cells based on bilinear pooling, which can be used iteratively .

Representations	Deep learning, as a special area in representational learning, studies the use of artificial neural networks (ANNs) with many hidden layers to automatically discover the representations or features suitable for specific tasks from the raw data . In practice, it is often found that better representations can simplify the subsequent learning tasks and therefore has a great value. Over the past decade, it becomes feasible to learn effective and robust representations for single modalities, such as text , , , , , , , , , ,  and image , , , , , , , due to the availability of large data and development of deep learning. For multimodal representations, though attracting more and more attentions, it still remains a challenging problem due to the complex cross-modal interactions and possible mismatch between the training and test data of each modal. In this section, commonly used types of single modal representations, such as text and image, are first reviewed which often serve as cornerstones for learning multimodal representations. Afterwards, both supervised and unsupervised methods for learning a joint representation space for multiple modalities are introduced.

Applications ::: Visual Question Answering ::: Task definition	VQA extends text-based QA from NLP by asking questions related to the visual information presented in an image or a video clip. For image based VQA, it is often considered as a visual Turing test, in which the system is required to understand any form of natural language-based questions and to answer them in a natural way. However, it is often simplified as a classification task defined in different ways to focus on the core problem , , , , . Initial works generated the questions using templates or by converting from description sentences using syntax trees , . Later later studies focus on the use of free-form natural language questions authored either by human or powerful deep generative models, such as GAN and VAE , , , . Different from the open-ended questions presented in complete sentence form, possible answers are often presented as a large set of classes (e.g. 3000) related to yes/no, counts, object classes and instances etc. To focus on the core understanding and reasoning problems, VQA can be simplified as to classify visual and textual features into the answer related classes. Alternatively, VQA can be defined as to select among multiple (e.g.

Fusion ::: Bilinear Pooling-based Fusion ::: Factorization for bilinear pooling	Since bilinear representations are found to be closely related to the polynomial kernels, different low-dimensional approximations can be used to acquire compact bilinear representations . Count Sketches and convolutions can be used to approximate the polynomial kernels ,  that leads to the multimodal compact bilinear pooling (MCB) method . Alternatively, by enforcing a low rank to the weight tensor, multimodal low-rank bilinear pooling (MLB) factorizes the three-dimensional weight tensor for bilinear pooling into three two-dimensional weight matrices . More precisely, the visual and textual feature vectors are linearly projected to low-dimensional modality-specific factors by the two input factor matrices, which are then fused using element-wise multiplication followed by a linear projection with the third matrix, the output factor matrix. Multimodal factorized bilinear pooling (MFB) modifies MLB by using an extra operation to pool the element-wise multiplication results by summing the values within each non-overlapped one-dimensional window . Multiple MFB models can be cascaded to model high-order interactions between input features and is called multi-modal factorized high-order pooling (MFH) .

Applications ::: Visual Question Answering ::: Integrating external knowledge source	Since most of the VQA questions in these data sets are about simple counting, colors, and object detections that do not need any external knowledge to answer, a further development of the task is to include more difficult questions that require knowing more than what the questions entail or what information is contained in the images. Both knowledge-based reasoning for VQA (KB-VQA) and fact-based VQA (FVQA) data sets incorporate structured knowledge base, which often requires extra steps to query the knowledge base that makes the method no longer an end-to-end trainable approach , . Different from the structured knowledge bases, outside knowledge VQA (OK-VQA) uses external knowledge in the form of natural language sentences collected by retrieving Wikipedia articles with search queries extracted from the question, and an extra ArticleNet model is trained to find the answers from the retrieved articles .

Applications ::: Visual Question Answering ::: Common data sets and approaches	The first VQA data set, DAQUAR, uses real-world images with both template-based and human annotated questions . COCO-QA has more QA pairs than DAQUAR by converting image descriptions from the MS COCO data set into questions . Such questions are in general easier since they allow the model to rely more on the rough image rather than logical reasoning. VQA v1 and v2 are the most popular data sets for VQA consisting of open-ended questions and both real and abstract scenes , . A VQA Challenge based on these data sets is held annually as a CVPR workshop since 2016. Visual7W is a part of the Visual Genome data set for VQA with multiple choices . It contains questions related to “what”, “who”, and “how” for spatial reasoning, and “where”, “when”, and “why” questions for high-level common-sense reasoning. The 7th type of the questions in Visual7W are the “which” questions, which are also termed as the pointing questions, whose answer choices are associated with bounding boxes of objects in the image.

Fusion ::: Simple Operation-based Fusion	In deep learning, vectorized features from different information sources can be integrated using a simple operation, such as concatenation or weighted sum, which often has only a few or even no parameter associated since the joint training of deep models can adapt the layers for high-level feature extractions to adjust for the required operation. Concatenation can be used to combine either low-level input features , ,  or high-level features extracted by the pre-trained models , , . For weighted sum with scalar weights, an iterative method is proposed  that requires the pre-trained vector representations for each modality to have the same number of elements arranged in an order that is suitable for element-wise addition. This is often achieved by jointly training a fully connected layer for dimension control and reordering for each modality, together with the scalar weights for fusion. A recent study  employs neural architecture search with progressive exploration , ,  to find suitable settings for a number of fusion functions. Each fusion function is configured by which layers to fuse and whether to use concatenation or weighted sum as the fusion operation. Other weak functions can also be used to fuse multiple layers from each modality .

We finish this section by reviewing the relation networks (RN), which has a simple structure that uses an ANN as the function to model the relationship between any pair of visual and textual features, and the resulted output values are accumulated and transformed by another ANN . Though RN merely models the relationship without any form of induction reasoning, it achieves very high VQA accuracy on CLEVR. This inspires a re-thinking of the connection between correlation and induction.

Section SECREF2 reviews the recent progress on developing representations for single or multiple modalities. Section SECREF3 introduces the commonly used fusion methods, particularly attention and bilinear pooling. Applications including caption generation, text-to-image synthesis, VQA, and visual reasoning are introduced in Section SECREF4, followed by conclusions.

Representations ::: Multimodal Representations ::: Methods for zero-shot learning	Zero-shot learning often applies to vision related tasks due to the difficulty to acquire sufficient labelled images for training for all possible object categories. Not all types of multimodal representations are suitable for zero-shot learning since they may require pair-wised data from both modalities to present at the same time. Here we review methods that rely on extra language source to remedy this issue. Deep learning based zero-shot learning started by training a linear mapping layer between different pre-trained embeddings , . The deep visual-semantic embedding (DeViSE) model is built upon Skip-Gram word embedding and AlexNet visual features and allows both pre-trained models to be jointly trained with the linear mapping layer . It achieved a large-scale test with 1000 seen classes and 2000 unseen classes. Better representations could be learnt when correlated autoencoders are used to reconstruct the representations for each modality, which improves one-shot and few-shot image retrieval performance comparing to DeViSE . Richer information source can be used for both modalities, including words selected from Wikipedia articles and features derived from multiple CNN layers .

Applications ::: Visual Reasoning ::: Neural module network based methods	Neural module network (NMN) is a method which composes a collection of jointly trained neural “modules” into a deep model for answering the question . A dependency parser first helps to convert the natural language question into a fixed and rule-based network layout, and specify both the set of modules used to answer the question and the connections between them. Then a deep model is assembled based on the layout to produce the prediction of the answers. SHAPES, a synthetic dataset consists of complex questions about simple arranges of ordered shapes, was also proposed to focus on the compositional phenomena of questions . A later study learns the model layout predictor jointly with the parameters of the modules by re-ranking a list of layout candidates using reinforcement learning, which is termed as dynamic NMN (D-NMN) . Modules such as “find” or “relate” operation uses attention models to focus on one or two regions of the input image and makes the forwarding of the assembled deep model similar to running a functional program .

Fusion ::: Attention-based Fusion ::: Other attention like mechanisms	Gated multimodal unit is a method that can be viewed as the attention of image and text based on gating . It performs weighted sum of visual and textual feature vectors based on dimension-specific scalar weights generated dynamically by the gating mechanism. Similarly, element-wise multiplication can be used to fuse visual and textual representations, which is used to create the building blocks of a multimodal residual network (MRN) based on deep residual learning . Dynamic parameter prediction network (DPPnet) uses a dynamic weight matrix to transform the visual feature vectors, whose parameters are dynamically generated by hashing the text feature vector .

Further studies found correlating image regions/fragments with sentence fragments or attribute words generates fine-grained multimodal embeddings , by finding the alignments of the image and sentence fragments automatically.  unifies the embeddings of concepts at different levels, including objects, attributes, relations and full scenes.  proposed a stacked cross attention network (SCAN) to learning fine-grained word and image-object aligned embedding for image-text matching.  employs a deep attentional multimodal similarity model (DAMSM) extending DMSM with attention models to measure the similarity between image sub-regions and words as an additional loss for text-to-image generation.

Applications ::: Caption Generation	Image captioning  requires to generate a description of an image and is one of the earliest task that studies multi-modal combination of image and text. We mainly review the deep learning based methods for caption generation. Image captioning, such as , , , divide the task into several sub-tasks and generate caption in a step-by-step manner. Authors in  first trained a deep CNN model to detect the words from images, then built a log-linear language model to compose the words into sentences. Similarly,  fed the image feature into a log-linear language model to generate sentences. In contrast,  tried to find the exact matching of objects in images and words in sentences to determine if an image and a sentence match with each other. Similar to the RNN-based encoder-decoder methods for machine translation , , ,  propose to generate captions from images in an end-to-end manner via the encoder-decoder architecture. In those models, a CNN, typically pre-trained on ImageNet , encoded the image into a continuous vector, which is then fed into a RNN/LSTM decoder to generate the caption directly.

Those works all followed the same architecture, but varied slightly the choice of CNN architecture and how the image vector was fed into the decoder. Though powerful and convenient, the encoder-decoder architecture lacks to ability capture the fine grained relationship between objects in images and words in sentences. To overcome this, attention-based encoder-decoder model  was proposed and has become the standard benchmark for this task since then. In the attention encoder-decoder model, before generating the next word, the decoder first calculates the matching score (attention) with objects in the image, then conditions on the weighted image feature to generate the next token. There has been lots of work that tried to improve the attention model by incorporating more structures. For example,  added a gate at every decoding step to determine if the next word should be generated using image information;  combined detected words and image features as inputs to the decoder network. More recently, there has been a lot works that add more structure/knowledge from either image  or text side . Specifically,  used an object detector to localize the features for image object and then generates the caption based on the localized features. It improved the previous state of art model by a large margin in a variety of evaluation metrics.

Representations ::: Multimodal Representations ::: Unsupervised training methods	Joint embeddings for multimodal data can be learnt by simply reconstructing the raw input using multiple streams of deep Boltzmann machines or autoencoders with a shared layer as the shared representation space , , . Alternatively, with the development of methods for single modal representations, the shared representation space can be constructed based on those of the involved single modalities. For example, in , Fang et al. propose a deep multimodal similarity model (DMSM), which extended the text-modal DSSM to learning embedding representations of text and image in an unified vector space. ,  perform simple fusion of the word and image embeddings with addition or concatenation.  learns to increase the similarity between corresponding Skip-Gram word embedding and AlexNet derived image features. ,  maximize the correlation and mutual information between embeddings of different modalities respectively.  modifies the distance between CBOW word embeddings according to the similarities between their visual instantiations, which are found by clustering abstract scenes in an unsupervised way.

Visual reasoning methods for VQA are reviewed in the end. Detailed task specifications, data sets, and selected work for each task will be introduced in this section.

Fusion ::: Attention-based Fusion ::: Image attention	 extends an LSTM model for text question processing with an image attention model conditioned on the previous LSTM hidden state, whose input is a concatenation of the current word embedding with the attended image feature. The final LSTM hidden state is regarded as the fused multimodal representation to predict the answer for pointing and grounded VQA. The attention model for sequence-based encoder-decoder model is used to attend to the image features for image captioning . Further for VQA, attention model conditioned on both image and query feature vectors is applied to pinpoint the image regions relevant to the answer . Similarly, stacked attention networks (SANs) are proposed to use multiple layers of attention models to query an image multiple times to infer the answer progressively by simulating a multi-step reasoning procedure . At each layer, a refined query vector is generated and send to the next layer by adding the previous query vector to the attended image vector produced using the current attention model. Spatial memory network (SMem) is a multi-hop method for VQA, which aligns words to image regions in the first hop and performs image attention based on the entire question in the second hop to derive the answer .

Approaches designed for these data sets often focus on fusing image and question vectors with the previously discussed attention- and bilinear pooling-based methods, such as SAN, co-attention, Up-Down, MCB, MLB, and BAN etc.

Applications	In this section, selected applications for multimodal intelligence that combine vision and language are discussed, which include image captioning, text-to-image generation, and VQA. It is worth noting that there are other applications, such as text-based image retrieval , , , and visual-and-language navigation (VLN) , , , , , , , , , that we have not included in this paper due to space limitation. Caption generation is a task that aims to automatically generate a natural language description of an image. It requires a level of image understanding beyond normal image recognition and object detection. A reverse of caption generation is text-to-image synthesis, which often generates image pixels according to a description sentence or some key words provided by human. VQA is related to caption generation, which often takes an image as the input and a free-form, open-ended natural language question about the image, to output a classification result as the output of the answer. Natural language understanding is required as the questions are in free form. Other capabilities such as knowledge based reasoning and common-sense reasoning can be important since the questions are open-ended. Visual reasoning can be included in all of the aforementioned tasks.

The naïve Inception structure is a concatenation of CNN layers with $1\times 1$, $3\times 3$, and $5\times 5$ sized kernels, and a max pooling layer with $3\times 3$ kernels, and can be viewed as a sparsely connected convolutional architecture to reduce overfitting and computational cost. Later versions of the Inception models improve the structures further by factorizing the kernels and adding residual connections. AlexNet, GoogLeNet, and ResNet are the winners of the 2012, 2014, and 2015 ImageNet Large Scale Visual Recognition Competition for image classification respectively , . Alternatively, features with more direct relationships with the semantics can be used as visual embeddings, such as convolutional features and the associated class labels from selected regions found by object detection models, such as region with CNN features (R-CNN) , Fast R-CNN , and Faster R-CNN  etc.

Fusion ::: Attention-based Fusion	Attention mechanism is widely used for fusion, which often refers to weighted sum a set of vectors using scalar weights dynamically generated by a small “attention” model at each time-step , . Multiple glimpses (output heads) are often used by the attention model to generate multiple sets of dynamic weights for the summation, whose resulted values can be concatenated to reserve more information. When applying attention mechanism to an image, image feature vectors relevant to different regions are weighted differently to produce an attended image vector.

MUTAN, a multimodal tensor-based Tucker decomposition method, uses Tucker decomposition  to factorize the original three-dimensional weight tensor operator with a small-dimensional core tensor and the three two-dimensional weight matrices used by MLB . The core tensor models the interactions across modalities. Comparing to MUTAN, MCB can be seen as MUTAN with fixed diagonal input factor matrices and a sparse fixed core tensor, while MLB is MUTAN with the core tensor set to identity. Recently, BLOCK, a block superdiagonal fusion framework is proposed to use block-term decomposition  to compute bilinear pooling . BLOCK generalizes MUTAN as a summation of multiple MUTAN models to provide a richer modeling of interactions between modalities. The MUTAN core tensors can be arranged as a superdiagonal tensor, similar to the submatrices of a block diagonal matrix. Furthermore, bilinear pooling can be generalized to more than two modalities, such as  and  that use outer products to model the interactions among the representations for video, audio, and language.

The object-driven attentive GAN (Obj-GAN) improves the two-stage generation idea using a pair of object-driven attentive image generator and object-wise discriminator . At every generation step, the generator uses the text description as a semantic layout and synthesizes the image region within a bounding box by focusing on the words that are most relevant to the object in it. ObjGAN is found to be more robust and interpretable, and significantly improves the object generation quality for complex scenes.

To ensure the generated image semantically match the text precisely,  proposed attentional GAN (AttnGAN), which also stacks three GANs for different image resolutions , and while the first GAN is conditioned on the sentence embedding, the next two GANs are conditioned on bimodal embeddings produced by attention models fusing word-level features with low-resolution images. It is shown attention mechanism can help GAN to focus on words that are most relevant to the sub-region drawn at each stage. Apart from stacking the generators,  shows that high resolution images can also be generated with a dynamic memory module.

Fusion ::: Bilinear Pooling-based Fusion ::: Bilinear pooling and attention mechanism	Bilinear pooling can be used along with attention mechanism. MCB/MLB fused bimodal representation can be used as the input feature of an attention model to derive the attended image feature vector, which is fused with the textual feature vector using MCB/MLB again to form the final joint representation , . MFB/MFH can be used for alternating co-attention to learn the join representation , . Bilinear attention network (BAN) uses MLB to fuse image and text to produce a bilinear attention map as the attention distributions, which is used as the weight tensor for bilinear pooling to fuse the image and text features again .

Representations ::: Multimodal Representations ::: Supervised training methods	Supervisions can be used to improve multimodal representation learning.  factorizes the representations into two sets of independent factors: multimodal discriminative factors for supervised training and intra-modality generative factors for unsupervised training. The discriminative factors are shared across all modalities and are useful for discriminative tasks, whereas the generative factors can be used to reconstruct missing modalities. With detailed text annotations,  proposed to learn word embeddings from their visual co-occurrences (ViCo) when applying to the same natural scene image or image region. ViCo is found to be complementary to the GloVe embedding by better representing similarities and differences between visual concepts that are difficult to obtain from text corpora alone.  applies multiple supervised training tasks to different layers of the vision-language encoder. The order of the training tasks is arranged following the idea of curriculum learning to increase the complexity of training objective step-by-step.

Representations ::: Unimodal Embeddings	The representations obtained using ANN models are often distributed, which entails that elements composing the representations can be set separately to allow more concepts to be encoded efficiently in a relatively low-dimensional space . This can be compared with the symbolic representations, such as the one-hot encoding that uses an element with value one to indicate the presence of the associated symbol or category, and value zero for the rest elements. In deep learning, the term embedding often refers to a mapping from a one-hot encoding representing a word or an image category to a distributed representation as a vector of real-valued numbers.

Fusion ::: Attention-based Fusion ::: Attention in bimodal Transformer	Recall Section SECREF14, the bimodal extensions to BERT rely on different tokens to indicate whether a vector is a word piece or an image, and the attention models fuse images with words in bimodal input sequences , , , , . OmniNet uses the gated multi-head attention model in each decoder block to fuse the vectors from the other modalities with that produced for the current modality by the previous layers in the the block . LXMERT uses independent encoders to learn the intra-modality features for each modality, and a cross-modality encoder sitting above them to learn the cross-modality features using extra cross-attention layers . ViLBERT extends BERT to include two encoder streams to process visual and textual inputs separately, which can interact through parallel co-attention layers .

Fusion ::: Bilinear Pooling-based Fusion	Bilinear pooling is a method often used to fuse a visual feature vector with a textual feature vector into a joint representation space by computing their outer product, which allows a multiplicative interaction between all elements in both vectors and is also termed as second order pooling . Comparing to simple vector combination operations (assuming each vector has $n$ elements), such as weighted sum, element-wise multiplication, or concatenation that result in $n$ or $2n$ dimensional representations, bilinear pooling leads to an $n^2$ dimensional representation by linearizing the outer product resulted matrix into a vector and is therefore more expressive. The bilinear representation is often linearly transformed into an output vector using a two-dimensional weight matrix, which is equivalent to use a three-dimensional tensor operator to fuse the two input feature vectors. Each feature vector can be extended with an extra value one to reserve input single modal features in the bilinear representation via outer product . However, given its high dimensionality, typically on the order of hundreds of thousands to a few million, bilinear pooling often requires decomposing the weight tensor to have the associated model to be trained properly and efficiently.

Fusion ::: Attention-based Fusion ::: Image and text co-attention	Different from the aforementioned image attention methods, co-attention mechanism has a symmetric attention structure to generate not only an attended image feature vector, but also an attended language vector . The parallel co-attention uses a joint representation to derive the image and language attention distributions simultaneously; alternating co-attention, on the other hand, has a cascade structure that first generates the attended image vector using the linguistic features, followed by the attended language vector generated using the attended image vector. Similar to the parallel co-attention, dual attention network (DAN) estimates attention distributions for image and language simultaneously to derive their attended feature vectors . The attention models are conditioned on both feature and memory vectors of the relevant modality. This is a key difference to co-attention since the memory vectors can be iteratively updated at each reasoning step by repeating the DAN structure. The memory vectors can be either shared for VQA or modality-specific for image-text matching. Stacked latent attention (SLA) improves SAN by concatenating the original attended image vector with values from earlier layers of the attention model to retain the latent information from intermediate reasoning stages .

To empower the model to handle data samples with some missing modality, zero-shot learning problem is studied to increase the similarity of the representational spaces across the involved modalities. At last, inspired by the great success of adapting pre-trained LMs for downstream tasks in NLP, methods that leverage large unimodal data sets to improve the learning of multimodal representations are also discussed.

The method of global vectors (GloVe) shows that effective text-embedding can be learnt using a global log-bilinear regression model based on the co-occurrence counts of words . Meanwhile, a serious of deep structured semantic models (DSSM) were proposed since 2013 for sentence level embedding learning through optimizing semantic-similarity driven objectives, with various neural network structures in a pseudo-simense network setting , , , , , , . More recently, in order to transfer to use in downstream natural language understanding tasks without much training data, studies focus on learning general text embeddings by predicting word probabilities using NNLMs with complex structures on a large text corpus. Embeddings from language models (ELMo) uses a combined embedding from multiple layers of bidirectional LSTMs for forward and backward directions . Generative pre-training (GPT) and bidirectional encoder representations for Transformers (BERT) use the decoder and encoder part of the Transformer model to estimate the probability of the current subword unit , . Other technologies, such as masked language model and multi-task training are used in these methods nowadays .

Representations ::: Multimodal Representations ::: Transformer based methods	Transformer is a prevalent sequence-based encoder-decoder model formed by stacking many blocks of feedforward layers with multi-head self-attention models . The parameters in all blocks shared across time similar to the time-delayed neural networks  and quasi-RNN  without an explicit temporal order. Compared with RNN based encoder-decoder models , it can have higher training efficiency due to the additional parallel degree across-time and superior performance on longer sequences benefited from the removing of first-order Markovian assumption imposed to the RNNs. BERT, the encoder part of Transformer pre-trained on a large text corpus as a masked LM, becomes a standard choice for word piece embeddings for downstream tasks, particularly since it utilizes both past and future information easily. It is natural to generalize the text-only BERT to cover images as well that can be used as the pre-trained multimodal embeddings. A straightforward way to extend the unimodal BERT to bimodal, is to include new tokens to indicate the input of visual features, such as Unicoder-VL , VL-BERT , VisualBERT , VideoBERT , and B2T2 .

Applications ::: Visual Question Answering ::: Discounting language priors	Though significant achievements have been made, recent studies point out that the common VQA benchmarks suffer from strong and prevalent priors – most bananas are yellow and mostly the sky is blue, which can often cause the VQA model to over-fit to these statistical biases and tendencies from the answer distributions, and largely circumvent the need to understand the visual scenes. Based on the objects, attributes, and relations provided through the scene graphs from Visual Genome, a new data set, GQA, was created to greatly reduce such biases by generating questions with a functional program that controls the reasoning steps . New splits for VQA v1 and VQA v2 are generated to have different answer distributions for every question of the training and test sets, which are referred to as VQA under challenging priors (VQA-CP v1 and VQA-CP v2) . Recent methods propose to handle the biased priors with adversarial training or additional train only structures , .

Representations ::: Multimodal Representations	Although significant progress has been made in the learning of representations for vision or language, it is theoretically insufficient to model a complete set of human concepts using only unimodal data. For example, the concept of “beautiful music” is clearly grounded in the auditory perceptron and one can be struggled to describe this by natural language or other approaches. Therefore, it is important to learn a joint embedding to leverage the complementarity from multimodal data to represent the concepts better. Both supervised and unsupervised training approaches are of broad interest and can be applied to tasks with different data availability. Meanwhile, by assuming the corresponding representations to have similar neighbourhood structures across modalities, the representation of a concept with zero training sample in one modal can be found based on its representations grounded in other modalities which have training data. For instance, when using zero-shot training for image labelling, the closest word vectors can be retrieved as labels by projecting images of objects unseen in the training set onto the linguistic space. More recently, inspired by Transformer and BERT from NLP, it becomes increasingly popular to apply these models to develop better bimodal representations combining vision and language.

Applications ::: Visual Question Answering ::: Other issues	Another problem that current VQA methods suffers from is the low robustness against linguistic variations from the questions. A data set, VQA-Rephrasings, modified the VQA v2.0 validation set with human authored rephrasing of the questions . A cycle-consistency  based method that improves the linguistic robustness by enforcing consistencies between the original and rephrased questions, and between the true answer and the answers predicted based on the original and rephrased questions.  suggests that attention mechanism can cause VQA models to suffer from counting the object proposals, and an extra model component was proposed as a solution. Moreover, it is the fact that the current VQA methods cannot even read text from images. A method is proposed to address this problem by fusing not text extracted from the image using optical character recognition . VizWiz is a goal oriented VQA data set collected by blind people taking possibly low quality pictures and asking questions in spoken English, which also include many text related questions .

His research interests are mainly in deep learning, natural language processing, speech recognition, computer vision, information retrieval, and multimodal intelligence. He has held editorial positions on multiple IEEE Journals and the Transactions of the ACL, and served in the organizing committee/program committee of major speech and language processing conferences. He is a member of the IEEE SLTC for the term of 2015-2017 and the Chair of the IEEE Seattle Section in 2016. He received the Bachelor degree from Tsinghua University in 1996, MS degree from Chinese Academy of Sciences in 1999, and the PhD degree from the University of Missouri – Columbia in 2003. Li Deng has been the Chief Artificial Intelligence Officer of Citadel since May 2017. Prior to Citadel, he was the Chief Scientist of AI, the founder of the Deep Learning Technology Center, and Partner Research Manager at Microsoft and Microsoft Research , Redmond (2000-2017). Prior to Microsoft, he was an assistant professor (1989-1992), tenured associate (1992-1996), and full professor (1996-1999) at the University of Waterloo in Ontario, Canada.

Representations ::: Unimodal Embeddings ::: Language representations	Text embeddings can be derived from a neural network language model (NNLM) , which estimates the probability of a text sequence by factorizing it into word probabilities based on the chain rule of probability. A feedforward neural network with a linear projection layer and a non-linear hidden layer is often used as an implementation of an $n$-gram LM, which takes the previous $n-1$ words as the input to predict the probability of the current word. Each word is presented in one-hot encoding based on the vocabulary and converted into a real-valued vector, the word embedding, using the projection layer. An improved NNLM is to replace the feedforward model with an RNN, such as a long short-term memory (LSTM)  or gated recurrent unit (GRU)  model, which allows the use of information from all past words stored in a fixed-length recurrent vector when predicting the current word. Apart from NNLMs, continuous bag-of-words model (CBOW) and skip-gram model are two simple feedforward structures that learn word embedding either by predicting the current word based on the past and future context words or vice versa .

Besides word and subword levels, text embedding can be learnt at phrase, sentence, and even paragraph levels, such as the skip-though vectors that extends the skip-gram method to the sequence-to-sequence framework , . It uses two decoders to predict the previous and next sentences given the embedding of the current sentence generated by the encoder.

Representations ::: Unimodal Embeddings ::: Vector arithmetic for word and image embeddings	It is well-known that word embeddings can learn not only syntactic but also the semantic regularities. A famous example showed vector(“King”)$-$vector(“Man”)$+$vector(“Woman”) results in a vector closest to vector(“Queen”) where vector($\cdot $) denotes the vector representation of a word learnt by a RNN LM . Similar phenomenon has been observed for vision embeddings. It was shown that when using a generative adversarial network (GAN) , there exists a similar vector arithmetic that the representation of an image with a man wearing glasses subtracted by that of a man without glasses and finally add the representation of a woman without glasses will lead to the representation of a woman wearing glasses . This reveals GAN can capture image representation that distangles the concept of gender from the concept of wearing glasses. Such encouraging progress in text and image representations encouraged further studies on the joint representations of these two modalities. More details about GAN based image generation can be found later in Section SECREF31.

Representations ::: Unimodal Embeddings ::: Visual representation	The image embeddings can be acquired as the output values from the final CNN layers from models that classify images into categories, such as AlexNet , VGG nets , and residual neural network (ResNet) . AlexNet is a model with five CNN layers with rectified linear unit (ReLU) activation functions whose kernel sizes are $11\times 11$, $5\times 5$, and $3\times 3$. A VGG net often has 16 or 34 CNN layers, with all of them using very small $3\times 3$ kernels. ResNet can have a depth of up to 152 layers, mostly with $3\times 3$ kernels, due to the invention of residual connections. Comparing to the aforementioned models, GoogLeNet has a more different structure formed by stacking multiple Inception structures .

For other NLP tasks without much training data, such as question answering (QA) and machine reading comprehension, generative pre-training that transfers parameters from a language model (LM) pre-trained on a large out-of-domain data set using unsupervised or self learning then followed by fine-tuning on small in-domain data sets, achieved record-breaking results over a set of tasks , , . Despite the advances in vision, speech, and language processing, many problems in artificial intelligence involve more than one modality, such as an intelligent personal assistant (IPA) that should understand human communicative intentions embedded not only in spoken language, but also in body and pictorial languages . Therefore, it is of broad interests to study the modeling and learning approaches across multiple modalities . Benefiting from the advances in image processing and language understanding , a set of tasks that combine both image and text have drawn much attention, which include visual grounding tasks like referring expression understanding and phrase localization , , , image captioning , , , visual QA (VQA) , , , text-to-image generation , , , and visual-language navigation  etc.

He also received dozens of best paper and patent awards for the contributions to artificial intelligence, machine learning, information retrieval, multimedia signal processing, speech processing and recognition, and human language technology. He is an author or co-author of six technical books on deep learning, speech processing, pattern recognition and machine learning, and, the latest, natural language processing (Springer, June 2018).

Rather than direct text attribute input, sentence embedding generated by recurrent models can be used as the text interface for zero-shot learning to achieve competitive results . Moving beyond empirical findings, recent study analyzed the properties of deep learning based cross-modal mapping using a similarity measure .

LXMERT , ViLBERT , and OmniNet  modify the Transformer model by introducing an extra encoder or attention structures for visual features. More details about the modified structures can be found from Section SECREF18. Furthermore, recent NLP study found that multitask training can improve the generalization ability of the BERT representations . Most of the aforementioned bimodal BERT style models adopt multitask training to improve their performance on downstream tasks like VQA, image and video captioning etc. Although it would be useful to rigorously compare the performance of these models to understand the impact of different design choices, it is hard to do so since different amount of parameters and pre-training data are used across papers.

Chao Zhang is an advisor of JD.com speech team, and a research associate in speech and natural language processing at the University of Cambridge. He received his B.E. and M.S. degrees in 2009 and 2012 respectively, both from the Department of Computer Science & Technology, Tsinghua University, and a Ph.D. degree in 2017 from Cambridge University Engineering Department. Zichao Yang is a quantitative researcher at Citadel. Prior to that, he received his Phd in computer science from Carnegie Mellon University. His research interests are in machine learning, deep learning and their applications in natural language processing and computer vision. He has published dozens of papers in NeurIPS, ICML, CVPR, ICCV, EMNLP, NAACL etc. Xiaodong He (IEEE Member 2003, Senior member 2008, Fellow 2019) is the Deputy Managing Director of JD AI Research, and Head of the Deep learning, NLP and Speech Lab. He is also Affiliate Professor of ECE at the University of Washington (Seattle).

Applications ::: Visual Reasoning ::: Other types of end-to-end reasoning methods	Another end-to-end approach is the memory, attention, and composition (MAC) network that decomposes the question into a series of attention-based reasoning steps and perform each of them using a recurrent MAC cell that maintains a separation between the control and memory hidden states. Each hidden state is generated by an ANN model constructed based on attention and gating mechanisms . More recently, both deterministic symbolic programs and probabilistic symbolic models have been used as the execution engine for the generated programs to improve the transparency and data efficiency, which result in the neural-symbolic VQA (NS-VQA) and probabilistic neural-symbolic models (prob-NMN) respectively , . As an extension of NS-VQA, the neuro-symbolic concept learner (NS-CL) uses a neuro-symbolic reasoning module to execute programs on the scene representation. NS-CL can have its program generator, reasoning module, and visual perception components jointly trained in an end-to-end fashion without requiring any component-level supervisions . Its perception module learns visual concepts based on the language descriptions of the objects and facilitates learning new words and parsing new sentences.

An end-to-end version of NMN (N2NMN) used an RNN question encoder to convert the input question into a layout policy without requiring the aid of a parser . The work is based on a more recent data set called compositional language and elementary visual reasoning diagnostics (CLEVR). As its name suggests, CLEVR is a synthetic diagnostic data set testing a range of visual reasoning abilities of objections and relationships with minimal biases and detailed annotations describing the kind of reasoning each question requires . Other implementations of NMN include the program generator and execution engine method (PG+EE) that shares generic design among some operations , stack-NMN that improves the parser and incorporates question features into the modules , and transparency by design network (TbD-net) that redesigns some modules from PG+EE to maintain the transparency of the reasoning procedure .

4) choices, and each choice is associated with each answer presented in the form of a natural language sentence . This setup can be implemented as a classification to the choices based on features of the image, question, and answer candidates . There exist other types of VQA task definitions, such as the Visual Madlibs dataset that requires to answer the questions by “fill-in-the-blanks” . Furthermore, visual dialogue can be viewed as to answer a sequence of questions grounded in an image , , which extends VQA by requiring to generate more human like responses and to infer the context from the dialogue history.

Applications ::: Text-to-Image Synthesis ::: Other topics	In addition to the layout, other types of fine-grained control in image generation have also been studied in literature. Attribute2Image  studies the use of attributes in face generation, such as age and gender etc.  uses the same idea for face editing, such as to remove the beard or change the hair color. Text-adaptive GAN  allows semantic modification of input images for birds and flowers via natural language.  enforces to learn the representation content and style as two disentangled variables using a dual inference mechanism based on cycle-consistency for text-to-image synthesis. The success of these methods validate GAN is able to learn some semantic concepts as disentangled representations, as in Section SECREF9. Text2Scene is another interesting work that generates compositional scene representation from natural langauge step-by-step without using GANs . It is shown with minor modifications, Text2Scene can generate cartoon like, semantic layout, and real image like scenes. Dialogue based interaction is studied to control image synthesis, in order to improve complex scene generation progressively , , , , . Meanwhile, text-to-image synthesis is extended to multiple images or videos, where visual consistency is required among the generated images , , .

Applications ::: Text-to-Image Synthesis ::: Semantic layout control for complex scenes	With the success in the generation of realistic and semantically consistent images for single objects, such as birds  or flowers , state-of-the-art text-to-image synthesis methods still struggle to generate complex scenes with many objects and relationships, such as those in the Microsoft COCO data set . In the pioneering work , not only text descriptions but also locations of objects specified by keypoints or bounding boxes are used as the input. Later, detailed semantic layout, such as a scene graph, is used to replace the natural language sentence as a more direct description of objects and their relationships , , . Meanwhile, efforts are made to keep natural language input while incorporating the idea of semantic layout.  includes extra object pathway to both generator and discriminator to explicit control the object locations.  employs a two-stage procedure that first builds a semantic layout automatically from the input sentence with LSTM based box and shape generators, and then synthesizes the image using image generator and discriminators. Since fine-grained word/object level information is not explicitly used for generation, such synthesized images do not contain enough details to make them look realistic.

Applications ::: Text-to-Image Synthesis ::: Generating semantically consistent images	To improve the semantic consistency between relevant image and text features, DAMSM is proposed for AttnGAN .  tackles the same problem by leveraging hierarchical representations with extra adversarial constraints to discriminate not only real/fake image pairs, but also real/fake image-text pairs at multiple image resolutions in the discriminator, and is named as hierarchically-nested discriminator GAN (HDGAN). Similarly, text conditioned auxiliary classifier GAN (TAC-GAN) introduces an extra image classification task to the discriminator , whereas text-conditioned semantic classifier GAN (Text-SeGAN) alternates the classifier with a regression task to estimate the semantic relevance between image and text . Analogous to cycle consistency , MirrorGAN is proposed to improve the semantic consistency between the two modalities using an extra image captioning module .

Applications ::: Text-to-Image Synthesis ::: Generating high quality images	Though basically reflecting the meaning of the descriptions, it is found the images produced by GAN-INT-CLS do not have necessary details and vivid object parts, and therefore leads to the StackGAN method . StackGAN decomposes image synthesize into more manageable sub-problems through a sketch-refinement process by stacking two CGANs trained separately. The first GAN produces 64$\times $64 low-resolution images by sketching the primitive shape and colors of the object based on the text, and the second GAN is trained after to generate 256$\times $256 images by rectifying defects and adding compelling details in the low-resolution image. StackGAN$++$ improves this idea by adding an extra GAN to generate 128$\times $128 images in between and training all GANs jointly.

To evaluate whether a generated image is semantically consistent with the input text description, R-precision  and visual-semantic similarity  are used as the metrics.

Applications ::: Text-to-Image Synthesis ::: GAN based methods	Comparing to VAE, conditional-GAN (CGAN) is found to be able to synthesis highly compelling images of specific categories that a human might mistake for real , . A GAN model consists of a generator that synthesize candidates based on input noises and a discriminator that evaluates them. Adversarial training is employed to make the generator to capture the true data distribution so that the discriminator can no longer discriminate the synthesized data from the real ones . CGAN extends the standard GAN structure by conditioning on extra category labels for both generator and discriminator. GAN-INT-CLS allows to synthesize visually plausible 64$\times $64 images using the embeddings of natural language descriptions to replace the category labels in CGAN . The automatic evaluation of the quality of text conditioned images can be less straightforward. To find the discriminability of GAN generated images, inception score (IS)  and Fréchet inception distance  (FID) are often used. Multi-scale structural similarity (MS-SSIM)  is commonly used to evaluate the diversity of images.

Applications ::: Text-to-Image Synthesis	Text-to-image synthesis or generation that relies on natural language to control image generation, is a fundamental problem in computer vision. It is considered as a difficult problem since it least involves two tasks: high quality image generation and language understanding. The generated image is required to be both visually realistic and semantically consistent to the language description. Deep learning based text-to-image synthesis can perhaps be dated back to the use of LSTM for iterative hand-writting generation . This iterative image generation idea is later extended to form the deep recurrent attentive writer (DRAW) method that combines an LSTM based sequential variational auto-encoder (VAE) with a spatial attention mechanism . alignDRAW modifies DRAW to use natural language based descriptions to synthesis images with general content . An attention model is used to compute the alignment between the input words and the patches drawn iteratively.

Image captions with richer information could be generated when incorporated with external knowledge. For example, based on an model that can recognize celebrities , a CaptionBot app is developed which can not only describe the facts (such as activities) in a picture, but also describe who is doing that if the person in the picture is recognized . Further, beside simply generating a factual description of the image, other approaches were also proposed for explicitly controlling the style , semantic content , and diversity  of the generated caption.

He also held faculty or research positions at Massachusetts Institute of Technology (Cambridge, 1992-1993), Advanced Telecommunications Research Institute (ATR, Kyoto, Japan, 1997-1998), and HK University of Science and Technology (Hong Kong, 1995). He is a Fellow of the Academy of Engineering of Canada, a Fellow of the Washington State Academy of Sciences, a Fellow of the IEEE, a Fellow of the Acoustical Society of America, and a Fellow of the International Speech Communication Association. He has also been an Affiliate Professor at the University of Washington, Seattle. He was an elected member of the Board of Governors of the IEEE Signal Processing Society, and was Editors-in-Chief of IEEE Signal Processing Magazine and of IEEE/ACM Transactions on Audio, Speech, and Language Processing (2008-2014), for which he received the IEEE SPS Meritorious Service Award. In recognition of the pioneering work on disrupting speech recognition industry using large-scale deep learning, he received the 2015 IEEE SPS Technical Achievement Award for “Outstanding Contributions to Automatic Speech Recognition and to Deep Learning”.

A parallel co-attention like twin stream structure is also included to attend to both image and language features that also allows to reason iteratively using multiple SLA layers. Dual recurrent attention units (DRAU) implements the parallel co-attention structure with LSTM models for text and image to attend to each input location of the representations obtain by convolving image features with a stack of CNN layers . To model high-order interactions between modalities, high-order correlations between two data modalities can be computed as the inner product of two feature vectors and used to construct high-order attention models to derive the attended feature vectors for both modalities .

In , dynamic memory network (DMN) is augmented to use separate input modules to encode the question and image, which uses attention based GRUs to update episodic memory iteratively to retrieve the required information. Bottom-up and top-down attention method (Up-Down), as its name suggested, simulates human visual system using a combination of two visual attention mechanisms . The bottom-up attention mechanism proposes a set of salient image regions found by a Faster R-CNN, and the top-down attention mechanism uses a concatenation of visual and linguistic features to estimate the attention weights and produce the attended image feature vector for image captioning or VQA. The attended image feature vector can be fused with the linguistic feature again using an element-wise product. Complementary image features derived from different models, such as ResNet and Faster R-CNN, are used for multiple image attention mechanisms . Moreover, the reverse of image attention that generates attended text feature with image and text input is used for text-to-image synthesis in  and .

Introduction	Significant progress has been made in the field of machine learning in the past years due to the rapid development of deep learning , , , , , . Dating back to the dramatic increase in the accuracy of large-scale automatic speech recognition (ASR) using fully connected deep neural networks (DNN) and deep auto-encoders around 2010 , , , , , , , , , , , and followed by a set of breakthroughs in computer vision (CV) using deep convolutional neural network (CNN) models  for large-scale image classification around 2012 , , ,  and large-scale object detection , ,  around 2014, a set of major milestones have been achieved in pattern recognition with single input modality. Subsequently, in natural language processing (NLP), recurrent neural network (RNN) based semantic slot filling methods  achieved new state-of-the-art in spoken language understanding, and RNN-encoder-decoder models with attention mechanism , also referred to as sequence to sequence models , produced superior performance in machine translation in an end-to-end fashion , .

Question: What are special architectures this review focuses on that are related to multimodal fusion?
Answer: attention mechanism, bilinear pooling