Question: What is the main focus?
Evidence: Significant progress has been made in the field of machine learning in the past years due to the rapid development of deep learning BIBREF0, BIBREF1, BIBREF2, BIBREF3, BIBREF4, BIBREF5. Dating back to the dramatic increase in the accuracy of large-scale automatic speech recognition (ASR) using fully connected deep neural networks (DNN) and deep auto-encoders around 2010 BIBREF6, BIBREF7, BIBREF8, BIBREF9, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15, BIBREF16, and followed by a set of breakthroughs in computer vision (CV) using deep convolutional neural network (CNN) models BIBREF17 for large-scale image classification around 2012 BIBREF18, BIBREF19, BIBREF20, BIBREF21 and large-scale object detection BIBREF22, BIBREF23, BIBREF24 around 2014, a set of major milestones have been achieved in pattern recognition with single input modality. Subsequently, in natural language processing (NLP), recurrent neural network (RNN) based semantic slot filling methods BIBREF25 achieved new state-of-the-art in spoken language understanding, and RNN-encoder-decoder models with attention mechanism BIBREF26, also referred to as sequence to sequence models BIBREF27, produced superior performance in machine translation in an end-to-end fashion BIBREF28, BIBREF29. For other NLP tasks without much training data, such as question answering (QA) and machine reading comprehension, generative pre-training that transfers parameters from a language model (LM) pre-trained on a large out-of-domain data set using unsupervised or self learning then followed by fine-tuning on small in-domain data sets, achieved record-breaking results over a set of tasks BIBREF30, BIBREF31, BIBREF32.
Despite the advances in vision, speech, and language processing, many problems in artificial intelligence involve more than one modality, such as an intelligent personal assistant (IPA) that should understand human communicative intentions embedded not only in spoken language, but also in body and pictorial languages BIBREF33. Therefore, it is of broad interests to study the modeling and learning approaches across multiple modalities BIBREF34. Benefiting from the advances in image processing and language understanding BIBREF35, a set of tasks that combine both image and text have drawn much attention, which include visual grounding tasks like referring expression understanding and phrase localization BIBREF36, BIBREF37, BIBREF38, image captioning BIBREF39, BIBREF40, BIBREF41, visual QA (VQA) BIBREF42, BIBREF43, BIBREF44, text-to-image generation BIBREF45, BIBREF46, BIBREF47, and visual-language navigation BIBREF48 etc. In these tasks, natural language plays a key role in helping the machine to “understand” the content of the images, where “understand” means to capture the underlying correlations between the semantics embedded in language with the visual features obtained from the images. In addition to text, vision can be combined with speech as well. Such tasks include audio-visual speech recognition BIBREF49, BIBREF50, BIBREF51, speaker recognition BIBREF52, BIBREF53, BIBREF54, as well as speech diarisation BIBREF55, BIBREF56, separation BIBREF57, BIBREF58 and enhancement BIBREF59, which mostly focused on the use of visual features to improve the robustness of the audio-only methods.
In this paper, a technical review of the models and learning methods for multimodal intelligence is provided. The main focus is the combination of CV and NLP, which has become an important area for both research communities covering many different tasks and technologies. To provide a more structured perspective, we organize the methods selected in this technical review according to three key topics: representation, fusion, and applications.
Learning representations for the input data is a core problem for deep learning. For multimodal tasks, collecting paralleled data across all modalities can be quite difficult and leveraging pre-trained representations with desired properties, such as suitable for zero-shot or few-shot learning, is often an effective solution to the issue. Both supervised and unsupervised training based multimodal representation learning methods are reviewed.
The fusion of the features or representations of the single modalities is undoubtedly a centric problem of any multimodal task. Different from previous studies that often categorise the related work into early, middle and late stage methods based on the stage that fusion happens in the procedure, we classify them according to the actual operation used in the fusion, such as attention and bilinear pooling, since it becomes difficult to classify some recent complex approaches into stages.
Three types of applications are reviewed in this paper, namely image captioning, text-to-image synthesis and VQA. This is to give an idea how representation learning and fusion can be applied to specific tasks, and to provide a viewpoint of the situation of the current development of the multimodal applications, especially those integrating vision with natural languages. Visual reasoning methods for VQA are also discussed in the end.
This paper is organised as follows. Section SECREF2 reviews the recent progress on developing representations for single or multiple modalities. Section SECREF3 introduces the commonly used fusion methods, particularly attention and bilinear pooling. Applications including caption generation, text-to-image synthesis, VQA, and visual reasoning are introduced in Section SECREF4, followed by conclusions.
The main focus is the combination of CV and NLP, which has become an important area for both research communities covering many different tasks and technologies.
Answer: The combination of CV and NLP.