# Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend
## 8 Conclusions  
This paper discusses the methodology, benchmark, and trend of deep vision multimodal learning, which is supported by references to some of the more influential papers of recent years.  
The methodology is divided into learning architecture, learning paradigms, and multimodal data analysis. In terms of learning architectures, we looked at different steps of a typical deep learning process: feature extraction, modality aggregation, and multimodal loss function. We are surprised that the transformer-based models have played a significant role in multimodal learning in recent years. There is endless potential to exploit the unity and flexibility of the attention mechanism across multiple modalities. As for loss function, most of the loss functions for multimodal learning are extended by some previous unimodality loss functions, which are complemented according to the cyclic consistency or alignment between modalities.  
We considered some common learning paradigms in deep multimodal learning, including semi-supervised learning, self-supervised learning, and transfer learning. We find that the mutual complementation and alignment of modalities can serve as good alternatives when fully supervised signals are not available. Multimodal transfer learning is more robust than single modality. We also found that knowledge transfer between modalities can be achieved utilizing knowledge distillation, etc., which is an important research direction.  
In terms of multimodal data analysis, we focus on two kinds of data issues that multimodal learning may have in a practical situation: missing modalities and noisy modalities. These two cases reflect the imbalance between the modes in different aspects. Not much work has been done in this direction, and it is a development focus on how to utilize strong and weak modalities in multimodal learning reasonably.  
For multimodal learning applications and benchmarks, we collect mainstream tasks in the vision-language multimodal learning field, including media captioning, visual question answers, multimodal machine translation, text-image retrieval, and text-to-image generation. These tasks are made up of combinations of modalities and are of practical value. For each task, we list the widely used datasets and some state-of-the-art models on them and analyze the reasons for the success of these models. We hope that more meaningful tasks and datasets will be published.  
After our thorough research, we have proposed some possible future research directions in the deep vision multimodal learning field: pretraining strategy, unified framework, missing and noisy modality, and multimodal task diversity. We believe that a unified large-scale multitask pretraining framework will become mainstream, as already demonstrated in natural language processing. Additionally, as we mentioned before, the task of making fuller use of unbalanced modal information and more diversity should be given full attention.  
In conclusion, this review examines some of the influential work in deep vision multimodal learning in recent years and explores future trends. We hope this review will be helpful for beginners or researchers entering the field. Multimodal learning will be a significant trend of deep learning in the future, and more interested people are needed to get involved.  
Conceptualization, W.C. and G.W.; Writing--original draft, W.C.; Writing--review & editing, W.C.; Supervision, G.W. All authors have read and agreed to the published version of the manuscript.  
Funding: This research received no external funding. Institutional Review Board Statement: Not applicable. Informed Consent Statement: Not applicable. Conflicts of Interest: The authors declare no conflict of interest.