# Deep Vision Multimodal Learning: Methodology, Benchmark, and Trend
## 2 Learning Architectures
### Modality Aggregation  
After extracting multimodal features, it is important to aggregate them together. The more typical fusion methods [25; 26] are divided into early fusion and late fusion. Early fusion refers to directly splicing the multimodal input signals and then sending them into a unified network structure for training. Late fusion [27; 28] refers to the fusion of the features of each modal signal after feature extraction. Finally, the network structure at the end is designed according to the different downstream tasks. As early and late fusion inhibit intra- and intermodal interactions, current research focuses on intermediate fusion methods, which allow these fusion operations to be placed in multiple layers of deep learning models. Different modalities can be fused simultaneously into a shared presentation layer or executed incrementally using one or more modalities at a time. Depending on the specific task and network architecture, the multidodal fusion method can be very flexible. [PERSON] et al. [29] uses a \"slow-fusion\" network where extracted video stream features are gradually fused across multiple multimodal fusion layers, which performs better in a large-scale video stream classification task. [PERSON] et al. [30] consider the difference in information level between modalities and fuse the information of different modalities step by step, which are visual input modalities, then motion input modals, then audio input modalities. [PERSON] et al. [31] propose a method to automatically search for an optimal fusion strategy regarding input data. In terms of specific fusion architecture, existing fusion methods can be divided into matrix- or MLP-based, attention-based, and other specific methods. **Matrix-based fusion**. [PERSON] et al. [32] use \"tensor fusion\" to obtain richer feature fusion methods with three types of tensors: unimodal, bimodal, and trimodal. [PERSON] et al. propose the low-rank multimodal fusion method, which improves efficiency based on \"tensor fusion\". [PERSON] et al. [33] integrate multimodal features by considering higher-order matrices. **MLP-based fusion**. [PERSON] et al. [34] propose a framework consisting of three parts: a compositional semantics language model, a deep video model, and a joint embedding model. In the joint embedding model, they minimize the distance of the outputs of the deep video model and compositional language model in the joint space and update these two models jointly. [PERSON] et al. [35] first encode all modalities, then use decoding to restore features, and finally calculate the loss between features. **Attention-based fusion**[36]. [PERSON] et al.use \"delta-memory attention\" and \"multiview gated memory\" to simultaneously capture temporal and intermodal interactions for better multiview fusion. The purpose of using the memory network is to save the multimodal interaction information at the last moment. In order to capture the interactive information between multimodalities and unimodality, the multi-interactive attention mechanism [37] is further used; that is, textual and visual will fuse information through attention when they cross the multiple layers. [PERSON] et al. [38] use a shared token between two transformers so that this token becomes the communication bottleneck of different modalities to save the cost of computational attention. In addition to these specific fusion methods, there is some auxiliary work. [PERSON] et al. [39] help find the best fusion architecture for the target task and dataset. However, these methods often suffer from the exponential increase in dimensions and the number of modalities. The low-rank multimodal fusion method [40] performs multimodal fusion using low-rank tensors to improve efficiency. [PERSON] et al. [41] propose a novel regularization term based on the functional entropy. Intuitively, this term encourages balancing each modality's contribution to the classification result.