Query: Multimodal fusion survy

title: Multiscale multilevel context and multimodal fusion for RGB D salient object detection
abstract: Red green blue and depth (RGB D) saliency detection has recently attracted much research attention; however, the effective use of depth information remains challenging. This paper proposes a method that leverages depth information in clear shapes to detect the boundary of salient objects. As context plays an important role in saliency detection, the method incorporates a proposed end to end multiscale multilevel context and multimodal fusion network (MCMFNet) to aggregate multiscale multilevel context feature maps for accurate saliency detection from objects of varying sizes. Finally, a coarse to fine approach is applied to an attention module retrieving multilevel and multimodal feature maps to produce the final saliency map. A comprehensive loss function is also incorporated in MCMFNet to optimize the network parameters. Extensive experiments demonstrate the effectiveness of the proposed method and its substantial improvement over state of the art methods for RGB D salient object detection on four representative datasets.

title: Deep multimodal fusion for semantic image segmentation: A survey
abstract: Recent advances in deep learning have shown excellent performance in various scene understanding tasks. However, in some complex environments or under challenging conditions, it is necessary to employ multiple modalities that provide complementary information on the same scene. A variety of studies have demonstrated that deep multimodal fusion for semantic image segmentation achieves significant performance improvement. These fusion approaches take the benefits of multiple information sources and generate an optimal joint prediction automatically. This paper describes the essential background concepts of deep multimodal fusion and the relevant applications in computer vision. In particular, we provide a systematic survey of multimodal fusion methodologies, multimodal segmentation datasets, and quantitative evaluations on the benchmark datasets. Existing fusion methods are summarized according to a common taxonomy: early fusion, late fusion, and hybrid fusion. Based on their performance, we analyze the strengths and weaknesses of different fusion strategies. Current challenges and design choices are discussed, aiming to provide the reader with a comprehensive and heuristic view of deep multimodal image segmentation.

title: Multimodal Fusion for Objective Assessment of Cognitive Workload: A Review
abstract: Considerable progress has been made in improving the estimation accuracy of cognitive workload using various sensor technologies. However, the overall performance of different algorithms and methods remain suboptimal in real world applications. Some studies in the literature demonstrate that a single modality is sufficient to estimate cognitive workload. These studies are limited to controlled settings, a scenario that is significantly different from the real world where data gets corrupted, interrupted, and delayed. In such situations, the use of multiple modalities is needed. Multimodal fusion approaches have been successful in other domains, such as wireless sensor networks, in addressing single sensor weaknesses and improving information quality/accuracy. These approaches are inherently more reliable when a data source is lost. In the cognitive workload literature, sensors, such as electroencephalography (EEG) electrocardiography (ECG) and eye tracking, have shown success in estimating the aspects of cognitive workload. Multimodal approaches that combine data from several sensors together can be more robust for real time measurement of cognitive workload. In this article, we review the published studies related to multimodal data fusion to estimate the cognitive workload and synthesize their main findings. We identify the opportunities for designing better multimodal fusion systems for cognitive workload modeling.

title: Deep Binocular Fixation Prediction using a Hierarchical Multimodal Fusion Network
abstract: RGB D data are increasingly being used for myriad computer vision tasks. For such tasks, most methods simply concatenate or add feature vectors from RGB images and depth maps and allow the two modalities to complement each other mutually. However, such a fusion strategy results in inefficient and inadequate performance. In this study, we propose deep binocular fixation prediction based on a hierarchical multimodal fusion network that suitably combines RGB and depth maps. In the proposed method, a novel convolutional block attention module completely extracts image texture features and retains spatial information. In addition, a pyramid dilated convolution module refines feature information, further improving the fusion of RGB and depth maps. Experimental results indicate that the proposed network achieves state of the art performance on the NUS and NCTU datasets.

title: MinD at SemEval 2021 Task 6: Propaganda Detection using Transfer Learning and Multimodal Fusion
abstract: We describe our systems of subtask1 and subtask3 for SemEval 2021 Task 6 on Detection of Persuasion Techniques in Texts and Images. The purpose of subtask1 is to identify propaganda techniques given textual content, and the goal of subtask3 is to detect them given both textual and visual content. For subtask1, we investigate transfer learning based on pre trained language models (PLMs) such as BERT, RoBERTa to solve data sparsity problems. For subtask3, we extract heterogeneous visual representations (i.e. face features, OCR features, and multimodal representations) and explore various multimodal fusion strategies to combine the textual and visual representations. The official evaluation shows our ensemble model ranks 1st for subtask1 and 2nd for subtask3.

title: 3D Point of Intention Determination Using a Multimodal Fusion of Hand Pointing and Eye Gaze for a 3D Display
abstract: This paper proposes a three dimensional (3D) point of intention (POI) determination method using multimodal fusion between hand pointing and eye gaze for a 3D virtual display. In the method, the finger joint forms of the pointing hand sensed by a Leap Motion sensor are first detected as pointing intention candidates. Subsequently, differences with neighboring frames, which should be during hand pointing period, are checked by AND logic with the hand pointing intention candidates. A crossing point between the eye gaze and hand pointing lines is finally decided by the closest distance concept. In order to evaluate the performance of the proposed method, experiments with ten participants, in which they looked at and pointed at nine test points for approximately five second each, were performed. The experimental results show the proposed method measures 3D POIs at 75 cm, 85 cm, and 95 cm with average distance errors of 4.67% 5.38% and 5.71% respectively.

title: Multimodal Fusion Refiner Networks
abstract: Tasks that rely on multi modal information typically include a fusion module that combines information from different modalities. In this work, we develop a Refiner Fusion Network (ReFNet) that enables fusion modules to combine strong unimodal representation with strong multimodal representations. ReFNet combines the fusion network with a decoding/defusing module, which imposes a modalitycentric responsibility condition. This approach addresses a big gap in existing multimodal fusion frameworks by ensuring that both unimodal and fused representations are strongly encoded in the latent fusion space. We demonstrate that the Refiner Fusion Network can improve upon performance of powerful baseline fusion modules such as multimodal transformers. The refiner network enables inducing graphical representations of the fused embeddings in the latent space, which we prove under certain conditions and is supported by strong empirical results in the numerical experiments. These graph structures are further strengthened by combining the ReFNet with a Multi Similarity contrastive loss function. The modular nature of Refiner Fusion Network lends itself to be combined with different fusion architectures easily, and in addition, the refiner step can be applied for pre training on unlabeled datasets, thus leveraging unsupervised data towards improving performance. We demonstrate the power of Refiner Fusion Networks on three datasets, and further show that they can maintain performance with only a small fraction of labeled data.

title: A central multimodal fusion framework for outdoor scene image segmentation
abstract: Robust multimodal fusion is one of the challenging research problems in semantic scene understanding. In real world applications, the fusion system can overcome the drawbacks of individual sensors by taking different feature representations and statistical properties of multiple modalities (e.g. RGB depth cameras, multispectral cameras) In this paper, we propose a novel central multimodal fusion framework for semantic image segmentation of road scenes, aiming to effectively learn joint feature representations and optimally combine deep neural networks with statistical priors. More specifically, the proposed fusion framework can automatically generate a central branch by sequentially mapping multimodal features into a common space, including both low level and high level features. Besides, in order to reduce the model uncertainty, we employ statistical fusion to compute the final prediction, which leads to significant performance improvement. We conduct extensive experiments on various outdoor scene datasets. Both qualitative and quantitative experiments demonstrate that our central fusion framework achieves competitive performance against existing multimodal fusion methods.

title: Interpretable Multimodal Fusion Networks Reveal Mechanisms of Brain Cognition
abstract: The combination of multimodal imaging and genomics provides a more comprehensive way for the study of mental illnesses and brain functions. Deep network based data fusion models have been developed to capture their complex associations, resulting in improved diagnosis of diseases. However, deep learning models are often difficult to interpret, bringing about challenges for uncovering biological mechanisms using these models. In this work, we develop an interpretable multimodal fusion model to perform automated diagnosis and result interpretation simultaneously. We name it Grad CAM guided convolutional collaborative learning (gCAM CCL) which is achieved by combining intermediate feature maps with gradient based weights. The gCAM CCL model can generate interpretable activation maps to quantify pixel level contributions of the input features. Moreover, the estimated activation maps are class specific, which can therefore facilitate the identification of biomarkers underlying different groups. We validate the gCAM CCL model on a brain imaging genetic study, and demonstrate its applications to both the classification of cognitive function groups and the discovery of underlying biological mechanisms. Specifically, our analysis results suggest that during task fMRI scans, several object recognition related regions of interests (ROIs) are activated followed by several downstream encoding ROIs. In addition, the high cognitive group may have stronger neurotransmission signaling while the low cognitive group may have problems in brain/neuron development due to genetic variations.

title: Multimodal Fusion for Image and Text Classification with Feature Selection and Dimension Reduction
abstract: Internet has become an important information platform, and it is very important to accurately understand the multimedia information of the Internet. In this paper, our main task is to do classification based on pictures and texts collected from the Internet, which is a classification problem of multimodal fusion in practice. However, when multimodal data is put together, there may occur the dimension disaster problem. We apply feature selection (FS) and dimension reduction (DR) in feature levels both in later fusion and early fusion to solve this problem. The classification accuracies in different models obtain improvements in different levels respectively. We also discuss the relation between single modals and multimodal in later fusion. In our experiments, images and text can be classified by multimodal models under FS/DR, and of which with the help the multimedia information from the Internet can be analysed better to help enterprises provide better services and products, and then carry out better network marketing and promotion.