Tackling Modality Heterogeneity with Multi-View Calibration Network for Multimodal Sentiment Detection
Year: 2023

Abstract:
With the popularity of social media, detecting sentiment from multimodal posts (e.g. imagetext pairs) has attracted substantial attention recently. Existing works mainly focus on fusing different features but ignore the challenge of modality heterogeneity. Specifically, different modalities with inherent disparities may bring three problems: 1) introducing redundant visual features during feature fusion; 2) causing feature shift in the representation space; 3) leading to inconsistent annotations for different modal data. All these issues will increase the difficulty in understanding the sentiment of the multimodal content. In this paper, we propose a novel Multi-View Calibration Network (MVCN) to alleviate the above issues systematically. We first propose a text-guided fusion module with novel Sparse-Attention to reduce the negative impacts of redundant visual elements. We then devise a sentiment-based congruity constraint task to calibrate the feature shift in the representation space. Finally, we introduce an adaptive loss calibration strategy to tackle inconsistent annotated labels. Extensive experiments demonstrate the competitiveness of MVCN against previous approaches and achieve state-of-the-art results on two public benchmark datasets.

heading: 1 Introduction
text: Multimodal sentiment detection(Xu, 2017; Xu and Mao, 2017) aims to explore the sentiment embedded in the multimodal contents, such as text, images, and videos. With the growth of social media, it shows broad applications for the understanding of one’s position, attitude, or opinion towards an entity, person, or topic, which has attracted substantial attention from both academic and industrial
∗Both authors contribute equally to this work. † Corresponding authors. Email:{yuanshaozu@jd.com,
longbiao wang@tju.edu.cn.}
communities. In this paper, we focus on detecting the sentiment of multimodal posts in social media. As shown in Figure 1, the model is required to infer the human sentiments given the image and text pairs.
Previous works mainly focus on how to integrate modalities and have achieved astonishing progress. Earlier works apply concatenation to fuse different features, such as Xu (2017); Xu and Mao (2017). To promote the fusion, Yang et al. (2020); Yu and Jiang (2019); Kumar and Vepa (2020); Xu et al. (2018) construct different modules to realize deeper multimodal interaction. However, these methods of viewing different modalities in the same light neglect the modality heterogeneity, hindering the performance of the model. Although CLMLF (Li et al., 2022) tries to apply contrastive learning to alleviate this problem, it is in coarse granularity and only considers feature level, which is insufficient to address the issue of feature shift.
Modality heterogeneity is mainly caused by the modality gap, which has been discussed in some related multimodal research (Hazarika et al., 2020; Lin and Hu, 2022; Liang et al., 2022; Radford et al., 2021). In multimodal sentiment detection, it is because image modality often has less information and more redundant features compared to text modality. As shown in Figure 1, only a few visual
5240
elements reveal sentiment information(e.g., people smile, ill dog), causing redundant visual elements and low information density for the sentiment. In contrast, text modality is more indicative of sentiment owing to higher information density (Tsai et al., 2019; Sun et al., 2021). Consequently, this heterogeneity across different modalities will bring difficulty in understanding the sentiment of multimodal content.
Concretely, the ignorance of modality heterogeneity will directly cause three problems: a) It takes sparse and noisy visual features into fusion and causes confusion in understanding sentimental information. b) Integrating modalities of different properties will further lead to the multimodal feature shift and makes the model capture spurious correlations between multimodal features and sentiments. c) As mentioned by Niu et al. (2016); Yang et al. (2021), it will affect the data annotators to vote inconsistent labels for unimodal, which leads to uncertain annotated labels and weaken the label confidence. Taking MVSA-Single (Niu et al., 2016) for example, approximately 47% of the samples suffer from inconsistent annotated labels.
To tackle the above problems systematically, we propose a Multi-View Calibration Network (MVCN) from three different views:
(1) To avoid sparse and redundant visual features of direct integrating modalities (Li et al., 2022), we propose a Text-Guided Fusion (TGF) module to leverage text data to dominate the fusion process. Specially, we propose Sparse-Attention mechanism using sparsemax (Martins and Astudillo, 2016) to automatically eliminate redundant visual features and capture the essential parts of the image with respect to the sentiment.
(2) To further calibrate the feature shift, we propose a Sentiment-based Congruity Constraint (SCC) task to restrain representation space. In the SCC task, we propose relative distance to gather multimodal features around the corresponding sentimental centroids estimated with samples’ labels. In addition, to overcome the limitation of the minibatch, we also introduce Accumulating Calibration (AC) strategy to accumulate sampling information, thus computing the sentiment centroids from a global perspective. Compared to contrastive learning (Li et al., 2022), SCC has more strength to calibrate the feature shift for it brings sentimental semantic labels from a global perspective.
(3) To alleviate uncertain annotated labels mis-
leading the model during the training stage, we also introduce an adaptive loss calibration (ALC) strategy to calibrate the training loss in the sentiment detection task, where the detection model is forced to be less confident for uncertain annotated labels. The experiments conducted on different benchmark datasets (Niu et al., 2016; Cai et al., 2019) show that MVCN has significantly improved the performance on all metrics compared with previous state-of-the-art models. In summary, the key contributions of this paper are as follows:
• We introduce a novel Multi-View Calibration Network (MVCN) including Text-Guided Fusion module, Sentiment-based Congruity Constraint, and Adaptive Loss Calibration strategy for multimodal sentiment detection to systematically solve the problems of modality heterogeneity from different views.
• The thorough experiments show that MVCN improves the performance over all metrics and achieves state-of-the-art on two benchmark datasets (Niu et al., 2016; Cai et al., 2019).

heading: 2 Methodology
text: The architecture of the proposed multi-view calibration network (MVCN) is shown in Figure 2. Generally, MVCN consists of text-guided fusion module and two paralleled sub-tasks. The two paralleled sub-tasks are sentiment classification and sentiment-based congruity constraint respectively.

heading: 2.1 Text-Guided Fusion Module
text: As shown in Figure 2, Text-Guided Fusion Module contains three components: Unimodal Encoders, Text-Guided Unit, and Reduction Unit.
Unimodal Encoders. To obtain visual and textual features for multimodal fusion, we apply two different unimodal encoders to extract their representations. For text modality, we use the pretrained BERT (Jacob Devlin, 2019) model as the text encoder to obtain the text representation. Given a sequence of text T = {t1, t2, ..., tnt}, where nt is the number of text length, the output of the BERT model can be defined as:
Xt = {EC , E1, E2, ..., Ent} = BERT (T ; θbertt ) (1)
where EC ∈ Rdt is the embedding of the CLS token and θbertt denotes the parameters of the BERT model. For image modality, we use the pretrained
ViT model (Dosovitskiy et al., 2020) as the image encoder to obtain the image representation. Given an image I , the output of the ViT model can be defined as:
Xi = {IC , I1, I2, ..., Ini} = V iT (I; θviti ) (2)
where ni is the length of the image representation and θV iTt denotes the parameters of the ViT model.
Text-Guided Unit (TGU). Taking the extracted image features Xi and the text features Xt as inputs, we perform multimodal fusion by passing the inputs through N layers stacking Text-Guided Unit (TGU). For each layer of TGU, Xt is first fed into a self-attention (α) as query, key, and value to generate text-aware feature Xf . We then input Xi and Xf to a Sparse-Attention (γ) layer to obtain text-guided visual sparse features Xg, and finally apply a feed-forward network (FFN) for Xg to produce the TGU output. Here, we design Sparse-Attention in two places: (1) We use the textual features to attend to visual features and get text-guided features. (2) We normalize attention weight with sparsemax (Martins and Astudillo, 2016), thus obtaining sparse posterior attention weights, where the weight for redundant visual features are set to 0. Moreover, inspired by previous works (Rahman et al., 2019; Yuan et al., 2022) that leverage pretrained language model to enhance the ability of capturing multimodal context, we initialize TGU with the weight of the pretrained BERT. In the formula, the output of N th layer TGU can be defined as: TGU(Xt, Xi)
N = [Xf = α(Xt, Xt, Xt), Xg = γ(Xf , Xi, Xi), FFN(Xg)].
Reduction Unit (RU). To get the multimodal representation Q ∈ Rdt for sentiment classification, we follow Li et al. (2022) and use stacked attention layer and fully-connected layer with GELU activate function (Hendrycks and Gimpel, 2016) as Reduction Unit to perform dimensionality reduction for the features.

heading: 2.2 Sentiment Congruity Constraint (SCC)
text: Intuitively, samples’ features Q with the same sentimental labels should be closer in the representation space. However, feature shift caused by modality heterogeneity hinders this trend and makes it difficult to capture correlations between multimodal features and sentiments. To address this problem, we take advantage of label information to estimate sentimental centroids, then restrict the distance between the samples and their centroids, thus calibrating the feature shift.
Here, we first employ a fully-connected layer to map multimodal representation Q to normalized representations Q́ for the SCC task. Then, we compute the positive centroid (C0), the neutral centroid (C1) and the negative centroid (C2) of the representation Q́ during training stage. Let 0,1,2 denote the positive label, neutral label, and negative label, the formula for computing centroids is defined as:
C(0|1|2) = ∑B j=1 I(Y (j) = (0|1|2)) · Q́j∑B
j=1 I(Y (j) = (0|1|2)) (3)
where B is the number of all the candidate computing samples, I(.) is an indicator function, and Q́j and Y (j) are the representation and Ground-Truth label respectively of the j-th sample.
To restrict the distance between the samples and their corresponding centroids, an intuitive idea is to minimize absolute distance, such as L2 loss. However, we find it causes all the samples to get too close to their centroids and completely eliminates data distribution, which makes the training hard to converge. To tackle this issue, we propose a relative L2 loss to measure the distance. Here, we first calculate the absolute L2 distance D = {d1, ..., dB} between the centroid C and the multimodal representation Q́ for a batch:
D = {∥Q́i − CY (i)∥22}Bi=1√
ι (4)
where Y (i) represents the ground-truth label of the ith sample, ι is the feature dimension, and √ ι serve as a scale factor. We then normalize the absolute distance to a relative distance and optimize the SCC task with the following formula:
Lscc = − B∑
i=1 log( exp(−di)∑B i=1 exp(−di) ) (5)
Accumulating calibration strategy. Nevertheless, there still exist two problems to optimize the SCC task because of the limitation of the minibatch. On the one hand, computing centroid for iterating mini-batch results in a frequently updated centroid. On the other hand, the samples in a minibatch are insufficient to estimate an accurate centroid. To solve this problem, we propose an accumulating calibration strategy to enlarge the computing space and narrow the change for the samples.
We first employ an auxiliary accumulating TGF module (denoted as TGF’) to produce sufficient representations Qm as candidate computing samples for centroids in advance. To accumulate computing information, we then build a queue to restore all the representations. The queue is dynamically updated by replacing the premier mini-batch with the current mini-batch during training for each iteration. Thus, we can estimate stable and slowupdated centroids with Equation 3 from a more global perspective. Here, to guarantee training stability, we leverage the momentum optimization (He et al., 2020; Li et al., 2021) to slowly update TGF’, which can be defined as:
θm ← βθm + (1− β)θt (6)
where θm denotes the parameters of the accumulating TGF’, θt denotes the parameters of the TGF module and β ∈ [0, 1) is a balance parameter.

heading: 2.3 Sentiment Classification (SC)
text: For the sentiment classification task, we feed multimodal representation Q ∈ Rdt into the fully connected layer with softmax function to predict the logits. However, directly optimizing this task with Cross Entropy (CE) as previous work still suffers from the issue of uncertain annotated labels during the training stage.
Adaptive loss calibration (ALC) strategy. To overcome the above issue, we design a simple but effective strategy, adaptive loss calibration. It forces the detection model to decrease confidence for the training examples of inconsistent annotated labels, thus calibrating the loss. To better elaborate the strategy, we first define cross entropy loss utilized in most previous works. Suppose that pi ∈ {p1, p2, ..., pK} denotes the predicted logit and yi ∈ {y1, y2, ..., yK} represents the ground-truth, CE loss can be defined as L = −∑Ki yilog(pi), where yi ∈ {0 , 1} , pi ∈ [0 , 1] and K denotes the number of categories. Different from label smoothing (Szegedy et al., 2016), we first leverage unimodal labels provided by the datasets to adaptively adjust the confidence factor α ∈ {0, 0.1}, then normalize the ground-truth probability. Intuitively, it makes the model becomes confident about its ground-truth by setting α as 0. In the formula, we normalize the ground-truth probability ŷi for each label as:
ŷi = (1− α) · I(yi = 1) + ( α
K − 1) · I(yi = 0) (7)
where I(.) is a indicator function. Moreover, the loss function Lsc can be reformatted as:
Lsc = ∑
((1− α) ∗ Li · I(yi = 1) +α · Li · I(yi = 0))
(8)

heading: 2.4 Training Loss
text: We optimize the above two tasks with total loss L:
L = λscLsc + λsccLscc (9)
where λsc and λscc are hyper-parameters to balance the different training losses.

heading: 3 Experiments
text: In this section, we first introduce the experimental setup and report the experimental results, then conduct the ablation study and visualization analysis.

heading: 3.1 Datasets
text: All our experiments were conducted on three public datasets: MVSA-Single (Niu et al., 2016), MVSAMultiple (Niu et al., 2016) and HFM (Cai et al., 2019). Here, we give a brief introduction to these datasets and dataset statistics are shown in Table 2.
MVSA-Single, MVSA-Multiple. MVSASingle and MVSA-Multiple are popular text-image sentiment datasets crawled from Twitter, where MVSA-Multiple is an upgraded version of MVSASingle and contains more text-image pairs. Both of them have three categories: positive, neutral, and negative. For fair comparison, the two MVSA datasets are processed in the same way as Xu and Mao (2017).
HFM. HFM has two sentimental categories: positive and negative. We follow Cai et al. (2019) and adopt the same preprocessing method for experiments, which has been widely used in previous works.

heading: 3.2 Implementation Details
text: For feature extraction, we use 12-layer visual transformer ViT-B/16 (Dosovitskiy et al., 2020) as visual encoder and BERT-base (Jacob Devlin, 2019) as textual encoder. The text-guided fusion encoder is composed of 6 stacked text-guided units. In ALC, the confidence factor is set as 0 or 0.1 according to unimodal labels. For Sentiment Congruity Constraint, the queue sizes of accumulating calibration are set as 3611, 13624, and 19816 for MVSASingle, MVSA-Multiple, and HFM respectively.
During the training stage, the learning rate is set to 2e-5. We train the model for 10 epochs
1Considering CLMLF brings extra data for data augmentation, to evaluate capacity of the model, we also report the results without extra data for fair comparison.
with batch size of 16, 32, and 32 for MVSASingle, MVSA-Multiple, and HFM. In addition, we adopt AdamW optimizer with ϵ of 1e− 8 and β of (0.9, 0.999). For loss function, both hyperparameters λsc and λcc are set to 1.0. Following previous settings, we adopt ACC and Weighted F1 as the evaluation metrics for MVSA datasets and Macro-F1 and ACC for HFM to evaluate the performance of the model.

heading: 3.3 Baselines
text: To fully validate the performance of MVCN, we select both unimodal and multimodal baselines.
Unimodal Baselines. For text modality, we choose CNN (Kim, 2014), Bi-LSTM (Zhou et al., 2016) and BERT (Jacob Devlin, 2019) as baselines since they are popular models for text classification. For image modality, ResNet (He et al., 2016) and ViT (Dosovitskiy et al., 2020) are selected for their superior capability for image classification.
Multimodal Baselines. For MVSA-∗ datasets, the compared baselines include: MultiSentiNet (Xu and Mao, 2017) that designs an attentionbased semantic network for multimodal sentiment analysis; HSAN (Xu, 2017) applying a hierarchical semantic attentional network for multimodal sentiment analysis; Co-MN-Hop6 (Xu et al., 2018), employing a co-memory network to iteratively model the interactions between multiple modalities; MGNNS (Yang et al., 2021) utilizing a multi-channel graph neural networks with sentiment-awareness for image-text sentiment detection. CLMLF (Li et al., 2022) is the previous SOTA model that aligns and fuses the text and image modalities through contrastive learning. For HFM datasets, we compare two variants of Concat (Schifanella et al., 2016): Concat(2) concatenates text and image, while Concat(3) brings extra image attribute features; MMSD (Cai et al., 2019) fusing text, image, and image attributes with a multimodal hierarchical framework; and D&R Net (Xu et al., 2020) that fuses text, image, and visual attributes by the decomposition and relation network.

heading: 3.4 Main Results
text: The comparison between MVCN and the baselines is demonstrated in Table 1. Obviously, the textonly model is more competitive with image-only methods. It indicates that the image modality is less helpful and contains more redundant features compared to text modality, which supports our intuition of eliminating redundant visual features. In
addition, the multimodal models surpass the unimodal models because of fusing more information. Overall, MVCN achieves state-of-the-art with a considerable performance gain over other methods, which indicates the necessity of tackling modality heterogeneity from different views. Specially, we find that MVCN achieves better results on MSVASingle compared to the other two datasets. We conjecture that small dataset suffers more of the modality heterogeneity problem due to lack of data diversity.

heading: 3.5 Ablation Study
text: To investigate the effectiveness of each module, we conduct an ablation study in Table 3. Firstly, compared to the MFS model that equally fuses the image and text features, it is straightforward that TGF module can aid sentiment detection since it eliminates redundant visual features. In addition, the model equipped with Sentiment-based Congruity Constraint (SCC) brings a significant improvement,
implying the importance of calibrating feature shift with congruity constraint. And accumulating calibration (AC) strategy by additionally augmenting SCC with more accurate and stable centroids, consistently improves performance. Furthermore, it can be observed that ALC strategy can further boost performance, demonstrating ALC is an effective way to reduce the impact of the uncertain annotated labels. Finally, MVCN equipped with all novel modules achieves the best performance, illustrating the effectiveness of all the above modules.

heading: 4 Analysis

heading: 4.1 Discussion of Components
text: Variants of Text-Guided Unit. To verify the design for Text-Guided Unit in TGF, we evaluate the performance of TGU variants in Table 4. Here, “w/ softmax” denotes Self-Attention, while “w/ spm” represents we normalize attention weight with sparsemax. “w/ pretrain” and “w/o pretrain” indicate whether the guided unit initialized with the pretrained BERT weight or not. From the table, we observe “w/ spm” shows superiority compared to “w/ softmax”, indicating Sparsemax-Attention boosts the performance of the model by eliminating noisy visual features. Additionally, “w/ pretrain” can further promote the performance, which is consistent with previous works that pretrained language models can enhance the capacity for capturing multimodal context. Effectiveness of SCC. In Figure 3, we plot the curves during training to explore why we apply relative L2 distance to optimize SCC. From Figure 3 (a), “w/ SCC(aL2)” that applies absolute L2 dis-
TGF MVSA-Single HFM Acc F1 Acc F1 w/ softmax 0.7365 0.7339 0.8476 0.8427 w/ spm 0.7396 0.7355 0.8493 0.8451 w/o pretrain 0.7232 0.7148 0.8306 0.8217 w/ pretrain 0.7396 0.7355 0.8493 0.8451
Table 4: Ablation results of TGF and we remove ALC and SCC modules here.
tance dramatically narrows the distance between the samples and the centroids to an extremely low level. However, it hinders the training and eventually leads to the model being hard to optimize in Figure 3 (b). Because it will completely eliminate data distribution and make the data samples lose correlation with each other by directly minimizing L2 distance. Conversely, “w/ SCC(aL2)” that utilizes relative L2 distance can reduce the semantic distance to a reasonable extent, and more importantly, maintain the data distribution. Therefore, it ensures the SCC task can be optimized and the accuracy of “w/ SCC(rL2)” in Figure 3 (b) gradually increases during training.
Different Loss Calibration Strategies. To explore the effectiveness of ALC, we compare different loss calibration strategies in Table 5. It shows label smoothing (LS) only brings slight improvement since it avoids overfitting and strengthens the robustness of the model. However, LS can not handle the label problem caused by modality heterogeneity. Compared to the LS, our proposed ALC exhibits superiority and outperforms LS in performance, verifying it is necessary to mitigate the adverse effects of inconsistent labels.
Dataset Strategy ACC F1
MSVD-Single - 0.7563 0.7422
LS 0.7579 0.7436 ALC 0.7606 0.7455
Table 5: Results with adaptive loss calibration (ALC) and label smoothing (LS). Note − represents the model without any strategy.

heading: 4.2 Visualization
text: Sparse-Attention Visualization. To verify the advantage of Sparse-Attention in the TGF module, we visualize the attention weight in Figure 4. Compared to Self-Attention, the sampling cases show Sparse-Attention captures the essential parts of the image with respect to the sentiment and meanwhile attenuates the negative effect of redundant visual features. As the example in Figure 4(a), the model paid more attention on the “ill dog” in the image for it reflects negative sentiment, certifying that the model can focus the sentimental regions in the image and avoid the interference of irrelevant objects. This also confirms it is necessary to eliminate redundant visual features, reinforcing the importance of Sparse-Attention. Feature Distribution Visualization. In order to visually demonstrate the superiority of SCC task with AC strategy, we visualize the feature distribution on the Multiple-Single dataset with contrastive learning (Li et al., 2022) and SCC. Here, we apply T-SNE2 algorithm to perform dimensionality reduction for the feature, obtaining a 2-dimensional feature vector distribution visualized in Figure 5. From Figure 5(b), we observe that the SCC task forces samples belonging to the same category to
2https://github.com/mxl1990/tsne-pytorch
gather around their corresponding centroids. Conversely, Figure 5(a) shows that when we remove SCC task, the degree of data aggregation is less obvious. The reason for this phenomenon is that, compared to contrastive learning, the SCC taking sentimental labels into consideration to constrain the distribution from a more global perspective, is more capable of calibrating the feature shift. Hence, this strength facilitates the MVCN to learn the distinguished features in representation space and improve the performance of the model.

heading: 4.3 Case Study
text: In Figure 6, we conduct a case study for MVCN and the previous SOTA model CLMLF. We find that for these complicated and confusing cases, it is difficult for CLMLF to capture the user’s sentiment of samples because of its limitations in handling modality heterogeneity. For example, in the first case, although the phrases (e.g., vibrant colors) in texts represent specific sentiments, CLMLF was still misled by sparse visual sentimental informa-
tion. For the following two cases in Figure 6, we find the text may express negative sentiments, and images contain elements (e.g., serious man, smiling man) that reflect sentiments. However, CLMLF fails to predict the real sentiment due to modality heterogeneity. In contrast, our model can distinguish these cases for the corresponding sentiment, which also proves the advantage of tackling modality heterogeneity from different views.

heading: 5 Related Work
text: In this section, we introduce the background of multimodal sentiment detection and modality heterogeneity.

heading: 5.1 Multimodal Sentiment Detection
text: Multimodal sentiment detection has become a significant research topic and previous works mainly focus on fusing multimodal features with different strategies. Some earlier works fuse the multimodal features by concatenation, such as HSAN (Xu, 2017) and MultiSentiNet (Xu and Mao, 2017). Later relevant works mainly focus on how to integrate modalities and promote better inter-modal interaction. For example, CoMN (Xu et al., 2018) designs a co-memory network to iteratively model the interactions between image and text features, TomBERT (Yu and Jiang, 2019) fuses the multimodal representations with a bilinear interaction layer, Kumar and Vepa (2020) proposes to use gating mechanism and attention to perform deep multimodal interaction, and MVAN (Yang et al., 2020) adapts memory network to fuse the multimodal features via perceptron and stacking-pooling module. Recent work (Li et al., 2022) applies contrastive learning and data augmentation to handle this problem. Different from these works, we aim to im-
prove this task by tackling modality heterogeneity from multiple views.

heading: 5.2 Modality Heterogeneity
text: Modality heterogeneity is reflected in that multiple modalities show different properties. Typically, heterogeneity usually exists in human-generated signals (language) with high information density and other natural signals modalities (e.g., image, video, audio) with heavy redundancy. In the past few years, some works have been done in multimodal representation learning to alleviate the impact of modality heterogeneity. Hazarika et al. (2020) project different modalities to modality-invariant representation space and learn their commonalities. Furthermore, Wu et al. (2021); Zhao et al. (2021) employs the modality translation method to convert the source modality to the target one in order to learn the commonalities between the different modalities. However, due to the huge modality gap, it is insufficient to handle multimodal heterogeneity by projecting different modal features into the same representation space. To overcome the modality gap and encourage learning useful features, Li et al. (2022); Lin and Hu (2022) design various contrastive learning tasks to predict cross-modal representation in an implicit way. However, they are feature-level alignment and in coarse granularity, which is not sufficient to address the problem of feature shift.

heading: 6 Conclusion
text: In this paper, we present a Multi-View Calibration Network (MVCN) to address the modality heterogeneity problem for text-image sentiment detection from different views. Specially, we respectively introduce the text-guided fusion module to calibrate multimodal fusion and reduce the negative impacts of redundant visual elements, a sentiment-based congruity constraint task to further calibrate the feature shift in representation space and an adaptive loss calibration strategy to calibrate the training loss in terms of uncertain annotated labels. The thorough experiments show the MVCN achieves state-of-the-art performance on two benchmark datasets.
Limitations
Considering Modality Heterogeneity can promote many related multimodal applications, it is worth continually exploring. In this paper, we propose
Text-Guided Fusion (TGF) module equipped with Sparse-Attention to integrate different modalities in representation aspects, which is an implicit way to build the relations of fine-grained features, such as visual objects, and textual words. Previous work (Khademi, 2020; Wang et al., 2020) has proven that Graph Convolutional Network (GCN) (Scarselli et al., 2008) shows advantages in modeling the relations among visual and textual elements. Inspired by these works, we argue that explicitly introducing the relationship of fine-grained features via GCN can better guide the model to eliminate redundant features. Thus it can further narrow the modalities gap and facilitate fusion for multimodal content understanding. In the future, we will bring GCN to learn multimodal relationships and boost the performance of the model.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 6

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 3

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Scetion 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 3

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 3
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section 3
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The data that was collected/used doesn’t contain any information that names or uniquely identifies individual people or offensive content.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 3.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 3.1
C 3 Did you run computational experiments? Section 3
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 3.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 3.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? we don’t use human annotators.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
7 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? we don’t have any paid participants.
7 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? we don’t collect data via crowdsourcing.
7 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No ethics review board was involved
7 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Our data doesn’t include any protected information.

heading: Future Work
text: In the future, we will bring GCN to learn multimodal relationships and boost the performance of the model