Uncertainty-Guided Modal Rebalance for Hateful Memes Detection (2024)

Abstract: Hateful memes detection is a challenging multimodal understanding task that requires comprehensive learning of vision, language, and cross-modal interactions. Previous research has focused on developing effective fusion strategies for integrating hate information from different modalities. However, these methods excessively rely on cross-modal fusion features, ignoring the modality uncertainty caused by the contribution degree of each modality to hate sentiment and the modality imbalance caused by the dominant modality suppressing the optimization of another modality. To this end, this paper proposes an Uncertainty-guided Modal Rebalance (UMR) framework for hateful memes detection. The uncertainty of each meme is explicitly formulated by designing stochastic representation drawn from a Gaussian distribution for aggregating cross-modal features with unimodal features adaptively. The modality imbalance is alleviated by improving cosine loss from the perspectives of intermodal feature and weight vectors constraints. In this way, the suppressed unimodal representation ability in multimodal models would be unleashed, while the learning of modality contribution would be further promoted. Extensive experimental results demonstrate that the proposed UMR produces the state-of-the-art performance on four widely-used datasets. Disclaimer: This paper contains discriminatory content that may be disturbing to some readers.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4361–4371 August 11-16, 2024 ©2024 Association for Computational Linguistics
Disclaimer: This paper contains discriminatory content that may be disturbing to some readers.

heading: 1 Introduction
text: Memes, a form of user-generated content on social media platforms, have become a prevalent way for expressing opinions. Generally, memes consist of an image paired with a humorous caption. However, against a backdrop of current political and socio-cultural fragmentation, a sharply increasing
* Corresponding author
number of individuals are exploiting this format to propagate hate content on platforms by adeptly combining image with text. Therefore, detecting and curbing hateful memes is a particularly urgent research issue.
Previous research on hateful memes detection has employed pre-trained vision-language models for learning vision, language, and cross-modal interactions comprehensively (Das et al., 2020; Muennighoff, 2020; Zhou et al., 2021). Meanwhile, sophisticated fusion techniques (Kiela et al., 2020; Lee et al., 2021; Yang et al., 2023) and external knowledge enhancement methods (Zhu, 2020; Yang et al., 2022; Cao et al., 2022, 2023) have been proposed to further learn the discriminative features of memes. Although the above studies have produced promising progress, they excessively rely on multimodal fusion features, where the inherent uncertainty and imbalance between modalities have not been explicitly considered.
The modality uncertainty is caused by the contribution degree of each modality to hate sentiment. As illustrated in Figure 1, the text in the left meme narrates an incredible story, while the image shows two smiling individuals. The text and image convey completely opposite sentiments. In this case, the
4361
multimodal fusion feature can provide additional discriminative information to detect hate sentiment against religion in the meme. Conversely, the text and image in the right meme convey a consistent sentiment, allowing the meme to be identified as non-hateful. However, the learning of the crossmodal fusion features would cause an interaction between black in the text and woman in the image, leading the model to incorrectly classify it as sexist (Lee et al., 2021). Therefore, quantifying the inherent uncertainty is crucial for determining when unimodal information suffices and when the integration of cross-modal information becomes necessary.
The modality imbalance is caused by the dominant modality suppressing the optimization of another modality. Through experimental analysis, we find that there is a modality imbalance phenomenon between the unimodal features in hateful memes. As shown in Figure 2, the performance of text is closer to multimodal representation compared to image, but the performance of text and image modality in multimodal models is clearly worse than that of the image-only and text-only models, respectively. The above phenomenon indicates that the dominant text modality leads to the suppression of image modality optimization, further resulting in the inability of multimodal models to fully unleash the corresponding discriminative capabilities, thereby affecting the judgment of modality contributions.
To address the above issues, this paper proposes an Uncertainty-guided Modal Rebalance (UMR) framework for hateful memes detection. Specifically, we incorporate a probability distribution to produce a stochastic representation for individual
samples, diverging from the deterministic point embeddings employed in current approaches. To simplify modeling, we associate each meme with a Gaussian distribution in a latent space defined by mean and variance parameters. The mean represents the feature, whereas the variance measures the uncertainty. By modeling uncertainty, we flexibly combine discriminative cross-modal and unimodal features. Furthermore, we introduce a crossmodal feature fusion module based exclusively on MLP to capture semantic between images and texts, providing complementary features for hateful memes. Finally, we improve the cosine loss to alleviate the modality imbalance by considering both weight norm and inter-modal constraints. Releasing the unimodal representation ability in multimodal models through modal rebalancing further promotes the learning of modality contributions. The main contributions are summarized as follows:
• We formulate the modality uncertainty and imbalance problems, two critical challenges to hateful memes detection, and present an uncertainty-guided modal rebalance framework to quantify the uncertainty through Gaussian distribution modeling.
• To alleviate the adverse effects of modality imbalance, we improve cosine loss by conducting modality-specific L2 normalization on both features and weights, fully releasing the representation ability of unimodal in multimodal models to achieve modal rebalancing.
• Extensive experimental results demonstrate that 1) UMR produces the state-of-the-art performance on four widely-used datasets; 2) UMR provides consistent improvement on four vision-language backbones.

heading: 2 Related Work

heading: 2.1 Hateful Memes Detection
text: The hateful memes detection task aims to identify detrimental content, including hate, harm, and offense speech. Facebook first proposes the Hateful Memes Challenge (Kiela et al., 2020) to prompt researchers to pinpoint specific categories of hateful content. Prior research has delved into classic dual-stream models that integrate visual and textual features derived from image and text encoders through attention-based mechanisms and various fusion techniques for hate speech classification (Kiela et al., 2020; Das et al., 2020; Lippe
et al., 2020; Yang et al., 2023). Recent research has also endeavored to employ data augmentation (Zhou et al., 2021; Zhu, 2020; Lee et al., 2021; Cao et al., 2022, 2023; Yang et al., 2022) and ensemble strategies (Velioglu and Rose, 2020; Sandulescu, 2020) to improve the performance of classifying hateful memes. With the development of hateful memes detection communities, Pramanick et al.(Pramanick et al., 2021a) have expanded the categories of hatefulness and introduced two benchmarks pertaining to COVID-19 and US politics. Subsequently, Zhang et al.(Zhang et al., 2023) propose TOT to uncover the underlying harm in memes scenario through topology-aware optimal transport. Suryawanshi et al.(Suryawanshi et al., 2020) also create a dataset of offensive memes containing abusive messages. Based on this dataset, Lee et al.(Lee et al., 2021) propose the DisMultiHate model to disentangle visual and textual representations of memes for understanding. However, the above works overly rely on cross-modal fusion features, where modality uncertainty and imbalance are ignored.

heading: 2.2 Uncertainty Learning
text: The present popular representation learning techniques involve the extraction of features as point representations and aim to position these points as close as possible to the ground truth within a high-level representation space. Nevertheless, there typically exist multiple appropriate point representations, indicating the uncertainty present in representation learning. To tackle this issue, researchers have proposed the use of probability distribution representations to infer diverse solutions and enhance robustness, thereby preventing model overfitting to a single answer. In the domain of natural language processing, Gaussian distribution has been employed to represent words because it effectively captures asymmetric relationships among words (Vilnis and McCallum, 2014). Since then, researchers have explored the use of various distribution families for word representations (Athiwaratkun and Wilson, 2017; Li et al., 2018). In the computer vision domain, to model visual uncertainty, some studies have introduced Gaussian representations into specific tasks, such as person re-identification (Yu et al., 2019), pose estimation (Sun et al., 2020), and face recognition (Chang et al., 2020). More recently, the construction of distributions has yielded progress in generating diverse predictions for cross-modal retrieval in the
multimodal field (Chun et al., 2021). However, the uncertainty modeling in the hateful memes detection community remains blank. We are the first to attempt to define the inherent uncertainty between modalities and model each meme as a Gaussian distribution. Furthermore, we consider the issue of modal imbalance and promote the uncertainty learning of memes through modal rebalancing, thereby enhancing the diversity and robustness of the hate detection process.

heading: 3 Methodology

heading: 3.1 Cross-Modal Feature Encoder
text: The proposed UMR is illustrated in Figure 3. Taking CLIP (Radford et al., 2021) as an example, for a given image-text pair, we use Vision Transformer and Text Transformer to encode them respectively, and then map them to the same dimension. The encoded image feature is represented as I ∈ Rl×d, and the text feature is represented as T ∈ Rk×d.

heading: 3.2 Modal Rebalance Module
text: As discussed in Figure 2, the inconsistency in performance between modalities demonstrates the imbalance, where the modality with worse performance is particularly suppressed. Recently, cosine loss (Ranjan et al., 2017) has been proven effective in reducing intra-class imbalance by L2 normalization or maximizing cosine similarity scores on features in multimodal fine-grained tasks (Liu et al., 2017; Deng et al., 2019; Xu et al., 2023). Inspired by this prior work, we extend it to the hateful community to alleviate the modality imbalance in hateful memes, focusing on weight norm and inter-modal constraints.
Specifically, we first concatenate the features from the image encoder and the text encoder and obtain the logit score of the intermediate process through a fully connected layer. The vanilla softmax loss can be represented as follows:
Lvani = − 1
N
N∑
i=1
log eW
⊤ yi [Ii;Ti]+byi
∑n j=1 e W⊤j [Ii;Ti]+bj , (1)
where N represents the batch size, W ∈ R2d×n and b ∈ Rn represent fully connected layer weight and bias, respectively. n represents the hateful class number. We further divide W into two modalitywise module weights W Iand W T . In this way, we can obtain the logit output f(xi)j as follows:
f (xi)j = W I⊤ j Ii +W T⊤ j Ti, (2)
where f(xi)j represents the j-th class of the i-th sample, and bias b is omitted for simplicity. Next, we transform W I
⊤ j Ii + W T⊤ j Ti in the logit out-
put to cosθIj + cosθ T j , where cos θ I j =
W I ⊤
j Ii
∥W Ij ∥·∥Ii∥ and similar for cos θTj . θj is the angle between the weight and the feature. Following previous cosine loss (Wang et al., 2018; Deng et al., 2019; Xu et al., 2023) in fine-grained learning, We fix the modalitywise weights to 1 through L2 normalization and re-scale the embedding features to s. Normalizing the features and weights ensures that the prediction only depends on the angle between the feature vector and weight vector. Finally, the loss of the modal rebalancing module is defined as:
Lmrm = − 1
N
N∑
i=1
log es·(cos θ I yi +cos θTyi)
∑n j=1 e s·(cos θIj+cos θTj ) .
(3) Through the naive trigonometric transformation,
equation 2 can be rewritten as follows:
f (xi)j = cos θ I j+cos θ T j = 2 cos(
θIj + θ T j
2 )·cos(θ I j − θTj 2 ).
(4) This formula suggests that the logit score will only reach a high value when both modalities exhibit high confidence, meaning θIj and θ T j are both small. This acts as a cooperative constraint. Additionally, θIj and θ T j must be similar, with their difference remaining small, adhering to the symmetric constraint. This necessitates a more balanced op-
timization of unimodal features. Therefore, the modal rebalancing module can fully unleash the unimodal representation ability in multimodal models, providing support for subsequent learning of cross-modal features and modality contributions.

heading: 3.3 Cross-Modal Fusion Module
text: Cross-modal fusion has the capability to capture semantic interactions between different modalities, providing complementary features for hate memes detection. This is particularly valuable when image and text feature representations within memes convey conflicting sentiments. Recently, Multilayer Perceptron (MLP)-based models are proposed for vision tasks. By substituting MLP (e.g., MLPmixer (Tolstikhin et al., 2021) and ResMLP (Touvron et al., 2022)) for the self-attention mechanism, significantly reducing computational costs while maintaining high performance. However, the above models only contain two independent MLPs, one processing the sequential length and another processing the dimension size. CubeMLP (Sun et al., 2022) is the first to transfer it to multimodal feature processing. It adds an additional MLP module to handle modality features.
Inspired by the above MLP-based models, we naturally extend it to feature fusion of hateful memes. Specifically, we first concatenate the unimodal features to comprise a multimodal tensor C ∈ RS×M×D, where S represents the sequential length, M denotes the number of modalities, and
D indicates the size of the feature dimension. Then, the multimodal features are passed to the stacked three MLP units for mixing. Each MLP unit comprises two fully-connected layers followed by a GELU (Hendrycks and Gimpel, 2016) nonlinear activation to mix the multimodal features along the respective axes. A residual connection is employed in the unit according to (Touvron et al., 2022). Taking the first sequential-mixing MLP as an example, tensor C ∈ RS×M×D can be viewed as a set of vectors of C∗,m,d ∈ RS×1×1, where (m, d) ∈ {(1, 1), (1, 2), ..., (2, 1), (2, 2), ..., (M,D)}. Here, C∗,m,d represents the vector of m-th modality and d-th dimension. Each fully-connected layer in the sequential-mixing MLP unit can be represented as:
FCS(C∗,m,d) = WSC∗,m,d + bS , (5)
where S′ is the reduced dimension along the S-axis, which is set as a hyperparameter. All C∗,m,d share the parameters WS and bS . Therefore, the complete sequential-mixing MLP can be represented as:
U∗,m,d = LayerNorm(FCS(GELU(FCS(C∗,m,d)))
+ C∗,m,d), (6)
where the output tensor U ∈ RS′×M×D can be considered as a set of vectors of U∗,m,d ∈ RS
′×1×1. Similar to the first MLP unit along the S-axis, the output V ∈ RS′×M ′×D from the second MLP unit along the M -axis can be regarded as a set of vectors Vs,∗,d ∈ R1×M
′×1. The output G ∈ RS′×M ′×D′ from the third MLP unit on the D-axis can be regarded as a set of vectors Gs,m,∗ ∈ R1×1×D′ . Here, M ′ and D′ are the reduced dimensions along the M -axis and D-axis, respectively. The indices (s, d) range over {(1, 1), (1, 2), ..., (2, 1), (2, 2), ..., (S′, D)} and (s,m) range over {(1, 1), (1, 2), ..., (2, 1), (2, 2), ..., (S′,M ′)}. Finally, the modality-mixing MLP and the dimension-mixing MLP can be represented as:
Vs,∗,d = LayerNorm(FCM (GELU(FCM (U∗,m,d)))
+ U∗,m,d), (7)
Gs,m,∗ = LayerNorm(FCD(GELU(FCD(Vs,∗,d)))
+ Vs,∗,d), (8)
where G ∈ RS′×M ′×D′ is the mixed cross-modal feature representation.

heading: 3.4 Uncertainty-Guided Module
text: As shown in Figure 1, hateful memes detection should be aware of the uncertainty between modalities. However, for each given input sample, the unimodal features is deterministic. Therefore, we use the probability distribution Pz|x to capture the uncertainty of input embeddings, using the embeddings z (representing image I or text T ) as estimates for the mean of the desired distribution Pz|x. The distribution Pz|x can be represented as a parametric distribution Pz|x(z|ẑ, θ̂) where the parameters can be estimated (Lakshminarayanan et al., 2017; Upadhyay et al., 2023). Therefore, we introduce two modality-specific components to estimate parameters ẑ, θ̂. To ensure that the mean of the distribution estimated by modality-specific components aligns the point estimates generated by the frozen encoders, we establish a probabilistic reconstruction task for the embeddings within each modality. Specifically, for a given sample x, we extract the embedding z using the frozen encoder. Subsequently, the modality-specific component learns to reconstruct the z, producing a reconstruction denoted as ẑ). The modality-specific component is trained by maximizing the likelihood.
ζ∗ = argmax ζ
N∏
i=1
β̂ie −(|ẑi−zi|/α̂i)β̂i
2α̂iΓ ( 1/β̂i ) , (9)
where ζ represents the parameters of the component. Γ represents the Gamma function. β̂ie
−(|ẑi−zi|/α̂i)β̂i 2α̂iΓ(1/β̂i) represents the Generalized Gaussian Distribution (GCD), denoted as G, which is capable of modeling heavy-tailed distributions. It’s worth noting that the Gaussian and Laplace distributions are special cases of G with parameters α = 1, β = 2 and α = 1, β = 1, respectively. The variables ẑ, α̂, β̂ denote the predicted mean, scale, and shape parameters of G obtained from modality-specific components for the given input zi. We determine modality-specific optimal parameters by minimizing negative log-likelihood, equivalent to Equation 9. Given z and the predicted values ẑ, α̂, β̂, the loss can be expressed as:
Lugm(ζ) = ( |ẑ− z|
α̂
)β̂ − log β̂
α̂ + log Γ
( 1
β̂
) , (10)
where image-specific component loss is represented as LIugm(ζI), and text-specific component loss is represented as LTugm(ζT ).
Moreover, there is a phenomenon in hateful memes where the same image corresponds to different texts and the same text corresponds to different images. To this end, we ensure that the output distributions of image and text embeddings related to similar concepts remain close to each other.
LI→Tugm (ζI , ζT ) = ( |ẑI − zT |
α̂I
)β̂I − log β̂I
α̂I + log Γ
( 1
β̂I
) ,
(11)
LT→Iugm (ζI , ζT ) = ( |ẑT − zI |
α̂T
)β̂T − log β̂T
α̂T + log Γ
( 1
β̂T
) .
(12)
The overall objective of the uncertainty-guided module is designed as:
Lugm (ζI , ζT ) = LIugm(ζI) + LTugm(ζT ) + LI→Tugm + LT→Iugm . (13)
Finally, the uncertainty of different modalities in the sample xi can be quantified by estimating the unimodal distribution as follows:
σ̂2I = α̂2IΓ(3/β̂I)
Γ(1/β̂I) , (14)
σ̂2T = α̂2TΓ(3/β̂T )
Γ(1/β̂T ) , (15)
λi = Sigmoid
( σ̂2I + σ̂ 2 T
2
) , (16)
where λi represents the uncertainty score. The sigmoid function is employed to map these scores to the range [0, 1]. The uncertainty score λi serves as a weight that governs the fusion of unimodal and cross-modal features. Specifically, the uncertaintyguided module adaptively emphasizes cross-modal features and reduces the influence of unimodal features when uncertainty is high, and does the opposite when uncertainty is low.

heading: 3.5 Hateful Memes Detector
text: We flatten the mixed multimodal features and utilize the uncertainty score λi to guide the feature fusion process. Specifically, the cross-modal feature is multiplied by λi and each unimodal feature is multiplied by 1 − λi. The resulting fused feature Fi is then fed into the hateful memes classifier. Cross-entropy loss Ltask is employed for hateful memes detection.
Fi = λiGi ⊕ (1− λi)Ii ⊕ (1− λi)Ti, (17) ŷi = Softmax(FC(Fi)), (18)
Ltask = − 1 N
N∑
i=1
yi log(ŷi). (19)
Table 1: Statistics of Hateful, Harmful-C, Harmful-P and Offensive memes datasets.
Datasets #Training #Validation #Test
Hateful Hateful(3,050) Non-Hateful(5,450) Hateful(250) Non-Hateful(250) Hateful(500) Non-Hateful(500)
Harmful-C Harmful(1064) Non-Harmful(1949) Harmful(61) Non-Harmful(116) Harmful(124) Non-Harmful(230)
Harmful-P Harmful(1486) Non-Harmful(1534) Harmful(86) Non-Harmful(91) Harmful(173) Non-Harmful(182)
Offensive Offensive(187) Non-Offensive(258) Offensive(58) Non-Offensive(91) Offensive(58) Non-Offensive(91)
Finally, we combine the aforementioned modules to optimize the overall objective function of UMR framework:
LLoss = Ltask + Lmrm + Lugm. (20)

heading: 4 Experiments

heading: 4.1 Datasets
text: The experiment is conducted on four publicly available datasets as follow: Hateful memes which was created as part of the Hateful Memes Challenge 2020 for multimodal hateful detection and published in (Kiela et al., 2020), containing 10K memes with binary labels. Harmful-C memes and Harmful-P memes are respectively related to COVID-19 and United States politics, and published in (Pramanick et al., 2021a) and (Pramanick et al., 2021b) for multimodal harmful detection, each containing nearly 3.5K memes with binary labels. Offensive memes is related to the 2016 United States presidential election and published in (Suryawanshi et al., 2020) for multimodal offensive detection, containing nearly 1K memes with binary labels. The statistics are shown in Table 1.

heading: 4.2 Implementation Details
text: In the cross-modal feature encoder, we use four multimodal backbone models to initialize the image and text encoders, including CLIP ViT-L/14 (Radford et al., 2021), ALBEF ViT-B/16 (Li et al., 2021), BLIP ViT-B/16 (Li et al., 2022) and BLIP-2 ViT-L FlanT5XL (Li et al., 2023). In the crossmodal fusion module, we set S to 100, which involves zero padding shorter sequences and truncating longer sequences for sequence size matching. The value of M is fixed at 2, as we only have two involved modalities. The dimension D is set to 256, consistent with the output dimension of the projection layer. Additionally, The S′, M ′ and D′ are set
to 10, 2 and 32, respectively. In the uncertaintyguided module, both the image and text components consist of Multi-Layer Perceptrons. Each MLP comprises an input layer, transforming the embedding dimension to 256, a hidden layer of size 256, and an output layer, converting from 256 back to the embedding dimensions. For the hateful memes detector, the intermediate feature dimension of the detector is 64 and the dropout rate is set to 0.4. For the above backbone models, the initial learning rate is set to 1e-5, 3e-5, 2e-5 and 1e-4, respectively. The size of the minibatch is set to 16. Each dataset is trained for 10 epochs. The UMR framework is trained on a single A800 GPU.

heading: 4.3 Evaluation Metrics
text: For the evaluation of Hateful memes, we adopt the methodology outlined in (Kiela et al., 2020), employing the Area Under the Receiver Operating Characteristic curve (AUROC) and accuracy (Acc.) as evaluation metrics. The AUROC serves as the primary metric. For Harmful-C and Harmful-P memes, we adhere to the evaluation protocol established by (Pramanick et al., 2021b). Here, we utilize (Acc.), Macro-F1 (F1), and Macro-Averaged Mean Absolute Error (MMAE) as evaluation metrics. In the case of Offensive memes, we follow the evaluation procedure described in (Suryawanshi et al., 2020), employing F1 score, precision (Pre.), and recall (Rec.) as evaluation metrics.

heading: 4.4 Experimental Results
text: Comparison with the baselines. To assess the efficacy of the proposed UMR framework, we uti-
ison methods used for each dataset vary. These methods are outlined below: For hateful memes, CDKT (Yang et al., 2022) is employed, which is a cross-domain knowledge transfer model. It leverages sarcasm domain knowledge to provide additional discriminative information for the relatively small attack samples. For harmful memes, TOT (Zhang et al., 2023) is utilized. TOT deciphers implicit harm in memes scenarios using topologyaware optimal transport. For offensive memes, DisMultiHate (Lee et al., 2021) is employed. DisMultiHate disentangles target information from memes to improve offense content classification. Compared to CDKT, the proposed UMR demonstrates higher performance without requiring additional domain data. UMR achieves this by adaptively aggregating unimodal and cross-modal features through estimating uncertainty between modalities, without the need for complex feature representation (such as entities and demographic information) as required by TOT and DisMultiHate.
Furthermore, we observe that BLIP-2 could provide optimal performance on offensive memes, while showing disappointing results on hateful memes. The primary reason for this discrepancy lies in the construction of the datasets. Compared to the offensive dataset, the hateful dataset introduces benign confounding factors (Kiela et al., 2020) to confuse hate memes, which is particularly rare.

heading: 4.5 Quantitative Analysis
text: Effectiveness of each component. To assess the influence of each component in UMR, we conduct a series of ablation studies, as depicted in Table 5. It can be observed: 1) Removing the modal
rebalance module (w/o mrm), the performance decreases greatly, indicating that modality imbalance will weaken the effectiveness of subsequent feature fusion and uncertainty learning; 2) Removing the uncertainty-guided module (w/o ugm), performance decreases the most, demonstrating that considering the contribution of each modality to hate sentiment is the most critical factor for hateful memes detection task; 3) Removing the crossmodal fusion module (w/o cfm) and using the attention mechanism to capture dependencies between modalities, performance has no significant changes, verifying that MLP-based cross-modal fusion could maintain higher performance while reducing computational costs. Uncertainty-guided analysis. Table 6 shows the performance of various uncertainty measurement methods. It is evident that all UMR variants exhibit superior performance, underscoring the importance of uncertainty learning in hateful memes detection. Notably, UMR outperforms UMR-COS and UMR-DIS. This is primarily because UMR generates a stochastic representation for each sample using a Gaussian distribution, whereas UMR-COS and UMR-DIS rely on fixed unimodal representations to calculate distance, failing to capture the uncertainty of distributions. Cross-modal fusion analysis. As illustrated in Table 7, the performance degradation of UMR-CAT is evident, suggesting that merely concatenating unimodal features without modeling cross-modal interactions is inadequate for effective multimodal representation. UMR-CNN, on the other hand, tends to capture locally confined semantic interactions due to the limited size of the convolution kernel. In contrast, UMR can explore these interactions more globally, leading to improved performance.

heading: 4.6 Qualitative Analysis
text: The purpose of UMR is to model modality uncertainty and alleviate modality imbalance for hateful memes detection. To further understand UMR in-
tuitively, we show some cases in Figure 4-5. Modality uncertainty analysis. We refer to the visualization method in ALBEF and adopt GradCAM (Selvaraju et al., 2017) to visualize the third layer cross-attention map of the multimodal encoder in ALBEF. As shown in Figure 4, the left image is ALBEF and the right image is UMR. The keyword is black, and the darker the color, the higher the attention of black to the image area. The visualization experiment results show that in ALBEF, black and the black women in the image, including the specular reflection, are highly focused, leading to misclassification. In UMR, black focuses more on the black parts in the image, making the model distinguish it as non-hateful sample. Modality rebalance analysis. As shown in Figure 5, the performance of text and image in UMR is relatively stable and comparable. This indicates that UMR greatly improves modality imbalance in hateful memes. In addition, the performance of text and image modality in multimodal models has significantly improved compared to image-only and text-only models, respectively.

heading: 5 Conclusion
text: In this paper, a hateful memes detection framework (UMR) is proposed to address the challenges of modality uncertainty and modality imbalance. By extracting stochastic embeddings from a Gaussian distribution to quantify uncertainty, the framework adaptively aggregates cross-modal and unimodal features. By improving cosine loss from weight norm and inter-modal constraints, modality imbal-
ance can be alleviated. Moreover, UMR demonstrates scalability by accommodating various multimodal models as backbones. Experimental results on four widely-used datasets reveal that UMR consistently outperforms baselines and achieves competitive performance compared to existing methods for hateful memes detection. Quantitative analysis further validates the rationality of each component.
Limitations
We would like to highlight some limitations of the proposed method and suggest potential future directions. Firstly, as reported in the experimental results, our method decreases the robustness of the model due to inconsistent backbone performance across different datasets. Secondly, although we have shown the effectiveness of UMR through case studies in this paper, a more comprehensive analysis is required. For example, future work can explore more advanced and interpretable backbone models to enhance the interpretability of hate speech.

heading: Future Work
text: Limitations
We would like to highlight some limitations of the proposed method and suggest potential future directions. Firstly, as reported in the experimental results, our method decreases the robustness of the model due to inconsistent backbone performance across different datasets. Secondly, although we have shown the effectiveness of UMR through case studies in this paper, a more comprehensive analysis is required. For example, future work can explore more advanced and interpretable backbone models to enhance the interpretability of hate speech