Venue: NAACL 2025

# See-Saw Modality Balance: See Gradient, and Sew Impaired

Vision-Language Balance to Mitigate Dominant Modality Bias

JuneHyoung Kwon1 , MiHyeon Kim1*2 , Eunju Lee2 , Juhwan Choi1 , YoungBin Kim1,2

Equal contribution.Currently at: KT CORPORATION, minhyeon.gim@kt.com.

Footnote 1: footnotemark:

###### Abstract

Vision-language (VL) models have demonstrated strong performance across various tasks. However, these models often rely on a specific modality for predictions, leading to "dominant modality bias." This bias significantly hurts performance, especially when one modality is impaired. In this study, we analyze model behavior under dominant modality bias and theoretically show that unaligned gradients or differences in gradient _magnitudes_ prevent balanced convergence of the loss. Based on these findings, we propose a novel framework, **BalGrad** to mitigate dominant modality bias. Our approach includes inter-modality gradient reweighting, adjusting the gradient of KL divergence based on each modality's contribution, and inter-task gradient projection to align task _directions_ in a non-conflicting manner. Experiments on UPMC Food-101, Hateful Memes, and MM-IMDb datasets confirm that BalGrad effectively alleviates over-reliance on specific modalities when making predictions.

## 1 Introduction

Vision-language (VL) models combine image and text modalities, resulting in powerful multi-modal representations. Owing to this integration of two modalities, these models can achieve higher performance in vision-language tasks. Recently, leveraging extensive datasets, VL models have demonstrated remarkable performance across various tasks such as image captioning Hu et al. (2022), visual question answering Khademi et al. (2023), and cross-modal retrieval Liu et al. (2023), showing their capability to harness the complementary strengths of visual and textual data.

However, these models often rely on a single modality rather than treating and utilizing them equally, leading to the dominance of a certain modality on the overall performance. A conceptual overview of this effect can be seen in Figure 1. This phenomenon, where a specific modality disproportionately influences the model's outcomes, is referred to as "dominant modality bias" Woo et al. (2023). For instance, VL models tend to be biased towards the text modality when recognizing hate expressions Kiela et al. (2020); Aggarwal et al. (2024), thereby limiting the VL model's ability to effectively integrate and interpret images.

This bias behaves particularly detrimentally when one modality is impaired, such as when data is noisy and it is difficult to gather paired data Garg et al. (2022); Woo et al. (2023); Yang et al. (2024). This issue is common in real-world scenarios due to privacy-related data sharing restrictions or stringent data storage policies Voigt and Von dem Bussche (2017) and can severely degrade the model's performance. Additionally, the failure to sufficiently explore the weak modality limits the overall performance of the VL model Wang et al. (2020); Huang et al. (2022); Peng et al. (2022), highlighting the need for robust solutions to mitigate dominant modality bias.

To address this issue and balance the information between modalities, numerous studies have

Figure 1: Conceptual visualization of dominant modality bias. The key modality differs by task: **(a)** For the hate recognition task, text descriptions of memes lead, while **(b)** for the food classification task, food images play a crucial role in prediction.



been conducted. Several studies have focused on modulating the gradients of each encoder based on the confidence of individual modalities (Peng et al., 2022; Li et al., 2023). Other approaches have involved training multimodal models using the best-performing learning rates from unimodal models Yao and Mihalcea (2022). However, these methods often induce negative transfer Wang et al. (2019); Yu et al. (2020), which occurs when the model's performance decreases with the addition of modality data compared to solely using unimodal data.

We first analyze the behavior of models after the dominant modality bias has taken root. Our analysis reveals that certain modalities are more crucial to target performance and observes that the dominant and weak modalities converge at different rates during training. Additionally, we theoretically demonstrate that the balanced convergence of the loss is influenced by both the _magnitude_ and _direction_ of the gradient. Based on these findings, we propose **BalGrad****(Balancing Gradients)** to mitigate dominant modality bias. Firstly, we adopt a mutual KL divergence between the predictions of each modality to ensure balanced updates. However, a naive approach that equally aligns the distributions of two modalities can hinder the representation learning of each modality. To address this, we introduce **inter-modality gradient reweighting**, which adjusts the _magnitude_ of the gradient of the KL divergence term based on the learning status of each modality. Additionally, we propose **inter-task gradient projection**, which updates the gradient of the target task to establish a balance between both modalities. We project the target task's gradient in a _direction_ orthogonal to the KL divergence gradient if a conflict between the gradients occurs, encouraging stabilized training between the two modalities.

We evaluate the effectiveness of BalGrad on models using three vision-language datasets: UPMC Food-101 Wang et al. (2015), Hateful Memes Kiela et al. (2020), and MMIMDb Arevalo et al. (2017). To simulate the influence of individual modalities, we conduct experiments under conditions where specific modalities are missing or impaired by noise. The experimental results demonstrate that the proposed method reduces the gap between the modalities while avoiding negative transfer. The contributions of our proposed method are as follows:

* We analyze the dominant modality bias and theoretically demonstrate that the balanced convergence of loss is influenced by both the _magnitude_ and _direction_ of the gradient.
* We propose BalGrad, which reweights the gradients between modalities to ensure stable convergence and projects the target task's gradient to avoid conflicts that hinder balanced learning.
* Experimental results across UPMC Food-101, Hateful Memes, and MM-IMDb under different impaired conditions confirm the effectiveness of our proposed method in mitigating dominant modality bias.

## 2 Related Work

In multimodal models, such as VL models, a bias towards a preferred or easier-to-learn modality often leads to the under-exploration of others Wang et al. (2020); Huang et al. (2022); Peng et al. (2022). Studies have analyzed this, noting that multimodal models are prone to overfitting and show discrepancies in generalization across modalities Wang et al. (2020). Differences in convergence speeds also contribute to this bias Yao and Mihalcea (2022); Wu et al. (2022). An early study in this field finds that certain modalities, correlating with their network's random initialization, dominate the learning process Huang et al. (2022), while other researchers attribute the preference to unimodal representation margins and insufficient integration of modalities Yang et al. (2024). Another line of study highlights that spurious correlations with instance labels cause imbalances in modality utilization Guo et al. (2023). In this paper, we identify that the dominant modality bias in VL models arises from the influence of gradient magnitude and direction on the model's loss function, hindering balanced learning across modalities.

In response to the challenge of balancing modalities in multimodal learning, various strategies have been proposed. MSLR suggests using different optimal learning rates for each modality during multimodal learning to enhance performance Yao and Mihalcea (2022). Another approach involves using a conditional utilization rate to re-scale modality features, ensuring balanced contributions from each modality Wu et al. (2022). Gradient blending optimizes the mixing of modalities based on the model's overfitting behavior Wang et al. (2020) 

2020). OGM-GE adaptively controls the optimization process using modality-specific confidence scores (Peng et al., 2022). AGM employs Shapley values to modulate gradients through mono-modal responses, aiming to balance the learning process across modalities (Li et al., 2023). However, these methods often lack consideration of negative transfer and may introduce adverse effects. In this paper, we propose BalGrad, which reweights gradients considering the learning status of each modality and projects the gradients to mitigate dominant modality bias without disrupting the balance between modalities.

## 3 Method

In this section, we analyze the dominant modality bias and propose BalGrad to mitigate such bias. In Section 3.1, we observe the behavior of VL models and theoretically demonstrate the factors influencing balanced loss convergence. In Section 3.2, based on these findings, we introduce BalGrad, which reweights and projects gradients to ensure balanced learning across modalities.

### Analysis of Dominant Modality Bias

We introduce a controlled experiment to analyze the behavior of VL models biased by dominant modality. We denote the training dataset as \({\cal D}=\{(x_{i},y_{i})\}_{i=1}^{N}\), where \(x_{i}=(x_{i}^{v},x_{i}^{l})\) is a pair of data from the image and text modalities, respectively, and \(y_{i}\) represents the label. We extract features from the image and text encoders, passing them through their respective embedding layers, \(h_{v}(\cdot)\) and \(h_{l}(\cdot)\). These embeddings are then fused via concatenation and passed through a classifier, \(f_{\mathcal{T}}(\cdot)\), to yield the predicted probability \(p_{\mathcal{T}}\). Details on the architecture and training scheme are provided in the Appendix B.

**Analysis on Performance Gap.** To analyze the impact of individual modalities on the performance of VL models, we mute one modality by inputting empty values at the data level, rendering it noninformative. This method is applied while testing on the UPMC Food-101, Hateful Memes, and MM-IMDb datasets. The experimental results in Figure 2 (a) show a significant performance drop when a specific modality is missing. In the case of UPMC Food-101, the image modality significantly influences the overall performance, while in Hateful Memes, the text modality plays a more crucial role. Conversely, the performance drop is relatively minor when the weak modality (_text_ for _UPMC Food-101_ and _image_ for _Hateful Memes_) is missing. In contrast, for MM-IMDb, the performance drop is similar when either modality is missing, indicating that the model is not biased towards a specific modality.

**Analysis on Training Dynamics.** To observe the loss dynamics of each modality during the training phase, we add linear classifiers \(f_{v}(\cdot)\) and \(f_{l}(\cdot)\) on top of the image and text embedding layers, respectively. These classifiers output probabilities \(p_{i}^{v}\) and \(p_{i}^{l}\), which are then used to predict the label \(y_{i}\), and each target objective is represented as \(\mathcal{L}_{\mathcal{T}}^{w}\) and \(\mathcal{L}_{\mathcal{T}}^{lr}\), respectively. We find that the loss of the dominant modality decreases rapidly, while the loss of the weak modality decreases relatively slowly, as shown in Figure 2 (b). For MM-IMDb specifically, the loss gap decreases as training iterations increase, demonstrating that the model is not biased toward any single modality. This indicates that, during training, one modality is overly exploited while the other modality is relatively underexplored, which is consistent with previous re

Figure 2: Experimental results on the UPMC Food-101, Hateful Memes, and MM-IMDb datasets in the presence of dominant modality bias. **(a)** Performance visualization under different missing conditions (full, image only (missing text), text only (missing image)) for each dataset. **(b)** Illustration of learning curves for each modality across datasets.

 

search (Wang et al., 2020; Huang et al., 2022; Peng et al., 2022). We conjecture that this phenomenon appears inherently task-dependent, with the VL model inclined to update based on the easy-to-learn modality that can quickly reduce the loss (Arpit et al., 2017; Nam et al., 2020).

**Theoretical Analysis of Gradient Influence.** To theoretically analyze why VL models struggle to balance the utilization of both modalities, we examine the loss reduction in terms of gradient updates. The loss function for a target is defined as \({\cal L}(\theta_{v},\theta_{l},\theta_{\mathcal{T}})\), where \(\theta_{v}\) and \(\theta_{l}\) are the parameters of image and text embedding layers, respectively, and the \(\theta_{\mathcal{T}}\) represents the parameters of the classifier \(f_{\mathcal{T}}(\cdot)\). The objective is to find the optimal parameters \(\Theta=\left\{\theta_{v},\theta_{l},\theta_{\mathcal{T}}\right\}\) that minimize \({\cal L}(\theta_{v},\theta_{l},\theta_{\mathcal{T}})\). To analyze how each modality contributes to the overall loss reduction, we decompose the target task loss gradient with respect to the model parameters \(\Theta\) into modality-specific components, denoted by \(\mathcal{G^{\tau}}=\left\{g_{l},g_{v},g_{\mathcal{T}}\right\}\). These partial gradients capture the influence of linguistic, visual, and task-related parameters, respectively, under standard gradient-descent updates. Additionally, \(g_{\mathcal{T}}=\sum_{i\in\left\{v,l\right\}}\triangledown_{\theta_{i}} \hat{y}\triangledown_{g\mathcal{T}}\nabla_{p_{\mathcal{T}}}{\cal L}=\sum_{ i\in\left\{v,l\right\}}g_{\mathcal{T}}^{i}\) denotes the gradient for parameters \(\theta_{\mathcal{T}}\) of the linear classifier \(f_{\mathcal{T}}(\cdot)\), where \(g_{\mathcal{T}}^{i}\) denotes the gradient of each modality in \(f_{\mathcal{T}}(\cdot)\). We theoretically analyze how the target objective is influenced by the varying magnitudes and directions of gradients for each modality.

**Proposition 1**.: **(Gradient Effect on Change of Loss)** Let the parameters \(\theta_{v},\theta_{l}\), and \(\theta_{\mathcal{T}}\) of a multimodal model be updated with gradients \(g_{v},g_{l},\) and \(g_{\mathcal{T}}\) using a sufficiently small step size \(\lambda>0\), resulting in updated parameters \(\hat{\theta}_{v},\hat{\theta}_{l},\) and \(\hat{\theta}_{\mathcal{T}}\). Then the change in the loss function satisfies

\[\begin{split}\triangle{\cal L}&~{}=~{}-2\,\lambda \,\big{(}g_{\mathcal{T}}^{v}\cdot g_{\mathcal{T}}^{l}\big{)}\\ &~{}-\lambda\,\sum_{i\in\left\{v,l,\mathcal{T}\right\}}\!\Big{(} g_{i}\cdot g_{i}~{}+~{}g_{\mathcal{T}}^{i}\cdot g_{\mathcal{T}}^{i}\Big{)}~{}+~{}O( \lambda^{2}),\end{split} \tag{1}\]

where the cross term \(-2\,\lambda\,(g_{\mathcal{T}}^{v})\cdot(g_{\mathcal{T}}^{l})\) captures the interaction between the visual and language gradients and the magnitudes and directions of each gradient \(g_{\mathcal{T}}^{v}\) and \(g_{\mathcal{T}}^{l}\) governs how much the overall loss is reduced.

Proof.: See A.1 

If the gradients for the two modalities \(g_{\mathcal{T}}^{v}\) and \(g_{\mathcal{T}}^{l}\) do not align well, meaning they have conflicting directions or have significantly different magnitudes, the loss reduction will not be balanced. Gradients with larger magnitudes substantially impact loss reduction, while gradients with directions that align more closely between modalities facilitate more effective joint learning. Consequently, the loss is likely to decrease more under the influence of the dominant modality, leading to an uneven contribution from each modality.

### BalGrad

Based on the findings above, we propose BalGrad to mitigate the dominant modality bias, which consists of two components: inter-modality gradient reweighting and inter-task gradient projection. Inter-modality gradient reweighting addresses the imbalance caused by different gradient _magnitudes_, ensuring more equal contributions from each modality. Inter-task gradient projection aligns the gradient _directions_ of the modalities, facilitating more effective joint learning and preventing the dominant modality from disproportionately influencing loss reduction. The overall process of BalGrad can be seen in Figure 3.

#### 3.2.1 Inter-modality Gradient Reweighting

Standard VL models lack the consideration to ensure that both modalities are updated equally, leading to the stronger modality dominating the training phase, as we observed in the previous section. Therefore, inspired by knowledge distillation Hinton et al. (2014); Zhang et al. (2018); Phuong and Lampert (2019), we aim to balance the gradients received from each modality by aligning the distributions of their predictions. To achieve this, we compute the mutual Kullback-Leibler (KL) divergence between the predictions \(p_{i}^{v}\) and \(p_{i}^{l}\) of the two modalities. This involves aligning the predictions of the image modality with those of the text modality and vice versa. The KL divergence from \(p_{i}^{l}\) to \(p_{i}^{v}\) is as follows:

\[{\cal L}_{kl}^{l}=-\sum_{i}p_{i}^{l}log\frac{p_{i}^{v}}{p_{i}^{l}} \tag{2}\]

We also compute \({\cal L}_{kl}^{v}\) in the same manner. We represent the gradients of \({\cal L}_{kl}^{l}\) and \({\cal L}_{kl}^{v}\) as \(g_{kl}^{l}=\nabla{\cal L}_{kl}^{l}\) and \(g_{kl}^{v}=\nabla{\cal L}_{kl}^{v}\), respectively. In this way, each modality's embedding layer learns to correctly predict the label and match the probability estimate of other modalities, thereby alleviating the severe imbalance. However, symmetrically aligning the distributions between the two modalities 

overlooks the differences in their convergence status, as observed in Section 3.1. This can cause the layers of the faster-converging modality to be hindered in their representation learning, leading to performance degradation. Therefore, we propose an **inter-modality gradient reweighting** method that adjusts the magnitude to which each modality receives the KL divergence gradient based on its contribution to the learning objective. We reweight the gradient of the KL divergence term for \(p_{i}^{l}\) to \(p_{i}^{v}\) and \(p_{i}^{v}\) to \(p_{i}^{l}\) using the following terms, respectively:

\[\mathcal{W}^{l}=\frac{\mathcal{L}_{\mathcal{T}}^{l}}{\mathcal{ L}_{\mathcal{T}}^{v}+\mathcal{L}_{\mathcal{T}}^{l}},~{}\mathcal{W}^{v}=\frac{ \mathcal{L}_{\mathcal{T}}^{v}}{\mathcal{L}_{\mathcal{T}}^{v}+\mathcal{L}_{ \mathcal{T}}^{l}} \tag{3}\]

In this configuration, if the target task loss for a modality is low (i.e., it has converged more), the gradient receives a lower weight. This ensures that the gradient of the weak modality is updated more toward matching the dominant modality's prediction, thereby reducing the training gap. In contrast, the dominant modality receives less influence from the underperforming predictions, allowing it to effectively learn its representation. Additionally, to ensure that each modality is trained for the target task independently, we introduce an additional term that increases the reweighting factor as iteration progresses. This ensures that the impact of mutual learning grows over time, allowing individual encoders to learn effectively in the initial stages and progressively encouraging balanced learning between modalities. The final reweighted gradient for the KL divergence is as follows:

\[g_{kl}=(\gamma+\frac{\gamma}{1+e^{-t}})(\mathcal{W}^{l}g_{kl}^{ l}+\mathcal{W}^{v}g_{kl}^{v}) \tag{4}\]

\(\gamma\) is the initial weighting factor, and we set \(\gamma=1/2\).

#### 3.2.2 Inter-task Gradient Projection

Proposition 1 highlights that properly aligning gradients with different directions and magnitudes is crucial for effective joint learning. However, when gradients are not aligned and exhibit negative cosine similarity, known as conflicting gradients, the optimization process becomes suboptimal (Yu et al., 2020; Shi et al., 2023). Such conflicts can arise between the gradients of different tasks, potentially causing the dominant gradient to overwhelm the optimization process at the expense of the other task's performance.

For our case, as confirmed in Section 3.1, the target task, which is \(\mathcal{L}_{\mathcal{T}}\) alone, fails to balance the modalities and fully explore the weak modality. Therefore, we introduce the balance between the predictions of each modality as an additional task. However, as mentioned earlier, naive joint training can cause conflict between the gradient of the target task and KL divergence (i.e., \(g_{\mathcal{T}}\) and \(g_{kl}\)).

 



**Proposition 2.** (Gradient Conflicts on Loss Reduction with KL Loss) Let \({\cal G}^{\tau}=\left\{\,g_{v},\,g_{l},\,g_{\tau}\right\}\) and \({\cal G}^{kl}=\left\{\,g_{v}^{kl},\,g_{l}^{kl},\,0\right\}\) be the gradients from a target loss \({\cal L}_{\tau}\) and a KL loss \({\cal L}_{kl}\), respectively, with parameters \(\theta=[\theta_{v},\theta_{l},\theta_{\tau}]^{\top}\). Assume the parameters are updated by gradient descent with a small step size \(\lambda>0\): \(\theta_{y}^{\prime}=\theta_{v}-\lambda\left(g_{v}+g_{v}^{kl}\right),\theta_{ l}^{\prime}=\theta_{l}-\lambda\left(g_{l}+g_{l}^{kl}\right),\theta_{\tau}^{ \prime}=\theta_{\tau}-\lambda\,g_{\tau}\). Then, for the combined loss \({\cal L}={\cal L}_{\tau}+{\cal L}_{kl}\), the change in the loss is

\[\begin{split}\Delta{\cal L}&={\cal L}\big{(} \theta^{\prime}\big{)}-{\cal L}\big{(}\theta\big{)}\\ &=-\lambda\Big{(}\|{\cal G}^{\tau}\|^{2}+\,\|{\cal G}^{kl}\|^{2}+ \,2\,({\cal G}^{\tau})^{\top}{\cal G}^{kl}\Big{)}\\ &+O(\lambda^{2}).\end{split} \tag{5}\]

In particular, if \(({\cal G}^{\tau})^{\top}{\cal G}^{kl}<0\), the gradients from the target and KL losses _conflict_, reducing the effective loss reduction.

Proof.: See Appendix A.2 

Building upon Proposition 2, we aim to ensure that the gradient of the target task does not disrupt the balance between modalities. Specifically, we propose **inter-task gradient projection**, which projects \(g_{\mathcal{T}}\) onto \(g_{kl}\) in a non-conflicting manner. First, we consider the relationship between the two gradients to determine if they conflict and compute the cosine similarity between the two gradients. If \(g_{\mathcal{T}}\cdot g_{kl}\geq 0\), we assume that \(g_{\mathcal{T}}\) is being updated in a direction that aligns with modality balance, and we use the original \(g_{\mathcal{T}}\) for updating the model. Conversely, if \(g_{\mathcal{T}}\cdot g_{kl}<0\), indicating a potential disruption to the balance between modalities, we project \(g_{\mathcal{T}}\) in a direction orthogonal to \(g_{kl}\). This process can be represented as follows:

\[g_{\mathcal{T}}^{\perp}=\begin{cases}g_{\mathcal{T}}-\left(\frac{g_{\mathcal{ T}}\cdot g_{kl}}{||g_{kl}||^{2}}\right)g_{kl},&\text{if $g_{\mathcal{T}}\cdot g_{ kl}<0$}\\ g_{\mathcal{T}},&\text{otherwise}\end{cases} \tag{6}\]

This projection ensures that \(g_{\mathcal{T}}^{\perp}\) is adjusted to maintain the balance between the modalities while preventing conflicts with \(g_{kl}\). In a nutshell, the proposed BalGrad allows for extensively learning different modalities and tasks, effectively optimizing the target task while maintaining the balance between the modalities.

## 4 Experiments

### Experimental Setup

We conduct experiments on three vision-language datasets: UPMC Food-101 Wang et al. (2015), Hateful Memes Kiela et al. (2020), and MMDb Arevalo et al. (2017). For image and text encoding, we utilize ViT Dosovitskiy et al. (2021) and BERT Devlin et al. (2019), respectively, employing a late concatenation architecture for final predictions. To minimize extensive fine-tuning, we adopt linear probing, freezing all encoder parameters and training only the embedding and classifier layers. Further implementation details are provided in Appendix B.

To assess the robustness of the VL model against dominant modality bias, we introduced two impaired conditions: missing and noisy. For the missing modality, empty strings were used for text and zero pixels for images Lee et al. (2023). In the noisy condition, 30% salt and pepper noise is added to images Lim et al. (2023), and 15% of text tokens were randomly deleted Manolache et al. (2021); Yuan et al. (2023). All experiments were conducted with the model trained on unimpaired full modality data, with impairments applied to the entire data of a specific modality during testing. Further implementation details are provided in Appendix C.

### Experimental Results

We train with full modality data and evaluate the performance of the VL model under conditions where one modality is entirely impaired across three datasets, as shown in Table 1. "Full" refers to the scenario where no modalities are impaired during testing. For the impaired cases (missing and noisy), each modality is impaired according to the specified method. "_Avg._" denotes the average performance when each modality is impaired individually, while "\(\Delta_{Gap}\)" represents the performance difference between the image-impaired and text-impaired conditions. A smaller \(\Delta_{Gap}\) indicates a more balanced model that does not overly rely on a single modality, thereby exhibiting less dominant modality bias.

For the UPMC Food-101 dataset, BalGrad demonstrates the highest performance across all conditions--full, missing image, and missing text. Notably, it improves the performance on the weak modality, text, by 12.5%p compared to the baseline. Additionally, it achieves the highest average performance and exhibits the smallest gap, effectively mitigating bias despite the dominant influence of the image modality. In the noisy condition, our method shows robustness comparable to AGM Li et al. (2023) and achieves the highest _Avg._

BalGrad exhibits the highest performance

in conditions where the dominant text modality is missing, as well as in the full modality, _Avg._, and \(\Delta_{Gap}\) for the Hateful Memes dataset. OGMGE Peng et al. (2022) and AGM perform better in the image missing condition than in the full modality condition, indicating a heavy reliance on the text modality, with performance increases of 0.13%p and 1.56%p, respectively. In other words, adding the image modality results in a decrease in performance compared to using text alone, exhibiting negative transfer Wang et al. (2019). In the noisy condition, BalGrad demonstrates the highest _Avg._ performance and the smallest \(\Delta_{Gap}\), showcasing that BalGrad sufficiently explores the image modality.

Furthermore, BalGrad maintains the balance between the modalities even despite the absence of any dominant modality. For the MM-IMDb dataset, our proposed method shows slightly lower performance compared to the baseline but exhibits the second-smallest \(\Delta_{Gap}\), indicating balanced results without a dominant modality. Although OGM-GE demonstrates high performance, it exhibits a significant imbalance between modalities, as evidenced by the considerably higher gap, which is 10.83%p more than our method. BalGrad achieves the highest average performance and the lowest gap in the noisy condition, showcasing that our proposed method effectively explores both modalities without being biased towards one.

Additionally, to investigate the robustness under varying degrees of impairment, we mute a specific modality according to the missing ratio \(r\), and the results are shown in Figure 4. For each dataset, we randomly drop a certain percentage \(r\%\) of the data from each modality and measure the resulting performance \(\Delta_{Gap}\). We set missing ratios \(r\in\{0.2,0.4,0.6,0.8\}\). Experimental results indicate that BalGrad consistently exhibits a lower gap compared to existing methods across varying missing ratios, demonstrating robustness to impaired modalities. While BalGrad exhibits a slightly larger gap compared to the baseline, it is noteworthy that BalGrad significantly reduces the gap for datasets with dominant modality bias. Additionally, it introduces a small gap for datasets where dominant modality bias is not present.

Additional experimental results on various fusion mechanisms, backbone models, and datasets are provided in Appendix C. The results demonstrate that BalGrad consistently delivers robust performance across different biases, modality types, datasets, and perturbed conditions, underscoring its effectiveness in synergistically integrating modalities to prevent negative transfer and ensure reliable, real-world multimodal learning.

Figure 4: Evaluation on robustness to different missing ratio \(r\) of BalGrad and existing methods on UPMC Food-101, Hateful Memes, and MM-IMDb datasets.

 



### Ablation and Analysis

**Analysis of Each Component.** We conduct ablation experiments to assess the impact of gradient reweighting and projection as shown in Table 2. While gradient reweighting shares a common approach with existing methods Peng et al. (2022); Li et al. (2023), helps mitigate modality imbalance, it induces negative transfer in the Hateful Memes dataset and leaves the MM-IMDb dataset overly reliant on text. In contrast, incorporating gradient projection eliminates negative transfer and balances modality use. By aligning the gradient of the target loss with the KL loss term, we reduce reliance on any single modality, effectively preventing negative transfer. These points clarify how our approach differs from existing work and address the gaps in empirical validation and mitigation of negative effects.

**Evaluation on Text Decoder-based Vision-Language Model.** To examine BALGRAD's effectiveness in text decoder-based architectures, we conduct additional experiments using BLIP Li et al. (2022), which generates textual outputs from visual inputs via a text decoder. This setup differs from encoder-only VL models and aligns with autoregressive language modeling approaches. As shown in Figure 5, BALGRAD achieves the lowest \(\Delta_{Gap}\) across all datasets, indicating its ability to balance modality contributions in decoder-based VL models. These results highlight BALGRAD's potential for extension to decoder-only LLMs, as it effectively mitigates dominant modality bias across different VL architectures.

**Ablation on Inter-modality Gradient Reweighting.** To validate the efficacy of inter-modality gradient reweighting, we track the training loss dynamics for each modality on datasets with dominant modality bias (UPMC Food-101 and Hateful Memes), as shown in Figure 6. Without reweighting, weights are fixed at \(\mathcal{W}^{v}=1/2\) and \(\mathcal{W}^{l}=1/2\), equally distilling information between modalities. Experimental results show that reweighting leads to faster and more stable convergence of loss for each modality. This supports Proposition 1 in Section 3.1, indicating that gradient reweighting optimizes the exploration of individual modalities while maintaining balance in the VL model.

Figure 5: Bar plots comparing the performance of existing methods and BalGrad using BLIP. Each bar represents \(\Delta_{Gap}\)(%), defined as the performance difference between missing image and missing text conditions.

\begin{table}
\begin{tabular}{c|c c c|c c c|c c c|c c} \hline \multirow{4}{*}{Modality} & \multicolumn{4}{c|}{**UPMC Food-101**} & \multicolumn{4}{c|}{**Hateful Memes**} & \multicolumn{4}{c}{**MM-IMDb**} \\  & Baseline & w/Gradient w/ Gradient weighting projection & **BALGRAD** & Baseline & w/Gradient w/ Gradient weighting projection & **BalGRAD** & Baseline & w/gradient w/ Gradient weighting projection & **GALGRAD** \\ \hline Full & 76.01 & 78.17 & 76.20 & **80.32** & 65.10 & 65.80 & 66.30 & **67.35** & 44.09 & **44.30** & 42.30 & 43.19 \\ \hline \multirow{3}{*}{Missing} & Image 12.99 & 22.30 & 19.82 & **25.49** & 64.34 & 66.37\({}^{a}\) & 65.40 & 65.86 & 18.85 & **21.48** & 18.87 & 18.81 \\  & Test & 63.52 & 64.10 & 63.76 & **65.03** & 55.60 & 57.03 & 56.20 & **57.48** & 18.40 & 17.20 & **18.80** & 17.47 \\  & Avg.1 & 38.26 & 43.20 & 41.79 & **45.26** & 59.97 & **61.70** & 60.80 & 61.67 & 18.63 & **19.34** & 18.64 & 18.14 \\  & Gap & 50.53 & 41.80 & 43.94 & **39.54** & 8.74 & 9.34 & 9.20 & **8.38** & 0.45 & 4.28 & **0.33** & 1.34 \\ \hline \end{tabular}
\end{table}
Table 2: Ablation study results compares performance with and without inter-modality gradient reweighting and inter-task gradient projection to evaluate their impact on modality balance and transfer effects on UPMC Food-101, Hateful Memes, and MM-IMDB datasets. The best results are highlighted in bold, the second-best in italics, and values shown in gray® indicate negative transfer. “\(\Delta_{Gap}\)” is reported in %p, while all other values are in %.

Figure 6: Training iteration loss curves for image and text modalities on the UPMC Food-101 and Hateful Memes datasets, comparing the effects of the existence of inter-modality gradient reweighting.





**Analysis on Inter-task Gradient Projection.To assess the impact of inter-task gradient projection, we visualize the cosine similarity between the gradients of KL divergence (\(g_{kl}\)) and the target task (\(g_{\mathcal{T}}\)) throughout the entire training process using histograms, as shown in Figure 7. Without gradient projection, negative similarity between gradients is prevalent throughout training, resulting in imbalanced updates to the target task. Conversely, BalGrad, incorporating inter-task gradient projection, shows a positive mean cosine similarity between gradients, indicating fewer conflicts during training. This suggests that the gradients for the target task are more balanced between the two modalities, leading to more balanced convergence. This reduction in conflicts narrows the performance gap between image and text modalities, mitigating over-reliance on any specific modality, aligning with our analysis in Section 3.1.**

To further quantify this effect, we conduct an ablation study measuring the frequency of conflicting gradients with and without projection across three datasets, as shown in Table 3. The fraction indicates the percentage of gradient conflicts that occur between the gradients of KL divergence (\(g_{kl}\)) and the target task (\(g_{\mathcal{T}}\)) throughout the entire training process. The results demonstrate that there is a high incidence of conflicting gradients across all datasets without projection. In contrast, the use of projection significantly reduces gradient conflicts, especially in datasets with dominant modality bias, such as UPMC Food-101 and Hateful Memes.

## 5 Conclusion

In this paper, we addressed the challenge of dominant modality bias, where a VL model disproportionately relies on one modality, undermining the contributions of others. Our analysis shows that unaligned gradients and differences in gradient _magnitudes_ hinder balanced loss convergence. Based on these findings, BalGrad mitigates this bias by incorporating inter-modality gradient reweighting, which adjusts the KL divergence gradient based on each modality's contribution, and inter-task gradient projection to align task _directions_ nonconflictingly. Experiments on UPMC Food-101, Hateful Memes, and MM-IMDb datasets demonstrate that BalGrad effectively reduces dominant modality bias, enhances model robustness, and improves accuracy. These results highlight the potential for more stable and balanced training in VL models, paving the way for future advancements.

### Limitation

While BalGrad has shown efficacy in mitigating dominant modality bias in VL models, extending this approach to multimodal models with more than two modalities presents additional challenges. When dealing with three or more modalities, the training cost rapidly increases due to the need to consider the relationships between the gradients of each pair of modalities. This increased complexity in gradient management makes the balancing process more computationally intensive and difficult to maintain effectively. Thus, while BalGrad is effective in bi-modal settings, its application in multimodal scenarios requires further refinement to manage the higher computational demands and ensure balanced performance across all modalities.

Figure 7: Histogram visualization of the frequency of gradient conflicts between image and text gradients during training iterations on the UPMC Food-101 and Hateful Memes datasets. \(\mu_{w/o}\) and \(\mu_{w/}\) represent the average cosine similarity values w/o and w/ projection, respectively.

\begin{table}
\begin{tabular}{l|c c c c|c c} \hline  & \multicolumn{2}{c|}{**UPMC**} & \multicolumn{2}{c|}{**Hateful**} & \multicolumn{2}{c}{**MM-IMDb**} \\  & \multicolumn{2}{c|}{**Food-101**} & \multicolumn{2}{c|}{**Memes**} & \multicolumn{2}{c}{} \\ \cline{2-6}  & Fractioni & \(\Delta_{Gup}\) & Fractioni & \(\Delta_{Gup}\) & Fractioni & \(\Delta_{Gup}\) \\ \hline w/o Projection & 0.66 & 43.27 & 0.78 & 10.21 & 0.28 & 4.21 \\ w/ Projection & **0.36** & **39.54** & **0.32** & **8.28** & **0.26** & **4.04** \\ \hline \end{tabular}
\end{table}
Table 3: Ablative results show the fraction of conflicting gradients and \(\Delta_{Gap}\) on the UPMC Food-101 and Hateful Memes datasets, comparing scenarios without inter-task gradient projection (“w/o Projection”) and with standard BalGrad (“w/ Projection”).





## Acknowledgment

This research was supported by the Institute for Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2021-0-01341, Artificial Intelligence Graduate School Program, Chung-Ang University). This research was also supported by the MSIT (Ministry of Science and ICT), Korea , under the Graduate School of Metaverse Convergence support program (IITP-2025-RS-2024-00418847) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation).

## References

* Aggarwal et al. (2024) Piush Aggarwal, Jawar Mehrabanian, Weigang Huang, Ozge Alacam, and Torsten Zesch. 2024. Text or image? what is more important in cross-domain generalization capabilities of hate meme detection models? In _Findings of the Association for Computational Linguistics: EACL 2024_, pages 104-117.
* Arevalo et al. (2017) John Arevalo, Thamar Solorio, Manuel Montes-y Gomez, and Fabio A Gonzalez. 2017. Gated multimodal units for information fusion. In _Proceedings of the International Conference on Learning Representations: Workshop Track_.
* Arpit et al. (2017) Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. 2017. A closer look at memorization in deep networks. In _Proceedings of the International Conference on Machine Learning_, pages 233-242.
* Cheplygina et al. (2019) Veronika Cheplygina, Marleen De Bruijne, and Josien PW Pluim. 2019. Not-so-supervised: a survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. _Medical image analysis_, 54:280-296.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4171-4186.
* Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In _Proceedings of the International Conference on Learning Representations_.
* Garg et al. (2022) Muskan Garg, Seema Wazarkar, Muskaan Singh, and Ondrej Bojar. 2022. Multimodality for nlp-centered applications: Resources, advances and frontiers. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 6837-6847.
* Guo et al. (2023) Yangyang Guo, Liqiang Nie, Harry Cheng, Zhiyong Cheng, Mohan Kankanhalli, and Alberto Del Bimbo. 2023. On modality bias recognition and reduction. _ACM Transactions on Multimedia Computing, Communications and Applications_, 19(3):1-22.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 770-778.
* Hinton et al. (2014) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the knowledge in a neural network. In _Proceedings of the NeurIPS 2014 Deep Learning and Representation Learning Workshop_.
* Hu et al. (2022) Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022. Scaling up vision-language pre-training for image captioning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 17980-17989.
* Huang et al. (2022) Yu Huang, Junyang Lin, Chang Zhou, Hongxia Yang, and Longbo Huang. 2022. Modality competition: What makes joint training of multi-modal network fail in deep learning? (provably). In _Proceedings of the International Conference on Machine Learning_, pages 9226-9259.
* Khademi et al. (2023) Mahmoud Khademi, Ziyi Yang, Felipe Frujeri, and Chenguang Zhu. 2023. Mm-reasoner: A multimodal knowledge-aware framework for knowledge-based visual question answering. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 6571-6581.
* Kiela et al. (2019) Douwe Kiela, Suvrat Biooshan, Hamed Firooz, and Davide Testugine. 2019. Supervised multimodal bitransformers for classifying images and text. _arXiv preprint arXiv:1909.02950_.
* Kiela et al. (2020) Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringsha, and Davide Testuggine. 2020. The hateful memes challenge: Detecting hate speech in multimodal memes. In _Advances in Neural Information Processing Systems_, pages 2611-2624.
* Kruk et al. (2019) Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, Dan Jurafsky, and Ajay Divakaran. 2019. Integrating text and image: Determining multimodal document intent in instagram posts. _arXiv preprint arXiv:1904.09073_.
* Kumar and Nandekumar (2022) Gokul Karthik Kumar and Karthik Nandekumar. 2022. Hate-clipper: Multimodal hateful meme classification based on cross-modal interaction of clip features. _arXiv preprint arXiv:2210.05916_

Yi-Lun Lee, Yi-Hsuan Tsai, Wei-Chen Chiu, and Chen-Yu Lee. 2023. Multimodal prompting with missing modalities for visual recognition. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 14943-14952.
* Li et al. (2023) Hong Li, Xingyu Li, Pengbo Hu, Yinuo Lei, Chunxiao Li, and Yi Zhou. 2023. Boosting multi-modal model performance with adaptive gradient modulation. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 22214-22224.
* Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In _Proceedings of the International Conference on Machine Learning_, pages 12888-12900. PMLR.
* Lim et al. (2023) Jongin Lim, Youngdong Kim, Byungjai Kim, Chanho Ahn, Jinwoo Shin, Eunho Yang, and Seungju Han. 2023. Biasadvv: Bias-adversarial augmentation for model debiasing. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 3832-3841.
* Liu et al. (2024) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2024. Visual instruction tuning. _Advances in Neural Information Processing Systems_, 36.
* Liu et al. (2023) Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv, Zhiyuan Liu, and Ge Yu. 2023. Universal vision-language dense retrieval: Learning a unified representation space for multi-modal retrieval. In _Proceedings of the International Conference on Learning Representations_.
* Manolache et al. (2021) Andrei Manolache, Florin Brad, and Elena Burceanu. 2021. Date: Detecting anomalies in text via self-supervision of transformers. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 267-277.
* Menini et al. (2020) Stefano Menini, Alessio Palmero Aprosio, and Sara Tonelli. 2020. A multimodal dataset of images and text to study abusive language. In _Proceedings of the Seventh Italian Conference on Computational Linguistics, CLiC-it 2020_, volume 2769. CEUR-WS. org.
* Mishra et al. (2023) Shreyash Mishra, Suryavardam, Parth Patwa, Megha Chakraborty, Anku Rani, Ashshwarya Reganti, Aman Chadha, Amitava Das, Amit Sheth, Manoj Chinnakotla, et al. 2023. Memotion 3: Dataset on sentiment and emotion analysis of codemixed hindindigenlism memes. _arXiv preprint arXiv:2303.09892_.
* Nam et al. (2020) Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. 2020. Learning from failure: Debiasing classifier from biased classifier. In _Advances in Neural Information Processing Systems_, pages 20673-20684.
* Peng et al. (2022) Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu. 2022. Balanced multimodal learning via on-the-fly gradient modulation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 8238-8247.
* Phuong and Lampert (2019) Mary Phuong and Christoph Lampert. 2019. Towards understanding knowledge distillation. In _Proceedings of the International Conference on Machine Learning_, pages 5142-5151.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sasruty, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In _Proceedings of the International Conference on Machine Learning_, pages 8748-8763. PMLR.
* Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. _arXiv preprint arXiv:1910.01108_.
* Shi et al. (2023) Guangyuan Shi, Qimai Li, Wenlong Zhang, Jiaxin Chen, and Xiao-Ming Wu. 2023. Recon: Reducing conflicting gradients from the root for multi-task learning. In _Proceedings of the International Conference on Learning Representations_.
* Voigt and Bussche (2017) Paul Voigt and Axel Von dem Bussche. 2017. The eu general data protection regulation (gdpr). _A Practical Guide, 1st Ed., Cham: Springer International Publishing_, 10(3152676):10-5555.
* Wang et al. (2020) Weiyaoug Wang, Du Tran, and Matt Feiszli. 2020. What makes training multi-modal classification networks hard? In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 12695-12705.
* Wang et al. (2015) Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. 2015. Recipe recognition with large multimodal food dataset. In _Proceedings of the 2015 IEEE International Conference on Multimedia & Expo Workshops_, pages 1-6.
* Wang et al. (2019) Zirui Wang, Zihang Dai, Barnabas Poczos, and Jaime Carbonell. 2019. Characterizing and avoiding negative transfer. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 11293-11302.
* Welinder et al. (2010) Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. 2010. Caltech-ucsd birds 200. _Technical Report CNS-TR-2010-001, California Institute of Technology_.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remii Loor, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45 

* Woo et al. (2023) Sangmin Woo, Sumin Lee, Yeeonju Park, Muhammad Adi Nugroho, and Changick Kim. 2023. Towards good practices for missing modality robust action recognition. In _Proceedings of the AAAI Conference on Artificial Intelligence_, pages 2776-2784.
* Wu et al. (2022) Nan Wu, Stanislaw Jastrzebski, Kyunghyun Cho, and Krzysztof J Geras. 2022. Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks. In _Proceedings of the International Conference on Machine Learning_, pages 24043-24055.
* Yang et al. (2024) Zequn Yang, Yake Wei, Ce Liang, and Di Hu. 2024. Quantifying and enhancing multi-modal robustness with modality preference. In _Proceedings of the International Conference on Learning Representations_.
* Yao and Mihalcea (2022) Yiqun Yao and Rada Mihalcea. 2022. Modality-specific learning rates for effective multimodal additive latefusion. In _Findings of the Association for Computational Linguistics: ACL 2022_, pages 1824-1834.
* Yu et al. (2020) Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. In _Advances in Neural Information Processing Systems_, pages 5824-5836.
* Yuan et al. (2023) Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Fei Huang, and Songfang Huang. 2023. Hype: Better pre-trained language model fine-tuning with hidden representation perturbation. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics_, pages 3246-3264.
* Zhang et al. (2018) Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. 2018. Deep mutual learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 4320-4328.

## Appendix A Appendix of Propositions

### Proof of Proposition 1

**Proposition 1.** (Gradient Effect on Change of Loss) Let the parameters \(\theta_{v},\theta_{l}\), and \(\theta_{\mathcal{T}}\) of a multi-modal model be updated with gradients \(g_{v},g_{l}\), and \(g_{\mathcal{T}}\) using a sufficiently small step size \(\lambda>0\), resulting in updated parameters \(\hat{\theta}_{v},\hat{\theta}_{l},\) and \(\hat{\theta}_{\mathcal{T}}\). Then the change in the loss function satisfies

\[\begin{split}\triangle\mathcal{L}&=~{}-2\,\lambda \left(g^{v}_{\mathcal{T}}\cdot g^{l}_{\mathcal{T}}\right)\\ &-\,\lambda\sum_{i\in\{v,l,\mathcal{T}\}}\Big{(}\,g_{i}\cdot g _{i}~{}+~{}g^{i}_{\mathcal{T}}\cdot g^{i}_{\mathcal{T}}\Big{)}~{}+~{}O( \lambda^{2}),\end{split} \tag{7}\]

where the cross term \(-2\,\lambda\left(g^{v}_{\mathcal{T}}\right)\cdot(g^{l}_{\mathcal{T}})\) captures the interaction between the visual and language gradients and the magnitudes and directions of each gradient \(g^{v}_{\mathcal{T}}\) and \(g^{l}_{\mathcal{T}}\) governs how much the overall loss is reduced.

Proof of Proposition 1.: Let the \(\theta_{v},\theta_{l},\theta_{\mathcal{T}}\) be updated in the direction of negative gradients \(g_{v},g_{l},g_{\mathcal{T}}\) with step size \(\lambda>0\). Then the updated \(\hat{\theta}_{v},\hat{\theta}_{l},\hat{\theta}_{\mathcal{T}}\) are \(\theta_{v}-\lambda g_{v},\theta_{l}-\lambda g_{gl},\theta_{\mathcal{T}}- \lambda g_{\mathcal{T}}\). In that case, the change in the loss function with updated parameters is

\[\triangle\mathcal{L}=\mathcal{L}(\hat{\theta}_{v},\hat{\theta}_{l},\hat{ \theta}_{\mathcal{T}})-\mathcal{L}(\theta_{v},\theta_{l},\theta_{\mathcal{T}})\]

By the first-order taylor expansion with a point \((\theta_{v},\theta_{l},\theta_{\mathcal{T}})\),

\[\begin{split}&\mathcal{L}(\hat{\theta}_{v},\hat{\theta}_{l}, \hat{\theta}_{\mathcal{T}})-\mathcal{L}(\theta_{v},\theta_{l},\theta_{ \mathcal{T}})\\ &=\mathcal{L}(\theta_{v}-\lambda g_{v},\theta_{l}-\lambda g_{l}, \theta_{\mathcal{T}}-\lambda g_{\mathcal{T}})\\ &\quad-\mathcal{L}(\theta_{v},\theta_{l},\theta_{\mathcal{T}})\\ &=\mathcal{L}(\theta_{v},\theta_{l},\theta_{\mathcal{T}})\\ &\quad+(\theta_{v}-\lambda g_{v}-\theta_{v})^{T}g_{v}(\theta_{ l}-\lambda g_{l}-\theta_{l})^{T}g_{l}\\ &\quad+(\theta_{\mathcal{T}}-\lambda g_{\mathcal{T}}-\theta_{ \mathcal{T}})^{T}g_{\mathcal{T}}\\ &\quad-\mathcal{L}(\theta_{v},\theta_{l},\theta_{\mathcal{T}})++ O(\lambda^{2})\\ &=-\lambda(g^{T}_{v}\cdot g_{v}+g^{T}_{l}\cdot g_{l})\\ &\quad-\lambda(g^{l}_{\mathcal{T}}+g^{v}_{\mathcal{T}})^{T}(g^{l }_{\mathcal{T}}+g^{v}_{\mathcal{T}})\\ &\quad+O(\lambda^{2})\\ &=-2\lambda g^{v}_{\mathcal{T}}\cdot g^{l}_{\mathcal{T}}-\lambda \sum_{i\in\{v,l,\mathcal{T}\}}(g_{i}\cdot g_{i}+g^{i}_{\mathcal{T}}\cdot g^{ i}_{\mathcal{T}})\\ &\quad+O(\lambda^{2})\end{split}\]

Influence of Fusion Methods.The term \(\big{(}g^{v}_{\mathcal{T}}\big{)}^{\top}\big{(}g^{l}_{\mathcal{T}}\big{)}\) captures how the visual and language gradients interact within the classifier parameters \(\theta_{\mathcal{T}}\). Different fusion methods yield different dependencies of \(g^{v}_{\mathcal{T}}\) and \(g^{l}_{\mathcal{T}}\) on \(v\) and \(l\):

* **Addition:** The fused input is \(x=v+l\). Because \(v\) and \(l\) are merged by simple addition, their representations feed directly into the same part of the classifier. Consequently, \(\big{(}g^{v}_{\mathcal{T}}\big{)}^{\top}\big{(}g^{l}_{\mathcal{T}}\big{)}\) often remains significant due to the shared pathway.
* **Concatenation:** The fused input is \(x=[v;l]\). Each modality is placed in distinct segments of the classifier's input vector, reducing direct interactions. As a result, \(g^{v}_{\mathcal{T}}\) and \(g^{l}_{\mathcal{T}}\) may be more independent, potentially lowering the cross term \(\big{(}g^{v}_{\mathcal{T}}\big{)}^{\top}\big{(}g^{l}_{\mathcal{T}}\big{)}\).
* **Attention:** The fused input is \(x=\mathrm{Attention}(v,l)\). This method can create strong interdependence between \(v\) and \(l\) within \(\theta_{\mathcal{T}}\). Hence, it \(\big{(}g^{v}_{\mathcal{T}}\big{)}^{\top}\big{(}g^{l}_{\mathcal{T}}\big{)}\) can become highly influential since changes \(v\) affect \(l\) and vice versa through the attention mechanism 

Hence, the sign and magnitude of the cross term reflect how strongly the parameters for the two modalities are tied together under each fusion strategy.

### Proof of proposition 2

**Proposition 2.** (Gradient Conflicts on Loss Reduction with KL Loss) Let \({\mathcal{G}^{\tau}}=\left\{\,g_{v},\,g_{l},\,g_{\tau}\right\}\) and \({\mathcal{G}^{kl}}=\left\{\,g_{v}^{kl},\,g_{l}^{kl},\,0\right\}\) be the gradients from a target loss \({\mathcal{L}_{\tau}}\) and a KL loss \({\mathcal{L}_{bl}}\), respectively, with parameters \(\theta=[\theta_{v},\theta_{l},\theta_{\tau}]^{\top}\). Assume the parameters are updated by gradient descent with a small step size \(\lambda>0\): \(\theta_{v}^{\prime}=\theta_{v}-\lambda\left(g_{v}+g_{lv}^{kl}\right),\quad \theta_{l}^{\prime}=\theta_{l}-\lambda\left(g_{l}+g_{l}^{kl}\right),\quad \theta_{\tau}^{\prime}=\theta_{\tau}-\lambda\,g_{\tau}\). Then, for the combined loss \({\mathcal{L}}={\mathcal{L}_{\tau}}+{\mathcal{L}_{kl}}\), the change in the loss is

\[\begin{split}\Delta{\mathcal{L}}&={\mathcal{L}} \big{(}\theta^{\prime}\big{)}-{\mathcal{L}}\big{(}\theta\big{)}\\ &=-\lambda\Big{(}\|{\mathcal{G}^{\tau}}\|^{2}+\,\|{\mathcal{G}^{ kl}}\|^{2}+\,2\,({\mathcal{G}^{\tau}})^{\top}{\mathcal{G}^{kl}}\Big{)}\\ &+O(\lambda^{2}).\end{split} \tag{8}\]

In particular, if \(({\mathcal{G}^{\tau}})^{\top}{\mathcal{G}^{kl}}<0\), the gradients from the target and KL losses _conflict_, reducing the effective loss reduction.

**Proof of Proposition 2.**

Proof.: Because \({\mathcal{L}}={\mathcal{L}_{\tau}}+{\mathcal{L}_{kl}}\), its gradient is

\[\nabla_{\theta}{\mathcal{L}}={\mathcal{G}^{\tau}}+{\mathcal{G}^{kl}}.\]

Under a small step size \(\lambda\), a first-order Taylor expansion about \(\theta\) gives \(\Delta{\mathcal{L}}\approx-\lambda\,\|{\mathcal{G}^{\tau}}+{\mathcal{G}^{kl}} \|^{2}\). Since \({\mathcal{G}^{\tau}}=\{g_{v}^{\tau},g_{l}^{\tau},g_{\tau}^{\tau}\}\) and \({\mathcal{G}^{kl}}=\{g_{v}^{kl},g_{l}^{kl},0\}\), the relevant parameters are updated as:

\[\begin{split}\theta_{v}^{\prime}&=\theta_{v}- \lambda\,(g_{v}^{\tau}+g_{v}^{kl})\\ \theta_{t}^{\prime}&=\theta_{l}-\lambda\,(g_{l}^{ \tau}+g_{l}^{kl})\\ \theta_{\tau}^{\prime}&=\theta_{\tau}-\lambda\,g_{ \tau}^{\tau}.\end{split}\]

By decomposing norm:

\[\begin{split}\Delta{\mathcal{L}}=&-\lambda\Big{(}\| g_{v}^{\tau}\|^{2}+\|g_{v}^{kl}\|^{2}+2\,g_{v}^{\tau})^{\top}(g_{v}^{kl})\\ &+\|g_{l}^{\tau}\|^{2}+\|g_{l}^{kl}\|^{2}+2\,g_{l}^{\tau})^{\top} (g_{l}^{kl})+\|g_{\tau}^{\tau}\|^{2}\Big{)}\\ &+O(\lambda^{2})\end{split}\]

Hence, if either \((g_{v}^{\tau})^{\top}(g_{v}^{kl})<0\) or \((g_{l}^{\tau})^{\top}(g_{l}^{kl})<0\), the negative cross-term reduces the effective loss decrease for that modality. 

## Appendix B Further Implementation Details

### Dataset and Evaluation Metrics

**UPMC Food-101**Wang et al. (2015) is a food classification dataset with 101 categories and 90,840 image-text pairs, involving the classification of food items using both images and textual recipe descriptions; to create a validation split, we extracted 5,000 samples from the training set Kiela et al. (2019), as the dataset only provides training and testing sets.

**Hateful Memes**Kiela et al. (2020) is designed to detect hate speech by combining image and text modalities, comprising 8,500 training samples, 1,000 validation samples, and 500 test samples.

**MM-IMdb**Arevalo et al. (2017) is a multi-label movie genre classification dataset that incorporates poster images and plot descriptions, containing 23 genre tags with 15,552 training samples, 2,608 validation samples, and 7,799 test samples.

We utilize classification accuracy, AUROC, and F1-Macro as evaluation metrics for the UPMC Food-101, Hateful Memes, and MM-IMDb datasets, respectively.

### Architecture and Training Scheme

In all comparative experiments, we employ ViT Dosovitskiy et al. (2021) and BERT Devlin et al. (2019) as image and text encoders, respectively. We adopt a late concatenation architecture where the embeddings from each modality are concatenated to make the final prediction. We employ linear probing as our fine-tuning strategy, which freezes all the encoder parameters and trains only the embedding and classifier layers.

We adopt this modular architecture and fine-tuning scheme for several key reasons: First, the modular design of BalGrad allows it to extend to various encoders, easily accommodating different architectures. This flexibility is crucial in real-world scenarios where resources are often constrained. Our structure supports a range of scalable encoder configurations, ensuring adaptability to different resource availability and application requirements. Additionally, in some cases, data access is restricted due to privacy concerns, necessitates the use of pre-extracted features Cheplygina et al. (2019); Kruk et al. (2019); Menini et al. (2020), making the application of early fusion-based large VLMs Liu et al. (2024) impractical. Also, to focus improvements on BalGrad's gradient reweighting and projection, we adopt linear probing as a 

fine-tuning strategy, ensuring that the gains were not merely due to the encoders' inherent capabilities but to our method's effectiveness.

As a baseline, we adopt a standard linear probing approach and compare our proposed method against existing methods designed to balance modalities in VL models, specifically MSLR Yao and Mihalcea (2022), OGM-GE Peng et al. (2022), and AGM Li et al. (2023).

### Implementation Details

We use vit-base and bert-base-uncased checkpoints as the image and text encoders, respectively, loading them from Transformers library Wolf et al. (2020). The embeddings extracted from each encoder have a dimensionality of 768, and we concatenate these embeddings to form a 1568-dimensional vector, which is then passed to a final classifier. We resize all images to \(224\times 224\) and apply a random horizontal flip for augmentation. For text, the maximum sequence lengths are set to 1024 for MM-IMDb, 512 for UPMC Food-101, and 128 for Hateful Memes. We use the Adam optimizer with a momentum of 0.9 for all experiments, training for 20 epochs with a batch size of 128.

## Appendix C Additional Experimental Results

### Experimental Results on Different Fusion Mechanisms

The way embeddings from different modalities are fused can significantly impact a model's ability to capture and leverage cross-modal interactions. We conducted experiments on different fusion strategies in the baseline and BalGrad, specifically exploring element-wise addition and attention-based fusion mechanisms following previous work Kumar and Nandakumar (2022). We tested these mechanisms on the UPMC Food-101, Hateful Memes, and MM-IMDb datasets, evaluating the \(\Delta_{Gap}\) in performance under conditions where either the image or text modality was missing. Results for addition and attention are presented in Figure 8. Across all datasets, BalGrad demonstrated the smallest \(\Delta_{Gap}\) with both fusion mechanisms, effectively mitigating dominant modality bias. This confirms that BalGrad effectively captures and leverages cross-modal interactions across different fusion mechanisms.

### Experimental Results on Different Backbone Models

We conduct extensive experiments across diverse backbone models, underscoring its consistent per

Figure 8: Bar plots illustrating the performance of existing methods and BalGrad with different fusion mechanisms: **(a)** addition and **(b)** attention, evaluated on the UPMC Food-101, Hateful Memes, and MM-IMDb datasets. Each bar indicates \(\Delta_{Gap}\)(%), which quantifies the performance variation between missing image and missing text conditions.

Figure 9: Bar plots presenting the performance comparison between existing methods and BalGrad across different backbone models: **(a)** ResNet and DistilBERT, and **(b)** CLIP, on the UPMC Food-101, Hateful Memes, and MM-IMDb datasets. Each bar represents \(\Delta_{Gap}\)(%), measuring the performance discrepancy under missing image and missing text conditions.

 

formance and adaptability to varying architectures and computational resources. Specifically, we employ lower-capacity models--ResNet-50 (He et al., 2016) for the image encoder and DistilBERT Sanh et al. (2019) for the text encoder--to assess robustness concerning model size. Additionally, we leverage the widely-used multimodal pretrained VLM, CLIP Radford et al. (2021), for further evaluation due to its strong ability to seamlessly integrate visual and textual information, providing a rigorous test for BalGrad. Experiments were carried out on the UPMC Food-101, Hateful Memes, and MM-IMdb datasets, assessing the performance gap under conditions where either the image or text modality was missing. Results for ResNet-DistilBERT and CLIP are presented in Figure 9. Across all datasets, BalGrad consistently exhibited the smallest \(\Delta_{Gap}\), effectively balancing the contributions between modalities.

Intriguingly, while earlier experiments using ViT and BERT encoders on the MM-IMDb dataset showed no over-reliance on a specific modality, our additional studies reveal that conventional methods tend to heavily rely on the text modality, when employing ResNet and DistilBERT. These findings indicate that such bias is influenced not only by the task but also by the choice of backbone model. Our comprehensive experiments affirm that BalGrad effectively mitigates bias irrespective of the backbone model employed, showcasing its superior scalability.

### Experimental Results with Additional Datasets

To validate the generalizability of BalGrad, we conduct experiments on two additional datasets: Memotion Mishra et al. (2023) and CUB-200-2011 Welinder et al. (2010). The Memotion dataset, used for classifying the humor level of meme images based on their descriptions, includes annotations such as "not funny", "funny", "very funny", and "hilarious". The CUB-200-2011 dataset is a fine-grained bird classification dataset, requiring the categorization of 200 bird species based on images and descriptions. We evaluate each dataset using weighted F1 score and classification accuracy.

The results for the Memotion dataset, presented in Table 4, show that when the text modality is missing, performance drops significantly more than when the image modality is missing, indicating a bias toward the text modality. BalGrad not only achieves the highest performance with the image modality alone but also excels in the _Avg._ and \(\Delta_{Gap}\) metrics, demonstrating effective modality balance.

As shown in Table 5, the CUB-200-2011 dataset exhibits a strong reliance on the image modality. BalGrad outperforms AGM by more than 4%p in accuracy when the image modality is missing and achieves the smallest \(\Delta_{Gap}\) at 8.63%, demonstrating its superiority in handling fine-grained classification tasks even under challenging conditions.

\begin{table}
\begin{tabular}{r c|c c c c c} \hline \hline \multirow{2}{*}{Modality} & \multicolumn{7}{c}{**CUB-200-2011**} \\ \cline{2-7}  & Baseline & MSLR & OGM-GE & AGM & BalGrad \\ \hline Full & 70.55 & 70.36 & 70.28 & **71.12** & 70.77 \\ \hline \multirow{4}{*}{Missing} & Image & 58.34 & 59.24 & **59.66** & 59.54 & 59.48 \\  & Text & 49.29 & 51.32 & 50.38 & 51.44 & **52.78** \\ \cline{1-1}  & \(Ay_{\text{F}}\!\cdot\!1\) & 53.82 & 55.28 & 55.02 & 55.49 & **56.13** \\ \cline{1-1}  & \(\Delta_{Gap}\!\cdot\!1\) & 4.53 & 3.96 & 4.64 & 4.05 & **3.35** \\ \hline \hline \end{tabular}
\end{table}
Table 4: The experimental result of BalGrad on the Memotion dataset. The best result in each test dataset is boldfaced, and the second best is presented with underline. “_Avg._” represents the average performance under conditions where one of the modality is missing, while “\(\Delta_{Gap}\)(%)” indicates the performance difference.

\begin{table}
\begin{tabular}{r l|l l l l l} \hline \hline \multirow{2}{*}{Modality} & \multicolumn{7}{c}{**CUB-200-2011**} \\ \cline{2-7}  & Baseline & MSLR & OGM-GE & AGM & BalGrad \\ \hline Full & 74.71 & 72.12 & 75.15 & **76.28** & 75.84 \\ \hline \multirow{4}{*}{Missing} & Image & 37.38 & 40.21 & 39.49 & 41.24 & **45.47** \\  & Text & 61.24 & 60.20 & 59.14 & 61.42 & **62.72** \\ \cline{1-1}  & Avg.\(\uparrow\) & 49.31 & 50.21 & 49.32 & 51.33 & **54.10** \\ \cline{1-1}  & \(\Delta_{Gap}\!\cdot\!1\) & 11.93 & 9.99 & 9.83 & 10.09 & **8.63** \\ \hline \hline \end{tabular}
\end{table}
Table 5: The results of BalGrad on the CBU-200-2011 dataset are presented. The highest performance in each test dataset is shown in bold, with the second-highest underlined. “_Avg._” reflects the average performance when one modality is absent, and “\(\Delta_{Gap}\)(%)” denotes the performance difference.