See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language
  Balance to Mitigate Dominant Modality Bias
Authors: name: JuneHyoung Kwon; name: MiHyeon Kim; name: Eunju Lee; name: Juhwan Choi; name: YoungBin Kim
Published: 2025-03-18 02:17:41

Abstract: Vision-language (VL) models have demonstrated strong performance across
various tasks. However, these models often rely on a specific modality for
predictions, leading to "dominant modality bias.'' This bias significantly
hurts performance, especially when one modality is impaired. In this study, we
analyze model behavior under dominant modality bias and theoretically show that
unaligned gradients or differences in gradient magnitudes prevent balanced
convergence of the loss. Based on these findings, we propose a novel framework,
BalGrad to mitigate dominant modality bias. Our approach includes
inter-modality gradient reweighting, adjusting the gradient of KL divergence
based on each modality's contribution, and inter-task gradient projection to
align task directions in a non-conflicting manner. Experiments on UPMC
Food-101, Hateful Memes, and MM-IMDb datasets confirm that BalGrad effectively
alleviates over-reliance on specific modalities when making predictions.