See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias
Authors: Junehyoung Kwon, MiHyeon Kim, Eunju Lee, Juhwan Choi, YoungBin Kim


Reviewer paper summaries: The paper proposes a gradient reweighting method to finetune late fusion VL encoders (the encoders themselves are frozen) for three specific tasks. The method’s goal is to avoid unimodal collapse and make the role of the two modalities more balanced.

**Comments after Author Discussion**: Thanks to the authors for the many responses. However, I cannot reliably guarantee the soundness of the contributions and the methodology of the paper at its current state, I would like to keep my scores and suggest that the authors revise their work.
Summary of strengths: 1. The method’s results show a fair, although not incredibly high gain in performance, but given the weaknesses of this paper, the usefulness of this method can be quite unconvincing.
Summary of weaknesses: 1. *Limited applicability*: The paper works with quite simple baseline models (BERT and ViT encoders and simple fusion mechanism, namely concatenation) trained specifically for one task and it is quite unclear how this would impact modern, task-overarching and more powerful VL encoders (ViLBERT and the like) and especially VL decoders (LLaVA and the like). The task-specific training is not even full, because the ViT and BERT encoders are frozen and just the embedding layers are trained. It would have been more useful to conduct such a study on more relevant and generalist models, that have been shown to be very imbalanced, even given training on multiple tasks and more complex modality fusion architectures. https://arxiv.org/abs/2404.18624  

1. *Limited novelty*: It is unclear how this paper differs from others. Gradient reweighting to encourage a better balance between modalities during learning has been conducted before, as the paper itself also cites in L170-173. However, starting L173 reads: “However, these methods often lack empirical validation on large capacity models and may introduce negative effects.” Given that this work too does not work with modern and generalist models (see weakness 1. above), and the lack of explanation of why and what negative effects related work introduced, I am completely unconvinced that this approach differs significantly from prior work.
Comments, suggestions and typos: The title is a bit hard to comprehend. I think it was meant to sound witty, but it is just obfuscated. 

L044 “dominant modality bias” has been also called “unimodal collapse” in the literature, see https://aclanthology.org/2023.acl-long.223.pdf . 

L121: do you mean “the loss”? 

L210: What exactly does muting one modality mean? Zero-ing out? Replacing with noise? MASK/PAD tokens? Something else? Please be more specific in a technical paper. 

Figure 6 caption does not mention what the cosine similarity compares. I assume from the main body it is between gradients. It would be helpful to add this.
Overall assessment scores: 2