Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving
  Vision-Linguistic Compositionality
Authors: name: Youngtaek Oh, name: Jae Won Cho, name: Dong-Jin Kim, name: In So Kweon, name: Junmo Kim

In this paper, we propose a new method to enhance compositional understanding
in pre-trained vision and language models (VLMs) without sacrificing
performance in zero-shot multi-modal tasks. Traditional fine-tuning approaches
often improve compositional reasoning at the cost of degrading multi-modal
capabilities, primarily due to the use of global hard negative (HN) loss, which
contrasts global representations of images and texts. This global HN loss
pushes HN texts that are highly similar to the original ones, damaging the
model's multi-modal representations. To overcome this limitation, we propose
Fine-grained Selective Calibrated CLIP (FSC-CLIP), which integrates local hard
negative loss and selective calibrated regularization. These innovations
provide fine-grained negative supervision while preserving the model's
representational integrity. Our extensive evaluations across diverse benchmarks
for both compositionality and multi-modal tasks show that FSC-CLIP not only
achieves compositionality on par with state-of-the-art models but also retains
strong multi-modal capabilities. Code is available at:
https://github.com/ytaek-oh/fsc-clip.