Discriminative Fine-tuning of LVLMs
Authors: name: Yassine Ouali; name: Adrian Bulat; name: Alexandros Xenos; name: Anestis Zaganidis; name: Ioannis Maniadis Metaxas; name: Georgios Tzimiropoulos; name: Brais Martinez
Published: 2024-12-05 17:54:27

Abstract: Contrastively-trained Vision-Language Models (VLMs) like CLIP have become the
de facto approach for discriminative vision-language representation learning.
However, these models have limited language understanding, often exhibiting a
"bag of words" behavior. At the same time, Large Vision-Language Models
(LVLMs), which combine vision encoders with LLMs, have been shown capable of
detailed vision-language reasoning, yet their autoregressive nature renders
them less suitable for discriminative tasks.
  In this work, we propose to combine "the best of both worlds": a new training
approach for discriminative fine-tuning of LVLMs that results in strong
discriminative and compositional capabilities. Essentially, our approach
converts a generative LVLM into a discriminative one, unlocking its capability
for powerful image-text discrimination combined with enhanced language
understanding.
  Our contributions include: (1) A carefully designed training/optimization
framework that utilizes image-text pairs of variable length and granularity for
training the model with both contrastive and next-token prediction losses. This
is accompanied by ablation studies that justify the necessity of our
framework's components. (2) A parameter-efficient adaptation method using a
combination of soft prompting and LoRA adapters. (3) Significant improvements
over state-of-the-art CLIP-like models of similar size, including standard
image-text retrieval benchmarks and notable gains in compositionality.