Title: Variational Adapter for Cross-modal Similarity Representation
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Variational Adapter, cross-modal similarity representation, cross-modal retrieval, domain generalization

Abstract: The core of vision-language models lies in measuring cross-modal similarity within a unified representation space. However, most image-text matching or multi-class image classification datasets lack fine-grained cross-modal matching annotations, forcing the continuous similarity space into binary classification boundaries. This compression induces false negative samples and significantly impairs the generalization performance of cross-modal tasks. While prior research has attempted to mitigate this by modeling intra-modal ambiguity, it often overlooks inherent annotation flaws, leading to suboptimal uncertainty allocation.
To address these challenges, we propose a Variational Adapter for Cross-modal Similarity Representation (VACSR). This approach reformulates image-text matching under fine-grained semantic scarcity as a variational inference problem. It constructs a latent space for cross-modal similarity and uses regularization techniques to mitigate overfitting to binary annotations. Additionally, we introduce a distributional optimization loss to eliminate erroneous gradients caused by false negative samples.
We validate the effectiveness of VACSR in image-text retrieval tasks using the COCO Caption dataset and two extended datasets: CxC and ECCV Caption. Furthermore, we conduct domain generalization experiments on ImageNet and its four variant datasets, further demonstrating the robustness of VACSR. Finally, correlation analysis between uncertainty and retrieval accuracy confirms that the uncertainty learned by VACSR is highly interpretable and provides a reliable basis for assessing the credibility of retrieval results.