ReCLIP: Refine Contrastive Language Image Pre-Training with Source Free
  Domain Adaptation
Authors: name: Hu. Xuefeng; name: Zhang. Ke; name: Xia. Lu; name: Chen. Albert; name: Luo. Jiajia; name: Sun. Yuyin; name: Wang. Ken; name: Qiao. Nan; name: Zeng. Xiao; name: Sun. Min; name: Kuo. Cheng-Hao; name: Nevatia. Ram
Published: 2023-08-04 18:11:40

Abstract: Large-scale Pre-Training Vision-Language Model such as CLIP has demonstrated
outstanding performance in zero-shot classification, e.g. achieving 76.3% top-1
accuracy on ImageNet without seeing any example, which leads to potential
benefits to many tasks that have no labeled data. However, while applying CLIP
to a downstream target domain, the presence of visual and text domain gaps and
cross-modality misalignment can greatly impact the model performance. To
address such challenges, we propose ReCLIP, the first source-free domain
adaptation method for vision-language models, which does not require any source
data or target labeled data. ReCLIP first learns a projection space to mitigate
the misaligned visual-text embeddings and learns pseudo labels, and then
deploys cross-modality self-training with the pseudo labels, to update visual
and text encoders, refine labels and reduce domain gaps and misalignments
iteratively. With extensive experiments, we demonstrate ReCLIP reduces the
average error rate of CLIP from 30.17% to 25.06% on 22 image classification
benchmarks.