TransAgent: Transfer Vision-Language Foundation Models with
  Heterogeneous Agent Collaboration
Authors: name: Yiwei Guo; name: Shaobin Zhuang; name: Kunchang Li; name: Yu Qiao; name: Yali Wang
Published: 2024-10-16 03:01:44

Abstract: Vision-language foundation models (such as CLIP) have recently shown their
power in transfer learning, owing to large-scale image-text pre-training.
However, target domain data in the downstream tasks can be highly different
from the pre-training phase, which makes it hard for such a single model to
generalize well. Alternatively, there exists a wide range of expert models that
contain diversified vision and/or language knowledge pre-trained on different
modalities, tasks, networks, and datasets. Unfortunately, these models are
"isolated agents" with heterogeneous structures, and how to integrate their
knowledge for generalizing CLIP-like models has not been fully explored. To
bridge this gap, we propose a general and concise TransAgent framework, which
transports the knowledge of the isolated agents in a unified manner, and
effectively guides CLIP to generalize with multi-source knowledge distillation.
With such a distinct framework, we flexibly collaborate with 11 heterogeneous
agents to empower vision-language foundation models, without further cost in
the inference phase. Finally, our TransAgent achieves state-of-the-art
performance on 11 visual recognition datasets. Under the same low-shot setting,
it outperforms the popular CoOp with around 10% on average, and 20% on EuroSAT
which contains large domain shifts.