Title: TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration
Venue: NeurIPS 2024
Accepted: yes
Keywords: Vision-Language Models

Few-shot Transfer Learning

Heterogeneous Agent Collaboration
Abstract: Vision-language foundation models (such as CLIP) have recently shown their power in transfer learning, owing to large-scale image-text pre-training. However, target domain data in the downstream tasks can be highly different from the pre-training phase, which makes it hard for such a single model to generalize well. Alternatively, there exists a wide range of expert models that contain diversified vision and/or language knowledge pre-trained on different modalities, tasks, networks, and datasets. Unfortunately, these models are "isolated agents" with heterogeneous structures, and how to integrate their knowledge for generalizing CLIP-like models has not been fully explored. To bridge this gap, we propose a general and concise TransAgent framework, which transports the knowledge of the isolated agents in a unified manner, and effectively guides CLIP to generalize with multi-source knowledge distillation. With such a distinct framework, we flexibly collaborate with 11 heterogeneous agents to empower vision-language foundation models, without further cost in the inference phase. Finally, our TransAgent achieves state-of-the-art performance on 11 visual recognition datasets. Under the same low-shot setting, it outperforms the popular CoOp with around 10\% on average, and 20\% on EuroSAT which contains large domain shifts.

TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration

Abstract

Vision-language foundation models (such as CLIP) have recently shown their power in transfer learning, owing to large-scale image-text pre-training. However, target domain data in the downstream tasks can be highly different from the pre-training phase, which makes it hard for such a single model to generalize well. Alternatively, there exists a wide range of expert models that contain diversified vision and/or language knowledge pre-trained on different modalities, tasks, networks, and datasets. Unfortunately, these models are "isolated agents" with heterogeneous structures, and how to integrate their knowledge for generalizing CLIP-like models has not been fully explored. To bridge this gap, we propose a general and concise TransAgent framework, which transports the knowledge of the isolated agents in a unified manner, and effectively guides CLIP to generalize with multi-source knowledge distillation. With such a distinct framework, we flexibly collaborate with 11 heterogeneous agents to empower vision-language foundation models, without further cost in the inference phase. Finally, our TransAgent achieves stateof-the-art performance on 11 visual recognition datasets. Under the same low-shot setting, it outperforms the popular CoOp with around 10% on average, and 20% on EuroSAT which contains large domain shifts. The code will be released at https://github.com/markywg/transagent .

1 Introduction

Recently, Vision-Language (V-L) foundation models are mainly pre-trained by contrastive learning with massive image-text pairs from web [61, 38, 15]. As a result, they show the potential on a number of downstream visual recognition tasks, by transferring their representations with prompt learning [87, 86] and/or model adaptation [31, 80]. However, target domain data in the open world are diversified, e.g. , EuroSAT [35] refers to satellite images that are highly different from web images in the pre-training. With this large domain shift, it is challenging to achieve good generalization only by adopting such a single model ( e.g. , CLIP), especially under low-shot regime. Alternatively, with the fast development in vision and NLP, there arises a wide range of expert models [33, 9, 43, 49, 7, 18, 64, 11, 46, 12] which contain rich knowledge by pre-training on different modalities, tasks, networks, and datasets. Hence, the natural question is, is it possible to integrate such knowledge to boost vision-language foundation models?

To answer this question, we should further analyze the form of knowledge in these models. The simplest form is the model output, which explicitly exhibits what kind of tasks these models can tackle. However, these models have heterogeneous network structures and outputs, making direct knowledge combinations infeasible. Hence, existing works often choose cascades of these models

(") $%&'()*+%, -*.ℎ 0123 &4.ℎ%5+

(() 164)6*4- %7 2)(,+384,.

Visual

Prompt

Textual

Prompt

DINO

MAE

ViTDet

SAM

89:;<= >?@ABC9D9AC

E?C:? FG?H9@D9AC

Vision	Agents

GPT-3

Vicuna

BERT

Language Agents

Vision-Language

Foundation	Model

PixArt-P

Stable	Diffusion

BLIP2

X2Y `?C?G<D9AC

Y2X a<bD9AC

Multi-modal	Agents

X?[D eC@AH?G

YZ<B?

X?[D

Frozen

Trainable

X?[D

YZ<B?

Visual

Knowledge

Textual

Knowledge

Multi-modal

Knowledge

Knowledge

Versatility

Transfer

Flexibility

Deployment

Efficiency

>?@A

BCDEF

×

×

×

√

×

√

DG?HAIJK>

@LGM

√

√

×

√

√

NOPQ

RSTQU

VW

XTQYOZQ X[[\OY[]

+".$%%

+".'(%

+".%$%

)*+,

-+./01*, 2)34

>C^@

×

√

×

×

[81, 72, 47], according to their output forms. Apparently, such a design lacks flexibility in transfer learning, based on tool invocation in sequence. Moreover, the resulting pipeline is unfriendly for deployment, due to the ensemble of various models in the inference phase.

Another form is the latent representation which implicitly encodes data knowledge in these models [36, 4, 1]. Compared to the explicit output, this implicit representation has a key advantage, i.e. , it is the feature vector that has a homogeneous form among different models. In other words, such vectorized knowledge opens the possibility for a unified integration of these heterogeneous agents. Based on this observation, we propose a general TransAgent framework in Figure 1. To our best knowledge, it is the first unified distillation framework for generalizing vision-language foundation models with efficient heterogeneous agent collaboration. Compared to the previous works mentioned above, our TransAgent contains three distinct technical contributions.

Knowledge Versatility . In our TransAgent, we leverage 11 heterogeneous agents from vision, language and multi-modal research, which comprehensively covers diversified knowledge that is complementary with CLIP-like models, from visual recognition to dense prediction, from chatbot to text encoder, from multi-modal generation to caption.

Transfer Flexibility . First, we provide a generic knowledge extraction method for each modality, allowing us to flexibly extend more agents if necessary in the future. Especially for multi-modal agents, we design a novel manner to extract the prediction score vector of classes as multi-modal knowledge in the target domain, via elaborate mining of vision-language alignment in these models. Second, we introduce a mixture-of-agents gating mechanism for integrating external knowledge of different agents in each modality. This allows our TransAgent to automatically select agents via soft weighting so that it can adaptively tackle few-shot settings in different domains of target datasets.

Deployment Efficiency . We leverage multi-source distillation to transfer knowledge of these heterogeneous agents into CLIP. Since all these pre-trained models are frozen, the fine-tuning effort is neglectable with a few learnable prompts. More importantly, we can unload all the external agents after distillation, i.e. , the inference pipeline with the enhanced CLIP is just the same as the original one, achieving deployment efficiency without a heavy model ensemble.

Finally, we conduct extensive experiments on 11 visual recognition benchmarks, where our TransAgent achieves the state-of-the-art under the same low-shot transfer setting, e.g. , via knowledge collaboration, it outperforms the well-known CoOp [87] with around 10% on average and 20% on EuroSAT which contains large domain shifts. Our method also achieves better results than CaFo [81], which adopts a model ensemble strategy.

2 Related Work

Foundation models. The rapid advancements in deep learning methods have brought abundant pre-trained models to the research area. We group these models into four categories and further demonstrate their ideas below. (i) Vision models: Vision foundation models [33, 9, 3, 59, 32, 13] pre-trained on ImageNet [21] have shown outstanding transfer capability in visual recognition by fine-tuning on downstream datasets. Moreover, various models [43, 49, 16, 17, 8] can be specialists in dense prediction tasks by pre-training on task-relevant domain data. (ii) Large language models: The emergence of large language models (LLMs) [7, 18, 69, 70, 27] has been raising increasing attention from the research community and the public. The astonishing comprehension ability of the LLMs is credited to the linguistic knowledge which can be further applied to solve vision tasks [51, 20, 88, 79]. (iii) Text-to-image generative models: Text-conditioned generation task requires high-level understanding of the given prompts. Recently, diffusion-based generative models [37, 23, 64, 11, 56, 62, 66] have become the state-of-the-art. These models can follow the text conditions faithfully and generate desired outcomes, owing to the semantic knowledge learned during the pre-training stage. (iv) Image-to-text captioning models: These models typically integrate visual knowledge into LLMs to obtain multi-modal understanding abilities [46, 12, 15, 2, 14, 25, 47], offering better experience in referential dialog scenario. In this work, we excavate the underlying knowledge in these heterogeneous models to empower the VL foundation models.

Few-shot adaptation. To efficiently transfer vision-language foundation models like CLIP [61] to downstream tasks, researchers have proposed various adaptation methods, which are primarily based on prompt learning [87, 86, 39-41, 65, 85, 53, 50] or adapter [31, 80, 68, 48, 81, 75, 89]. Lu et al. [53] explore the potential of collaborating CLIP's architectural variants and propose adaptive ensemble strategies to enhance the generalization performance. PromptKD [50] adapts a larger CLIP teacher to downstream datasets and distills the knowledge to a smaller student in an unsupervised manner, separating the need for labeled domain data during transfer. TaskRes [75] proposes to decouple the prior knowledge of the pre-trained models and the task-specific knowledge, enabling reliable old knowledge preservation and flexible new knowledge exploration. GraphAdapter [48] further utilizes the dual-modality structure knowledge for better adaptation in downstream tasks.

Agent collaboration. Considering the complementary knowledge of diverse pre-trained models specialized in different domains or tasks, several works have been proposed to solve vision tasks with agent collaboration [74, 82, 72, 52, 71, 28]. The most relevant to our work is CaFo [81], which transfers the external knowledge using cache models [80]. However, such an ensemble manner introduces further costs in the inference stage. On the contrary, we adopt heterogeneous agent collaboration to aggregate the knowledge, and distillation strategy to inject the knowledge into CLIP, which demonstrates better performance and guarantees deployment efficiency.

Multi-teacher distillation. To improve the effectiveness of knowledge distillation [36], recent works [10, 30, 26, 76-78, 83, 54, 63] attempt to integrate the knowledge from multiple teacher networks. To be noted, how to perform multi-teacher distillation is not trivial, and how to extract and collaborate knowledge from various heterogeneous teachers has not been fully explored for CLIP-like foundation models. In this work, we devise a generic knowledge extraction method and flexible knowledge collaboration mechanism to further enhance the generalization ability of VL foundation models.

3 Method

In this section, we introduce our TransAgent in detail. As shown in Figure 1, it consists of visionlanguage foundation models and agents from different modalities. In the following section, we will illustrate how to collaborate with them for downstream visual recognition tasks. To start with, we briefly review the V-L foundation models by using the well-known CLIP [61]. Specifically, CLIP consists of two branches. In the vision branch, an image is first divided into L non-overlapping equal-sized patches and then projected as input visual tokens V in , which are fed to the vision encoder to obtain image feature. In the language branch, a text description is projected as input textual tokens T in which are then processed by the text encoder to generate text feature. Through contrastive learning over massive image-text pairs, CLIP achieves good alignment between the two modalities.

More interestingly, such large-scale V-L models show the potential in visual classification on the downstream tasks. By converting each class label into a text template such as "a photo of a {class

What

does a [CLS] look like?

BERT

GPT-3

Vicuna

Text Encoder

Gate

Encoder

Student Encoder

Encoder

Decoder

Encoder

Decoder

Encoder

Decoder

Prompts

Vision Encoder

Output

Visual

Tokens

Gated

Visual

Tokens

DINO

MAE

ViTDet

SAM

BERT

Gated

Textual Tokens

𝑁𝑁

×

𝐶𝐶

𝑁𝑁

𝑐𝑐𝑐𝑐𝑐𝑐

×

𝐶𝐶

𝑁𝑁

𝑐𝑐𝑐𝑐𝑐𝑐

×

𝐶𝐶

𝑁𝑁

𝑐𝑐𝑐𝑐𝑠𝑠

×

𝐶𝐶

𝑁𝑁

×

𝐶𝐶

𝑁𝑁

𝑐𝑐𝑐𝑐s

×

𝐶𝐶

A [CLS] looks like…………

…………………………………………

…………………………………………

A [CLS] looks like…………

…………………………………………

…………………………………………

A photo of a [CLS]

𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫

𝑁𝑁

×

𝐶𝐶

Output

Textual

Tokens

𝑁𝑁

×

𝐶𝐶

𝑁𝑁

×

𝐶𝐶

𝑁𝑁

×

𝐶𝐶

𝑁𝑁

𝑐𝑐𝑐𝑐𝑠𝑠

×

𝐶𝐶

𝑁𝑁

𝑐𝑐𝑐𝑐𝑠𝑠

×

𝐶𝐶

Gate

𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫𝑫

(

𝑏𝑏

)

𝐿𝐿𝐶𝐶𝐶𝐶𝑇𝑇𝑉𝑉𝐶𝐶𝑇𝑇𝑚𝑚

𝑆𝑆𝑇𝑇𝑚𝑚𝐶𝐶𝑤𝑤

𝐶𝐶𝐶𝐶𝑉𝑉𝑉𝑉𝐶𝐶𝑏𝑏𝐶𝐶𝐶𝐶𝐶𝐶𝑤𝑤𝐶𝐶𝐶𝐶𝐶𝐶

(

𝐶𝐶

)

𝑉𝑉𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶

𝑆𝑆𝑇𝑇𝑚𝑚𝐶𝐶𝑤𝑤

𝐶𝐶𝐶𝐶𝑉𝑉𝑉𝑉𝐶𝐶𝑏𝑏𝐶𝐶𝐶𝐶𝐶𝐶𝑤𝑤𝐶𝐶𝐶𝐶𝐶𝐶

name}", these models can easily achieve zero-shot inference. To further enhance their generalization ability under the few-shot settings, prompt learning methods are proposed, which introduce a number of learnable prompts while freezing the pre-trained CLIP [87, 86, 39-41, 45]. Following previous works, we add a set of learnable textual prompts P T ∈ R N ctx × C , where N ctx is the number of learnable prompts, in the language branch and concatenate them with the textual tokens, which are then processed by the text encoder to obtain the prompted textual feature T ∈ R N cls × C for all N cls categories. Similarly, a set of learnable visual prompts P V are inserted in the image branch to generate the prompted visual feature V ∈ R N × C where N denotes the number of images:

Consequently, we compute the prediction score vectors S = { S c } for the image samples, where S c is the cosine similarity between the visual feature V and the textual feature T c of specific class c . As a result, we minimize the cross entropy loss between the score vectors and the ground truth labels:

to fine-tune the learnable prompts P T and P V for visual recognition. However, as mentioned in the introduction, it is difficult to achieve good generalization by adapting such a single CLIP model, especially when the domain shift of the target datasets is large. Hence, we propose to transfer diversified knowledge from heterogeneous agents in different modalities for better adaption of CLIP.

3.1 Vision Agent Collaboration (VAC)

One important component of CLIP is its vision branch in Figure 2. Hence, we consider to enhance this branch by transferring visual knowledge from various vision agents. To achieve this goal, we have to answer three critical questions. The first question is which models should be used for collaboration. As we know, CLIP mainly establishes image-level alignment between the two modalities, neglecting visual details in the pixel space. To fill this gap, we choose vision agents from two aspects. On one aspect, we choose vision models pre-trained with self-supervision such as MAE [33] and DINO [9]. Both agents focus on detailed image modeling via image masking [33] or patch self-distillation [9]. On the other aspect, we choose vision models built on dense supervision such as ViTDet [49] and SAM [43]. Both agents work on instance-level prediction with bounding boxes and masks. Detailed information on these models can be found in the supplementary.

The second question is how to extract visual knowledge by collaborating with these agents. As mentioned in the introduction, the latent feature is a common knowledge form among these heterogeneous models. Hence, given an input image, we extract the intermediate visual features

{ V A ( i ) ∈ R N × C } from the vision encoders of these agents. Moreover, the contribution of different agents may vary among different domains. To fully exploit these agents in a unified manner, we introduce a Mixture-of-Agents (MoA) gating mechanism to adaptively integrate { V A ( i ) } as the visual knowledge. Specifically, we concatenate all the agent features along the channel dimension C , and feed them into an MLP network to generate the gating weight W V . Next, we obtain the gated visual features V A by computing the weighted sum over { V A ( i ) } :

The final question is how to transfer the visual knowledge to enhance CLIP's vision encoder. We adopt feature distillation [36] where we compute the L1 loss between the prompted visual features V in Eq. 1 and the gated visual features of the vision agents:

Since the original CLIP is frozen, the above loss term allows us to fine-tune the learnable visual prompts P V in Eq. 1 and MLP gating network in Eq. 3, which enables us to adaptively transfer external visual knowledge to the visual prompts to empower generalization ability.

3.2 Language Agent Collaboration (LAC)

The other important component of CLIP is its language branch in Figure 2. Similar to the vision branch, we consider three critical questions to transfer the textual knowledge from various language agents. Recent studies [60, 42] have shown that it is coarse to use a simple template ( e.g. , "a photo of a {class name}") to describe a certain category. Hence, we first interact with the popular chatbots such as GPT-3 [7] and Vicuna [18] to enrich the class descriptions using queries like "What does a {class name} look like?". After obtaining the detailed descriptions from these chatbots, we use a text encoder ( e.g. , BERT [22]) to extract the text features { T A ( j ) ∈ R N cls × C } of all descriptions. To adaptively integrate { T A ( j ) } as the textual knowledge, we also utilize the MoA gating mechanism:

Finally, for each category, we perform feature distillation where we compute the L1 loss between the prompted textual feature in Eq. 1 and the gated textual feature from language agents in Eq. 5:

By fine-tuning the learnable textual prompts P T in Eq. 1 and MLP gating network in Eq. 5, we adaptively transfer the rich textual knowledge to enhance CLIP's textual representations.

3.3 Multi-modal Agent Collaboration (MAC)

Through vision and language agent collaboration, we can enhance the learnable visual and textual prompts respectively. Considering the key success in vision-language foundation models is credited to the multi-modal alignment, we investigate how to further align the visual and textual prompts with external multi-modal agents. First, there exist two types of multi-modal agents, including Text-toImage (T2I) generative models [64, 11] and Image-to-Text (I2T) captioning models [46, 12]. Since both types of agents involve conversion from one modality to the other, we believe they implicitly achieve vision-language alignment. Based on the above discussion, we choose our T2I agents built upon two mainstream structures including Stable Diffusion [64] in UNet style and PixArtα [11] in DiT style. Moreover, we choose our I2T agents specialized in two mainstream tasks including BLIP-2 [46] for general captioning and Shikra [12] for grounded dialogue.

Next, we consider what kind of knowledge can represent vision-language alignment. The most direct form may be the probability score vector that shows the prediction confidence over target classes. Hence, for each training image, we investigate such knowledge in these multi-modal agents.

In the T2I agents, cross attention effectively encodes relations between image and text [34, 84]. Hence, we leverage such prior to extract the score vector. Specifically, we use the template description of a class c as text. Then, we extract cross attention value M c k between the text and the k -th token of the input image from the pre-trained T2I agents. Consequently, we sum over all the tokens to

Vision	Encoder

Text	Encoder

UNet Style

DiT Style

Attention	Maps

Per	Class

Vision	Encoder

Q-Former

LLM

Vision	Encoder

MLP

LLM

@2B CDEFGH

B2@ CDEFGH

Learned	Score	Vectors

Gated	Score	Vectors

g×g

!$%

g×g

!$%

g×g

!$%

g×g

!$%

Gate

Block

Block

Down	Block

Up	Block

Stable Diffusion

PixArt-P

BLIP2

Shikara

];=D9b=9@<D9AC

];=D9b=9@<D9AC

g×a

g

!$%

×a

g

!$%

×a

g×a

CLS

'

CLS

(

CLS

)

CLS

'

CLS

(

CLS

)

CLS

'

CLS

(

CLS

)

CLS

'

CLS

(

CLS

)

g

!"#

×a

g

!"#

×a

g×a

g

!$&

×a

g×g

!$%

g×g

!$%

CLS

'

CLS

(

CLS

)

CLS

'

CLS

(

CLS

)

89:;9<<=;9>?

Attention	Maps

Per	Class

obtain the prediction score of the image w.r.t. class c : S c T 2 I = log( ∑ k exp( M c k )) , where we adopt LogSumExp (LSE) pooling [5] to provide more accurate matching scores.

In the I2T agents, there exists a projection module ( e.g. , Q-Former in BLIP-2 [46] and MLP in Shikra [12]) to adapt the visual features for the large language model. Hence, we extract the visual feature of an input image via the projection and the textual features of all the classes from the LLM. By computing the cosine similarity between them, we can obtain the prediction score S I 2 T of the image over all the classes. Next, we leverage the MoA gating mechanism to adaptively summarize the prediction score vectors from all the multi-modal agents M A = Concat ( { S T 2 I , S I 2 T } ) :

Finally, we explore how to transfer S A to enhance the learnable prompts in CLIP. Specifically, after being processed by the vision and text encoders of CLIP, the learnable visual and textual prompts P V and P T are transformed into Q V and Q T . Unlike P V and P T which are universal, Q V is relevant to the image samples and Q T is relevant to the target classes. Hence, we can compute the cosine similarity between Q V and Q T to obtain the learned score vectors S P ∈ R N × N cls of the input images over all the classes. We perform score distillation between S P and S A :

Through computing the KL divergence, we can leverage external multi-modal knowledge S A as a semantic guidance to align the learnable visual and textual prompts.

3.4 Multi-Source Knowledge Distillation

Finally, we combine all the distillation loss from multiple sources, achieving heterogeneous agent collaboration for knowledge transfer:

where λ 1 , λ 2 , λ 3 are hyperparameters. Since all the pre-trained models are frozen in the training phase, we only need to fine-tune the learnable vision and language prompts with negligible cost. More importantly, owing to the distillation strategy, all the agents can be unloaded and the modality-specific gates can be abandoned in the inference phase. We can simply use the enhanced CLIP just like the original one, which largely boosts deployment efficiency without a model ensemble.

4 Experiments

Datasets and Metrics. We evaluate our proposed method on 11 commonly used datasets covering a wide range of recognition tasks, including ImageNet [21], Caltech101 [29], OxfordPets [58], StanfordCars [44], Flowers102 [57], Food101 [6], FGVCAircraft [55], SUN397 [73], UCF101 [67], DTD [19] and EuroSAT [35]. We explore two typical low-shot scenarios to evaluate the performance. (i) Base-to-novel generalization: The datasets are equally split into base and novel classes. The model is trained on base classes and evaluated on the test set of both classes. We report the base and novel class accuracy and the harmonic mean (HM) of the results. (ii) Few-shot classification: We assess the accuracy trained with 1/2/4/8/16 shot(s) per class to examine the model's learning capacity.

Implementation Details. We adopt CLIP ViT/B-16 as our backbone and conduct all experiments using 3 different seeds to obtain an averaged result, following previous works [40, 41, 50]. Our method ensembles knowledge from heterogeneous agents, including pre-trained vision models, LLMs, T2I generative models, and I2T captioning models. Detailed information on the agents and training settings are shown in Appendix A.

0

1

2

4

8

16

Number of training samples per class

45

50

55

60

65

70

75

80

Accuracy (%)

Average over 11 datasets

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

0

1

2

4

8

16

Number of labeled training examples per class

35

40

45

50

55

60

65

70

75

Accuracy (%)

ImageNet

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

0

1

2

4

8

16

Number of labeled training examples per class

80.0

82.5

85.0

87.5

90.0

92.5

95.0

Accuracy (%)

Caltech101

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

0

1

2

4

8

16

Number of labeled training examples per class

50

60

70

80

90

Accuracy (%)

OxfordPets

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

0

1

2

4

8

16

Number of labeled training examples per class

40

50

60

70

80

Accuracy (%)

StanfordCars

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

0

1

2

4

8

16

Number of labeled training examples per class

70

75

80

85

90

95

Accuracy (%)

Flowers102

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

0

1

2

4

8

16

Number of labeled training examples per class

50

60

70

80

Accuracy (%)

Food101

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

0

1

2

4

8

16

Number of labeled training examples per class

20

25

30

35

40

45

50

Accuracy (%)

FGVCAircraft

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

0

1

2

4

8

16

Number of labeled training examples per class

40

45

50

55

60

65

70

75

Accuracy (%)

SUN397

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

0

1

2

4

8

16

Number of labeled training examples per class

35

40

45

50

55

60

65

70

75

Accuracy (%)

DTD

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

0

1

2

4

8

16

Number of labeled training examples per class

50

60

70

80

90

Accuracy (%)

EuroSAT

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

0

1

2

4

8

16

Number of labeled training examples per class

55

60

65

70

75

80

85

Accuracy (%)

UCF101

Zero-shot CLIP

Linear probe CLIP

CoOP

CaFo

TransAgent

4.1 Comparison with State-of-the-Art

In Table 1, we compare the base-to-novel generalization performance of our proposed TransAgent with state-of-the-art prompt learning methods, including CoOp [87], CoCoOp [86], MaPLe [40], RPO [45] and PromptSRC [41]. All approaches use the same CLIP ViT-B/16 during the evaluation stage. Our method demonstrates superior performance across 11 datasets, surpassing all competing methods in terms of base and novel accuracy, as well as the harmonic mean, particularly excelling on the EuroSAT dataset. In Figure 4, we present the few-shot classification performance of TransAgent and previous methods. Our method consistently outperforms the counterparts and demonstrates growing learning capability when the training samples increase. To be mentioned, TransAgent outperforms CaFo [81] with much lower deployment costs (see Table 13). Detailed comparisons on cross-dataset evaluation and domain generalization are provided in Appendix B.

4.2 Ablative Analysis

In this section, we present the ablative analysis of our agent collaboration designs for each modality. We adopt IVLP [40, 41] as our baseline model.

MAE

DINO

SAM

ViTDet

Caltech

Pets

Cars

Flowers

Food

Aircraft

DTD

EuroSAT

UCF101

SUN397

ImageNet

Datasets

GPT-3

Vicuna

Agents

Caltech

Pets

Cars

Flowers

Food

Aircraft

DTD

EuroSAT

UCF101

SUN397

ImageNet

SD

Pixart

BLIP2

Caltech

Pets

Cars

Flowers

Food

Aircraft

DTD

EuroSAT

UCF101

SUN397

ImageNet

Effectiveness of individual agent. In Table 2 - Table 4, we show the results of introducing an individual agent as a teacher to supervise the baseline model in the beginning rows. Nearly all vision agents contribute to the improvement in accuracy, except for DINO which is behind. However, we observe that DINO performs well on most datasets, except on EuroSAT (with a 5.60% decline). Nonetheless, we still keep DINO as one of the vision agents. While for the language and the multi-modal agents, they all boost the performance of the baseline model respectively.

Distillation strategy. In the middle rows of Table 2 - Table 4, we ablate the alignment strategy for agent collaboration of each modality. (i) For V AC, we choose to adopt feature distillation with either the average-pooled feature at the last layer or all features at all layers. As the results suggest, layer-wise distillation of all features performs better. (ii) For LAC, two special tokens from the output of the text encoder are considered to compute Equation 6. These tokens are inserted into the text sequence to compose a complete prompt. [ EOS ] performs better owing to the causal attention module in Transformer blocks, so that it aggregates more information. (iii) For MAC, either the prompted logits of CLIP or the learned scores (Eq. 8) are chosen to align with the semantic logits from multi-modal agents. Since the prompted logits need to be aligned with ground-truth labels as well (Eq. 2), there might be confusion to align it with the external knowledge simultaneously. Experimental results show that using learned scores to align with semantic logits yields better results.

Fusion design. In the last rows of Table 2 - Table 4, we ablate the fusion design for each modality. "Average" refers to simply calculating the average of all output features from the agents along the channel dimension; "Add" refers to calculating the distillation loss separately for each agent; "Gating" denotes our proposed MoA gating mechanism. As can be seen, gating fusion achieves the best results for all collaboration designs, since it adaptively selects the useful information from the agents.

Effectiveness of collaboration module(s). As presented in Table 5, the first row indicates the baseline model. Each individual module is beneficial for improving the generalization ability of the foundation models, where the last column shows the relative increase. Results in Table 6 demonstrate that the combinations of collaboration modules further boost the results.

4.3 Visualization

We calculate the averaged gating weights of each agent with 16-shot training samples from all classes of a certain dataset and present the results with heat maps. As shown in Figure 5, different agents do not contribute equally towards certain target datasets, which also verifies the superiority of our gating fusion design. For vision agents, DINO experts in recognizing general objects while lags behind on some fine-grained datasets ( e.g. , EuroSAT), whereas the others that focus more on details perform better. Language agents provide domain-specific linguistic knowledge accordingly. I2T agents consistently provide their knowledge thanks to the grounding nature, while T2I agents demonstrate better capability in fine-grained scenarios.

5 Conclusion

We propose TransAgent, a unified framework to transfer vision-language foundation models through heterogeneous agent collaboration. By adaptively integrating the external knowledge of agents from different modalities viac the MoA gating mechanism, TransAgent achieves state-of-the-art performance on 11 datasets under the low-shot scenarios.

Limitations. Although TransAgent collaborates with heterogeneous agents in a unified manner, transferring the external knowledge through distillation may harm the original CLIP's representations even using prompt learning methods because the domain knowledge from agents are diversified, and irrelevant information may also be introduced. Moreover, one of the key characteristics of these agents is large-scale pre-training, few-shot scenarios may not meet their data-hungry nature. So our future work would be integrating the knowledge with directed focus and further unleashing the potential of the agents even when the domain-specific knowledge ( e.g. , labels) is absent.

Acknowledgments and Disclosure of Funding

This work was supported in part by the National Key R&D Program of China (NO.2022ZD0160505), the National Natural Science Foundation of China under Grant (62272450)

References

R. Adriana, B. Nicolas, K. S. Ebrahimi, C. Antoine, G. Carlo, and B. Yoshua. Fitnets: Hints for thin deep nets. In International conference on learning representations , 2015.

J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:23716-23736, 2022.

H. Bao, L. Dong, S. Piao, and F. Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254 , 2021.

Y. Bengio, A. Courville, and P. Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence , 35(8):1798-1828, 2013.

P. Blanchard, D. J. Higham, and N. J. Higham. Accurately computing the log-sum-exp and softmax functions. IMA journal of numerical analysis , 41(4):2311-2330, 2021.

L. Bossard, M. Guillaumin, and L. V. Gool. Food-101-mining discriminative components with random forests. In European conference on computer vision , pages 446-461. Springer, 2014.

T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. Advances in neural information processing systems , 33: 1877-1901, 2020.

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020.

M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, October 2021.

Y. Chebotar and A. Waters. Distilling knowledge from ensembles of neural networks for speech recognition. In Interspeech , pages 3439-3443, 2016.

J. Chen, Y. Jincheng, G. Chongjian, L. Yao, E. Xie, Z. Wang, J. Kwok, P. Luo, H. Lu, and Z. Li. Pixartα : Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International conference on learning representations , 2024.

K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao. Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195 , 2023.

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the international conference on machine learning , pages 1597-1607. PMLR, 2020.

X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794 , 2022.

Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, Z. Muyan, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238 , 2023.

B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems , 34:17864-17875, 2021.

B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1290-1299, 2022.

W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/ .

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3606-3613, June 2014.

W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems , 36, 2024.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 248-255. IEEE, 2009.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

P. Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International conference on learning representations , 2021.

D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378 , 2023.

S. Du, S. You, X. Li, J. Wu, F. Wang, C. Qian, and C. Zhang. Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. Advances in neural information processing systems , 33:12345-12355, 2020.

Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103.10360 , 2021.

X. Fan, T. Ji, C. Jiang, S. Li, S. Jin, S. Song, J. Wang, B. Hong, L. Chen, G. Zheng, et al. Mousi: Poly-visual-expert vision-language models. arXiv preprint arXiv:2401.17221 , 2024.

L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE/CVF conference on computer vision and pattern recognition workshop , pages 178-178. IEEE, 2004.

T. Fukuda, M. Suzuki, G. Kurata, S. Thomas, J. Cui, and B. Ramabhadran. Efficient knowledge distillation from an ensemble of teachers. In Interspeech , pages 3697-3701, 2017.

P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better vision-language models with feature adapters. International journal of computer vision , 132(2): 581-595, 2024.

K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9729-9738, 2020.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16000-16009, June 2022.

X. He, W. Feng, T.-J. Fu, V. Jampani, A. Akula, P. Narayana, S. Basu, W. Y. Wang, and X. E. Wang. Discriminative diffusion models as few-shot vision and language learners. arXiv preprint arXiv:2305.10722 , 2023.

P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE journal of selected topics in applied earth observations and remote sensing , 12(7):2217-2226, 2019.

G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the international conference on machine learning , volume 139, pages 4904-4916. PMLR, 2021.

M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim. Visual prompt tuning. In European conference on computer vision , pages 709-727, 2022.

M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 19113-19122, June 2023.

M. U. Khattak, S. T. Wasim, M. Naseer, S. Khan, M.-H. Yang, and F. S. Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF international conference on computer vision , pages 15190-15200, October 2023.

M. U. Khattak, M. F. Naeem, M. Naseer, L. Van Gool, and F. Tombari. Learning to prompt with text only supervision for vision-language models. arXiv preprint arXiv:2401.02418 , 2024.

A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollar, and R. Girshick. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4015-4026, October 2023.

J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE/CVF international conference on computer vision workshop , pages 554-561, December 2013.

D. Lee, S. Song, J. Suh, J. Choi, S. Lee, and H. J. Kim. Read-only prompt optimization for vision-language few-shot learning. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1401-1411, October 2023.

J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the international conference on machine learning , volume 202, pages 19730-19742. PMLR, 23-29 Jul 2023.

K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo, Y. Wang, L. Wang, and Y. Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 , 2023.

X. Li, D. Lian, Z. Lu, J. Bai, Z. Chen, and X. Wang. Graphadapter: Tuning vision-language models with dual knowledge graph. Advances in neural information processing systems , 36, 2024.

Y. Li, H. Mao, R. Girshick, and K. He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision , pages 280-296, 2022.

Z. Li, X. Li, X. Fu, X. Zhang, W. Wang, and J. Yang. Promptkd: Unsupervised prompt distillation for vision-language models. In Proceedings of the IEEE/CVF international conference on computer vision , 2024.

H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems , 36, 2024.

M. Liu, S. Roy, W. Li, Z. Zhong, N. Sebe, and E. Ricci. Democratizing fine-grained visual recognition with large language models. In International conference on learning representations , 2023.

Z. Lu, J. Bai, X. Li, Z. Xiao, and X. Wang. Beyond sole strength: Customized ensembles for generalized vision-language models. arXiv preprint arXiv:2311.17091 , 2023.

Z. Ma, J. Dong, S. Ji, Z. Liu, X. Zhang, Z. Wang, S. He, F. Qian, X. Zhang, and L. Yang. Let all be whitened: Multi-teacher distillation for efficient visual retrieval. In Proceedings of the AAAI conference on artificial intelligence , volume 38, pages 4126-4135, 2024.

S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.5151 , 2013.

A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Indian conference on computer vision, graphics & image processing , pages 722-729. IEEE, 2008.

O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3498-3505. IEEE, 2012.

Z. Peng, L. Dong, H. Bao, Q. Ye, and F. Wei. Beit v2: Masked image modeling with vectorquantized visual tokenizers. arxiv 2022. arXiv preprint arXiv:2208.06366 , 2022.

S. Pratt, I. Covert, R. Liu, and A. Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF international conference on computer vision , pages 15691-15701, October 2023.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the international conference on machine learning , volume 139 of PMLR , pages 8748-8763. PMLR, 18-24 Jul 2021.

A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 1(2):3, 2022.

M. Ranzinger, G. Heinrich, J. Kautz, and P. Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 12490-12500, 2024.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, June 2022.

S. Roy and A. Etemad. Consistency-guided prompt learning for vision-language models. In International conference on learning representations , 2024.

C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems , 35: 36479-36494, 2022.

K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Y.-L. Sung, J. Cho, and M. Bansal. Vl-adapter: Parameter-efficient transfer learning for visionand-language tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5227-5237, 2022.

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

X. Wang, Y. Zhang, O. Zohar, and S. Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. arXiv preprint arXiv:2403.10517 , 2024.

C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 , 2023.

J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3485-3492. IEEE, 2010.

P. Ye, C. Huang, M. Shen, T. Chen, Y. Huang, Y. Zhang, and W. Ouyang. Merging vision transformers from different tasks and domains. arXiv preprint arXiv:2312.16240 , 2023.

T. Yu, Z. Lu, X. Jin, Z. Chen, and X. Wang. Task residual for tuning vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10899-10909, 2023.

F. Yuan, L. Shou, J. Pei, W. Lin, M. Gong, Y. Fu, and D. Jiang. Reinforced multi-teacher selection for knowledge distillation. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 14284-14291, 2021.

H. Zhang, D. Chen, and C. Wang. Confidence-aware multi-teacher knowledge distillation. In IEEE international conference on acoustics, speech and signal processing , pages 4498-4502. IEEE, 2022.

H. Zhang, D. Chen, and C. Wang. Adaptive multi-teacher knowledge distillation with metalearning. In IEEE international conference on multimedia and expo , pages 1943-1948. IEEE, 2023.

H. Zhang, X. Li, and L. Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858 , 2023.

R. Zhang, W. Zhang, R. Fang, P. Gao, K. Li, J. Dai, Y. Qiao, and H. Li. Tip-adapter: Trainingfree adaption of clip for few-shot classification. In European conference on computer vision , pages 493-510. Springer, 2022.

R. Zhang, X. Hu, B. Li, S. Huang, H. Deng, Y. Qiao, P. Gao, and H. Li. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15211-15222, 2023.

S. Zhang, D. Huang, J. Deng, S. Tang, W. Ouyang, T. He, and Y. Zhang. Agent3d-zero: An agent for zero-shot 3d understanding. arXiv preprint arXiv:2403.11835 , 2024.

S. Zhao, X. Wang, and X. Wei. Mitigating accuracy-robustness trade-off via balanced multiteacher adversarial distillation. IEEE transactions on pattern analysis and machine intelligence , (01):1-14, 2024.

W. Zhao, Y. Rao, Z. Liu, B. Liu, J. Zhou, and J. Lu. Unleashing text-to-image diffusion models for visual perception. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5729-5739, October 2023.

Z. Zheng, J. Wei, X. Hu, H. Zhu, and R. Nevatia. Large language models are good prompt learners for low-shot image classification. In Proceedings of the IEEE/CVF international conference on computer vision , 2024.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 16816-16825, June 2022.

K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International journal of computer vision , 130:2337-2348, July 2022.

D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 , 2023.

X. Zhu, R. Zhang, B. He, A. Zhou, D. Wang, B. Zhao, and P. Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. In Proceedings of the IEEE/CVF international conference on computer vision , pages 2605-2615, 2023.

A Additional Implementation Details

Training Details. The number of learnable vision and language prompt tokens are both set to 4, and the prompt depth is set to 9 for base-to-novel generalization and few-shot classification, and 3 for cross-dataset and domain generalization. The learnable text prompts of the first layer are initialized with the word embeddings of "a photo of a", while the other learnable prompts are randomly initialized with a normal distribution. For vision agent collaboration, we adopt online distillation where the models are loaded during the training process. For language and multi-modal agent collaboration, due to the huge model parameters, we extract their knowledge offline before our training launches. For few-shot classification, we train the models for 50 epochs under different low-shot settings (ranging from 1 to 16). We re-implement the compared methods in Figure 4 with Vision Transformer [24] backbone for fair comparison. For the other benchmarks, we observe that PromptKD [50] adopts a transductive setting which uses all training samples in an unsupervised manner and obtains better results. However, such a setting is beyond our scope. In our work, all models are trained for 20 epochs using 16-shot samples with a fixed batch size of 4 and a learning rate of 0.0025 with SGD as the optimizer. We set λ 1 = 1 , λ 2 = 25 and λ 3 = 1 in Eq. 9 after extensive hyperparameter search to balance the total loss. The memory and training time needed under different settings on each dataset are provided in Table 8 using a batch size of 4. At least 48GB is required to extract knowledge from language and multi-modal agents. All experiments are conducted on a single Nvidia A6000 GPU.

Information of agents. Table 7 lists all the agents we have collaborated in our TransAgent framework. (i) Vision agents: DINO provides robust representations through image contrastive (IC) learning while MAE presents powerful capability via mask image modeling (MIM). SAM and ViTDet are specialists in image segmentation (IS) and object detection (OD) respectively. (ii) Language agents: Both GPT-3 and Vicuna are excellent chatbots that faithfully follow human instructions and respond with convincing answers by text generation (TG). We use BERT, which is just CLIP's text encoder pre-trained using mask language modeling (MLM), to process the generated captions from chatbots before transferring these linguistic knowledge. (iii) Multi-modal agents: Stable Diffusion and Pixartα share different model architectures, but both T2I agents demonstrate astonishing image generation (IG) capability. The I2T agents are pre-trained with multiple tasks including image-text contrastive (ITC), image-text matching (ITM), and image-grounded text generation (ITG) over large-scale datasets. These agents demonstrate outstanding multi-modal understanding ability.

B Additional Experiments

Cross-Dataset Evaluation . We compare the cross-dataset performance in Table 9. On the source dataset, TransAgent achieves better performance than previous few-shot prompt learning methods under the same training settings. We credit the fine-fitted result to the ImageNet pre-trained vision experts, which transfer their knowledge of the source dataset during the training process. More surprisingly, our method does not seem to overfit on the source dataset. It still presents competitive results against MaPLe, which is the SOTA few-shot method on cross-dataset generalization, leading to an overall improvement over the previous methods.

Domain Generalization . Table 10 presents the results on domain generalization. TransAgent outperforms previous few-shot prompt learning methods with the highest average accuracy. This suggests that our method improves the robustness of VLMs against out-of-distribution data.

C Additional Ablative Analysis

MACloss type. We scrutinize the loss function used to compute the distillation loss in Eq. 8. As shown in Table 11, using L1 loss or MSE loss underperforms using KL loss by a large margin. Such channel-wise loss is incompatible with the form of the score vectors, which are per-class probability distributions, and using such loss makes it hard to converge. On the contrary, KL loss provides smoother training due to the soft matching between the probability distributions ( i.e. , the score vectors).

Pooling type. In Table 12, we ablate the pooling strategy to aggregate the cross attention value from T2I agents to obtain the score vectors. The pooling operations are performed on the spatial dimension of the cross attention maps, which results in the semantic logits of a certain image over all the candidate categories. As is shown in Table 12, average and max pooling lag behind since they introduce information loss during the operations. However, LogSumExp (LSE) pooling yields the best performance as it is more robust and provides more accurate matching scores [34].

1

2

4

8

16

Shots

55

60

65

70

75

80

85

90

95

EuroSAT

CoOp

TransAgent

1

2

4

8

16

Shots

15

20

25

30

35

40

45

50

FGVCAircraft

CoOp

TransAgent

1

2

4

8

16

Shots

83.5

84.0

84.5

85.0

85.5

86.0

86.5

87.0

Food101

CoOp

TransAgent

1

2

4

8

16

Shots

92

93

94

95

96

Caltech101

CoOp

TransAgent

1

2

4

8

16

Shots

90

91

92

93

94

OxfordPets

CoOp

TransAgent

1

2

4

8

16

Shots

70.0

72.5

75.0

77.5

80.0

82.5

85.0

UCF101

CoOp

TransAgent

1

2

4

8

16

Shots

67.5

70.0

72.5

75.0

77.5

80.0

82.5

85.0

StandfordCars

CoOp

TransAgent

1

2

4

8

16

Shots

80.0

82.5

85.0

87.5

90.0

92.5

95.0

97.5

Flowers102

CoOp

TransAgent

1

2

4

8

16

Shots

50

55

60

65

70

75

DTD

CoOp

TransAgent

Deployment efficiency. Table 13 presents the comparison of inference time and performance evaluated on the test set of ImageNet [21]. All models adopt Vision Transformer [24] backbone. Our method demonstrates superior deployment efficiency compared to CaFo [81] which adopts a cumbersome model ensemble. To be mentioned, the learnable prompts in the training phase are frozen during the inference stage, which carries the knowledge of heterogeneous agents to enhance CLIP's generalization ability. The inference time of our TransAgent is a little behind the zero-shot CLIP, but it still achieves a good accuracy-efficiency tradeoff.

D Additional Visualization

To verify the robustness of TransAgent, we run few-shot classification experiments over 3 seeds and visualize the variance on 9 datasets in Figure 6 (ImageNet and SUN397 are excluded since the variance is small for both methods.). We can observe that TransAgent consistently outperforms CoOp in robustness and excels in most low-shot scenarios.

E Examples of Generated Descriptions from Chatbots

We interact with the language agents ( i.e. , chatbots) as described in Section 3.2 to obtain rich descriptions of the categories in the target domain, following [42]. To further unleash the linguistic potential of these language agents, we gather their knowledge adaptively via the MoA gating mechanism. In fact, these language agents give descriptions in different styles with their respective unique understanding. Below we show some examples of the descriptions given by different chatbots for certain classes to demonstrate their characteristics.

Class: Pomeranian (OxfordPets).

GPT-3: "A pet Pomeranian typically has a thick coat of fur that can be either straight or curly."

Vicuna: "The image shows a small dog with a fluffy white coat. The dog has a round face with big brown eyes and a black nose. Its ears are small and pointy, and its tail is curled up over its back."

Class: Industrial Buildings (EuroSAT).

GPT-3: "A satellite photo of industrial buildings might look like a patchwork of utilitarian structures."

Vicuna: "The image shows a large industrial building with a flat roof and several windows on the front facade. The building is made of concrete and has a gray color. The roof is also gray and appears to be made of a similar material."

Class: Club Sandwich (Food101).

GPT-3: "A club sandwich is a sandwich made with ham, turkey, bacon, cheese, and tomato, with lettuce in between slices of bread."

Vicuna: "The image shows a golden brown club sandwich, with a thick slice of bread on the bottom, and two thinner slices on top. The bread is toasted and has a crispy texture."

Class: Office Building (SUN397).

GPT-3: "An office building is usually a tall building with many floors."

Vicuna: "An office building is a structure that is designed for business-related activities. It typically has multiple floors and a variety of rooms and spaces to accommodate different functions, such as open-plan offices, meeting rooms, and private offices."

Class: 747-200 (FGVCAircraft).

GPT-3: "The 747-200 is a jumbo jet produced by Boeing."

Vicuna: "The Boeing 747-200 is a large, wide-body commercial airliner that was first introduced in the 1970s."

Class: 2012 Mercedes-Benz E-Class Sedan (StandfordCars).

GPT-3: "The Mercedes-Benz E-Class Sedan 2012 can be identified by its distinctive grille, sleek headlights, and classic Mercedes-Benz logo."

Vicuna: "The 2012 Mercedes-Benz E-Class Sedan is a luxury car that is known for its sleek and sophisticated appearance."

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract synoptically demonstrates our proposed method, which aims to transfer VL foundation models with heterogeneous agent collaboration. The introduction section further claims the background and our contributions.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the potential limitations of our proposed method in Section 5. Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The results of the paper are mainly experimental.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The implementation details are provided in the main paper (Section 4) as well as in the supplementary (Appendix A).

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The training and evaluation codes are released at https://github.com/ markywg/transagent , with detailed instructions on how to download data and reproduce the experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The training details and hyperparameters setting are listed in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Since all experiments are run over 3 different seeds, we provide the variance figures in the supplementary (Appendix D) to display the robustness of our proposed method.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: As stated in Appendix A, all experiments are conducted using a Nvidia A6000 GPU. The memory and training time required on each dataset are presented in Table 8.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conduct the research project following NeurIPS Code of Ethics faithfully. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The work does not seem to raise any (negative) societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our method does not involve data or models that might lead to intended misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have made acknowledgements to the assets used in our work.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We heve prepared the documentation of our codes for future reproduction, which is released alongside the codes at https://github.com/markywg/transagent .

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the TransAgent framework, which unifies and transports knowledge from isolated agents to guide CLIP in generalizing through multi-source knowledge distillation. This framework allows flexible collaboration with 11 heterogeneous agents to enhance vision-language foundation models. Importantly, this approach incurs no additional cost during the inference phase.\n\nSTRENGTHS:\n**[Interesting idea]** Leveraging heterogeneous agent collaboration for generalized vision language models is intriguing.\n\n**[Good presentation]** The paper is well-written and organized, which is easy to follow.\n\n**[Well-illustrated figures]** The figures shown in this paper are clear enough to tell the workflow of the method.\n\nWEAKNESSES:\n**[Overclaimed statements]** The paper mentions that “it is the first unified transfer framework for generalizing vision-language foundation models with heterogeneous agent collaboration.” However, CaFo [67] had done similar things by adopting multiple heterogeneous agents. This statement may need a revision to differentiate from CaFo.\n\n**[Some SOTA methods are not compared]** Some works to be compared or discussed in related works are listed as follow: “PromptKD: Unsupervised Prompt Distillation for Vision-Language Models, CVPR2024”, “DePT: Decoupled Prompt Tuning, CVPR2024”, “Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models, ICML2024”, “Consistency-guided Prompt Learning for Vision-Language Models, ICLR2024”, “GraphAdapter: Tuning Vision-Language Models with Dual Knowledge Graph, NeurIPS2023” and “Task Residual for Tuning Vision-Language Models, CVPR2023”.\n\n**[Need further explorations]** The knowledge distillation technique used in this work is too straightforward. It would be nice to see an advanced distillation with heterogeneous agents.\n\n**[Inadequate experiments]** There are some experiments to be done for verifying the effectiveness of the proposed method. (i) Training time comparisons with other methods and more comparisons for inference time in Table 10; (ii) It would be nice to see the performance by increasing each agent gradually. This can help better understand which agent is more effective and which one is useless. (iii) Ablation study conducted using VAC, LAC, or MAC individually, as well as in combinations of any two.\n\n**[Disorganized reference format]** Please reformat the references, e.g., “Conference on Computer Vision and Pattern Recognition”, “Proceedings of the IEEE/CVF conference on computer vision and pattern recognition” and “Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)”.\n\nQUESTIONS:\nPlease refer to the weakness section.\n\nLIMITATIONS:\nYes, they have provided the limitation section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces TransAgent, a novel framework designed to enhance vision-language foundation models like CLIP through the integration of knowledge from diverse, pre-trained expert models. These experts, which include vision, language, and multi-modal models, possess rich knowledge acquired from different modalities, tasks, networks, and datasets. TransAgent addresses the challenge of generalizing these models to new domains, especially under low-shot conditions, by proposing a unified transfer framework that leverages heterogeneous agent collaboration. The framework employs a mixture-of-agents gating mechanism to adaptively integrate external knowledge and uses multi-source distillation to transfer this knowledge into CLIP, enhancing its generalization ability without additional inference cost. Experiments demonstrate TransAgent's state-of-the-art performance on 11 visual recognition datasets, significantly outperforming methods like CoOp under low-shot settings and showing remarkable results on EuroSAT with large domain shifts. The paper's contributions lie in its innovative approach to knowledge transfer, the flexibility of its framework, and its significant improvements in model generalizability and efficiency.\n\nSTRENGTHS:\nThe paper presents a highly original approach to enhancing vision-language foundation models by proposing the TransAgent framework. This framework innovatively integrates knowledge from a diverse set of pre-trained expert models into a unified system. \n\nThe use of a mixture-of-agents gating mechanism and multi-source distillation is a unique contribution that sets this work apart from existing methods.\n\nThe authors have conducted extensive experiments on 11 visual recognition datasets, demonstrating the effectiveness of TransAgent under various low-shot scenarios. The comparison with state-of-the-art methods like CoOp and the detailed ablation studies provide a solid understanding of the framework's capabilities and the impact of different components. \n\nThe paper is well-structured, with a clear abstract and introduction that succinctly summarize the contributions and scope of the work.\n\nWEAKNESSES:\nAfter changing the expert agent selection, the model needs to be retrained, and this structure is not plug-and-play.\n\nThe use of integrated features from multiple teacher models as supervision for distillation is widely used in other fields, and the authors have transferred this approach to the domain of this paper, which lacks sufficient novelty\n\nQUESTIONS:\nCould the authors provide insights into how the TransAgent framework generalizes to domains outside of the tested benchmarks, especially those with significantly different characteristics?\n\nThe paper mentions the potential introduction of irrelevant knowledge during distillation. What strategies are in place or could be considered to mitigate this issue?\n\nThe role of learnable prompts seems pivotal. How were the prompts engineered, and what impact did they have on the model's performance?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the challenge of vision-language foundation models (e.g., CLIP) struggling to generalize to diverse target domain data in downstream tasks. It highlights the potential of using expert models, which are pre-trained on various modalities, tasks, networks, and datasets, to improve generalization. The proposed TransAgent framework integrates the knowledge of these isolated expert models through a unified approach, enhancing CLIP's performance via multi-source knowledge distillation. TransAgent collaborates with 11 heterogeneous agents to empower vision-language models without adding inference phase costs. The framework achieves good performance on some visual recognition datasets.\n\nSTRENGTHS:\n1. The problem is of practical importance.\n2. The experiments are sufficient.\n3. The gains of knowledge distillation seem strong.\n\nWEAKNESSES:\n1. In my opinion, the proposed method leverages multiple pretty strong 'experts' for knowledge distillation, e.g., SAM, MAE, DINO, ViTDet, GPT, and Vicuna. However, almost all the baselines to be compared do not rely on external models. Hence, I think the majority of comparisons (e.g., Table 1 and Figure 4) may be unfair. The proper baselines should be other distillation methods (using the same external models as this paper), which the proposed method is actually orthogonal to most of the current baselines. \n\n2. On top of 1, I think that if some significant technical contributions regarding knowledge distillation are not proposed, simply 'applying knowledge distillation methods to CLIP' may not be an acceptable novel contribution for me.\n\n3. Given that multiple pretty strong 'experts' (e.g., SAM, MAE, DINO, ViTDet, GPT, and Vicuna) for knowledge distillation have been employed, the current gains of performance seem limited.\n\nQUESTIONS:\nPlease refer to 'Weaknesses'.\n\nLIMITATIONS:\nThe authors have addressed the limitations and potential negative societal impacts of their work.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-18"}

comments: SUMMARY:
This paper aims to handle heterogeneous foundation model combination across different pretrained backbones. Instead of using vanilla ensemble, it proposes to use a distillation process to transfer knowledges from different agents. More specifically, it uses a learnable gate module to integrate different knowledge sources. Extensive experiments show the propose method outperforms other related models.

STRENGTHS:
1. Efficiently combining several pretrained foundation models is a valuable research direction.
2. Overall, the writing is clear to read and follow, such as motivation, methodology, and empirical results.
3. Compared with baseline, the proposed model does not need to involve extra inference cost, benefitted by a distillation process.
4. Empirical results show the model superiority.

WEAKNESSES:
1. Overall, I think this is more like a jointly distillation technique, instead of so-called agent collaboration, which may lead to certain misunderstanding.
2. Some figures are a little confused to understand. A clean and informative figures to show the whole working pipeline will be helpful.
3. Even if the empirical results are good, the proposed method still lacks of research novelty.

QUESTIONS:
See the weaknesses section above.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-20

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a TransAgent framework, which guides CLIP to generalize with multi-source knowledge distillation. The framework contains three kinds of collaboration, including vision models, language models and muti-modal models. A Mixture-of-Agents (MoA) gating mechanism is proposed to adaptively integrate the knowledge. SOTA is achieved on 11 datasets under the few-shot scenarios.\n\nSTRENGTHS:\nThis method guarantees the deployment efficiency, as knowledge from heterogeneous agents are distillated and injected into CLIP.\nThis framework can be extended flexibly. Any expert can be introduced as a teacher model. \nThe paper is easy to follow. Figures and experiments are clearly presented.\n\nWEAKNESSES:\nThe method applies multi-teacher distillation in clip with prompts, and popular large models are exploited as teachers. Thus, the novelty is limited. \nGiven that the framework comprises three components — VAC, LAC, and MAC—the reviewer insists that the  results of every module (only-VAC, only-LAC, only-MAC) will be more persuasive to demonstrate their respective effectiveness (i.e. only-VAC's, only-LAC's, only-MAC's).\nThe clip absorbs various knowledge from 11 excellent agents. In addition to few-shot learning on 11 downstream datasets, experiments on general dataset are also encouraged, like WebQA and CIRR.\nIn the related works, research on muti-teacher distillation should be included. Baselines in the experiments should be introduced in the related work.\nTypos in line 259: knowledege -> knowledge\n\nQUESTIONS:\n11 agents contain different domain knowledges. Irrelevant information or inconsistent information may be induced. Is there any challenge in the training and how to deal with challenges?\nHow to prevent overfitting when distilled from 11 pretrained models? \nThe λ_2 is far greater than other λ values, does it mean that the loss of LAC is more important than other losses?\n\nLIMITATIONS:\nThe authors have listed the limitation in the paper. It is encouraged to provide more analysis and solutions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-26"}