TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration
Authors: Yiwei Guo
Shaobin Zhuang
Kunchang Li
Yu Qiao
Yali Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Vision-Language Models
Few-shot Transfer Learning
Heterogeneous Agent Collaboration
Primary area: machine_vision

Abstract: Vision-language foundation models (such as CLIP) have recently shown their power in transfer learning, owing to large-scale image-text pre-training. However, target domain data in the downstream tasks can be highly different from the pre-training phase, which makes it hard for such a single model to generalize well. Alternatively, there exists a wide range of expert models that contain diversified vision and/or language knowledge pre-trained on different modalities, tasks, networks, and datasets. Unfortunately, these models are "isolated agents" with heterogeneous structures, and how to integrate their knowledge for generalizing CLIP-like models has not been fully explored. To bridge this gap, we propose a general and concise TransAgent framework, which transports the knowledge of the isolated agents in a unified manner, and effectively guides CLIP to generalize with multi-source knowledge distillation. With such a distinct framework, we flexibly collaborate with 11 heterogeneous agents to empower vision-language foundation models, without further cost in the inference phase. Finally, our TransAgent achieves state-of-the-art performance on 11 visual recognition datasets. Under the same low-shot setting, it outperforms the popular CoOp with around 10\% on average, and 20\% on EuroSAT which contains large domain shifts.

gent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration
Yiwei Guo!.2 Shaobin Zhuang 334 Kunchang Li Yu Qiao
Yali Wang
Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences
$Shanghai Al Laboratory Shanghai Jiao Tong University
Abstract
Vision-language foundation models (such as CLIP) have recently shown their power in transfer learning, owing to large-scale image-text pre-training. However, target domain data in the downstream tasks can be highly different from the pre-training phase, which makes it hard for such a single model generalize well Alternatively, there exists a wide range of expert models that contain diversified vision and/or language knowledge pre-trained on different modalities, tasks, networks, and datasets. Unfortunately. these models are "isolated agents" with heterogeneous structures, and how integrate their knowledge for generalizing CLIP-like models has not been fully explored. To bridge this gap, we propose a general and concise TransAgent framework, which transports the knowledge of the isolated agents in a unified manner, and effectively guides CLIP to generalize with multi-source knowledge distillation. With such distinct framework, we flexibly collaborate with 11 heterogeneous agents to empower vision-language foundation models, without further cost in the inference phase. Finally, our TransAgent achieves state- of-the-art performance on 11 visual recognition datasets. Under the same low-shot setting, outperforms the popular CoOp with around 10% on average, and 20% on EuroSAT which contains large domain shifts. The code will be released at https:://itthrr wwt.///om////eaaaaeeeeeee htwpyy/ggggggtaaaannttme
Introduction
Recently, Vision-Language (V-L) foundation models are mainly pre-trained by contrastive learning with massive image-text pairs from web [61
38 15). As result, they show the potential on number of downstream visual recognition tasks, by transferring their representations with prompt learning [87] [6) and/or model adaptation 31
80 However, target domain data in the open world are diversified, e.g., AT (35) refers to satellite images that are highly different from web images in the pre-training. With this large domain shift, it is challenging to achieve good generalization only by adopting such single model (e.g., CLIP), especially under low-shot regime. Alternatively, with the fast development in vision and NLP, there arises wide range of expert models
33 43 49. [8. 64 11 46, [2] which contain rich knowledge by pre-traue pre-training on different modalities, tasks, networks, and datasets. Hence, the natural question is, is it possible to integrate such knowledge to boost vision- language foundation models?
To answer this question, we should further analyze the form of knowledge in these models. The simplest form is the model output, which explicitly exhibits what kind of tasks these models can tackle. However, these models have heterogeneous network structures and outputs, making direct knowledge combinations infeasible. Hence, existing works often choose cascades of these models
Interns at Shanghai Al Laboratory.
CCorresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Viston Agents
Viruus Recugratten
Prediction
Knowledge Versatiity
Pleatalliy
Deplsymett
Bifidency
VITIME
SAM
Visual
x
x
Knowledge
modal Agents
Jouge
Gesernzion
Prompi5BC
Dislee-Language Foundation Modal
CaFD
x
X
Mexal
Phompt
Notzal
Eassderr
Multi-rodal Kazwludge
Stalie Dillasion
Pronat
BLIP2
Avenage ArenageArcuurs Accuure
Knowledge
3
Chuclist
Passier
Ø
Langsage Agunts
Tloce
Novel
IIM
(a) Overview of TransAgent
(6) Comparison with. SOT methods
Figure I: An overview of our TransAgent. (a) Trans/gent transfers multi-source knowledge from heterogeneous agents to enhance the generalization ability of vision-language foundation models. It demonstrates knowledge versatility, transfer flexibility and deployment efficiency through elaborate agent collaboration and knowledge ensemble strategy. (b) SOTA comparison for base-to-novel generalization on 11 visual recognition benchmarks. Our method outperforms previous SOTA, especially on the more diversified target domains.
81 72 47,, according to their output forms. Apparently, such design lacks flexibility in transfer learning, based on tool invocation in sequence. Moreover, the resulting pipeline is unfriendly for deployment, due to the ensemble of various models in the inference phase.
Another form is the latent representation which implicitly encodes data knowledge in these models (36 4 M. Compared to the explicit output, this implicit representation has a key advantage, i.e., the feature vector that has homogeneous form among different models. In other words, such vectorized knowledge opens the possibility for a unified integration of these heterogeneous agents. To our best
Based on this observation, we propose general TransAgent framework in Figure [] knowledge, it is the first unified distillation framework for generalizing vision-language foundation models with efficient heterogeneous agent collaboration. Compared to the previous works mentioned above, our TransAgent contains three distinct technical contributions.
(1) Knowledge Versatility. In TransAgent, leverage 11 heterogeneous agents from vision, language and multi-modal research, which comprehensively covers diversified knowledge that is complementary with CL.IP-like models, from visual recognition to dense prediction, from chatbot to text encoder, from multi-modal generation to caption.
(2) Transfer Flexibility. First, provide generic knowledge extraction method for each modality, allowing us to flexibly extend more agents if necessary the future. Especially for multi-modal agents, design novel manner to extract the prediction score vector of classes knowledge the target domain, via elaborate mining of vision-language alignment these models. Second, introduce mixture-of-aggents gating mechanism for integrating external knowledge of different agents in each modality. This allows our TransAgent to automatically select agents via soft weighting that it can adaptively tackle few-shot settings in different domains of target datasets.
mudti-modal
(3) Deployment Efficiency. We leverage souti-source distillation to transfer knowledge these heterogeneous agents into CLIP. Since all these pre-trained models are frozen, the fine-tuning effort neglectable with a few learnable prompts. More importantly, we can unload all the external agents after distillation, i.e., the inference pipeline with the enhanced CLIP is just the same the original one, achieving deployment efficiency without a heavy model ensemble.
Finally, we conduct extensive experiments on 11 visual recognition benchmarks, where our TransAgent achieves the state-of-the-art under the same low-shot transfer setting, egg. via knowledge collaboration, it outperforms the well-known CoOp [87] with around 10% average and 20% on FuroS which contains large domain shifts. Our method also achieves better results than CaFo 81] which adopts a model ensemble strategy.

---

2 Related Work
Foundation models. The rapid advancements in deep learning methods have brought abundant pre-trained models to the research area. We group these models into four categories and further demonstrate their ideas below. (i) Vision models: Vision foundation models [33
3 59
32 13
pre-trained on ImageNet [2] have shown outstanding transfer capability in visual recognition by
fine-tuning on downstream datasets. Moreover, various models [43 49 16 17 8 can be specialists in dense prediction tasks by pre-training on task-relevant domain data. (ii) Large language models: The emergence of large language models (LLMs) 77 [8] 69, 70] 27 has been raising increasing attention from the research community and the public. The astonishing comprehension ability of the LLMs is credited to the linguistic knowledge which can be further applied to solve vision tasks [31] 20 88 79). (iii) Text-to-image generative models: Text-conditiones generation task requires level understanding of the given prompts. Recently, Liffusioo--bsse generative models 56 62 66) have become the state-of-the-art. These models can follow the text conditions faithfully and generate desired outcomes, owing to the semantic knowledge learned during the pre-training stage. (iv) Image-to-text captioning models: These models typically integrate visual knowledge into LLMs to obtain multi-modal understanding abilities [46] 15 2 25 47), offering better experience in referential dialog scenario. In this work, we excavate the underlying knowledge in these heterogeneous models to empower the VL foundation models.
37 23
Few-shot adaptation. To efficiently transfer vision-language foundation models like CLIP [6]
to downstream tasks, researchers have proposed various adaptation methods, which are primarily
based on prompt learning [87 86 (3 655 85]
(33 (50] or adapter [31 80 688
48 81 75 [9] Lu
al. [ explore the potential of collaborating CLIP's architectural variants and propose adaptive
ensemble strategies to enhance the generalization performance. PromptKD (50) adapts a larger CLIP
teacher to downstream datasets and distills the knowledge to smaller student in an unsupervised manner, separating the need for labeled domain data during transfer. TaskRes (75] proposes to decouple the prior knowledge of the pre-trained models and the task-specific knowledge, enabling reliable old knowledge preservation and flexible new knowledge exploration. GraphAdapter [48] further utilizes the dual-modaa moddaity structure knowledge for better adaptation in downstream tasks.
Agent collaboration. Considering the complementary knowledge of diverse pre-trained models specialized in different domains or tasks, several works have been proposed to solve vision tasks 52 71 28). The most relevant to our work is CaFo [81]. which with agent collaboration 774 82 72
transfers the external knowledge using cache models [0) However, such ensemble manner introduces further costs in the inference stage. On the contrary, we adopt heterogeneous agent collaboration to aggregate the knowledge, and distillation strategy inject the knowledge into CLIP, which demonstrates better performance and guarantees deployment efficiency.
Multi-teacher distillation. To improve the effectiveness of knowledge distillation 10, 30 26 76
36 recent works
83 54 63) attempt to integrate the knowledge from multiple teacher networks. To be noted, how to perform multi-teacher distillation is not trivial and how to extract and collaborate knowledge from various heterogeneous teachers has not been fully explored for CLIP-like foundation models. In this work, we devise a generic knowledge extraction method and flexible knowledge collaboration mechanism to further enhance the generalization ability of VI. foundation models.
3 Method
In this section, we introduce our TransAgent in detail. As shown in Figure it consists of vision- language foundation models and agents from different modalities. In the following section, we will illustrate how to collaborate with them for downstream visual recognition tasks. To start with. briefly review the V-L foundation models by using the well-known CLIP 61]. Specifically, CLIP consists of two branches. In the vision branch, an image is first divided into L non-overlapping equal-sized patches and then projected as input visual tokens Vin. which are fed to the vision encoder to obtain image feature. In the language branch, text description is projected input textual tokens Ti, which are then processed by the text encoder to generate text feature. Through contrastive learning over massive image-text pairs, CLIP achieves good alignment between the two modalities.
More interestingly, such large-scale V-L. models show the potential in visual classification on the downstream tasks. By converting each class label into a text template such as "a photo of a (class
3

---

Studen Enasder
GPT-3
BERT
DINO
Escoder
*
Decsder
Whet clone (CLS) look ltke?
[GL5] looks
Baad
Encoder
Gited
+
Gate
Tiken
Vicuna
BERT
VItDet
Eaceder
Procepes
Demnter
[CLS]
SAM
Bacoder
Vocil Veiens
Visto Bocoder
Distillation
Test Uncoder
Distillation
W
phote of [CUS]
Muu
Mau
(a) Viston Agent Collaboration
(b) Language Agent Collaboration
Vision Agent Collaboration and Language Agent Collaboration. (a) VAC integrates visual knowledge via MoA gating and transfers the knowledge through layer-wise feature distillation. (b) LAC enhances the textual representations through class-specific feature distillation between the prompted textual feature and the gated textual feature.
Figure 2:
name]", these models can easily achieve zero-shot inference. To further enhance their generalization ability under the few-shot settings, prompt learning methods are proposed, which introduce number
of learnable prompts while freezing the pre-trained CLIP 87 86 49 45). Following previous works, we add set of learnable textual prompts Pr E Nexx where Netr the number of learnable prompts, in the language branch and concatenate them with the textual tokens, which are then processed by the text encoder to obtain the prompted textual feature T E Naxx'C for all
Nets categories. Similarly, set of learnable visual prompts Py are inserted in the image branch to generate the prompted visual feature V E IR- XC where N denotes the number images:
T TextEncoder(TT... PT), V VisionEncoder((V in. Pv).
(1)
Consequently, we compute the prediction score vectors {S*} for the image samples, where S the cosine similarity between the visual feature V and the textual feature T of specific class c. As a result, we minimize the cross entropy loss between the score vectors and the ground truth labels:
LCr. = CrossEntrropyssofttaa
Y):
to fine-tune the learnable prompts Pr and Py for visual recognition. However, mentioned in the introduction, is difficult to achieve good generalization by adapting such a single CLIP model, especially when the domain shift of the target datasets is large. Hence, propose to transfer diversified knowledge from heterogeneous agents in different modalities for better adaption CLIP.
3.1 Vision Agent Collaboration (VAC)
One important component of CLIP is its vision branch in Figure Hence, we consider to enhance this branch by transferring visual knowledge from various vision agents. achieve this goal, we have to answer three critical questions. The first question is which models should be used for collaboration. As we know, CLIP mainly establishes image-level alignment between the two modalities, neglecting visual details in the pixel space. To fill this gap, choose vision agents from two aspects. On one aspect, we choose vision models pre-trained with self-supervision such as MAE 33] and DINO 9]. Both agents focus on detailed image modeling via image masking [3 or patch self-distillation On the other aspect, we choose vision models built on dense supervision such as ViTDet [49 and SAM [43. Both agents work instance-level prediction with bounding boxes and masks. Detailed information on these models can be found in the supplementary.
9
The second question is how to extract visual knowledge by collaborating with these agents. As mentioned the introduction, the latent feature a common knowledge form among these het- erogencous models. Hence, given input image, extract the intermediate visual features

---

(Va(i) E I ] from the vision encoders of these agents. Moreover, the contribution of different agents may vary among different domains. To fully exploit these agents in unified manner, we introduce Mixture-of-Agent visual knowledge. Specifically, we concatenate all the agent features along the channel dimension C, and feed them into an MLP network to generate the gating weight Wv. Next, we obtain the gated visual features V by computing the weighted sum over (i)):
(MoA) gating mechanism to adaptively integrate (V^(i)} as the
Wy MLP(Concat(( (V)())) VA= Σ. (r())a(()
(3)
The final question is how to transfer the visual knowledge to enhance CLIP's vision encoder. We adopt feature distillation (36] where we compute the L1 loss between the prompted visual features V in Eq. and the gated visual features of the vision agents:
CvcC V Val.
Since the original CLIP is frozen, the above loss term allows us to fine-tune the learnable visual prompts Py in Eq. and MI.P gating network in Eq. 3 which enables us to adaptively transfer external visual knowledge to the visual prompts to empower generalization ability.
3.2 Language Agent Collaboration (LAC)
The other important component of CLIP is its language branch in Figure 2 Similar to the vision branch, we consider three critical questions to transfer the textual knowledge from various language agents. Recent studies [60] 42 have shown that is coarse to use a simple template (e.g.. "a photo of (class name)") ") to describe certain category. Hence, first interact with the popular chatbots such as GPT-3 7 and Vicuna (18) to enrich the class descriptions using queries like "What does (class name] } look like?". After obtaining the detailed descriptions from these chatbots, we use a text encoder (e.g., BERT (22)) to extract the text features (T) E RNaxxC] j of all descriptions. To adaptively integrate (J) the textual knowledge, we also utilize the MoA gating mechanism:
WT MLP(CCnnca(( (Tx()))
T T Σ, WT())T
Finally, for each category, we perform feature distillation where we compute the LI loss between the prompted textual feature in Eq. and the gated textual feature from language agents in Eq. 5
LLAC T AT
(6)
By fine-tuning the learnable textual prompts Pr in Eq. [ and MLP gating network in Eq. 5 we
adaptively transfer the rich textual knowledge to enhance CLIP's textual representations.
3.3 Multi-modal Agent Collaboration (MAC)
Through vision and language agent collaboration, we can enhance the Iearnable visual and textual prompts respectively. Considering the key success in vision-language foundation models is credited to the multi-modal alignment, we investigate how to further align the visual and textual prompts with external multi-mode agents. First, there exist two types of multi-modal agents, including Text-to-
Image (T2I) generative models [64]] [1] and Image-to- Text (I2T) captioning models (46 12] Since both types of agents involve conversion from one modality to the other, we believe they implicitly achieve vision-language alignment. Based the above discussion,
Texte
choose our T21 agents built
upon two mainstream structures including Stable Diffusion
(4 in UNet style and Art-rt [TT]
in DiT style. Moreover, we choose our 12T agents specialized in two mainstream tasks including BLIP-2 [46 for general captioning and Shikra [2] for grounded dialogue.
Next, we consider what kind of knowledge can represent vision-language alignment. The most direct form may be the probability score vector that shows the prediction confidence over target classes. Hence, for each training image, we investigate such knowledge in these multi-modal agents.
In the T2I agents, cross attention effectively encodes relations between image and text R4 Hence, we leverage such prior to extract the score vector. Specifically, we use the template description of a class e as text. Then, we extract cross attention value M between the text and the k-th token of the input image from the pre-trained T2I agents. Consequently, we sum over all the tokens to
5

---

Block
Viilon Enceder
Vision Encoder
Styyl
Style
Q-Vormer
Up Block
L.M
Stable Didfusion
721 Agents
PixArt-4
BLIP2
12T Agrnts
Shulkara
Muxic
WKG
Mays
Atteriin Mays
Mhitpticatio
Spasia Phsiie
Spotial Proting
dity sola
Gate
A
Vision Bnooder
C
Mwi
MaukC
Naclie
Distillution
Nuyac
CLS, (5 055
Leased Sanchatav
Galol Sore Roar
Text Encoder
Figure 3: Multi-modal Agent Collaboration. Top left: We first extract the cross attention maps from the T2I agents and then obtain the score vectors through LSE pooling. Top right: We compute the score vectors from the 12T agents as the cosine similarity between the projected visual feature and the LLM's textual feature. Finally, we perform score distillation between the learned score vectors and the gated score vectors to further align the learnable prompts.
obtain the prediction score of the image w.r.t class c: S = log( Ex exp(M;)), where we adopt LogSumExp (LSE) pooling [5] to provide more accurate matching scores.
In the 12T agents, there exists a projection module (e.g., Q-Former in BLIP-2 46 and MLP in Shikra [12] to adapt the visual features for the large language model. Hence, we extract the visual feature of an input image via the projection and the textual features of all the classes from the LLM. By computing the cosine similarity between them, we can obtain the prediction score S127 of the image over all the classes. Next, we leverage the MoA gating mechanism to adaptively summarize the prediction score vectors from all the multi-modal agents MA- Concat (Sr21, S12T));
Ws MLP(MA), SA SA=E, (s())MA(n).
(7)
Finally, we explore how to transfer S to enhance the learnable prompts in CLIP. Specifically, after being processed by the vision and text encoders of CLIP, the learnable visual and textual prompts Py and P are transformed into Qy and Qr- Unlike Py and Py which are universal, Qv is relevant to the image samples and Q is relevant to the target classes. Hence, similarity between Qv and Qr to obtain the learned score vectors Sp R. Nat N of the input images over all the classes. We perform score distillation between Sp and SA:
compute the cosine
LMAC (L.(sofmax (Sp) |softmaxx (S))
(8)
Through computing the KL divergence, we can leverage external multi-modal knowledge S as semantic guidance align the learnable visual and textual prompts.
3.4 Multi-Source Knowledge Distillation
Finally, we combine all the distillation loss from multiple sources, achieving heterogeneous agent collaboration for knowledge transfer:
LrransAgen Lce + Xy + AeLLAC + A3LMaC,
(9)

---

Table I: Accuracy comparison with state-of-the-art methods base-to-novel generalization. All methods use CLIP's ViT-B/16 as the vision encoder. Our TransAgent exhibits strong generalization ability and outperforms previous SOTA on all datasets. The best results are bolded.
Average
ImageNet [2]
Caltech101
29
OxfordPets
58
Method
Base
Novel
HM
Basc
Novel
HM
Base
Novel
HM
Base
Novel
HM
CLIP 61
69.34
74.22
71.70
72.43
68.14 14
70.22 96.84
94.00
95.40
91.17
97.26
94.12
CoOp g
82.69
63.22
71.66
76.47
67.88
71.92
98.00
89.81
93.73
93.67
95.29
94.47
CoCoOp 86
80.47
71.69
75.83
75.98
70.43
73.10
97.96
93.81
95.84
95.20
97.69
96.43
MaPLe [40]
82.28
75.14
78.55
75.40
70.32
72.72
98.27
93.23
95.68
95.43
97.83
96.62
RPO [45]
81.13
75.00
77.78
76.60
71.57
74.00
97.97
94.37
96.03
94.63
97.50
96.05
PromptSRC
41
84.26
76.10
79.97
77.60
70.73
74.01
98.10
94.03
96.02
95.33
97.30
96.30
TransAgent
85.29
77.62
81.27
78.07
70.57
74.13
98.90
95.23
97.03
96.33
98.13
97.22
StanfordCars
44
Flowers102
[77
Food101
6
FGVCAireraft [55
Method
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CLIP 61
63.37
74.89
68.65
72.08
77.80
74.83
90.10
91.22
90.66
27.19
36.29
31.09
CoOp 87
78.12
60.40
68.13
97.60
59.67
74.06
88.33
82.26
85.19
40.44
22.30
28.75
CoCoOp 88
70.49
73.59
72.01
94.87
71.75
81.71
90.70
91.29
90.99
33.41
23.71
27.74
MaPLe
40
74.70
71.20
72.91
97.70
68.68
80.66
90.30
88.57
89.43
36.90
34.13
35.46
RPO 45
73.87
75.53
74.69
94.13
76.67
84,50
90.33
90.83
90.58
37.33
34,20
35.70
PromptSRC
41
78.27
74.97
76.58
98.07
76.50
85.95
90.67
91.53
91.10
42.73
37.87
40.15
TransAgent
79.53
74.73
77.06
98.37
77.13
86.46
90.87 92.20
91.53
43.77
39.00
41.25
SUN397
73
DTD
19
EuroSAT
35
UCF101
67
Method
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
Base
Novel
HM
CLIP 61
69.36
75.35
72.23
53.24
59.90
56.37
56.48
64.05
60.03
70.53
77.50
73.85
CoOp g7
80.60
65.89
72.51
79.44
41.18
54.24
92.19
54.74
68.69
84.69
56.05
67.46
CoCoOp 36
79.74
76.86
78.27
77.01
56.00
64.85
87.49
60.04
71.21
82.33
73.45
77.64
MaPLe 40
78.47
76.93
77.79
80.67
56.48
66.44
83.90
66.00
73.88
85.23
71.97
78.04
RPO 451
80.60
77.80
79.18
76.70
62.13
68.61
86.63
68.97
76.79
83.67
75.43
79.34
PromptSRC
41
82.67
78.47
80.52
83.37
62.97
71.75
92.90
73.90
82.32
87.10
78.80
82.74
TransAgent
82.90
79.30
81.06
84.37
63.67
72.57
97.43
83.43
89.89
87.60
80.47
83.88
where Å1. A2. Ag are hyperparameters. Since all the pre-trained models are frozen in the training phase, we only need to fine-tune the learnable vision and language prompts with negligible cost. More importantly, owing to the distillation strategy, all the agents be unloaded and the modality-specinf gates can be abandoned in the inference phase. We can simply use the enhanced CLIP just like the original one, which largely boosts deployment efficiency without model ensemble.
4 Experiments
Datasets and Metrics. We evaluate our proposed method on 11 commonly used datasets covering
a wide range of recognition tasks, including ImageNet (21) Caltech101 29], OxfordPets [58]
StanfordCars
44, Flowers 102 [57], Food101 66. FGVCAircraft [55] SUN397 773. UCF101 67
DTD 19] and EuroSAT [35). We explore two typical low-shot scenarios to evaluate the performance. (i) Baseeto-novel generalization: The datasets
equally split into base and novel classes. The model is trained on base classes and evaluated on the test set of both classes. We report the base and novel class accuracy and the harmonic mean (HM) of the results. (ii) Few-shot classification: We assess the accuracy trained with 1/2/4/8/16 shot(s) per class to examine the model's learning capacity.
Implementation Details. We adopt CLIP ViT/B-16 16 as our backbone and conduct all experiments using different seeds to obtain an averaged result, following previous works 40
41 50 Our
method ensembles knowledge from heterogeneous agents, including pre-trained vision models, LLMs, T21 generative models, and I2T captioning models. Detailed information on the agents and training settings are shown in Appendix
7

---

Average over 11 datasats
Imageliet
Caltcch011
925-
Zeoand CJT
8
Besans CJT Liaar pots
Zaro thit CLP
pubs CJ*
Unerr probe tur Cade
Cato
Cate
45-
Tursgget
Turndgent
16
15
wortkss per
per
60-
Zeo over cup
Zero dvir C.P
Zero
cuT
Linar cu'
Linur prote
50
Cede
Cato
40
Coon
Cafe
35-
CodP
Cole
16
Number oflubded tuiring sourssh per des
Nambur of tua uonska per dess
Nambur Libded uurtrg per dass
Food101
FOVCAirraatt
SUN397
AcCIFOr TM
co-
Zeo our
Zero thet CiP
Zamo over CuP
50
Coop
Coop
50-
Cafo
Cate
45-
Cafe
Turssgent
20
Tursgget
Tursgget
Kumber of mwwiteoonnse dass
cttbeeesttiiinnnneeee. per doot
of mistgromite
EuroSAT
UCF101
70-
55-
Accuscy
Zeo tue CuP Liorr pross C.P CodP
Zero C.P Livar pioze CJP Colp
Zero
Lieer poss Cup
Coop
PansAger
TaamAgee
TarnAgee
16
Kumber ddbibeed wwwsteeoooeeeeeeeeoooo... per disss
Number htt.oo/eeemmiitoooooseee doos
Kunber Aiiiees wwwteeoosseeeeeeeeeoo.o. doos
Figure 4: Accuracy comparison in few-shot classification. Trans/gent demonstrates state-of-the- art performance for all few-shot settings on different datasets, which proves promising learning capability even under extremely limited supervision.
4.1 Comparison with State-of-the-Art
In T we compare the base-to-nove gemeralization performance of our proposed TransAgent with state-of-the--n prompt learning methods, including CoOp [87) CoCoOp [86). MaPLe [40] RPO 45 and PromptSRC [T]. All approaches use the same CLIP VYT-B/16 during the evaluation stage. Our method demonstrates superior performance across 11 datasets, surpassing all competing methods in terms of base and novel accuracy, as well as the harmonic mean, particularly excelling on the EuroSAT dataset. In Figure 4 we present the few-shot classification performance of TransAgent and previous methods. Our method consistently outperforms the counterparts and demonstrates growing learning capability when the training samples increase. To be mentioned, TransAgent outperforms CaFo [8I] with much lower deployment costs (see Table [3). Detailed comparisons on cross-dataset evaluation and domain generalization are provided in Appendix B
4.2 Ablative Analysis
In this section, we present the ablative analysis of our agent collaboration designs for each modality. We adopt IVLP [40 [1] our bascline model.
8

---

Caltech
Pats
Cars
Cars
Cars
Plowers
Flowers
Flowers
Datasets
Food
Food
Aircraft
Aireraft
Atreraft.
DTD
DTD
DTD
ETT
EuroSAT
EuroSAT
UCF101
UCF101
SUN387-
SUN397
SUN297
ImageNet
ImageNNe
ImageNNt
MAE
DINO
SAM
VitDet
GPT-3 Vicuna
SD
Pixart
BLIP2
Shikra
Agents
Figure 5: Averaged gating weights of each agent on different datasets. Deeper color indicates more contributions to the gated feature(s) vectors.
Table 2: VAC Design.
Table 3: LAC Design.
Table 4: MAC Design.
Models
Base
Novel
HM
Models
Base
Novel
HM
Models
Base
Novel
HM
baseline
84.21
71.79
77.51
baseline
84.21
71.79
77.51
buseline
84.21
71.79
77.51
MAE
84.51
71.66
77.55
Stable Diffissio
84,91
73.11
78.57
DINO
84.60
70.90
77.15
GPT-3
85.15
79.50
Pixart-a
84,78
73.47
78.72
SAM
84.73
71.87
77.77
Vicuna
85.35
74.70
79.67
BLIP2
84,87
73.43
78.73
ViTDet
84.45
72.04
77.75
SOOS
84.19
75.25
79.47
Shikra
84,97
72.90
78.47
Last layer
84.47
75.92
79.97
EOS
85.29
77.62
81.27
Prompted logits
84.45
75.18
79.54
Layer-wise
85.29
77.62
81.27
Learndd scores
85,29
77.62
81.27
Add
Average
84.40
84,20
75.17
74.79
79.22
79.52
Add
Average
82.44
84.23
75.98
78.48
79.89
Add
Average
84,33
84.37
75.35
75.04
79.61
Gating
85.29
77.62
Gating
85.29
77.62
81.27
Gating
85,29
77.62
81.27
Table 5: Performance w/ individual module.
Table 6: Performance w/ module combinations.
VAC
LAC
MAC
HM
A
VAC
LAC
MAC
HM
A
77.51
/
/
x
80.02
+2.51
79.04
+1.53
x
/
79.79
+2.28
X
/
x
79.90
+2.39
x
/
80.40
+2.89
x
X
/
79.61
+2.10
/
81.27
+3.76
Effectiveness of individual agent. In Table Table 4 we show the results of introducing an individual agent as a teacher supervise the baseline model in the beginning rows. Nearly all vision agents contribute to the improvement in accuracy, except for DINO which is behind. However, we observe that DINO performs well on most datasets, except on EuroSAT (with 5.60% decline). Nonetheless, we still keep DINO as one of the vision agents. While for the language and the multi-modal agents, they all boost the performance of baseline model respectively.
Distillation strategy. In the middle rows of Table 2 Table we ablate the alignment strategy for agent collaboration of each modality. (i) For VAC, we choose adopt feature distillation with either the seerage-pooleet feature at the last layer or all features at all layers. As the results suggest, layer-wise distillation of all features performs better. (ii) For LAC, two special tokens from the output of the text encoder are considered to compute Equation 6 These tokens are inserted into the text sequence to compose a complete prompt. [EOS] performs better owing to the causal attention module in Transformer blocks, so that it aggregates more information. (iii) For MAC, either the prompted logits CLIP or the learned scores (Eq. g are chosen align with the semantie logits from multi-modal agents. Since the prompted logits need to be aligned with ground-truth labels as well (Eq. 2). there might be confusion align it with the external knowledge simultaneously. Experimental results show that using learned scores to align with semantic logits yields better results.

---

Fusion design. In the last rows of Table 2 Table we ablate the fusion design for each modality. "Average" refers simply calculating the average all output features from the agents along the channel dimension; "Add" refers calculating the distillation loss separately for each agent; "Gating" denotes our proposed MoA gating mechanism, As can be seen, gating fusion achieves the best results for all collaboration designs, since it adaptively selects the useful information from the agents.
Effectiveness of collaboration module(s). As presented in Table 5 the first row indicates the baseline model. Each individual module is beneficial for improving the generalization ability of the foundation models, where the last column shows the relative increase. Results in Table 6 demonstrate that the combinations of collaboration modules further boost the results.
4.3 Visualization
We calculate the averaged gating weights each agent with 16-shot training samples from all classes of certain dataset and present the results with heat maps. As shown in Figure S different agents do not contribute equally towards certain target datasets, which also verifies the superiority our gating fusion design. For vision agents, DINO experts in recognizing general objects while lags behind on some fine- pre-grained datasets (e.g., EuroSAT), whereas the others that focus more on details perform better. Language agents provide domain-specific linguistic knowledge accordingly. I2T agents consistently provide their knowledge thanks to the grounding nature, while T2I agents demonstrate better capability in fine-grained scenarios.
Conclusion
We propose Trent,gent, unified framework transfer vision-language foundation models through heterogeneous agent collaboration. By adaptively integrating the external knowledge of agents from different modalities viac the MoA gating mechanism, TransAgent achieves state-of-the-art performance on 1I datasets under the low-shot scenarios.
Limitations. Although TransAgent collaborates with heterogeneous agents in unified manner, transferring the external knowledge through distillation may harm the original CLIP's s representations using prompt learning methods because the domain knowledge from agents are diversified, and irrelevant information may also be introduced. Moreover, one of the key characteristics of these agents is large-scale pre-training, few-shot scenarios may not meet their data-hungry nature. So our future work would be integrating the knowledge with directed focus and further unleashing the potential of the agents even when the Nomain-specifie knowledge (e.g., labels) is absent.
Acknowledgments and Disclosure of Funding
This work supported in part by the National Key R&D Program of China (NO.2022ZD0160505)) the National Natural Science Foundation of China under Grant (62272450)
References
[1] R. Adriana, B. Nicolas, S. Ebrahimi, C. Antoine, G. Carlo, and B. Yoshua. Fitnets: Hints for thin deep nets. International conference on learning representations, 2015.
[2] J.BB. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23783 2022.
[3] H. Bao, L. Dong, Piao, and Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv: 106..082264 2021.
[4] Y. Bengio, A. Courville, and P. Vincent. Representation learning: review and new perspectives. IEEE transactions pattern analysis and machine intelligence, 35(8): 1788-1888 2013.
[5] Blanchard, J. Higham, and J. Higham. Accurately computing the log-sum-exp and softmax functions. IMA journal of numerical analysis, 41(4):2311 1-133330,111300 2021.
10

---

[6] L. Bossard, M. Guillaumin, and L. V. Gool. Food-101- mon-iig discriminative components with random forests. In European conference on computer vision, pages 146-461. Springer. 2014.
[7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, Sutskever, and D. Amodei. Lan- guage models are few-shot learners. Advances in neural information processing systems, 33: 1877-1901, 2020.
[8] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[9] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, Bojanowski, and Joulin. Emerging properties in self- supervisdd vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, October 2021.
[10] Y. Chebotar and A. Waters. Distilling knowledge from ensembles of neural networks for speech recognition. In Interspeech, pages 3439-3443, 2016.
[11] J. Chen, Y. Jincheng, G. Chongjian, L. Yao, E. Xie, Z. Wang, J. Kwok, P. Luo, H. Lu, and Z. Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International conference learning representations, 2024.
[12] K. Chen, Z. Zhang, W. Zeng, Zhang, F. Zhu, and R. Zhao. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic. arXiv preprint 2xXi::3306.1195 2023.
[13] Chen, S. Kornblith, M. Norouzi, and G. Hinton. simple framework for contrastive learning of visual representations. In Proceedings of the international conference on machine learning. pages 1597-1607. PMLR, 2020.
[14] X. Chen, X. Wang, S. Changpinyo, A. Piergiovanni, P. Padlewski, D. Salz, S. Goodman, A. Grycner, B. Mustafa, L. Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint 2XXi:2209.06794 2022.
[15] Z. Chen, Wu, W. Wang. Su, G. Chen, S. Xing. Z. Muyan, Q. Zhang, Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.00 14238, 2023.
[16] B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation, Advances in neural information processing systems, 34:17864-17875, 2021.
[17] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attentin mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 1290-1299 2022.
[18] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng. Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang. J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL htttts://mmmy::
03- 00-30--0000000000000000000000000 -0---i0vvviii vicuna
[19] M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1606-3613, June 2014.
[20] W. Dai, Li, D. Li, A. M. H. Tiong, J. Zhao, Wang, B. Li, P.N. Fung, and S. Hoi. Instructblip: Towards general-pure purppose vision- vision-language models with instruction tuning. Advances in neural information processing systems, 36, 2024.
[21] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 248-255 IEEE, 2009.

---

[22] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
[23] Dhariwal and A. Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780- 8794, 2021.
[24] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for
image recognition at scale. In International conference on learning representations, 2021.
[25] D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint wwXiv:2303.03378, 2023.
[26] S. Du, S. You, X. Li, J. Wu, F. Wang, C. Qian, and C. Zhang. Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. Advances in neural information processing systems, 33: 00112255-12335 2020.
[27] Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang. Glm: General language model pretraining with autoregressive blank infilling. arXiv preprint arXiv:2103. 10360, 2021.
[28] X. Fan, Ji, C. Jiang, S. Li, S. Jin, S. Song, J. Wang. B. Hong, L. Chen, G. Zheng, et al. Mousi: Poly-visualeexperr vision- language models. arXiv preprint arXiv:2401.17221, 2024.
[29] L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In IEEE/CVF conference computer vision and puttern recognition workshop, pages 178-178. IEEE, 2004.
[30] T. Fukuda, M. Suzuki, G. Kurata, S. Thomas, J. Cui, and B. Ramabhadran. Efficient knowledge distillation from an ensemble of teachers. In Interspeech, pages 3697-3701, 2017.
[31] P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better vision-language models with feature adapters. International journal of computer vision, 132(2): 581-595, 2024.
[32] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages
2020.
[33] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, June 2022.
[34] X. He, W. Feng, T.-J. Fu, V. Jampani, A. Akula, P. Narayana, S. Basu, W. Y. Wang, and X. E. Wang. Discriminative diffusion models as few-shot vision and language learners. arXiv preprint arXiv:2305. 10722, 2023.
[35] Helber, Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE journal of selected topics in applied earth observations and remote sensing. 12(7):2217-223) 2019.
[36] G. Hinton, O. Vinyals, and Dean. Distilling the knowledge in a neural network. arXiv preprint ArXiv:1503.025: 2015.
[37] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 6851, 2020.
[38] C. Jia, Y. Yang, Y. Xia, Y. Chen, Z. Parekh, H. Pham, Q. V. Le, Y. Sung, Z. Li, and T. Duerig. Scaling up visual and vision- language representation learning with noisy text supervision. In Proceedings the international conference on machine learning, volume 139, pages 4904-4916. PMLR, 2021.
[39] M. Jia, I. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim. Visual prompt tuning, In European conference on computer vision, pages 709-727, 2022.
12

---

[40] M. U. Khattak, H. Rasheed, M. Maaz, S. Khan, and F. S. Khan. Maple: Multi-modie prompt learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- lion, pages 191 13-19122, June 2023.
[41] M. U. Khattak, S. T. Wasim, M. Naseer, S. Khan, M.-H. Yang, and F. S. Khan, Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF international conference on computer vision, pages 15110-15200, October 2023.
[42] M. U. Khattak, M. F. Naeem, M. Naseer, L. Van Gool, and F. Tombari. Learning to prompt with text only supervision for vision-language models. arXiv preprint wxXiv:2401002410 2024.
[43] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. P. Dollar, and R. Girshick, Segment anything. In Proceedings of the JEEF/CVF international conference on computer vision, pages 4015-4026, October 2023.
[44] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE/CVF international conference on computer vision workshop, pages 554-561, December 2013.
[45] D. Lee, S. Song. J. Suh, J. Choi, S. Lee, and H. J. Kim. Read-only prompt optimization for vision-language few-shot learning. In Proceedings of the IEEE/CVE international conference on computer vision, pages 1401-1411, October 2023.
[46] J. Li. D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the international conference on machine learning, volume 202, pages 19730- 19742. PMLR, 23-29 Jul 2023.
[47] K. Li, Y. He, Y. Wang, Y. Li, W. Wang, P. Luo. Y. Wang. L. Wang, and Y. Qiao. Vidcochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06353 2023.
[48] X. Li, D. Lian, Z. Lu, Bai, Z. Chen, and X. Wang. Graphadapter: Tuning vision-language models with dual knowledge graph. Advances in neural information processing systems, 36, 2024.
[49] Y. Li, H. Mao, R. Girshick, and K. He. Exploring plain vision transformer backbones for object detection. In European conference on computer vision, pages 280-296, 2022.
[50] Z. Li, X. Li, X. Fu, X. Zhang, W. Wang, and J. Yang. Promptkd: Unsupervised prompt distilla- tion for vision-lanp Laiiun-angguage models. In Proceedings of the IEEE/CVF international conference on computer vision, 2024.
[51] H. Liu, C. Li, Q. Wu, and J. Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[52] M. Liu, S. Roy, W. Li, Zhong, N. Sebe, and E. Ricci. Democratizing fine fine-grained visual recognition with large language models. In International conference on learning representations, 2023.
[53] Z. Lu, J. Bai, X. Ii, 7. Xiao, and X. Wang. Beyond sole strength: Customized ensembles for generalized vision-language models. arXiv preprint ar arXiv:2311.1700 2023.
[54] Z. Ma, J. Dong, S. Ji, Z. Liu, X. Zhang, Z. Wang. S. He, F. Qian, X. Zhang, and L. Yang. Let all be whitened: Multi-teacher distillation for efficient visual retrieval. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 4126-4135, 2024.
[55] S. Maji, E. Rahtu, J. Kannala, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv: 2013.
[56] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 1rXiv:2112.107411 2021.
[57] M.-F. Nilsback and A. Zisserman. Automated flower classification over large number of classes. Indian conference on computer vision, graphics & image processing, pages 722-729. IEEE, 2008
13

---

[58] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3498-3505, IEEE, 2012.
[59] Z. Peng, Dong, Bao, Q. Ye, and F. Wei. Beit v2: Masked image modeling with vector- quantized visual tokenizers. arxiv 2022. arXiv preprint arXiv:2208.06:333. 2022.
[60] Pratt, Covert, R. Liu, and A. Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In Proceedings of the IEEF/CVF international conference on computer vision, pages 15691-15701, October 2023.
[61] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the international conference machine learning, volume 139 of PMLR, pages 9748-8763. PMLR, 18-24 Jul 2021.
[62] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hicrarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2004.06125, 1(2):3 2022.
[63] M. Ranzinger, G. Heinrich, J. Kautz, and P. Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 12490-12506 2024.
[64] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, June 2022.
[65] S. Roy and A. Etemad. munsistency-guideer prompt learning for vision-language models. In International conference on learning representations, 2024.
[66] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gon- tijo Lopes, B. Karagol Ayan, T. Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35: 36479-36494, 2022.
[67] K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
[68] Y.-L. Sung, J. Cho, and M. Bansal. VI-adapter: Parameter-efficient transfer learning for vision- and-language tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5227-5237, 2022.
[69] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro. F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.
2023.
[70] H. Touvron, L. Martin, K. Stone, Albert, A. Almahairi. Y. Babaei, N. Bashlykov, Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.002288 2023.
[71] X. Wang, Y. Zhang, O. Zohar, and S. Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. arXiv preprint arXiv:2403. 10517, 2024.
[72] C. Wu, S. Yin, W. Qi, X. Wang, Z. Tang, and N. Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.046) 043371 2023.
[73] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 3485-3492. IEEE, 2010.
[74] Ye, C. Huang, M. Shen, T. Chen, Y. Huang, Y. Zhang, and W. Ouyang. Merging vision transformers from different tasks and domains. arXiv preprint arXiv: arX1v:2312.16240, 2023.
14

---

[75] T. Yu, Z. Lu, X. Jin, Z. Chen, and X. Wang. Task residual for tuning vision- models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10899-10909, 2023.
[76] F. Yuan, L. Shou, J. Pei, W. Lin, M. Gong, Y. Fu, and D. Jiang. Reinforced multi-teacher selection for knowledge distillation. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14284-14291, 2021.
[77] H. Zhang, D. Chen, and C. Wang. Confidence-awerd multi-teacher knowledge distillation. In IEEE international conference on acoustics, speech and signal processing, pages 4498-4502. IEEE, 2022.
[78] H. Zhang, D. Chen, and C. Wang. Adaptive multi-teacher knowledge distillation with meta- learning. In IEEE international conference multimedia and expo, pages 1943-1948. IEEE, 2023.
[79] H. Zhang, Li, and L.. Bing. Video-llama: An instruction-uun -unned audio-visual language model for video understanding, arXiv preprint Sttii::30600285555 2023.
[80] R. Zhang, W. Zhang, R. Fang, P. Gao, K. Li, J. Dai, Y. Qiao, and H. Li. Tip-adapter: Training- free adaption of clip for few-shot classification. In European conference on computer vision, pages 493-510. Springer, 2022.
[81] R. Zhang, X. Hu, B. Li, S. Huang, H. Deng, Y. Qiao, Gao, and H. Li. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15211-15222, 2023.
[82] S. Zhang, Huang, Deng, Tang, W. Ouyang, T. He, and Zhang. Agent3d-zero: An agent for zero-shot 3d understanding. arXiv preprint arXiv:2403. 1135, 2024.
[83] S. Zhao, X. Wang, and X. Wei. Mitigating accuracy-robustnes trade-off via balanced multi- teacher adversarial distillation. IEEE transactions on pattern analysis and machine intelligence, (01):1-14, 2024.
[84] W. Zhao, Y. Rao, Z. Liu, B. Liu, J. Zhou, and J. Lu. Unleashing text-to-image diffusion models for visual perception. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5729-5739, October 2023.
[85] Z. Zheng, Wei, X. Hu, H. Zhu, and R. Nevatia. Large language models are good prompt learners for low-shot image classification. In Proceedings of the IEEE/CVF international conference on computer vision, 2024.
[86] K. Zhou, Yang, C. Loy, and 7.. Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages (6816-16825, June 2022.
[87] K. Zhou, Yang. C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International journal of computer vision, 130:2337-2348, July 2022.
[88] D. Zhu, Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arX4v:2304. 10592, 2023.
[89] X. Zhu, R. Zhang, B. He, A. Zhou, D. Wang, B. Zhao, and P. Gao. Not all features matter: Enhancing few-shor clip with adaptive prior refinement. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2605-2615 2023.
15

---

Table Demonstration of heterogeneous agents specialized in different domains or tasks.
Pre-training
Model
Parameters
Model Type
Knowledge
Tasks
Datasets
DINO
86M
ViT
IC
ImageNet-IK
Vision
MAE 33
86M
ViT
MIM
ImageNet-IK
Vision
SAM 43
86M
ViT
IS
SA-IB
Vision
ViTDet 49
86M
ViT
OD
COCO
Vision
GPT-3
175B
LLM
TG
Language
Vicuna 18
13B
LLM
TG
Language
BERT 22
38M
Transformer
MLM
Language
Stable Diffusion
64
0.86B
UNet
IG
LAION-2B
Multi-modal
Pixart-o
0.6B
DiT
IG
Multi-modal
BLIP-2 46
2.7B
MILM
ITC+TM+IIGG ITCHTM+ITG
Multi-modal
Shikra
12
7B
MILM
Multi-modal
Table 8: Memory and training time required for each dataset.
Setting
ImageeNtt
Cultech
101
Oxford
Pets
Standford
Cars
Flowers
102
Food101
Aircraft
FGVC
SUN397
DTD
Euro SAT
UCF101
Memory (MB)
Fow-shot (1 (16-shot)
to- fo-nevel
19978
4542
6892
6892
5488
4708
5488
5486
10110
7708
4700
4124
5484
Time (ms/batch)
Basc-to-novel
400
226
211
209
219
273
191
190
Fow-shol (16-shot)
688
214
206
262
217
215
221
356
196
205
220
A Additional Implementation Details
Training Details. The number of learnable vision and language prompt tokens are both set to 4, and the prompt depth is set to 9 for base-to-novel generalization and few-shot classification, and 3 for cross-dataset and domain generalization. The learnable text prompts of the first layer are initialized with the word embeddings of "a photo of a", while the other learnable prompts are randomly initialized with a normal distribution. For vision agent collaboration, we adopt online distillation where the models are loaded during the training process. For language and multi-modal agent collaboration, due the huge model parameters, we extract their knowledge offline before our training launches. For few-shot classification, we train the models for 50 epochs under different low-shot settings (ranging from 1 to 16). We re-implement the compared methods in Figure with with Vision Transformer (24] backbone for fair comparison. For the other benchmarks, we observe that PromptKD [50] adopts transductive setting which uses all training samples an unsupervised manner and obtains better results. However, such setting is beyond our scope. In our work, all models are trained for epochs using 16-shot samples with a fixed batch size of and a learning rate of 0.0025 with SGD as the optimizer. We set X = 1, X2 25 and A3
in Eq. 9 aftte extensive hyperparameter search to balance the total loss. The memory and training time needed under different settings each dataset provided in Table & using batch size of 4. At least 48GB is required to extract knowledge from language and multi-modal agents. All experiments are conducted on a single Nvidia A6000 GPU.
Information of agents. Table lists all the agents we have collaborated in TransAgent frame- work. (i) Vision agents: DINO provides robust representations through image contrastive (IC) learning while MAE presents powerful capability via mask image modeling (MIM). SAM and ViT- Det are specialists in image segmentation (IS) and object detection (OD) respectively. (ii) Language agents: Both GPT-3 and Vicuna are excellent chatbots that faithfully follow human instructions and respond with convincing answers by text generation (TG). We use BERT, which is just CLIP's text encoder pre-trained using mask language modeling (MLM), to process the generated captions from chatbots before transferring these linguistic knowledge. (iii) Multi-modaa agents: Stable Diffusion and Pixart-a share different model architectures, but both T21 agents demonstrate astonishing image generation (IG) capability. The 12T agents are pre-trained with multiple tasks including image-text contrastive (ITC), image-text matching (ITM), and image- -rggndddd text generation (ITG) over large-scale datasets. These agents demonstrate outstanding multi-mode understanding ability.
16

---

Table 9: Accuracy comparison with previous methods cross-dataset evaluation.
Source
Target
Method
ImageNet
Caltech
Oxford Pets
Standford Cars
Flowers
Food101
FGVC
SUN397
DTD
Euro SAT
UCF101
Avg.
101
102
Aircraft
CLIP
66.72
92.94
89.07
65.29
71.30
86.11
24.87
62.62
44.56
47.69
66.77
65.12
CoOp
71.51
93.70
89.14
64.51
68.71
85.30
18.47
64.15
41.92
46.39
66.55
63.88
CoCoOp
71.02
94.43
90.14
65.32
71.88
86.06
22.94
67.36
45.73
45.37
68.21
65.74
MaPLe
70.72
93.53
90.49
72.23
86.20
24.74
68.69
66.30
PrompiSRC
71.27
93.60
90.25
65.70
70.25
86.15
23,90
67.10
46.87
45.50
68.75
65.81
TransAgent
72.00
94.37
90.33
65.43
71.40
86.47
23.20
66.20
45.30
52.13
69.93
66.48
Table 10: Accuracy comparison on domain generalization.
Source
Target
Method
ImageNet
-V2
-S
-A
-R
Avg.
CLIP
66.73
60.83
46.15
47.77
73.96
57.18
CoOp
71.51
64.20
47.99
49.71
75.21
59.28
CoCoOp
71.02
64.07
48.75
50.63
76.18
59.91
MaPL.e
70.72
64.07
49.15
50.90
76.98
60.27
PromptSRC
71.27
64.35
49.55
50,90
77.80
60.65
TransAgent
72.00
64.87
49.63
51,23
77.53
60.82
Table 11: MAC loss type.
Table 12:
Pooling type.
Table 13: Inference time.
Loss
Base
Novel
Pooling
Base
Novel
HM
Method
Time
Accuracy
LI
84.93
74.69
79.48
Average
84.97
76.12
80.30
Zero-shot CLIP
1min06s
66.70
MSE
84.91
74.40
79.31
Max
84.69
75.12
79.62
CaFo
16min14s
72.46
KL
85.29
77.62
81.27
LogSumExp
85.29
77.62
81.27
TransAgent
2min35s
73,50
B Additional Experiments
Cross-Dataset Evaluation. We compare the cross-dataset performance Table On the source dataset, TransAgent achieves better performance than previous few-shot prompt learning methods under the same training settings. We credit the fine-fitted result to the ImageNet pre-trained vision experts, which transfer their knowledge of the source dataset during the training process. More surprisingly, our method does not seem to overfit on the source dataset. still presents competitive results against MaPLe, which is the SOTA few-shot method cross-dataset generalization, leading to an overall improvement over the previous methods.
Domain Generalization. Table 10 presents the results domain generalization. TransAgent outperforms previous few-shot prompt learning methods with the highest average accuracy. This suggests that our method improves the robustness of VLMs against disttiibbitinnonnnn data.
C Additional Ablative A Analysis
MAC loss type. We scrutinize the loss function used to compute the distillation loss Eq, 8 As shown in Table [T] using I.1 loss or MSE loss underperforms using KI. loss by large margin. Such channel-wise loss is incompatible with the form of the score vectors, which are per-class probability distributions, and using such loss makes it hard to converge. On the contrary, KL loss provides smoother training due to the soft matching between the probability distributions (i.., the score vectors).
Pooling type. In Table 12 we ablate the pooling strategy to aggregate the cross attention value from T2I agents to obtain the score vectors. The pooling operations are performed on the spatial dimension of the cross attention maps, which results in the semantic logits of certain image over all the candidate categories. As is shown in Table 12] average and max pooling lag behind since they introduce information loss during the operations. However, LogSumExp (LSE) pooling yields the best performance as it is more robust and provides more accurate matching scores (34).
17

---

EuroSAT
FGVCArcrattt
Food101
011
66
Oude
Oyde
Co0g Imzsgges
Shott
Shots
Shots
Caltech101
OxfordPets
UCFID1
Accurszy
Orde
Orde
C40g Smasgge
Shots
StandFardCars
Flawers102
DTD
803
Accuraay
Acpunity
Culp TrEDAAgs
CuOs Tranwkger
Gedp
010
Figure 6: Variance and performance of TransAgeet compared with CoOp. TransAgent demon- strates better robustness and outperforms CoOp on most low-shot cases.
Deployment efficiency. Table 13 presents the comparison of inference time and performance evaluated the test set of ImageNet (21). All models adopt Vision Transformer (24) backbone. Our method demonstrates superior deployment efficiency compared to CaFo [8] which adopts
cumbersome model ensemble. To be mentioned, the learnable prompts in the training phase are
frozen during the inference stage, which carries the knowledge of heterogeneous agents enhance
CLIP's generalization ability. The inference time of our TransAgent is little behind the zero-shot CLIP, but still achieves a good accuracy-efficiency tradeoff.
D Additional Visualization
To verify the robustness of TransAgent, we run few-shot classification experiments over seeds and visualize the variance datasets in Figure ( and SUN397 excluded since the variance is small for both methods.). We can observe that TransAgent consistently outperforms CoOp in robustness and excels in most low-shot scenarios.
E
Examples of Generated Descriptions from Chatbots
We interact with the language agents (i.e., chatbots) described in Section 3.2 to obtain rich
descriptions of the categories in the target domain, following [42]. To further unleash the linguistic potential of these language agents, we gather their knowledge adaptively via the MoA gating mechanism. In fact, these language agents give descriptions in different styles with their respective unique understanding. Below show examples of the descriptions given by different chatbots for certain classes to demonstrate their characteristics.
Class: Pomeranian (OxfordPets).
18

---

GPT-3: "A pet Pomeranian typically has a thick coat of fur that can be either straight or curly."
Vicuna: "The image shows small dog with fluffy white coat. The dog has round face with big brown eyes and a black nose. Its ears are small and pointy, and its tail is curled up over its back."
Class: Industrial Buildings (EuroSAT).
GPT-3: "A satellite photo of industrial buildings might look like patchwork of utilitarian structures."
Vicuna: "The image shows large industrial building with a flat roof and several windows on the front facade. The building is made of concrete and has gray color. The roof is also gray and appears to be made of similar material."
Class: Club Sandwich (O1)
GPT-3: "A club sandwich is sandwich made with ham, turkey, bacon, cheese, and tomato, with lettuce in between slices of bread."
Vicuna: "The image shows golden brown club sandwich, with thick slice of bread on the bottom, and two thinner slices on top. The bread is toasted and has a crispy texture."
Class: Office Building (SUN397).
GPT-3: "An office building is usually a tall building with many floors."
Vicuna: "An office building is structure that is designed for business-related activities. typically has multiple floors and a variety of rooms and spaces to accommodate different functions, such as open-plan offices, meeting rooms, and private offices."
Class: 747-200 (FGV ..Crerra.........
GPT-3: "The 747-200 is jumbo jet produced by Boeing."
Vicuna: "The Boeing 747-200 is large, wide-body commercial airliner that was first introduced in the 1970s."
Class: 2012 Beereddes-eezz E-Class Sedan (StandfordCars).
GPT-3: "The Mercedes-Bent E-Class Sedan 2012 can identified by its distinctive grille, sleek headlights, and classic Mercedes-Benz. logo."
Vicuna: "The 2012 Mercedes-Benz E-Class Sedan luxury car that is known for its sleek and sophisticated appearance."
19

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract synoptically demonstrates our proposed method, which aims to transfer VL foundation models with heterogeneous agent collaboration. The introduction section further claims the background and our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
20

---

Justification:
We discuss the potential limitations of our proposed method in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer:
[NA]
Justification:
The results of the paper are mainly experimental.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The implementation details are provided in the main paper (Section 4 as well in the supplementary (Appendix A)
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., s., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The training and evaluation codes are released at /tttts:///itie com/ markywg/ haagyygg/rrasaaggg transagent] with detailed instructions on how to download data and reproduce the experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ttpps.cc/ubllc//uuddee htw.a//iicc/iddee/ddeeeeuuiiiiii /Ct////ooeeeeeaeeaaaeee ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
22

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The training details and hyperparameters setting are listed in Appendix A
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core paper to a level detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Since all experiments are run over different seeds, we provide the variance figures in the supplementary (Appendix D) to display the robustness of proposed method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: As stated in Appendix A, all experiments are conducted using a Nvidia A6000 GPU. The memory and training time required each dataset are presented in Table
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We conduct the research project following NeurlPS Code Ethics faithfully.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
The work does not seem to raise any (negative) societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]

---

Justification:
Our method does not involve data or models that might lead to intended misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: have made acknowledgements to the assets used our work.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL..
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, thhors thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: We heve prepared the documentation of codes for future reproduction, which released alongside the codes at tattps://githen com hatmm//aaayyttttaaaaggggge
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
25

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "WGLwN7LnyH", "reviewer": "Reviewer_QaNB", "summary": "This paper introduces a TransAgent framework, which guides CLIP to generalize with multi-source knowledge distillation. The framework contains three kinds of collaboration, including vision models, language models and muti-modal models. A Mixture-of-Agents (MoA) gating mechanism is proposed to adaptively integrate the knowledge. SOTA is achieved on 11 datasets under the few-shot scenarios.", "questions": "11 agents contain different domain knowledges. Irrelevant information or inconsistent information may be induced. Is there any challenge in the training and how to deal with challenges?\nHow to prevent overfitting when distilled from 11 pretrained models? \nThe λ_2 is far greater than other λ values, does it mean that the loss of LAC is more important than other losses?", "limitations": "The authors have listed the limitation in the paper. It is encouraged to provide more analysis and solutions.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This method guarantees the deployment efficiency, as knowledge from heterogeneous agents are distillated and injected into CLIP.\nThis framework can be extended flexibly. Any expert can be introduced as a teacher model. \nThe paper is easy to follow. Figures and experiments are clearly presented.", "weaknesses": "The method applies multi-teacher distillation in clip with prompts, and popular large models are exploited as teachers. Thus, the novelty is limited. \nGiven that the framework comprises three components — VAC, LAC, and MAC—the reviewer insists that the  results of every module (only-VAC, only-LAC, only-MAC) will be more persuasive to demonstrate their respective effectiveness (i.e. only-VAC's, only-LAC's, only-MAC's).\nThe clip absorbs various knowledge from 11 excellent agents. In addition to few-shot learning on 11 downstream datasets, experiments on general dataset are also encouraged, like WebQA and CIRR.\nIn the related works, research on muti-teacher distillation should be included. Baselines in the experiments should be introduced in the related work.\nTypos in line 259: knowledege -> knowledge", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments. We provide our feedbacks as follows.\n\n**Q1: The method applies multi-teacher distillation in clip with prompts, and popular large models are exploited as teachers. Thus, the novelty is limited.**\n\n**A1:** To be noted, how to perform multi-source distillation is not trivial, since how to **extract and collaborate** knowledge from various heterogeneous teachers has not been explored for CLIP-like foundation models. We would like to emphasize the distinct technical novelty and contributions of our approach, specifically in the context of *Transfer Flexibility* (Introduction, Lines 49-55).\n\n(1) **Generic Knowledge Extraction Method:** Our method introduces a novel approach for extracting knowledge from heterogeneous agents, particularly the multi-modal ones. We design a unique method to extract prediction score vectors as multi-modal knowledge. This involves elaborate mining of vision-language alignment within these models. The efficacy is demonstrated in Table 4 (Page 9).\n\n(2) **Flexible Knowledge Collaboration Mechanism:** We propose a mixture-of-agent gating mechanism to integrate external knowledge from different agents. This mechanism allows TransAgent to dynamically select agents via soft weighting, enabling it to adapt effectively to few-shot settings across various target datasets. The effectiveness is detailed in Table 2-4 (Page 9).\n\nBeyond these technical innovations, TransAgent offers two additional significant advantages. (1) *Knowledge Versatility* (Line 45-48): To our best knowledge, TransAgent is **the first framework** to enhance CLIP-like models comprehensively using 11 heterogeneous agents, covering a wide range of vision, language, and multi-modal research areas. (2) *Deployment Efficiency* (Line 56-60):\nTransAgent allows all external agents to be unloaded after distillation. This guarantees the inference pipeline of the enhanced CLIP to be consistent with the original one, without the need for a cumbersome model ensemble, as evidenced in Table 10 (Page 16).\n\n**Q2: The results of every module (only-VAC, only-LAC, only-MAC) will be more persuasive to demonstrate their respective effectiveness.**\n\n**A2:** The ablation is conducted below. As observed in the table, each module is effective and their combination further boosts the overall performance on 11 benchmarks (the same setting as Table 1 in the paper).\n\n|Module|Base Acc.|Novel Acc.|HM|\n|:-|:-:|:-:|:-:|\n|baseline|84.21|71.79|77.51|\n|VAC|84.96|73.90|79.04|\n|LAC|85.23|75.20|79.90|\n|MAC|85.01|74.85|79.61|\n|TransAgent|85.29|77.62|81.27|\n\n**Q3: In addition to few-shot learning on 11 downstream datasets, experiments on general dataset are also encouraged, like WebQA and CIRR.**\n\n**A3:** Note that, these 11 downstream datasets have been widely used to evaluate the generalization capacity of foundation models [1-4], since they are actually the general datasets arranging from image to video task, from natural to satellite images,\nfrom object to scene understanding, from regular to fine-grained recognition. Hence, we follow the mainstream setting for evaluation.\n\n**Q4: In the related works, research on muti-teacher distillation should be included. Baselines in the experiments should be introduced \nin the related work. Typos in line 259: knowledege -> knowledge.**\n\n**A4:** Thanks for your suggestions on related works. We will include baseline methods and research on multi-teacher distillation [5-7] in the related work in our final version. Additionally, we will fix the typos in the revision.\n\n**Q5: 11 agents contain different domain knowledge. \nIrrelevant information or inconsistent information may be induced. \n(1) Is there any challenge in the training and how to deal with challenges? \n(2) How to prevent overfitting when distilled from 11 pretrained models? \n(3) The \\\\(\\lambda_2\\\\) is far greater than other \\\\(\\lambda\\\\) values, does it mean that the loss of LAC is more important than other losses?**\n\n**A5:** Thanks for raising the concerns.\n\n(1) The primary challenge lies in the varying importance of the 11 agents for different target datasets. Treating all the knowledge equally would indeed hinder generalization. To address this, we implement a **mixture-of-agent gating mechanism** in the few-shot training. This mechanism enables us to adaptively weight the contributions of the agents for different target datasets. The visualization is provided in Figure 5, and its effectiveness is demonstrated at the bottom of Tables 2-4 (Page 9).\n\n(2) We mitigate the risk of overfitting, since all pre-trained models (including CLIP and 11 agents) are frozen during the training process. Fine-tuning primarily works on a few learnable prompts and the gating mechanism, which involves minimal adjustments (conducted over 20 epochs).\n\n(3) The main reason is that, the absolute value of LAC loss is smaller than other losses. To ensure that the LAC loss contributes effectively to the training process, we choose a greater \\\\(\\lambda_2\\\\). This adjustment ensures that the weighted values of all the losses are comparable, and be effective when training our TransAgent. We ablate the value of \\\\(\\lambda_2\\\\) below.\n\n|\\\\(\\lambda_2\\\\)|Base|Novel|HM|\n|:-:|:-:|:-:|:-:|\n|1.0|84.89|74.36|79.28|\n|10.0|84.75|75.47|79.84|\n|20.0|84.97|76.50|80.51|\n|25.0|85.29|77.62|81.27|\n|30.0|85.15|77.31|81.04|\n\n**References.**\n\n[1] Learning to prompt for vision-language models. (IJCV-22)\n\n[2] Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. (CVPR-23)\n\n[3] Maple: Multi-modal prompt learning. (CVPR-23)\n\n[4] Self-regulating prompts: Foundational model adaptation without forgetting. (ICCV-23)\n\n[5] Mitigating Accuracy-Robustness Trade-Off Via Balanced Multi-Teacher Adversarial Distillation. (TPAMI-24)\n\n[6] Let All be Whitened: Multi-teacher Distillation for Efficient Visual Retrieval. (AAAI-24)\n\n[7] AM-RADIO: Agglomerative Vision Foundation Model Reduce All Domains Into One. (CVPR-24)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dIuss3maQ5", "reviewer": "Reviewer_gY7M", "summary": "This paper aims to handle heterogeneous foundation model combination across different pretrained backbones. Instead of using vanilla ensemble, it proposes to use a distillation process to transfer knowledges from different agents. More specifically, it uses a learnable gate module to integrate different knowledge sources. Extensive experiments show the propose method outperforms other related models.", "questions": "See the weaknesses section above.", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Efficiently combining several pretrained foundation models is a valuable research direction.\n2. Overall, the writing is clear to read and follow, such as motivation, methodology, and empirical results.\n3. Compared with baseline, the proposed model does not need to involve extra inference cost, benefitted by a distillation process.\n4. Empirical results show the model superiority.", "weaknesses": "1. Overall, I think this is more like a jointly distillation technique, instead of so-called agent collaboration, which may lead to certain misunderstanding.\n2. Some figures are a little confused to understand. A clean and informative figures to show the whole working pipeline will be helpful.\n3. Even if the empirical results are good, the proposed method still lacks of research novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments. We provide our feedbacks as follows.\n\n**Q1: The proposed method is more like a jointly distillation technique, instead of so-called agent collaboration, which may lead to certain misunderstanding.**\n\n**A1:** We would like to clarify this misunderstanding. Joint distillation is our goal, while agent collaboration is our method to achieve this goal. Specifically, we introduce a generic gating mechanism for each modality (outlined in Equation 3, 5 and 7). This mechanism allows us to flexibly weight the contributions of various teacher models, thereby creating a summarized knowledge vector that is subsequently used for distillation within the given modality. We describe this process as **\"agent collaboration\"** because it involves all the agents (models) working together adaptively towards a shared objective, i.e., constructing a knowledge vector for effective distillation. This collaborative aspect distinguishes our method.\nWe will clarify this in the revision.\n\n**Q2: Some figures are a little confused to understand. A clean and informative figures to show the whole working pipeline will be helpful.**\n\n**A2:** Thanks for your suggestions. Figure 1 actually shows the whole pipeline of our TransAgent, where\nwe extract external knowledge from heterogeneous models in each modality and leverage the knowledge of all the modalities to boost vision-language foundation models. The current figures mainly focuse on \"what\" knowledge are integrated. We will add more informative figures to show \"how\" to integrate these knowledge in the revision.\n\n**Q3: Even if the empirical results are good, the proposed method still lacks of research novelty.**\n\n**A3:** To be noted,\nhow to perform multi-source distillation is not trivial, since how to **extract and collaborate** knowledge from various heterogeneous teachers has not been explored for CLIP-like foundation models.\nWe would like to emphasize the distinct technical novelty and contributions of our approach, specifically in the context of *Transfer Flexibility* (Introduction, Lines 49-55).\n\n(1) **Generic Knowledge Extraction Method:**\nOur method introduces a novel approach for extracting knowledge from heterogeneous agents, particularly the multi-modal ones. We design a unique method to extract prediction score vectors as multi-modal knowledge. This involves elaborate mining of vision-language alignment within these models. The efficacy is demonstrated in Table 4 (Page 9).\n\n(2) **Flexible Knowledge Collaboration Mechanism:**\nWe propose a mixture-of-agent gating mechanism to integrate external knowledge from different agents. This mechanism allows TransAgent to dynamically select agents via soft weighting, enabling it to adapt effectively to few-shot settings across various target datasets. The effectiveness is detailed in Table 2-4 (Page 9).\n\nBeyond these technical innovations, TransAgent offers two additional significant advantages. (1) *Knowledge Versatility* (Line 45-48):\nTo our best knowledge, TransAgent is **the first framework** to enhance CLIP-like models comprehensively using 11 heterogeneous agents, covering a wide range of vision, language, and multi-modal research areas. (2) *Deployment Efficiency* (Line 56-60):\nTransAgent allows all external agents to be unloaded after distillation. This guarantees the inference pipeline of the enhanced CLIP to be consistent with the original one, without the need for a cumbersome model ensemble, as evidenced in Table 10 (Page 16).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lGG1rcRIWd", "reviewer": "Reviewer_wEnm", "summary": "The paper focuses on the challenge of vision-language foundation models (e.g., CLIP) struggling to generalize to diverse target domain data in downstream tasks. It highlights the potential of using expert models, which are pre-trained on various modalities, tasks, networks, and datasets, to improve generalization. The proposed TransAgent framework integrates the knowledge of these isolated expert models through a unified approach, enhancing CLIP's performance via multi-source knowledge distillation. TransAgent collaborates with 11 heterogeneous agents to empower vision-language models without adding inference phase costs. The framework achieves good performance on some visual recognition datasets.", "questions": "Please refer to 'Weaknesses'.", "limitations": "The authors have addressed the limitations and potential negative societal impacts of their work.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The problem is of practical importance.\n2. The experiments are sufficient.\n3. The gains of knowledge distillation seem strong.", "weaknesses": "1. In my opinion, the proposed method leverages multiple pretty strong 'experts' for knowledge distillation, e.g., SAM, MAE, DINO, ViTDet, GPT, and Vicuna. However, almost all the baselines to be compared do not rely on external models. Hence, I think the majority of comparisons (e.g., Table 1 and Figure 4) may be unfair. The proper baselines should be other distillation methods (using the same external models as this paper), which the proposed method is actually orthogonal to most of the current baselines. \n\n2. On top of 1, I think that if some significant technical contributions regarding knowledge distillation are not proposed, simply 'applying knowledge distillation methods to CLIP' may not be an acceptable novel contribution for me.\n\n3. Given that multiple pretty strong 'experts' (e.g., SAM, MAE, DINO, ViTDet, GPT, and Vicuna) for knowledge distillation have been employed, the current gains of performance seem limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments. We provide our feedbacks as follows.\n\n**Q1: Almost all the baselines to be compared do not rely on external models. Hence, I think the majority of comparisons (e.g., Table 1 and Figure 4) may be unfair.**\n\n**A1:** Thank you for raising the concern.\n\n(1) Table 1 and Figure 4 are provided for the state-of-the-art (SOTA) comparison, rather than an ablation study focusing solely on the distillation method. Our aim is to identify **the most effective method**, regardless of the specific techniques employed. The results demonstrate that TransAgent achieves superior performance, highlighting its advancements in few-shot generalization.\n\n(2) During the inference phase, we unload all external models, ensuring that the inference process of TransAgent is identical to that of the original CLIP with learnable prompts. Therefore, it is fair to compare our method with others using learnable prompts, as they share **the same inference setting**, as presented in Table 1.\n\n(3) Our TransAgent also outperforms CaFo, which similarly relies on external models, as shown in Figure 4. This comparison further underscores the effectiveness and fairness of our approach.\n\n**Q2: The proper baselines should be other distillation methods (using the same external models as this paper), which the proposed method is actually orthogonal to most of the current baselines.**\n\n**A2:** Thanks for your insightful feedback.\n\n(1) We have included baselines that involve distillation with external models. As indicated in the second row of Tables 2-4 (Page 9), the use of distillation from each external model contributes to performance improvement. Our proposed heterogeneous agent collaboration (the Gating results shown in Tables 2-4) consistently achieves the best results. This fairly demonstrates the effectiveness of our method in leveraging multiple external models.\n\n(2) For the other distillation method in the research of CLIP [PromptKD, CVPR2024], it is not feasible to use the exact same set of external models as in our study due to differences in frameworks and settings. Specifically, the referenced method employs a transductive setting, which differs from our approach. Nonetheless, we have included this method as a state-of-the-art (SOTA) baseline for comparison.\nFor instance, on the EuroSAT benchmark, which involves a **significant domain shift** with satellite images, our method achieves an accuracy of 83.43 on novel classes, compared to 82.08 for PromptKD. Notably, we achieve this with only 16-shot labeled data in the base class, whereas PromptKD utilizes 16-shot labeled data in the base class and the full training set without labels in the base and novel classes. This demonstrates our method's superior performance.\n\n**Q3: If some significant technical contributions regarding knowledge distillation are not proposed, simply 'applying knowledge distillation methods to CLIP' may not be an acceptable novel contribution.**\n\n**A3:** To be noted, how to perform multi-source distillation is not trivial, since how to **extract and collaborate** knowledge from various heterogeneous teachers has not been explored for CLIP-like foundation models.\nWe would like to emphasize the distinct technical novelty and contributions of our approach, specifically in the context of *Transfer Flexibility* (Introduction, Lines 49-55).\n\n(1) **Generic Knowledge Extraction Method:**\nOur method introduces a novel approach for extracting knowledge from heterogeneous agents, particularly the multi-modal ones. We design a unique method to extract prediction score vectors as multi-modal knowledge. This involves elaborate mining of vision-language alignment within these models. The efficacy is demonstrated in Table 4 (Page 9).\n\n(2) **Flexible Knowledge Collaboration Mechanism:**\nWe propose a mixture-of-agent gating mechanism to integrate external knowledge from different agents. This mechanism allows TransAgent to dynamically select agents via soft weighting, enabling it to adapt effectively to few-shot settings across various target datasets. The effectiveness is detailed in Table 2-4 (Page 9).\n\nBeyond these technical innovations, TransAgent offers two additional significant advantages.\n(1) *Knowledge Versatility* (Line 45-48):\nTo our best knowledge,\nTransAgent is **the first framework** to enhance CLIP-like models comprehensively using 11 heterogeneous agents, covering a wide range of vision, language, and multi-modal research areas.\n(2) *Deployment Efficiency* (Line 56-60):\nTransAgent allows all external agents to be unloaded after distillation. This guarantees the inference pipeline of the enhanced CLIP to be consistent with the original one, without the need for a cumbersome model ensemble, as evidenced in Table 10 (Page 16).\n\n**Q4: Given that multiple pretty strong `experts' for knowledge distillation have been employed, the current gains of performance seem limited.**\n\n**A4:** Thank you for raising this concern. The performance gains achieved by our TransAgent are indeed significant, as highlighted by the following observations. *First*, TransAgent operates similarly to CoOp [2] during inference, utilizing learnable prompts while unloading all external models.\nWith the same inference pipeline, TransAgent significantly outperforms the popular CoOp by around **10%** on average, and by as much as **20%** on the EuroSAT dataset which contains large domain shifts, under the same low-shot setting in Table 1. *Second*, when compared to CaFo [3], which also utilizes external models, TransAgent achieves approximately **5%** higher accuracy on average, while using only **1/10** of the inference time as shown in Table 10.\n\n**References.**\n\n[1] PromptKD: Unsupervised Prompt Distillation for Vision-Language Models (CVPR-24)\n\n[2] Learning to prompt for vision-language models. (IJCV-22)\n\n[3] Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. (CVPR-23)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ak0gxvrtR0", "reviewer": "Reviewer_A3Ab", "summary": "The paper introduces TransAgent, a novel framework designed to enhance vision-language foundation models like CLIP through the integration of knowledge from diverse, pre-trained expert models. These experts, which include vision, language, and multi-modal models, possess rich knowledge acquired from different modalities, tasks, networks, and datasets. TransAgent addresses the challenge of generalizing these models to new domains, especially under low-shot conditions, by proposing a unified transfer framework that leverages heterogeneous agent collaboration. The framework employs a mixture-of-agents gating mechanism to adaptively integrate external knowledge and uses multi-source distillation to transfer this knowledge into CLIP, enhancing its generalization ability without additional inference cost. Experiments demonstrate TransAgent's state-of-the-art performance on 11 visual recognition datasets, significantly outperforming methods like CoOp under low-shot settings and showing remarkable results on EuroSAT with large domain shifts. The paper's contributions lie in its innovative approach to knowledge transfer, the flexibility of its framework, and its significant improvements in model generalizability and efficiency.", "questions": "Could the authors provide insights into how the TransAgent framework generalizes to domains outside of the tested benchmarks, especially those with significantly different characteristics?\n\nThe paper mentions the potential introduction of irrelevant knowledge during distillation. What strategies are in place or could be considered to mitigate this issue?\n\nThe role of learnable prompts seems pivotal. How were the prompts engineered, and what impact did they have on the model's performance?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents a highly original approach to enhancing vision-language foundation models by proposing the TransAgent framework. This framework innovatively integrates knowledge from a diverse set of pre-trained expert models into a unified system. \n\nThe use of a mixture-of-agents gating mechanism and multi-source distillation is a unique contribution that sets this work apart from existing methods.\n\nThe authors have conducted extensive experiments on 11 visual recognition datasets, demonstrating the effectiveness of TransAgent under various low-shot scenarios. The comparison with state-of-the-art methods like CoOp and the detailed ablation studies provide a solid understanding of the framework's capabilities and the impact of different components. \n\nThe paper is well-structured, with a clear abstract and introduction that succinctly summarize the contributions and scope of the work.", "weaknesses": "After changing the expert agent selection, the model needs to be retrained, and this structure is not plug-and-play.\n\nThe use of integrated features from multiple teacher models as supervision for distillation is widely used in other fields, and the authors have transferred this approach to the domain of this paper, which lacks sufficient novelty", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments. We provide our feedbacks as follows.\n\n**Q1: After changing the expert agent selection, the model needs to be retrained, and this structure is not plug-and-play.**\n\n**A1:** We would like to clarify that the training effort required by our framework is minimal. All pre-trained models (including CLIP and the 11 agents) are frozen. Fine-tuning is performed on the learnable prompts and the gating mechanism, which requires only a small amount of computational effort. For example, under the 16-shot setting, training for 20 epochs on a single A6000 GPU takes less than 20 minutes for most benchmarks.\nThis efficient process means that our framework can be **quickly adapted** to changes in expert agent selection or the testing of new downstream datasets. \nAdditionally,\nour method is actually **'plug-and-play'**, \ni.e.,\nthis concise collaboration framework can be integrated into various prompt-learning based methods with ease.\nWe will further investigate this in future work.\n\n**Q2: The authors have transferred the distillation approach to the domain of this paper, which lacks sufficient novelty.**\n\n**A2:** To be noted,\nhow to perform multi-source distillation is not trivial, since how to **extract and collaborate** knowledge from various heterogeneous teachers has not been explored for CLIP-like foundation models.\nWe would like to emphasize the distinct technical novelty and contributions of our approach, specifically in the context of *Transfer Flexibility* (Introduction, Lines 49-55).\n\n(1) **Generic Knowledge Extraction Method:**\nOur method introduces a novel approach for extracting knowledge from heterogeneous agents, particularly the multi-modal ones. We design a unique method to extract prediction score vectors as multi-modal knowledge. This involves elaborate mining of vision-language alignment within these models. The efficacy is demonstrated in Table 4 (Page 9).\n\n(2) **Flexible Knowledge Collaboration Mechanism:**\nWe propose a mixture-of-agent gating mechanism to integrate external knowledge from different agents. This mechanism allows TransAgent to dynamically select agents via soft weighting, enabling it to adapt effectively to few-shot settings across various target datasets. The effectiveness is detailed in Table 2-4 (Page 9).\n\nBeyond these technical innovations, TransAgent offers two additional significant advantages.\n(1) *Knowledge Versatility* (Line 45-48):\nTo our best knowledge,\nTransAgent is **the first framework** to enhance CLIP-like models comprehensively using 11 heterogeneous agents, covering a wide range of vision, language, and multi-modal research areas.\n(2) *Deployment Efficiency* (Line 56-60):\nTransAgent allows all external agents to be unloaded after distillation. This guarantees the inference pipeline of the enhanced CLIP to be consistent with the original one, without the need for a cumbersome model ensemble, as evidenced in Table 10 (Page 16).\n\n**Q3: Could the authors provide insights into how the TransAgent framework generalizes to domains outside of the tested benchmarks, especially those with significantly different characteristics?**\n\n**A3:** The generalization capacity of our TransAgent framework is primarily credited to the **integration of diversified knowledge** from a wide range of heterogeneous agents that are pre-trained on different modalities, tasks, networks, and datasets.\nTo substantiate this claim, we follow [1-5] to verify TransAgent on 11 downstream datasets spanning diverse domains arranging from image to video task, from natural to satellite images, from object to scene understanding, from regular to fine-grained recognition. The remarkable performance across these datasets demonstrates that TransAgent can **generalize effectively**, even when faced with **significant domain shifts**.\nWe will investigate more diverse domain data in future work.\n\n**Q4: The paper mentions the potential introduction of irrelevant knowledge during distillation. What strategies are in place or could be considered to mitigate this issue?**\n\n**A4:** Thanks for your concern.\nTo mitigate the introduction of irrelevant knowledge during distillation, we employ a **mixture-of-agent gating mechanism** (outlined in Equation 3, 5, and 7). This mechanism enables our TransAgent to adaptively select and weight the knowledge from different agents, thereby reducing the impact of irrelevant information.\nThe visualization of this process is provided in Figure 5, and the effectiveness of this design is demonstrated in the bottom sections of Tables 2-4 (Page 9). This adaptive selection ensures that only the most relevant knowledge is distilled, enhancing the overall performance and robustness of the model.\n\n**Q5: The role of learnable prompts seems pivotal. How were the prompts engineered, and what impact did they have on the model's performance?**\n\n**A5:** To avoid the complex design of prompt engineering, we simply leverage the widely-used CoOP [1] method with a number of learnable prompt vectors, as mentioned in Lines 102-110 in the paper. Our method works well with such a simple prompting method, showing its effectiveness and flexibility. The importance of the learnable prompts as well as the prompt design have been studied thoroughly by previous works [3, 4, 5], and we simply follow the de-facto standard.\n\n**References.**\n\n[1] Learning to prompt for vision-language models. (IJCV-22)\n\n[2] Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. (CVPR-23)\n\n[3] Maple: Multi-modal prompt learning. (CVPR-23)\n\n[4] Self-regulating prompts: Foundational model adaptation without forgetting. (ICCV-23)\n\n[5] Read-only prompt optimization for vision-language few-shot learning. (ICCV-23)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9XYmJGqjAq", "reviewer": "Reviewer_DUuV", "summary": "This paper proposes the TransAgent framework, which unifies and transports knowledge from isolated agents to guide CLIP in generalizing through multi-source knowledge distillation. This framework allows flexible collaboration with 11 heterogeneous agents to enhance vision-language foundation models. Importantly, this approach incurs no additional cost during the inference phase.", "questions": "Please refer to the weakness section.", "limitations": "Yes, they have provided the limitation section.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**[Interesting idea]** Leveraging heterogeneous agent collaboration for generalized vision language models is intriguing.\n\n**[Good presentation]** The paper is well-written and organized, which is easy to follow.\n\n**[Well-illustrated figures]** The figures shown in this paper are clear enough to tell the workflow of the method.", "weaknesses": "**[Overclaimed statements]** The paper mentions that “it is the first unified transfer framework for generalizing vision-language foundation models with heterogeneous agent collaboration.” However, CaFo [67] had done similar things by adopting multiple heterogeneous agents. This statement may need a revision to differentiate from CaFo.\n\n**[Some SOTA methods are not compared]** Some works to be compared or discussed in related works are listed as follow: “PromptKD: Unsupervised Prompt Distillation for Vision-Language Models, CVPR2024”, “DePT: Decoupled Prompt Tuning, CVPR2024”, “Beyond Sole Strength: Customized Ensembles for Generalized Vision-Language Models, ICML2024”, “Consistency-guided Prompt Learning for Vision-Language Models, ICLR2024”, “GraphAdapter: Tuning Vision-Language Models with Dual Knowledge Graph, NeurIPS2023” and “Task Residual for Tuning Vision-Language Models, CVPR2023”.\n\n**[Need further explorations]** The knowledge distillation technique used in this work is too straightforward. It would be nice to see an advanced distillation with heterogeneous agents.\n\n**[Inadequate experiments]** There are some experiments to be done for verifying the effectiveness of the proposed method. (i) Training time comparisons with other methods and more comparisons for inference time in Table 10; (ii) It would be nice to see the performance by increasing each agent gradually. This can help better understand which agent is more effective and which one is useless. (iii) Ablation study conducted using VAC, LAC, or MAC individually, as well as in combinations of any two.\n\n**[Disorganized reference format]** Please reformat the references, e.g., “Conference on Computer Vision and Pattern Recognition”, “Proceedings of the IEEE/CVF conference on computer vision and pattern recognition” and “Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments. We provide our feedbacks as follows.\n\n**Q1: The paper mentions that “it is the first unified transfer framework for generalizing vision-language foundation models with heterogeneous agent collaboration.” However, CaFo [67] had done similar things by adopting multiple heterogeneous agents. This statement may need a revision to differentiate from CaFo.**\n\n**A1:** Thanks for your suggestion.\nCaFo [67] leverages cascades of external models based on their output forms, which necessitates careful sequential arrangement of the models. This design inherently lacks **flexibility** due to the required model arrangement.\nIn contrast, our TransAgent framework avoids this issue by employing a unified distillation process for heterogeneous agents, eliminating the complexity of sequential model arrangement. Additionally, TransAgent unloads all external models during inference, achieving **deployment efficiency** without the burden of a heavy model ensemble as seen with CaFo.\nBased on these distinctions, we will revise the statement to: “it is the first unified *distillation* framework for generalizing vision-language foundation models with *efficient* heterogeneous agent collaboration.”\n\n**Q2: Some works to be compared or discussed in related works are listed as follow: \nPromptKD (CVPR2024), \nDePT (CVPR2024), \nT\\\\(_{En}\\\\) (ICML2024), \nCoPrompt (ICLR2024), \nGraphAdapter (NeurIPS2023),\nTask Residual(CVPR2023).**\n\n**A2:** Thanks for this suggestion.\nWe list the comparison on base-to-novel generalization of 11 benchmarks.\n(1) **PromptKD (CVPR2024)** also leverages distillation but uses the full training set of base and novel classes without labels. To align with the mainstream setting, we re-implemented its official code using only 16-shot labeled data. Our TransAgent clearly outperforms PromptKD under these conditions.\n(2) **T\\\\(_{En}\\\\) (ICML2024)** employs a model ensemble that must be maintained during inference, leading to inefficient deployment. In contrast, our TransAgent unloads all external models after training, with the inference pipeline identical to CLIP. This results in better performance with only **1/3** complexity during inference compared to **T\\\\(_{En}\\\\)**.\n(3) Compared to other prompt-learning based methods such as **DePT (CVPR2024)** and **CoPrompt (ICLR2024)**,\nour TransAgent simply achieves better performance.\n(4) For adapter-based methods such as **GraphAdapter (NeurIPS2023)** and **Task Residual(CVPR2023)**, we will refer them in the related works since the model and data settings are different.\n\n|Method|Training Data|Inference Model|Base Acc.|Novel Acc.|HM|\n|:-|:-|:-:|:-:|:-:|:-:|\n|PromptKD (CVPR2024)|16-shot + unlabeled|87M|86.96|80.73|83.73|\n|PromptKD (CVPR2024)|16-shot|87M|77.75|71.69|74.59|\n|T\\\\(_{En}\\\\) (ICML2024)|16-shot|268M|85.48|77.17|81.11|\n|DePT (CVPR2024)|16-shot|86M|85.19|76.17|80.43|\n|CoPrompt (ICLR2024)|16-shot|91M|84.00|77.23|80.48|\n|TransAgent (ours)|16-shot|86M|**85.29**|**77.62**|**81.27**|\n\n**Q3: It would be nice to see an advanced distillation with heterogeneous agents.**\n\n**A3:** Thanks for your insightful comments.\nTo be noted,\nour primary goal is to introduce a **generic knowledge integration** method to enhance CLIP using heterogeneous agents. The main contribution lies in how we collaborate these heterogeneous agents to generate the knowledge vector for distillation, rather than the distillation operation itself.\nSpecifically, we propose a novel **mixture-of-agent gating mechanism** (outlined in Equation 3, 5, and 7) for collaborating heterogeneous agents in each modality. This mechanism allows TransAgent to adaptively select and weight the knowledge from various agents, effectively reducing the introduction of irrelevant knowledge during distillation.\nThe visualization of this process is provided in Figure 5, and the effectiveness of this design is demonstrated in the bottom sections of Tables 2-4 (Page 9). This innovative approach represents the advancement in our collaboration method.\n\n**Q4: More Experiments \n(i) Training time comparisons with other methods and more comparisons for inference time; \n(ii) It would be nice to see the performance by increasing each agent gradually. \n(iii) Ablation study conducted using VAC, LAC, or MAC individually, as well as in combinations of any two.**\n\n**A4:** (i) Since PromptKD (CVPR2024) also leverages distillation. We further make the comparison with it. The cost comparison on ImageNet is presented below, where all the models are trained for 5 epochs under 16-shot setting on a single A6000 GPU.\nClearly, our model achieves a better performance with higher efficiency.\n|Method|Training Time (min)|Inference Time (sec)|HM|\n|:-|:-:|:-:|:-:|\n|PromptKD|146|201|71.29|\n|TransAgent|72|54|73.93|\n\n(ii) Note that we have already conducted the experiment of **whether use** a specific agent can help improve the performance in the second row of Table 2-4. The visualization of the gating weights in Figure 5 may also provide insights of which agent is more useful for a specific dataset. For instance, as for vision agents, DINO experts in recognizing general objects while lags behind on some fine-grained datasets (e.g., EuroSAT), where the other agents like SAM which focus more on details perform better.\n\n(iii) The results are presented below. As observed, each individual module is beneficial for improving the generalization ability of the foundation models. And the combinations of these modules further boost the results.\n\n|Module|Base Acc.|Novel Acc.|HM|\\\\(\\Delta\\\\)|\n|:-|:-:|:-:|:-:|:-:|\n|baseline|84.21|71.79|77.51|-|\n|VAC|84.96|73.90|79.04|+1.53|\n|LAC|85.23|75.20|79.90|+2.39|\n|MAC|85.04|74.85|79.61|+2.10|\n|VAC + LAC|85.31|75.36|80.02|+2.51|\n|VAC + MAC|85.11|75.10|79.79|+2.28|\n|LAC + MAC|85.56|75.84|80.40|+2.89|\n|TransAgent|85.29|77.62|81.27|**+3.76**|\n\n**Q5: Disorganized reference format.**\n\n**A5:** Thanks for your advice. We will check and reformat the references in our final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their constructive comments. 
We are delighted to receive positive feedback such as 
"the idea is intriguing" (**DUuV**), 
"valuable research direction" (**gY7M**), 
"can be extented flexibly" (**QaNB**), 
"unique contribution" (**A3Ab**), 
"of practical importance" (**wEnm**), 
"figures are clearly presented" (**DUuV, QaNB**), 
"easy to follow" (**QaNB, gY7M, DUuV**) 
and "well-structured" (**A3Ab, DUuV, gY7M**).
We have carefully addressed all the concerns raised by the reviewers in the individual response section.

Meta-review: This paper introduces a unified framework that utilizes various agents to integrate their knowledge for enhancing CLIP-like models. The framework is flexible, achieving superior results compared to state-of-the-art approaches, without incurring additional inference costs compared to the baseline. The writing is clear and easy to follow, and the authors have provided a thorough rebuttal addressing the reviewers' questions and concerns.
Decision: Accept (poster)