ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Experts
Authors: Yuanchen Wu
Junlong Du
Ke Yan
Shouhong Ding
Xiaoqiang Li
Venue: ICLR 2025 Poster (2025)
Keywords: Vision-language Modeling
Knowledge Transfer
Vision Experts
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Vision-language (VL) learning requires extensive visual perception capabilities, such as fine-grained object recognition and spatial perception. Recent works typically rely on training huge models on massive datasets to develop these capabilities. As a more efficient alternative, this paper proposes a new framework that Transfers the knowledge from a hub of Vision Experts (ToVE) for efficient VL learning, leveraging pre-trained vision expert models to promote visual perception capability. Specifically, building on a frozen CLIP image encoder that provides vision tokens for image-conditioned language generation, ToVE introduces a hub of multiple vision experts and a token-aware gating network that dynamically routes expert knowledge to vision tokens. In the transfer phase, we propose a "residual knowledge transfer" strategy, which not only preserves the generalizability of the vision tokens but also allows selective detachment of low-contributing experts to improve inference efficiency. Further, we explore to merge these expert knowledge to a single CLIP encoder, creating a knowledge-merged CLIP that produces more informative vision tokens without expert inference during deployment. Experiment results across various VL tasks demonstrate that the proposed ToVE achieves competitive performance with two orders of magnitude fewer training data.

Published as a conference paper at ICLR 2025
TOVE: EFFICIENT VISION-LANGUAGE LEARNING VIA KNOWLEDGE TRANSFER FROM VISION EXPERTS
Yuanchen W Junlong Duu, Ke Shouhong Ding", and Xiaoqiang Lift
School of Computer Engineering and Science, Shanghai University, Shanghai
Tenneet Youtu Lab, Shanghai
(yuanchenwu, 8 whs.shueee .edu.edd.... edu.cn, en,
( [jeffdu, kewwikyan, http::/www..wtt.....e.ee.oeeeeee
ABSTRACT
Vision-language (VL) learning requires extensive visual perception capabilities, such as fine-grained object recognition and spatial perception. Recent works typ- ically rely on training huge models on massive datasets to develop these capabil- ities. As more efficient alternative, this paper proposes new framework that Transfers the knowledge from a hub of Vision Experts (ToVE) for efficient VL learning, leveraging pre-trained vision expert models to promote visual percep- tion capability. Specifically, building on frozen CLIP encoder that provides vi- sion tokens for image-conditioned language generation, ToVE introduces a hub of multiple vision experts and token-aware gating network that dynamically routes expert knowledge vision tokens. the transfer phase, we propose a "resid- ual knowledge transfer" strategy, which not only preserves the generalizability of the vision tokens but also allows detachment prove inference efficiency. Further, we explore to merge these expert knowledge to a single CLIP encoder, creating a snoowedge-mergeee CLIP that produces more informative vision tokens without expert inference during deployment. Experi- ment results across various VI. tasks demonstrate that the proposed ToVE achieves competitive performance with two orders of magnitude fewer training data.
low-contributing experts to im-
INTRODUCTION
The integration of visual perception with language processing, referred to as vision-language (VL) learning, is critical frontier in multi-modal research. Compared to standalone language processing, a comprehensive super-set that necessitates additional visual perception capability. Many VL tasks, such image captioning (Lin et al., 2014) and visual question answering (VQA) (Antol et al., 2015), require the model to be capable of content understanding, fine-graine grained recognition, and spatial perception. Recent works have predominantly relied on massive datasets (sometimes over billions of image-text pairs) with large-scale model architectures (Wang et al., 2022a; Li et al., 2023a; Wang al., 2023) develop these capabilities from scratch. However, the dependency massive dataset presents significant challenges, particularly in specialized domains such as medical imaging where acquiring vast amounts of data is not feasible.
To achieve efficient VL learning, one direct approach is to train from scratch using a small-scale dataset and model architectures. However, the overall visual perceptual capabilities of these mod- els exhibits a significant degradation due to insufficient learning of diverse visual perceptual skills. Although some studies (Dai et al., 2022; Liu al., 2024b) have attempted address this issue by transferring the image-text pre-trained CL.IP (Radford et al., 2021) to VI. tasks, recent findings indi- cate that CLIP's visual perception capability also limited (Li al., 2022a; Tong al., 2024). As illustrated in Figure 1, the recent advanced efficient Vision LanguaeL Model (VLM) equipped with CLIP, Prismer-Z (Liu et al., 2024b), struggles with spatial reasoning and fine-grained perception, often misinterpreting spatial relationships and failing to differentiate between visually distinct ob- jects. Moreover, tasks such as image captioning, this model is prone to visual hallucinations,
"This work was done during an internship at Tencent Youtu Lab. 1 "oorreppnndence Ke Yan and Xianqiang Li..

---

Published as a conference paper at ICLR 2025
Image
Depth
Normal
CLIP
DINO
Question: the car below the parking meter?
Prismer-Z: Yes. X
ToVE: No.
Prismer-Z: white cake
penguins on top of
three tiered bunch of little fipurines on it.
Figure 1: The comparison between Prismer-Z and the proposed To on Novel Object Caption and Vision Spatial Reasoning.
Figure 2: Different vision experts can provide rich visual prior knowledge, which can be transferred to VL learning, and efficiently improve visual perception capability with limited, small-scale data.
wherein it incorrectly imagines details about an image. Given the availability of numerous pre- trained vision models from public repositories, our intuition that "why not fully utilizing these vision experts and transfer their knowledge to enhance the visual perception capability?" As shown in Figure different experts exhibit distinct vision properties for the same inputs, and each can contribute uniquely when their knowledge is transferred to VL learning (Geman et al., 1992).
To this end, we propose VLM that Transfers the knowledge from hub of Vision Experts (ToVE) for efficient VL. learning. Building recent VLM designs (Liu et al., 2024b; Li et al., 2023a), where frozen CLIP image encoder provides vision tokens for image-conditioned language generation, we establish model hub that includes multiple domain-specific vision experts and a token-aware gat- ing network that dynamically routes "expert knowledge" into every vision token. To preserve the generalizability of the vision tokens from CLIP, we develop "residual knowledge transfer" strategy when transferring the knowledge to vision tokens. Since the experts are not coupled the ToVE framework, we selectively detach experts with minimal contributions to enhance inference effi- ciency based on their average gating weights during the training stage. Further, since the knowledge from vision experts acts as complement calibration for the vision tokens, we merge the expert knowledge to the vanilla CLIP vision encoder via the proposed "Inowledge merging" approach. This approach eliminates the need for expert inference, significantly boosting inference efficiency while maintaining robust vision perception capabilities in VL tasks. In summary, our main contributions are follows:
Token-aware Knowledge Transfer from Vision Experts. Compared with previous works relying on large-scale models and datasets to develop the vision capabilities required by VL tasks from scratch, we construct model hub from readily available vision experts, transferring their knowledge to tasks for efficient learning. The proposed ToVE can dynamically route optimal vision knowledge respective vision tokens and adopts residual transfer strategy to enhance the original vision tokens while maintaining their generalizability. Consequently, To VE can achieve competitive performance with two orders of magnitude less training data.
Pluggable Vision Experts and Knowledge Merging. Since the vision experts ToVE are not coupled and their knowledge serves to complement calibrate each vision token, this allows to selectively detach the low-contributing experts to improve inference efficiency. Furthermore, with the vision tokens enriched with expert knowledge, we introduce "knowledge merging" approach to adapt this knowledge into a single vision encoder. This approach eliminates the need for vision expert inference while achieving promising performance without any vision experts.
2 RELATED WORK
Vision-language Learning. Vangunee (VL) learning represents the integration of visual and language processing capabilities. This field typically follows dual-phase approach: pre-training
2

---

Published as a conference paper at ICLR 2025
and sasc-specific fine-tuning. During the pre-training phase, models are trained image-text pairs, enabling them to learn visual perception aligned with the texts, thereby enhancing performance in downstream VL tasks. Fine-tuning involves transferring this model knowledge to specific VL tasks, such as image captioning (Lin et al., 2014) and visual question answering (VQA) (Antol et al., 2015). Notably, the pre-training phase is data-intensive, often requiring billions of image-text pairs to achieve satisfactory performance (Wang et al., 2022a; Alayrac et al., 2022; Wang et al., 2023). Recent advancements have seen some studies propose efficient training methods that offer improved performance even with smaller models and fewer data requirements, such as Prismer (Liu et al., 2024b), MAMO (Zhao et al., 2023), and EVE (Chen et al., 2024a). The work closest to ours Pris- mer (Liu et al., 2024b), which requires more data and additional training of "Resampler" (Alayrac et al., 2022) to learn to implicitly synthesize expert knowledge into auxiliary vision tokens. con- trast, we aim to transfer the vision knowledge from the vision experts to the original vision tokens. The transfer phase is explicit and interpretable, with a token-awart gating network that dynamically routes expert knowledge to these vision tokens.
Conditional Computation. The process of knowledge transfer from vision experts is close to con- ditional computation (Yang al., 2019; Chen et al., 2020; Han al., 2021). Recent works intro- duce the idea into mixture-of-expert (MoE) (Riquelme et al., 2021; Fedus et al., 2022; Wang et al., 2024) within Transformer architectures (Vaswani et al., 2017), where multiple Feed Forward Net- works (FFNs) serve as experts and a gating network selectively activates these experts to process the given input tokens. Different from these models which create mixture-of-experts from learnable FFNs with random initializations, we seek to transfer the knowledge from diverse, pre-trained vision experts to VL learning. Furthermore, MoE models typically prioritize load balancing (Fedus al., 2022) to ensure the full utilization of each expert, while our approach focuses on adaptively learning the optimal assignment of experts to efficiently transfer the vision knowledge for various VL tasks.
Learning from Models. Given the abundance of pre-trained models (referred to as experts) trained on diverse datasets, learning from models aims to leverage the knowledge gained from existing models to enhance model performance, rather than training from scratch with raw data. Traditional methodologics, such as fine-tuning and knowledge distillation, are frequently used but often fail to fully utilize the knowledge from existing models. To address their limitations, various model merging techniques have been developed to amalgamate or edit the weights from different homoge- neous models, such model soup (Wortsman et al., 2022), task arithmetic (Ilharco et al., 2022), and DARE (Yu et al., 2023). Recent efforts have also explored the ensemble of multiple heterogeneous models to tackle tasks such as classification (Shu al., 2022), domain adaptation (Li et al., 2022c), and language generation (Jiang et al., 2023). These approaches typically collect models from sim- ilar tasks and ensemble them at the task level. In contrast, we leverage vision experts from distinct domains, transferring their extensive and diverse vision knowledge to VL learning. In ToVE, we develop a "residual knowledge transfer" strategy to dynamically transfer optimal expert knowledge for each vision token, thereby enhancing vision token representations.
3 METHODOLOGGY
3.1 THE OVERALL FRAMBWORK
The ToVE framework, as illustrated in Figure 3, integrates CLIP (Radford al., 2021) vision encoder Evis, a fine-grained gating network G. model hub with K vision experts {E,, Ez, .... E EK] sourced from public repositories, and a language decoder Diung- For each image, its vision tokens are encoded by E,i. and are then sent to the gating network G to obtain the optimal assignment of expert knowledge at the token level. The expert knowledge is fused and transferred to the vision tokens via the proposed residual knowledge transfer strategy. After that, the knowledge-enhance vision tokens are sent to the language model through cross-attention to condition the language generation.
3.2 TRANSFER KNOWLEDGE FROM HUB OF VISION EXPERTS
Expert Token Projector. Since the vision experts vary in all the aspects including tasks, datasets, learning paradigms, and the network architectures, expert token projectors are required to align these experts within a unified embedding space before the knowledge transfer phase. The projector is
3

---

Published as a conference paper at ICLR 2025
Routing
Routing:
Vision Tokens
Token-aware
weighting
Image-conditioned generation
Expert
Expert K
Gating Network
Token-awore Expert Knowledge
on desk
keyboard.
Model Hub
Gating Network
text output t
DONO
EVA
MLP
CLIP
Shared MLP
Language Model
Adapter
Expert
Expert
Depth
MLP
casual attention mask
Gating Network
Vision Encoder
Visian
A cat sits on desk next to keyboard. 

CLIP
MLP
Tokens
CLIP
Figure 3: The overall framework of ToVE. The vision tokens processed by the vision encoder Evis are assigned expert knowledge through the gating network, then enhanced with a 'residual knowledge transfer" strategy before interacting with the language model. For the gating network, it dynamically assigns the optimal expert knowledge each vision token for VI. learning.
multi-layer perceptron network (MLP) with a GeLU (Hendryeks Gimpel, 2016) non-linearity, where the first layer is expert-specific and the second layer is shared among all vision experts. Specifically, given the k-th (k < K) vision expert Eg, its projector parameterized as: 1 (wff; 1/2). where v denotes the weights of the first layer specific to E; and 1/2 denotes the weights of the second layer shared across experts. The projection function of can delineated Ha,, E A Red Radue where dg is the token dimension of the expert E₂ and daang the token dimension of the language model D Dang-
Token-aware Expert Knowledge Ensemble. The representation of each vision token is distinet, carrying distinct knowledge (e.g., foreground objects, depth, spatial positions) for complex VL tasks. This necessitates a specialized strategy to transfer unique expert knowledge to each to- ken. To achieve this, we introduce token-aware gating network parameterized as 6, with routing function defined as Go E A Rdd.. + R where dvis denotes the length of the vision tokens. For each vision token t E Rdd,, the gating network takes it input and computes its routing score [F1, ..., ['] et al., 3021; Fedus al., 2022) that activates the expert with the top-1 routing score, we argue that knowledge from single expert domain is insufficient, and, in fact, the gating network can adap- tively learn the assignment of these experts. Therefore, we propose to apply an ensemble of the tokens tj derived from the vision experts to produce the expert knowledge for each vision to- ken. Specifically, for the vision token E normalize its routing score r; by softmax function Softmax(r), = e*s / T e"t, imposing a relative competition among the vision experts. That is, the final ensemble weight of expert knowledge is computed as w; Softmax(r; + e). where we empirically add small noise c sampled independently ~ A(0,, R entry-wise to improve the exploratory behavior of the gating network and the robustness of the model. Finally, the expert knowledge token for ty can be computed as:
] E R for each expert. Different from previous MOE works (Riquelme
K
wyy H (th) for i 11.2....,
(1)
exp
where N is the total number of vision tokens.
Residual Knowledge Transfer. Transferring expert knowledge into vision tokens can be achieved through two straightforward strategies: (a) directly integrating texp into vision token via addition; (b) appending texp as the auxiliary vision tokens; The former strategy (a) preserves the count of vision tokens but may lead an over-reliance on expert knowledge, potentially overwhelming the generalizable vision tokens from CLIP. contrast, the latter strategy (b) introduces additional com- putational burden and increases the complexity of learning from these auxiliary expert tokens. To address the above limitations, we introduce a residual knowledge transfer method, inspired by the residual architectures employed in modern foundation models (He et al., 201 6; Devlin et al., 2018).
4

---

Published as a conference paper at ICLR 2025
For each vision token Eiss its expert knowledge Exp is transferred via a residual addition, which is defined as follows:
tiis + x M6 (tesp),
(2)
vis
where Ma(-) denotes a two-layer MLP function : A Rdue + Rdu, parameterized by ó = [ Φ2]. Here, is the coefficient to reconcile the proportion of expert knowledge transferred into original vision tokens. By incorporating expert knowledge as residual addition term adjusted by À and an MLP function, rather than a direct alteration of the existing CLIP vision tokens, this strategy seamlessly integrates expert knowledge into the vision-language learning process, maintaining both the generalizability of the CLIP vision tokens and computational efficiency of the model.
Pluggable Vision Experts. From our analysis in Section 4.4, there is positive correlation between the average ensemble scores w and the experts' contributions. Since ToVE does not couple expert knowledge in the transfer phase, it is easy to detach low-contributing experts from the architecture to improve the inference efficiency. Suppose choose the top-k contributing experts & (|E) < K) for inference, the routing scores T; of the detached vision experts are set to -O0:
Gg((tn i E E
T; = Supp (Go where fupa (Go(t)))) - 1
(3)
otherwise
After applying the softmax function, only the ensemble weight of top-k contributing experts will be maintained and reconciled. Experts not included in & do not need to participate in the inference process. This operation allows the flexibility detach any number of experts without additional training, achieving a balance between computational resources and model performance.
3.3 VISION-LANGUAGE LEARNING OF TOVE
Language Modeling Pre-training. Based on an image-text pair dataset (I, E D, ToVE is pre-trained with unified language modeling loss without image-text contrastive and image-text matching losses commonly employed in other works (Li et al., 2021; 2022b). The loss of ToVE is formulated follows:
Ci 0, 0) E(LTY~D E (Diang (Eviss p).T)
(4)
where Evis is the knowsedge-transferred vision tokens of image I, l is the cross-entropy function between the predictions from the language decoder and the actual text description (i.e., caption) of the images, and is the prompt (i.c., prefix) of the language model. Lim ensures the alignment between the knowledge-transee transffrred vision tokens and text generation, leading the framework to explore optimal fusion configurations across diverse vision experts.
Enhaneing Exploration of Vision Experts. When optimizing solely with Lim, the gating network G.(-) is prone to overfit and fall into local minimum by trivially assigning ensemble weights to a few specific vision experts. One main reason is that different experts have dis- tinct domain knowledge, and vary in the dif- the
Image-conditioned generation
Experts
distillation
A cat sits on desk next keyboard. </5>
text output 1
Vision Encoder
ficulty of transferring their knowledge vision tokens through H.. This issue makes ToVE difficult explore other potentially valu- able vision knowledge. Thus, to ensure the sufficient learning of He before exploring the optimal routing of the vision experts, we em- ploy an auxiliary load balancing loss Laux(() on the gating network Go. followed by (Shazeer et al., 2017; Lepikhin et al., 2020; Fedus et al., 2022). The specifics of this implementation are detailed in Appendix A.3. During the train- ing phase, the vision experts can gradually pro- vide effective knowledge to vision tokens, hence, we set a relaxing coefficient a with a co- sine schedule progressively reduce Laux learn the optimal routing of vision experts: aa
CLIP
MLPs
Language Model
Adopter
merge
casual
attention mask
RUDA
CLIP
MLPs
<>> A cat sits on a desk next
keyboard.
Figure The overview of expert knowledge merging. The CLIP vision encoder enables the merging of expert knowledge into itself by pre- dicting the knowledge-transferres vision tokens as an auxiliary learning target.
5

---

Published as a conference paper at ICLR 2025
Pre-train
Models
VSR
POPE-R
POPE-A
Model
CIDEr
SPICE
(#f pairs)
MiniGPT-4
50.7
78.9
71.4
SimVLM
L8B (600x)
101.4
LLaVA
56.3
68.7
67.0
BLIPI
129M (43x)
113.2
14.8
BLIP-2p-78
50.0
BLIP-20PT2.
4M (1.3x)
111.9
14.5
InstructBLIPx--2
54.3
89.3
78.5
PrismenyARsK
12.7M (4.2x)
107.9
14.8
Prismer-2 (12.7M)
63.2
84,9
81.2
PrismeraasE
12.7M (4.2x)
87.5
13.0
Prismer-Z Z" (3M)
55.3
81.7
80.5
VinVL
5.7M (1.9x)
95.5
13.5
ToVE-Lite (3M)
65.9
86,6
81.9
OSCART
4M (1.9))
80.9
11.3
ToVE (3M)
67.7
87.4
82.5
ToVE (no experts)
3M
92.1
13.3
ToVe-Lite
3M
104.1
14.4
Table Zero-shot performance on VSR and POPE. Both Prismer-Z and ToVE are fine- tuned on VQAv2, and tested in a zero-shot manner. *: the results reproduced by (us- ing the same dateset as ToV V-7B: Vicuna- 7B (Zheng et al., 2024); Accuracy is adopted on VSR, and Fl-score is adopted on POPE.
ToVE
3M
110.2
14.9
Zero-shot caption performance on Novel Object Caption (NoCaps). "I": the model is fine-tumed on COCO caption dataset, and then conduct zero-shot caption test on NoCaps.
Table 1:
og x 0.5 x (1 cos (Tr x +)). where ar represents the coefficient at any given training epoch I, T is the total training epochs, and og is the initial coefficient the training start.
Learning Objective. With the incorporation of language modeling loss and the load balancing loss, the final optimization problem becomes:
arg min Lim + a Laux- (0,0,0)
(5)
Our To framework simultaneously explores the knowledge transfer to vision tokens and vision- language learning, which can be trained in an end-to-end manner.
3.4 REDUCE ALL EXPERTS INTO ONE
As more experts are integrated into the model hub, the computation load is increasingly intensi- fied. Therefore, inspired by distillation techniques that bridge the gap between training and infer- ence (Gou et al., 2021), we propose to merge the expert knowledge to the CLIP vision encoder, dubbed "expert knowledge merging". The proposed "residual knowledge transfer" strategy es- sential for effective knowledge merging, as it transfers the expert knowledge to the vision tokens without significantly altering the original representations. That is, there a small representation gap 9 Eyiss tvis between the original vision tokens and the knowledge-transfers and the learning target of "cxpert knowledge merging" is to minimize this gap g. As depicted in Fig- ure 4, maintain the language modeling loss while simultancously merging the expert knowledge to the vision encoder through the 12-norm loss:
vision token,
Liransfer ELTTY [e (Dtang (Ivis, p), T) + iyis tiis 112)
where Eyi and Evis denote the original and the wwwwwlege-transfereee vision tokens, respectively. In this transfer stage, only the CLIP vision encoder with its MLPs are online updated, while other components in ToVE remain frozen. inference stage, only the knowledge-merged CLIP provides the vision token for language generation.
4 EXPERIMENTS
4.1 IMPLEMENT DETAILS
ToVE utilizes VIT (Dosovitskiy et al., 2020) pre-trained by CLIP (Radford et al., 2021) as the frozen vision encoder, and RoBERTa (Liu al., 2019) as the frozen language decoder, following the prac- tice in (Liu et al., 2024b). Both of them have 12 (base-size) Transformer blocks. In the pre-training stage, the image resolution is set to 256 256. We random resized cropping and horizontal

---

Published as a conference paper at ICLR 2025
Pre-train (# pairs)
COCO
NoCaps
Pre-train (# pairs)
Model
Model
test-dev
test-std
B@4
S
Out
Overall
LEMON
200M
40.3
23.3
107.9
106.8
BETT3
3.1B
77.7
BLIP
129M
39.4
105.7
110.0
BLIP
129M
78.2
78.2
BLIP
14M
38.6
102.4
105.1
Prismer
12.7M
76,8
77.0
Prismer-Z
12.7M
39.7
24.1
105.8
107.5
OSCAR
8.9M
73.2
73.4
Prismer
12.7M
40.1
24.1
111.7
109.1
MAMO
4M
76.1
76.2
GIT
10M
40.4
23.0
89.6
96.6
MaskVLM
4M
75.5
VinVL
8.9M
38.2
23.6
83.8
94.3
ALBEF
4M
74.5
74.7
OSCAR
6.5M
36.5
23.1
77.6
81.1
Triple
4M
74.9
74.9
ToVE-Lic
3M
39.5
24.1
108.2
106.7
ToVE-Lite
3M
74.1
74.0
ToVE
3M
40.3
24.5
113.1
112.5
ToVE
3M
75.4
75.8
Table 3: Fine-tuned caption performance on COCO (Karpathy split) and NoCaps (validation set). These are all base-size models. CIDEr is adopted on NoCaps. B@:: BLEU-4; S: SPICE.
Table 4: Fine-tuned VQA performance on
VQA v2 (test set). These are all base-size VLM models. Accuracy is adopted to evalu- ate the VQA performance.
flipping for data augmentation. The pre-training dataset is composed of two in-domain datasets (i.e., COCO (Lin al., 2014) and Visual Genome (Krishna al., 2017)) and web dataset (i.e.. CC3M (Sharma et al., 2018)). The web dataset is filtered and re-captioned by a pre-trained image captioner (Li et al., 2022b). In the fine-tuning stage, the image resolution is increased to 480 480, and the load balancing loss is not employed to further exploit the optimal expert assignment to spe- eific VL tasks. To enhance the model's capability process lightweight MLP adaptors (Houlsby et al., 2019) are integrated within each transformer layer of the language model. More training details are provided in Appendix A.8.
wowledge-transferree vision tokens,
4.2 VISION EXPERTS IN THE MODEL HUB
Low-level Vision Experts. ToVE is equipped with three low-level vision experts from the domains of depth (Ranftl et al., 2021), surface normal (Bae et al., 2021), and edge (Poma et al., 2020). These predicted labels are conducted patch embedding operations through randomly initialized convolu- tional layers. Specifically, we employ five convolutional layers with a small [3 x 3] kernel to encode their respective expert knowledge. The details of these vision experts and encoding convolution layers can be viewed in Appendix A.2.
Embedding Vision Experts. We include two embedding vision experts, i.e., DINO (Caron et al., 2021) and EVA (Fang et al., 2023). trained from self-supervised and image-text contrast- ing paradigms, respectively. Compared to the low-level vision experts, we utilize their patch tokens as the vision knowledge. To align the number of tokens with the vision encoder (i.e., CLIP) for knowledge transfer, we apply an interpolation operation to their patch tokens. The details of these vision experts can be viewed in Appendix A.2.
4.3 RESULTS ON VISION-LANGUAGE TASKS
Zero-shot Performance on Novel Object Captioning. By adopting a unified language model- ing objective pre-train ToVE, naturally generate descriptions for images without requiring fine-tuning, thereby enabling zero-shot generalization. In Table 1, we compare ToVE against sev- eral prior arts, including SimVLM (Wang al., 2022b). BLIP/BLIP2 (Li et al., 2022b; 2023a), VLnVL (Zhang et al., 2021), OSCAR (Li al., 2020), and Prismer (Liu et al., 2024b). Despite using fewer data pairs and smaller models, ToVE consistently demonstrates superior zero-shot captioning abilities. For instance, under a model scale similar to PrismerBASE, ToVE achieves CIDEr score of 110.2 and SPICE score of 14.9, surpassing PrismerBASE by a large margin (+22.7 CIDEr) while utilizing an approximately 4x smaller pre-training dataset. For the variant of ToVE after expert knowledge merging, dubbed "ToVE-Lite", outperforms the baseline model without any ex- pert participation by CIDEr score of +12.0, attaining 95.6% of the original ToVE's performance. Additionally, the zero-shot performance of To VE models often exceeds the fine-tuned performance of certain other VLMs, such as LEMON and BLIP. More results can be viewed in Appendix A.5.
7

---

Published as a conference paper at ICLR 2025
Fine-tuned Performance COCO Caption, NoCaps, and VQAv2. We summarize the fine- tuned captioning and VQA evaluations in Table and Table 4, respectively. With a smaller training dataset, ToVE demonstrates strong results for both COCO and NoCaps benchmarks while using fewer image-text pairs. Especially for NoCaps, with out-domain samples with novel objects, we make significant improvements over the prior arts, with CIDEr of 113.1. In VQA, We additionally compare ToVE against several prior arts, BEiT3 (Wang et al., 2023), MAMO (Zhao et al., 2023), MaskVLM (Kwon et al., 2023), (Li et al., 2021), and Triple (Yang al., 2022). ToVE also achieves comparable performance with the prior arts. This indicates that the knowledge transfer from various vision experts is primarily responsible for good robustness and generalization.
Vision Perception Capabilities. We evaluate the VLM's vision capabilities using the Visual Spa- tial Reasoning (VSR) (Liiu et al., 2023) and POPE (Li et al., 2023b) benchmarks. VSR evaluates the reasoning about the relative positions of different objects, while POPE evaluates the "object hallu- cination" issue. From Table 2, when the dataset is scaled down from 12.7M to 3M, we can observe the recent efficient VLM, Prismer-Z (Liu et al., 2024b), shows a marked decline (-7.9% in VSR) in visual perception capabilities with smaller datasets, with a more severe issue of object hallucination. Conversely, when transferring the knowledge from vision experts to VL, ToVE significantly out- performs Prismer-Z Z with the same 3M dataset (+12.4% % in VSR), with improvements in mitigating object hallucination (+2.0% in POPE-A). Additionally, our comparison with recent multi-modal large language models (including MiniGPT-4 (Zhu et al., 2023), LLaVA (Liu et al., 2024a), and InstructBLII, 70 (Dai et al., 2024)) demonstrates that VLMs with large-scale pre-training generally has better visual perception capabilities. Notably, ToVE shows a substantial advantage over current MLLMs, especially in vision spatial reasoning.
4.4 ANALYSIS OF TOVE
Visualization of the routing maps from the Gating Net- work G. As presented in Figure 6, we visualize the routing maps of the images from the COCO caption dataset de- termined by ToVE's gating network. It showcases the gat- fu-
:40
90
132.8
13211
130
40
126.5
ing network's s adeptness in learning restance-dependes sion weights at the patch level, crucial for transferring the
Ensemble weight
:20
most beneficial vision knowledge from the vision experts to
CIDEr
the vision tokens. For the low-level experts (columns 2-4), a consistent pattern is observable across their routing maps, where they predominamtly enhance the areas around objects. This pattern indicates that these experts significantly enrich the CLIP tokens with low-level information, such depth perception around objects, edges, and surface orientations, which bolsters the model's ability perceive fine details. In contrast, for the embedding experts (columns 5-6). we notice distinct patterns knowledge contributions: DINO primar- ily enhances the patch tokens associated with the main ob- jects in the captions, whereas EVA contributes significantly to the understanding of the image backgrounds. More rout- ing maps can be viewed in Appendix A.4.
:10
110.02
11933
108.
108.6
1043
300
collapse
ensemble weight
COCO
Nocaps
90
None Edge Depth Narmal DINO EVA
Removed Experts
Figure 5: Impact of iteratively moving vision experts on zero-shot caption performance.
Pluggable Vision Experts. In Section 3.2, we explore the strategy of detaching less-contributing experts to enhance inference efficiency. is built on a positive correlation between the ensemble scores w and the performance benefits contributed by these vision experts. Based on the average ensemble scores, interpreted as "contribution", counted during the training phase, we iteratively remove experts in descending order of their average contributions. As depicted in Figure 5, it is observed that the performance declines noticeably as experts with increasingly significant contribu- tions are removed. The model exhibits higher sensitivity the in-domain dataset, COCO caption, compared to that on a hold-out dataset, NoCaps. Upon the complete removal of all experts, we notice a model collapse where the model fails in normal language generation. We attribute this collapse to the substantial discrepancy between the vision tokens without expert knowledge and the input space of the language model. We also conduct a similar experiment on the VQAv2 benchmark, the results can be viewed in Appendix A.6.
8

---

Published as a conference paper at ICLR 2025
Image
Depth
Normal
Edge
DINO
EVA
Caption Results
GT: man flying through the air while riding skateboard.
FoVE: snowboarder gliding down snowy slope.
The infamous big clock underneath cloudy sky,
FoVE: clock tower underneath cloudy sky.
GT: A red bird being fed from person's hand.
FOVE: person is holiding
Figure 6: Visualization of ToVE's routing maps and the corresponding caption results on COCO caption without fine-tuning, Brighter patches indicate higher activation of the corresponding expert.
Tasks
Baseline
Depth
Edge
Normal
DINO
EVA
ToVE
132.8
126.5
127.1
COCO
116.8
120.7
121.1
120.3
128.9
130.6
132.8
120
NoCaps
92.1
98.2
96.3
98.7
105.1
109.1
110.2
110.2
116.8
Baseline
VQAv2
70.0
70.1
70.1
72.9
74.4
75.8
CIDEr
92.1
VSR
54.8
59.8
60.1
61.0
65.3
63.8
67.7
POPE-R 80.7
85.9
86.4
86.3
85.8
85.7
55,1
POPE-A
80.3
80.7
80.8
81.2
82.0
80.8
82.5
COCO
Table VL performance of the variants equipped with different vision experts. Baseline: the ablated variant without transferring any vision experts to VI. learning.
0.3
0.5
lambda
Figure 7: Ablation study on A.
ABLATION STUDIES
Different Vision Experts Transferred to VL Learning. We evaluate the performance of each vi- sion expert transferred VL tasks. As shown in Table 5, we observe that each vision expert yields performance gains compared to the baseline without any vision experts. Specifically, the low-level experts significantly enhance visual perception capabilities, although their improvement in VQA, which requires strong multi-modal reasoning. is relatively marginal. On the other hand, the embed- ding experts contribute more substantially across all VL tasks, consistent with our average ensemble shown in Figure 5. DINO improves visual perception capabilities, demonstrating significant gains VSR and POPE. Conversely, EVA promotes content understanding, showing notable per- formance improvements captioning and VQA tasks. Also, further compare the results of using EVA as the base vision encoder, the discussions can be viewed in Appendix A. 10. After transferring the knowledge from all experts, ToVE achieve significant performance enhancements across all VL tasks compared to the baseline without any experts, possessing both stronger visual perception and content understanding capabilities.
in Residual Knowledge Fusion. À controls the proportion of transferring vision expert knowledge the vision tokens. depicted in Figure 7, our experiments demonstrate that relatively small À results in superior performance compared to a larger one. For in-domain COCO captions. the proportion of expert knowledge fusion yields a relatively marginal impact. However, for NoCaps, which requires the model's generalizability on the novel objects, the model performance sharply declines (from 110.2 to 35.9) as À increases beyond 0.1. This underscores the inherent strong gen- eralization capabilities of CLIP for VL tasks. An excessive fusion of expert knowledge, in turn, diminishes its generalizability. Therefore, only a modest complement of expert knowledge is req- uisite for achieving satisfactory results.

---

Published as a conference paper at ICLR 2025
Knowledge Transfer and Merging. In knowledge transfer strategies, evaluate three approaches: (A1) direct addition of the expert knowledge tokens to the vision tokens, (A2) concatenation with the vision tokens, and (A3) residual addition to the vision tokens (ours). We observe that Al results in poor performance on NoCaps, which can be considered a special case where À is set to 1.0, as discussed in "À in residual knowledge fusion". While A3 slightly underperforms in comparison residual addition (A3), it does not deteriorate the generality of the CLIP model as Al does.
In knowledge merging strategies, we evaluate three approaches: (B1) direct distillation be- tween the original vision tokens and the to- kens with fused expert knowledge using L2- norm loss, (B2) language modeling to align the vision tokens with the input space of the language model that accepts the knowledge- transferred vision tokens, and (B3) combina- tion of language modeling and L2-norm dis- tillation (ours). The results manifest that B1 yields poor distillation performance, indicating significant gap between the language model and the distilled CLIP. Although B2 can di- rectly optimize the gap between the distilled CLIP and the language model, we can observe that combining language modeling with L2-norm (B3) achieves the optimal performance.
Ablation
Strategies
NoCaps COCO
Direct addition
26.8
126.3
Transfer
Concatenation
108.2
130.4
Residual addition
110.2
132.8
L2-norm
10.5
45.6
Merging
LM
96.2
118.7
LM + L2-norm
106.7
128.6
Table 6: Ablation study on knowledge transfer and merging strategies.
6 CONCLUSION
This paper proposes the ToVE framework for efficient vision-language learning by transferring knowledge from a hub of pre-trained vision experts. It utilizes a gating network with a resid- ual knowledge transfer strategy dynamically route expert knowledge to vision tokens, ensuring enhanced visual perception while preserving generalizability through residual knowledge transfer. This method allows for the selective detachment of low-contributing experts improve inference efficiency. Additionally, we introduced knowledge merging approach to merge expert knowledge into single vision encoder. Experiments across various VL tasks demonstrate ToVE's competi- tive performance with significantly less training data, excelling in zero-shot captioning and visual spatial reasoning tasks. The visualization of gating network outputs and the analysis pluggable vi- sion experts highlight the effectiveness of transferring diverse vision knowledge to VL tasks, which presents a promising alternative to large-scale models and datasets.
REFERENCES
Jean-Bape nan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716- 23736, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433. 2015.
Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Estimating and exploiting the aleatoric uncer- tainty in surface normal estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13137-13146, 2021.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Junyi Chen, Longteng Guo, Jia Sun, Shuai Shao, Zehuan Yuan, Liang Lin, and Dongyu Zhang. Eve: Efficient lision-language pre-training with masked prediction and modality-aware moe. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 11 1110-1110 2024a.
10

---

Published as a conference paper at ICLR 2025
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large rision-language models? arXiv preprint arXiv:2403.20330 2024b.
Yinpeng Chen, Xiyang Dai, Mengehen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 11000-11039, 2020.
Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation clip via vision-language knowledge distillation. arXiv preprint arXiv:2203.0633 063366 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances Neural Information Processing Systems, 36, 2024.
Jacob Devlin, Ming- Wing-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880,, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint DrX1v:2010.11929, 2020.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. arXiv preprint arXiv:2303. 11331, 2023.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120):1-39, 2022.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma.
Neural computation, 4(1):1-58, 1992.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 179(6):189-1189, 2021.
Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, and Yulin Wang. Dynamic neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(11): 7436-7456, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp-
770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv:1606.08415. 2016.
arXiv preprint
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-eeffiieet transfer learning for nlp. In International conference on machine learning, pp. 2790-2799, PMLR, 2019.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. arXiv preprint arXiv:2212 04089, 2022.
11

---

Published as a conference paper at ICLR 2025
Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LIm-blender Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint arXiv:2306.02561, 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. jonpnnttiona.ournaa of computer vision, 123:32-73, 2017.
Gukyeong Kwon, Zhaowei Cai, Avinash Ravichandran, Erhan Bas, Rahul Bhotika, and Stefano Soatto. Masked vision and language modeling for multi-modal representation learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xie, Yueting Zhuang, Qi Tian, and Siliang Tang. Fine-grained semantically aligned vision-language pre-training. Advances in neural information processing systems, 35:7290-7303, 2022a.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705, 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023a.
Xiujun Li, Xi Yin, Chumyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28,
2020, Proceedings, Part XXX 16. pp. 121-137. Springer, 2020.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023b.
Ziyue Li, Kan Ren, Xinyang Jiang, Yifei Shen, Haipeng Zhang, and Dongsheng Li. Simple: Special- ized model-sample matching for domain generalization. In The Eleventh International Conference on Learning Representations, 2022c.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Associ- ation for Computational Linguistics, 11:635-663 6511, 2023.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024a.
Shikun Liu, Linxi Fan, Edward Johns, Zhiding Yu, Chaowei Xiao, and Anima Anandkumar. Pris- mer: A languaela model with multi-task experts. Transactions on Machine Learning Re- search, 2024b. ISSN 2835-8856.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: arXiv: 1692, 2019.
12

---

Published as a conference paper at ICLR 2025
Xavier Soria Poma, Edgar Riba, and Angel Sappa. Dense extreme inception network: Towards a robust enn model for edge detection. In Proceedings of the IEEE/CVE winter conference on applications of computer viston, pp. 1923-1932, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1 188, 2021.
Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583-8595, 2021.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemnyee, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 2018
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-expprts layer. arXiv preprini arXiv: 70633338 2017.
Yang Shu, Zhangjie Cao, Ziyang Zhang, Jianmin Wang, and Mingsheng Long. Hub-pathway: trans- fer learning from hub of pre-trained models. Advances in Neural Information Processing Sys- tems, 35:32913-32927,, 2022.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint arXiv:2401.0622 066209 2024.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Haotao Wang, Ziyu Jiang, Yuning You, Yan Han, Gaowen Liu, Jayanth Srinivasa, Ramana Kom- pella, Zhangyang Wang, et al. Graph mixture of experts: Learning on large-scale graphs with explicit diversity modeling, Advances in Neural Information Processing Systems, 36. 2024.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 1200002005.14000, 2022a.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng. Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, al. Image foreign language: Beit pretraining for vision and lasion-language tasks. In Proceedings of the IEEFJCVF Confer- ence on Computer Vision and Pattern Recognition, pp. 19175-19186, 2023.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022b.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965-23998. PMLR, 2022.
Brandon Yang, Gabriel Bender. Quoc V Le, and Jiquan Ngiam. Condconv: Conditionally parame- terized convolutions for efficient inference. Advances in neural information processing systems, 32, 2019.
13

---

Published as a conference paper at ICLR 2025
Jinyu Yang. Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision- langonlanguuag pre-training with triple contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15671-15680, 2022.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models free lunch. arXiv preprint arXiv:2311.0309, 2023.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5579-5588, 2021.
Zijia Zhao, Longteng Guo, Xingjian He, Shuai Shao, Zehuan Yuan, and Jing Liu. Mamo: Fine- grained vision-language representations learning with masked multimodal modeling. In Proceed- ings of the 46th International ACM SIGIR Conference Research and Development in Informa- tion Retrieval, pp. 1528-15388 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 MODEL ARCHITECTURE
Table provides an overview the architecture details for both ToVE-Lite and ToVE models. The vision encoder in both models utilizes ViT-B/16 backbone with 12 layers. The language decoder is based on ROBERTARASE. also with 12 layers and a width of 768. For the ToVE-Lite model, the total number of trainable parameters is 91 million, with total parameter count of 260 million when including non- von-trainable parameters. The ToVE model, which incorporates five vision experts, has 103 million trainable parameters. The total parameter count for the ToVE model. including the combined 1 billion parameters from the vision experts, sums up to 1.2B. These details illustrate the comprehensive design and the scalable nature of our models, accommodating varying levels of complexity and inference capabilities.
Vision Encoder
Language Decoder
Trainable
Total
Model Type
Backbone
Layers
Backbone
Layers
Width
Params.
Params.
ToVE-Lite
VIT-B/16
12
RoBERTa BASE
12
768
91M
260M
ToVE
VIT-B/16
12
RoBERTa
12
768
103M
1.2B
Table 7: ToVE- and ToVE architecture details. We detail the backbone, number of layers, and width for each architecture size, as well as the trainable and total parameters. For data inference, we include the total parameters, which encompass five vision experts with combined parameter size of IB in our ToVE model.
A.2 VISION EXPERTS
The details of the vision experts are provided in Table 8. For low-level vision experts, the expert labels are initially processed using randomly initialized convolutional layers to encode their respec- tive vision knowledge. Specifically, we employ five convolutional layers with a small [3 3] kernel, which demonstrates superior performance compared to single layer with larger kernel, as evi- denced the Vision Transformer (Dosovitskiy al., 2020). For embedding experts, we select their large-size models. They have patch size of 14, with input images sized at 224, producing 256 patch tokens. To achieve the residual knowledge transfer, their token quantity matches that of the vision encoder, which uses the Base-size model with patch size of 16 and input images sized at 256.
Task
Dataset
Model
Params,
Post-Processing
Depth Estimation
MIX-6
DPT (Ranftl al., 2021)
123M
Re-normalised to | 1] and use convolution layers to encode the vision knowledge
Surface Normal
ScanNet
NLL-AngMF (Bae al., 2021)
72M
Edge Detection
BIPED
DexiNed (Poma et al., 2020)
35M
Self-supervised
LVD-142M
DINO-v2 (Caron et al., 2021)
304M
The input image size is resized to 224 x 224 and encoded tokens.
MIM CLIP
Merged-38M EVA-CLIP 02 (Fang et al., 2023)
430M
Table 8: Selected Tasks and Vision Experts with Parameters and Post-Processing Techniques.
A.3 LOAD BALANCING LOSs
To encourage a balanced assignment of vision tokens across different experts, we incorporate an auxiliary loss into the gating network. This auxiliary loss is beneficial to ensure the sufficient learning of H. before exploring the optimal routing of the vision experts. Its consists of two parts: Importance loss and Load loss. The importance of the k-th expert is defined as the normalized
15

---

Published as a conference paper at ICLR 2025
gating routing scores corresponding to the k-th expert, summed over the vision tokens tyis of an image I:
Imp, (tris)
Coftmax((Go(tia)) E
The importance loss over the vision tokens Livis from an input image T defined as:
std(Imp((tis)) meaan (Imp(txss)
Limp (trvi))
In addition to the importance loss, also introduce a load loss to ensure balanced routing results. The load of an expert k given vision token Ei E Evs defined as the probability (frequency) of routing to expert k, summed over the vision tokens tyis of image I:
Loadæ(tvis)
Σ
Pk (tii))
(9)
E/s Et
(x(() P((G (Cya)) 2 thresholdy (Go (L)
(10)
The load loss over the vision tokens Lvis from an input image I is defined as:
(oo(Lod((4vs)) imean(Lodd(ta::)
Croad(tvis
(11)
The total auxiliary loss of the gating network is then given by:
1 (Lmp + Lload)
Caux
(12)
A.4 ROUTING THE GATING NETWORK
Figure provides additional visualizations of the routing maps generated by ToVE's gating network for images from the COCO caption dataset. These maps highlight how the gating network assigns expert knowledge at the patch level. Low-level experts (Depth, Normal, Edge) predominantly en- hance object-related areas, enriching the CLIP tokens with essential visual information. Embedding experts (DINO, EVA) show distinct contributions, with DINO focusing on main objects and EVA en- hancing background understanding. These visualizations demonstrate the gating network's adaptive capability in optimizing vision token enhancement.
A.5 VISUALIZATION OF CAPTION AND VQA RESULTS
75
75.4
Figure 10 illustrates the performance of ToVE and ToVE-Lite on zero-shot image captioning tasks using the NoCaps dataset and fine-tuned VQA tasks VQAv2. Both models perform well, but ToVE consistently provides more detailed and con- textually rich descriptions. For instance, ToVE specifies "hot chocolate" instead of just "coffee" and adds context like "be- ing loaded onto flatbed trailer." In VQA tasks, ToVE con- sistently delivers accurate and concise answers, demonstrat- ing superior visual and language understanding capabilities. However, it is important to note that ToVE requires higher in- ference costs due to the integration of multiple vision experts. In contrast, Vo-L- while slightly less detailed, still per- forms admirably with lower computational overhead. This trade-off between performance and inference cost should be considered when choosing between the two models based on the specific requirements of the application.
74.3
23.8
Ensemble weight
Accuracy
/2.4
22
Baseling
70
collapse
ensemble weight
None Edge Depth Normal DINO EVA
Removed Experts
Figure 8: Impact of sequentially
moving vision experts on VQA.
16

---

Published as a conference paper at ICLR 2025
Image
Depth
Normal
Edge
DINO
EVA
Figure 9: Visualization of To'E's routing maps and the corresponding caption results on COCO caption without fine-tuning. Brighter patches indicate higher activation of the corresponding expert.
A.6 PLUGGABLE VISION EXPERTS ON VQA
In further analysis, we investigate the impact of sequentially detaching experts on the visual question answering task using the VQAv2 dataset. The green line in Figure illustrates the performance trajectory as vision experts are iteratively removed in descending order of their average ensemble weight, interpreted as their relative contribution. This trend mirrors the observations made on the image captioning task, underscoring the positive correlation between expert's ensemble weight and its performance utility across diverse vision-language tasks. Notably, the degradation appears even more severe for VQAv2 compared to the COCO caption benchmark, implying heightened reliance on the visual experts for this question answering challenge.
A.7 LOAD BALANCING LOSS AND NOISE
In our exploration of strategies to enhance the effec- tive use of vision experts, we evaluate the impact of load balancing loss and the introduction of noise e in Table 9. The results show that both load balanc- ing loss alone (C1) and noise alone (C2) improve performance. Load balancing loss enhances perfor- mance on NoCaps to 109.9 and COCO to 132.1 by promoting balanced expert utilization. Noise im- proves performance to 109.4 on NoCaps and 130.9 COCO by preventing overfitting to specific
Ablations
NoCaps COCO
Load Balancing Noise
109.9
132.1
109.4
130.9
Load Balancing + Noise
110.2
132.8
Table 9: Ablation study on Load Balancing Loss and noise
17

---

Published as a conference paper at ICLR 2025
ToVE-Lite:
What does the dispenser the wall give? ToVE: Toilet paper.
The car is on flatbed trailer.
ToVE:
Q: What colors are the tile on the wall? ToVE: Black and white.
A silver car being loaded onto flatbed trailer.
ToVE-Lite: A cup coffee with whipped cream on top,
Q: What is the players number? ToVE: 22.
cup hot chocolate with whipped cream on top.
ToVE:
Q: hit the ball?
ToVE: Yes.
ToVE-Lite: A military tank driving down the road.
Q: Can the dog drive?
ToVE: No.
ToVE: A military tank driving down a street next to trees.
Q: What breed the black dog?
ToVE: Pug.
ToVE-Lite: A table with bunch of drinks and box of coke.
Q: Where is this animal commonly found?
ToVE: table topped with a cup coffee and a pack starbucks energy drinks.
ToVE: Farm.
Q: What color is it's nose? ToVE: Black.
Figure 10: Visualisation of zero-shot image captioning on NoCaps and fine-tuned VQA results on VQAv2. ToVE can produce more detailed and semantically coherent captions than ToVE-Lite. We show failure case (yellow box) of Nocaps dataset.
perts. Combining both strategies (C3) yields the best results, with 110.2 on NoCaps and 132.8 on COCO. Without load balancing, the gating network quickly converges to a trivial solution, predomi- nantly routing the EVA expert, making results similar to using EVA alone. Thus, the combination of load balancing and noise ensures a more effective and comprehensive integration of diverse expert knowledge, leading more robust and generalized vision-language representations.
A.8 TRAINING DETAILS
All our models are trained using the AdamW optimizer with a weight decay of 0.05. Automated data augmentation (AutoAug) is applied during both the pre-training and fine-tuning stages. For pre-training, the learning rate is set to 3e-4 with a total of 10 epochs. During fine-tuning for VQA, we use learning rate of le-5 and train for 10 epochs. For fine-tuning the captioning model, the learning rate is le-5 with a total of 3 epochs.
A.9 TOVE WITH LLMS
With the rapid advancements in LLMs numerous LLM-based Vision-Language Models (LVLMs) have emerged. In this work, we also extend ToVE to language models to explore its potential in the LVLM domain. Specifically, we implemented ToVE within the LA-155 framework (Liu al., 2024a) (dubbed as "ToVE Vicuna") with LLaVA's training data. During our implementation, we randomly sampled three-quarters of the dataset and utilized the entire instruction-tuning dataset. In the ToVE design, we integrated two domain experts DINO (Caron et al., 2021) and Depth (Ranftl et al., 2021)-and employed QLoRA (Dettmers et al., 2024) to reduce computational overhead. To evaluate ToVE Vicuna, we conducted experiments on the MME (perception) (Fu et al., 2023) and MMStar (perception) (Chen et al., 2024b) benchmarks to validate the improvement in visual capa- bilities brought by expert knowledge. As shown in Table 10, the results demonstrate significant enhancement in perception capabilities through knowledge transfer. For instance, the MME percep- tion score increased from 1434.5 to 1523.1, while the fine-grained perception subset of MMStar saw
18

---

Published as a conference paper at ICLR 2025
Models (QLoRA)
MME p
MMStar (Overall)
MMStar (Coarse)
MMStar (Fine-grained)
LLaVA-1.5-7B
1434.5
34.6
61.6
27.6
ToVE Vicuna
1523.1
35.8
64.0
31.2
Table 10: Performance comparison of Vicunic and LLaVA baseline on MME and MMStar.
an improvement of 3.6 points. These findings support the potential of ToVE in the LVLM domain. the future, we will conduct further exploration this direction.
Benchmark
CLIP Encoder
EVA Encoder
CLIP + EVA expert
NoCaps
92.1
95.7
109.1
VQAv2
70.0
70.5
74.4
VSR
54.8
51.7
63.8
Table 11: Comparison of performance using different encoders.
A.10 COMPARISON OF BASE VISION ENCODERS
We compared the performance of ToVE when using CLIP and EVA as the base vision encoders, with results summarized Table 11. As shown, the two models exhibit comparable performance. Inter- estingly, we observed notable improvement when combining CLIP as the vision encoder and EVA as the expert. We reckon that this enhancement can be attributed to EVA's capability to effectively process background representations, as evidenced by Figures and 9. The visualizations illustrate the gating maps for each vision expert, revealing that EVA's gating activations predominantly oc- cur in background regions, while showing minimal activation in subject areas. This observation suggests that, while low-level experts and DINO primarily focus on visual perceptual knowledge (as supported by their performance in visual perception tasks), their contributions to understanding background context remain limited. contrast, EVA significantly improves semantic comprehen- sion of these regions for the base vision encoder, thereby enhancing overall model performance.
A.11 LIMITATIONS
Despite its advancements, the ToVE framework has several limitations. It heavily depends on the availability and quality of pre-trained vision experts, which may not always available for certain domains tasks. The initial integration and training with multiple experts can be computationally intensive, posing challenges in resource-constrainee environments. Estimating the contribution of each expert relies empirical methods that might not always yield optimal configurations, poten- tially affecting performance. While To demonstrates robust generalization within its pre-training datasets, its effectiveness completely unseen domains may vary, necessitating additional fine- tuning. Furthermore, the complexity of integrating multiple experts and dynamic mechanisms adds the implementation and debugging challenges, especially for practitioners with limited experience in multi-modal learning frameworks. Addressing these limitations in future work could involve de- veloping more adaptive methods for expert integration, optimizing computational efficiency, and enhancing generalization across diverse tasks and domains.
19


Reviews:
[{"review_id": "2gWL9b2QYw", "reviewer": "Reviewer_dYR6", "summary": "This paper presents the ToVE (Transfer of Vision Experts) framework, leveraging pre-trained vision models to enhance vision-language learning efficiency by transferring expert knowledge via a token-aware gating network, resulting in competitive performance on vision-language tasks with significantly reduced data requirements.", "questions": "As shown in Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The ToVE framework introduces a novel approach for efficient vision-language learning by utilizing a hub of pre-trained vision experts. This method promotes the effective transfer of knowledge, addressing the challenge of limited data availability in specialized domains.\n\n2. The experimental results demonstrate that ToVE achieves competitive performance across various vision-language tasks using significantly less training data compared to existing models. \n\n3. The paper includes visualizations of the gating network's routing decisions, illustrating how expert knowledge is allocated across different image regions. This enhances interpretability, showing how ToVE leverages expert knowledge in a token-specific manner to improve performance on complex visual tasks.", "weaknesses": "1.  The process of routing expert knowledge to vision tokens, particularly the token-aware gating network, adds complexity. Although the authors propose methods for detaching low-contributing experts to improve efficiency, the initial setup and training remain computationally intensive, which may hinder practical application.\n\n2.  The paper introduces a load-balancing loss to ensure a balanced use of experts, but the effectiveness of this loss in preventing over-reliance on certain experts is not extensively validated, leaving questions about how it affects the model’s performance and robustness.\n\n3.  The paper emphasizes the efficiency benefits of transferring pre-trained expert knowledge, yet it lacks a detailed discussion on how ToVE handles potential domain mismatches between the knowledge of vision experts and specific downstream vision-language tasks, such as highly specialized applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jytQd0UBkj", "reviewer": "Reviewer_q7GH", "summary": "This paper introduces ToVE, a selection method for vision encoders in vision-language models. ToVE utilizes multiple vision encoders, each pre-trained differently, and uses a gating network to select their output tokens, which are then added to the tokens from a CLIP encoder as visual features. Notably, these vision encoders can be distilled into a single encoder to reduce inference computation.", "questions": "See weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes a novel method for enhancing vision-language models by leveraging multiple vision encoders.\n2. Experiments demonstrate the superiority of using multiple vision encoders.", "weaknesses": "1. My primary concern is the significance of this approach. Using multiple vision encoders will multiply the parameter count and computational cost, making comparisons with other base-sized VLMs unfair. On the other hand, if distillation is used to merge them into a single encoder, the resulting composite encoder does not performs obviously better than simply using EVA as the encoder.\n\n| dataset  | EVA    | ToVE-lite |\n| ----- | ----- | ----- |\n| NoCaps | 109.1 | 104.1|\n| VQAv2   | 74.4   | 74.0|\n| VSR        |  63.8 | 65.9|\n| POPE-R | 85.7 | 86.6|\n| POPE-C | 80.8 | 81.9|\n| average |82.76|82.50|\n\n\nThis suggests that using ToVE-lite might be less effective than carefully selecting a single, well-performing encoder.\n\n2. As mentioned above, due to the difference in parameter counts, comparisons with other VLMs may be unfair. Additionally, although the amount of VL data used for pretraining is indicated, the vision encoders and language models themselves are trained on large datasets, enhancing their individual visual and linguistic capabilities, which in turn boosts multimodal performance. Therefore, the amount of pretraining data for both the vision and language models should also be specified.\n\n3. The setting used in this paper seems somewhat outdated, as the current trend in VLMs is toward general-purpose multimodal LLMs, such as LLaVA. I recommend that the authors implement ToVE within the LLaVA setting to demonstrate its effectiveness in broader, more contemporary scenarios.\n\n4. The datasets used for comparison are somewhat limited. I suggest adding more datasets, such as GQA, TextVQA, and VizWiz, to provide a more comprehensive evaluation. Additionally, Table 3 should also report the CIDEr score for COCO.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5tqXtuXAo9", "reviewer": "Reviewer_B56D", "summary": "This paper presents a novel way to assimilate knowledge from different visual experts trained for different visual tasks into a pre-trained ViT  for solving visual language tasks. The idea of expert knowledge assimilation might not be new, but it is not trivial to align depth, surface normal or edge information along with a clip-based ViT model for vision language tasks. The authors achieve it by a carefully constructed architecture pipeline, where the experts and the ViT model are frozen, a token-based gating network selects which expert to distil knowledge from into that visual token, and then a language decoder utilizes these modified tokens for the final task. They further go on to merge all the expert knowledge into their vision encoder to relieve using multiple experts during inference.", "questions": "1. The paper is not written that carefully. I would suggest the authors to properly rewrite the paper considering grammatical mistakes, notation inconsistencies, etc.\n\n2. How is a token-level knowledge extracted from low-level vision experts? This is not described properly either in main paper or appendix.\n\n3. What would be proper baselines to compare against, since BLIP and others do not use knowledge sharing from other experts? I think the tables need a bit of redesign.\n\nPlease address these comments as well as the weaknesses. I am inclined to accept this paper since I like the architecture but the paper needs quite a bit of rework.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of fusing knowledge from multiple experts into the vision encoder is good.\n\n2. The architecture design is novel, relating knowledge fusion to mixture of experts idea.\n\n3. Although similar to Prismer in idea, ToVE does have better results\n\n4. Ablations about types of knowledge merging, expert detachment are quite interesting and insightful", "weaknesses": "1. There are quite a few grammatical and sentence-construction mistakes throughout the paper, for e.g. \"The projection function of can be delineated as\" this sentence doesn't make sense. Also, the mapping function from d_k to d_lang should be about the feature dimension, since this is achieved by projection function MLPs, but it is written as token lengths, which is not the same as token feature dimension. Is this a writing mistake or the authors are changing token lengths, i.e., number of tokens?\n\n2.  \"Different from MOEs, which commonly activates the expert with the top-1 routing score,\" this is not true. The \"original\" MoE paper (Shazeer et al) did not have top -1, mainly switch transformer has top-1, and sparse MoE has top-k. Also, the authors employ a load balancing loss exactly similar to the Shazeer et al paper. So, stating that is a bit misleading. \n\n3. After eq 6, it says t_clip and t_fuse, which does not even appear in the equation.\n\n4. The method requires token-specific output from each expert (eq 1). So, how are tokens obtained for an image from low-level vision experts?\n\n5. In Table 1 and 2, ToVE is not sota but its results are marked bold. (e.g. BLIPv2 CIDEr score in Table1, InstructBLIP POPE-R score in table 2).  Since comparison is tough due to architecture modification, expert merging, it is essential to atleast show number of training parameters to understand the comparison validity", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Xq5mrBks5c", "reviewer": "Reviewer_6BHS", "summary": "This paper proposes a method called ToVE, which fuses visual information from different visual encoders as the input of language model in a vision-language model. Specifically, the authors use CLIP-ViT as the main input and use it to weight the tokens from other backbones. Furthermore, the authors proposed a distillation algorithm to teach knowledge from different visual backbones to CLIP-ViT. Experiments show that the proposed method can bring certain improvements to the performance.", "questions": "1) This article is confusing in several ways:\n\n    a) The author needs to further explain why setting the weights of tokens from unnecessary backbones to -inf in Formula 3 can improve computational efficiency. And authors should clarify if/how they are able to avoid activating all experts.\n\n    b) In the section ``Enhancing Exploration of Vision Experts,'' the authors use the concept of L_aux but only cite it, lacking an explanation of how to apply it in this method. The authors should provide some explanation in the main text.\n\n2. The motivation behind the manuscript and the source of the proposed method's performance need further clarification. From Fig. 8 and Fig. 9, it appears that the strong visual backbone, EVA, plays the primary role in most scenarios. Have the authors considered re-running the baseline experiment using EVA as the sole visual feature extractor? This will help clarify the contribution of the other experts beyond EVA and provide deeper insights into the model's overall performance.\n\n3. There are some methods that ToVE should compare with:\n\n    [1] Zi-Yi Dou, Yichong Xu, Zhe Gan, et.al.. An Empirical Study of Training End-to-End Vision-and-Language Transformers. CVPR 2022.\n\n    [2] Wonjae Kim, Bokyung Son, Ildoo Kim. ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision. ICML 2021", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) The proposed method is easy to understand.", "weaknesses": "1) The organization of this paper is somewhat confusing, especially the methods section.\n2) The computation overhead can be siginifcantly higher than other methods.\n3) The improvement of model capabilities may come from the introduction of a stronger visual backbone, rather than each model playing its own role in its professional field.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)