VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-10 07:41:53

Abstract: Finetuning a pretrained vision model (PVM) is a common technique for learning downstream vision tasks.
However, the conventional finetuning process with randomly sampled data points results in diminished training efficiency.
To address this drawback, we propose a novel approach, Vision-language Collaborative Active Finetuning (VeCAF). 
With the emerging availability of labels and natural language annotations of images through web-scale crawling or controlled generation, VeCAF makes use of these information to perform parametric data selection for PVM finetuning. VeCAF incorporates the finetuning objective to select significant data points that effectively guide the PVM towards faster convergence to meet the performance goal. This process is assisted by the inherent semantic richness of the text embedding space which we use to augment image features. 
Furthermore, the flexibility of text-domain augmentation allows VeCAF to handle out-of-distribution scenarios without external data.
Extensive experiments show the leading performance and high computational efficiency of VeCAF that is superior to baselines in both in-distribution and out-of-distribution image classification tasks. 
On ImageNet, VeCAF uses up to 3.3$\times$ less training batches to reach the target performance compared to full finetuning, and achieves an accuracy improvement of 2.8\% over the state-of-the-art active finetuning method with the same number of batches.

Main paper: 
 related work 2.1 active learning: 
 The learning algorithm in active learning is allowed to choose the data from which it learns. There are two main selection criteria: uncertainty [3, 27, 44, 46] and diversity [1, 5, 40] . Uncertainty of the model can aid selection of the most difficult unlabeled data.Early works estimate the uncertainty with various heuristics such as posterior probability [21, 45] , entropy [17, 29] and classification margin [37] . Previous works [13, 31, 36 ] also formulate active learning as an optimization problem. They typically operate in a discrete space that trivially matches the sample distribution of a dataset [10, 15] . However, discrete optimization is harder to solve than the one in continuous space. Also, most previous methods are designed for a from-scratch training without the pretraining stage. Bengar et al. [4] reveals drawbacks of such setting without unsupervised pretraining. Xie et al. [41] addresses both shortcomings with the proposed continuous-space ActiveFT method that applies selected samples to the finetuning of the pretrained model in a single pass. VeCAF extends ActiveFT to the practical setting of data selection from a large labeled dataset. VeCAF framework enhances the training efficiency with a training objective-aware data selection and achieves optimal finetuning results with minimal training batches.Recent developments in language models, especially vision-language models [22, 23] demonstrate their effectiveness in aligning the embedding space of vision and language to achieve cross-modal generalization. For example, BLIP-2 [22] addresses the modality gap using a lightweight Querying Transformer, while Shikra [7] handles spatial coordinate inputs and outputs in natural language and excels in referential dialogue and general vision-language tasks. Many prior work exploit the connection between the image and text modalities, where they explore the use of language in training better vision models. For example, Ma et al. [30] leverage pretrained language models to design a distribution alignment objective. This objective guides the vision model to learn linguistic representations specific to the task under a semi-supervised setting. Similarly, Fahes et al. [11] utilize CLIP to optimize affine transformations of source domain features. This optimization aligns these features with the target text embeddings while preserving their content and semantics. In our work, we use the language embeddings of image captions to perform text-space augmentations, achieving better sample selection quality in active learning setting. 
proposed method: 
 This section introduces the details of the proposed VeCAF framework. As illustrated in Figure 2 , we start by selecting a subset of training data with the PVM image feature space using the proposed ODS method, as introduced in Section 3.1. Then, the selected samples pass through pretrained language model to get semantically-rich text embeddings, which augment the image features with our proposed CEA technique, as formulated in Section 3. {� � }The image features a man in a wetsuit holding a large tench fish in a body of water, possibly a lake. image features are used for PVM finetuning as candidates in future rounds of data selection. Finally, we show that VeCAF can overcome challenges of out-of-distribution data in Section 3.3.Consider a PVM 𝑓 (•; 𝑤) with 𝑤 weights and a user-defined finetuning objective L. Given a labeled training set D : {X, Y}, where X is the set of image 𝑥 and Y is the set of corresponding label 𝑦. The goal of our active data selection is to find a subset S : {X 𝑆 , Y 𝑆 } ⊂ D, such that PVM finetuning with this subset for a fixed number of iterations leads to the largest reduction of the training objective. Formally, we formulate the optimization problem of our objective-aware data selection (ODS) process asEQUATIONwhere𝛿 S = ∇ 𝑤 E 𝑥 𝑠 ,𝑦 𝑠 ∈ S [L (𝑓 (𝑥 𝑠 ; 𝑤), 𝑦 𝑠 )]is the gradient accumulated by finetuning with S and 𝛽 is the learning rate.In a practical setting, we cannot compute ∇ 𝑤 L (𝑓 (𝑥; 𝑤), 𝑦) for each training example (𝑥, 𝑦) before the data selection, as the gradient computation cost almost equals to the cost of full finetuning, which contradicts the purpose of active data selection. In this sense, we make an assumption that a data point with a larger loss contributes more to the convergence speed of the model.Intuitively, this assumption leads a naive data selection policy of selecting the data points with Top-𝐾 training losses. However, previous active learning work [41] has discovered that the diversity of the selected data is also important in order to cover corner cases in the dataset and to avoid overfitting. Therefore, we design the ODS algorithm with the following principles: 1) data point with a larger loss L (𝑓 (𝑥; 𝑤), 𝑦) shall be selected with a higher probability; and 2) maintaining the diversity of data points selected in S. Analytically, we formulate our ODS objective asEQUATIONwhere 𝐷 𝐾𝐿 (•||•) is the KL divergence, 𝑅(•) is a diversity metric,The distribution of the selected data 𝑝 𝑆 (S) is determined by the data selection model. As a sanity check, the naive "Top-𝐾 training losses" serves as the optimal solution for Equation 2 without considering diversity when 𝜆 = 0.To enable continuous optimization, we optimize Equation 2 using a parameterized data selection model 𝜃 𝑆 . For the simplicity, we follow previous work [41] to model the data selection distribution 𝑝 𝑆 (S) in the lower-dimension image embedding space, where embedding 𝑒 is produced by the PVM from the input 𝑥 at a hidden layer. 𝜃 𝑆 consists of 𝐾 centroids in the image embedding space, each selecting the nearest data point. We define the probability of a data point 𝑥 𝑖 with the corresponding embedding 𝑒 𝑖 being selected asEQUATIONwhere ⟨•, •⟩ denotes the cosine distance, and 𝜃 𝑐 𝑖 𝑆 is the closest centroid in 𝜃 𝑆 to 𝑒 𝑖 . We derive the parameterized distribution distance 𝐷 (𝜃 𝑆 ) := 𝐷 𝐾𝐿 (𝑝 L ||𝑝 𝑆 ) for optimization objective Equation 2 EQUATIONwhere 𝐶 and 𝛼 are the constants omitted in the formulation.For the diversity metric 𝑅(•), we follow the diversity regularization term proposed in [41] asEQUATIONBy substituting Equation 5and Equation 6into Equation 2 and by removing constant terms, our final objective in terms of 𝜃 𝑆 parameters can be written asEQUATION)The optimization on 𝜃 𝑆 is conducted via gradient descent. To further resolve the dependency of the optimized data selection model to its initialization as observed in [41] , we consider an independentlyinitialized data selection model ensemble[], denoted as {𝜃 𝑒 } 𝐸 𝑒=1 , in the optimization. We empirically set 𝐸 = 5 to balance the performancecost tradeoff. After optimizing each data selection model 𝜃 𝑒 independently using Equation 7, we can then remove the bias in the initialization by utilizing the mean 𝜇 and covariance Σ calculated on 𝜃 * 𝑒 . Specifically, given an optimized data selection model 𝜃 * 1 , we achieve the final unbiased data selection model as 𝜃 * 𝑆 = 𝜃 * 1 -Σ -1 (𝜃 * 1 -𝜇). The optimization in Equation ( 7) is performed before each finetuning "loop" with the current model weights 𝑤. The training data with the closest embedding to each 𝜃 𝑆 centroid is selected to form a "finetuning set" with 𝐾 elements. Then, this set is used to finetune the PVM until the next loop starts. We update ODS weights 𝑤 after a predetermined number of training batches.We empirically find that samples selected by the ODS process tend to consist of multiple objects in both foreground and background, which is a result of coarse image embeddings produced by the PVM. To further improve quality of image embeddings for both data selection and finetuning, we propose Cross-attentive Embedding Augmentation (CEA). Given a training image, CEA leverages a pretrained text encoder transfer the image caption to the corresponding text embedding. Then, the text embedding is used to augment the image embedding with attention-based method.Given selected sample {𝑥 𝑆 𝑖 } 𝐾 𝑖=1 in the previous ODS run, the corresponding image caption can be denoted as {𝑐 𝑆 𝑖 } 𝐾 𝑖=1 . Then we feed the caption into a frozen text encoder, e.g. BERT [9] , to convert the captions into the text embeddings {𝑡 𝑖 } 𝐾 𝑖=1 . We use text embeddings with the same dimensions as image embeddings for easier fusion.Inspired by [16] , CEA is conducted as mapping the image embedding 𝑒 𝑖 towards the corresponding text embedding 𝑡 𝑖 . To decide the 𝛼 𝑖 = Softmax( 𝑒 𝑗 • 𝑡 𝑗 ∥𝑒 𝑗 ∥ 2 ∥𝑡 𝑗 ∥ 2 ) = exp 𝑒 𝑖 •𝑡 𝑖 ∥𝑒 𝑖 ∥ 2 ∥𝑡 𝑖 ∥ 2 𝐾 𝑗=1 exp 𝑒 𝑗 •𝑡 𝑗 ∥𝑒 𝑗 ∥ 2 ∥𝑡 𝑗 ∥ 2 . (8)The attention score 𝛼 𝑖 helps to derive the augmented embedding 𝑒 𝑎𝑢𝑔 𝑖 using the image embedding 𝑒 𝑖 and the corresponding text embedding 𝑡 𝑖 as 𝑒EQUATIONwhere 𝜂 is the fixed step size. The proposed CEA method enriches the semantic information and improves performance after finetuning as shown in experiments. We further present the pseudo code of VeCAF in Algorithm 1, which specifies the complete procedure of the proposed Vision-language Collaborative Active Finetuning (VeCAF) framework.In addition to achieving more efficient finetuning on in-distribution (ID) tasks, one of the additional benefits of introducing the text modality is the ability to artificially modify the corresponding image captions as a semantic augmentation. By leveraging the strong language capability provided by the text encoder, we can implicitly alter our requirements for the selected data. This capability enables us to facilitate domain transfer to out-of-distribution (OOD) scenarios with only ID data. For example, leveraging the CEA technique, we can add descriptive phrases like "It's a {target_domain} day!" to the image captions. This modification shifts the PVM image embedding towards the "snow" distribution while highlighting the ID sample images that have snowflakes or similar patterns during the data selection process, as demonstrated in Figure 3 . By incorporating such textual cues, we can guide the selection process toward images that possess specific characteristics or attributes, so the finetuned model can better generalize in out-of-distribution scenarios. 
experiments: 
 In this section, we present results of conducted experiments on multiple image classification tasks. First, we introduce the experiment setup and evaluation protocols in Sec. 4.1. Main results in Sec. 4.2 show the effectiveness and efficiency of the proposed VeCAF when compared to other sample selection methods. Particularly, we demonstrate advantages of the proposed VeCAF in both in-distribution and out-of-distribution scenarios where it improves the finetuning efficiency by up to 3.3× and 2.7% accuracy on ImageNet-1K. In Sec. 4.3, we present further analysis including ablation studies, Ve-CAF generality and qualitative results.Datasets. For model training, we conduct experiments using three image classification datasets: CIFAR-10 [19] , class-unbalanced Cal-tech101 [12] , and ImageNet-1K [8] . Details of the dataset can be found in Appendix A. For out-of-distribution evaluation, we evaluate on ImageNet-C [14] which consists of various synthetically generated corruptions applied to the ImageNet validation set.Implementation details. In our main experiments, we use the DeiT-B model [38] pretrained with DINO [6] on ImageNet-1K [8] as the PVM for finetuning. Additionally, we present experiments for different PVM architectures and model sizes in Section 4.3 to demonstrate VeCAF generalizability. For all experiments, we resize input images to 224×224 to ensure consistency during both the data selection and the finetuning. In the ODS process, we optimize the data selection model parameters 𝜃 using the Adam optimizer with a learning rate of 0.001 until convergence.We adopt the standard protocols outlined in [38] to finetune the DeiT-B model. We use the SGD optimizer for supervised finetuning with the following hyperparameters: 5e-4 learning rate (lr), 1e-4 weight decay and 0.9 momentum. The total number of finetuning batches for CIFAR-10, Caltech101 and ImageNet is set to be 750, 1500 and 12500, respectively. The total batch numbers are set such that baseline active learning methods can converge to a reasonable performance on each dataset. All experiments are conducted on two Tesla-A100 GPUs with a batch size of 256. We finetune the model following the settings provided in the official DeiT code 1 , with cosine learning rate decay applied. Baselines. We compare VeCAF with three active learning baselines LearnLoss [43] , TA-VAAL [18] , ALFA-Mix [32] , ActiveFT [41] and the conventional approach of full data finetuning. The details of the baselines are summarized as follows:• LearnLoss [43] predicts target losses for unlabeled inputs to select data points with potential incorrect predictions. • TA-VAAL [18] selects data based on an estimation of the data distribution of labeled and unlabeled pools, which is further enhances by incorporating a loss prediction ranking. • ALFA-Mix [32] employs interpolations between labeled and unlabeled instances to uncover unrecognized features, which leads to an efficient data selection policy with a closed-form solution to identify the optimal interpolation that induces changes in predictions.• ActiveFT [41] performs data selection in the embedding space of a pretrained model, where the selection is converted to a continuous optimization to match the distribution of the data pool while maintaining diversity within selected subsets. • Full Data finetuning (FT) randomly samples all data points from the training set to perform finetuning of the pretrained vision models following the conventional paradigm. In-distribution performance. We present the image classification results in Table 1 . The results demonstrate the effectiveness of the proposed VeCAF method when compared to other active learning approaches. Under the fair comparison of the multi-run setting, VeCAF consistently outperforms other methods across all three datasets, irrespective of the employed sampling ratios. Remarkably, even with low data sampling ratios, our method excels in selecting highly representative samples. When compared to full data finetuning with limited training batches, VeCAF achieves higher accuracy even with only 1% of the data being used for finetuning in each epoch. Additionally, it is noteworthy that the performance gain brought by VeCAF over previous active learning methods increases with the training set being more complex. For example, VeCAF gains 2.7% accuracy over ActiveFT on ImageNet-1K compared to less than 1% on CIFAR-10, which proves that the VeCAF is suitable for more complex learning tasks. In summary, these results demonstrate Ve-CAF to be more effective with the pretraining-finetuning paradigm when compared to previous active learning methods.Finegrained objective-aware training. The proposed ODS method offers additional flexibility to accommodate finegrained training objectives. For verification, we evaluate VeCAF under 1-vs.-all finetuning objective on multiple CIFAR-10 classes. Specifically, given a target class, we set the loss as a binary classification, with target class being positive and all others being negative. The modified loss is used in ODS optimization as in Equation (7) . A naive objectiveaware baseline of selecting samples with the largest loss is also included in the comparison. Results in Table 2 show that VeCAF performs better with the finegrained objectives for target categories, and the naive addition of top-𝐾 loss is the inadequate approach for data selection with finegrained objectives.Out-of-distribution generalization. We further conduct experiments using ImageNet-C validation set to verify the capability of VeCAF on helping a model finetuned only on the source domain to generalize towards out-of-distribution (OOD) scenarios. Specifically, we consider the domain adaptation scenarios of clear→snowy and clear→foggy. The results are presented in Table 3 .In these experiments, we finetune a PVM classifier using only uncorrupted (source) ImageNet data points. The only exception is CLIPstyler [20] domain images by leveraging the knowledge of pretrained CLIP models. Finetuning settings are the same in the previous experiments on ImageNet with 1% of the training data being used in each finetuning loop for ActiveFT and VeCAF. We can see from Table 3 that VeCAF consistently outperforms three baselines by up to 4.2%. CLIPstyler helps OOD generalization when compared to Full Data FT, but the presence of artifacts in the stylized images limits its performance. Despite that ActiveFT has good performance on in-distribution data, it underperforms in the OOD setting. This is a result of overfitting because ActiveFT selects data only based on the source domain distribution. On the other hand, the proposed domain-specific text embedding augmentation in VeCAF allows the source domain data that resembles the target domain style to be selected in the finetuning process (see Figure 3 ). This leads to better generalization in OOD scenarios.Efficiency analysis. To provide a deeper understanding of the potential training efficiency improvement introduced by VeCAF, we illustrate the required batch numbers to achieve the target accuracy (B2A) for different approaches on the three datasets. Specifically, we report training batches needed for each method with B2A@45,50 (i.e., top-1 accuracy of 45% and 50%) for ImageNet, B2A@65 for Caltech101, and B2A@90 for CIFAR-10, respectively. Figure 5 highlights the efficiency of VeCAF in comparison to other methods using such batch numbers. On ImageNet, VeCAF achieves 3.3× acceleration over full data finetuning (3075 v.s. 10250 batches) and outperforms other baselines with less batches as the target accuracy grows. Additionally, we plot the training loss as a function of the batch number for different approaches during the finetuning process in Figure 4 . This visualization further highlights the faster convergence and performance of VeCAF when compared to other methods. These figures provide a comprehensive view of the performance and efficiency of VeCAF, emphasizing its enhanced efficiency in achieving the desired accuracy levels.To verify that the amount of training batches serve as a good proxy for the overall computational cost of the finetuning process, we further analyze the overhead of using language encoder and performing data selection in each data selection loop in Table 4 . We follow the descriptions in the PyTorch report [24] to estimate the computational cost of training and inference. The CLIP-ViT-L text encoder model we use in the experiments requires about 11× the FLOPs of the DeiT model being finetuned, but only needs to be inferenced once in each data selection loop. This leads to the FLOPs overhead of CEA to be approximately 4.5% of the total finetuning cost. The resulting reduction ratio of total FLOPs brought by VeCAF (7.01/2.21 = 3.17×) is therefore similar to the batch number reduction ratio (10/3 = 3.33×). Note that this analysis is consistent across datasets as all data are resized to 224×224 for ViT inference. This result verifies the computational efficiency of VeCAF as shown by the batch number reduction in previous experiments. Generality of VeCAF. The proposed VeCAF framework can be used to finetune various PVMs with the help of different language encoders (LMs). Table 5 reports results on ImageNet-1K using various pairs of PVMs (DeiT-S/B [38] , Swin-Transformer-S/B [28] , and XciT-M/L [2] ) and LMs (BERT-L [9] , mT5-L [42] , and GPT2-L [34] ). VeCAF demonstrates its ability to adapt to different PVM model architectures and the flexibility to be used with various text encoder models. These results prove VeCAF to be a general active data selection method for efficient PVM finetuning.Embedding visualization. In Figure 6 , we present the image embedding visualization of the CIFAR-10 training set using UMAP dimension reduction. With each method selecting 1% of data from the training set, black dots represent the samples selected by both ActiveFT and VeCAF, red stars denote samples only selected by VeCAF, and blue stars denote samples only selected by ActiveFT. While maintaining the diversity of data selection, samples chosen by VeCAF appear to be closer to the boundaries compared to those selected by ActiveFT. This confirms that the proposed ODS helps to select important samples around the decision boundaries. This is a result of our selection strategy that incorporates training objective and, therefore, helps the PVM to learn the subtle differences between categories more efficiently during the finetuning phase.Ablation study. We first verify the importance of proposed ODS and CEA techniques in Table 6 . Specifically, ODS can be disabled by removing the L term in Equation (7) . ODS significantly improves classification accuracy, especially with limited data, and expedites Table 7 presents a study on the impact of the loop count in the multi-run selection setting using CIFAR-10 dataset. The total number of training batches is the same in all experiments with uniform division by the loop count. It is clear that the model performance is enhanced with the increase in the number of loops. However, performing additional loops of data selection leads to overhead in the total finetuning time. Therefore, we set the number of loops to three throughout our experiments to balance the trade-off between the model performance and the finetuning time. 
conclusions: 
 In this paper, we proposed novel active data selection framework called VeCAF that improves the efficiency of model finetuning with two major components. First, VeCAF finds a subset of training data that leads to faster convergence using the objective-aware selection model. Second, it utilizes the text-domain knowledge in pretrained language encoders to augment image embeddings during selection.Extensive experiments demonstrated advantages of the proposed approach such as more accurate in-distribution classification, robustness to out-of-distribution data and high computational efficiency. In future, we envision VeCAF can be extended to other multimedia domains with the potential of text-domain augmentation and by controllable active data generation for additional improvements. 
: 
 pretraining data and conserving valuable training resources during the finetuning stage, this paradigm has archived remarkable adoption in practical applications.In the real-world deployments, practitioners aim to adapt deep learning models to a certain scenario or tune a model towards a specific performance target with minimal efforts for data selection and with quick training. Training with all available downstream task data can be not only costly but also can lead to a biased or degraded performance in case of improperly collected data. This motivates the proposal of a data selection framework that can actively select the optimal data subset for finetuning. Previous work on active learning [4, 41] has shown the feasibility of PVM finetuning with only a small subset (e.g., less than 5%) of training data while achieving high performance metrics in downstream tasks. However, this line of work is often limited by the setting of low label availability which hinders its effectiveness to meet the user-specified objectives.With the growing feasibility to gather large amounts of images with labels and natural language captions in the target domain through web-scale data crawling [35] or controlled generation [25] , we find it is practical to explore a novel setting of active finetuning using annotated data. Then, we aim to select an optimal subset of training data for finetuning while having faster convergence and/or higher performance metrics as shown in Figure 1 . The selection can further be performed in a loop to accommodate the changing model performance during the finetuning process. To this end, we propose to perform an Objective-aware Data Selection (ODS) using a parameterized data selection model. This ODS model reweighs training data distribution according to the downstream objective and selects a subset that is both diverse and representative to the task.The pursuit of objective-awareness brings new challenges to the data selection. Intuitively, images with misleading object appearances and complicated backgrounds, as illustrated in Figure 2 , often provide more informative supervision. However, the image features extracted by the PVM may not fully capture all the semantic information present in the image. Therefore, PVM image features may miss useful information for the data selection and finetuning process as in previous ActiveFT [41] work. To address this limitation, we propose to leverage semantically rich language embedding spaces of the text encoders (e.g., CLIP [33] , mT5 [42] , BERT [9] etc.) in our novel Vision-language Collaborative Active Fine-tuning (VeCAF) approach. Specifically, we extract the text embeddings of the captions associated with each image. These captions may be sourced directly from original datasets such as COCO-Caption or alternatively generated by a multimodal Large Language Model (LLM) e.g., BLIP-2 [22] . Then, we propose a Cross-attentive Embedding Augmentation (CEA) to augment the image features extracted by the PVM such that the augmented features can focus more on the rich semantic information of the training samples. Therefore, the CEA facilitates both the active data selection and the finetuning process.Empirically, we demonstrate improved efficiency and performance across different scenarios. VeCAF is evaluated on three image classification datasets including CIFAR-10 [19], Caltech101 [12] , and ImageNet-1K [8] with the pretraining-finetuning paradigm where base models are pretrained on ImageNet-1K. Results on ImageNet demonstrate that VeCAF can significantly accelerate the PVM convergence speed to the target performance and saves up to 3.3× computational cost when compared to finetuning with all training set. Importantly, VeCAF also addresses out-of-distribution (OOD) scenarios, where we augment image features by target-domain text embeddings derived from the generated image captions. By leveraging the alignment between text and images embeddings, VeCAF increases the likelihood of selecting images that possess the characteristic features of the target domain from the training dataset as shown in Figure 3 . In addition, we verify the OOD generalization ability on the corrupted ImageNet-C dataset where VeCAF improves accuracy by over 6% when compared to state-of-the-art active learning methods. Our main contributions are summarized as follows:• We propose a novel framework, VeCAF, to improve computational efficiency of PVM finetuning using both the training objective and the language-embedded knowledge. • We propose the Objective-aware Data Selection (ODS), where a parameterized data selection model is optimized for the userspecified objectives and selects a subset that contributes to faster convergence and higher performance metrics. • We further employ pretrained language encoders with the proposed Cross-attentive Embedding Augmentation (CEA) to enrich semantic information in image features and to provide explicit semantic guidance for data selection and finetuning. 
 
 Appendix: https://github.com/facebookresearch/deit 


Decision: This paper ultimately received three positive "borderline accept" scores. After the rebuttal, all reviewers were satisfied with the authors' responses and the additional experimental results. Considering that the reviewers have no further questions, I recommend accepting this paper. I also suggest that the authors include the additional experiments in the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a VeCAF strategy to select significant data points and only uses these selected data for efficient downstream tuning. Extensive experiments show the leading performance and high computational efficiency of VeCAF.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to follow.
2. Using text embedding to augment the image embedding and improve out-of-distribution scenarios is interesting and insightful.
3. The work has conducted very thorough and comprehensive experiments, which is worthy of praise.
limitations: limitations: 1. The details on how text embedding can help with data selection and fine-tuning are not clear enough. Based on the information provided in the paper (lines 399-403 and Algorithm 1), it seems that the text embedding is used to further enhance the already selected images, rather than to help with the initial data selection process. This makes me a little unclear about what it role is in the data selection process.
2. Introducing text embedding may raise some fairness issues. Specifically, the use of additional external information leads to unfairness in performance comparisons. Furthermore, I would like to know if this also incurs additional computational resource overhead.
paper_summary: paper_summary: In this paper, the authors proposed novel active data selection framework called VeCAF that improves the efficiency of model finetuning with
two major components. First, VeCAF finds a subset of training data that leads to faster convergence using the objective-aware selection
model. Second, it utilizes the text-domain knowledge in pretrained language encoders to augment image embeddings during selection.
Extensive experiments demonstrated advantages of the proposed approach.
strength_weakness: strength_weakness: a）The experiment in this paper is thorough, and the presentation is relatively clear.

b)  On ImageNet, VeCAF uses up to 3.3× less training batches to reach the target performance compared to full finetuning, and achieves an accuracy improvement of 2.7% over the state-of-the-art active finetuning method with the same number of batches.
limitations: limitations: a) Please attach the publication time of the compared model  in the Table 1 for easy reading.

b)  The presentation of Figure 2 is not very clear compared to the corresponding textual expression,  and further polish is necessary.
paper_summary: paper_summary: This paper proposed a novel Vision-language Collaborative Active Finetuning (VeCAF) approach to efficiently select training subsets that are most significant to downstream finetuning objective. By introducing additional semantic information via pretrained language encoders to construct Cross-attentive Embedding Augmentation (CEA), VeCAF demonstrated superior performance and computational efficiency compared to baseline methods.
strength_weakness: strength_weakness: 1.	Text-domain augmentation in image data selection.
2.	Efficient and effective end-to-end active fine-tuning framework.
limitations: limitations: 1.	The concept of Objective-aware Data Selection (ODS) is like that of ActiveFT. It is recommended that the authors emphasize the difference between this part of the design and ActiveFT.
2.	The method for generating image captions for image classification datasets (CIFAR-10, Caltech101, and ImageNet-1K) is unclear. It is recommended that the authors add ablation experiments on the effect of image depiction quality on the performance of VeCAF.
3.	The generalizability experiment of the VeCAF method is incomplete. The effectiveness of VeCAF on other visual tasks (e.g., semantic segmentation) and backbone network structures (e.g., CNNs) is unknown.