Selective Vision-Language Subspace Projection for Few-shot CLIP

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-21 15:54:10

Abstract: Vision-language models such as CLIP are capable of mapping the different modality data into a unified feature space, enabling zero/few-shot inference by measuring the similarity of given images and texts.
However, most existing methods overlook modality gaps in CLIP's encoded features, which is shown as the text and image features lie far apart from each other, resulting in limited classification performance.
To tackle this issue, we introduce a method called **S**elective Vision-Language **S**ubspace **P**rojection (**SSP**), which incorporates local image features and utilizes them as a bridge to enhance the alignment between image-text pairs.
Specifically, our SSP framework comprises two parallel modules: a vision projector and a language projector. Both projectors utilize local image features to span the respective subspaces for image and texts, thereby projecting the image and text features into their respective subspaces to achieve alignment. Moreover, our approach entails only training-free matrix calculations and can be seamlessly integrated into advanced CLIP-based few-shot learning frameworks. Extensive experiments on 11 datasets have demonstrated SSP's superior text-image alignment capabilities, outperforming the state-of-the-art alignment methods. The code is available at: [https://anonymous.4open.science/r/SSP-D3EC/main_our.py](https://anonymous.4open.science/r/SSP-D3EC/main_our.py)

Main paper: 
 introduction: 
 The topic of pre-trained vision-language models (VLMs) [16, 21] has attracted significant research interest due to their exceptional performance in various multimodal tasks [12, 44, 50] . Among these, CLIP (Contrastive Language-Image Pretraining) [35] stands out in classification tasks, especially in zero/few-shot scenarios. For a given test image, CLIP computes the similarity between the image feature and the text features of all class labels, in the form of "a feature alignment Recently, prompt tuning [52, 53] and adapter tuning [9, 41, 51] techniques have been applied in CLIP to further explore its generalization ability in few-shot tasks. These methods add learnable embeddings or adapter layers to either the visual encoder or textual encoder of CLIP. While they have improved the performance, they overlook the modality gaps existed in multi-modal models. As discussed in [24, 32, 40] and illustrated in Figure 1 (a), modality gaps refer to different modality embeddings, e.g., CLIP's encoded text features (pink cone) and image features (blue cone), are located in two separated regions in a hypersphere. To better understand these gaps, we use von Mises-Fisher (vMF) distribution [1, 6] to fit the text and image distributions, considering both modality features are normalized to unit length and lie on a unit hypersphere, where the parameters 𝝁 and 𝜅 in vMF are analogous to the mean and standard deviation of Gaussian distribution, respectively (details of vMF formalization can be found in the Supplementary Material). Figure 1(c ) summarizes the results, where the original text and image features show different distributions, e.g., a large angle between 𝝁 tex and 𝝁 vis , and a large ℓ 1 -norm between 𝜅 tex and 𝜅 vis . This phenomenon is unexpected, as paired text-image features are optimized during the pre-training stage to be closely located to each other while being separated from other paired image-text features. To further investigate these gaps, we visualize the activation maps by calculating the similarity maps between the text features and local image features (feature map) in Figure 2 , the second row shows that the text features focus on the unrelated regions, not just the foreground objects. This also demonstrates the gaps between the text and image features. In general, several reasons can cause [23] , while our SSP-aligned features primarily foreground objects.the modality gaps, broadly categorized as follows: (1) differences in downstream dataset distribution from the training data distribution [48] , (2) different random model initializations cause the different feature cones [24] , (3) CLIP model does not incorporate the finegrained relationship between text tokens and image patches [8, 25] , while these works on analyzing the modality gaps, they do not concentrate on CLIP's few-shot generalization capability.Based on the observation and analysis, we propose a trainingfree method called Selective Vision-Language Subspace Projection (SSP), that leverages the local image features as a bridge to align image and text features. Our method aims at reducing the modality gap so that the paired text-image features are no longer farther apart from each other, as shown in Figure1(b) (the pink cone and blue cone are closer). To be specific, our SSP consists of two parallel modules, namely, a vision projector and a language projector. In the vision projector module, we utilize the regions of local image features that are similar to the image features to create a unified vision subspace due to the common structural and textural elements present in images, and then all image features are projected onto this vision subspace to achieve alignment. The language projector module operates in a similar manner, where we use local image features to construct the language subspace for each class. Subsequently, text features extracted from CLIP are aligned by projecting them onto their respective language subspaces. During the inference stage, all projected features are employed to classify the test image. The effectiveness of SSP can be seen from the metric results in Figure 1 (c), by narrowing the modality gap, the text-image features aligned by SSP exhibit a higher cosine similarity, e.g., 0.7755 vs. 0.8325, as well as the smaller difference of 𝜅, and they also obtain a closer distribution distance by comparing the KL divergence. Besides, the visualized activation maps in the last row of Figure 2 show that text features processed by SSP can mainly focus on the foreground object. Overall, our SSP method only involves trainingfree matrix calculations, which can enhance the CLIP's capability in few-shot scenarios by improving the alignment between the paired text-image features.The main contributions of our SSP are summarized as follows:(1) We propose a training-free method SSP to reduce the modality gaps in CLIP's encoded features, which improves the CLIP's few-shot generalization ability. (2) We design vision and language projectors, which leverage regions of the local image features to align the image and text features via subspace projection. (3) Our SSP is flexible and can be applied to various CLIPbased methods, improving their performance across diverse benchmarks, even in state-of-the-art methods, e.g., an average accuracy improvement of 0.63% over APE [55] in 16-shot. 
related work: 
 In this section, we first introduce the Vsion-Language Models. Then we present CLIP with prompt tuning and adapter tuning in the few-shot scenario and compare our SSP with related methods.VLMs have gained significant advances in recent years [26, 49] . These models bridge the modalities of vision and language and are typically pre-trained on a large dataset. With image-text pairs, VLMs use an image encoder and a text encoder to extract image and text features, and then learn the vision-language correlation by using certain pre-training objectives. Moreover, a range of VLMs, including CLIP (Contrastive Language-Image Pretraining) [23, 35] , CoCa (Contrastive Cross-modal Learning) [45] , and BLIP (Bidirectional Learning of Image and Text Priors) [20] , can be leveraged for various downstream tasks, such as object recognition [51, 54] , object detection [10] , and image captioning [2, 30] . For instance, CLIP is pre-trained on a vast dataset of web-based image-text pairs and learns to align their representations through contrastive loss, which enables it to recognize unseen data by matching the embeddings of any given images and texts. In this paper, we aim to use CLIP to address few-shot classification problems. The overview of our approach. The training images and extended labels are sent to the frozen visual encoder and textual encoder to extract features, respectively. Subsequently, the related local image features (features maps) are employed to construct the vision subspace and language subspaces, which are performed to align the extracted image and text features through subspace projection. Finally, a projected testing feature along with projected training features are inputted into the classification framework to predict results.The concept of prompt tuning is initially introduced in the natural language processing area, which refers to utilizing a fixed part of the text input as learnable embeddings and fine-tuning its parameters based on the downstream task data. CoOp [53] uses learnable word embeddings to generate context prompts automatically, eliminating the need for manual prompt templates. CoCoOp [52] proposed a meta network to learn image features that served as conditions added to prompt embeddings to further enhance the model's generalization ability. ProDA [27] aims to learn the distributions of the prompt embedding. ProGrad [54] used zero-shot prediction results to direct the model gradient update, preventing conflicts between few-shot models and general knowledge while mitigating overfitting issues. However, the above-mentioned methods, although they enhance the classification performance of CLIP in few-shot scenarios, involve the introduction of learnable parameters and increase training consumption. In contrast, our SSP only involves matrix calculation and does not introduce any learnable parameters.Adapter tuning techniques are applied in VLMs [9, 41, 51, 55] to enhance downstream generalization ability by freezing the original model parameters and only updating the parameters of added adapter module. CLIP-Adapter [9] utilized a fully connected layer to adapt the features outputted from the frozen CLIP. Tip-Adapter [46] leverages a cache model to measure the relations between image and text features to construct a classifier based on few-shot training data. APE [55] represents an enhanced version of Tip-adapter, selecting the most discriminative feature channels via statistical analysis. SuS-X [41] relies on the category names from the training set to generate image samples based on stable diffusion [37] . Cross-Modal Adapter (CMA) [17] achieves cross-modal interaction by sharing adapter weights between two modalities. While the aforementioned methods adapt text-image features for the downstream tasks, they do not account for their modality gaps. Our SSP method strives to reduce the gaps to better align the paired text-image features and can be integrated into the aforementioned methods. 
methodology: 
 In this part, we provide a detailed explanation of our methodology. Firstly, we offer a brief overview of the zero-shot inference ability of CLIP in Section 3.1. Then, we delve into the specifics of our vision projector, discussed in Section 3.2, and language projector discussed in Section 3.3. Following that, we present the classification process of SSP in Section 3.4. Lastly, we conduct a comparative analysis between our SSP approach and related methods in Section 3.5.The pre-trained CLIP model can be adapted to the downstream tasks, i.e., zero-shot classification. This process involves extending the "[class name]" to the template "a photo of a [class name]". Subsequently, the image feature 𝒇 ∈ R 𝑑 and the class-extended text feature 𝒕 𝑖 ∈ R 𝑑 are extracted from CLIP's visual encoder and textual encoder, respectively. The classification is determined by the cosine similarity ⟨𝒇, 𝒕 𝑖 ⟩:EQUATIONwhere 𝑁 represents the number of classes. In the few-shot setting of CLIP, there are given 𝑁 classes, each class containing 𝐾-shot training images. In our method, apart from the encoded image feature 𝒇 and the text feature 𝒕, we also exploit local image features 𝒙 ∈ R ℎ𝑤×𝑑 , where ℎ and 𝑤 represent the height and width of the local image feature, respectively. The overview of our method is depicted in Figure 3 . We select local image features that exhibit strong correlations with both image and text features, respectively, as determined by cosine similarity. Then we utilize these selected local features to construct the vision subapace and language subspaces, respectively. In the inference stage, the test image feature is projected into the vision and language subspaces, respectively, and then the projected training text-image features and the projected test features are sent to the classifier to predict the result.In the vision projector module, the cosine similarity between the image feature and the local image features is calculated for each sample in the training set as follows:EQUATIONHere, 𝒇 𝑖,𝑗 is the image feature of 𝑗-th sample from class 𝑖, and 𝒙 𝑖,𝑗 denotes the corresponding local image features. 𝐾 indicates the number of samples within each class. 𝒔 𝑖,𝑗 ∈ R ℎ𝑤 denotes the vector of similarity scores, with each element indicating the correlation between the image feature and the local region feature. Previous studies [23, 29] have demonstrated that local image features play a crucial role in capturing object and semantic information for vision and language, respectively. Consequently, a higher value in 𝒔 𝑖,𝑗 indicates that this local region contains more discriminative information for correct classification. Conversely, a smaller value suggests that the region includes less useful or irrelevant information for the corresponding class. Based on these insights, we utilize regions with top-𝑄 largest values to construct the vision subspace:EQUATIONwhere 𝑄 is a hyper-parameter validated in Section 4. EQUATIONX comprises the informative features for vision patterns, and we decompose the X by Singular Value Decomposition (SVD):EQUATIONwhere 𝑼 vis ∈ R ( (𝑁 +1)𝐾 ×𝑄 ) × ( (𝑁 +1)𝐾 ×𝑄 ) and 𝑽 vis ∈ R 𝑑 ×𝑑 is the left singular vectors and right singular vectors, respectively, and they are corresponded to the singular values 𝚺 vis ∈ R ( (𝑁 +1)𝐾 ×𝑄 ) ×𝑑 sorted in the descending order. We employ the principal vectors of 𝑽 vis , denoted as Vvis , to span the visual subspace [38] , and the corresponding projection matrix is calculated as [11, 47] :EQUATIONwhere 𝑷 vis = 𝑷 T vis ∈ R 𝑑 ×𝑑 . Depending on the constructed vision subspace and projection matrix, we align the image features via subspace projection:EQUATIONwhere 𝑭 = [𝒇 1,1 , • • • , 𝒇 𝑁 ,𝐾 ] ∈ R 𝑑 ×𝑁The language projector module has the same effects as the vision projector, and they exhibit a similar procedure. The cosine similarity between the text feature and local image features is calculated as:EQUATIONwhere 𝒕EQUATIONwhere 𝑚 = {𝑚 1 , 𝑚 via SVD, as shown below:EQUATIONwhere 𝑼 𝑖 tex and 𝑽 𝑖 tex are the left and right singular vectors, respectively, corresponding to the singular values 𝚺 𝑖 tex sorted in descending order. Ṽ𝑖 tex denotes the primary singular vectors of 𝑽 𝑖 tex , which forms the basis for the 𝑖-th language subspace. The corresponding projection matrix is denoted as 𝑷 𝑖 tex = Ṽ𝑖 tex ( Ṽ𝑖 tex ) T ∈ R 𝑑 ×𝑑 . Subsequently, the text features are projected by their corresponding projection matrix as follows:EQUATIONThe projected text features T and the projected image features F belonging to the same class lie closely to each other, as depicted in Figure 1 (b), Thereby, these paired text-image features are utilized for classifying test images during the inference stage.Our SSP doesn't involve the parameters training process and can seamlessly build on top of existing adapter-based methods, such as LFA [32] , Tip [51] , and APE [55] , to achieve improved classification results, as summarized in Table2 and Table 3 . Specifically, given a test image 𝒇 test ∈ R 𝑑 , it's projected into the vision subspace and language subspace, respectively. The vision subspace projection process is straightforwardly calculated as:EQUATIONwhere 𝒇 vis test denotes the projected feature. However, for the language subspace projection, we encounter a challenge since we lack categorical information to determine the appropriate language projection matrix to use. To address this, we rely on projection theory [47] to choose the language projection matrix based on minimizing the ℓ 2 norm of the orthogonal projected features:EQUATIONIf the 𝒇 test lies in the 𝑖-th language subspace entirely, the projection operation doesn't change the norm of the test feature, i.e., 𝒇 test = 𝑷 𝑖 tex 𝒇 test . After aligning the test feature via our vision and language subspace projection. The aligned test features along with the aligned image and text features are inputted into the various classification frameworks the get the prediction. For example, if we choose the LFA [32] as the classifier, we calculate the transformation matrix 𝑾 based on aligned image features F and text features T . We then use 𝒇 tex test to compute the final classification logits, given by T •𝑾 •𝒇 tex test . If we choose the Tip [51] or APE [55] as the classification frameworks, we utilize both 𝒇 vis test and 𝒇 tex test to calculate the prediction results, which are given by F • 𝒇 vis test + T • 𝒇 vis test .Our method focuses on aligning the image and text features and can be easily built on top of various classifiers as described above. This sets our SSP approach apart from adapter-based methods such as Tip [51] , APE [55] , and LFA [32] . Although these methods operate on adapting the feature encoded from CLIP, they can also be viewed as classification frameworks due to their direct calculations between the training and testing samples. As shown in Figure4, the adapterbased methods are depicted in the green box at the bottom. They employ various techniques to calculate the relationship among 𝑻 , 𝑭 , 𝑳, and the test sample 𝒇 test to achieve classiffication, where 𝑳 signifies the one-hot labels for 𝑭 . In contrast, our method, illustrated in the gray section at the top, leverages local image features 𝑿 as a bridge to align the image features, text features, and test features, resulting in T , F , 𝒇 vis test , and 𝒇 tex test . These aligned features are substituted for the original ones in the computations to derive the classification results. 
experiments: 
 In this section, we first describe the experimental settings, and then we conduct ablation studies to analyze the effectiveness of different components within our method. Additionally, we compare the performance of our approach with other state-of-the-art (SOTA) methods. Finally, we evaluate the robustness of our method by testing it on out-of-distribution datasets. The experiments aim to address the following research questions (RQ): RQ1: What is the optimal number of selected regions for image features and text features, respectively? RQ2: How many principal singular vectors should be used to construct the vision projection matrix and language projection matrix, respectively? RQ3: What is the impact of the projection operation on the classification results? RQ4: How does the performance of our approach compare to that of state-of-the-art methods? RQ5: What is the generalization ability of our method? RQ6: How does the computational efficiency of our method compare to that of other methods? By addressing these research questions systematically, we aim to provide a comprehensive evaluation of the effectiveness and robustness of our proposed methodology.Datasets. In our experimental evaluation, we conducted experiments on 11 widely-used image classification benchmarks, covering a diverse range of object, scene, texture, and fine-grained categories. The datasets used in our experiments comprise ImageNet [5] , Cal-tech101 [7] , DTD [4] , EuroSAT [13] , FGVCAircraft [28] , Flowers102 [31] , Food101 [3] , OxfordPets [33] , StanfordCars [19] , SUN397 [43] , and UCF101 [39] . Additionally, we adopted variants of ImageNet, namely, ImageNetV2 [36] , ImageNet-Sketch [42] , ImageNet-A [15] , Accuracy (%) and ImageNet-R [14] , to assess the out-of-distribution generalization ability of methods, followed by [52] . Implementation Details. Our experiments are conducted using pre-trained CLIP [35] , where the visual encoder is adopted as ResNet-50 (RN50), and the textual encoder is a Transformer. The weights of these encoders remain frozen during both training and inference stages. We follow the previous works [51] to set 𝐾 = 1, 2, 4, 8, and 16, respectively, and follow the data preprocessing protocol in CLIP, including random cropping, resizing, and random horizontal flip. To incorporate textual information, we insert the class names into the standard templates, (e.g., "a photo of [class name]"). In comparisons, we evaluate our method against LFA [32] , Tip [51] , and APE [55] . To ensure a fair comparison, we use the same classification approach as these methods, when comparing with APE, we utilize the textual prompt provided in CuPL [34] . Training Details. When compared with LFA [32] , our approach follows the same fine-tuning stage described in [32] , using AdamW [18] optimizer with a cosine scheduler [32] , a learning rate of 5e-4 and a weight decay of 5e-4, for 50-200 iterations, and a cosine scheduler.In this section, we use RN50 on ImageNet [5] to evaluate the contributions and effectiveness of different components in our approach.The selected regions of the local image features are crucial in our method, as they determine which salient regions of the image contribute to the projection subspace. These selected local features should contain discriminative patterns relevant to the category while excluding irrelevant areas. To identify the optimal number of regions for vision subspace and language subspace, we vary the number of 𝑄 and 𝐶 from 15 to 49, respectively. The results are depicted in Figure 5 (a) and Figure 5 (b). We observe that as 𝑄 and 𝐶 increase, the averaged accuracy initially increases and then decreases. The highest accuracy is achieved in the range of 35-45. Furthermore, we analyze the disparity between the selected regions for image and text, respectively. To achieve this, we calculate the similarity between the text features and the selected regions based on image features, as well as the similarity between the image features and the selected regions based on text features. The results are shown in Figure 5 (c) and Figure 5 (d). In Figure 5 (c), as the regions become more complete (blue curves), the similarity between the selected region features and the text label feature (red curve) decreases. Similarly, in Figure 5 (d), as the number of selected regions increases, the similarity between the selected region features and image features (blue curve) also decreases. These observations and results indicate that the responded regions for image and text features are different, and selections should be performed separately for each of them.The effects of subspace construction ( RQ2). In this experiment, we investigate the effects of using different numbers of singular vectors in the construction of the vision projection matrix (𝑷 vis ) and language projection matrix (𝑷 𝑖 tex ). Considering a feature dimension of 𝑑 = 1024, we vary the number of singular vectors from 550 to 1000. The results are presented in Figure 6 (a) and Figure 6(b) . Firstly, we observe that using a small number of singular vectors in both vision and language subspace constructions leads to poor classification performance. To be specific, for the vision subspace, when the number exceeds 700, the results converge across all shot settings. This indicates that using 700 singular vectors is sufficient to span the entire vision subspace. In the case of the language subspace, we observe that the projection operation brings noticeable 6(d) . We find that the vision subspace projection error decreases significantly when the number of singular vectors exceeds 700, while the language subspace projection error gradually decreases. However, when the number exceeds 1000 for the language subspace, the projection error decreases to almost zero. This is because the projection matrix (𝑷 𝑖 tex ≈ 𝑰 ) is nearly an identity matrix, resulting in an invalid projection operation (𝒇 test = 𝑰 • 𝒇 test ) and a projection error close to zero ((𝑰 -𝑷 𝑖 tex )𝒇 test ≈ 0). Based on the above analysis, we choose to employ 900 singular vectors for both constructing the vision subspace and the language subspace.The effects of vision and language projector ( RQ3). In this ablation experiment, we aim to demonstrate the impact of vision projector and language projector individually. The performance outcomes across different shot settings are summarized in Table 1 .The results indicate that both vision subspace projection and language subspace projection lead to improvements in classification performance compared to the baseline (CLIP). Specifically, under 𝐾 = 16, vision projection yields a 3.61% accuracy improvement, while language projection achieves a 3.01% accuracy improvement. Furthermore, it is worth noting that combining both projection operations yields the best performance, as indicated by the last row of Table 1 . This outcome validates the effectiveness of utilizing both vision and language subspace projections in our method.We conducted a comprehensive comparison of our method with the latest approaches, including CoOp [53] , TaskRes [46] , GraphAdapter [22] , LFA [32] , APE [55] , and Tip [51] , across 11 image classification datasets. We first perform experiments using different visual encoders, including RN50, RN101, ViT-B/32, and ViT-B/16, on the Ima-geNet with a 16-shot setting. The results, as summarized in Table 2 , indicate that our SSP method leads to performance improvements compared to Tip, LFA, and APE. Notably, when built upon the APE method, our SSP outperforms all other methods. Furthermore, we extended our comparisons to evaluate our method across different shot settings on various datasets, with the results outlined in Table 3. While our method can not achieve the highest performance on some individual datasets, it consistently outperforms the other methods when considering the average accuracy across all datasets. This demonstrates the robustness of our SSP method. Specifically, compared to LFA, our method achieves a significant average accuracy improvement of 3.4% in the 1-shot setting. Furthermore, even when compared to APE, our method consistently exhibits slightly better performance across all shot settings. These results emphasize the effectiveness of our approach in improving few-shot image classification performance. For further comparisons, we also evaluated our method against Tip-F [51] and APE-T [55] . The detailed results of these comparisons can be found in the Supplementary Material.To assess the generalization ability of our method, we conducted an experiment using the ImageNet dataset as the source dataset, which provides 16-shot training images for each category. We then tested our method on four different variants of the ImageNet dataset: ImageNetV2 [36] , ImageNet-Sketch [42] , ImageNet-A [15] , and ImageNet-R [14] . These test datasets share the same class labels as ImageNet but exhibit semantic gaps. The classification results of our method on these test datasets are summarized in Table 4 . Our method achieved a significant improvement of 0.73% over the LFA method specifically on the ImageNet-R dataset. Furthermore, our method demonstrated an average performance improvement of 0.38% across all out-of-distribution (OOD) datasets. These results demonstrate our SSP is robust in domain adaptation.We conducted a computation efficiency analysis by comparing the computation time and parameters between our SSP and other approaches on the ImageNet dataset, under the 16-shot setting. The results are summarized in Table 5 . It's worth noting that our SSP does not introduce any learnable parameters and primarily incurs computation time during matrix operations. To align image features, SSP requires a single SVD operation, typically taking a few seconds that could be ignored. As for text features alignment, the number of SVD operations depends on the total categories in the dataset (e.g., 1000 categories in ImageNet). When combined with LFA, Tip, or APE, our SSP needs nearly an extra 1 minute computation time, but it consistently improves the classification accuracy, particularly by 0.72% over Tip. 
conclusion: 
 In this paper, we have proposed a method named Selective Vision-Language Subspace Projection to align the different modality features extracted for pre-trained CLIP through subspace projection. This alignment strengthens the generalization ability of CLIP in the few-shot scenarios. Our SSP is seamlessly integrated into various classification frameworks and does not introduce any additional learnable parameters. The extensive experiments validate the effectiveness of our proposed method. However, it's worth noting that the performance improvements are limited on the ImageNet dataset. In our future work, we plan to explore more general alignment techniques by incorporating other pre-trained models, such as large language models, diffusion models, etc.  
 
 Appendix:  


Decision: The manuscript introduces the so-called Selective Vision-Language Subspace Projection (SSP) method to address modality gaps in vision-language models like CLIP. The main idea is to utilize local image features for better text-image alignment. The proposed SSP framework include a vision projector and a language projector, which are used to enhance alignment through training-free matrix calculations and integrate seamlessly into existing few-shot learning frameworks. Extensive experiments demonstrate SSP's superior performance over state-of-the-art methods. 

The manuscript received two borderline accept and two accept ratings. Strengths include the innovative, practical approach, comprehensive evaluation, and code availability, while weaknesses involve implementation details and concerns from borderline reviews.  The authors' rebuttals partially addressed these weakness. After considering these factors, I recommend accepting this paper. Authors should add detailed implementation explanations, address reviewer concerns, and highlight future research directions in the final version.

Reviews: review: paper_summary: paper_summary: This manuscript introduces Selective Vision-Language Subspace Projection (SSP) to solve the alignment issue in the vision and language feature spaces of image-text pairs. The proposed SSP utilizes vision and language projectors to align the image and text features via subspace projection.
strength_weakness: strength_weakness: 1. This manuscript focuses on an important alignment issue in existing CLIP-based methods and introduces an effective model by incorporating local image features and subspace projector to tackle the issue.
2. The experimental results demonstrate the effectiveness of the proposed approach. And the visualization examples also show a better concentration on important regions. 
3. The ablation experiments presented in the manuscript are comprehensive.
limitations: limitations: 1. Some unclear explanation: the manuscript lacks some explanation about the details of the methodology. For instance,
  (1) what are the candidate regions from which the selected regions are chosen? Could the authors provide more details about the generation process of such regions?
  (2) According to Figure 1, the proposed method can largely alleviate the misalignment issue, but the final results only show a relatively limited improvement. Besides, the original CLIP-based methods can also achieve good results with such large modality gap. It seems that removing modality gap brings limited improvement. Could the authors give an explanation.
paper_summary: paper_summary: The paper introduces a technique titled Selective Vision-Language Subspace Projection, which aligns the multimodal features derived from the pre-trained CLIP model via a subspace projection approach. This alignment strengthens the generalization ability of CLIP in the few-shot scenarios. The proposed module can be effortlessly incorporated into a variety of classification frameworks, adding no extra trainable parameters. Comprehensive experimental results validate the effectiveness of the proposed method across diverse benchmarks.
strength_weakness: strength_weakness: - The proposed method has good generality and has demonstrated effectiveness on multiple datasets.
- The paper is well-written and easy to follow.
- The experiments are comprehensive, and the results demonstrate that the proposed method can be effectively integrated into various classification frameworks, leading to performance improvements.
limitations: limitations: - When constructing the language subspaces, local image features extracted by the visual encoder are used. Can we still call them language subspaces? This somewhat undermines the persuasiveness of Figure 1 and Figure A.
- According to Figure 2 (middle), the text features focus on unrelated regions, and these regions are selected via Top-C. Why can language subspaces constructed with the assistance of local image features from these unrelated areas achieve the results shown in Figure 2 (bottom)?
- In Table 2 and Table 5, the performances of APE and APE+SSP are inconsistent.
paper_summary: paper_summary: This paper proposes a training-free method called Selective Vision-Language Subspace Projection (SSP) method for few-shot CLIP. The approach combines two parallel modules: a vision projector and a language projector to achieve alignment. Experimental results on several benchmark datasets are reported.
strength_weakness: strength_weakness: A novel problem definition and a well motivated solution.
The paper is overall well-written and easy to follow.
Both quantitative and qualitative interpretations are discussed in detail.
limitations: limitations: Some concerns and suggestions to improve this work are as follows:
1. The methods mentioned in the abstract are unclear, and there is also a lack of corresponding explanation in Section I, e.g. “However, most existing methods overlook modality gaps in CLIP’s encoded features, which is shown as the text and image features lie far apart from each other, resulting in limited classification performance”.
2. The summary of the contributions in the manuscript is not sufficiently clear. It would be beneficial to explicitly outline the novel aspects and specific contributions of the proposed method or findings.
3. The paper lacks specific details regarding the practical implementation.
4. Reference formatting is inconsistent. Please carefully revise the references, ensuring consistent use of either full names or abbreviations for journal and conference names throughout the manuscript.
paper_summary: paper_summary: This paper proposes a selective vision-language subspace projection method to enhance the alignment between image-text pairs by utilizing local image features. The proposed approach reduces the modality gaps in CLIP’s encoded features, which improves the CLIP’s few shot generalization ability. Sufficient experimental evaluations are provided.
strength_weakness: strength_weakness: 1. The paper summarized a comprehensive review of the current progress of related work and pointed out the limitations. 
2. The method has achieved significant improvements in experiments. 
3. This paper is well organized and written.
limitations: limitations: Some concerns and suggestions to improve this work are as follows:
1. The introduction and related work sections would benefit from including a more comprehensive set of references, particularly few-shot clip studies [1, 2], to provide a more thorough overview of the existing literature in the field.
[1] Domain Aligned CLIP for Few-shot Classification, Workshop on Applications of Computer Vision, 2024.
[2] Contrastive knowledge-augmented meta-learning for few-shot classification, Workshop on Applications of Computer Vision, 2023.
2. The main idea is easy to follow. The author proposes to use the regions of the local image features to align the image and text features via subspace projection. Could you please give some qualitative analysis to explain this? 
3. It is recommended that the authors check any potential citation errors and enhance the manuscript's overall quality.