FewVS: A Vision-Semantics Integration Framework for Few-Shot Image Classification

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 05:48:26

Abstract: Some recent methods address few-shot image classification by extracting semantic information from class names and devising mechanisms for aligning vision and semantics to integrate information from both modalities. However, class names provide only abstract information, which is insufficient to capture the visual details present in images. As a result, this vision-semantics alignment is inherently biased, leading to sub-optimal integration outcomes. In this paper,  we avoid this biased vision-semantics alignment by introducing CLIP, a natural bridge between vision and semantics, and enforcing unbiased vision-vision alignment as a proxy task. Specifically, we align features encoded from the same image by both the few-shot encoder and CLIP's vision encoder. This alignment is accomplished through a linear layer, with a training objective formulated using optimal transport-based assignment prediction. Thanks to the inherent alignment between CLIP's vision and text encoders, the few-shot encoder is indirectly aligned to CLIP's text encoder, which serves as the foundation for better vision-semantics integration. In addition, to further improve vision-semantics integration at the testing stage, we mine potential fine-grained semantic attributes of class names from large language models. Correspondingly, an online optimization module is designed to adaptively integrate the semantic attributes and visual information extracted from images. Extensive results on four datasets demonstrate that our method outperforms state-of-the-art methods.

Main paper: 
 related work: 
 Existing FSIC methods can be mainly divided into two categories, i.e., vision-based methods and semantics-based methods.Vision-based methods leverage support samples to extract class-related features for classification. To achieve this, several methods [3, 16, 38, 51] with different optimization strategies learn an initial model that can rapidly adapt to novel tasks with a few steps of updating. However, due to limited support samples, such methods often suffer from overfitting. To address this issue, some methods [9, 31, 39, 47, 53] learn a metric space where novel samples are classified utilizing a prevalent prototype-based classifier [39] . Other methods [19, 27] focus on acquiring a robust few-shot encoder by employing a transformer-based backbone [15] and incorporating techniques such as distillation [17] and self-supervised learning [6, 18] .Semantics-based methods [8, 26, 44, 45, [48] [49] [50] extract semantic information from class names and integrate visual and semantic information to improve FSIC performance. Recent methods [8, 26, 45, 50] introduce implicit or explicit modality alignment mechanisms to bridge the gap between visual and semantic modalities for improved integration performance. For example, Xu et al. [44] introduced a conditional variational autoencoder (CVAE) to generate visual features conditioned on semantic features, which implicitly aligns the visual and semantic modalities in the latent space. Li and Wang [26] enforced explicit alignment by aligning latent distributions of the visual and semantic modalities and minimizing the cross-modal reconstruction errors. Zhang et al. [50] designed a semantic revolution module to augment class names to encode robust semantic features, achieving more accurate visionsemantics alignment. However, information derived from class names is insufficient to represent the visual details in diverse images. Consequently, the vision-semantics alignment in the above methods is inherently biased, leading to suboptimal integration outcomes. 
proposed method: 
 In this section, we first revisit the definition of FSIC and briefly introduce the testing pipeline of CLIP. We then elaborate on the training (Section 3.2) and testing (Section 3.3) pipelines of FewVS. C 𝑏 and C 𝑛 correspond to the base and novel class sets, respectively, with the condition that they are disjoint, i.e., C 𝑏 ∩ C 𝑛 = ∅. To provide semantic information, we map 𝑦 to its class name 𝑐, such as 'House finch' and 'Jellyfish'.Adhering to a conventional 𝑁 -way 𝐾-shot setting [16, 39] , each episode samples 𝑁 classes from C 𝑛 , with 𝐾 samples per class, to construct the support set: S = {(𝐼 𝑠 𝑖 , 𝑦 𝑠 𝑖 )} 𝑁 ×𝐾 𝑖=0 . Similarly, 𝑁 × 𝑄 samples from these 𝑁 classes are sampled to form the query set:Q = {(𝐼 𝑞 𝑖 , 𝑦 𝑞 𝑖 )} 𝑁 ×𝑄 𝑖=0 . Notably, S ∩ Q = ∅.The aim of FSIC is to train a model on D 𝑏 with strong generalization performance on Q by utilizing the labeled samples from S.Testing pipeline of CLIP. CLIP demonstrates remarkable performance in zero-shot image classification which is a task similar to FSIC. At the testing stage, given a query image 𝐼 𝑞 and a set of possible class names {𝑐 𝑖 } 𝑁 𝑖=1 , CLIP firstly encodes visual and semantic features from the image and class names via its vision encoder 𝐸 𝑣 (•) and text encoder 𝐸 𝑠 (•), respectively. It then computes the cosine similarity scores between 𝐼 𝑞 and each 𝑐 𝑖 to integrate vision and semantics, relying on its inherent vision-semantics alignment property. Finally, the probability that 𝐼 𝑞 belongs to class 𝑖 is computed via a softmax function:𝑃 (𝑦 = 𝑖 |𝐼 𝑞 ) = exp(< 𝐸 𝑣 (𝐼 𝑞 ), 𝐸 𝑠 (𝑐 𝑖 ) >) 𝑁 𝑗=1 exp(< 𝐸 𝑣 (𝐼 𝑞 ), 𝐸 𝑠 (𝑐 𝑗 ) >) (1)where < • > is the cosine similarity function. Note that CLIP's two encoders share the same feature dimension. Due to the alignment nature between the vision and text encoders, CLIP can classify images of novel classes based only on semantic information about these novel classes.The overall training pipeline of FewVS is shown in Figure 1 . In FewVS, we enforce vision-vision alignment as a proxy task to avoid biased vision-semantics alignment. This alignment equips FewVS with the capability to perform CLIP-like vision-semantics integration (as stated in Section 3.1.2) between a lightweight few-shot encoder and CLIP's text encoder. Specifically, we add a linear projection layer following the few-shot encoder to transform the features of the few-shot encoder into those of CLIP's vision encoder. To ensure the alignment quality, inspired by [5, 7, 33, 43] , we use an optimal transport strategy to compute the assignment predictions of the features of the few-shot encoder and CLIP's vision encoder, respectively, thus obtaining two kinds of assignment predictions. Then, we train the linear projection layer by maximizing the consistency between these two kinds of predictions. Next, we further provide the details of the training pipeline.N  1 N  1 N  1 W H C K N    • W H C  s z t z t Q s Q q x s X d K N  • q x ŝ X ˆA d K N  • d 1 d M N  • d  1Feature extraction. Given a batch containing 𝐵 images from D 𝑏 , we feed them into frozen CLIP's vision encoder 𝐸 𝑣 (•), and extract the global pooled hidden features, denoted asZ 𝑡 = {z 1 , ..., z 𝐵 } ∈ R 𝐵×𝑑 𝑡, which serves as the alignment target. Similarly, we feed the same images into the pretrained and frozen few-shot encoder 𝐸 𝑓 𝑒𝑤 (•). Subsequently, we transform the resulting featuresX = {x 1 , ..., x 𝐵 } ∈ R 𝐵×𝑑 𝑠into the same dimension of Z 𝑡 via a learnable linear projection layer ℎ(•), yielding the image projection denoted as Z 𝑠 , which serves as the source of alignment:EQUATION3.2.2 Assignment prediction. Different from previous works [2, 5, 33] , we sample 𝐿 entities describing visual objects from a vast corpus (namely WordNet [30] ), and encode their names via CLIP's text encoder to construct an entity embedding set E = {e 1 , ..., e 𝐿 } ∈ R 𝐿×𝑑 𝑡 which can be regarded as cluster centers. Then, we map Z t and Z 𝑠 to E to obtain the assignments Q 𝑡 = {q 𝑡 1 , ..., q 𝑡 𝐵 } ∈ R 𝐵×𝐿 and Q 𝑠 = {q 𝑠 1 , ..., q 𝑠 𝐵 } ∈ R 𝐵×𝐿 , respectively. Similar to [2, 5, 33] , we optimize Q 𝑡 and Q 𝑠 to maximize the similarity between the features of CLIP's vision encoder and the entity embeddings and the similarity between the features of the few-shot encoder and the entity embeddings, respectively. The above process can be modeled as an optimal transport problem with the following objectives: maxEQUATIONmaxEQUATION)where 𝐻 is the entropy function,𝐻 (Q) = -𝑖 𝑗 Q 𝑖 𝑗 log Q 𝑖 𝑗 ,and 𝜖 is a smoothness parameter of the assignment. Following [5] , we implement an equal partition by constraining the matrix Q to belong to the transportation polytope: where 1 𝐿 denotes the vector of ones in dimension 𝐿. The two constraints in Eq. ( 5) ensure that on average each entity embedding is selected at least 𝐵 𝐿 times in a batch. The optimized assignments Q 𝑠 * and Q 𝑡 * , namely the solutions of Prob. (3) and Prob. (4) over Q, take the form of a normalized exponential matrix [11] :Q = {Q ∈ R 𝐿×𝐵 + |Q1 𝐵 = 1 𝐿 1 𝐿 , Q ⊤ 1 𝐿 = 1 𝐵 1 𝐵 } (5)EQUATIONEQUATIONwhere u and v are renormalization vectors in R 𝐿 and R 𝐵 , respectively, which are computed using the iterative Sinkhorn-Knopp algorithm [11] . More details are provided in the supplementary material.To train the linear projection layer, following [5] , we construct a "swapped" problem to predict the assignments of the features of CLIP's vision encoder based on the features of the few-shot encoder, and vice versa. Specifically, we predict Q 𝑠 from Z 𝑡 and Q 𝑡 from Z 𝑠 , which can be formulated as the crossentropy loss between the assignments and the probabilities that the corresponding features belong to the entity embeddings. The loss function is defined as:L 𝑎𝑙𝑖𝑔𝑛 = - 1 𝐵 𝐵 ∑︁ 𝑏=1 𝐿 ∑︁ 𝑙=1 [Q 𝑠 * 𝑏𝑙 log p 𝑡 𝑏𝑙 + Q 𝑡 * 𝑏𝑙 log p 𝑠 𝑏𝑙 ],whereEQUATIONThis loss function ensures that the transformed features of the few-shot encoder contain the similar information with the features derived from CLIP's vision encoder, making the few-shot encoder indirectly aligned to CLIP's text encoder.In this section, we first briefly introduce the overall testing pipeline of FewVS which is shown in Figure 1 . Subsequently, we elaborate on the fine-grained semantic attribute mining and the online visionsemantics integrator.Overall testing pipeline. In a standard 𝑁 -way 𝐾-shot episode with a support set S = {(𝐼 𝑠 𝑖 , 𝑦 𝑠 𝑖 )} 𝑁 ×𝐾 𝑖=0 and a query image 𝐼 𝑞 , we first feed both S and 𝐼 𝑞 into the pretrained few-shot encoder𝐸 𝑓 𝑒𝑤 (•) to obtain visual support features X 𝑠 = {x 𝑠 1 , ..., x 𝑠 𝑁 •𝐾 } ∈ R 𝑁 •𝐾 ×𝑑and visual query feature x 𝑞 ∈ R 𝑑 . Then, we transform X 𝑠 and x 𝑞 into X𝑠 ∈ R 𝑁 •𝐾 × d and x𝑞 ∈ R 𝑁 •𝐾 × d via a linear projection layer ℎ(•), where d is the feature dimension of CLIP's text encoder 𝐸 𝑠 (•). Subsequently, we map the support labels back into class names, and mine 𝑀 fine-grained semantic attributes for each class using GPT-3, the details of which are provided later in Section 3.3.2. Then, we encode these semantic attributes via 𝐸 𝑠 (•) and obtain a semantic feature setA = {a 1 , ..., a 𝑁 •𝑀 } ∈ R 𝑁 •𝑀 × d .The final probability 𝑃 of a query sample 𝐼 𝑞 belonging to class 𝑛 is computed from the weighted sum of visual logits 𝑙 𝑣𝑖𝑠 ∈ R 𝑁 and House finch: l small bird l red, brown, or grey l conical beak l notched tail l typically found feeding in urban or suburban areas American robin: l medium-sized bird l orange or reddish breast l gray back and wings l white lower belly l often seen on the groundl slim and graceful sighthound l long, thin limbs l long, feathered ears l short, silky coat l varying colors Ladybug: l round or oval body shape l red or orange with black spots l black head with white patches l six legs l small size semantic logits 𝑙 𝑠𝑒𝑚 ∈ R 𝑁 :EQUATIONwhere 𝛼 is a weight factor determining the balance between the visual and semantic information.To obtain 𝑙 𝑣𝑖𝑠 𝑛 , we follow [39] to extract the prototype of class 𝑛, denoted as 𝑝 (𝑛), and compute the similarity between x q and 𝑝 (𝑛):EQUATIONwhere X 𝑠 𝑛 is a set of support features belonging to class 𝑛 in X 𝑠 , and < • > denotes the cosine similarity function.To obtain 𝑙 𝑠𝑒𝑚 𝑛 , a straightforward method is to compute the crossmodal similarity like CLIP:EQUATIONwhere A 𝑛 is a set of semantic features belonging to class 𝑛 in A, and p (𝑛) can be considered as the semantic prototype of class 𝑛. However, naively computing 𝑙 𝑠𝑒𝑚 𝑛 is suboptimal, due to the noisy semantic attributes introduced by semantic attributes mining (detailed later in Section 3.3.2). Therefore, we design an online optimization module to adaptively integrate visual and semantic information for the computation of 𝑙 𝑠𝑒𝑚 𝑛 , the details of which are provided later in Section 3.3.3.Fine-grained semantic attribute mining. To further improve vision-semantics integration performance, we propose to mine fine-grained semantic attributes corresponding to each class name, which describe potential visual details of this class. Similar to [29] , we automatically mine these fine-grained semantic attributes by prompting a large language model, such as GPT-3 [4] , to query the visual details. We prompt the language model with the input: "Q: What are the visual details for distinguishing a [class name] in a photo? A: There are several useful visual details of a [class name] in a photo:", Further implementation details can be found in the supplementary material.As depicted in Figure 2 , the fine-grained semantic attributes describe potential visual details of a class, such as colors, shapes, object components, and scene contexts. Nevertheless, the process of mining these semantic attributes is agnostic to the visual modality and may introduce noisy semantic attributes that mismatch novel classes. For instance, we prompt GPT-3 with three class names "harvestman", "Newfoundland dog", and "bell", obtaining the corresponding semantic attributes. Among them, semantic attributes "lacks venom or silk glands" associated with "harvestman", "males typically weighing 130-150 pounds" associated with "Newfoundland dog, " and "emits sound when moved by wind" associated with "bell" contain irrelevant noises. Obviously, semantic features encoded from these noisy attributes may have a negative impact on vision-semantics integration.( ) ( M N K N •  • ) ( 1 M N •  N K N  • N K N  • N  1  w w q x ˆs X Â d K N  • d 1 d M N  •As shown in Figure 3 , the key point behind this module is to online learn an importance weight vector 𝜔 ∈ R 𝑁 •𝑀 to adjust the contribution of each semantic attribute in A, with the support set S as supervision.For each episode, we randomly initialize 𝜔 0 = 0 ∈ R 1×𝑁 •𝑀 , and infer 𝜔 * using labeled support samples via inner-loop optimization. To be more specific, we first construct a cross-modal similarity map by computing the pair-wise similarity between the transformed support features X𝑠 and semantic features A as S 𝑠 ∈ R 𝑁 •𝐾 ×𝑁 •𝑀 , where each element in S 𝑠 is obtained by s 𝑖 𝑗 =< x𝑠 𝑖 , a 𝑗 >, 𝑖 = 1, ..., 𝑁 • 𝐾, and 𝑗 = 1, ..., 𝑁 • 𝑀. Similarly, we compute the pair-wise similarity between the query image 𝐼 𝑞 and A as S 𝑞 ∈ R 𝑁 •𝑀 .Then, we adjust the contribution of each semantic attribute to classification by performing broadcast and addition operations on 𝜔, thus obtaining an adjusted similarity map S𝑠 = S 𝑠 + [1 𝑁 •𝐾 ×1 • 𝜔]. In this way, the similarities associated with each semantic attribute are re-weighted via 𝜔. Subsequently, the semantic logits (denoted as 𝑙 𝑠 𝑛 ) of the 𝑖th support sample 𝐼 𝑠 𝑖 belonging to class 𝑛 can be computed as follows:EQUATIONwhere S𝑠 𝑖𝑛 is a set of similarity logits in S𝑠 associated with both 𝐼 𝑠 𝑖 and semantic attributes of class 𝑛. Then, the class prediction 𝑦 ′ 𝑖 of 𝐼 𝑠 𝑖 can be computed as:𝑦 ′ 𝑖 = softmax({𝑙 𝑠 𝑛 } 𝑁 𝑛=1 ). Given that y ′ = {𝑦 ′ 𝑖 } 𝑁 •𝐾𝑖=1 is dependent on 𝜔, we optimize 𝜔 to re-weight semantic attributes' classification contributions using support labels y = {𝑦 𝑖 } 𝑁 •𝐾 𝑖=1 as supervision, and formulate an online optimization objective as follows:EQUATION)This objective is optimized iteratively through gradient descent to obtain the optimal weight vector 𝜔 * that gives the insights into the positive/negative impact of each semantic attribute on classification. Finally, we compute 𝑙 𝑠𝑒𝑚 𝑛 as follows:EQUATIONwhere S𝑞 𝑛 is a set of similarity logits in S𝑞 associated with semantic attributes of class 𝑛. 
experiments: 
 To test the performance of FewVS, we conducted experiments on four public datasets including miniImageNet, tieredImageNet, CIFAR-FS, and FC100.miniImageNet [42] and tieredImageNet [37] are both subsets of ImageNet [12] , while CIFAR-FS [3] and FC100 [31] are derived from CIFAR100 [22] Architecture. In all experimental configurations, we utilized ResNet-12 (Res12) and ViT-Small (ViT-S) as the few-shot encoders. Specifically, we implemented and pretrained the Res12 encoder using the same settings as described in FEAT [47] . For the ViT-S encoder, we employed the training strategy reported in SMKD [18] . We established two baseline methods using these vanilla few-shot encoders, namely Baseline-Res and Baseline-Trans, in which we performed prototype-based classification following [39] . For the Res12 encoder, visual features were derived by averaging the outputs from the final residual block. For the ViT-S encoder, visual features were obtained by averaging the hidden states from the last transformer block. Additionally, we utilized both the vision and text encoders from Res50x4 CLIP, and employed a single linear layer to construct the linear projection layer ℎ(•).Training details. During the training phase of FewVS, we kept the pretrained few-shot encoder and CLIP's vision encoder frozen and focused only on optimizing the linear projection layer ℎ(•). For FewVS based on Res12, we adhered to traditional FSIC methods [9, 31, 39, 47, 53] by resizing the input images to 84 × 84. We also adjusted the stride of the last downsampling layer in CLIP's vision encoder to 1 to accommodate smaller input sizes. For FewVS based on ViT-S, we resized the input image to 320 × 320 for miniImagenet and tierdImagene, and 224 × 224 for CIFAR-FS and FC100, maintaining the consistency with SMKD [18] . We set the batchsize 𝐵 = 128 and trained the linear projection layer for 20 epochs, using the AdamW optimizer [28] with a learning rate of 1e-3. To construct the entity embedding set E used in Eq. (3) and Eq. ( 4), we encoded the names of 1,000 entities sampled from WordNet [30] via CLIP's text encoder.Testing details. At the testing stage, we mined five fine-grained semantic attributes for each class using GPT-3 (i.e., 𝑀 = 5). We online optimized 𝜔 in Eq. ( 13) via iterative gradient decent, employing an SGD optimizer with a learning rate of 5e-2. The number of iterations was set to 1 for the 1-shot task, and to 5 for the 5-shot task. Additionally, in FewVS based on Res12, we set 𝛼 in Eq. ( 9) to 2 and 3 for the 1-shot and 5-shot tasks, respectively. In FewVS based on ViT-S, we set 𝛼 to 2 and 10 for the 1-shot and 5-shot tasks, respectively. Evaluation. To ensure the stability of our evaluation results, we tested on 2,000 randomly sampled episodes from each benchmark test set, and reported the average performance with a 95% confidence interval. In each episode, we randomly selected 15 query features per class to assess the performance in both the 5-way 1-shot and 5-way 5-shot tasks.The classification results are reported in Table 1 and Table 2 . From Table 1 and Table 2 , compared with the two baselines, the performance of FewVS is remarkably better. In particular, in the 1-shot task, FewVS achieves accuracy improvement by 13.3% ∼ 18.7%, 10.8% ∼ 14.6%, 6.1% ∼ 12.6%, and 12.1% ∼ 16.2% on miniImageNet, tieredImageNet, CIFAR-FS, and FC100, respectively. This is because the semantic information derived from class names effectively assists in constructing knowledge of novel classes in the extreme absence of support samples.Compared with other semantics-based competitors, FewVS-Trans surpasses the existing SoTA method (i.e., SemFew [50] ) by 0.5% ∼ 7.8% on the four datasets. This is because FewVS enforces unbiased alignment, mines fine-grained semantic attributes, and adaptively integrates these semantic attributes with visual features via an online optimization strategy. In summary, FewVS-Trans achieves new SoTA performance compared with previous FSIC methods across both 1-shot and 5-shot tasks.In this section, our analysis is mainly based on the large challenging dataset, i.e., tieredImageNet, and all the experiments were performed using Res12 as the few-shot encoder. More ablation studies, such as experiments w.r.t. hyperparameters and visualization analysis, are provided in the supplementary material. In FewVS, we proposed the optimal transport-based alignment, which maximized the consistency between assignment predictions computed from the output features of the few-shot encoder and those of CLIP's vision encoder. To study the impact of this alignment, we tested two variants. In the first variant (w/o alignment), we removed the alignment module and degenerated FewVS into a baseline. In the second variant (with L 𝐿2 ), we defined the alignment objective as the minimization of the L2 distance between the features from the two encoders. As shown in Table 3 , compared with FewVS, the performance of these two variants drops significantly, verifying the effectiveness of the optimal transport-based alignment.4.4.2 Impact of the entity-based cluster centers in assignment prediction. Different from previous optimal transport-based approaches [1, 2, 5, 33] , we constructed the cluster centers (i.e., E in Eq. (3) and Eq. ( 4)) for feature assignment using entity embeddings. These embeddings were encoded from the names of entities in WordNet through the text encoder of CLIP. To evaluate the efficacy of these entity-based cluster centers, we tested two variants. In the first variant (w/o entity embed), we defined E as 3K learnable prototypes following [1, 2, 5] . In the second variant (with word embed), we defined E as the word embeddings of CLIP's text encoder following [33] . As shown in Table 4 , compared with these two variants, the entity-based cluster centers achieve substantial performance improvement, which is possibly because the cluster centers constructed from entities are better to represent image contents, thus producing more accurate feature assignments.Impact of mining fine-grained semantic attributes. In FewVS, we mined fine-grained semantic attributes from GPT-3 to enrich the semantic information of class names. To demonstrate its effectiveness, we tested a variant (w/o mining), in which we removed the semantic attribute mining process. In this variant, the semantic prototype p (𝑛), defined in Eq. ( 11), only relies on the semantic feature encoded from the class name 𝑛. Therefore, there is no need to re-weight the semantic attributes, and the classification is straightforwardly conducted using Eq. (11) . As shown in Table 5 , the performance of FewVS notably deteriorates in the absence of fine-grained semantic attributes, indicating that mining semantic attributes from large language models does enrich semantic information on class names.Impact of the online vision-semantics integrator. In FewVS, we designed an online vision-semantics integrator that adaptively integrated vision and semantics by re-weighting the classification contribution of each semantic attribute. This was accomplished by iteratively optimizing 𝜔 in Eq. (13) . To investigate the rationality of this module, we tested a variant (w/o optimization), in which the number of optimization iterations was set to 0. From Table 6 , FewVS involving online optimization performs better than its variant. This demonstrates the rationality of the online vision-semantics integrator, which can identify the positive/negative impact of each semantic attribute on classification. We compared our FewVS with a CLIP-based approach, i.e., Tip-Adapter [52] . Tip-Adapter adapts CLIP to few-shot scenarios using a training-free key-value cache model [20, 32] . The original few-shot setting in Tip-Adapter differs from general FSIC. Therefore, we modified Tip-Adapter for FSIC by restricting the cache model to only store images from an 𝑁 -way 𝐾-shot episode as the key-value database during the testing phase.As reported in Table 7 , FewVS outperforms Tip-Adapter by a large margin on tieredImageNet and has fewer parameters. This is because compared with the heavy CLIP vision encoder (namely Res50x4), FewVS utilizes a lightweight few-shot encoder. This encoder can be easily fine-tuned or trained from scratch using methodologies specifically designed for FSIC, facilitating the encoding of task-specific visual information. 
conclusion: 
 In this paper, we proposed a framework, called FewVS, to integrate visual and semantic information for improving FSIC. In FewVS, we indirectly achieved unbiased vision-semantics alignment by introducing CLIP and enforcing vision-vision alignment as a proxy task, with an optimal transport-based objective. Furthermore, we mined fine-grained semantic attributes from large language models, and designed an online optimization module to adaptively integrate these semantic attributes with information extracted from images. Experiments showed that FewVS effectively improved baseline methods and outperformed the state-of-the-art methods on four challenging datasets.  
: 
 tackle this problem, existing methods [8, 26, 44, 45, 48, 50] propose to extract semantic features from class names via a pretrained text encoder [13, 35, 36] as prior information of novel classes, and then design different cross-modal interaction modules to integrate the visual and semantic features for improving the FSIC performance. However, these methods ignore the inherent modality gap between the visual and semantic modalities, thus only achieving modest integration performance.Recognizing this limitation, recent methods [8, 26, 45, 50] introduce various modality alignment mechanisms to bridge this modality gap for improved integration. Additionally, these methods leverage the semantic branch of CLIP, a vision-language pretrained model, as a powerful text encoder to obtain reliable semantic features. For instance, BMI [26] explicitly aligns the few-shot encoder with CLIP's text encoder by minimizing the discrepancies in latent distributions between the visual and semantic modalities, as well as the cross-modal reconstruction errors. SemFew [50] designs a semantic revolution module to augment class names to encode robust semantic features, achieving more accurate alignment between vision and semantics. However, the semantic information derived from class names is limited, which falls short of capturing the visual details of diverse images. Consequently, this leads to biased alignment between vision and semantics, producing suboptimal integration outcomes.Actually, CLIP [36] can serve as a natural bridge between vision and semantics. It is trained on a vast dataset comprising over 400 million image-caption pairs using a contrastive learning paradigm, ensuring an inherent alignment between its vision and text encoders. In this paper, leveraging CLIP's inherent alignment, we enforce vision-vision alignment as a proxy task to avoid biased vision-semantics alignment. To be specific, we align features encoded from the few-shot encoder and CLIP's vision encoder on the same image at the training stage, by learning a linear projection layer with an optimal transport-based training objective. Due to the inherently aligned nature of CLIP's vision and text encoders, the few-shot encoder is indirectly aligned with CLIP's text encoder. In this way, we can perform CLIP-like vision-semantics integration using the few-shot encoder and CLIP's text encoder. Notably, compared with the heavy CLIP's vision encoder, the few-shot encoder is designed to be lightweight and can be trained from scratch using methodologies tailored for FSIC, such as meta-learning [16] , to specifically learn and extract task-specific visual information.To further improve the vision-semantics integration at the testing stage, we mine fine-grained semantic attributes for each class name from large language models, such as GPT-3 [4] . Such semantic attributes can describe potential visual details and provide richer prior information for novel classes than class names. However, the process of mining these semantic attributes is agnostic to the visual modality, possibly introducing noisy semantic attributes that mismatch novel classes. Therefore, drawing inspiration from [19] , we design an online optimization module to adaptively integrate these fine-grained semantic attributes with the visual information from query images. This is achieved through online optimizing a learnable importance weight for each semantic attribute's classification contribution using the labels of support samples as supervision. Through this process, we can identify the positive/negative impact of each semantic attribute on our classification objective and then re-weight its contribution, which guides an adaptive integration for more effective query classification.The main contributions of this paper are summarized as follows:• We propose a novel Few-shot Vision-Semantics integration framework, namely FewVS, to integrate vision and semantics for improving FSIC. In FewVS, we indirectly achieve unbiased vision-semantics alignment by introducing CLIP and enforcing vision-vision alignment as a proxy task, with an optimal-transport based training objective. • To further enhance integration performance, at the testing stage, we mine fine-grained semantic attributes for each class by querying large language models, and design an online optimization module to adaptively integrate vision and semantics.• Extensive experiments on four public benchmarks (miniIma-geNet, tieredImageNet, CIFAR-FS, and FC100) demonstrate that our FewVS outperforms state-of-the-art methods. 
 
 Appendix:  


Decision: This paper introduces CLIP, a bridge between vision and semantics. 

The reviewers highlight a number of relevant issues, including limited novelty and clarity. Nevertheless, based on the rebuttal, all reviewers converged in borderline accept (with two of them moving from reject to accept). We encourage the authors to take into account all the feedback for the camera-ready version of the article.

Reviews: review: paper_summary: paper_summary: This paper aims to align the few-shot feature encoder with the CLIP vision encoder to indirectly align with CLIP's text encoder. The authors propose that this approach addresses the issue of biased vision-semantics alignment.
strength_weakness: strength_weakness: 1. The paper is well-written and easy to understand.

2. The novel idea of leveraging CLIP to bridge the gap between vision and semantics is simple, interesting, and intuitive.

3. The experiments are extensive, and the proposed method achieves promising performance in few-shot image classification across four datasets.
limitations: limitations: 1) From the experiment, I observed that the Swin-T backbone performs well across the four datasets. Have the authors tried using the proposed method based on this backbone?

2) In the table, with L2, the 1-shot performance drops significantly, but the 5-shot performance does not drop as much. It would be interesting to provide more analysis on this.
paper_summary: paper_summary: This paper tackles the few-shot image classification problem by aligning with CLIP from two perspectives. First, the method utilizes vision-vision alignment with a training objective formulated using optimal transport-based assignment prediction as a proxy task to avoid biased vision-semantics alignment. Further, the method proposes to use fine-grained semantic attributes instead of the single class name for classification. The proposed method achieve SOTA performance on four datasets.
strength_weakness: strength_weakness: 1. The paper is easy to follow.
2. The proposed method achieves SOTA performance on various benchmarks.
limitations: limitations: 1. This work aims to align with CLIP in an unbiased way. Why not directly use the CLIP visual encoder to extract the feature as CLIP has shown impressive zero-shot image classification ability. Finetuning CLIP with some adapters or other techniques could possibly be another solution. In Table 7, the proposed method outperforms than a CLIP-based approach Tip-Adapter. Unfortunately, there is no detailed explanation.
2. This paper performs the vision-vision alignment at the output of the encoders. Perhaps aligning intermediate feature maps can provide more useful details. Typically, in the few shot setting, aligning with more samples can significantly increase the performance as shown in Table 3. Thus, exploring the intermediate feature map alignment is reasonable. Moreover, I think aligning with CLIP in the pretraining phase can further improve the performance.
3. The pretrained few-shot encoder is frozen during the training which is not reasonable as only one projection layer may not be sufficient to align with CLIP. Perhaps more tunable parameters can better align the model with CLIP.
4. More detailed captions under Figure 1 and Figure 3 can help readers better understand the total pipeline.
5. There are some typos in the manuscript, for example \hat{x}^q \in R^{\hat{d}} in Line 403. Please check them carefully.
paper_summary: paper_summary: This paper proposes a proxy task of vision-vision alignment to improve the effectiveness of CLIP for few-shot classification, where the proxy task is to align the visual features from the CLIP and few-shot encoder. In this way, the visual features can achieve a better alignment with the text encoder of CLIP. Furthermore, an online vision-semantics integrator is introduced to provide a weight for feature fusion.
strength_weakness: strength_weakness: 1. The motivation of the paper is clear and convincing;
2. The idea of vision-vision alignment is a neat way to better use CLIP;
3. The proposed method is effective with a significant improvement.
limitations: limitations: 1. It seems that the core idea of this paper is a method similar to CLIP distillation, so the contribution appears to be limited;
2. I wonder if is there a comparison of directly inputting the CLIP's visual features into the online vision-semantics integrator;
3. Both vision-vision alignment and fine-grained semantic attributes mining are convincing ideas, but these techniques may have already been introduced in previous works or other tasks, lacking specific improvements and modifications for few-shot image classification accordingly;
4. The ablation studies seems to have been conducted only on the ResNet backbone.
paper_summary: paper_summary: In this paper, the authors propose a few-shot vision semantics integration framework (FewVS). FewVS addresses data scarcity and insufficient information fusion issues by aligning a pre-trained few-shot encoder with CLIP’s visual encoder on the same image, thereby indirectly aligning the few-shot encoder with CLIP’s text encoder.
strength_weakness: strength_weakness: 1. The proposed method addresses the challenge of few-shot learning by enriching the feature space with language-derived context, which is often lacking in purely visual methods.
2. The writing in this paper is clear and effectively communicates the proposed framework and experimental results. The authors also provide clear explanations of the methodology.
3. The proposed method is theoretically reasonable using feature alignment based on optimal transmission. This method also ensures accurate feature alignment between the few-shot encoder and the CLIP visual encoder, thereby enhancing the effectiveness of FewVS.
limitations: limitations: 1. The FewVS framework relies on large pre-trained models (such as CLIP and GPT-3) for visual and semantic integration. This dependency may result in high computational costs and resource requirements.
2. Although the paper introduces the FewVS framework, its approach mainly involves combining existing technologies rather than introducing genuinely novel or innovative ideas or techniques. For example, while advanced models like CLIP are used, the integration method does not introduce significant innovation and lacks breakthrough improvements compared to some previous work.
3. The FewVS framework relies on the CLIP model for visual and semantic integration, but does not address or remove noise in the CLIP model. As large language models (such as CLIP) may contain noise, this could affect the accuracy of final feature alignment and the quality of integration.
4. Some references have inconsistent and non-standard citation formats. For instance, the conference names in reference [36] are capitalized while they are lowercase in reference [31]. Reference [36] uses conference abbreviations, whereas the full names are used in reference [42]. Some references lack page numbers, suggesting careful revision is needed.