Title: VT-FSL: Bridging Vision and Text with LLMs for Few-Shot Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Few-Shot Learning

Multimodal Fusion

LLM
Abstract: Few-shot learning (FSL) aims to recognize novel concepts from only a few labeled support samples. Recent studies enhance support features by incorporating additional semantic information  (e.g., class descriptions) or designing complex semantic fusion modules. However, these methods still suffer from hallucinating semantics that contradict the visual evidence due to the lack of grounding in actual instances, resulting in noisy guidance and costly corrections. To address these issues, we propose a novel framework, bridging Vision and Text with LLMs for Few-Shot Learning (VT-FSL), which constructs precise cross-modal prompts conditioned on Large Language Models (LLMs) and support images, seamlessly integrating them through a geometry-aware alignment mechanism. It mainly consists of Cross-modal Iterative Prompting (CIP) and Cross-modal Geometric Alignment (CGA). Specifically, the CIP conditions an LLM on both class names and support images to generate precise class descriptions iteratively in a single  structured reasoning pass. These descriptions not only enrich the semantic understanding of novel classes but also enable the zero-shot synthesis of semantically consistent images. The descriptions and synthetic images act respectively as complementary textual and visual prompts, providing high-level class semantics and low-level intra-class diversity to compensate for limited support data. Furthermore, the CGA jointly aligns the fused textual, support, and synthetic visual representations by minimizing the kernelized volume of the 3-dimensional parallelotope they span. It captures global and nonlinear relationships among all representations, enabling structured and consistent multimodal integration. The proposed VT-FSL method establishes new state-of-the-art performance across ten diverse benchmarks, including standard, cross-domain, and fine-grained few-shot learning scenarios. Code is available at https://github.com/peacelwh/VT-FSL.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper claims to be addressing the semantic hallucination problem when class names or LLM generated descriptions of classes are used in few-shot learning. The paper argues that features corresponding to LLM description of classes or the class attributes (generated based on the class names) must be aligned with the support images and generated synthetic images. The paper uses a prompting strategy to select the best K (e.g., 4) synthetic images from multiple images generated by JanusPro based on the textual description given by Qwn2.5-VL-32B using a pair-wise comparison. The vision encoder is Visformer-Tiny and the text encoder is ViT-B/16 CLIP. It aligns textual features, features of generated images, and the features of the support set images. \n\nThe results---standard few shot classification on miniImageNet and tiredImageNet, CIFAR-FS FC100, fine-grained few-shot classification on CUB-200-2022, Stanford-Dogs, Stanford-Cars, cross-domain few-shot classification on miniImageNet to CUB, miniImageNet to Places, miniImageNet to Plantae---are superior.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tSuperior results. \n2.\tPrompting using the features of both support-set images and textual descriptions. \n3.\tAligning the embeddings of textual descriptions, support image embeddings, and generated image embeddings. \n\nWeaknesses:\n\n1.\tUsing a large language model and a large image generator in addition to the Visformer-Tiny visual encoder gives an unfair advantage to this model. It would be important to compare training and inference times with a few existing papers. \n2.\tPicking the best K image generations probably gives very clean images representing the class in question. Fig. 6b shows such clean generations. The ablation in Table 5 shows that the contribution of the P_vision alone is the largest. This too is, in my opinion, an unfair advantage.  \n3.\tWhat do the numbers in row 1 of Table 5 represent? Is it the few-shot classification accuracy of Visformer-Tiny?\n4.\tUsing a carefully generated descriptive caption embedding with prototype embeddings exists in works cited in this paper, e.g., [27], and [26]. In this context, the novelties are 1. Cross-modal prompting, and 2. Geometric alignment. I am concerned whether this novelty is sufficient. Does the ablation show prompting with textual data only?\n\nQUESTIONS:\n1.\tRelated to items 1 and 2 above, justify the use of several large models. Report on the timing. \n2.\tAddress the concern about the insufficient novelty. \n3.\tImprove clarity.\n\nLIMITATIONS:\nReliance on the LLM and the image generator.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method named VT-FSL, which aims to bridge vision and text modalities using a large language model (LLM) to enhance few-shot learning (FSL). The authors argue that prototype-based FSL methods often suffer from non-discriminative class representations, resulting in semantic deviation from the true class centers. To address this, the proposed VT-FSL framework incorporates two key components: Cross-modal Iterative Prompting (CIP) and Cross-modal Geometric Alignment (CGA).\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper is well-structured and clearly motivated, making it easy to follow the authors' logic and contributions.\n2. The idea of leveraging both visual and textual modalities for FSL is novel and timely, particularly with the growing capability of LLMs.\n3. Experimental results demonstrate significant improvements over state-of-the-art methods, validating the effectiveness of the proposed approach.\n\nWeakness:\n1. The title of the paper is too generic and does not effectively convey the unique contributions or technical highlights of the work. A more specific and informative title would help better position the paper.\n2. Some notations in the paper are unclear or inconsistent. For example,  Eq.(4) lacks the definition of the $z_v$. Additionally, the usage of $Z_t$ and $z_s$ and $Z_s$ is confusing—it's unclear why uppercase and lowercase letters are used differently. A clarification or unification of these notations is needed.\n3. The paper would benefit from an ablation study on the loss function, e.g., evaluating the performance using only the cross-entropy loss, only $L_{D2A}$ and only $L_{A2D}$.\n4. A major concern lies in the fairness of the experimental comparisons. The image generation process appears to introduce additional samples, which may compromise the integrity of the few-shot setting. Moreover, it is unclear whether the generative model has been exposed to the novel classes during training, which could further undermine the validity of the results. This point should be explicitly discussed and clarified.\n5. The generation process seems to effectively double the number of shots in each class, which may lead to an unfair advantage. I strongly recommend the authors to conduct ablation studies that vary the number of generated images and report the impact on performance. This would help assess whether the performance gain is due to the proposed framework or simply the increased data volume.\n\nQUESTIONS:\nAll of my questions and concerns are outlined in the “Weaknesses” section above. If the authors are able to address these issues satisfactorily, I would be open to revising my score.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses a large language model and a text-to-image generation model to produce precise category descriptions and visual cues, serving as auxiliary information to enhance the support set images. Specifically, two key modules: Cross-modal Iterative Prompting, which utilizes category names and images to generate detailed textual descriptions and synthesized images; and Cross-modal Geometric Alignment, which employs a kernel-based volume minimization approach to align the representations across modalities are introduced. Experimental results validate the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n1. While previous works have primarily focused on incorporating auxiliary semantic information, this paper introduces the novel perspective of additionally leveraging visual information.\n2. To better align support images, descriptive texts, and synthesized images, the paper proposes an interesting volume-based alignment method, which offers mathematical interpretability.\n3. The experimental evaluation is relatively comprehensive and complete, demonstrating strong performance across benchmarks.\n\n**Weaknesses**:\n1. Regarding the selection of top-k synthesized images, how is the reliability of the selection ensured? The paper employs a LLM as a discriminator to perform pairwise comparisons. However, I wonder whether this process involves the support set at all. If not, can the visual consistency between the generated images and the support set images truly be guaranteed?\n2. I understand this method as an attempt to transfer knowledge from large models to smaller ones, thereby enriching existing representations. The integration of a large model undoubtedly introduces additional computational overhead during training. I am curious about how the paper addresses or justifies this concern.\n3. The term fusion factor k can be easily confused with k-shot. Moreover, the paper lacks an ablation study on this hyperparameter. I would like to know how the model performance changes as k varies from 0 to 1. Additionally, what is the optimal value of k for each dataset? These details could be included in the supplementary material.\n4. Since your model internally utilizes the CLIP architecture, why not directly perform parameter-efficient fine-tuning based on CLIP?\n\nQUESTIONS:\nPlease refer to the weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper provides a novel way to efficiently extract semantic features from an image in a few-shot regime. Authors perform this extraction by proposing an approach containing two novel procedures: first, they augmentate the available examples by providing an iterative procedure to extract the semantic features of an image as text and use them to generate a new semantic-grounded image; second, they provide a geometry-based loss function to extract those features in a way they are aligned with the semantics of the class.\n\nSTRENGTHS AND WEAKNESSES:\n= Strengths:\n* The approach provided by the authors is (as to my knowledge) novel and well implemented, although it can be seen as a more complex version of the work proposed in https://arxiv.org/pdf/2311.18649 . \n* The overall framework (backbone + pre-training + fine-tuning) is unarguably better than the baselines shown. \n\n= Weaknesses:\n* I found it a little difficult to understand the overall system and I think that its description (mainly section 3.3 \"Cross modal fusion\") can be improved. For instance, it is not clear from figure 2 if the \"Block\" is shared in all cases, or where the $\\beta$ vector appears on the figure. Other than that, the paper is generally well written.\n* My main concern is the conclusions taken by comparing with the used baselines. I understand that they are taken directly from the corresponding table of the previous works, but there seems to be a lot of variance in the pre-training data, the backbone model and the fine-tuning procedure. That makes it difficult to determine if the framework is better because, for instance, the tieredImageNet was used in the pre-training stage or because of the loss function designed. As such, claims like the ones in line 213-214 (\"These results highlight the effectiveness and the advantage of constructing complementary multimodal prompts with global alignment across all features for full semantic integration.\") are not necessarily true, since there is (as to my understanding) no direct comparison between different fine-tuning procedures for the same pre-training stage.\n\nQUESTIONS:\n* I do not understand table 5. Does each row represent a different baseline with the specified properties (i.e., text/vision/aling)? In that case, an additional column with the name would be handy. If not, does each row correspond to the same pre-training stage and different fine-tuning procedures? In this case, that may help to clarify the observation made in the \"weaknesses\" section. \n\n* Baselines used for each table differ in some cases. Is there any particular reason for this? For instance, SemFew and ECER are the second best methods in table 2 but they don't appear in table 3. Similarely, SUITED is the second best method of table 3 but it is not shown in table 4. Is there any reason why those methods could not be used in other scenarios?\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}