Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMs
Authors: Hari Chandana Kuchibhotla
Sai Srinivas Kancheti
Abbavaram Gowtham Reddy
Vineeth N. Balasubramanian
Venue: Accepted by TMLR ()

Abstract: Fine-grained Visual Recognition (FGVR) involves distinguishing between visually similar categories, which is inherently challenging due to subtle inter-class differences and the need for large, expert-annotated datasets. In domains like medical imaging, such curated datasets are unavailable due to issues like privacy concerns and high annotation costs. In such scenarios lacking labeled data, an FGVR model cannot rely on a predefined set of training labels, and hence has an unconstrained output space for predictions. We refer to this task as Vocabulary-Free FGVR (VF-FGVR), where a model must predict labels from an unconstrained output space without prior label information. While recent Multimodal Large Language Models (MLLMs) show potential for VF-FGVR, querying these models for each test input is impractical because of high costs and prohibitive inference times. To address these limitations, we introduce Nearest-Neighbor label Refinement (NeaR), a novel approach that fine-tunes a downstream CLIP model using labels generated by an MLLM. Our approach constructs a weakly supervised dataset from a small, unlabeled training set, leveraging MLLMs for label generation. NeaR is designed to handle the noise, stochasticity, and open-endedness inherent in labels generated by MLLMs, and establishes a new benchmark for efficient VF-FGVR.

Published in Transactions on Machine Learning Research (08/2025)
Efficient Vocabulary-Free Fine-Grained Visual Recognition in the Age of Multimodal LLMs
Hari Chandana Kuchibhotla
ai20resch 1006@0th..0o
Indian Institute of Technology Hyderabad, India
Sai Srinivas Kancheti Indian Institute of Technology Hyderabad, India
cs21resch01000 (httth000.@iiioooo
Abbavaram Gowtham Reddy CISPA Helmholtz Center for Information Security, Saarbrücken, Germany
Vineeth N Balasubramanian
vineeth. nub motteshnnnmmmmooo....cooo com & vineeth.nb@cse.itee www.bbeeeeeehee....e.. ac. ic.in
Microsoft Research India Indian Institute of Technology Hyderabad, India
Reviewed on OpenReview: https: // openreview. net/ forun? id- FuAOUMu9X2
Abstract
Fine-grained Visual Recognition (FGVR) involves distinguishing between visually similar categories, which is inherently challenging due to subtle inter-class differences and the need for large, datascts unavailable due issues like privacy concerns and high annotation costs. In such scenarios lacking labeled data, an FGVR model cannot rely training labels, and hence has an unconstrained output space for predictions. We refer to this task as Vocabulary-Fres FGVR (VF-FGVR), where a model must predict labels from an unconstrained output space without prior label information. While recent Multimodal Large Language Models (MLLMs) show potential for VF-FGVR, querying these models for each test imput is impractical because of high costs and prohibitive inference times. To address these limitations, introduce Ncarost-Neighbor Label Refinement (NeaR), novel approach that fine-tunes downstream CLIP model using labels generated by an MLLM. Our approach constructs weakly supervised dataset from small, unlabeled training set, leveraging MLLMs for label generation. NeaR is designed to handle the noise, stochasticity, and open-endedness inherent in labels generated by MLLMs, and establishes new benchmark for efficient VF-FGVR.
datasets. In domains like medical imaging, such curated
predefined set of
1 Introduction
Fine-Grained Visual Recognition (FGVR) is a task in computer vision that focuses on distinguishing between highly similar categories within a broader class (Wei al., 2021). This task has gained increased importance in recent years given the success of foundation models on coarso-grained classification tasks. Traditional image classification aims to differentiate between dogs, cats, and birds, while FGVR aims distinguish between different subcategories, such Tennessee Warbler, Yellow- among birds as an example. Fine-grained understanding is essential for wide range of applications, including biodiversity studies (Liu et al., 2024b; Yang et al., 2020), medical diagnosis (Ridzuan et al., 2022; Sohoni et al., 2020), manufcttuing (Yang et al., 2022), fashion retail (Cheng et al., 2021), and agriculture (Yang et al., 2020).
Warbler and Orange- crowned Warbler Orange-crouned
FGVR poses significant challenges due to the subtle differences between categories and the need for large annotated datasets. Typically, fine-grained classification datasets are annotated by domain experts who

---

Published in Transactions on Machine Learning Research (08/2025)
meticulously examine each image and assign a corresponding label. However, when such domain experts are unavailable, not only is labeled data unavailable, but the underlying fine-grained categories of interest may also be unknown. We refer to this task as Vocabulary- FGVR (VF-FGVR), where the vocabulary of fine-grained labels is not provided. In this context, pre-trained Multimodal Large Language Models (MLLMss), which are trained on vast corpora of image-lext data and possess extensive world knowledge, provide a contemporary solution (Reid et al., 2024). MLLMS excel at zero-shot multimodal tasks such as Visual Question Answering (VQA). By recasting FGVR as VQA problem, where we ask the question 'What is the best fine -aae-ngaieedd class label for this image?', MLLMs can predict fine- ane-grained labels in a zero-shot manner without prior training on a specific dataset. The ability of MLLMs to operate in VF setting presents a promising approach for domains where curated, labeled datasets are scarce or unavailable. However, querying such large models for each test input is computationally expensive and time-consuming, mak- ing their usage impractical at scale. As shown in Table 1, performing inference with GPT-4o (Achiam et al., 2023) on 32, 203 test images from benchmark FGVR datasets takes R 17.5 hours of querying and incurs cost of x USD $100.
Method
Training Time
Inference Time
Cost ($) cACC
FizeR (Liu et al., 202
10
1.12
57.0
CPT-4
17.5 h
100
59.2
25-CLIP GPT-40
0.03
54.6
NeaR + GPT-4o (Ours)
1.57
0.03
67.6
LLLMA
9.12 h
0
18.1 1
2S.CLIP LLaMA NeaR LLAA (Ours)
Hence, especially considering the need for sustainable Al systems, there is an imminent need for an efficient VF-FGVR system that performs on por with MLLMs on the fine-grnined understanding lask, while being efficient in terms of time and computational resources required. We focus on this problem in this work. To this end, we only consider access to a small unlabeled set of training images belonging to the classes of interest, with no information regarding the individual class names or even the total number of classes. Such unlabeled datasets are relatively easy to obtain for many domains for instance, a collection of unlabeled photos of exotic birds. We empirically show in 4 that having about 3 images per class is sufficient to learn an FGVR system that can perform inference for any number of lest samples in the considered experiments. To solve this VF-FGVR task, we propose to label this training set by querying an MLLM for each image. Such a noisily labeled training set can be used in different ways a simple strategy would be to utilize this MLLM supervision to build a zero-shot classifier over the set of generated labels using a pre-trained CLIP (Radford et al., 2021) model. Another way would be to naively fine-tune the CLIP model using the generated labels. Tn both these approaches, test images can now be classified by the CLIP model without the need for expensive forward passes through an MLLM. Although such simple methods are efficient in compute and time taken, they fall short on performance as shown in 4. This is because MLLM outputs are inherently noisy and open-ended, so the generated labels do not necessarily provide strong supervision.
1,77 A
0.03 h
65.0
Table 1: Performance and cost metrics for different methods on benchmark FGVR datasets (computed over 32, 503 images from the test sets). t= proprietary models, 1 = open-source models (both used only for inference). Our method NeaR achieves clustering accuracy (cACC described in $ 4) that exceeds even direct MLLM queries, at a fraction of cost and time taken.
To address these limitations, we propose Nearest-Neighhoor Label Refinement (NeaR), a method designed to learn using the noisy labels generated by an MLLM. Our approach first constructs a candidate label set for each image using the generated labels of other similar images. In line with prior work on lcarning with noisy labels (LNL) (Li et al., 2020), we partition the dataset into clean and noisy samples. We then design a label refinement scheme for both partitions that can effectively combine information from the constructed candidate set and the generated label. Finally, to address the open-ended nature of MLLM outputs, we incorporate a label filtering mechanism to truncate the label space. Our method NeaR thus enables us to handle the inherently noisy and open-ended labels generated by MLLMss allowing us to effectively fine-tune a downstream CLIP model. As shown in Table L, for GPT-4o, our approach can achieve performance exceeding that of direct inference while incurring only 1/100th of the total inference cost, and requiring a negligible fraction of inference time.
Our key contributions can be summarized as: (i) Differing from existing works, we study how contemporary models can be used to build cost-efficient vocabulary-free fine-grained visual recognition system, (ii) We propose a pipeline that can handle noisy and open-ended labels generated by an MLLM. Our proposed
2

---

Published in Transactions on Machine Learning Research (08/2025)
method NeaR. leverages similarity information to construct a candidate label set for each image which is used to mitigate the impact of label noise. We also design label filtering mechanism to improve classification performance. (iit) We perform a comprehensive set of experiments showing that NeaR outperforms existing works and the MLLM-based basclines we introduce for VF-FGVR (detailed in the baselincs paragraph of $ 4), achieving this in a cost-efficient way.
2 Related Work
GraenGr Visual Recognition. FGVR (Wah et al., 2011; Maji et al., 2013) aims to identify sub categories of an object, such as various bird species, aircraft type etc. FGVR has been extensively studied in prior work (Wei et al., 2021). A key limitation of these methods is their reliance on annotated datasets, which are often unavailable in many important domains like e-commerce and medical data. With advancements in Vision-Tanguage Models and MLLMs, the burden of dataset annotation can be alleviated, reducing the need for extensive human effort.
Foundation Models for VF-FGVR. Recent advancements in MLLMs have led to models demonstrating strong zero-shot performance across wide range of multimodal tasks (Li et al., 2023; Achiam et al., 2023; Reid ct al., 2024; Touvron et al., 2023; Liu et al., 2023). These MLLMs can be applied to VF-FGVR by framing the lask as a VQA problem. MLLMs are broadly categorized into two types: (1) Proprietary models, such as GPT-4o (Achiam et al., 2023) and GeminiPro (Reid et al., 2024), and (2) Open-source models, including BLIP-v2 (Li et al., 2023), LLaVA-1.5-BB (Liu et al., 2023), LLaMA-322-11B (Touvron ct al., 2023) and Qwen2-7B (Wang et al., 2024). Recent works (He et al., 2025; Zhang et al., 2024) examine the zero-shot fine-grained performance of such MLLMs in the closed-world setting and show that finetuning can help. Such approaches are not applicable in the VF-FGVR setup where labeled data as well as the label space, is unavailable. As shown in Table 1, for both types of MLLMs, performing inference for every test point remains computationally expensive and time-consuming. To address this, recent works have developed more efficient. solutions for VF-FGVR. For instance, (Liu et al., 2024a;c; Conti et al., 2023) propose pipelines that use cascades of MLLMs. FineR (Liu et al., 2024a) presents a pipeline combining VQA systems, Large Language Models (LLMs), and a downstream CLIP model, leveraging unsupervised data to build a multimodal classifier for inference. RAR (Liu et al., 2024c) uses a multimodal retriever with external memory, retrieving and ranking top-k samples using an LLM. RAC (Long et al., 2022) uses retrieval to refine features via a learned transformer using labeled memory bank, operaling under full supervision. In contrast with RAC, NeaR is lightweight, crekbone-agnostic and avoids complex memory- meserr- architectures. Furthermore both RAR and RAC operate under full supervision, making them unsuitable for the VF-FGVR task. CaSED (Conti et al., 2023) approaches VF-FGVR by accessing an external database to retrieve relevant text for & given image. Nevertheless, these methods are often complex and do not fully exploit the advancements in MLLMs, resulting in suboptimal performance.
Prompt Tuning. Prompt-tuning methods add small number of learnable tokens to the input while keeping the pretrained parameters unchanged. The tokens are fine-tuned to enhance the performance of large pre-trained models on specific tasks. Context Optimization (CoOp) (Zhou et al., 2021) was the first to introduce text-based prompt tuning, replacing manually designed prompts like 'a photo of a with adaptive soft prompts. We study the impact of our method under other prompt-tuning methods such as VPT (Jia et al., 2022) and IVLP (Rashoed et al., 2022) in Appendixξ A8.5.
Learning with Noisy Labels. (Arpit et al., 2017) demonstrated the memorization effect of deep networks, showing that models tend to learn clean patterns before fitting noisy labels. To mitigate this, (Han et al., 2018; Chen et al., 2019) introduce iterative learning methods to filter out noisy samples during training. (Arazo et al., 2019) proposed a mixture model-based approach to partition datasets into clean and noisy subsets, leading to more reliable training. Building on these insights, DivideMix (Li et al., 2020), a state-of-theeatt LNL. method, combines semi-supervised learning with data partitioning to achieve superior performance on noisy datasets. JoAPR (Guo & Gu, 2024) is a contemporary approach to fine-tune CLIP on noisy few-shot data. We compare against JoAPR in Tab 4.
3

---

Published in Transactions on Machine Learning Research (08/2025)
Training Stage
Inference Stage
Images
Step-1
Step-2
Step-d
Provide best grained clese biw for this image.
C:
(Rlddheeecee woodlpeecker: Marsh When. ...Cattird Dank Swallow)
Viee Kirkng kin
beleleeee vesageditr Fout woutroke)
Labil Fiboring
Red-heeded woodpecker
Image Rurning
Coa : (Rack-aaddd woodpicker Cathidd
Lanquage Modil
S
Teat Images
User
Label Ref nemeemmntt
Paramitars
Lynal + f(x)
and Cand date
Set Update
Traign Freeder
(Nin Xucl
woodserter
Harch Wran
Settin
Preduction
Figure 1: Overview of our proposed method, NeaR, for Vocabulary-Free Fine- Grained Visual Recognition (VF-FGVR). In the Training Stage, start with few unlabeled images. Step-1: An MLLM generates best-estimate fine-gratd ine-grained label (e.g.. (Red-headed woodpecker') for each image. Step-2: A candidate label set is constructed using K-Nearest Neighbors, capturing related fine-grained classes. Step-3: The model is fine-tuned using CLIP-based architecture. A GMM is applied to the loss to partition the data into clean and noisy samples. Based on this split, label refinement mechanism is used to further update and refine the labels. The final loss, Iifinat, is then computed, and the model parameters are updated accordingly. In the Inference Stage, we apply label filtoring to limit the label space (Step-4). Our approach handles the noise and open-ended nature of MLLMMgennrated labels, siguificantly reducing inference time and cost while maintaining performance.
3 Methodology
As shown in Table 1, although MLLMs are capable of performing VF-FGVR, labeling every test image is expensive and time consuming which limits their practical application. To address these limitations, we propose Nearest-Neighbor Label Refinement (NeaR), a method designed to leverage MLLMs efficiently for VF-FGVR. Our approach begins by constructing a candidate label set for each unlabeled training image as described in $ 3.2. Next we partition the data into 'clean' and 'noisy" samples using Gaussian Mixture Model (GMM) and the small-loss rule (Arpit et al., 2017). In $ 3.3, we show how to refine labels of noisy samples by incorporating information from the candidate sets. The final loss function is simple cross-entropy loss, where the refined labels serve as the targets. As outlined in $ 3.1, using MLLM outputs directly can result in an excessively large label space, which can hinder performance. To address this, NeaR incorporates a label filtering mechanism to truncate the label space, which boosts classification performance while improving efficiency. Once the CLIP model is fine-tuned, can classify test images without requiring additional expensive forward passes through an MLLM, significantly improving inference efficiency. Our methodology NeaR is able to exceed the performance of direct MI.LM-based classification at just a fraction of the cost. and compute. An overview of our methodology is presented in Figure 1, and the pseudocode is detailed in Algorithmš in the appendix. We begin by discussing the necessary preliminaries.
3.1 Preliminaries
Problem Formalization. We consider a setting where only small, unlabeled training set of n images X = (a.) 2; E X is available. We assume that this training set has at least m-shot samples for each class of the unknown ground-truth class name set G. We also study a more realistic scenario where there is class imbalance in $ 4.2 and observe that the performance of NeaR does not degrade. Note that no further information about G is known, including its cardinality. For each image I;, we obtain a class name 4 = L(x,,)) p) from an MILM L, where p a simple text prompt 'Provide a best Aine-grained class name for this image.' that guides the MLLM to generate a fine-grained class name for the image. The generated dataset. D (x, 1,) ) consists of image-label pairs, and the output space of class names is
4

---

Published in Transactions on Machine Learning Research (08/2025)
denoted by C = Ui=1 4. W.L.ogg. we assume C is lexicographically ordered and we denote the k = |C] labels by C = [(1, c,, Ct ). Let yi E [0. 13 E [n], denote the one-hot encoding of the text label 4 for image i. w/- 1f Cj and 0 otherwise. Due to the inherent noise and stochastic nature of MLLM generated labels, it is common to have |C] > [g]. Furthermore, the number of generated labels increases with the size of the training set and can become prohibitively large, hampering training and reducing efficiency of the downstream CLIP classífier.
CLIP Classifier: CLIP (Radford et al., 2021) consists of an image encoder I and text encoder T trained contrastively on image-text pairs. For VF-FGVR, we first query an MLLM to build a few-shot dataset 'D with label space C. CLIP classifies image x as ((x) = arg max cos cos(I(z)) (I)), T()). We call this baseline ZS-CLIP. Note that ZS-CLIP does not leverage the paired supervision in D and thus serves as a simple baseline. CLIP can benefit from fine-tuning on a small labeled datast. CoOp (Zhou et al., 2021) is prompt-tuning approach that adds a small number of learnable tokens to the class name. We denote CLIP's class predictions by fo(x) E A where A denotes the k simplex. The prompts are trained on the dataset D by minimizing
cross-entropy loss (C) (C= n ΣΣ (2 Naive finetuning using CoOp on dataset generated
by an MLLM can be susceptible to label noise. Our NeaR method mitigates this issue by refining labels through nearest-neighbor information. The following sections detail how NeaR constructs candidate sets, learns with these sets, and filters labels for effective downstream performance.
3.2 NeaR: Candidate Set Construction
The label generated by an MLLM in response to a prompt may vary significantly from the ground truth class label for each image. We propose to leverage local geometry to mitigate the noise in generated labels. More formally, we make the manifold assumption, which suggests that similar images should share similar or identical class labels (Iscen et al., 2022; Li et al., 2022). This is particularly useful when the label 4 assigned to image xy by the MLLM is incorrect, which we refer to as a noisy label. By constructing a candidate label set, increase the likelihood of including the true label or a semantically closer alternative in the candidate set rather than relying solely on the potentially incorrect label provided by the MLLM.
Table 2 shows that the semantic similarity between the best label in the candidate set and the ground truth is higher than that of the noisy single label for the training images, supporting our hypothesis. We use CLIP's pretrained image encoder I extract image features of the entire training set X. For each image I,, select the top-k most similar images (including 2; itself) and gather their corresponding labels to form the candidate set S: = (4,,
Method
Bird-200
Car-196
Dog-120
Flower-102
Pet-37
MLLM Labels
69.9
78.6
70.9
57.9
Ransoo CS
72.8
78.9
73.6
61.6
85.9
KNNN CS (ours)
78.0
8133
75.5
65.8
87.4
Table 2: The table compares the quality of labels gen- erated by the MLLM (LLaMA), a Random candidate set (CS) where we pick 2 other labels (for x = 3) at random from the set of all MILM generated labels, and K-NN CS using sACC. While Random CS mod- estly increases the likelihood of including the true label compared to using MLLM labels directly, K-NN CS significantly outperforms Random CS, generating a superior candidate set and validating our hypothesis.
In this work, we choose λ = 3. The resulting dataset is reconstructed as D. = {xx, 4, S,) incorporat- ing the candidate sets rather than single labels alone. An alternative way of noise mitigation is to have the MLLM directly generate candidate set of 'top-k' labels for each image, instead of just a single label. However this approach does not make use of similarity information between images, as each candidate set is now generated independently, leading to an excessively large label space. We empirically demonstrate the effectiveness of our nearest-neighhor based candidate set generation over other alternatives in Appendix & A8.4.
3.3 NeaR: Learning With a Candidate Set
We treat the candidate sets as a source of supplementary similarity information to be used in conjunction with the label. As explained below, for a noisy image, where the initial label 1 is incorrect, we propose
5

---

Published in Transactions on Machine Learning Research (08/2025)
relying the candidate set St to mitigate the impact of noise. Conversely, for clean image, we trust and utilize its generated label 4. Detecting Noisy Samples. It has been demonstrated in (Arpit et al., 2017) that models tend to learn clean samples before noisy ones, resulting in lower loss values for clean samples. Following DivideMix (Li et al., 2020), for every training epoch, we fit a two-component Gaussian Mixture Model (GMM) over the cross entropy loss values of all training samples {L(r, 4)] where [(x;,4) 4)) = - ΣE= vd log(fe (z;)), is the one-hot encoding of 4. The component with the smaller mean value models the clean samples, while the other component models the noisy ones. The posterior probability wa = PGMAN (nleanc x4) computed from the fitted GMM is used to model the likelihood that a sample z; clean. This GMM is refitted for every training epoch, cnabling dynamic cstimation of label noise over time. We now partition the training data into clean Xet {a, E wi 2 r and noisy Xne = X Xet sets based clean probability threshold т. We use the average clean posterior as adaptive threshold for every epoch, ie mer== 1 Σ ws. The effect of different thresholding strategies presented in Appendix & A8.6.
Warm-up. Warm-up strategies are commonly used to speedup convergence and stabilize training. As demonstrated in (Li et al., 2020), an initial warm-up phase allows the model to learn the clean samples better, resulting in better separation between the losses of clean and noisy samples. During warm-up, we train prompts for a few epochs (10 in our experiments) by minimizing the cross entropy loss over the generated labels I with one-hot representation y;
n k
I Tawarmup (0)
x log((s
==
This warm-up step lays the groundwork for effective training by allowing the model to initially focus on images labeled correctly by the MLLM.
Candidate Set Guided Label Refinement. Following the initial warm-up phase, we make a forward pass over the entire training set at each training epoch to fit GMM and partition data into clean and noisy samples Xet & Xus as described earlier. We model the confidence of the candidate set S by vector qi E R for each image i E [n], initialized qf = - confidence over classes belonging to the candidate set, and zero for non-members. A candidate set is derived from neighboring images and provides broader view of possibly correct labels. Our approach constructs refined labels to effectively leverage this additional information. We propose to construct refined labels for clean and noisy images differently. For an image x, with one-hot label ys and candidate set confidence (s, we construct a refined label y; by performing label-mixup using model predictions so as:
|.. if e₁ E S, and 0 otherwise. This initialization reflects uniform
V: = shrp((w: we - ii + (1 (1- (1-w;) ws) fo(x:), T), ir E Xa rsc(shrp(u; q + (1 w.)) fo TT. o/w
where w, is the GMM clean posterior probability, and fa,),) denotes the CLIP model class probabilities with learnable prompts 0. The label-mixup conditioned on clean posterior probability wi, enabling the model to
rely on the pseudo-label when w, is small. The sharpen function shrp(y, T)' (v') + Σ + as defined in (Berthelot et al., 2019), adjusts a probability distribution to be more confident using a temperature T. The rescale function is rscly, q)' = (y (v E (y (10)), where @ represents the hadamard product, rescales
probability y with the current confidence estimates of the candidate set g. This ensures that the refined label has non-zero probabilities only for the candidate labels. For both clean & noisy images, we update candidate set confidence to be used in the next epoch as 9 = rsc(fo(zs), 1[4]) where 1[q.] is 1 at non-zero indices. Prompts are learned by minimizing the cross-entropy loss between the refined labels ğ and CLIP model output.
-1
Linall (0)
ΣΣ 1=1 1=1
vl
log(5
n
6

---

Published in Transactions on Machine Learning Research (08/2025)
Method
Bird-200
Car-196
Dog-120
Flower-102
Pet-37
Average
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
CoOp-GT (Upper Bound)
81.1
06.6
64.0
79.9
74.6
82.1
88.2
92.5
70.0
CaSED
25.6
50.1
26.9
11.1
55.9
52.3
60.9
63.6
43.7
52.6
FineR
51.1
00.5
49.2
63.5
48.1
64.9
63.8
51.3
72.9
72.4
57.0
64.3
RAR
51.6
69.5
53.2
63.6
30.0
65.2
63.7
53.2
74.1
74.8
58.5
65.3
(GPT-40
68.8
85.2
37.4
61.5
71.1
80.4
50.5
51.6
68.2
63.5
50.2
72.4
ZS-CLIP-GPT50
48.8
72.5
42.9
59.5
43.8
69.1
18.2
53.0
68.2
78.7
54.6
66.6
CoOp-GPTT44
75.4
50.8
72.9
70.4
$3.5
64.1
69.2
NoaR-GPTio
55.8
75.6
57.0
60.0
61.6
74.4
80.6
52.1
82.9
84.0
67.6 67.6(+3365%
69.2
IGennini Pro
62.7
62.8
45.3
56.8
73.3
Pem
51.7
74.6
11.6
61.7
58.9
72.6
57.7
29.1
71.7
78.6
56.3
67.3
CuDp- GeminilPro
55.2
75.9
541.2
61.5
62.7
73.8
68.3
51.2
81.6
83.8
63.6
69.2
CessR-GeminlPro
55.9
76.0
54.9
61.1
61.7
75.4
77.9
53.2
79.4
80.8
(6+6()+)%
69. (++0.1%)
4Qwen2- VL-7B- IL-TB-Instru
D0wen2-VL--B-- DS-CLLP-Qwea2
53.0
75.3
45.6
61.77
68.7
78.8
84.8
72.7
77.7
85.1
66.2
76.1
41.0
06.0
00.8
39.3
70.5
66.7
55.2
72.4
77.2
58.0
65.9
CoOp-Qweu2
72.1
73.4
77.0
65.2
NeaR-Qwen2
48.9
72.0
65.6
63.2
02.0
73.3
81.4
68.0
84.6
86.8
66.5(+11.3%% +..%)
71.7(-0.3%)
!LLAMA-3.2-11B
11.4
70.6
144
61.6
35.0
71.8
66.0
65.1
48.1
78-CI.P-LLaMA
48.7
06.3
45.8
60.6
57.4
65.9
74.8
58.4
76.0
78.4
60.5
65.9
CoOp-LLaMA
49.2
68.7
45.5
00.7
58.4
68.1
76.9
59.8
74.4
79.2
60.7
67.4
No.R-LLaMA
78.6
83.5
65.0( +4.3%)
(1..8(+22) 2.4%%
Table ZS-Zero Shot, t proprietary models used for inference, open-source models used for inference. Our results shown here are for k = 3 and m = The first row is CoOp-GT, where we finetune CLIP model using ground-tre truuh labels, serving as an upper bound. The second partition consists of contemporary VF baselines of which FineR (Liu et al., 2024a) is best performing. We outperform FineR by large margin, even when using weaker open-source MLLMs. The next four partitions are for labels generated by various MLLMss We compare NeaR against CoOp within each partition, and highlight best numbers in bold. Our method NeaR outperforms all contemporary baselines, as well as ZS-CLIP and CoOp baselines for a variety of MLLMs.
Connection to PRODEN. Our loss is similar in spirit to losses designed for Partial Label Learning (PLL), such PRODEN (Feng et al., 2020), which allow learning when only candidate labels are present. However unlike the PLJ. setting, our candidate sets are constructed for every image using noisy MLLM outputs, and may not contain the true label. Furthermore, our method uniquely benefits from access to an initial 'best-cstimate' label 4 generated by the MLLM, which is not exploited by traditional PLL algorithms. This best estimate label allows us to differentiate clean samples and helps training convergence by transferring knowledge from clean to noisy samples through iterative updates of g.
3.4 NeaR: Label Filtering
Although we train on the entire label set C, we observe that many labels are noisy and can be removed from the inference time label space. Let Fetip = {cz 3x E X s.t arg max fz(} be a filtered set labels
]E[M]
which are predicted by CLIP on the training set. Let Feand = {c; s.t = arg max g be another filtored jE(A]
set of labels which are predicted using just the candidate scts. We propose to keep only those labels which belong to both sets. The evaluation time label space is Crest = Fetip n Foand and the inference time prediction
of an image r is ((x) arg max sim(I(xr, (t)), where i are the learned prompts. Label filtering is effective IeCren
shown in Table A14.
4 Experiments and Results
In this section, we comprehensively evaluate the classification performance of NeaR for the VF-FGVR task. We begin by describing the datasets, metrics and benchmark methods compare against.
7

---

Published in Transactions on Machine Learning Research (08/2025)
Datasets: We perform experiments on five benchmark fine-graince datasets: CaltechUCSD Bird-200 (Wah et. al., 2011); Stanford Car-196 (Khosla et al., 2011); Stanford Dog-120 (Krause et al., 2013), Flower- 102 (Nilsback & Zisserman, 2008), Oxford-IIIT Pet-37 (Parkhi et al., 2012). Following (Liu et al., 2024a), for each datasct, NeaR and other basclines only have access to m unlabeled training images per class. Unless specified otherwise, we assume m 3. 3. Results for 1 1 m S 10 are shown in Figure 2.
Baselines: We compare our method NeaR against four different classes of baseline methods. (i) Direct Inference on MLLMs. For every test image, we directly query an MLLM for fine-grained label using text prompt such as 'What is the best fine-grained class name for this image?'. We evaluate two proprictary MLLMs- GPT-4o (Achiam et al., 2023) and GeminiPro (Reid et al., 2024) and two strong open-source MLLMs, LLaMA-3.2-1 --iiiii--nnoo--sttrreee (Touvron et al., 2023) and Qwen2-VI. VL- 7B-Instruct (Wang et al., 2024). In the Appendix $ A8, we show results on two other weaker open-source MLLMs, BLIP-2 (Li et al., 2023) and LLaVA--15 (Liu et al., 2023). (ii) Contemporary VF Basclines. We consider three contemporary basclines which do not require expert annotations but use foundational models to perform VF-FGVR CaSED (Conti et al., 2023), FineR (Liu el al., 2024a) and RAR (Liu el al., 2024c). (iii) ZS-CLIP with MLLM label space. As described in 3.1, can perform zero-shot classification using pre-trained CLIP over the label space generated by querying various MLLMS on training images. We consider ZS- CLIP-LLLMM,, and ZS- ZS-CLIP-Qwen2. (iv) Prompt Tuning Baselines. Following CoOp as described in $ 3.1, we directly perform prompt-tuning using the labels generated by an MLLM. We consider four variants CoOp-GPT4o, Co0i-GeminiPro,, CoOp-LLaMA, and CoOp-Qwen2.
four variants of ZS-CLIP- - CS-CLLP-G DS-CLLP-GeminiPe
Evaluation Metrics: In the VF-FGVR setting, NeaR as well as all other baselines operate in an unconstrained label space, making accuracy an invalid metric since the predicted labels may never exactly match the ground-truth labels. Following FineR (Liu et al., 2024a) and CaSED (Conti et al., 2023), we evaluate performance using two complementary metrics: Clustering Accuracy (cACC) and Semantic Accurucy (sACC). cACG measures the ability of the model to group similar images logether. For M test
M Σ 1wt = p((i)), F=1
images with ground-truth labels y² and predicted labels 9, cACO is computed as max PEP())
M
where P()) is the set of all permutations of the generated labels. Since cACC disregards the actual label name, it does not measure if the predictions are semantically correct. Despite this limitation, cACC is a strong evaluation metric and is widely used in areas such as GCD (Vaze et al., 2022), where the goal is to assess consistency of predictions rather than exact label semantics. Semantic closeness is captured by sACC, which measures the cosine similarity between Gentence-BERT (Reimers & Gurevych, 2019) embeddings of the predicted and ground-truth labels. As observed in (Liu et al., 2024a), SACC is a more forgiving metric than cACC, because embedding based similarity methods can capture general semantics even for completely distinct labels. We hence consider cACC as representative of the model's performance, with sACC acting as a sanity check to ensure that the predicted labels remain meaningful.
Implementation Details: We use CLIP VIT-B/16 (Radford et al., 2021) as the VLM, whose image encoder we also use to find the n-nearest neighbors, with x = 3 by default. We strictly follow FineR(Liu et al., 2024a) for the default number of shots i.c m = 3 and for the few-shot training splits. For both the CoOp bascline and our method, we introduce 16 trainable context vectors. The same set of prompts are optimized during the warmup stage, and for the subsequent. training stage. We use SGD as the optimizer and train for 50 epochs, with 10 warmup epochs. We use a temperature of 2 in the sharpening function. Our batch size 32. We use the SGD optimizer with learning rate of 0.002, and use both constant learning rate scheduler and cosine annealing scheduler sequentially. The training hyperparameters are the same for CoOp and NeaR. We sample an equal batch of clean and noisy samples during every epoch. We run all our experiments on a single Nvidia Tesla V100-32GB GPU with an Nvidia driver version of 525.85.12. We use PyTorch 2.4.0 and CUDA 12.0. We utilize the publicly available meta-llama L 3.2-1a-...-11---- 1i.---ii--iiiio------ Instront model and Qwen/ Seenn//wen2-V--22 -Tnt2-- model from HuggingFace. We observe that instruction tuned MLLLMs generate better labels compared to base models. We perform inference using the HuggingPace transformers
8

---

Published in Transactions on Machine Learning Research (08/2025)
library (Wolf et al., 2019). Unless otherwise specifically stated, we use LLaMA-3.2 as our default MLLM. Our code is available at httpss//gituubb.om/NNeel
4.1 Main Results
In this section we compare NeaR against baselines on five fine-grained datasets. In addition to the considered baselines, we benchmark against JoAPR (Guo & Gu, 2024), a state-of-the-art noisy label learning method designed for CLIP, and against PRODEN (Feng et al., 2020), a widely used partial label learning algorithm.
Benchmarking NeaR Against Baseline Methods: We evaluate NeaR against the four categories of baselines introduced in & Direct MLLM inference, contemporary VF methods, zero-shot CLIP, and prompt-tuned CLIP. The results are shown in Table 3, with all numbers reported for 3-shot training images. The first partition of the table, CoOp-GT, is the performance of fine-tuned CLIP model when provided with the ground-truth label space, serving an upper bound. The next partition consists of contemporary methods that can perform VF-FGVR. Out of these, FineR (Liu et al., 2024a) is conceptually closest to ours as it uses a combination of an LLM and a VQA system to construct a training-free CLIP based classifier. We outperform FineR on all datasets by a margin of at least +8% in average cACC, even when using labels from open-source MLLMs.
Moreover, as shown in Table 1, NeaR is significantly more efficient in terms of computation time. The next. four partitions in Table 3 report results using labels generated by GPT-4o, GeminiPro, Qwen2 and LLaMA-3.2 respectively. Within each partition, we first present results for direct inference with the MLLM, followed by ZS-CLIP, CoOp, and finally NeaR. Across all MLLMs, NeaR performs the best on average cACC, showing gains of at least, +3% over the CoOp baseline for GPT-4o, GeminiPro and LLaMA-3.2, and a gain of +1.3% over CoOp for Qwen2. Furthermore, we observe a large performance gain for the difficult Car-196 dataset, where NeaR--LLMA shows a gain of +7.1% in cACC over CoOp-LLaMA. These results highlight that NeaR effectively learns from the imperfect labels generated by MLLMs, leading to robust and efficient. fine-grained classification.
Comparison against PRODEN: Our loss function resembles those used in Partial Label Learning (PLL), such as PRODEN (Feng et al., 2020), which are designed to handle learning with only candidate sets. To study the efficacy of traditional PI.L approaches, we replace the traditional cross-entropy loss used in CoOp with PRODEN, and learn prompts using the candidate sets directly. The results are shown in Table 4, where NeaR out performss PRODEN by a large margin of 4.3% in cACC and 3.2% in sACC. Unlike traditional PLL where candidate sets are assumed to include the correct label, our candidate sets are generated for each image using noisy MLLM outputs and may not always contain the true label. Also, NeaR uniquely benefits from an initial "best-estinate" label 4 from the MLLM, which traditional PLL methods do not exploit. As described in $ 3.3, this best-estimate label is used to find "clean' images which have a higher probability of being correctly labeled. Knowledge from these clean samples helps resolve the ambiguity in candidate sets, improving performance.
Comparison against JoAPR (Guo & Gu, 2024), a Contemporary Noisy Label Learning Method for CLIP: JoAPR is prompt-tuning method designed to fine-tune CLIP on noisy few-shot data. In Table 4, we show the results of using JoAPR to learn from noisy LLaMA generated labels. Our method NeaR outperforms JoAPR by +4.6% in average cACC, and by +0.9% in average sACC. For JoAPR we use the default configuration suggested in the paper. These gains highlight that generic noisy-label learning methods, which expect structured noise (such as flips) within a closed label set, do not fully address the challenges posed by open-ended MLLM outputs. By incorporating similarity information, performing candidate set guided label refinement, and performing label filtering, NeaR provides robust solution to the VF-FGVR.
4.2 Ablation Studies
We conducted a thorough ablation to evaluate the contribution of each component in our pipeline in Table 5. We split the components of "Candidate Set Guided Label Refinement' into label mixup, rescaling, and using the candidate set guidance. The best performance is observed when all components including warmup,
9

---

Published in Transactions on Machine Learning Research (08/2025)
Method
Bird-200
Car-196
Dog-120
Flower-102
Pet-37
Average
cACO
sAOC
cACC
sACC
cACO
sACC
cACC
sACC
eACC
sACC
cACC
sACC
JAAPR-LLaMA
49.2
70.0
42.8
60.6
59.5
70.6
76.7
60.1
83.3
60.4
68.9
PRODEN
48.3
67.6
45.9
60.6
57.9
67.0
75.2
59.0
75.8
78.5
60.6
66.6
LLMM
51.1
70.2
52.5
60.8
59.2
70.2
78.6
61.7
83.4
86.1
64.9
(+-4.3%)
69.8 (+3.2%)
Table 4: Comparison of NeaR with contemporary noisy label learning method, JoAPR (Guo & Gu, 2024), for CLIP using -- labels. NeaR outperforms JoAPR with an average improvement of +4.6% in cACC and +0.9% in sACC. These results indicate that directly applying LNL methods is insufficient to handle the challenges of noisy MLLM outputs. By incorporating better label refinement using candidate set, and by performing label filtering, NeaR provides a robust solution to the VF-FGVR problem. We also compare our method against PRODEN (Feng et al., 2020). We significantly outperform PRODEN on both cACC and sACC.
Components
Bird-200
Car-196
Dog-122
Flower-
Pet-37
Avg
W.
GMM
LM
R.
Cand.
LF
eACC
sACC
cACC
sACC
cACC
sACC
eACC
sACC
cACC
sACC
rACC
sACC
x
/
/
V
48.8
71.5
19.8
61.9
56.8
71.9
75.4
61.3
79.3
85.4
62.0
70.4
x
4
/
X
/
48.5
69.1
48.9
50.7
57.1
69.6
76.9
60.1
81.8
84.1
62.6
68.5
x
/
/
51.3
71.3
48.6
00.7
10.4
71.4
75.9
00.1
79.7
84.0
63.0
09.5
/
x
/
/
49.1
69.4
50.9
60.6
59.5
71.8
77.1
61.4
77.8
84.1
62.9
69.5
/
/
X
x
28.0
69.1
48.3
60.2
56.6
71.0
73.9
60.2
80.6
84.4
61.5
69.0
x
49.1
70.2
50.1
00.1
57.9
70.1
75.6
00.6
81.4
85.9
62.8
09.3
/
V
28.5
59.7
69.6
60.1
818
62.6
4
/
V
51.0
70.2
62.6
60.9
50.2
70.2
78.6
61.7
83.4
86.1
65.0
69.8
Table 5: Full ablation study across five datasets. Columns "W (Warm-up), "GMM", "LM" (Label Mixup), "R." (Rescaling), "Cand." (Candidate sct), and "LF" (Label Filtering) indicate whether cach module is enabled (V) or disabled (X). The best performance is achieved with all the components.
GMM-based partitioning, label mixup, rescaling, candidate set guidance, and post-training label filtering, are used. This configuration yields the highest average cACC (65.0%) and SACC (69.8%), re-confirming the effectiveness of proposed approach.
Removing the warm-up phase leads a noticeable drop in performance, as it causes the GMM to produce a less reliable clean/noisy split. GMM is important because it splits the data into clean and noisily labeled samples. Without this split, the candidate set guidance is no longer used and only the MLLM generated labels are used throughout. Label mixup blends the label (one-hot or candidate set) with the model's prediction, and acts as a regularizer. Rescaling ensures that the refined label for noisy sample has non-zero probabilitien only for the candidate set, enabling it to learn from a relevant set of candidate labels. Removal of this leads to a misleading label signal for the noisy samples. Disabling either candidate set guidance or label filtering leads to noticeable drops across most datasets. To study the effect of removal of the candidate set guidance, we refine the labels of the noisy samples as T. shrp(fo(2;), T), i.c we only used the sharpened CLIP pseudolabel. We also remove the candidate set based filtering Foand, defined in Sec 3.4. These ablations confirm that our components which are simple but carefully designed are integral for the performance gains.
We study the realistic scenario of class imbalance in the few-shot training data. We simulate a long-tail distribution where we randomly select a small number of head classes (10 classes for pet-37) which have 4 S m S 10 samples, and the remaining tail classes have m samples. We show results in Table 6. We observe that there is no degradation in performance for both CoOp and NeaR. Infact, we note slightly better cACC and sACC values for NeaR on account of the slight increase in the training data.
Imbalanced Training Data:
Random Data Distribution In this section we evaluate the performance of NeaR under extremely random data sampling. The results across all five datasets is shown in Table 7. Our method outperforms CoOp-LLaMA by 8.3% in cACC and 1.4% in sACC, demonstrating its robustness to data imbalance.
A Analysis on Number of Shots m in Training Data: We explore the effect of the number of images used per class, as presented in Figure 2. We consistently use = 3 for candidate set construction across all shots.
10

---

Published in Transactions on Machine Learning Research (08/2025)
Method
Bird-200
Car-196
Dog-120
Flower-102
Pet-37
Average
cACC
sACC
cACC
sACO
cACC
sACC
cACC
sACC
cACO
sACC
cACC
sACC
FineRt
46.2
66.6
48.5
62.9
42.9
61.4
58.5
48.2
63.4
67.0
51.9
61.2
ZS-CLIP-LLAMA
48.9
67.0
46.9
60.3
55.9
64.5
71.4
75.5
72.2
64.5
CoOp-LLaMA
47.9
69.9
45.6
60.6
54.2
67.8
74.0
60.2
78.0
71.0
60.0
06.5
NeaR-LLaMA
50.9
69.9
52.6
60.4
60.2
71.2
80.3
63.8
81.6
86.2
65.7 (+5.7%%
70.3 (+3388%)
Table 6: Performance comparison of NeaR-LLaMA with other basclines under long-tail class distribution. Both NeaR and CoOp relain performance on imbalanced data compared to balanced sampling. NeaR outperforms CoOp by +5.7% in cACC.
Method
Bird-200
Car-196
Dog-120
120
Flower- 102
Pet-37
Average
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
cACC
SACC
cACC
sACC
CoOp-LLaMA
12.1
52.8
42.0
36.7
42.1
54.0
60.3
43.6
54.1
56.3
48.1
52.7
NeaR-LLaMA
13.0
52.0
47.8
55.3
55.8
65.7
13.5
32.0
61.9
55.7
56.4
(+-8.3%)
54.1 (1.1%)
Table 7: Evaluation of NeaR under randomly sampled data distributions across five datasets. NeaR surpasses CoOp-LLaMA by 8.3% in cACC and 1.4% in sACC, highlighting its robustness data imbalance.
For m = 1, our method performs poorly due to excessive label filtering. However, as the number of shots increases, our candidate set is more informative and performance improves markedly. Our proposed method, NeaR, outperforms CoOp for all m 2, especially at higher shots where there are more noisy labels. We also observe that cACC drops with increasing m due to increase in the size of the test time label space. Despite this, NeaR consistently outperforms others at all m. For example, at m-10, NeaR-LLaMA achieves 61.8 cACC, vs 54.7 (CoOp=LLaMA) and 52.2 (FineR). NeaR is more robust with increasing m with only a 3.5% drop from m=3 compared to 6% for others.
4.3 Impact of Varying No. of Gearest-Neighhee K
We leverage similarity information to build candidate set for each image by augmenting its label with the labels of its к nearest-ncighhors In this section we study the effect of varying K from to 9 on the performance NeaR-LLaMA. We perform this experiment for 9-shot data from the Flowers- 102 dataset, to ensure that higher values of к give meaningful results. The results in Figure 3 show that NeaR performs well across a large range of K values, and justifies our choice of k = 3. Note that setting x = 1 is not the same CoOp-LLaMA, but is the result of NeaR. with q: = yi. The results also highlight two competing factors that influence the performance of NeaR as varies:
Improved Label Quality with Larger Candidate Sets A larger candidate more likely to involve semantically closer label. This is reflected in the upward trend of cACC from k 1 to = 3.
Inereased Noise with Larger Candidate Sets For higher values of k, while the likelihood of including better labels in the candidate set increases, it offset by the addition of irrelevant labels. noisier candidate set makes it harder for the algorithm to disambiguate the best label in the candidate set. This leads to plateau or even slight decrease in cACC for 3.
To generalize this idea, we conduct an experiment where m randomly sampled between and 10 as shown in Table 8, and we ablate over different values of k ranging from I to 9. We find that the cACC remains relatively stable across this range, with only minor variations. This indicates that our method is not particularly sensitive to the choice of k, which can be selected flexibly. In our implementation, we use k=3 as a practical default.
4.4 Choice of Vision Encoder for Candidate Set Construction
In this section we study how alternative choices of image embeddings for selecting top-x nearest neighbors affect the performance of NeaR. We the DINO (Caron et al., 2021) and MAE (He et al., 2021) encoders
11

---

Published in Transactions on Machine Learning Research (08/2025)
FineR
ZS-CLIP-LLaMA
CoOp-LLaMA
NeaR-LLaMA
65.0
Average cACC (%)
62.5
Average sACC (%)
70.0
68.0
57.5
66.0
55.0
64.0
52.5
62.0
60.0
5
8
3
6
9
of Shots
# of Shots
Figure 2: Effect of varying m, number of images per class in training data for labels generated by LLaMA. NeaR (in purple) outperforms CoOp (blue), 7SCI.IP-LIaMA (green) and FineR (red) for all m in average cACC & sACC, where there are more noisy labels. For m-1, our method performs poorly due to excessive label filtering.
Flowers-102
Method (k-nearest)
cACC
SACC
Accuracy (%)
69.8
50.1
75
k: 2
72.3
51.7
CACC
73.5
52.0
SACC
k =
74.7
53.9
k
74.0
53.1
h:
71.3
53.6
65
k:
73.6
53.8
1 2
5
6
7
8
k =
71.5
52.6
k:
70.9
52.0
Figure 3: Efffec of varying x (1 to 9) on the perfor- mance of NeaR-LLaMA for the 9-shot Flowers- 102 dataset. Note that setting CoOp-LLaMA, but is the result of NeaR with V4.
Table 8: To study the generalized idea of effect of the candidate-set size (k) on NeaR, we perform an experiment where m is randomly sampled between and 10 for Flower-102.
not the same
solely for the nearest neighbor candidate set construction step, while retaining our default VIT-B/16 CLIP backbone for prompt tuning. The results for Llama-3.2, presented in Table 9, show that method continues to achieve competitive performance even when the k-NN step is based on features from other backbones.
Image Encoder
Bird-200
Car-196
Dog-
Flower- 102
Pet-37
Average
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
DINO
50.3
70.5
50.6
61.8
60.4
72.6
79.7
60.8
82.4
88.3
64.7
70.8
MAE
48.6
70.1
50.9
61.5
59.9
72.2
76.6
59.8
84.8
88.6
64.2
70.4
CLIP VIT B/16
51.03
70.2
52.6
60.9
59.2
70.2
78.6
61.7
83.4
86.1
65.0
69.8
Table 9: Evaluation of NeaR-LLaMA when different image encoders used to perform nearest neighbor selection during candidate set construction. The default CLIP ViT-B/16 backbone is retained for the rest of the pipeline. We observe that performance remains invariant to the choice of image embeddings chosen to construct the candidate sets.
12

---

Published in Transactions on Machine Learning Research (08/2025)
Method
Bird-200
Car-196
Dog-120
Flower-102 102
Pet-37
Average
cACC
SACO
cACC
sACO
eACC
SACO
cACC
SACC
cACC
8ACC
cACC
sACC
RNS0
CuOp-LLaMA
13.9
44.1
8.6
46.6
18.2
47.9
15.6
31.7
36.7
54.7
18.6
45.0
NoaltIHaMA
50.5
9.8
48.9
49.9
20.5
36.6
63.3
22.7
(+4.1%)
49.8 (+4.8%%
RN101
CoOp-LLaMA
17.3
45.5
9.6
47.4
20.8
45.4
17.5
34.7
41.0
55.2
21.8
45.6
NoRR-LLMMA
18.6
49.6
11.4
50.8
22.4
51.8
18.9
36.1
47.0
63.9
23.6
(+ (+1.8))
50.5
(+4.9%)
VII-B B/16
CoOp-LLaMA
19.2
05.7
45.5
60.7
38.4
68.4
75.9
59.8
714
79.2
00.7
67.1
51.0
70.2
32.6
60.9
39.2
70.2
78.6
61.7
83.5
86.2
65.0
(+ (+433))
69.8 (42.4%)
VIT
VB3-B/32
CuOp-LLaMA
45.0
56.3
39.3
60.5
51.9
66.2
69.8
59.1
72.7
79.8
55.7
66.2
LaaM
65.4
47.8
60.0
56.4
68.9
75.0
61.3
77.3
(+5.44%)
68.1 (+1.9%)
V-T-L
CoOp-LLaMA
55.8
73.4
51.1
63.0
65.1
74.3
81.7
63.3
83.2
87.6
67.3
72.3
NeaR-LaaMA
54.5
73.1
62.5
62.3
67.9
74.6
82.4
64.7
84.9
87.3
70.5
(+3.2%)
72.4 (+0.1%)
VIT-G
CuOp-LLaMA
56.2
74.0
51.2
63.1
64.7
73.8
82.2
63.1
82.4
87.5
67.3
72.3
Nealt LLaMA
50.0
74.9
63.0
61.7
66.2
74.2
87.5
64.9
83.7
85.0
71.9
(+ (.4.6%)
72.1 (-0.2%)
SigLIP
CoOp-LLaMA
36.8
61.1
61.9
62.8
58.5
70.2
71.1
57.2
75.6
80.0
60.8
66.9
NeaR-LLaMA
34.0
63.0
71.5
62.9
37.7
70.2
70.9
56.7
72.9
78.6
61.4
(+0.6%)
66.3 (-0 (9))
Table 10: Performance comparison of NeaR-LLaMA with CoOp-LLaMA across different CLIP backbones, including ResNet-50 (RN50), ResNet-101 (RN101), ViT-B/32, ViT-L, ViT-G and a different pretraining startegy SigLIP. For completeness, results are also provided for the default backbone, ViT-B/16. NeaR. consistently putperooms CoOp-LLaMA, achieving gains of +4.1% and +4.8% in cACC and sACC for RN550, +1.8% and +4.9% in cACC and sACC for RN101, and +5.4% and +1.9% in cACC and SACC for ViT-B/32, +3.2% in cACC and -0.1% in SACC for VIT-L, +4.6% in cACC for ViT-G and +0.6% in cACC for SigLIP. NeaR consistently outperforms the baseline for CLIP models.
4.5 Performance of NeaR on Different CLIP Backbones:
All results in this paper are on the VIT-B/16 CLIP backbone. In this section we compare the performance of NeaR-LLaMA with CoOp-LLaMA across various other CLIP backbones and SigLIP. In Table 10 we present the results of NeaR. for ResNet-50 (He et al., 2015), ResNet-101, and ViT-B/32 vision-encoder based CLIP model. We use the same configuration for each backbone. We observe performance gains of 5.5% in cACC and 3.2% in SACC with VIT-L, and gains of 6.9% in cACC and 2.9% in SACC with ViT-G over the default CLIP VIT-B/16 backbone. Our method outperforms CoOp-LLaMA by 3.2% for ViT-L and by 4.6% for VIT-G, demonstrating its effectiveness across stronger backbone architectures. We also experiment with the SigLIP VIT-B/16 backbone which has a different pretraining objective compared to CLIP. Our results indicate that our method does not transfer effectively to SigL.P. We conjecture that our method fails because our final objective resembles the InfoNCE objective used by CLIP, and is different from the Sigmoidal loss employed by SigLIP. However, we are competitive with the CoOp baseline in avg cACC. These results highlight the effectiveness of our method over diverse range of CLIP architectures.
4.6 Study on Adapter based Finetuning
In this section we follow CLIP-Adapter (Gao et al., 2021) to finetune lincar adapters added on top of both frozen CLIP image and text encoders, as an alternative to prompt-luning. We begin by noting that prompt learning is more efficient in terms of parameter count. In our setup, we use 16 prompt tokens for a total of 16 512 learnable paramcters, whereas a fully-conneeted lincar layer has 512
parameters where K the number of labels/classes (eg. number of GPT-4o generated labels is K=300 for Bird-200). The label filtering step 3.4 in our formulation results in variable label space during test time, and thus we cannot directly
13

---

Published in Transactions on Machine Learning Research (08/2025)
train linear probe with fixed mumber of classes. The table below shows cACC results for 0.8 (from (Gao et al., 2021)) for GPT-4o generated labels. The first row simply shows the naive implementation of the CLIP Adapter (Gao et al., 2021) on the labels. For the next two rows, we observe that NeaR- Saal--Adateer-GGPTT outperforms the adapter baseline (59.98% vs 48.75% average cACC), but still falls short of the performance achieved by prompt-based NeaR-GGP440 (59.98% vs 67.6% average cACC). This suggests that adapter based tuning is unsuitable for the VF-FGVR setting, potentially due its reduced robustness to open-vocabulary label noise. This is consistent with findings from (Wu al., 2023) which shows that prompt tuning is more robust to label noise compared to linear classifiers.
Method
Bird-200
Car-196
Dog-120
Flower-102
Pet-37
Average
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
APPp
39.2
63.9
40.4
56.4
40.0
61.1
57.7
42.9
66.2
75.8
48.7
60.0
SoaltAdapter-OO GP
50.2
73.9
48.0
59.2
54.7
69.7
70.2
49.3
76.6
82.3
50.9
66.9
Noalt-GPT4o (Ours)
55.8
75.6
57.0
60.0
61.6
74.4
80.6
52.1
82.9
84.0
67.6
(+7.7%%
69.2 (+2.3%%
Table 11:
Comparison of prompt-based NeaR-GPTAL (Ours) with adapter based variant NeaR-- Adapter-
GPT4o and standard adapter finetuning.
Success Cases
Success Cases
Liv
Mabe
les Den
ECh Anyonideen
MLL.M MUL.R Ciral Jest
QUICE
Gall Bain
Own
Oer' Anyoddinn
Dae Meystiie
Our Aleriiis
Boor
Fallure Cases
Fallure Cases
OT DWAN
Beggle
emte Motee
Stnds
Ch Fowe
Calledee
Caratien
MILM Dnllonnen Popur
Dore Coleniie
Ocai Cyccnees
Dark
Oary Orieee
Oue Seser Scuee Omn Busien
Oyne denieer Forsher Det Searress
Dine Cperre May
Figure 4: Qualitative results showcasing both success and failure cases of Flower-102 dataset (left) and Pet-37 dataset (right). Success refers to instances where NeaR correctly predicts the class label while the MLLM fails. Failure cases illustrate examples where NeaR produces incorrect predictions.
4.7 Qualitative Results of NeaR.
We visualize selection of inference images and analyze their predictions for the Flower-102 and Pet-37 datasets in Figure 4. Specifically, compare the predictions of NeaR with those obtained from directly querying the MLLM. When comparing sACC, we observe that MLLMs (GPT4/ hetp///mmmii///////e LLLMMA) produce stronger labels (sACC:72.4/73.3 339.9) or predictions than FincR (sACC: 64.3). Therefore, we focus our comparisons on NeaR against MLLMs. In success cases (e.g., Blackberry Lily, Bee Balm), NeaR predicts correct labels consistently, while LLaMA produces inconsistent and unrelated labels like Carnation or Coral Honeysuckle, reflected in lower cACC. failure cases, NeaR often misclassifics flowers as other species with similar structures or colors (c.g., Globe Flower + Buttercup, Windflower - Wood Anemone), unlike LLaMA, which shows greater label spread. For classes that NeaR underperforms, find they harder overall, even
14

---

Published in Transactions on Machine Learning Research (08/2025)
for the MLLM. Even when incorrect, NeaR is more consistent (e.g., Windflower Wood Anemone in most test images), while LLaMA outputs 7 diverse labels for same set of images. We observe a similar trend for pets.
5 Conclusion
We addressed the challenge of Vocabulary-Frre Fine-Grained Visual Recognition (VF-FGVR) by introducing NeaR, a method What leverages MLLMs to generate weakly supervised labels for a small set of training images, to efficiently fine-tune downstream CLIP model. Our approach constructs a candidate label set for an image using generated labels of similar images, and performs label refinement for clean and noisy data differently. NeaR also proposes label fitterigg stralegy, effectively managing the open-ended and noisy nature of MLLM outputs. Experiments 4 MLLMs show that NeaR. significantly outperforms direct inference methods while dramatically reducing computational cost and inference time, setting new benchmark for efficient and scalable VF-FGVR.
Impact Statement
Our framework relies on MLLM-generated labels, dependency that is becoming increasingly feasible with advancements in MLLM accessibility. We demonstrate strong performance across both proprietary (GPT-4o, GeminiPro) and open-source (LLaMA-11B, Qwen2-7B) models, showing robustness to MLLMs of varying capacities. We see this work as a foundation for future research leveraging MLLMs for fine-grained recognition, with no direct societal or ethical risks.
Acknowledgements
Hari Chandana Kuchibhotla, Sai Srinivas Kancheti and Abbavaram Gowtham Reddy would like thank MoE for the generous PMRF fellowship support to each of them; Sai Srinivas Kancheti and Vineeth N Balasubramanian would like to thank Microsoft Research India for the MSR India PhD Award and the AFMR grant, under which this work was carried out. We thank the anonymous reviewers for their valuable feedback that improved the presentation of this paper.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkal, et al. Gpt-4 technical report. ar preprinl arXiv:2303.0877 08774, 2023.
Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics, 2020.
Eric Arazo, Diego Ortego, Paul Albert, Noel O'Connor, and Kevin Meguinness. Unsupervised label noise modeling and loss correction. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 312-321. PMLR, 09-15 Jun 2019.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron C. Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In International Conference on Machine Learning. 2017.
David Berthelot, Nicholas Carlini, lan J. Goodfellow, Nicolas Papernot, Avital Oliver, and Colin Raffel. Mixmatch: A holistic approach to semi-supervised learning. ArXiu, Abs/1995.02249, 2019.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervisod
learning of visual features by contrasting cluster assignments. ArXiv, abs/2006.09882, 2020. URL
attpss//apt: wwwantccccoolr.. sea.ttiicchlar...rgooopopps/l//p
15

---

Published in Transactions on Machine Learning Research (08/2025)
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J'egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 TEFE/CVF International Conference on Computer Vision (ICCV), pp. 9630 9640, 2021. URL https://api semanticscholaar
org/CorpusID: : 233444273.
Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. Understanding and utilizing deep neural networks trained with noisy labels. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1062 1070. PMLR, 09 15 Jun 2019.
Wen-Huang Cheng, Sijie Song, Chieh-Yur Yun Chen, Shintami Chusnul Hidayati, and Jiaying Liu. Fashion meets computer vision: A survey, ACM Computing Surveys, 54(4), 2021. doi: 10.1145/344720
Alessandro Conti, Enrico Fini, Massimiliano Mancini, Paolo Rota, Yiming Wang, and Elisa Ricci. Vocabulary- free image classification. Advances in Neural Information Processing Systems, 36:30662 30680, 2023.
Lei Feng, Jiaqi Lv, Bo Han, Miao Xu, Gang Niu, Xin Geng, Bo An, and Masashi Sugiyama. Provably consistent partial-label learning. In Proceedings of the 34th Infernational Conference on Neurol Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Jiao Qiao. Clip-adapter: Better vision- language models with feature adapters. ArXiv, abs/2110.0455 2021. URL https://apl. semanticscholar. org arg/CorpusID 238583492.
Yuncheng Guo and Xiaodong Gu. Joapr: Cleaning the lens of prompt learning for vision-language models. Proceedings of the IEEE/CVF Conference on Compuler Vision and Pallern Recognition, pp. 28695 28705, 2024.
Bo Han, Quanming Yao, Xingrui Yu, Gang NIu, Miao Xu, Weihua Hu, Ivor Wai-Hung Tsang, and Masashi Sugiyama. Co-teaching; Robust training of deep neural networks with extremely noisy labels. In Neural Information Processing Systems, 2018. URL https : //tpp/appp.aa sea...ssaaaniccoollllll wwtp/orrpsII::5000000000 52065462.
K. Han, Yandong Li, Sagar Vaze, Jie Li, and Xuhui Jia. What's in a name? beyond class indices for image recognition. ArXiv, abs/ /2////330.0222444 2023.
Hulingxiao He, Geng Li, Zijun Geng, Jinglin Xu, and Yuxin Peng. Analyzing and boosting the power of fine-grained visual recognition for modti-lodal large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreviiet http://pwwerreveeeeeee/////eetap wwt://oouuu/teeeeeaaknno....
Kaiming Hc, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2015. URL httppss/appp.ssmmniiicccaaaa
wwwantccscholar wtwp://sccccoaao.orrrrrrarro www/CorpgsroD......oo 206594692..
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll'an ar, and Ross B. Girshick. Masked autoencoders are scalable vision learners. 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition
(CVPR), pp. 15979-15988, 2021. URL https: http://ppp..mmmancccchooolllr www.oolaarrrgg/ooooooooo Brrg/CorpuIID.... :.D::22339980
Ahmet Iscen, Jack Valmadre, Anurag Arnab, and Cordelia Schmid. Learning with neighbor consistency for noisy labels. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4662-4671, 2022.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser Nam Lim. Visual prompt tuning. ArXio, abs/2203.121 2022.
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fino-grained image categorization: Stanford dogs. In Proc. CVPR workshop on fine-grained, visual categorization (FGVC), volume 2, 2011.
16

---

Published in Transactions on Machine Learning Research (08/2025)
Jonathan Krause, Michael Stark, Jin Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on compuler vision workshops, pp- 554 501,-61,111 2013.
Jichang Li, Guanbin Li, Feng Liu, and Yizhou Yu. Neighborhood collective estimation for noisy label identification and correction. ArXiv, abs/2208.03307,, 2022.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi-supervised learning. In International Conference on Learning Representations, 2020.
Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-inage pre-training
with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. 2024 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 26286 26296, 2023.
Mingxuan Liu, Subhankar Roy, Wenjing Li, Zhun Zhong, Nicu Scbe, and Elisa Ricci. Democratizing fine-grained visual recognition with large language models. In The Twelfth International Conference on Learning Representations, 2024a.
Yu Liu, Yaqi Cai, Qi Jia, Binglin Qiu, Weimin Wang, and Nan Pu. Novel class discovery for ultra-fine-grainee visual categorization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17679-17688, 2024b.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Wei Li, Pan Zhang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang. Rar: Retrieving and ranking augmented milms for visual recognition. arXiv preprint arXiv:240.. 10.000.118..... 18805, 2024c.
Alex Long, Wei Yin, Thalaiyasingam Ajanthan, Vu Anh Nguyen, Pulak Purkait, Ravi Garg, Alan Blair, Chunhua Shen, and Anton van den Hengel. Retrieval augmented classification for long-tail visual recognition. 2022 IEFF/CVF Conference on Computer Vision and Pattern Recognátion, (CVPR), pp. 6949 (099-695,,40 2022. URL https: //ppi. wwwanticschooeeeee.... crt.aaauuio//rC 2470058344.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. ArXiv, abs/1306.5151, 2013.
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision, graphics & image processing, pp. 722 729. IEEE, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference computer vision and pattern recognition, pp. 3498 3505. IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Tlya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Hanoona Abdul Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman H. Khan, and Fahad Shahbaz Khan. Finc-tuned clip models are efficient vidco learners. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2022.
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, dean-baptist Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05538) 2024.
Nils Reimers and Iryna Gurevych. Centenco-bert: Sentence embeddings using siamese bert-networks. arXiv preprint Xiv:19908. 10084, 2019.
17

---

Published in Transactions on Machine Learning Research (08/2025)
Muhammad Ridzuan, Ameera Bawazir, Ivo Gollini Navarrete, Ibrahim Almakky, and Mohammad Yaqub. Self-supervision and multi-task learning: Challenges in fine-grained covid-19 multi-class classification from chest x-rays. In Annual Conference on Medical Image Understanding and Analysis, pp. 234 250. Springer, 2022.
Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fino-grained robustncss in coarso-grained classification problems. In Advances in Neural Information Processing Systems, volume 33, pp. 19339 19352, 2020.
Ximeng Sun, Ping Hu, and Kate Saenko. Dualcoop: Fast adaptation to multi-label recognition with limited annotations. ArXiv, abs/2206.09544 09541, 2022. URL https://apts www.nti /ttp semaatttscchllaarooo//oooooou Org/CorpusID: 249890410.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXio, abs/230 1397/332.1397,, 2023.
Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In IEEE Conference on Computer Vision and Pattern Recognátion, 2022.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech uesd bird dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing fision-langual model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
Xiu-Shen Wei, Yi-Zhe Song, Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang, and Serge Belongie. Fino-grained image analysis with deep learning: A survey. IEEE transactions on pottern analysis and machine intelligence, 44:8927 8948, 2021.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers: State-of- the-art natural language processing. CoRR, abs/ 0bs/1910.03771, 2019.
Cheng-En En Wu, Yu Tian, Haichao Yu, Heng Wang, Pedro Morgado, Yu Hen Hu, and Linjie Yang. Why is prompt tuning for vision-language models robust noisy labels? 2023 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 15442- 15442-15451, 2023. URL httt::://api..smaaiiicest 260125914.
org org/CorpusII:
Xin Xing, Zhexiao Xiong, Abby Stylianou, S. Sastry, Liyu Gong, and Nathan Jacobs. Vision-language pseudo-labels for single-positive multi-label learning. 2024 IEEE/CH /CF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 7799-7808, 2023. URL bttps://apl. semanticscholar. org/Corpus : 264439112.
Guofeng Yang, Yong He, Yong Yang, and Beibei Xu. Fino-grained image classification for crop discase basod on attention mechanism. Frontiers in Plant Science, 11:600854, 2020.
Jing Yang, Jian Duan, Tianxiang Li, Cheng Hu, Jianqiang Liang, and Tielin Shi. Tool wear monitoring in milling based on fine graiied image classification of machined surface images. Sensors, 22(21), 2022. doi: 10.3390/s22218111)
Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Levng-Levy. Why are visually-grounded language models bad at image classification? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:/ httts:///penreview..n net/ forum? Simmm?ddd-mmmBVVW
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lasion-language models. International Journal of Computer Vision, 130:2337 2348, 2021.
18

---

Published in Transactions on Machine Learning Research (08/2025)
Appendix
This supplementary material contains additional details:
Summary of notations and their descriptions in $ A6
The overall NeaR algorithm is presented in $ A7
Further Comments on Efficiency of NeaR in A8.1.
Further descriptions of datasets used in $ A8.2.
Effect of label filtering on the size of label space in $ A8.3
Analysis on an alternative way to obtain candidate sets is shown in $ A8.4
Performance across different prompting strategies in shown in A8.5
The effect of choice of threshold is studied in $ A8.6
Comparison with clustering-based methods in $ A8.7
Performance across variations different few-shot splits is shown A8.8
Comparison with other Pscudo-labeling based Methods in $ A8.9
Analysis of retrieved images in $ A8.10
Visualizing noisy samples from GMM in $ A8.11
Discussion on other weaker open-source MLLMs is presented in $ A8.12
Prompts used to generate labels for different MLLMs in $ A8.13
A6 Summary of Notations.
We represent elements of a set by a subscript and vector component by a superscript. For instance V {o, 1 denotes the one-hot vector encoding of the class label of the i-th image, i [n], and of is the j-th component this encoding. Specifically y! = if the i-th image is assigned the j-th label in the label set and 0 otherwise. summary of notations given Table A12.
A7 NeaR Algorithm
We present the training algorithm of NeaR in Algorithm 1. We present the three steps of our method NeaR shown in Figure 1. In lines L1-L7 we generate possibly noisy labels from an MLLM. In lines L8-L12, we then generate a candidate set from к nearest-neighhors of each image. In lines L19-L27 we train prompt vectors 0 by minimizing the cross-entropy loss between CLIP predictions and the refined labels. The label refinement in T.24 effectively disambiguates the best label from the generated candidates. As training progresses, our
estimate of the candidate labels gi gets better. The sharpening function is shrp(y, T)' (v²)+ + Σ (1r) and
the rescale function defined rsc(y, 9) (y 1 /
9)

---

Published in Transactions on Machine Learning Research (08/2025)
Notation
Description
i-th unlabeled training image
X
set of training images (21.22,..... ]n
Number of shots of images for each class belonging to unknown ground- truth class name set.
4 M(..p)
The class label generated for my by MLLM I. with input prompt
The dataset generated by MLLM L consisting of image-class name pains Label space generated by MLLM The & lexicographically ordered class names C = (ci, (2,,..,,..,
U
h C
M c (o, 1)*
Oue-hot oncoding of the label 4 of æ; Image encoder of pre-trained CLIP Text encoder of pre-trained CLIP Learnable prompt vectors added to the input embeddings of class name k-dimensional probability vector of the prompted CLIP model's class predic-
T
fa(w) E A
tions for image z, ie fo(x)s and Σ fo(x)==1
S:
Candidate set created by gathering MLLM generated labels of nearest- neighbors of æ; Number of nearest-neighbors
D. (1,,,,, SOYE
Augmented dataset containing the generated label I and constructed candi- date set St The cross-entropy loss of CLIP model predictions for image i w.r.t y given by vo (a.)) (also written as L(x;, two-component Gaussian Mixture Model fit loss values of all training samples for every epoch
GMM
w; PGssa(cleanp
The posterior probability of image belonging to the "clean" component, Le component with lower mean
7 E 10
Threshold used to partition data clean and noisy sets Clean and noisy partitions of the training data based on Was Confidence of the candidates. We have that of > if S, of = 0
X. Xns
E R
otherwise, and Σ -d=1 1=1
Refined label for image constructed based whether x; is clean or noisy.
For noisy images we rescale the label have non-zero probabilities only for
the candidate labels
shrp(y, T) and rsc(y. 9)
sharpen distribution y using temperature rescale distribution y based on candidate confidence
Lpmat (0) Crest
The cross entropy loss between model predictions fo(z) and refined labels g Inference label space post. filtering
Table A12: List of notations used in our paper, and their descriptions.
2

---

Published in Transactions on Machine Learning Research (08/2025)
Algorithm NeaR algorithm: Training
Require:
m-shot training images X = {ad' MLLLM I; input prompt number of nearest neighhors CLIP model predictions fa with learnable text prompts O vorn camm_ppochs; warm__eoccks learning-rates y, fwarmi Temperature T
Ensure: Trained parameters
Step-1: Labeling training images with an MILM
1: D -
Label each image by prompting MLLM L with prompt p
2: for == 1,2,..., do
3:
D DU (( L(z,,)))
4: end
5: --==14
6: k = [c|
1* WLOG we consider C = fci, .2....,,... C where & = [C] to be lezicogrophically ordered. Let be the one-hot encoding of
label I; */-
7: y 10, and of e, = o/w
Step-2: Candidate Set Construction
8: D. -.+0
1 Augment each image 2; with candidate set S composed of labels of k-nearest neighbors */
9: for (.),,4) in D do
10: S. knn labels(z,,k)
11: D. I.DDL. (1)4,4,8;))
12: end for
1" We initialize candidate confidence, for all images uniformly' *
4 R* and of = [S.1 oy E S, and o/w fo. T):
Function Partition_datal
14:
(:= (L(6(
15:
Nc. Nn, fft GMM(())
16:
W (wi, w2...., wn) where un (ewra((deaa(s)))
Xa (i, C
7]
Nno XXd
return Nna,
Step-3: Fine-tume prompts
CLIP model
for 1,2,..... n........mmmmm.... nam__epochs
During warmap, the prompts are tuned on the cross-entropy loss Lee using MLLM generated labels D #/
if warm epochs then
6, - 9 - qwarm (D.
22:
else
1t Every epoch post warm-up, we partition the entire data into clean and noisy sets by fitting a GMM on cross-entropy loss
L +
23:
Xna, W --aartitionddatal data(D,, fo, 1:
24:
= shrp (w: .1 rsc(shrp(uu; qi + (1 w.) fo._, ())).)).:)) o/w
wi) Sec- (4,), T) ia E X
1t We update candidate confidence (for both clean and noisy samples) used nent epoch, 1[g]] is non-zero indices and a/w
25:
Ts < rsc(for- (qq))
26:
(finat(0e-1) = -1 Σ Σ - (ax))
==1==1
6, - 91-1 - 1I -
end
29: end for
30: Return: Draumapoche
3

---

Published in Transactions on Machine Learning Research (08/2025)
A8 Additional Results
In this scction start with further comments Efficiency of NeaR. in $ A8.1. Descriptions of the datascts in A8.2 used followed by additional results. i) Effect of label filtering on the size of label space in A8.3. ii) Analysis on alternative way to obtain candidate sets is shown in $ A8.4. iii) Performance across different prompting strategies in shown in $ A8.5. iv) The effect of choice of threshold T is studied in $ A8.6. v) Comparison with clustering-basee methods in $ A8.7. vi) Performance across variations in different few-shot splits is shown in $ A8.8. vii) Comparison with other Pseudo-labeling based Methods of retrieved images in $ 10. ix) Visualizing noisy samples from GMM in A8.11. x) Discussion on other weaker open-source MLLMs is presented in & A8.12. xi) Prompis used to generate labels for different. MLLMs in A8.13.
A8.9. viii) Analysis
A8.1 Further Comments on Efficiency of NeaR
Table comprehensively covers the different MLLM types and all baselines we designed. For instance, we report inference time, cost, and cACC for GPT-4o, which closely reflects performance for Gemini Pro as well. Similarly, results for LLaMA are representative of Qwen. For 7.--CT.P-M..MM, CoOp-MT.LM, and NcaR-MLLM, inference time and cost remain the same, with only a slight overhcad training time. We also note that the cost estimates presented were for GPT-4o in late 2024, and current. API costs may be cheaper as inference becomes more scalable. We acknowledge that our cost estimate $100 for 32k images) could be improved by leveraging batched inference APIs, which can indeed reduce the cost by up to 5x and ensure timely completion. Our original estimate was based on standard per-image API usage (e.g. GPT-4 or Gemini Pro), which many users adopt by default due to ease of integration. However, we agree that using optimized batch pipelines is a more efficient option, potentially reducing the cost to approximately $20 for 32k images. Even by using optimizod batched pipeline, our method always requires lesser costs.
A8.2 Description of Datasets Used.
We show results on datasets with fine-grained labels Bird-200, Car- bog-120, Flower-102, Pet-37. In Table A13, we show the number of images used for training and the size of the test set.
Bird-200
Car-196
Dog-120
Flower-102
Pet-37
Train Set
200
196
120
102
37
Test Set
5794
8041
8550)
6149
3669
Table A13: Train and test set sizes of the datasets used in this paper. The number of shots is denoted by m, with m = 3 used as the default in our experiments unless otherwise specified.
A8.3 Effect of Label Filtering the Size of Label Space.
As described in $ 4.2, label filtering is crucial to obtain good VF-FGVR, performance. In Table A A14, we present the number of classes in the final classification label spaces that each method operates in. The first row indicates the size of the ground-truth label space. We observe that our label filtering mechanism is essential to combat the open-endedness of MLLM labels.
Method
Average
Bid-2000
Car-196
Dog-120
Flower-102
Pet-37
Ground Truths
200
196
120
102
37
MLLM Labels
412
562
169
183
63
Finelt
202
286
97
112
NeaR LLaMA
239
305
129
Table A14: Label filtering is effective in reducing the size of MLLM generated label to manageable levels.
4

---

Published in Transactions on Machine Learning Research (08/2025)
A8.4 Analysis on Alternative Ways to Construct Candidate Sets
The labels generated by MLLMs can be noisy. To address this, we propose to construct candidate set. for each image by grouping class labels from the k nearest-neighborr of the image. In this section study an alternative approach to candidate set generation, where we query the MLLM itself to generate set of k. labels directly for each image. We present the results of this approach in Table A15, showing performance across three MLLMs: GPT-4o, GeminiPro, and LLaMA. The purpose of this experiment is to demonstrate the effectiveness of candidate set construction via KNN. For the Car-196 dataset, the generated candidate labels were highly
in union of approximately 1200 unique labels across all images which leads to OOM issues during text feature computation. As a workaround for the Car-196 dataset alone, we limited the candidate sets to the top-2 labels per image. Furthermore, for LLaMA, our K-nn based approach oulperforms the direct approach by substantial margin, achieving a +5.2% higher cACC, while being more efficient.
Method
Bird-200
Car-196
Dog-120
Flower-
Pct-37
Average
eACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
eACC
sACO
cACC
sACC
53.1
74.2
46.0
54.1
61.0
71.9
77.0
55.4
83.1
82.7
64.0
67.6
NeaR-CPT-40
55.8
75.6
57.0
60.0
61.6
74.4
80.6
52.1
82.9
84.0
67.6 (+3.6%)
69.2 (+1.6%
Baalt-GminniPo-DDD Direct
52.7
73.8
43.8
52.2
70.5
58.3
73.9
46.9
81.3
81.7
64.4
62.5
Neai-GeminiPrro
55.9
76.0
54.9
61.1
64.7
75.4
77.9
53.2
70.4
80.8
66.6
(+2.2%)
69.3 (4.688
NoaRt-LaMA-l MaMMA-Droot
43.8
68.2
44.7
56.0
56.4
70.6
70.9
57.7
83.2
87.5
59.8
68.0
NeaR-LLaMA
51.0
70.2
52.6
60.9
59.2
70.2
78.6
61.7
83.5
86.2
65.0
(+5.2%
69.8 (+ (+1%)
Table A15: Evaluation of NeaR-MLLM under different candidate set generation methods. We compare our bann- candidate set against directly querying the MILM for a candidate sct, referred to as NeaR- MLLM-Direct. For the Car-196 dataset, both GPT-4o and GeminiPro encounter Out-of-Memory (OOM) errors due to the larger label space. For NeaR-LLaMA, our k-nn-based approach outperforms the direct approach by an average margin of +5.2% in cACC while being more computationally efficient.
A8.5 Different Prompting Strategies.
We analyze the impact of our proposed approach using various prompting strategies, as presented in Table A16. We consider three distinct prompting methods involving fine-tuning across different modalities: (1) For text-only prompting, we use CoOp (Zhou et al., 2021); (2) For image-only promptingg we employ VPT (Jia et al., 2022); and (3) For both text and image prompting, we adopt hierarchical prompts introduced at different text and image layers (Rashoed et al., 2022) as the backbone. Our method demonstrates strong performance across all prompting strategies, achieving cACC improvements of 4.2%, 1.9%, and 4. 4.1%, respectively. This clearly demonstrates the effectiveness of our method across different fine-tuning methods.
Method
Bird-200
Car- 196
Dog- 120
Flower-
Pet-37
Average
eACC
sACC
cACC
sACC
cACC
SACC
cACO
SACC
<ACC
sACC
cACC
sACC
Text Prompling
CoOp-LLaMA
49.2
68.7
45.5
60.7
58.4
75.9
39.8
74.1
60.7
67.4
NeaRt-LLMA
51.0
70.2
52.6
60.9
59.2
70.2
78.6
61.7
83.5
86.2
65.0
(+ (+4.3%)
69.8 (+2.4%)
Visual Prompting
VPT-LLaMA
48.9
69.5
45.3
61.3
60.3
70.2
78.1
61.7
73.2
82.2
61.1
69.0
Nealt- VPT-LIMMA
50.2
68.5
45.5
60.1
58.4
71.4
78.3
61.3
81.1
84.3
62.9
(+ 1.8%)
69.1
(+0.11%)
Multimodal Promoptng
48.9
69.5
45.3
61.3
60.3
70.2
78.1
61.7
73.2
82.2
61.1
69.0
I LLAMA
52.5
61.D
58.6
62.1
83.7
86.4
65.2
(+4.1%)
69.9 (+0.9%)
Table A16: Evaluation of NeaR under different prompting strategies. In addition to based prompting, as shown in Table 3, we present results on Visual Prompting method VPT (Jia et al., 2022) and Multimodal Prompting method IVLP (Rasheed al., 2022). We outperform the baselines by +1.8% and +4.1% in cACC respectively.
5

---

Published in Transactions on Machine Learning Research (08/2025)
A8.6 Effect of Choice of Threshold
To address the noisy nature of MLLM generated labels, our method NeaR separates samples into clean and noisy sets using a threshold based on clean posterior probability w; of GMM fitted on loss values. Instead of using a fixed threshold, we make adaptive by setting it to the mean posterior probability, 7= = Σ W
==1
allowing dynamic estimation of label noise at. every training epoch. We study the effects of using a fixed threshold of т 0.5 for NeaR-LLaMA, NcaR-GPT-44, NeaR-GeminiPro and YeaR-Qwent in Table A17. We observe that for NeaR--GmmniPrro and NeaR-Qwen2,, we have a performance gain of +1.3% and +1.1% in average cACC, while a relatively lower performance gain of +0.2% in NeaR-LLaMa, and NeaR-GPT-44. These results show that our adaptive thresholding performs better than static threshold across a variety of MLLM choices, thus eliminating the need for luning the hyperparameter 7.
Method
Bird-200
Car-196
Dog-120
Flower-102
Pet-37
Average
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
cACC
sACC
NeaR-LLaMA (r=0.5)
51.7
70.5
53.3
60.5
59.0
68.6
78.1
61.7
82.2
85.9
64.8
69.5
NeaR-LLaMA
51.0
70.2
52.6
00.9
59.2
70.2
78.6
61.7
83.5
86.2
65.0 (+0.2%)
69.8 (+0.3%%
NeaR-GGPT44 (7=0.5)
54.7
74.5
57.9
59.7
62.1
74.6
79.6
52.1
83.0
83.8
67.4
NeaR-GPT-to
53.8
75.6
57.0
00.0
61.6
71.1
80.6
32.1
82.9
84.0
67.6
(+0.2%)
69.2 (+0.3%%
MeaR-GGeiiiPro (r=0.5)
52.8
73.5
53.7
61.1
61.8
75.2
77.6
53.3
77.7
81.0
65.3
68.8
Neall GeminiPo
55.9
76.0
54.9
61.1
64.7
75.4
77.9
53.2
79.4
80.8
66.6
(+1.3%
69.3 (+0.5%)
NeaR- Qweu2 (r=0.5)
34.3
65.0
57.0
64.0
55.2
71.3
74.0
61.4
74.1
76.4
58.9
67.6
Neall Qwen2
35.5
65.6
58.0
64.0
56.6
71.7
75.8
62.5
73.8
76.4
60.0
(+1.1%)
68.0 (+0.4%)
Table A17: Evaluation of our dynamic threshold T across different MLLMs compared to a static threshold T = 0.5. The use of dynamic threshold shows consistent improvements across all MLLMs, with gains in Nemii-Gem and NeaR-Qwen2, achieving increases of 1.6% and 1.1% in average cACC, respectively, and a minor gain of 0.2% in other cases. These results support the design choice to avoid the hyperparameter т, which can vary slightly across MLLMSs.
A8.7 Comparison with clustering based methods.
We compare against three clustering baselines discussed in FineR (Liu et al., 2024a): (1) K-Means (Ahmed et al., 2020) clustering CLIP features, (ii) Sinkhorn-based parametric clustering (Caron et al., 2020) using CLIP and DINO features, and (iii) SCD (Han et al., 2023), which performs non-parametric clustering followed by CLIP-based narrowing of a large vocabulary consisting of 119k WordNet nouns and 11k bird names from Wikipedia. Results are reported in Table A18. We observe that NeaR outperforms classical clustering methods by large margin using GPT-4o labels. Despite having the knowledge of the number of classes, the resulting cACC of K-Means algorithm is only 36.7%, showing that knowledge of class count alone does not yield high performance. The fact that alternative clustering variants built on different feature extractors achieve only marginal improvements further confirms that our performance gains come from the method design itself not from any prior knowledge of the class count.
Method
Avg. (cACC)
K-Means
36.7
CL.IP-Sinhhon
21.6
DINO-Sinkhorn
19.1
SCD
52.2
FineR
57.0
NeaR
67.6
Table A18: Comparison of average clustering accuracy (cACC) across methods.
6

---

Published in Transactions on Machine Learning Research (08/2025)
85
LSCLIP-LLAMA
2S-LIP-LLMA
MaRR-LLaMA
NeaR-LLaKA
75
70
A ACCC (%)
70
CACC
55
60
50
55
Bird-200
Car-198
Dog-120
Flower-102
Pet-37
Bird-200
Car-198
Dog-120
Flower-102
Pet-37
We report cACC and sACC under the effect of random sampling of training images across five datasets. The plot demonstrates minimal variance across datasets, highlighting the robustness of NeaR variations in data selection.
Figure A5:
A8.8 Performance across variations in different few-shot splits
We have conducted extensive experiments across three random seeds to evaluate the robustness and consistency of our approach as shown in Figure A5. Specifically, for each seed, we sampled unique set of m images per class, ensuring diversity in the training data distribution across different runs. The results consistently demonstrate that NeaR outperforms the bascline approaches across various independent samplings, with minimum variance across datasels, indicating its stability and generalizability. This robustness across different random seeds highlights the effectiveness our approach in handling variations in training data selection, further strengthening its practical applicability in real-world scenarios.
A8.9 Comparison with other Pseudo-labeling based Methods
this section we compare NeaR against two ggeudo-labeling based methods DualCoOp(Sunn et al., 2022) & VXin al., 2023). DualCoOp operates in a multi-label setting, where cach image can belong multiple classes, and the goal is to maximize activation across all true labels. On the other hand, in NeaR we operate in problem setting where each image comes with noisy label and we tend to find the best possible semantic label using the proposed pipeline. VLPL operates in a single-positive multi-label setting, where only one ground-trute label is provided per image, but the model is expected to predict multiple relevant labels. It leverages VLMs to generate pseudo-labels that enrich the supervision signal during training. In contrast, NeaR operates in a noisy single-label setting, where cach image is associated with a possibility of semantically incorrect label from an MLLM. Rather than expanding to multiple labels, NeaR focuses on identifying the most semantically accurate label. We implemented DualCoOp and VLPL for our problem setting. The results are shown in Table A19 for LLaMA generated labels. As anticipated, given their fundamentally different. objectives, both DualCoOp and VLPL underperform compared to NeaR.
Method
Bird-200
Car-196
Dog-120
Flower- 102
Pet-37
Average
cACC
sACC
cACC
sACC
cACC
sACC
eACC
sACC
cACC
sACC
cACC
sACC
DualCoOp
35.6
31.7
20.8
45.2
41.9
45.5
6,3
27.8
18.5
48.6
24.6
39.7
VLPL
35.2
36.2
23.4
49.8
37.6
38.4
5.2
34.1
30.2
46.1
26.3
40.9
NeaR-
51.0
70.2
32.6
60.9
59.2
70.2
78.6
61.7
83.4
86.1
65.0
69.8
Table A19: Comparison against pseudo-labeling based methods for labels generated by LLaMA3.2.
A8.10 Analysis of retrieved images.
We perform qualitative analysis on the top three retricved images (w 3) for the Flower- 102 and Bird-200 datasets in Figure A6 and A7. In the results, each row shows reference image the left, followed by its
7

---

Published in Transactions on Machine Learning Research (08/2025)
two nearest neighbors obtained using CLIP ViT-B/16 features. We show cases with both successful and unsuccessful neighbors. Successful neighbors are the ones which help in forming desired candidate set by including the ground-truth label. Unsuccessful candidate sets are the ones with no ground-truths.
Successful Nearest-Neighbors
Unsuccessful Nearest-Neighbors
Snapdragon
Snapdragon
Foxglove
Pink Primrose
Silverbush
Silverbush
Orange Dahlia
Orange Dahlla
English Marigold
Pincushion Flower
Bee Balm
Black-eyed Bladk-eyd Black-eyed Susan
Siam Tullip
Stemless Gentian
Moon Orchid
Figure A6: Qualitative results showcasing successful and unsuccessful neighbors in Flowcr-102 dataset.
Successful Neaihst-Neighhors
Unsuccessful Nearest-
Golden-etngee Warbler
Common Yellowthrost
Golden-efnged Wartber
Rad-eyyd
Tenneosee Warbler
Philadelphia Vireo
Purple Finch
Purple
Pine
Pomarine Janger
Laysan Albatross
Pled Kingfisher
Pled Kinglisher
Cockaded Woodseokke
Bewick Wren
Great Gray Shrike
Figure A7: Qualitative results showcasing successful and unsuccessful neighbors in Bird-200 dataset.
A8.11 Visualizing noisy samples from GMM
The qualitative results for visualizing noisy samples are shown in Figure A8. The figure displays several training examples the GMM marked noisy immediately after warm-up. For each such image, provide the ground-truth label, the model prediction (after the warm-up step), and the final post-training prediction. It can be clcarly seen that the predictions post training have become much more semantically inclined and better than the initial predictions. We however cannot directly draw any comparison between the train set predictions and the test set predictions due to the label filtering step post training (the label space changes during inference). To make fair comparison, we also study the scenario where we compare the test sample predictions of our model with the no-GMM ablation in Figure A9, We observe that the predictions of no-GMM ablation more noisy.

---

Published in Transactions on Machine Learning Research (08/2025)
Flower-102 Dataset
GT: Morning Glory
Post Warmup: Field Bindweed
Post Training:
Momnig Giory
QT: Palergonium
Post Warmup: Eustoma
Post Training: Wild Rose
OT: Bnapdragon
Post Warmup: Adenium Obesum
Post Training: Snapdraagon
GT: Bolero Deep Blue
Post Wormup: Babiana
Post Training: Gentiona
Figure A8: Qualitative results of training samples classified as noisy by the G.M. just after warmup and after full training.
Flower-102 Dataset
GT: Barbeton
W// GMM: Gerbara
Wio GMM: Tithonia
GT: Cyctamen
W/ GMM: Babiana
GMM:: Wood Anemone
foot
Canna Lily
Lily
Wile
Figure A9: Qualitative comparison of test image predictions between ours (with GMM) and no GMM ablation (w/o GMM).

---

Published in Transactions on Machine Learning Research (08/2025)
A8.12 Results Other Open-Source MLLMs
In order to study the impact of NeaR other open-source MLLMs, query two weaker open-source MLLMs, LLa (Liu et al., 2023)and BLIP2 (Li et al., 2023), to generate labels for our datasets. In the context addressing the VF-FGVR problem, we observe that these models produce generic labols that lack fine-grained detail. For instance, in the Bird-200 dataset, images from various fine-grained classes such as American Goldfinch, Tropical Kingbird, Blue-headed Vireo, Yellow-throated Vireo, Blue-winged Warbler, Canada Warbler, Cape-May Warbler, and Palm Warbler were all labeled simply 'Bird' by LLaVA. This lack of specificity results in low cACC of 9.8% for CoOp-LLaVA and 4.7% for NeaR-LLaVA. This trend is also observed with BLIP2. The inability of these ML.LMs to generate diverse fine-grained labels makes them poor choice to solve the VF-FGVR task.
A8.13 Prompts used to generate labels from MLLMs.
In Table A20, describe the prompts used to obtain the labels for both proprietary and open-source MLLMs. We give different prompts for different datasets. As part of future work, we would like to explore if different prompting strategies can give better labels.
MLLM
Prompt Structure
"You are a multimodal AI trained to provide the best fine-grained class label for a given <dataset> image. Provide the best fine-grained class label for the given <dataset> image. Do not return anything else.", <img>
GPT-4o, GeminiPro
"Give me a fine-grained label for this <dataset>. For example, <samplelabel>, Just print the label and nothing else.", <img>
LLaMA, Qwen
Table A20: A summary of prompts used for querying MILM models used in this paper. In these prompts, dataset E (bird, car, dog, flower, pet). img refers to the image being queried for fine-grained class label. Samplelabel for bird is Black Throated Sunbird, samplelabel for car is 2012 BMW M3 coupe, etc. We observe that open-source models like LLaMA require extra supervision in terms of sample labels for better performance.
10


Reviews:
[{"review_id": "cvMXGu9W4S", "reviewer": "Reviewer_6edX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The work is well-motivated and tackles a novel problem that is more practical than traditional zero-shot fine-grained visual recognition.\n- The presentation is curated, and the content is clearly explained.\n- The proposed approach consistently improves over the previous state-of-the-art.\n\n**Weaknesses**\n\n- The assumption of having exactly k samples per class (even without labels) is quite strong, as it implicitly requires knowledge of the class distribution during data selection, effectively using label information before applying the proposed method. Although the authors attempt to address this by considering a more \"realistic\" imbalanced setting, this scenario is still artificial: it assumes access to at least k samples from every class. This allows the total number of classes to be inferred simply by dividing the dataset size by k, which undermines the realism of the setup. Knowing or approximating the number of classes in advance enables the model to produce a similar number of output clusters, which can artificially boost cluster accuracy by ensuring a one-to-one mapping with ground-truth labels rather than a more challenging many-to-one or one-to-many situation. This may also be confirmed in Tab. 3, as the main gains are on cluster accuracy.\n- Instead of the imbalanced setting, I believe it would be more interesting to show situations where the images are randomly sampled from the training set, without any assumption on the presence of all the classes in the selection process.\n- There is a mismatch in the semantic similarity reported in Tab. 2 (i.e., the one demonstrating that k-nn filtering improves candidate selection) and the main results in Tab. 3. I understand that the difference is due to all the other components introduced to fine-tune the model after the initial selection procedure. However, this may highlight that the training is not optimized, as the semantic similarity shows an average reduction of approximately 8 points. Moreover, the average semantic similarity reported in Tab. 2 does not match any of the averages for the MLLMs in Tab. 3.\n- The approach used to construct refined labels feels very overfitted to the data, as it uses a lot of transformations (also different between the clean and the noisy subsets). Moreover, I believe there are no ablations on this step of the pipeline.\n- When increasing the number of shots, cluster accuracy decreases and semantic accuracy increases. Could the authors propose a hypothesis for this behaviour?\n- The selected backbones in one ablation study (RN50, RN101, ViT-B/16, and ViT-B/32) form two closely related pairs, limiting the diversity of the analysis. It may be more informative to explore how performance varies when scaling within a single architecture family (e.g., ViT-B to ViT-L or ViT-g) or when using models with different pretraining strategies (e.g., SigLIP).\n- Unclear how using the commercial models in Tab. 11 resulted in an out-of-memory issue. If the problem is the generation of too many candidates, then the authors could convert the problem into an image-to-text retrieval problem (similar to what CaSED did) and retrieve for each image only the top-k most similar candidates.\n\n\n**Minor weaknesses**\n\n- The authors state they are the first to explore using MLLMs for the task, but that may be incorrect, as FineR uses BLIP for generation, and CaSED has baselines with BLIP and ablations where they test generating the candidate list instead of doing retrieval.\n- The authors should probably add spaces for some of the paragraphs (e.g., in the Related section, where there are some bold words to introduce a paragraph but no newline).\n- I suspect the estimate of the costs for commercial models is not optimized, as the authors report ~100$ for 30k images. However, for these situations it is best to perform batched inference via the API, which costs ~5 times less and is guaranteed to complete in at most 24 hours.\n- Tab. 2 does not explain which MLLM was used as a baseline.\n- The default value of nearest-neighbours and the number of shots is reported twice in the implementation details. Also, the decision for k=3 is very arbitrary, as it is more common to have 1, 2, 4, 8, 16 shots in few-shot settings.\n- The reported relative improvements are difficult to interpret, as the tables use inconsistent reference baselines. In some cases, the reference is not the second-best performing method, which can exaggerate the perceived effectiveness of the proposed approach.\n- Tab. 5 has two columns (warmup, GMM) which are never ablated. I advise the authors to either remove them or (better) report the ablation also without those components.\n- About the imbalanced training setting, I am unsure if the scenario can be called \"long-tailed\" due to the range of occurrences of classes, i..e, 3-10.\n- Unclear why the qualitative are against an MLLM and not against baselines (e.g., FineR).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors tackle the problem of fine-grained visual recognition in the vocabulary-free setting, i.e., where the target label list is unknown to the models. To solve the problem, they consider having access to a small balanced set of unlabelled images of the dataset and using a multimodal large language model (MLLM) to pseudo-label them. They introduce Nearest-Neighbor Label Refinement (NeaR) to fine-tune a model for the task in the presence of noisy labels, i.e., those generated by the MLLM, considering refinement and filtering steps to mitigate the noise in the pseudo-labels. Finally, they fine-tune the model with prompt tuning, improving the performance on five fine-grained datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should focus on addressing the weaknesses I reported above, with most of the focus on the following points:\n\n- Discuss the possibility of their model \"gaming\" the cluster accuracy.\n- Demonstrate that their approach works without the reliance on the k-shots balanced on the dataset classes, by showing the performance with, e.g., random sampling from the training set.\n- Explain the mismatch between Tab. 2 and Tab. 3 and the ambiguities in the first.\n- Report an ablation study on the construction of refined labels, to demonstrate how it contributes to the overall performance, and confirm whether the selected transformations are not overfitted to the data.\n- Propose a hypothesis on why having more shots increases semantic accuracy and decreases cluster accuracy for most methods.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "O26WFB7Tn9", "reviewer": "Reviewer_xzFU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The proposed method achieves vocabulary-free classification with performance comparable to known label setting for CLIP-like models, although without any groundbreaking results or discoveries.\n\nWeaknesses\n\n1. The paper borrows and stitches ideas from various existing works. The only novelty being the use of MLLM for generating labels.\n2. It is difficult to understand the correct flow of the entire methodology. There are three distinct operations: 1) building the candidate set with MLLM and nearest neighbor image retrieval; 2) GMM for filtering out noisy samples; 3) Label filtering. My question is, does the order in which the operations are performed matters? Is GMM performed before label filtering? I guess the GMM and label filter operations are highly correlated. \n3. I don't agree with the validity of experiment done in Figure-A4. The authors select 9-shot data from the Flowers-102 dataset, and ablate on the value of k from 1 to 9. However, in the real world, one might not be guaranteed to have 9 examples for each class. One experiment done in Figure-2 uses fixed k for varying shots. A complement to these results would be a single graph that shows the performance of models with simultaneously varying k and m values. \n4. The paper misses discussion on the similarities and dissimilarities of their method with retrieval-augmented classification methods such as RAC [1].\n5. Please see `Requested Changes` section for further comments.\n\nReferences \n\n[1] Long, Alexander, et al. \"Retrieval augmented classification for long-tail visual recognition.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The proposed method enables CLIP-like models to perform vocabulary-free zero-shot image classification using labels generated from a Multimodal LLM (MLLM).\n2. The proposed method mitigates noisy labels generated by the MLLM by aggregating labels of similar images retrieved using CLIP's image features.\n3. Extensive experiments are done including various open-source and proprietary MLLMs, along with ablation studies showcasing the superior performance of the proposed model.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I would like to see an ablation that justifies the order of operations: 1) Nearest Neighbor selection; 2) GMM; 3) Label Filtering. Also, in Table-5, the paper should include results with and without GMM.\n2. The paper should include an ablation study showing performance of NeaR with and without prompt tuning.\n3. The authors should also include results of prompt tuning with ZS-CLIP-GT (Upper Bound) method. I believe that would denote the true upper-bound performance of CLIP.\n4. The paper should cite and discuss other pseudo-label based classification methods such as DualCoOp [1], VLPL [2] and CaSED [3]. Ideally, I would like to see NeaR compared with these methods especially VLPL, which is very similar to proposed method.\n5. The paper needs to visualize the noisy samples filtered out by the GMM method. Do these sample correlate with the failure cases of the model?\n\nReferences\n\n[1] Sun, Ximeng, Ping Hu, and Kate Saenko. \"Dualcoop: Fast adaptation to multi-label recognition with limited annotations.\" Advances in Neural Information Processing Systems 35 (2022): 30569-30582.\n\n[2] Xing, Xin, et al. \"Vision-Language Pseudo-Labels for Single-Positive Multi-Label Learning.\" 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). IEEE, 2024.\n\n[3] Conti, Alessandro, et al. \"Vocabulary-free image classification.\" Advances in Neural Information Processing Systems 36 (2023): 30662-30680.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5c9cCiOm0F", "reviewer": "Reviewer_Q8Kb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n1. The proposed method shows improved efficiency compared to MLLM-based baselines and better performance than contemporary VF baselines.\n\n2. The proposed method is computationally efficient and straightforward to implement.\n\n**Weaknesses**\n\n1. **The presentation could be improved:**\\\n    (1).  Several symbols and concepts are not clearly explained. These include: the random candidate set method mentioned in Table 2, the implementation details of prompt training in the warmup subsection, the construction of $q^j_i$ (specifically, how a set is converted into a score), and the circle-dot symbol used in the probability rescaling procedure.\n\n2. **The ablation studies are not comprehensive enough:** \\\n (1) Ablations on the choice of vision foundation model (e.g., DINO, MAE) for selecting the top-k similar images are not included.  \\\n (2) An ablation study for the value of k in the top-k selection is missing. \\\n (3) The paper is missing an ablation study for the proposed candidate set-guided label refinement strategy. \\\n (4) The primary ablation results are presented on a single dataset. It is recommended to perform these ablations on all evaluated datasets.\n\n3. **The claim of improved efficiency is only supported by data in Table 1**; efficiency metrics for the results in Table 3 are missing.\n\n4. **Some claims appear to be inappropriate or incorrect.** For example, the authors state in the contributions that their method outperforms MLLM baselines. However, the results in Table 3 show that the best overall performance is achieved by the Qwen2-VL-7B model, which is an MLLM.\n\n5. The scaling capabilities of the method on larger CLIP backbones (e.g., Large and Giant) are not validated. Furthermore, the generalization of the method to CLIP variants (e.g., SigLIP) has not been explored.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a framework for vocabulary-free, fine-grained visual recognition. Pseudo-labels, generated from Multimodal Large Language Models (MLLMs), are refined and enhanced by their k-Nearest Neighbors (k-NN) sets during training. These labels are then used to supervise a CLIP model. Additionally, techniques such as a warmup phase and candidate set-guided label refinement are employed to improve overall performance. A label filtering method is also proposed for further enhancement at test time. The proposed method demonstrates greater efficiency compared to MLLM-based baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the concerns raised in the weaknesses section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)