Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs
Authors: Yunqi Hong
Sohyun An
Andrew Bai
Neil Lin
Cho-Jui Hsieh
Venue: NeurIPS 2025 poster ()
Keywords: Fine-grained image classification
unsupervised prompt optimization
Primary area: applications

Abstract: Despite Multimodal Large Language Models (MLLMs) showing promising results on general zero-shot image classification tasks, fine-grained image classification remains challenging. It demands precise attention to subtle visual details to distinguish between visually similar subcategories—details that MLLMs may easily overlook without explicit guidance. To address this, we introduce AutoSEP, an iterative self-supervised prompt learning framework designed to enhance MLLM fine-grained classification capabilities in a fully unsupervised manner. Our core idea is to leverage unlabeled data to learn a description prompt that guides MLLMs in identifying crucial discriminative features within an image, and boost classification accuracy. We developed an automatic self-enhancing prompt learning framework called AutoSEP to iteratively improve the description prompt using unlabeled data, based on instance-level classification scoring function. AutoSEP only requires black-box access to MLLMs, eliminating the need for any training or fine-tuning. We evaluate our approach on multiple fine-grained classification datasets. It consistently outperforms other unsupervised baselines, demonstrating the effectiveness of our self-supervised optimization framework. Notably, AutoSEP in average improves 13\% over standard zero-shot classification and 3\% over the best-performing baselines.

Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs
Yunqi Hong', Sohyun An', Andrew Bai', Neil Y. C. Lin?, Cho-Jui Hsieh'
*Computer Science Department, University of California, Los Angeles Mechaniica and Aerospace Engineering Department, University of California, Los Angeles (yunqihong, sohyun0423, andrewbai, chohsieh)@cs. ucla.edu, neillingg wcla.
Abstract
Despite Multimodal Large Language Models (MLLMs) showing promising results on general zero-shot image classification tasks, fine-grained image classification remains challenging. It demands precise attention to subtle visual details to dis- tinguish between visually similar subcategories--detall that MI.LMs may easily overlook without explicit guidance. To address this, we introduce AutoSEP, an iterative self-supervised prompt learning framework designed to enhance MLLM fine-grained classification capabilities in fully unsupervised manner. Our core idea is to leverage unlabeled data to learn a description prompt that guides MLLMs in identifying crucial discriminative features within an image, and boosts classification accuracy. We developed an automatic self- eef-ncnhan prompt learning framework called AutoSEP to iteratively improve the description prompt using unlabeled data, based on instance-level classification scoring function. AutoSEP only requires black-box access to MI.LMs, eliminating the need for any training or fine-tuning. We evaluate our approach on multiple fine-grained classification datasets. It consis- tently outperforms other unsupervised baselines, demonstrating the effectiveness of our self-supervised optimization framework. Notably, AutoSEP in average im- proves 13% over standard zero-shot classification and 3% over the best-performing baselines. Code is available at httpp:/// thub. com com/y--moee menn/y-hoopg h AutoSEP.
Introduction
While Multimodal Large Language Models (MLLMs) exhibit impressive zero-shot image classifica- tion capabilities on general image datasets [1, 2, 8, 20, 4, 15, 40], their performance often declines on fine-grained classification tasks. These tasks, which involve distinguishing between visually similar subcategories like specific bird, animal, or plant species [12, 10, 21), pose a challenge for ML. The difficulty stems from the need distinguish subtle visual differences; without explicit guidance, MILMMs tend to overlook crucial details or misinterpret key characteristics [35, 18, 22).
Previous efforts to enhance MILMs' fine-grained classification abilities have largely relied on training the model with large-scale labeled datasets [12, 38), which are highly resource-intensive, demanding both significant computational resources and vast amounts of high-quality labeled data. This especially challenging for fine-grained classification tasks, where collecting and annotating data is expensive and consuming due the subtle visual distinctions between classes. In contrast, while unlabeled images are readily available at test time, there has been limited exploration into how such data could leveraged to enhance classification performance. This motivates us to explore whether ML.MMs can self- self-improve their fine- mme-graneed classification ability using only unlabeled data and with only black-box access to the MILMM.
But how can unlabeled data be leveraged to improve MLLMs' zero-shot prediction capabilities? While self-supervised learning literature [3. 13, 31] has demonstrated that unlabeled data can be used to
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

train encoders to extract better discriminative features, it is nontrivial to achieve this in a zero-shot and black-box setting. To address this challenge, we develop Automatic Self- Eaffnnnhnnning Prompt Learning (AutoSEP), novel framework for improving fine-grainet visual classification MILLM with unlabeled data (see Figure 1). AutoSEP introduces an intermediate image description step, in which the MILMM is prompted to generate textual description of an image's s visual features. The MLLM then makes the final classification based on the original image and the additional textual description. This image description step can be viewed mapping the raw image to text that highlights discriminative and relevant features for the task, analogous to the role of encoders in contrastive learning. To evaluate the quality of descriptions using unlabeled data, we design instance-leve classification scoring function that measures the MLLM's 's ability to retrieve the correct description for each image relative to those of other images. A prompt optimization algorithm is designed to iteratively improve the snneription-genrratiin prompt based on this scoring function, without requiring white-box access to the MILLMs. By focusing subtle yet critical visual differences, the MILLM iteratively improves its fine-grained classification performance without requiring any labeled training data.
MLLM Classification
Zero-shot
Image
Prediction
AutoSEP
Image
MLLM Description Generation
Description
MLLM Classification
Prediction
Optimizing with unlabeled data
Optimized Description Generation Prompt
Analyze the bird depicted the image, focusing on the following details:
""Beak"" *M Describe
(e.g. curved, pointed, blunt), any distinctive markings, "Head:"" Describe its color, any distinctive markings (e.g.. eye stripe, eyebrow, throat patch), and the presence or absence of crest.
langth,
"Wings " Note the color, pattern (e.g., banded, barred, spotted), and white patches or bands.
"Tail." Describe its length, shape (e.g.. graduated, rounded, square), and any distinctive patterns (e.g.. banding, barring).
"Body:"" Describe the overall body shape (e e.g. slender, stout), the color of the chest and belly, and any distinctive markings.
"Overall Size and Posture:" Mention the bird appears large small, and describe its posture (e.g.. upright, hunched, relaxed).
"Exclude any background details or other information from your description."
Figure I: Top: Typical zero-shot pipeline for MLLM classification. Middle: Illustration of AutoSEP pipeline. An example of an AutoSEP-optimizze description-generaation prompt.
Bottom:
To summarize, our key contributions
follows:
We introduce a new self-supervised, black-box framework, AutoSEP, based on instance-level description retrieval to improve MLLMs fine-grained image classification capabilities.
We show that AutoSEP significantly enhances MLLMs' fine-grained image classification capabilities (13% average improvement over zero-shot and 3% over the best-performing baselines) on 8 fine-grained tasks.
To the best of our knowledge, this is the first work studying how to use unlabeled data to improve zero-shot image classification performance in the black-box setting for MLLMs.
2 Related Work
Fine-grained image classification with MLLMs. MLLMs have demonstrated remarkable perfor- mance across broad spectrum of visual tasks, primarily due to pretraining on large-scale datasets 20, 19, 40, 27, 14, 30]. However, since these models are predominantly trained on web-scraped corpora, they tend to underperform on fine-grained classification tasks [12, 10, 21]. To mitigate this limitation, prior works [12, 38] have employed extensive training using large-scale datasets. While effective, these approaches are computationally expensive and heavily reliant on high high-quality labeled data, rendering them impractical in scenarios where ine-grained distinctions are critical. This motivates us to explore an alternative approach, which leverages readily available unlabeled images to improve fine-grate fine-grained classification MILMMs through lightweight framework that requires neither labeled data nor white-box access to the model.

---

Prompt optimization. Automatic prompt optimization techniques have seen substantial progress in recent years, demonstrating their effectiveness in black-box, computationally constrained settings with a wide range of strategies for searching for effective prompts [26, 7, 37, 39, 24]. Specifically, several approaches [39, 24, 32] utilize L.LMs to identify and summarize erroneous cases, which are then iteratively used to refine the prompts. Others [11, 9] adopt evolutionary strategies, generating diverse candidates through mutation and selection. Yang et al. [34] treats LLMs as optimizers, leveraging historical prompt trajectories and associated performance signals to guide prompt generation, while Yang et al. [36], Juneja et al. [17] decompose prompts into semantic segments for finer-grained control. Although these methods have shown strong performance, they largely depend on labeled data to supervise the optimization process. To mitigate this dependency, recent work such as SPO [33] formulates prompt optimization in self supervise manner by generating candidate prompts with LLMS and evaluating them through 4L-Maased pairwise preference comparisons of the resulting model outputs. In contrast, we proposed a higher-level framework for fine-graines visual classification that involves leveraging unlabeled images and explicit prompting for textual descriptions, whereas SPO is lower-level prompt optimization technique. Furthermore, we find that SPO does not perform well even when incorporated into our framework due to its choice of scoring function (see Section 4).
3 Method
The key insight of AutoSEP is leveraging instance-level classification signal, which does not require label information, to inform the optimization of the description generation prompt for improving fine- grained classification. In Section 3.2, we introduce the technique of generating textual descriptions and combining with image inputs to improve visual classification. In Section 3.3, we detail the procedures for learning the description--eenraatoo prompt in self-supervised manner from instance-level classification scoring.
3.1
Problem Setting
Let M denote the MILM and denote the classification prompt. The vanilla zero-shot prediction is
M (xr, q).
(1)
where x is the given image. While MLLMS often achieve strong performance on commonly seen basic-level classes, they tend underperform on fine-grained classification tasks. We assume there exists a small unlabeled dataset X = {I1, I2,.... ..., Inl (more detailed experiments on different dataset sizes can be found in Section 4.3.) Our goal is to utilize X improve the zero-sho prediction accuracy on fine- aine-grined categories. Also, we consider only black-box access to the MLLM, which means fine-tuning prohibited.
3.2 Image Description Generation for Fine-grained Classification
Traditional self-supervised learning (SSL) leverages unlabeled data to contrastively learn better representations [3, 13, 31]. Inspired by SSL, we introduce additional description generation step, where a "description generation prompt" p is used to generate a text description t for an image x. Both the description and the raw image then jointly passed to the MLLM for predicting the label (see Figure 1):
(zx [q, t)), where M (x,
(2)
Intuitively, M(.p) functions as encoder, translating the input into text description t. This description then helps the MLLM conduct the final prediction.
However, a naive description generation prompt yields non-informative descriptions, and even worse, many descriptions might be mislcading. Therefore, our method utilize unlabeled data X to iteratively refine how MLLMs analyze and describe images, guiding them away from generic or incorrect visual features towards distinguishing and precise attributes.
3.3 Automatic Self-enhancing Prompt Learning
To learn better description-genrrate prompt, we formalize an optimization objective based on a scoring function (X,,) that evaluates the effectiveness of each candidate prompt p:
pd
arg max (XX,), p),
(3)
PEC
3

---

where & represents the space of coherent natural language prompts.
Improve
Description &
Image
Description
Description
Instance-Loval
>Wrong
Prompt
Neg....
Description
Classification
Reflection
Negative Sample
Instance-Lovel Classification
Image
Description
Prompt
Text (deaeriptioo.
Text 1: [deacription.
Which description correcty describes image?
Figure 2: An illustration automatic self eahancinnni prompt learning with instance- classification.
3.3.1 Instance-level Classification for Unsupervised Scoring
Inspired by traditional self-supervised learning methods [3, 13, 31J, which learn the encoders to pull positive pairs closer and push negative pairs further apart contrastively, propose to optimize the prompt so that the MLLM generates highly distinctive descriptions for each individual unla- beled image. By doing so, our method maximizes the inter-image distinctiveness of the generated descriptions.
We frame the optimization an instance-level classification task, where the MILM learns to correctly associate each image with its own generated description while distinguishing from the descriptions of other images (See Figure 2). To quantify how well the MILM can distinguish between different
image descriptions, we define sample matching correctness indicator V (a,, xyy) Given an image a; and two candidate descriptions I; and (j. which are generated from images and Tj. respectively, we present the MILMM with a binary-choice prompt G(Ias 16). The prompt asks "Text (la). Text ). Which description correctly describes the image? The first the second?". where (ta, to) is either (t,, tÿ) or (t,, t;), randomly shuffled with equal probability. The correctness indicator function then formulated as:
1 MMa G(4,,1)) "first",,
2 =
V (T) =
1 [M G(tz,t4)) "second")) Z =
where Z ~ (ernoulli(055) determines the random ordering of descriptions prevent positional bias. The function Vix,, 2,) returns if the MLLM correctly selects the description corresponding to z,, and 0 otherwise.
We compute the unsupervised scoring function that measures the model's ability to correctly match descriptions to their respective images. Formally, the scoring function is defined as:
((Xp)) = Σ Σ V(zi,zz).
(5)
Z.EX 2{;)(((;)
To avoid iterating over all descriptions, ((XP)) can be approximated with random negative samples:
(X,
ΣΣ Σ V(n,
le
F.EX2,E00
where O, RandomSample(X33 (fx)) (.:},k)
represents a set of randomly sampled images from X, excluding Fú, acting as negative samples. k X normalizes the score by the total number of evaluations conducted.
3.3.2
Description--eenerttio Prompt Optimization
To begin the optimization process, we start with simple yet effective initial generation prompt that instructs the MILMM describe the object. The prompt can be found in Appendix A.
Feedback and modification. To iteratively refine the prompt, we adopt feedback-driven approach that identifies and addresses the weaknesses in the current generation prompt. An illustration of AutoSEP workflow and prompts can be found in Appendix B. First, collect error pairs

---

Jerror = ([zg,Z)) Iz, 2, E X,2; + 2,, (z),,zy) = 0}, where the generated descriptions failed to correctly distinguish between images. Then we sample subset of Jerror and construct diagnostic prompt to analyze potential shortcomings of the current prompt. Specifically, with the current prompt together with examples of mismatched descriptions, and ask to explain why the current prompt may have led to these errors. This process, referred to as the Reflect operator, Reflect(.), yields a set of potential reasons explaining the observed errors. Based on MLLM's feedback, we then prompt to revise the current prompt by explicitly addressing the identified shortcomings. This refinement step is denoted as the Modify operator, Modify(-) (-), which outputs an improved version of the generation prompt. Each instance of feedback produces candidate revised prompt, resulting in multiple refined versions aimed at enhancing the distinctiveness of the generated descriptions. It has been shown in the literature that this reflect-and-mokde mandmmddffy approach is effective in automatic prompt engineering [24].
present the MILM
Iterative optimization. Our AutoSEP optimization process follows an iterative framework where the generation prompt progressively refined to generate highly distinctive descriptions (Algorithm 1). Starting with an initial prompt, evaluate its effectiveness by generating descriptions for a batch of images and forming an instance-level classification set by sampling negative images, Errors in the instance-level classification are identified and used to expand the prompt by adjusting it to better capture distinctive attributes. At each iteration, a set of candidate prompts is evaluated using a scoring function defined in Section 3.3. 1, and only the top- cop-errfrmmnn prompts next round. This iterative adaptation ensures that the MLLM increasingly focuses on fine-grained details, improving its ability generate more informative descriptions. The process terminates after predefined number iterations N, and the best-performing prompt is selected as the final optimized prompt.
retained for the
By maximizing (XX, k). we iteratively refine the prompt p improve the distinctiveness of the generated descriptions, ultimately enhancing the MLLM's ability to perform ane-grained classifica- tion in self-supervised manner.
Algorithm AutoSEP: Automatic Self Self-Enhancing Prompt Learning
Require: Pot initial prompt, N: iterations, X: unlabeled dataset, k: negative samples per image, b: top prompts
retained per iteration, I: number of reflections, (r(-): scoring function
I: Pa t }
Initialize candidate prompt set
{20}, where " (Ms, e X
Generate descriptions
for = to do
4:
X, ~ U(X)
D Random minibatch from X
5:
P. < P1-1
6:
for p Pi-1 do
J {(01,2;) 2, E Xa,i 1}. where each æ., negative samples æ, are randomly & Construct instance-level classification set b Collect errors
selected
8:
Jorro = (2x, (Zi, E (Z,
(2. x,) = 0}
Sorror C Jamor a sampled subset for each I....,
10:
(91, 82: .. } U Reflect(p, Jerroz)
Reflect on the errors (Sec. 3.3.2) b> Modify prompts (Sec. 3.3.2)
11:
H = (h1,h2,... hu) = U U Modify(p. gi, Jé.oo)
12:
P. P.UH
13:
Tu ((M(a,,) x; X3 l HI
b Generate descriptions for new prompts
14:
end for
15:
S. = [ k) 1p P.J
Evaluate prompts
16:
Pt 6 {p E Pe ((X, k) 2 T]. where 7 is the highest score in S.
17:
end for
18: Return p arg maxpePy (X.pkk)
4 Experiments
4.1 Experiment Settings
Datasets. We conduct experiments on four fine-grained image classification datasets, including CUB-200-2011 1 [29] (bird classification), iNaturalist 2021 [28] (various wild species), Stanford
Dogs [6], and VegFru [25]. For each dataset, construct fine-grained classification tasks on subsets
5

---

of categories that are particularly challenging for MLLMs. We construct three bird subsets with high confusion rates: CUB cuckoo (Black-billed Mangrove, and Yellow-billed cuckoo), CUB oriole (Hooded, Orchard, and Scott's 's oriole), and CUB _irev (Philadelphia, Red-eyed, and Warbling vireo). For iNauuraist, we select two subsets comprising visually similar species: iNat butterfly (Symbren- thia lilaea, Claudina crescent, Elada checkerspot) and iNat _uppieeee (Arctic lupine, Silvery lupine, Arizona lupine). For Stanford Dogs, we focus on three closely related terriers: StanfordDogs _errrer (Lakeland terrier, Norwich terrier, Cairn terrier). For the VegFru dataset, we also select two subsets consisting of visually similar species: VegFru ggrenss (Dandelion, Shepherd's purse, Prickly lettuce) and VegFru allium (Leek, Green Chinese onion, Bunching onion). All the results reported are evaluated on datasets that are balanced in class distribution.
VegFru_greens
Baselines. We compare method with both optimization- and optimization-basees approaches. For optimization- free methods, we consider: (1) Vanilla zero-shot, where the MILM directly prompted without additional context; (2) Zero-shot with descriptions, using image descriptions generated from an initial human-crafted prompt; (3) Zero-shot with majority vote, where we prompt the MILM to generate multiple responses with high temperature and take the most frequent prediction as the final answer; (4) Few-shot with random labels, where the MILM is shown m images with randomly assigned labels and then asked to classify the (m + 1)th image: (5) Few-shot with MLLM labels, where the MLLM is shown m images with its own predicted labels and then asked to classify the (m + 1)th image: (6) Multiple images display, where the MLLM directly shown m unlabeled images before predicting the (m + 1)th image, as proposed by Jiang et al. [16]; (7) K-means clustering, where we cluster image features extracted by pretrained image encoder, and then within each cluster, use the most frequent zero-shot prediction as the final label for all images in that cluster.
For prompt optimization-based methods, we consider three approaches to adapt them to the unsu- pervised setting: (1) Optimization with random labels, where we randomly assign labels to the images and use the classification accuracy under these random labels as the optimization objective; (2) Optimization with majority vote, where we iteratively assign pseudo-labels to images based on majority voting over multiple high-temperature outputs from the MLLM, and use the classification accuracy with respect to these pseudo-labels as the optimization objective; (3) Self-Supervieed Prompt Optimization (SPO) [33], originally designed for text-only tasks, which perform pairwise comparisons with outputs generated by different prompts infer their relative quality as signals to guide the optimization. We adopt SPO to our multimodal setting by asking the MLLM to compare the quality of image descriptions generated by different prompts. Following the original setup, we use GPT-4o [14] to conduct these comparisons. Prompts used for these baseline methods can be found in Appendix F.
Models. We conduct experiments with three state-of-the-art MLLMs: Gemini 1.5 Flash [27], GPT-4o [14], and Qwen2-VV--7BB-Instuue [30], covering both proprietary and open-source models. We performed the experiments local servers with 64 CPU cores and 4 Nvidia A6000 GPUs.
4.2 Experiment Results
Experiment results in Table I demonstrate that our method consistently outperforms all baseline approaches across all tested MLLMs. Notably, it achieves an average improvement of 13% over standard zero-shot classification and 3% over the best-performing baselines. These consistent gains highlight the robustness and generalizability of our approach across different model architectures and capabilities. Furthermore, our method operates without requiring any labeled data, making particularly well-suited for low-resource or open-world scenarios.
rescription-generatto prompt benefits from optimization. We also observe that in some cases, the initial description generation prompt can slightly improve classification performance. However, these improvements are generally marginal. And some cases, generating descriptions even degrades the performance. This suggests that unguided or naive prompting alone for description generation insufficient. More principled guidance and optimization are necessary for the model to generate informative and task-relevant descriptions that more effectively support classification.

---

Table Main results (aceuracy %). AutoSEP outperforms baselines consistently across all tasks and MLLMs. Source of variability stems from samples in the evaluation dataset. Gemini refers to Gemini 1,5 Flash.
Model
Gemini
GPT-4o
Qwen2-VL.
Gemini
GPT-4o
Qwen2-VL
-UB_oriole
Optimization-free
Vanilla zero-shot
11.68
11.46
58.54
52. 2.99
74.44
43.60
Zero-shot with descriptions
52.20=2.49
68.29
51.46+2000
46.74=2.31
73.71
50.7942.92
Zero-shot with majority vote
51.22+204
61.95=1.95
59.02
54.38=0.55
75.28=201
46.7444955
Few-shot with random labels
38.05
13.90
49.83
40.05
56,85
36,6316.50
Few-shot with MLLM labels
53,41
61.22.12.2
52.93
57.75
64.94
36,85.14.79
Multiple images display [16]
48.05
52.68
65.12
59.78 16.60
71.46.8.75
46,7412.8
K-means clustering
53.17
19.5
56,5942226
51.46=5.85
67.19
44,94
Optimization-bel
random labels
19.15=1885
65.85
52.20
51.69=2.36
72.58=1x8
51.01+183
With majority vote
44.88=469
67.32+2211
53.41
17.90
73.03=1174
1..65+
SPO
55.98=1.34
66.59-22.68
56. 10.10
49.49 49.49
76.12
46.52+1..7
AutoSEP (Ours)
61.22:2230
75.12:1185
63.41
08.20
75.96=00.0
55.1311588
CUB_vire
iNat_ NNtt_butterfly
Optimization-fre
Vanilla shot
49.21
1..5
55.2813.0
61. 14.111
68.00=1.46
40.25
Zero-shot with descriptions
51.46.1.11
86.07
50.7913.22
56.57
71.14
48.2511448
Zero-shot with majority vote
49.21
76.69
55.061301
62.00
66.07
46.5611.59
Few-shot with random labels
45.62
57.08
43.8247.78
45.43=1338
64.00=10.
32.1348.47
Few-shot with MILM labels
84.94
72.36-10.5
46.97.173
51.43=3326
82.57:143
42.13:12.2
Multiple images display [16]
56. 18
13.54
63.82.15.4
11.71
12.29=122
32.25:3344
K means clustering
40.22.19.25
17.08
17.53
58.57
59.14
38.38
Optimization-base
With random labels
41.24
72.81
48.3144.44
53.24
29.86
45.13
With majority
$3.71
87.19
51.24+2272
=2.2
61.07=1.8
$1.50+3366
SPO
52.81
70.34
51.69+1.99
23.1442224
79.71
45.26+227
AutoSEP (Ours)
59.18=0.55
87.42+1.10
65.45+1224
66.57=1.42
82.71
55.38+1.00
iNat lupine
StunfordDogy terrier
Optimization-free
Vanilla zero-shot
62.00
70.00
44.22+374
60.80
91.39=0048
73.20+220
Zero-shot with descriptions
16.67=1.2
13.00=1225
44.0012249
66. 66.22:2241
91.42=1.17
76.00+2.53
Zero-shot with majority vote
1..78
72.33
47.33
62.67=0.73
90.89=1.0
76.27:2213
Few-shot with random labels
47.5627.31
48.33.15.8
42.0017.38
60.27=10.3
66.89
12.1318.90
Few-shot with MILMM labels
60.22=114
69.3343.9
46.44:1675
66.67=103
87.56-347
64.674684
Multiple images display [16]
63.47-771
74.00=5.54
50.44+286
65.87=404
79.78=9.4
77.734449
K-means clustering
26.11
50.0022.11
30.001
39.60=7.50
68.67=1.00
64.1345.66
Optimization-base
With random labels
58.44
73.67
46.00
53.33
90.00
1..73.1887
With majority vote
59.33
17.17
44.3
62.27 1.882
91.11
17.74
SPO
56.44
64.88
46.5613.4
65. (5.16..60
81.47
64.27
AutoSEP (Ours)
66.15
76.60
54.6743.33
69.28 99.9
92.18=1.09
81.70+1..0
VegFru greens
VegFru allium
Optimization-free
Vanilla zero-shot
86.00
78.82++0.54
68.89+306
72.67=33118
60.44=1.6
54.22+3.44
Zero-shot with descriptions
77. 33.19
77.33+1.94
63.61+182
7..44=1.98
64.89=151
59.17+1.98
Zero-shot with majority vote
8..83
78.00=0.44
71.39+2.3
75.56=1.88
62,44=257
56.89+237
Few-shot with random labels
74.44=6.30
70.2228.47
33.33-+000
67.33=3.18
61.33=100
33.33+000
Few-shot with MLLM labels
84.44
78. 19.819
68.4415.14
75. 25.11:270 =2739
73.1 1:9233
59.11415.1
Multiple images display [16]
4..44
72.89+5.14
70.22+
79.56=2.69
66 66.00=2.29
59.564422
K K-means clustering
67.78
67.56+4994
64.0045.19
51.33=14.1
50.89=9.3
42.6747.12
Optimization-baa
With random labels
80.5622433
79.78=m044
48.5242.77
71.56=2.06
64.22-1147
45.194448
With majority vote
81.19
74.00=a..0
55.561091
74.22::0.4
64.67=0083
16,94+1.21
SPO
84.37
78. 40
57.78:13144
73.78=11.81
65. 65.56=2.90
60.00
AutoSEP (Ours)
87.78
81.11:0.7
72.22.1091
78.89
69.11.10.83
62.4412248

---

Instance-level classification is an effective unsupervised scoring signal, outperforming alterna- tive unsupervised scoring strategies. A key component of our method is the use of instance-level classification accuracy the unsupervised scoring signal during optimization. To better understand its effectiveness, we compare against three alternative unsupervised scoring methods for optimiza- tion: (1) Classification accuracy with randomly assigned labels; (2) Classification accuracy with majority vote pseudo-labels, where pseudo-labels are obtained by aggregating high-temperature predictions from the MLLM; (3) Pairwise output evaluation in SPO [33], which relies on pairwise comparisons of model outputs to estimate relative prompt quality.
While these alternatives sometimes yield performance gains over optimization-fie free baselines, they exhibit limited reliability and overall lower performance compared our instance-level classification approach. Majority vote both methods fall short stability and effectiveness. Although SPO has demonstrated strong performance on several text-only reasoning and question answering tasks, it does not transfer well to fine-grained image classification. The results suggest that while LLMSs can effectively compare the quality of reasoning paths in purely textual contexts, MLLMs struggle to consistently evaluate the informativeness image descriptions for fine-gras fine-grained classification. As a result, such evaluations offer weak and often noisy feedback signals for optimization. These results highlight the importance of instance-level classification as a robust and effective unsupervised signal.
tends to perform better than using random labels, but
4.3 Discussion and Analysis
Learning dynamics. We study how the classification performance evolves throughout the optimiza- tion process and how many iterations are usually required for the algorithm take effect. Full details of experimental settings are mentioned in Appendix C.I. The average results and their variances reported in Figure 3a and 3b computed by averaging these classification performances across all three runs. The Max results are computed by taking the best performance among the four prompts at each iteration, then averaging across the three independent runs.
Both the instance-level (Figure 3a) and class-wise (Figure 3b) classification performance consistently improves with more optimization iterations. More demonstrations other datasets be found in Appendix C. The correlational evidence suggests that instance-level classification serves as an effective unsupervised optimization signal to guide the model toward better class-wise discrimination. These results validate our design of leveraging instance-level feedback to drive prompt optimization for tine-grained classification in the absence of labeled data.
CUB cuckoo
Gemini
CUB cuckoo
Gemini
CUB cuckoo
0.82
0.62
0.60
0.80
0.55
00.78
0.60
0.50
0.76
Accuracy
0.58
0.45
0.74
0.72
0.56
0.40
0.35
0.70
0.68
Max
Average
0.54
0.52
Average
Max
0.30
0.25
0.20
Gamin
Owen
GP7-4o
0.66
2
4
5
1
2
Iteration
Iteration
Iteration
(a) Instance- Classification
(b) Class- wise Classification
(c) Prompt word diversity
Figure 3: Evolution of metrics with incrcasing optimization iterations.
Correlation between instance-level classification and class-wise classification. To further support the effectiveness of instance-level classification optimization signal, also compute the statis- tical Pearson correlation between instance-level and class-wise classification performance. Table shows strong positive Pearson correlation across different settings, suggesting that improvements in instance-level accuracy are closely associated with better class-wise classification performance. This empirical evidence reinforces our hypothesis that instance-level supervision can serve powerful proxy to guide the description optimization for fine-grained image classification in the absence of labeled data.
Diversity of generation prompts. optimization process, we evaluate the diversity of the generated prompts using quantitative diversity
To analyze how much new information is introduced during the

---

Table 2: Correlation between instance-level classification and class-wise classification.
Datasets
cu__cockoo
CUB_oriole
CUB CUB_vireo
iNat buft_rfy
Wat_lupine
Gemini 1.5 Flash
0.72
0.47=0024
0.70=0.19
0.63#0022
0.53=0002
Qwen2-VL-77B-Immmne Insttrcct
0.63
0.38
0.64
0.84
0.70
score. Specifically, for each prompt, we count the number of semantic keywords (lemmas of nouns, verbs, and adjectives excluding stop words), well as the number of unique words, defined words that appear only in that specific prompt and not in any other prompt throughout the optimization. These counts are normalized to compute the diversity score for each prompt. Detailed definition can be found in Appendix F.
As shown in Figure 3c, the diversity score steadily increases over iterations, indicating that the optimization process encourages the generation of increasingly varied and semantically rich prompts. This trend suggests that the model is progressively exploring a broader and more informative prompt space, which may potentially contribute to improved classification performance.
Computational complexity. Let b denote the number of top prompts retained, / is the number of prompt reflections generated per retained prompt, is the number of negative samples used per image in the instance-level classification, and is the number of unlabeled images. Each iteration of our algorithm involves generating 6- reflections, and b modified candidate prompts accordingly. For each of these prompts, we generate descriptions for all images, resulting in 6 / quiries the MLLM for description generation. Additionally, instance-level classification requires comparing each image's 's description against i negative samples under each prompt, leading to b 1- k n further MLLM queries. Therefore, the total number of MLLM queries per iteration is (2 + + kn)- bl. In our implementation, we typically set 4,7 S 5, and k resulting in an approximate query complexity of O(60n) per iteration.
For optimization-fres baselines, the computational cost is relatively low, since they do not require iterative prompt refinement. For optimization- based baselines, we used the same experimental setup as AutoSEP, resulting in comparable computational complexity. More specifically, optimization with random labels takes approximately O + n) bl) = O(40n) MLLM queries per iteration, as it involves only a single classification per image. Optimization with majority vote takes approximately O(2(1 + n. bl + en) O (45n), with = 5 denoting the number of majority votes collected for updates in each iteration. SPO [33] is more lightweight, requiring only ((2n) queries per iteration, since it only compares with the previous prompt and does not rely on several candidate prompts.
The actual runtime depends the inference speed of the MILMs. For example, one optimization iteration on 60 images with Gemini 1.5 Flash takes 12 minutes in average, demonstrating that AutoSEP is computationally feasible for practical use.
More samples, stronger instance-level signals, more stable optimization. We investigate how the number of unlabeled samples used during optimization impacts the performance of our method. Figure 4 shows a clear trend that the classification performance improves with increased number of samples. More demonstrations on other datasets can be found Appendix C.2.
CUB cuckoo Gemini
0.58
Accuracy
0.56
0.54
When the sample size is too small, instance-level classifica- tion constrained by limited variation and coverage, making it difficult to reliably evaluate a prompt's ability to generate distinctive and meaningful descriptions. This leads subopti- mal performance and introduces more noise and instability into the optimization process. In contrast, with more samples, the instance-leved signal becomes more robust, enabling more sta- ble optimization and resulting in consistent gains in class-wise classification accuracy. Notably, our method begins to perform reliably with as few as 60 unlabeled samples, demonstrating its practicality and scalability even in limited-data settings.
0.52
0.50
n=30
0.48
n-69
2
6
Iteration
Figure 4: Classification accuracy of Gemini with various number of sam- ples for optimization.

---

5 Conclusion
In this paper, we present a self-supervised prompt learning framework that improves the fine-grained image classification capabilities of MLLMs using only unlabeled data and black-box MLLM access. By optimizing the prompt for image description generation through instance-level classification feedback, our method enables MLLMs to focus on subtle, discriminative features critical for distin- guishing between visually similar categories. Extensive experiments across multiple fine-graine mne-graieed classification tasks and MLLMs demonstrate the robustness of our approach.
Limitations. Our method relies on the implicit assumption that subtle, dlass-diccimminatiee visual features can be accurately captured and described using natural language. This may not hold in cases where visual distinctions are extremely fine-grat une-grained or difficult to verbalize. Additionally, although our approach is fully unsupervised, it still requires sufficient number of unlabeled samples to ensure stable and effective prompt optimization. The absence of labeled data inherently limits the performance compared to fully supervised methods that benefit from explicit class-level guidance.
Acknowledgments and Disclosure of Funding
This work is supported by NSF 2048280, 2325121, 2244760, 2331966 and ONR N00014- 23-1- 2300:P00001.
References
[1] Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cahi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacoh Menick, Sehastian Borgeaud, Andrew Brock, Aida Nematzadch, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: visual language model for few-shot learning. 2022. org/ abaaaba//222 2204.. 14198.
URL attpps//aaaae
[2] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308. 12966, 1())3, 2023.
[3] Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607, PmLR, 2020.
[4] Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12)220001, 2024.
[5] Gheorghe Comanici, Eric Bieher, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with ad- vanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.0626/ 2025.
[6] F. Dataset. Novel datasets for fine-grained image categorization. In First workshop on fine grained visual categorization, CVPR, Citeseer. Citeseer. Citeseer, volume 5, page 2. Citeseer, 2011.
[7] Mingkai Deng, Jianyu Wang. Cheng-Ping Hsieh, Yihan Wang. Han Guo, Tianmin Shu, Meng Song, Eric P. Xing, and Zhiting Hu. Riprompt: Optimizing discrete text prompts with reinforcement learning, 2022. URL https: htmps:///rxxii: org/ abs/2205. 12548.
[8] Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. intemlm-xcompossee Mastering free-form text-image composition and comprehension in vision-language large model. arXiv preprint 24X.:24401116422 2024.
[9] Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder Self-referential self-improvement via prompt evolution, 2023. URL https://arxiv org/ aaaaabbb23399 16797.
[10] Gregor Geigle, Radu Timofte, and Goran Glavaš. African or european swallow? benchmarking large lingune models for fine-grained object classification. arXiv preprint mxXiv22406.14990 2024.
10

---

[11] Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song. Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Evoprompt: Connecting ilms with evolutionary algorithms yields powerful prompt optimizers, 2025. URL htttt:://aaatttttee org/ abs/2309... 08532.
[12] Hulingxiao He, Geng Li, Zijun Geng, Jinglin Xu, and Yuxin Peng. Analyzing and boosting the power of Lin-ggraied visual recognition for multi multi-modal modal large language models. arXiv preprint WxXv:2501.151400 2025.
[13] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 9729-9738, 2020.
[14] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.2127. 2024.
[15] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https::/ http///xxxivoo...aaa//// org/ abs//310. 2010/0080.........
[16] Yixing Jiang. Jeremy Irvin, Ji Hun Wang, Muhammad Ahmed Chaudhry, Jonathan H Chen, and Andrew Y Ng. Many-shot in-context learning in multimodal foundation models. arXiv preprint (rXiv:2405.0999) 2024.
[17] Gurusha Juneja, Nagarajan Natarajan, Hua Li. Jian Jiao, and Amit Sharma. Task facet learning: A structured approach to prompt optimization, 2024. URL https::/t///tttt http://aavvvv..raaaaaa//
10504.
[18] Sachyung Lee, Seunghyun Yoon, Trung Bui, Jing Shi, and Sungroh Yoon. Toward robust hyper-detailed image captioning: A multiagent approach and dual evaluation metrics for factuality and coverage. arXiv preprint 2Xi::2412.15484, 2024.
[19] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023. URL https: httppp::://aavv.../ae http://awwww..ogaa////aaaaaaa/ 12597.
[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 1.33882-349966 2023.
[21] Mingxuan Liu, Subhankar Roy, Wenjing Li, Zhun Zhong, Nicu Scbe, and Elisa Ricci. Democratizing fine-grained visual recognition with large language models. arXiv preprint arXiv:2401. 13837, 2024.
[22] Yexin Liu, Zhengyang Liang. Yueze Wang, Xianfeng Wu, Feilong Tang. Muyang He, Jian Li, Zheng Liu, Harry Yang. Sernam Lim, and Bo Zhao. Unveiling the ignorance of milms: Sceing clearly, answering incorrectly, 2025. URL https: / /awxia.vvvv... argg/bs/240666 10638.
[23] Rithesh Murthy, Ming Zhu, Liangwei Yang, Jielin Qiu, Juntao Tan, Shelby Heinecke, Caiming Xiong, Silvio Savarese, and Huan Wang. Promptomatix: An automatic prompt optimization framework for large language models. arXiv preprin! arXiv:2507. 2a..i25077.44,,,, 2025.
[24] Reid Pryzant, Dan Tter, Jerry Li, Yin Tat Lee, Chenguang 7hu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and heam search. arXiv preprint (rXix:300500349 2023.
[25] Yushan Feng Saihui Hou and Zilei Wang. Vegfru: domain-specifé datasct for fine-grained visual categorization. In IEEE International Conference on Computer Vision, 2017.
[26] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts, 2020. URL https: //arxiv. org abw///10...a.2/aae .
[27] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai. Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
[28] Grant Van Hom, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 12884 12893, 2021.
11

---

[29] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
[30] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- -aiiuala model's perception of the world any resolution. arXiv preprint 1rX1v:2409.22111, 2024.
[31] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on hypersphere. International conference on machine learning, pages 9929 9939. PMLR, 2020.
[32] Xinyuan Wang, Chenxi Li, Zhen Wang, Bai, Haotian Luo, Jiayou Zhang, Nebojsa Jojic. Eric Xing. and Zhiting Hu. Promptagent: Strategic planning with language models enables expert-level prompt optimization, 2023, URL https: //axxiv org/abs/eee awtpaass/.../////////.aa/aaaaaaa 16427.
[33] Jinyu Xiang, Jiayi Zhang. Zhaoyang Yu. Fengwei Teng, Jinhao Tu, Xinbing Liang, Sirui Hong. Chenglin Wu, and Yuyu Luo. Self-supervised prompt optimization. arXiv preprint arXiv:2502.0688 2025.
[34] Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe V. Le, Denny Zhou, and Xinyun Chen. Large language models as optimizers, 2024, URL ptttps//aariive wrt/ooo/////aa/e/ 222g/99../300....// 2ww.//399.03........
[35] Dingchen Yang, Bowen Cao, Guang Chen, and Changjun Jiang. Pensieve: Retrospect-then-coomppe mitigates visual hallucination. arXiv preprint arXiv:2403. 14401, 2024.
[36] Muchen Yang, Moxin Li, Yongle Li, Zijun Chen, Chongming Gao, Junqi Zhang, Yangyang Li, and Fuli Feng. Dual- -hase accelerated prompt optimization, 2024. URL https: arxiv. org/ abs/2406.... 13443.
[37] Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. Tempera: Test-time prompting via reinforcement learning, 2022. URL https://arxix: org/ abs /bb1/2221... 2212211199..
[38] Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. Why are visually -gguunly language models bad at image classification? arXiv preprint arXiv:2405.8415, 2024.
[39] Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models human-level prompt engineers, 2023. URL https://arxiv. org/abs/ 2211.01911
[40] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper limitations, but those are not discussed the paper.
The authors encouraged create separate "Limitations" scction their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specifiennt specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their hest judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: [NA]
Guidelines:
The NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
13

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (tttpe://nips..c/ppblee
Sid.dee//oeSuumiii ssionPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible,
so "No" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //np///p...///////// cc/public/ mul.dees/oode@uubmiesssoooooo for more details.
14

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4,1, Section 4,3 Computational Complexity, and Appendix E.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: Table 1.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
15

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips. cttss.ccppuuu httpppllcc/thhcssuudeeee inne??
Answer: [Yes]
Justification:
[Yes]
Guidelines:
The answer means that authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts the work performed?
Answer: [NA]
Justification: The paper introduces a technique for improving visual fine- graingr classification, which is net positive impact according to our understanding.
Guidelines:
The answer NA means that there is societal impact of the work performed.
the authors answer NA No. they should explain why their work has societal impact why the paper does address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point out. For example, the quality generative models could used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
legitimate to point out that an improvement in
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that been put in place responsible release data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No risk that are aware of.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should he released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
16

---

We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Sources are properly cited.
Guidelines:
The answer means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should he provided.
If assets released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Code anonymized and included in the supplementary materials.
Guidelines:
The NA means that the paper does release
Researchers should communicate the details of the dataset/code/model as part of their sub- missions structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing.
Guidelines:
answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent research with human subjects
Does the paper describe potential risks incurred by study participants. whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
17

---

Justification:
No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important. original. or non non-standard component of the core methods in this research? Note that if the L.M is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the rescarch, declaration is not required.
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https / htt:://eeriisscc//eeeee http:///ooeeeeeeeee//2////////// for what should or should not be described.
18

---

A Examples of Descriptions
Image
Generated
Desciipion
Descriptons
Tha st lia cra nitig
Partally contect Saar se Ourion in rad un and sose taun king
hax polred and knail
Partiity connool The oarion
long,
haok thrugh
heod
"portuff enl la secunie 'Smoal head it necine conercatio
plenager oike usd
the buck of reds.
edutticc white
rock paich The Nutale paid is not fessur this condie
eyertrp
shart ale
duleedivo bving "Meeat and skender is plauable key hattaoo
The bants hoer is biown w ih
Portually connect Heeris towen Fun iore, Eut epeeding trat's rec. clsse imanectie.
The tao ling alande wift
Incorent Bhack-dllss bu dh apda unddera de didinnztinn This owntnsss
Inttal promet
un regrly diarnde whroans that aprnal
ber (pooe the exclgmund or other oxtormtion
thr cadi of neck wiich key
Incerned The ned ofeerrual a Bluck baket Cushoo. Thene ro neck paich.
deeilling
Opsivined
digitly rained dark book
dightt
Anayos the nod ofpored in the Inage, trostingan
vyu-ing dowmsndry and tisk (oook) roecive owfteryy
sased pocteet cunnd) any vociiivv cerkicgs
"Beak" mo Seacthe coli lirgh, shape
headd browe wht prominer
Vouily comoed The laed sven, Thu gyv cuns will peor mone nel
purking p. ariae syebrie dhoal pactts and
redloppeiing
bared spotrol any whte patches sernes "f
"Wingas" hode
parte banded
sightly rounded tp.
king
lanard udgeat
decciption
ruantel whan lodkl, which
langth
lousded aqcam) and any strtnclvv pattenn in.
Correct bens tee uccurate stsemutiin.
"Bordy"" Dvsonier anath body shape clucde: stuet the coborol the chorr and baly. and dirincive
collnd maaking
"Oversl Stre and Pormonet stertor the nod ung.
The sidd body a siender and brown. with lig unders
Correct The image charry shows slender body brown upperpars. pulslight
uedget runcket moredd
"Facheee any maca dtttac onfveration
tranch, appenting
assorpaiis
rnlood and
arightonn branch befanicr
(a) Black-billed Cuckoo
Imnaye
Gerernted
Fact
zachoo with wlong. ealnder
Conrect Hat
haa hing pointed land sistion stodt book
Incorrect Whik the tail induus lona the bosk to nct short sbout
The head anal and round with uid staiiii slick and wiile cadkoms
Quokoss The lace to mos plan grassn beswein with subtee white thoott cresi.
The vey promineet
Incomrent The
there tud is not boiil baak carpo lkk in enikee or cone witt
The and' pia is mudied whitt,, with rodd aldroww paidr la back.
dianty seauts aledeww undeepaen clean white
upperport
Doants the iv the gios avago in dhise becaaro
bnd Jynons toolground olte anbrraiios
and
Lich
Cesorbe the bu the gloee peage is dessc fecuwop
yolloe and sleck
mardill unall darker andicie by the Vullow baliaaa Custook
pabuutr Aushars counpais desccis the
peosenc or apoamec of whec ahroat purto, color
syyee brown.
Correct The bird has darks eyea, and the natches the typicall care town color spades.
bnoigeoand or other sefuraadors
rounded
Ieo rounded shape lyecal of hallowollile Cuchooa
The feultirn unila Lodly urea bnow and
Cornect The Upprperr Buch and wingal ane unlfor brown...
indersatt wriie
(b) Yellow-billed Cuckoo
Figure 5: Examples showcasing descriptions generated from two different prompts using Gemini 1.5 Flash. Attributes highlighted in green indicate correct information, while those red denote incorrect attributes.
Figure shows two exampls of image descriptions generated by Gemini 1.5 Flash [27] using prompts before and after optimization. The initial descriptions tend to be vague or misleading, often misrepresenting key visual attributes. In contrast, the optimized prompts lead to more accurate and reliable descriptions that better reflect the distinctive features of the images.
19

---

B Detailed AutoSEP Workflow and Prompts
Inital
Eroor Collection
Prompt
dit imago
Cornect Usacrition: cooreect, cencription)
Weong Desor plont deseciptiion
image
Candidate
pooott (weerng Seacziphaonn
Enror Rellection Reflect))
I'm rying w a prompe used to genaebe description for the hied the image, whern the generated deseriatin con onhence the acro shot olatsilicaion of the bed in irage. three types arc Bloos Cuckoo, Nangrove Cuclico,
Cuckoo.
Collection
summet (proage)
but in prompt pareentees image csscripiion thet are too simple,, sims be and vagus, marinng to clatrquusn description cormenty matchee imagge
wrong
sttzlng)
Give reasone why the prompt could have gatten these eoamples wrong
Reflection
New Prompis Midlty))
write prompt used genamata descriptin the bed where penenated description
entence the zert-shot classiication of the tid in the image. The thrme types of binds ane A. Black-llled Cutinno Mangrowe Cuckenn Yellwwbllled Curkoo
Select Top-k
New Prompts
But ih prompt generates imaga ceeostol novr ther mokiee a dificcul is didiing ohrich desciptin ramnecly moithee the traege and weiiing the wring dearti from being chowen kiowing esarppiee: cernor cormor__ttii .rtring)
Generate
accemplee
prompp
Basadd on aboowa rnfomatioon wrsta cifarrnn rmproved Each prampt xSIAK KEN2X
Candidets Descriptions
new prompt
Instance-Lavvl Claxaftication
Ldeacraption, 31
Tedt (deccription_2)
Instance-Level Classilicatinn
Which description commetty describes the image? The first last the secord luad?
Figure 6: Workflow of AutoSEP with Optimization Prompts. Take CUB cuckoo as example.
Figure shows the detailed workflow of AutoSEP and prompts used in optimization. The prompt learning procedure is introduced in Section 3.3.2.
Additional Ablation Studies
C.1 Learning Dynamics
We study how the classification performance evolves throughout the optimization process and how many iterations are usually required
algorithm to effect. For each experimental setting, we perform three independent runs. In each run, we retain the top four candidate prompts at every iteration based on the scoring function. We use the average classification performance of these four prompts to represent the performance for that iteration. The average results and their variances reported in Figure 7 are computed by averaging these classification performances across all three runs. The Max results computed by taking the best performance among the four prompts at each iteration, averaging across three independent runs.
C.2 Number of Samples
More experiments different datasets conducted study the effect of the number of samples on AutoSEP framework (See Figure 8). results are quite consistent across different classification tasks. When the sample size is too small, the optimization process is unstable with large variance. In contrast, with more samples, the instance-level signal becomes more robust, resulting in more stable optimization and consistent gains in class-wise classification accuracy.
C.3 Different Sampling Methods
We conducted this ablation study to examine how the sampling strategy affects the final results. Specifically, replaced the random image sampling in AutoSEP with two alternatives: sampling only same-class images
20

---

CUB cuckoo Gemini
CUB vireo Qwen
INat butterfly Gemini
0.82
0.92
0.80
0.90
0.78
Accuracy
0.88
0.76
0.74
0.70
0.84
0.72
0.65
0.82
0.70
Max
Max
0.80
Max
0.68
Average Std
0.60
Average Std
0.78
Average Std
0.66
2
Iteration
Iteration
Iteration
(a) Instance-level Classification
(b) Instance-level Classification
(c) Instance- Classification
CUB cuckoo Gemini
CUB vireo Qwen
iNat butterfly Gemini
0.62
0.650
0.68
Accuracy
0.58
0.60
0.600
0.625
Accuracy
0.64
0.56
0.575
0.62
0.550
0.54
Max
0.525
Max
0.58
Max Average
0.52
Average
0.500
Averaae
0.56
1
2
5
5
2
Iteration
Iteration
Iteration
(d) Class-win wisc Classification
(c) Class- wisc Classification
(f) Class-wise Classification
Figure 7: Classification accuracy as function of iteration.
CUB oriole Gemini
CUB vireo Gemini
iNat butterfly Gemini
0.60
0.575
0.550
0.58
0.625
Accura
0.525
8 0.575
0.600
0.500
0.550
0.475
0.50
0.525
0.450
0.400
0.425
na:30
0-58
0.46
0.48
n=30
0-54
0.450
0.475
0.500
1-90
na:00
0-60
Iteration
iteration
Iteration
(a) Class-wise Classification
(b) Class-wise Classification
(c) Class-wise Classification
Figure 8: Classification accuracy of Gemini with various number of samples for optimization.
or only different-class images for instance-level classification. As shown in Table 3, neither same-class nor different class sampling leads to notable change in performance.
The goal of our deseription genera- tion process is clicit detailed, fine-
Table 3: Results (accuracy %) of different sampling processes.
grained understanding of each im-
age, instead of enforcing class-level discrimination. This image-specific descriptive richness is then used to guide the classification process in separate classification prompt. this context, encountering visually similar images-eve from the same class-is not a flaw but opportunity: it encourages the model to uncover subtle, discriminative visual cues that may otherwise he overlooked. In traditional self-supervised learning methods, similar images from the class are often sampled together to learn meaningful feature representations through contrastively learning. Our framework is inspired by this idea and pushes the model to verbalize these subtle differences. which ultimately improves the downstream classification.
Gemini 1.5 Flash
CUB_cuckoo
AutoSEP with different-class image sampling
60.98
AutoSEP with same-class image sampling AutoSEP with random image sampling (ours)
61.58+3.111
61.22:+2.30
D Additional Experiment Results
D.1 Results with Gemini 2.0 Flash
21

---

We conducted additional experi- ments with Gemini 2.0 Flash [5], with the results presented ble 4. While Gemini 2.0 Flash demonstrates substantial improve- ment over Gemini 1.5 Flash, its per- formance does not surpass that of GPT-4o on our benchmark tasks. Nevertheless, our AutoSEP method consistently improves the performance Gemini 2.0 Flash across both datasets, further confirming its effectiveness across different MLLMs.
Table 4: Results (accuracy %) on Gemini 2.0 Flash.
Gemini 2.0 Flash
CUB vireo
StanfordDogs Jerrie
Vanilla zero-shot
14.27=1.31
74.60+0:47
Zero-shot with descriptions
64.27=1.65 69.05=1.78
14.12:11.21
AutoSEP (ours)
85,94+1.05
D.2 Results with smaller MLLMs
We conducted additional experiments us- [30]. Since AutoSEP relies on strong agent for flection modification, we adopted a hybrid setup in which Gemini 1.5 Flash [27] served the Reflect and Modify op- erators well for image description generation, while was used for instance-level classification and evaluation. This setup follows com-
ing Dwen2-VL-7BB-nsssee
Table 5: Results (aceuracy %) on Qwen2-VL-7B -L-7BBLL---BBB
Qwen2-VL Lonn--LL-7B VL-7B-Instruct
CUB_cuckov
CUB oriole
Vanilla zero-shot Zero-shot with descriptions
40.241344
39.78:7741
39.27:12.68
50.45:
AutoSEP (ours)
61.79:+2.1
54.02=1.66
mon practice in prior prompt optimization studies [32, 33, 231. As shown in Table 5, our method also yiclds
substantial performance gains Qwen2-VL-7BBInstnun demonstrating its effectiveness smaller models.
E Prompt Details
this section, we provide the prompts used for baselines our experiments. Blue boxes show those used directly for classification; orange box shows description generation prompt before optimization; red box shows the prompt used for SPO [33].
Zero-shot (CUB_cuckoo)
Deternine what kind of bird this image shove from the following opticus:
Black-billed Cuckoo
В. Mangrove Cackoo
Yelloz-billed Cackno
prediction.
answer
Zero-shot with image description (CUB _uckoo)
Task
Determine what kind of bird this image shous from the following options:
B. Mangrove Cackoo Yelloz-billed
Anewer the letter from & to C prediction.
Prediction
image following features: (description)
The ansver is:
Few-shot with random/MLLM labels (CUB cuckoo)
Your task is to classify the image to three birds: A. Black-billk billed Cuckoo, B. Mangrove Cackoo, C. Yellov-billed Cuckoo.
The classification the inage ie:
22

---

The classification the image in: claesification inage
The clasaification the last image 18: (Answer Letter or or C)
Multiple images display [16] (CUB cuckoo)
Given the firat inage examplea, anaver the following question:
Weat kind of bird does the last inage chow: Black-billed Cuckso, B. Mangrove Cuckoo, C.
< Yollor-billed Cuckoo.
Directly Answer Letter & or B or C without any Turther information
Initial image description generation (CUB cuckoo)
Taak bird given inage detail, focusing on highly distinctive attributes are typical to this bird. Ignore the background or other information.
SPO [33] description quality evaluation (CUB_ccckoo)
I'm trying to urite prompt used generate description for the bird in the image, where the genarated description enhance the zero-shot clanstfication The three typea of birds are: Black-billed Cuckoo, B, Mangrove Cuckoo, c. Yellow-billed Cuckoo.
image.
There descriptions of the images generated by different proupts:
Text (description_1)
Text 2: {description_2)
Which description better describes the image? The first text or the second text? Provide your analysis and the choice you beliave better, XML tags encapsulata your response.
<anslyse>Sone mualysie</analye>> <choose>Firat/Bacomu (the batter anszer in p/chhoe))/choos>>>
F
Diversity Score
Quantifying the amount of semantic content introduced during optimization is essential for understanding the evolution prompts. To end, design a diversity score that captures two aspects:
Semantic richness, measured by counting the number of unique lemmas of content words (nouns, verbs, adjectives) after stop-word removal, denoted as(T(p));
Uniqueness across prompts, defined as number of words that appear exclusively given prompt and not any others, denoted ap))
To balance the contributions of these components, we normalize each by the maximum value observed across all prompts in the optimization:
T(p)
U())
T(p)
(P)
max; T(p))
max, U(p))
The final diversity score is then computed as the sum of these normalized terms:
D(p) T(p) + (p).
(8)
This metric captures inter-prompt uniqueness, which is central our analysis of prompts diversify throughout the optimization process.
23


Reviews:
[{"review_id": "K0LOYIx5u6", "reviewer": "Reviewer_gnDg", "summary": "This paper introduces AutoSEP, a method that aims to improve the zero-shot fine-grained classification ability of MLLM via automatic prompt optimization.  The proposed method requires the MLLM to generate a detailed image description and, based on that description, to conduct fine-grained classification. The core idea is to iteratively improve the description-generation prompt using unlabelled data.  To achieve this, it samples positive and negative image-text pairs (like contrastive learning) and asks MLLM to align the generated description with the correct image. Once fails, it asks the MLLM to refine the initial prompt. Experiments show the effectiveness of the proposed method.", "questions": "Since the method is simple and the paper is clear, I do not have any questions to clarify the paper.  It is noticeable that GPT-4o outperforms Gemini by a large margin across all benchmarks, while in my own experience, Gemini 2.0/2.5 performs much better than GPT on seeing image details.  I would appreciate it if you could verify this.  Please also see the weakness.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "[Strength]\n1. The paper is easy to follow and well organized.  \n2. The proposed method is logically clear and very simple.  It is an elegant combination of automatic prompt optimization with contrastive learning in fine-grained classification task, and I am happy to see that such a simple method actually brings consistent improvement across multiple models and benchmarks\n3. The experiments are sufficient and convincing.\n\n[Weakness]\nAlthough it is an elegant combination of two methods, the novelty of this method is somewhat limited.  The core of this method is still automatic prompt optimization, which is not a new idea.  Fitting automatic prompt optimization to fine-grained image classification is a delightful attempt, but cannot be recognized as a significant creation.  The scope of fine-grained classification also limits the significance of the proposed method.", "was_revised": true, "final_justification": "I will slightly raise my rating since some of my concerns have been resolved.", "rebuttal": "We’re very grateful to the reviewer’s valuable feedback. Below is our response to their concerns.\n\n**1. On method novelty**\n\n> **Reviewer’s concern**: Although it is an elegant combination of two methods, the novelty of this method is somewhat limited. The core of this method is still automatic prompt optimization, which is not a new idea. Fitting automatic prompt optimization to fine-grained image classification is a delightful attempt, but cannot be recognized as a significant creation. The scope of fine-grained classification also limits the significance of the proposed method.\n\n**Our Response**:\nWe appreciate the reviewer’s perspective and agree that automatic prompt optimization is not a new concept in itself. However, our contribution is not a reintroduction of this idea but rather a novel framework for its application in a new and challenging domain. Specifically, our contributions are:\n\n* To show **how unsupervised images can improve zero-shot prediction performance of LLMs**, a previously underexplored research direction. This setting is important for several real applications, such as medical image diagnosis where labeled data is scarce but unlabeled data are readily available.\n* To achieve this goal, since only unlabeled images are given, **traditional prompt optimization algorithms that rely on supervised data cannot be applied**. To resolve this issue, we propose a new framework to connect unsupervised prompt optimization with contrastive learning which uses **pairwise reasoning and self-supervised feedback** to iteratively improve descriptions that guide classification. This is the main technical contribution of the paper.\n\nOur work **directly addresses a persistent and underexplored weakness of current MLLMs**: their performance is limited in domains with insufficient or no training exposure. This issue continues to arise, as new tasks and data distributions emerge. There will always be scenarios where labeled training data is scarce or inaccessible due to cost, rarity, or privacy constraints. Existing methods provide no effective or practical solution for adapting MLLMs in such settings.\n\nOur work specifically targets these underperforming domains by enabling unsupervised prompt optimization without label reliance. Fine-grained classification serves as a prime example, but our method is **broadly applicable** to other datasets and tasks where the MLLM's training exposure is limited.\n\n\n**2. Results of better Gemini models**\n\n> **Reviewer’s concern**: It is noticeable that GPT-4o outperforms Gemini by a large margin across all benchmarks, while in my own experience, Gemini 2.0/2.5 performs much better than GPT on seeing image details. I would appreciate it if you could verify this.\n\n**Our Response**: We appreciate the reviewer’s comment and agree that Gemini 2.0/2.5 models have demonstrated strong image understanding capabilities in many scenarios. To further investigate this, we conducted additional experiments using Gemini 2.0 Flash, as shown in the Table below:\n\n**Table: Results on Gemini 2.0 Flash.**\n| CUB_vireo                   | Gemini 2.0 Flash   | StanfordDogs_terrier        | Gemini 2.0 Flash   |\n|:-------------|:-------------|:-------------|:-------------|\n| Vanilla zero-shot           | 64.27 ± 1.31       | Vanilla zero-shot           | 74.60 ± 0.47       |\n| Zero-shot with descriptions | 64.27 ± 1.65       | Zero-shot with descriptions | 74.12 ± 1.21       |\n| AutoSEP (ours)              | **69.05** ± 1.78   | AutoSEP (ours)              | **85.94** ± 1.05   |\n\nWhile we observe that Gemini 2.0 Flash significantly outperforms Gemini 1.5 Flash, our results do not reflect a performance advantage over GPT-4o on our benchmark tasks. Nonetheless, the key takeaway is that our AutoSEP method **consistently improves** the performance of Gemini 2.0 Flash across both datasets, reaffirming its effectiveness across different MLLMs. We will include these new results in our revised paper."}, {"review_id": "EQcUcsVCJ2", "reviewer": "Reviewer_Eaom", "summary": "The paper presents AutoSEP, a training-free approach to enhance MLLLM’s fine-grained image classification performance. The key idea is to use some unlabeled data to optimize the description prompt for classification. Empirical evaluation demonstrates performance improvement on multiple fine-grained classification datasets.", "questions": "1. \\[Line 106\\] The equation there should be numbered as well like (2).    \n2. \\[Line 128\\] Is \\<IMG\\> missing from the prompt?   \n3. The second question also applies to Figure 2  \n4. It’s better to specify the Gemini version (1.5 Flash) in Table 1\\.  \n5. Is there any data resampling or are all datasets balanced in class distribution? If not, the use of accuracy should be justified. The description in Section 4.1 is insufficient.   \n6. \\[Line 274\\] It’s great you address the perspective of computational complexity. It’s useful to comment on the compute for baselines as well.", "limitations": "See Weakness 4", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1. The problem setup of improving fine-grained image classification performance with black-box only access to MLLM is practical and useful.   \n2. It’s commonly observed that MLLM tend to under-perform for fine-grained classification, so the focus on fine-grained classification is also well motivated.   \n3. Leveraging the idea of contrastive learning for prompt optimization seems good.\n\nWeaknesses\n\n1. The focus on zero-shot classification is not well motivated given we typically have access to a small number of demonstration examples.  \n2. Given we have access to unlabeled data, it’s important to add a semi-supervised method as baseline as well. For example, we can use the MLLM to label those unlabeled data and use those cases as demonstration examples. The current baseline (4) only uses random labels.   \n3. \\[Line 57\\] It’s important to acknowledge there’s some previous work such as [https://arxiv.org/abs/2405.09798](https://arxiv.org/abs/2405.09798) showing adding unlabeled images directly to the context window boosts zero-shot classification performance. It’s important to add that as a baseline besides vanilla zero-shot. It’s similar to baseline (5) but without the prompt “\"The first m images show distinct types of {specie}.\"”  \n4. The possibility of data leakage is not discussed, especially given most datasets used are old.", "was_revised": true, "final_justification": "I've reviewed the rebuttal and comments from other reviewers. The two baselines added are convincing, so I've raised the score.", "rebuttal": "We really appreciate the reviewer for their detailed and valuable feedback and constructive suggestions. We address all of the concerns below.\n\n**1. The motivation of using limited unlabeled data**\n\n> **Reviewer’s concern**: The focus on zero-shot classification is not well motivated given we typically have access to a small number of demonstration examples.\n\n**Our Response**:\nWe appreciate the reviewer’s comment and would like to clarify our motivation and contribution. While it is true that a few labeled examples may sometimes be available, this is **not guaranteed** in many real-world scenarios, particularly in fine-grained domains (e.g., biodiversity, rare diseases, or emerging scientific categories), where **labeled data is expensive or infeasible to obtain**. Our method specifically addresses the *fully unsupervised setting*, where no labeled data is available, but a small number of unlabeled images can be collected. This setting is practically relevant and underexplored in the context of MLLMs.\n\nFor example, in nature science, researchers often collect unlabeled wildlife photos from camera traps but lack taxonomic annotations for rare species. Our method can be perfectly used in this case to increase the MLLM fine-grained image understanding and prediction precision.\n\n\n****\n**2 & 3. Adding new baselines**\n\n> **Reviewer’s concern**:\n> * Given we have access to unlabeled data, it’s important to add a semi-supervised method as baseline as well. For example, we can use the MLLM to label those unlabeled data and use those cases as demonstration examples. The current baseline (4) only uses random labels.\n> * It’s important to acknowledge there’s some previous work showing adding unlabeled images directly to the context window boosts zero-shot classification performance. It’s similar to baseline (5) but without the prompt \"The first m images show distinct types of {specie}.\"\n\n**Our Response**:\nWe sincerely thank the reviewer for their insightful suggestions regarding two new baseline comparisons. We list the results of the newly added baselines in the Table below.\n\n**Table: Results of new baselines.**\n| CUB_cuckoo      | Gemini 1.5 Flash | GPT-4o       | CUB_oriole      | Gemini 1.5 Flash | GPT-4o       |\n|:-------------|:-------------|:-------------|:-------------|:-------------|:-------------|\n| Few-shot with MLLM labels | 53.41 ± 5.31     | 61.22 ± 2.82      | Few-shot with MLLM labels | 57.75 ± 4.42     | 64.94 ± 7.96 |\n| Many-Shot ICL [1]         | 48.05 ± 6.84     | 52.68 ± 9.27      | Many-Shot ICL [1]         | **59.78** ± 6.60 | 71.46 ± 8.75 |\n| AutoSEP (ours)            | **61.22** ± 2.30 | **75.12** ± 1.83  | AutoSEP (ours)            | 58.20 ± 0.88     | **75.96** ± 0.90 |\n| **CUB_vireo**             |  |               | **iNat_butterfly**      |\n| Few-shot with MLLM labels | 44.94 ± 8.56     | 72.36 ± 10.5      |Few-shot with MLLM labels  | 51.43 ± 3.26     | 82.57 ± 14.3 |\n| Many-Shot ICL [1]         | 56.18 ± 7.95     | 68.54 ± 13.6      |Many-Shot ICL [1]          | 55.71 ± 1.56     | 72.29 ± 12.2 |\n| AutoSEP (ours)            | **59.18** ± 0.53 | **87.42** ± 1.10  |AutoSEP (ours)             | **66.57** ± 1.42 | **82.71** ± 2.43 |\n| **iNat_lupine**           |  |               |**StanfordDogs_terrier**      |\n| Few-shot with MLLM labels | 60.22 ± 11.4     | 69.33 ± 3.89      |Few-shot with MLLM labels  | 66.67 ± 10.3     | 87.56 ± 3.47 |\n| Many-Shot ICL [1]         | 63.47 ± 7.71     | 74.00 ± 5.54      |Many-Shot ICL [1]          | 65.87 ± 4.04     | 79.78 ± 9.04 |\n| AutoSEP (ours)            | **66.15** ± 1.47 | **76.60** ± 1.40  |AutoSEP (ours)             | **69.28** ± 0.97 | **92.18** ± 1.09 |\n\nWhile these two baselines are stronger than baseline (4) and (5) in our original paper, our proposed method still **consistently outperforms** them across multiple experiment settings. This further demonstrates the advantage of our self-enhancing prompt learning framework.\n\nWe will add these additional baselines and results to the revised paper to strengthen the effectiveness of our work.\n\n\n**Reference:**\n\n[1] Jiang, Yixing, et al. \"Many-shot in-context learning in multimodal foundation models.\" arXiv preprint arXiv:2405.09798 (2024).\n\n\n****\n**4. Adding new baselines**\n\n> **Reviewer’s concern**: The possibility of data leakage is not discussed, especially given most datasets used are old.\n\n**Our Response**:\nWe appreciate the reviewer’s concern regarding the possibility of data leakage. In our experiment, we specifically selected fine-grained classification tasks where the **MLLM’s zero-shot performance is low** and unsatisfying. This underperformance suggests that the model can’t do well in recognizing these fine-grained images and subtle visual distinctions​​, indicating a lack of prior exposure to the specific label sets or task formulations. While we acknowledge that it’s possible to have data leakage in the MLLMs, and complete verification of data leakage is challenging. However, the poor baseline performance provides empirical evidence that the model has not memorized these tasks. Moreover, our work focuses on improving zero-shot performance using only unlabeled data, regardless of whether the model has seen related data before.\n\n\n****\n**5. Addressing reviewer’s questions**\n\n**Q1 & Q4**:\n**Answer**: Thanks to the reviewer for pointing out these helpful suggestions. We will revise the paper to include the missing equation number for consistency and clarify the Gemini version (1.5 Flash) in Table 1.\n\n**Q2 & Q3**: [Line 128] Is `<IMG>` missing from the prompt?\n**Answer**: In our setup, we do send the image together with the prompt to the MLLM, but we do not explicitly include an `<IMG>` token in the text prompt. This is because models like Gemini 1.5 Flash handle multimodal inputs through dedicated APIs or structured inputs, where the image is passed as a separate input alongside the text, not embedded in the textual prompt.\n\n**Q5**: Is there any data resampling or are all datasets balanced in class distribution?\n**Answer**: All the results reported are evaluated on datasets that are **balanced** in class distribution. We will clarify this detail explicitly in Section 4.1 in the revised paper.\n\n**Q6**: [Line 274] It’s great you address the perspective of computational complexity. It’s useful to comment on the compute for baselines as well.\n**Answer**:\nFor optimization-free baselines, the computational cost is relatively low, since they do not require iterative prompt refinement. For optimization-based baselines, we used the same experimental setup as our proposed method, resulting in comparable computational complexity. More specifically, optimization with random labels takes approximately $O(2(1+n)·bl)=O(40n)$ MLLM queries per iteration, as it involves only a single classification per image. Optimization with majority vote takes approximately $O(2(1+n)·bl+cn)=O(45n)$ queries per iteration, with $c=5$ denoting the number of majority votes collected for updates in each iteration. SPO is more lightweight, requiring only $O(2n)$ queries per iteration, since it only compares with the previous prompt and does not rely on several candidate prompts. We will include these details in the revised paper to provide a clearer understanding of the computational costs."}, {"review_id": "nFZ8IIRcgi", "reviewer": "Reviewer_B2au", "summary": "This paper proposes AutoSEP, an automatic self-enhancing prompt learning framework that iteratively refines the description prompt to generate more discriminative descriptions for fine-grained zero-shot image classification.", "questions": "1. Is this method entirely training-free, with all steps performed at test time? More specifically, does each image have its own prompt that is iteratively optimized to generate a more discriminative description? If so, since the prompt is continuously changing, would this introduce instability or inconsistency during inference?\n2. The iterative prompt optimization appears to be potentially time-consuming for each image. How long does the inference process take in practice? It would be helpful if the authors could provide details on the inference pipeline and report the average inference time per image.\n\nOther questions and concerns are outlined in the “Weaknesses” section above. If the authors can address these issues satisfactorily, I would be open to revising my score.", "limitations": "Yes, but the paper should more explicitly address the limitations, as outlined in the Weaknesses and Questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The idea of self-enhancing prompt learning in a fully unsupervised setting is novel and effectively enriches instance-level semantic information.  \n2. The paper is well-structured and clearly written, making it easy to understand the methodology and contributions.  \n3. The experiments are comprehensive and demonstrate the effectiveness of the proposed framework across multiple fine-grained benchmarks.\n\nWeakness:\n1. Equation (3) is unclear. It is not obvious how the random variable $Z$ affects the evaluation of $V(x_i, x_j)$, since $Z$ is not explicitly included in the function's notation. This could be misleading to readers unfamiliar with the design.\n2. The definition of $G(t_a, t_b)$ is ambiguous. It would be helpful to clarify what $G$ represents and what its output is — for example, whether it is a textual prompt template or a more structured representation.\n3. It would be helpful to clarify the function of $M$ and what $M(x_i, G(t_i, t_j)) = \\text{``first''}$ actually means. Does ``first'' always refer to $t_i$, or is the order randomized depending on $Z$? Without clarification, it is difficult to interpret how correctness is determined and how $V(x_i, x_j)$ reflects model performance.\n4. During the sampling process in Eq.~(3), what happens if the model encounters visually similar images from the same class? Such cases may introduce ambiguity and make it difficult for the model to consistently distinguish between the corresponding descriptions, potentially affecting the reliability of the evaluation.\n5. In Section 3.3.2, it is unclear how to get $J_{error}$. How to identify that the generated descriptions failed to correctly distinguish between images? Is it manually defined?", "was_revised": true, "final_justification": "The supplementary explanations and ablation results have effectively addressed the concerns I raised earlier.r. Given these improvements, I am inclined to update my overall score accordingly.", "rebuttal": "We thank the reviewer for their thorough review and feedback. We’ll answer the questions one by one.\n\n**1, 2, 3, & 5. Regarding algorithm notations**\n\n**Our Response**:\nWe thank the reviewer for the detailed comments on the mathematical notations. We clarify each point regarding algorithm notations in detail below.\n\n* **Clarification of $V(x_i, x_j)$**: $V(x_i, x_j)$ is a binary correctness indicator that reflects whether the MLLM can correctly match image $x_i$ to its corresponding description $t_i$ over an incorrect alternative $t_j$. While the variable $Z \\sim \\text{Bernoulli}(0.5)$ is not explicitly included in the notation, it determines the random ordering of the two descriptions in the binary-choice prompt and thus influences the evaluation.\n* **Definition of $G(t_a, t_b)$**: $G(t_a, t_b)$ is a prompt template that formats two descriptions $t_a$ and $t_b$ into a binary-choice question posed to the MLLM, asking the MLLM whether the first description $t_a$ or the second description $t_b$ correctly describes the image. The MLLM is then expected to answer with either “first” or “second”.\n* **Role of $M$ and interpretation of $M(x_i, G(t_i, t_j)) = \\text{“first”}$**: We define the function of $M$ in Sec. 3.1, whose input is an image and a prompt, and output is the MLLM response. Specifically, $M(x_i, G(t_i, t_j))$ is the MLLM response (either \"first\" or \"second\") when shown image $x_i$ and prompt $G(t_i, t_j)$. To reduce positional bias, the order of $t_i$ and $t_j$ is randomly determined using $Z \\sim \\text{Bernoulli}(0.5)$. If $Z=0$, we prompt with $G(t_i, t_j)$, and the correct answer is “first”; if $Z=1$, we prompt with $G(t_j, t_i)$, and the correct answer is “second”. We use an indicator function to return $1$ if the model selects the correct description (i.e., matches the image to its ground-truth description), and 0 otherwise.\n* **Clarification of $J_{error}$**: $J_{error}$ denotes the set of image pairs where the MLLM fails to choose the correct description. That is, for a given pair $(x_i, x_j)$, if $V(x_i, x_j) = 0$, then $(x_i, x_j)\\in J_{error}$.\n\n\n****\n**4. Concern of sampling process**\n\n> **Reviewer’s concern**: During the sampling process in Eq.~(3), what happens if the model encounters visually similar images from the same class? Such cases may introduce ambiguity and make it difficult for the model to consistently distinguish between the corresponding descriptions, potentially affecting the reliability of the evaluation.\n\n**Our Response**:\nThe goal of our description generation process is to **elicit detailed, fine-grained understanding of each image**, instead of enforcing class-level discrimination. This image-specific descriptive richness is then used to guide the classification process in a separate classification prompt. In this context, encountering visually similar images--even from the same class--is not a flaw but an opportunity: it encourages the model to uncover subtle, discriminative visual cues that may otherwise be overlooked.\nIn traditional self-supervised learning methods, similar images from the same class are often sampled together to learn meaningful feature representations through contrastively learning. Our framework is inspired by this idea and pushes the model to verbalize these subtle differences, which ultimately improves the downstream classification.\n\nBesides, we conducted ablation studies to see how the sampling process affects the final results. We conducted our AutoSEP with only sampling different-class images or only sampling same-class images for the instance-level classification. The results are in the Table below. Same-class and different-class sampling both don’t affect the final results much.\n\n**Table: Results of different sampling processes.**\n| CUB_cuckoo                                     | Gemini 1.5 Flash |\n|:------------- |:-------------:|\n| AutoSEP with different-class image sampling    | 60.98 ± 3.59     |\n| AutoSEP with same-class image sampling         | 61.58 ± 3.11     |\n| AutoSEP with random image sampling (ours)      | 61.22 ± 2.30     |\n\n\n****\n**5. Addressing reviewer’s questions**\n\n> **Q1**: Is this method entirely training-free, with all steps performed at test time? More specifically, does each image have its own prompt that is iteratively optimized to generate a more discriminative description? If so, since the prompt is continuously changing, would this introduce instability or inconsistency during inference?\n\n**Answer 1**: To clarify, our method is **entirely training-free** in the sense that it does not require any finetuning of the model parameters. We only require black-box access to the MLLM, and perform optimization externally through prompt. **The prompt is not optimized per image.** Instead, we iteratively optimize a single description prompt using a small set of unlabeled images. This optimized prompt is then used consistently across all test-time images to generate improved image descriptions.\n\n> **Q2**: The iterative prompt optimization appears to be potentially time-consuming for each image. How long does the inference process take in practice? It would be helpful if the authors could provide details on the inference pipeline and report the average inference time per image.\n\n**Answer 2**: One optimization iteration on 60 unlabeled images using Gemini 1.5 Flash takes approximately 12 minutes on average, as reported in Sec. 4.3. This iterative prompt optimization is performed **once offline**, not per image. After the prompt is optimized, **inference is highly efficient**--each image requires only a single forward pass for description generation and one for prediction. As a result, the average inference time per image is comparable to standard zero-shot inference, with negligible additional overhead introduced by our method."}, {"review_id": "vAwXyub8B5", "reviewer": "Reviewer_AiFG", "summary": "For zero-shot fine-grained image classification with black-box multi-modal large language models (MLLM), this paper proposes to use an automatic prompt learning framework. It uses the MLLM to generate a description based on the image and a regular prompt and refines it with unlabeled data to generate a “distinctive descriptions for each unlabeled image”. It does this iteratively by, finally, giving the MLLM a binary choice prompt to select between two generated descriptions, one for the image in question and the other for another random image in the dataset.", "questions": "1.\tImprove the comparison of results as described above. \n2.\tJustify having access to the parent class name and subset creation. \n3.\tJustify the query complexity. \nI would be open to revising my rating if these concerns are addressed.", "limitations": "MLLM should be knowledgeable of the attributes defining the fine-grained classes. \nThe paper clearly lists two other limitations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tGetting an MLLM to generate prompts that increasingly focus on fine-grained details in this black-box MLLM fine-grained classification setting. \n2.\tThe method’s ability to use instance level classification signal for unsupervised prompt optimization. \n3.\tThe paper is, in general, clear. \n\nWeaknesses:\n\n1.\tThe initial prompts use the name of the parent class in the prompt, e.g., “the bird” in “Describe the bird in the given image in detail, focusing on highly distinctive attributes that are typical to this bird. Ignore the background or other information.”  While each dataset is limited to the parent class \"bird,\" providing this information explicitly to an MLLM can be an unfair advantage.\n2.\tThe number of times the MLLM gets queried is high. Initial prompt, iteration (e.g., 6), unlabeled image samples, prompts, prompts retained, reflection, modification, all involve prompting the MLLM. Computational complexity (query complexity), as described in lines 274 onwards, is high. \n3.\tThe paper constructs fine-grained classification tasks on subsets of categories in these datasets. Provide justification (from literature or otherwise) for this. Provide results similar to, e.g., Saha et al., Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions, CVPR 2024.\n4.\tThe experiments have been done with large MLLMs. It would be beneficial to know the performance of smaller variants. \n5.\tThe diversity score (Fig. 3c) is not clearly defined. Can’t the standard metrics be used? \n6.\tA major concern is the absence of comparison of results with existing work. The comparisons on variants of experiments (although logical) are crafted in this paper itself.", "was_revised": true, "final_justification": "The rebuttal successfully addresses my comments and, as far as I can tell, those of the other reviewers as well.", "rebuttal": "We’re very thankful to the reviewer’s detailed and valuable feedback. We answer all of the concerns and questions below.\n\n**1. Parent class name in initial prompt**\n> **Reviewer’s concern**: The initial prompts use the name of the parent class in the prompt, e.g., “the bird”. While each dataset is limited to the parent class, providing this information explicitly to an MLLM can be an unfair advantage.\n\n**Our Response**:\nWe thank the reviewer for bringing up this concern. To evaluate the impact of including the parent class name in the initial prompt, we replaced “the bird” with a more generic term, “the object,” and re-ran the experiments. The results are presented in the Table below.\n\n**Table: Results of different initial prompts.**\n| CUB_cuckoo | Gemini 1.5 Flash | Qwen2-VL-72B-Instruct |\n|:- |:-:|:-:|\n| AutoSEP with “the object” in initial prompt | 60.82 ± 3.43 | 63.72 ± 1.01  |\n| AutoSEP with “the bird” in initial prompt (ours)| 61.22 ± 2.30 | 63.41 ± 2.77  |\n\nWe observe that this change has minimal effect on the final performance, suggesting that explicitly including the parent class name in the initial prompt does not provide a significant advantage in our setting.\n\n****\n**2. Computational cost**\n> **Reviewer’s concern**: The number of times the MLLM gets queried is high. Computational complexity (query complexity) is high.\n\n**Our Response**:\nWhile our method involves multiple MLLM queries during the prompt optimization phase, this process is performed only once offline. 12 minutes per iteration with 60 examples using Gemini 1.5 Flash is **a modest runtime given the consistent and substantial performance gains** across diverse fine-grained classification tasks. Our method is practical, scalable, and requires no model fine-tuning or internal access, making it highly accessible and model-agnostic. It is also data-efficient, relying only on a small number of unlabeled examples. Compared to methods that require retraining, our approach offers a strong balance of efficiency and effectiveness, with minimal runtime overhead during inference (one forward pass per image for description generation and one for prediction).\n\n****\n**3. Data subset creation**\n> **Reviewer’s concern**: The paper constructs fine-grained classification tasks on subsets of categories in these datasets. Provide justification (from literature or otherwise) for this. Provide results similar to, e.g., Saha et al., Improved Zero-Shot Classification by Adapting VLMs with Text Descriptions.\n\n**Our Response**:\nWe thank the reviewer for raising this important point. Our decision to construct fine-grained classification tasks using subsets of categories is motivated by both practical considerations in the MLLM setting and some prior work.\n\nIn these fine-grained datasets, there’re over a hundred categories. Unlike CLIP-style models, which computes similarity scores across hundreds of categories to make predictions, asking an MLLM to choose from over a hundred fine-grained labels in a single prompt is rarely done and less reliable with the results. Instead, we focus on **challenging subsets of categories where zero-shot performance is low**, making them meaningful benchmarks for evaluating improvements. This allows us to study performance gains in hard fine-grained settings, rather than on categories that are easily separable by existing MLLMs.\n\nThis design is **consistent with prior work**. For example,\n* LLM-Mutate[1] used 6 species of Lichen (fungi), 5 species of Wrasse (fish) for classification.\n* Finedefics[2] constructed their dataset using 4 candidate options per image to test fine-grained classification performance.\n\n**Reference:**\n\n[1] Chiquier, Mia, Utkarsh Mall, and Carl Vondrick. \"Evolving interpretable visual classifiers with large language models.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.\n\n[2] He, Hulingxiao, et al. \"Analyzing and boosting the power of fine-grained visual recognition for multi-modal large language models.\" arXiv preprint arXiv:2501.15140 (2025).\n\n****\n**4. Results on smaller models.**\n> **Reviewer’s concern**: The experiments have been done with large MLLMs. It would be beneficial to know the performance of smaller variants.\n\n**Our Response**:\nWe appreciate this insightful suggestion to explore on smaller models. We conducted additional experiments using **Qwen2-VL-7B-Instruct**. Since automatic prompt optimization requires stronger agents for reflection and modification, we explored a hybrid setup where Gemini 1.5 Flash was used as the Reflect and Modify operators and image description generation, while the Qwen 7B model was used for instance-level classification and evaluation. This is a common setting to use a stronger model for reflection[3][4][5]. The results below indicate that our method also leads to **significant improvements** on Qwen2-VL-7B-Instruct.\n\n**Table: Results on Qwen2-VL-7B-Instruct.**\n| CUB_cuckoo | Qwen2-VL-7B-Instruct | CUB_oriole | Qwen2-VL-7B-Instruct |\n|:- |:-:|:-|:-:|\n| Vanilla zero-shot | 40.24 ± 3.43     |Vanilla zero-shot  | 39.78 ± 7.41  |\n| Zero-shot with descriptions| 39.27 ± 2.68 | Zero-shot with descriptions | 50.45 ± 1.56 |\n| AutoSEP (ours)  | **61.79** ± 2.31   |AutoSEP (ours)  | **54.02** ± 1.66 |\n\nThese results demonstrate that our method can effectively enhance fine-grained classification performance even for smaller models when paired with a stronger MLLM as the optimization engine. We will include these results to our revised paper to strengthen the comprehensiveness of our work.\n\n**Reference:**\n\n[3] Wang, Xinyuan, et al. \"Promptagent: Strategic planning with language models enables expert-level prompt optimization.\" arXiv preprint arXiv:2310.16427 (2023).\n\n[4] Xiang, Jinyu, et al. \"Self-supervised prompt optimization.\" arXiv preprint arXiv:2502.06855 (2025).\n\n[5] Murthy, Rithesh, et al. \"Promptomatix: An Automatic Prompt Optimization Framework for Large Language Models.\" arXiv preprint arXiv:2507.14241 (2025).\n\n****\n**5. Regarding diversity compute**\n> **Reviewer’s concern**: The diversity score (Fig. 3c) is not clearly defined. Can’t the standard metrics be used?\n\n**Our Response**:\nWe appreciate the reviewer’s feedback regarding the clarification of diversity score. Our goal was to quantify how much new semantic content is introduced throughout the optimization, which is important for understanding the evolution of prompts. To that end, our diversity score is a custom metric designed to capture two aspects:\n* Semantic richness, measured by counting the number of unique lemmas of content words (nouns, verbs, adjectives) after stop-word removal;\n* Uniqueness across prompts, defined as the number of words that appear only in a specific prompt and not in others.\n\nBoth counts are normalized to form the final diversity score. While standard metrics such as lexical diversity exist, they do not directly capture inter-prompt uniqueness, which is central to our goal of analyzing how the prompts diversify during optimization.\n\n****\n**6. Comparing existing work**\n> **Reviewer’s concern**: A major concern is the absence of comparison of results with existing work. The comparisons on variants of experiments (although logical) are crafted in this paper itself.\n\n**Our Response**:\nWe thank the reviewer for this observation. Our paper addresses **a novel problem setting**--self-enhancing prompt optimization for fine-grained image classification without labels using MLLMs--which, to the best of our knowledge, has **not been thoroughly explored in prior work**. As a result, direct comparisons with existing methods are limited.\n\nWe try to find some related methods. We applied the **SPO**[4] method from the text domain in our setting, whose results are reported in paper Table 1. We additionally applied the **Many-shot In-Context Learning (ICL)**[6] method to our setting by adding unlabeled images directly to the context window for prediction. We also add another baseline according to reviewer Eaom to use the MLLM to label some unlabeled data as demonstration examples. The results in the Table below show that our method outperforms these baselines in most of the settings.\n\n**Table: Results of new baselines.**\n| CUB_cuckoo  | Gemini 1.5 Flash | GPT-4o  | CUB_oriole | Gemini 1.5 Flash | GPT-4o   |\n|:-|:-|:-|:-|:-|:-|\n| Few-shot with MLLM labels | 53.41 ± 5.31 | 61.22 ± 2.82 | Few-shot with MLLM labels | 57.75 ± 4.42     | 64.94 ± 7.96 |\n| Many-Shot ICL [6]  | 48.05 ± 6.84     | 52.68 ± 9.27  | Many-Shot ICL [6]  | **59.78** ± 6.60 | 71.46 ± 8.75 |\n| AutoSEP (ours)  | **61.22** ± 2.30 | **75.12** ± 1.83  | AutoSEP (ours)  | 58.20 ± 0.88  | **75.96** ± 0.90 |\n| **CUB_vireo**  |  |   | **iNat_butterfly**  |\n| Few-shot with MLLM labels | 44.94 ± 8.56     | 72.36 ± 10.5      |Few-shot with MLLM labels  | 51.43 ± 3.26  | 82.57 ± 14.3 |\n| Many-Shot ICL [6]   | 56.18 ± 7.95     | 68.54 ± 13.6      |Many-Shot ICL [6]  | 55.71 ± 1.56     | 72.29 ± 12.2 |\n| AutoSEP (ours)  | **59.18** ± 0.53 | **87.42** ± 1.10  |AutoSEP (ours) | **66.57** ± 1.42 | **82.71** ± 2.43 |\n| **iNat_lupine**   |  |   |**StanfordDogs_terrier**    |\n| Few-shot with MLLM labels | 60.22 ± 11.4 | 69.33 ± 3.89  |Few-shot with MLLM labels  | 66.67 ± 10.3     | 87.56 ± 3.47 |\n| Many-Shot ICL [6]  | 63.47 ± 7.71  | 74.00 ± 5.54      |Many-Shot ICL [6]  | 65.87 ± 4.04     | 79.78 ± 9.04 |\n| AutoSEP (ours)  | **66.15** ± 1.47 | **76.60** ± 1.40  |AutoSEP (ours)  | **69.28** ± 0.97 | **92.18** ± 1.09 |\n\nWe believe these efforts provide a fair and informative benchmark, demonstrating the effectiveness and novelty of our method. We will add them to our revised paper.\n\n**Reference:**\n\n[4] Xiang, Jinyu, et al. \"Self-supervised prompt optimization.\" arXiv preprint arXiv:2502.06855 (2025).\n\n[6] Jiang, Yixing, et al. \"Many-shot in-context learning in multimodal foundation models.\" arXiv preprint arXiv:2405.09798 (2024)."}]

Author rebuttal:

Meta-review: The paper addresses the problem of fine-grained recognition in zero-shot setting for multimodal LLMs. The core idea is to leverage unlabelled data to iteratively optimize the description-generation prompt. 
Some notable strengths of the paper include:
- easy to read and well-organized paper
- proposed method is logically clear and elegant combination of existing components
- motivation for improving MLLM zero-shot performance in fine-grained scenarios is clear
- the ability of the method to use instance-level classification signal

Some important concerns of the paper were:
- limited novelty as it appears a combination of existing methods with only scope of fine-grained classification improvement
- no mention of semi-supervised baselines as the method leverage unlabelled data
- initial prompts use parent class information providing unfair advantage
- results with smaller variants of MLLMs
- comparison with some existing relevant works

After the post-rebuttal discussion period, reviewers acknowledged that their major concerns have been resolved. These include explanation on novelty, results on semi-supervised baselines, results on different sampling processes and smaller variants of MLLMs. Therefore, the decision is to recommend the acceptance and authors are recommended to include important reviewer's comments in the camera-ready version.
Decision: Accept (poster)