DiVE-k: DIFFERENTIAL VISUAL REASONING FOR FINE-GRAINED IMAGE RECOGNITION
Authors: Raja Kumar
Arka Sadhu
Ram Nevatia
Venue: ICLR 2026 Poster (2026)
Keywords: zero-shot image classification
visual reasoning
vision-language model
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: DiVE-k uses a model's own top-k predictions as MCQ options to convert open-ended recognition into a multiple-choice task, training using GRPO to elicit differential reasoning among options, boosting fine-grained visual recognition generalization.

Abstract: Large Vision Language Models (LVLMs) possess extensive text knowledge but struggle to utilize this knowledge for fine-grained image recognition, often failing to differentiate between visually similar categories. Existing fine-tuning methods using Reinforcement Learning (RL) with exact-match reward signals are often brittle, encourage memorization of training categories, and fail to elicit differential reasoning needed for generalization to unseen classes. To address this, we propose $\textbf{DiVE-k}$, $\textbf{Di}$fferential $\textbf{V}$isual r$\textbf{E}$asoning using top-$\textbf{k}$ generations, framework that leverages model's own top-k predictions as a training signal. 
For each training image, DiVE-k creates a multiple-choice question from the model's top-k outputs and uses RL to train the model to select the correct answer. This approach requires the model to perform fine-grained differential reasoning among plausible options and provides a simple, verifiable reward signal that mitigates memorization and improves generalization. 
Experiments on five standard fine-grained datasets show that our method significantly outperforms existing approaches. 
In the standard base-to-novel generalization setting, DiVE-k surpasses the QWEN2.5-VL-7B and ViRFT by 10.04% and 6.16% on the Harmonic Mean metric, respectively. Further experiments show similar gains in mixed-domain and few-shot scenarios.

Published as a conference paper at ICLR 2026
DIVE-K: DIFFERENTIAL VISUAL REASONING FOR FINE-GRAINEL IMAGE RECOGNITION
Raja Kumar', Arka Sadhu², Ram N University of Southern California, Meta
kumarrajeusce edu neva http:///ww.aaaaaaaooo/ooo/o///ta
ABSTRACT
Large Vision Language Models (LVLMs) possess extensive text knowledge but struggle to utilize this knowledge for fine-grained image recognition, often failing to differentiate between visually similar categories. Existing fine-tuning methods using Reinforcement Learning (RL) with exact string-match reward are often brit- tle, encourage memorization of training categories, and fail to elicit differential reasoning needed for generalization to unseen classes. To address this, we pro- pose DiVE-k, Differential Visual rEasoning using top-k generations, framework that leverages model's s own top-k predictions as a training signal. For each training image, DiVE-k creates a multiple-choice question from the model's top-k outputs and uses RL to train the model to select the correct answer. This approach requires the model to perform fine-grained differential reasoning among plausible options and provides simple, verifiable reward signal that mitigates memorization and improves generalization. Experiments on five standard fine-grained datasets show that our method significantly outperforms existing approaches. In the standard base-to-novel generalization setting, DiVE-k surpasses the QWEN2.5-VL- and ViRFT by 10.04% and 6. 6.16% on the Harmonic Mean metric, respectively. Further experiments show similar gains in mixed-domait and few-shot scenarios.
INTRODUCTION
We explore the task of zero-shot fine-grained image recognition, as visual reasoning task building on available Large Vision Language Models (LVL.Ms). Such capabilities are crucial for the gener- alization of vision systems. early zero-shot image recognition works, such as in CLIP [Radford ctt al. 2021), the visual embedding from image is matched against the text embedding of class names to determine the most likely label. LVLMs, such as QWEN2-WE VL (Wang et al.) 2024), contain Large Language Model (LLM) in themselves and able to their vast language knowledge with unified multimodal pre-training to achieve impressive capabilities in zero-shot recognition. However, the accuracy for fine-grained recognition is limited. We aim to improve accuracy by fine- tuning on a subset of categories of a new dataset (called the "base" set) and test on "novel" categories for which no training examples are seen. This setting common (Zhou et al. 2022b) and relevant for adapting models to new domains with limited training data.
Our approach is based on two key observations. First, the base model exhibits high variance across its PassēK performance (Figure Ip): the correct label often appears among the K sampled re- sponse, yet fails to get correct as Pass@ This indicates possible over-reliance on coarse, salient attributes shared by related categories and may benefit from a fine-grained, differential reasoning to separate semantically similar categories. The second observation that the LVL.Ms actually contain detailed knowledge about the parts and attributes of the base and novel categories which could be used for detailed, differential analysis.
Inspired by the success of Chain of Thought (CoT) reasoning (eei et al. 2022) using Reinforcement Learning (RL) (Guo et al.] 2025) for pure language tasks such as mathematics and coding (Shao et al. 2024b Jiang et al. 2023), ViRFT (Liu et al. 2025) extended this idea to vision tasks such as image classification. The key idea in Vs to construct a verifiable reward (Lambert et al. 2024
for image classification which allows visual reasoning RL training via GRPO (Shao al.
2024b

---

Published as a conference paper at ICLR 2026
Prompt:
image containing bird. Please identily the species of the bird based on the image'
Input
Sone rotel relving
genrto slant cant make fed pradition
Reasoning: The bird in the Image has small size, slender body, and pointed beak, which charscteristic features sparrows, The hird's posfure and the setting suggest could he tree sparrow. which known for its similor oppearance and habitot preferences. free Sparron
QWEN2.5-VL-73
Pass@kk accuracy QWENZ.5-VV
top-k as Options: The bird in the image has small size, slender body, and a short, conical beak which characleristic of sparrowa. Basedd
"chipping sparrow", "feld sparrow" "tree features, sparrow"; D. "clay-colored sparrow.
likely sparrows
Ours
white threst, which is cherscteristic of field typically has streaked appearence its back and darker crown. The tree sporrow usually has a. more distinct black cap and reddish-broww bresst. The doyy-colored sparrow is smaller and has more throat. Therefore, most likely bird in image field sparrow. Final Answer: field sparrow
Differential Ressoning: The bird a light brown coloration
Spph optiant
sparrow: The chipping
compan model
zerfrm diffrentie
uniform coloration without
(b) top-k options elictts differential reasoning
Figure I: For fine- seee-grrind image recognition task, most salient visual attributes are often insufficient to identify the correct category its common among similar categories. (a) This leads to signifi- cant performance gap in model's P Pass@1 and P Pass 820 accuracy. (b) differential reasoning can help indicate out the key visual attributes that can help distinguish among similar categories. Base model fails to use such discriminative features relying only on prominent visual features. We solve this by using top-k as options (the most likely categories base model confuses it for) and utilizes model's ' text knowledge to resolve this confusion using differential reasoning (highlighted in green).
However, their verifiable reward obtained through exact string match between the category name and the model's final answer is brittle: (i) requiring ad hoc string post-processing and model responses may use scientific or common names that cannot be validated by string matching: (ii) encourages memorization of the training category names (Section 4.2.T); and (iii) fails to incentivize attribute- level, discriminative reasoning (Figure [p). This leads to weak base to novel generalization and a tendency to ignore useful text knowledge when visual evidences alone are ambiguous.
To overcome these deficiencies, we propose DiVE-k framework (Differential Visual rEasoning us- ing top-k generations) that treats base model's top-k generations, obtained via K rollouts, training primitive that enables diffcrential visual reasoning. For each training image, we treat the top-k out- explicit hypotheses set and train the model using RL to resolve this set
puts of the base model by selecting the correct element.
We formulate this as Multiple- Hluttple-Chooie-- the model's own distribution. This yields two advantages (i) differential reasoning: presenting the model with its own top predictions options compels it to move beyond simple pattern recognition. Thus, the model learns to engage in fine-grained reasoning, identifying the specific attributes that differentiate the correct answer from other plausible alternatives, illustrated Figure (ii) easily verifiable reward signal: the reward for a correct prediction becomes trivially verifiable as model simply has to select the correct index from the given options. This contrasts with methods such as ViRFT, which rely on an exact string match. Our approach further mitigates the category name memorization issue and leads to better generalization on unseen categories.
Cwwi-hhhooieceeeeeeeee Question (MCQ) interface, using top-k as options leveraging
Experiments on five standard fine-grained image classification datasets show that DiVE-k outper- forms existing methods by a significant margin in two distinct zero-shot settings. For standard base-to-novel generalization, 10.04% and 6.16% on the Harmonic Mean (HM), respectively. This performance gain also extends to mixed-domain zero-shot base-to-novel generalization setting, where achieve improvements of 9.03% against QWEN2.5-VL. VL-7B and 4.02% against ViRFT. Further, we observe an average im- provement of 7.73% compared to ViRFT on 4-shot image classification.
method surpasses pre-trained QWEN2.5-VL-7B and ViRFT by
In summary, our main contributions are: (i) we propose DiVE-k framework which uses the top-k generations of base model training signal for fine-grained image classification, (ii) we demon-
2

---

Published as a conference paper at ICLR 2026
strate the benefits of using MCQ to distinguish among semantically similar categories, and (iii) we show improved performance on multiple fine- aeee-rained image classification datasets with detailed ablation studies.
2 PRIOR WORK
Zero-shot fine-grained image classification Vision Language Models (VLMs) (Radford et al.
2021
Li et al. 2022] Tschannen et al., [025] Yuan et al. 2021 use the idea of aligning image
with text to achieve zero-shot learning (Lampert al. 2013
Socher et al.
2013 Wang al. 2018
Zhang et al.] 2017). Although very competent for image-text alignment, these models have lim-
ited world knowledge unlike L.LMs (Radford et al. 2019 Brown al. 2020 Touvron al. 2023
Chowdhery et al., 2023). An early line of research proposes the idea of prompt learning (Zhou et al.
2022a] Khattak et al.) 2023b) where a prompt vector is learned for text prompt's context words. Zheng et al. (2024c) uses LLM's knowledge to learn the prompt vector. Another approach to bridge the knowledge gap by combining the perceptual strengths of VLMs with the linguistic abilities of
LLM (Esfandiarpoor & Bach, 2023, Menon & Vondrick, 2022, Pratt et al. [023]] Zeng et al. [0022] Novack et al., 2023] Roth et al., 2023). Our work in part inspired by FuDD (Esfandiarpoor & Bach; 2023) that uses multi-stage reasoning by using LLM knowledge to find pairwise discrimi-
native features to complement VLM. However, FuDD generates a fixed set of text prompts offline, limiting its ability to adapt its reasoning strategy to the specific difficulty of each input. Other related work tries to use LLM as a tool for reasoning through programming and language reasoner (Chen) et al. 2023] Gupta & Kembhavi] 2023 Zhang et al.) [023] Surís al., 2023). While these methods improved model performance, the separation between the vision and language modules creates a bottleneck, inhibiting seamless and integrated reasoning across modalities (Liu et 2023 2024).
Recent LVLMs (Bai et al. 2025: Hurst et al. 2024 Team et al. 2023 2025a) have excellent visual understanding, such as VQA (Shao et al.J 2024a), combining visual encoding directly into a LLM architecture. This opens up the possibility of joint vision and text reasoning capability (Team et al.,
2025c Bai et al. 2025 Team et al., 2025a). Additionally, recent success of RL for reasoning on maths and coding tasks (Jaech et al., 2024 Guo et al.) 2025 Team et al. (025b) have transformed the post- post-training reasoning research and there is growing interest to extend these ideas LVLMs.
Enhancing Reasoning with Reinforcement Learning. Building upon the seminal work of In- context learning (Brown et al. 2020) and CoT (Wei et al. 2022), the field has moved beyond static prompting by applying RI. to fine-tune these reasoning processes (Jaech et al.] [024]] Guo et al.) 2025 Team et al. 2025b). By treating the generation of a reasoning chain as a sequential decision- making problem. RL-based methods can train models to produce more accurate and explainable solutions. Recent breakthrough in DeepSeek-RI (Shao et al.) 2024b) showed the effectiveness of CoT based training further making it more efficient using their GRPO algorithm. Inspired by these success in text-based reasoning, a growing body of work apply RL to enhance the reasoning capabil- ities of LVLMs for vision-centric tasks, such as image classification (Liu et al. 2025 Li al., 2025). Object detection, Grounding (Liu et al. [025]] Shhn et al. 2025), and Visual Question Answering (Cao et al. 2025 Sarch et al.] [2025] Fan et al. (2025). Within our target domain of fine- fine-grained image recognition, the most pertinent work is ViRFT (Liu et al. 2025). which trains an LVLM using exact string matching reward to foster visual reasoning. However, this reward mechanism proves brittle, failing to generalize in base-to-nove settings. Furthermore, it does not fully leverage the LLM's inherent knowledge about fine-grained categories to incentivize the differential reasoning necessary for distinguishing among similar options. Zhu et al. ):055;; Chen et al. (2025) proposes to enhance LLM performance by improving Pass k accuracy. In contrast, we use model's own knowledge of Pass@k to improve their reasoning, specifically for vision-language task.
3 DIVE-K FRAMEWORK
DiVE-k framework employs a simple two-step strategy which elicits differential reasoning in LVLM using its top-k generations as training signal. An overview of our proposed method is shown offline top-k generation using the base model to construct a potential hypotheses set to be used for constructing Multiple Choice Questions (MCQs). In the second step (green box in 2). we use the MCQ dataset for RL training using GRPO.
in Figure 2 In the first step (red box in ( we perform
3

---

Published as a conference paper at ICLR 2026
african
Bengel
european african lymx bengal egyphion
Prongii pet identily pet Image"
Ref model
Aey
egyphion mou
Erputt Image 00
Ground frust celegery
(options)
A
Trainable
(heyy pat the Image from the given optioe Insliwl!
Prorpt: the neat
Pekay model
4
+ Frenen
Gradient
flow
ressering freces </hhiks
Figure 2: An overview of DiVE-k framework. First, we do an offline option mining (red box) where for each training image, we sample K rollouts from a pretrained LVLM and select top-k options by frequency, ensuring the ground-truth appears. Next we perform RL training using GRPO on MCQ prompts (green box): the model receives an image, natural language prompt, and k options as input and produces a reasoning chain and a final choice and is optimized with a simple, verifiable reward that combines MCQ correctness and format compliance.
Top-k as hypotheses set. In the first step of the DiVE-k framework, given an image I and a text query q, use the base policy model Te to rollout K responses y (V1, 1/. .., WK) sampled through a specific decoding strategy (e.g. top-p nucleus sampling).
y ~ To(I, 1, K)
Each of these responses (y: E )) can be represented as yi = (ra,ci, where is the reasoning trace, and e; the final predicted category name. Let C be the unique category names set within the K generated responses. Next, we count the frequencies of each category in C and using this frequency count, we construct the option set Ozop- by selecting the k most frequent categories. The value of k is set to k = min(m, [C]), where [C] is the number of unique categories, and we use = 5. Let è be the ground-truth category name. To ensure that the correct answer is always an option during training, adjust the set if necessary. If è & Otop-k- modify Otop-k by replacing the least frequent candidate with the ground-truth é
Finally, the option set Otop- & is structured into standard MCQ format. The options are enumerated and assigned labels (e.g.. A, B, C, ...). The options are randomly shuffled to avoid any option-order bias. The ground-truth label, o, is the label corresponding sample in our final dataset, D, is tuple (1, g, O Oenum à), where / is the image, is the query, Oenum the enumerated list of option strings, and the ground-truth label for the correct option.
the correct category é. Thus, each
To focus on challenging examples through hard-negative mining, we filter out trivial cases from the training set. Specifically, we exclude any sample for which the model generates only a single, correct category prediction (i.c., [C] = 1 and C= (c}). Note that our first step to generate options with the same base model used as the policy model during training is crucial, as it ensures the categories are drawn from the model's own distribution and yields optimal learning as shown in section 4.3.1
RL training using GRPO. We train the model using the MCQ dataset constructed in step one. Our task is defined by D consisting of (I,4, q, O Oemum; as explained in previous section. Our goal is to train LVLM as policy model ma which can generate (s, a) where is the intermediate reasoning tokens and is final answer. To achieve this, we train the model using GRPO algorithm (Shao al. 2024b). During training, for every data sample d,, model generate rollouts (Oo, OI.... On using the current policy model kg. For each of these responses, reward (ro, 71, "i,..,, ..., T'N are computed. These rewards for each group then used for group advantage estimation using equation 2
mean (r,,.... rN [N)
A std (
(2)
PN)
where 8 is small valued constant.
4

---

Published as a conference paper at ICLR 2026
Top-k loptionl
Trained
K rollouts
2012 SMW Xs
fed
polit frgi
2007 bew 25
from the bobow [sptions"
Ourt
sttleo The Image those BMW model, The design fashuras shick choractertette of the avenll body style, aäge desaty
diftective cospe-ibee reafline X0
grill
the grill shope x0. The hot more bayy and tradtiional SUV look, whih X6 hot choloor more dynudynenic profie. The oppeare o on older modal Batel suVcittiiko moders look, naggatting camwer claewer
polcy (rg)
image.
Figure 3: An example illustrate our inference pipeline (red arrows) and its comparison to exist- ing method (blue arrows). Similar to training phase, we perform inference in two steps (right of dotted line), where we first generate option by choosing top-k responses from K rollouts and then model picks the correct answer among the options unlike open-ended one step inference of existing methods (left of dotted line)
Our reward consists of two parts: first is MCQ reward (Tmeq) and second is format reward (r format). I'me checks for correctness in the response and rewards value of 1.0 if model predicts the correct option as answer and 0.0 otherwise as in equation 3
1.0,,
if à
T T'ooy (ā, a)
(3)
0.0 otherwise.
T format encourages the correct formatting for the output response to make it casier to extract <think> and <answer> tags. Our final reward (r) is defined the weighted sum of these two: r = format + Am I' mmmg. Our training objective function (o) defined as:
JGRPO (0)
Σ [min(s; clip(s;, e, +)A,) 3Dx (Te|
where e and | are the hyperparameters. Tie is the reference policy model (usually the pre-trained model) used to control the divergence of trained policy. 8; = na (o.1g)' where Tha is the old policy model before the update. Overall, this objective function aims to maximize the expected reward while keeping the policy close to the reference policy for stable learning.
=o(o::g)
inference pipeline and compares it to ViRFT which performs single-pass inference (left the dotted line) to directly predict the category names, while we use two-step pipeline (right of the dotted line) similar to training phase. We use the trained policy model to first generate potential options using top-k generations and then re-prompt the same policy to select the correct option among the provided options. It should be noted that we do not add ground-truth to the options if the first step fails to generate the groundtruth category as an option.
Inference. Figure 3 shows
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Baselines and Datasets. We compare our results against various baselines: 1) pre-trained QWEN2.5-VI-77B (Bai et al. 2025 2) Supervised Fine Tuning (SFT) (Zheng et al. 2024a) using full model and LoRA (Hu et al. (022) training. 3) ViRFT (Liu et al. (025) trained under the same base-novel setting. 4) OWEN2.5-VL-78 and ViRFT using our inference pipeline 5) Consistency as accuracy: During the two-step inference process, use the most consistent prediction from genera- tion final prediction. We also report the performance of the proprietary models, Gemini2.5-faar
lite(Comanici et al. 2025). GPT-5-mini (OpenAl (025) and Grok4-fast (xAl 2025), with closed weights accessible through APIs (OpenRouter 2024).
We evaluate our method on five standard fine-graine gree-graind image classification dataset across various domains, including OxfordFFlowess100 (Nilshack & Zisserman 2008), CUB-200 (Wah et al. 2011).
5

---

Published as a conference paper at ICLR 2026
OxfordPets-33 37 (Parkhi al. 2012), StanfordCars-196 (Krause et al. 2013) and FGVC Aircraft-100
Maji et al. 2013). For the base-novel split, we follow previous work (Kattaak et al. 2023a) to divide the categories into equal halves for base and novel. For instance in CUB-200, the first 100 categories are considered base and the other 100 are considered novel and similarly for other datasets. Note that neither images nor the category names of the novel classes seen during training.
Table 1: Quantitative comparison with existing methods on zero-shot base-to-novel generalization.
Our proposed methods shows strong generalization outprrformig existing methods. B + Base, N
+ Novel, H - Harmonic Mean, Geminí2.5-f-I - Semini2.5-ffash-liie flash-lite, QWEN2.5 - QWEN2.5-
VL-7B. Rows with are results of existing method using our inference pipeline. Consistency +
Most consistent prediction top-k generation as final answer.
Flowers
CUB
Pets
Cars
Airenftt
Avg
Method
Propretary Medels
Gemini2.54fl
94.1
91.6
92.8
75.3
60.3
91.5
93.0
73.9
92.3
82.1
50.7
65.4
62.4
78.0
80.0
GPT-S-mini
97.4
94.4
95.0
76.3
64.7
70.0
93.1
97.5
95.2
82.0
94.7
87.9
50.2
74.5
66.6
81.8
85.1
83.4
Grok Grok-4-tast
81.3
87.4
84.2
63.3
50.7
56.3
84.1
94.2
88.8
72.2
36.6
78.7
44.0
55.4
49.1
6P.DD
74,8
71.8
Method Compatison
CLIP
72.1
27.8
74.8
65.5
18.7
55.8
97.3
94.1
63.4
14.9
68.6
27.2
36.3
31.1
63.8
67.0
64.99
QWEN2.5
48.2
54.3
SET (Full)
93.7
62.6
75.1
$4.8
29.2
4334
95.8
42.6
39.0
81.2
40.6
54.1
64.7
10.1
17.5
84.0
37.0
49.8
SPT (LoRA)
84.5
83.9
84.2
62.2
49.2
$4.9
$5.7
90.3
89.3
59.9
71.2
65.0
29.5
54.3
$1.8
68.33
70.4
69.3
VIRFT
84.3
84.6
84.5
65.4
51.0
57.3
90.5
95.5
92.9
60.3
73.6
66.3
64.6
66.3
65.4
73.0
74.2
73.6
QWEN2.5*
90.7
89.3
58.7
89.5
69.4
75.6
Consistency
85.9
35.5
85.7
64.2
54,0
58.6
90.2
94.7
92.4
62.1
72.0
66.7
51.8
63.5
62.6
72.8
74.0
73.2
VIRFT*
89.5
87.9
88.7
70.0
60.0
64.8
92.6
92.8
92.7
64.8
76.9
70.4
66.4
67.8
67.1
76.8
77.1
76.9
Consistency
85.3
85.2
85.3
67.5
57.2
92.0
94,0
64.8
68.8
67.4
64.2
65.7
75.4
75.1
(ouns)
65.5
$9.1
91.6
72,4
69.1
Consisteacy
90.3
85.6
90.2
75.2
02.3
68.2
93.1
95.5
94.3
68.7
74.4
71.4
68.0
03.6
66.8
80.0
76.7
78.2
-13.1
44.3
48.4
+15.
+14.5
414.0
-14
-13
-13
48.7
46.1
43.5
+3.8
43.2
47.8
A W QWEN25
-13.2
65.1
48.9
+17.2
+17.3
417.5
+1.
40.9
41.3
+10.1
+3.
47.3
+17.8
14.8
+16.3
+11.9
+8.3
+10.1
Evaluation setting and Metric. We evaluate our method under two distinct zero-shot settingg. The first the standard zero-shot base-to-novel generalization, following (Zheng al.) 2024b), where a separate model is trained on the base classes of each dataset. The second is our proposed mixed- dataset setting, designed to assess cross-domain generalization capabilities. Here, a single model is trained on a unified dataset constructed by combining the base classes from all datasets. Addition- ally, we also evaluate our method under few-shot classification setting with 4 shots per class. For performance measurement, we report the classification accuracy on base and novel classes, along with their Harmonic Mean (HM).
For evaluation, use the LLM gemini-2.5-flash-- (Comanici et al. 2025 where we provide the ground-truth category name and model predicted category name and ask the LLM if they belong to the same fineggrained category not. This specifically helps evaluate better for the answers where model responds a scientific name and the provided ground-truth is common name and vice- versa. We provide more details in Appendix A.1.2
Implementation Details. We use the pre-trained Qwen2.5-VL-77--IE base model and perform the RL training using the proposed method. We K=20 during offline option generation using K rollouts and other details are provided in Appendix I.I.]] For Zero Shot Base-to-Novel training, we train the model for 400 steps, whereas for mixed data training, for 1 epoch. For few-shot training, we train each model for 200 steps following ViRFT. All models are trained on three A6000 GPUs with 48GB memory with an overall batch size of 6. For GRPO, total of 4 responses are generated for each input sample. Following ViRFT, we use learning rate of 10-6, AdamW optimizer, linear scheduler and set X,n = Ay 1. For ViRFT training, we use their official code from github with some modification discussed in Appendix A.1.1
(Bai et al. 2025 as our
train
4.2 RESULTS
4.2.1
TERO-SHOT -OSE-LO-NOVEL GENERALIZATION
We present the results for zero-shot to-vell generalization in Table I Across five fine- fineggrained benchmarks, DiVE-k shows the best generalization, yielding the highest average harmonic mean (HM) of 79.8 with average base/novel accuracies of 80.8/78.8. Relative to the strongest baseline

---

Published as a conference paper at ICLR 2026
(ViRFT), this corresponds to gains of +7.8 base, +4.6 novel, and +6.2 HM. Even when baselines run with our inference pipeline (ViRFT*), we retain a +4.0 base, +1.8 novel improvements. Notably, under inference pipeline, ViRFT gain over QWEN2.5-VL-75 model novel categories remains marginal by +0.1 (77.0-+77.1), suggesting their reward primarily reinforces base categories. In contrast, approach delivers substantive +1.8 boost novel classes under the same inference setting, indicating better generalization beyond the training categories.
The improvements are especially pronounced on CUB (+14.9 HM) and Oxford Flowers (+8.5 HM), and remain consistent on Stanford Cars (+6.1 HM) and FGVC Aircraft (+3.2 HM), indicating robust zero-shot transfer. We observe a small regression on Pet data compared to ViRFT, possibly due to more options leading the model to make more mistake during the second step. We show that our method outperforms ViRFT for smaller K and provide more analysis for this Ablation 4.3.2
We also evaluate supervised fine-tuning (SFT) as baseline. Although SFT yields strong accuracy gains on base categories, its performance deteriorates sharply on novel categories, with an average accuracy drop of 33.5% relative to the base model and a 19.9% reduction in HM. This sharp degra- dation highlights SFT's inability to generalize under the base-to-novel transfer setting. To provide broader perspective, we also include results from proprietary models which are likely much larger. In this context, our method surpasses Grok4-fast and is on par with Gemini2.5 1amini2.5-flash-litt though GPT-5-mini leads. We provide more results using Gemma-3 (Team al. 2025a) base model in Appendix A.3
Table 2: Quantitative comparison of our method with baselines under mixed-dataset base-to-novel generalization. QWEN2.5 - QWEN2.5-VL. VLL-B. Rows with are results of existing method using our inference pipeline, Consistency + Most consistent prediction in top-k generations as output.
Flowers
CUB
Pets
Cars
Airerañi
Avg
Method
N
B
H
B
N
H
B
N
QWEN2.5
83.8
63.3
48.2
90.3
58.9
72.9
65.1
54.3
52.3
70.5
VIRFT
87.2
85.9
86.5
65.7
$4.0
59.3
90.5
95.0
92.7
59.7
73.6
66.0
67.2
58.0
67.5
74.1
75.3
74.7
QWEN2,5
90.7
87.9
89.3
68.5
58.7
63.2
85.6
93.8
89.5
63.3
76.6
69.4
63.3
68.0
65.6
74.3
77.0
75.6
Consisency
92.4
62.6
67.8
ViRFT*
90.2
87.9
89.0
71.3
58.3
64.2
90.2
91.6
90.9
63.7
76.6
69.6
66.9
66.4
66.7
76.5
76.2
Consistency
85.2
85.1
85.2
68.0
$7.5
62.3
91.5
96.1
93.7
65.6
74.2
69.7
63.2
65.5
64.1
64.4
69.9
67.0
EXVESE (ours)
97.4
89.9
93.5
76,8
61.3
68.2
87.8
94.7
91.1
68.5
78.5
73.1
65.5
69.7
67.5
79.1
78.8
78.7
Consisiency
VIRFT
+10.2
+4.0
+7.0
+11.1
+3.3
2.7
0.3
-166
+8.8
+4.9
+1.7
0.0
+5.1
+3.
+4.0
QWEN25
+6.1
+13.5
+13.1
+9.5
+15.
+103
+83
4.2.2 MIXED-DATASE BASE-TO-NOVER GENERALIZATION
Training a single model on the union of base categories from all five datasets provide a strong eval- uation of mixed-dataset generalization. Table 2 shows the quantitative results for this setting. Our method attains the highest average harmonic mean (HM) of 78.7, improving over the pretrained QWEN2.5-VL-7B by +9.0 HM and over ViRFT by +4.0 HM, with average base/novel gains of +5.1/43.5. Under our two-step inference novel classes, ViRFT underperforms the pretrained QWEN2.5-VL-77 (76.2 vs. 77.0), while our approach reaches 78.8, a +1.8 improvement. This contrast suggests that ViRFT struggles to transfer when trained on the mixed base corpus, likely inforcing base-only cues, whereas our method maintains robust generalization to unseen categories.
Table 3: Quantitative comparison of our proposed method under 4-shot setting.
Model
Oxford Flowers
CUB
Oxford Pets
Stanford Cars
FGVC Aireraft
Average
OWEN2.5-VL-7B
78.43
51.62
79.05
57.91
52.50
63,90
ViRFT
81.12
51.75
85.81
$7.65
58.75
67.02
QWEN2.S.VI.7R
85.04
56.87
80.40
63.46
64.67
70.09
consistency
81,86
53.00
85.81
58.30
56,36
67.07
ViRET
84,80
consistency
83.33
54.12
87.83
59.02
54,71
67. 80
DIVE-k (Our))
88.72
63.87
85.14
66.90
69.10
74.75
consistency
85.78
58.00
86.48
60.17
64.92
71.07
A VIRFT
7.60
12.12
-0.67
9.25
10.35
7.73
A WILL OWENZ5S-VL-7B
10.29
12.25
6.09
8.99
16.60
10.85

---

Published as a conference paper at ICLR 2026
"The head composed of numerous small, delicate petals that arranged circular pattern. The color is soft pastel purple, which is characteristic of globe thistle (Scabiosa), which is known for its large, flat heads and delicate, spiky petals. The globe thistle fits the description of the Rower the image well. Final Answer: globe thistle
Reasoning:
image distinctive appearance with large, Aat, rounded
VIRFT
top-k Options: Alpincushioo Rower", 8. "sweet william", C. "globe- thistie" Differential Reasoning: The flower in the image has cluster of small, delicate petals that are arranged a circular pattern around central core, which is characteristic of pincushion Rowers (Scablosa). petels are light purple with slighlly ruffled oppearence, and overall structure matches the description of pincushion flower.
Promph:
"This is on image containing Flower. Please identify the species
Ours
william typically has larger, rounded petols, globe-thistie has spikier leaves and different Rower structure. Therefore, the most likely flowe in the image pincushion Rower.
the image"
Final Answer: pincushion Rlwwr
Figure 4: Qualitative comparison on fine-grained flower recognition (top: ViRFT; bottom: Ours). Top: ViRFT predicts "globe thistle," which is incorrect and reflects coarse judgment. Bottom: Our method enumerates close candidates and uses attribute- grrooutt-grodedd,.r differential reasoning such as capitulum/head shape, floret density and arrangement, bract patterning select the correct fine- grained label with justification aligned to the final choice.
4.2.3 FEW-SHOT CLASSIFICATION
DiVE-k also shows improvement under few-shot classification setting across datasets. As shown in Table 3 method achieves average HM classification accuracy of 74.75% improvement of 7.73% compared to ViRFT and 10.85% compared to QWEN2.5-VL-7B model, demonstrating the effectiveness of method even under data efficient training.
4.2.4
VISUALIZATION
In Figure 4 visualize and compare DiVE-k to ViRFT and defer additional visualizations Ap- We find that ViRFT directly latches onto high-level "thistle-like" cues and commits to an incorrect category ("global thistle")), without checking the discriminative attributes that separate neir-neig In contrast. DiVE-k first proposes small top-k shortlist and then rules candidates in/out through explicit, attribute-level comparisons, such as head geometry, the density//rrangemmntt of florets, and the presence/shape of bracts, before committing to final option. This process yields the correct species-level label together with a rationale that stays consistent with the answer reducing overgeneralization and improving interpretabilityy
pendix A.2
4.3
ABLATION STUDIES
4.3.1 THE EFFICACY OF TOP-K FOR MCQ OPTION GENERATION.
Our ablation into the MCQ option generation reveals that the option construction strategy is critical for model's performance. As detailed in Table 4 ran- domly selecting categories proved suboptimal, yield- ing only marginal gains and failing to instill robust reasoning capabilities. While employing a text em- bedding model from gemini embeddings (Lee et al. 2025) to generate semantically similar options of- fered some improvement, our proposed top-k as op- tions proves significantly more effective. By sam- pling options directly from the same base model's top-k generations achieves substantial clas- sification accuracy gain of 12% base classes and 6.9% novel classes compared to base model. This demonstrates that leveraging the base model's own knowledge distribution to generate options the optimal strategy for training, leading superior generalization both base and novel sets.
Table 4: Quantitative comparison of classi- fication accuracy CUB dataset when op- tions are sampled in different ways.
Base
Novel
HM
QWEN2.5-VL-7B
68.5
58.7
63.2
Random MCQ
72.5
57.5
64.1
Text Finh MCQ
76.0
59.3
66.8
zop-k MCQ
80.5
65.5
72.2
8

---

Published as a conference paper at ICLR 2026
Plowers
CLb
Aicrutt
12.3
72
00.3
613
60.3
53
60.3
68.3
01.37
56.
Figure 5: Change in classification accuracy for different values of K on different dataset. Our approach consistently outperforms the baselines for nearly all K.
4.3.2 ROLE TOP-K GENERATIONS DURING INFERENCE
We analyze the impact of the hyperparameter K, during K rollout for top-k generation, clas- sification accuracy, with results presented in Figure The plots show the HM of accuracy across base and novel split for different values of K during inference. Our method consistently surpasses both the baselines at nearly K, often by a large margin. As increases, accuracy generally rises and then saturates around K ~ 10 15, yielding mear-maximal performance at K = 15 20 for Flowers (~93%), CUB (~72.5%). Car (~73%), and Aircraft (~70%)), indicating diminishing returns beyond K 10-15. However, for Pet dataset it peaks small K (93.8% at K 2) and gradually declines for larger K, suggesting more options during MCQ leads to more mistakes for this specific dataset. Overall, these results show that approach achieves improved performance even for small K and thus can get benefit while avoiding extra computation.
4.3.3 ROLES OF VISION AND TEXT COMPONENTS
To investigate the individual contributions of the model's vision and text components to its reason- ing capabilities, conduct ablation to determine whether performance gains stem primarily from up- dating the vision features, refining the language model's ability to generate a correct chain of reason- ing tokens, or combination of both. The results of which are presented in Table
Table 5: Ablation effect of training dif- ferent part of the model on classification ac- curacy for CUB dataset; Full model training yields the best performance.
Base
Novel
HM
QWEN2,5 VL.-B
68.50
58.67
63.21
Vision only training
74.00
Text only training
74.33
60.33
66.60
Full model training
80.50
65.50
72.23
Our findings reveal distinct roles for each modal- ity. When we fine-tuned only the vision components
(freeze text decoder), the model's performance improved significantly on the base dataset (68.5-774)
but failed to generalize to the novel dataset, where performance slightly degraded (58.67 (88.67-57.83) This suggests that adapting visual features alone is insufficient for robust reasoning on new, unseen data. Conversely, training only the text components (freeze vision tower) improved performance on both the base (68.50 (68.50-774.33 and novel (58.67-460.3) sets. This indicates that enhancing the language model's ability to generate logical reasoning critical for generalization. However, the best performance was achieved through full model training, which yielded substantial gains on both base (8..50++80.50) and novel adaptation is key, a combination of both vision and text modules is necessary to unlock the model's full reasoning capability.
sets. This demonstrates that while language model
4.3.4 ACCURACY ANALYSIS OF THE TWo STEPS IN DIVE-K
As DiVE-k framework operates in two stages, the second stage can only succeed if the correct answer appears in the top-k rollouts from the first stage. Therefore, understanding the bottleneck between these two steps is crucial for diagnosing model performance.
Table 6 reports the top-k accuracy (Step of both the base QWEN2.5-VL-2 and the DiVE-k. We find that the base model already achieves strong top-k recall across most datasets, indicating that the correct answer is typically recoverable via sampling. DiVE-k training further improves this

---

Published as a conference paper at ICLR 2026
recall, yielding more consistent retrieval of the correct candidates. This improvement is particularly beneficial because Step 2 can only operate correctly when Step supplies the correct option.
Table 6: Top-k accuracy (step 1) across five datasets
Flowers
CUB
Pets
Cars
Aircralt
Avg
Method
N
N
B
N
N
B
N
OWEN2,5-VL
96.84
92.62
14.66
98.89
89.02
90.9
89.4
DIVE- (ouns)
98.99
92.62
89.66
77.50
98.94
16
89.
74
90:57
92.96
90.21
94.1
00.0
A
2,15
0.00
6.33
2,84
0.54
0.27
3,07
-0.27
3.94
0.00
3.2
0.6
Table 7 presents the Step 2 MCQ accuracy (differential reasoning). We observe average improve- ment of 4.3% on base and 1.6% on novel categories. Since Step 2 operates over the candidates generated in Step 1, high top-k recall directly strengthens the effectiveness of differential reason- ing. Improvements in candidate quality and reasoning accuracy reinforce each other, resulting in a compounding effect the final classification performance.
Across most datasets, Step accuracy is already high (often above 90%), which shifts the primary performance bottleneck to the differential reasoning stage. DiVE-k explicitly targets this challenge and achieves clear gains in MCQ accuracy. For the CUB dataset, however, top-k accuracy remains comparatively lower, leaving additional room for improvement in Step 1. This highlights that the two stages contribute differently depending on dataset difficulty. Some benchmarks are limited by candidate generation, while others limited by reasoning over those candidates.
Table 7: MCQ accuracy (Step 2) across five datasets
Flowers
CUB
Pets
Cars
Aircraft
Avg
Method
N
QWEN2,S-VL
93.65
94.90
82.20
78.62
94.85
32
81.4
65.6
(ouns)
98.40
98
89.78
84.52
90.05
95.00
76.00
73.25
76.59
85.7
87.2
A
4.75
1.08
7.58
5.90
3.06
0.15
3.87
-0.22
2.15
1.17
4.3
1.6
LIMITATIONS AND FUTURE WORK
While DiVE-k framework effectively leverages the model's intrinsic knowledge acquired during pre-training to improve its downstream accuracy, our two-step inference process incurs additional computational cost due to the requirement of two forward passes. The success of our approach demonstrated on QWEN2.5-VL, which exhibits high initial Passêk accuracy. However, the method's efficacy is contingent on this baseline performance: base LVLMs with lower intrinsic ac- curacy may not realize comparable gains. A potential direction mitigate this dependency, which we leave for future work, is to incorporate Pass@k accuracy reward signal directly into the training objective. Another promising avenue for future research involves the verification of gener- ated reasoning traces for factual correctness and their grounding in the input image.
CONCLUSION
In conclusion, introduced DiVE-k, novel framework that addresses the limitations of Large Vi- sion Language Models in fine-grained image recognition. By utilizing top-k generations as training primitive, our method requires the model perform differential reasoning among visually simi- lar categories using multiplechhoiee question format. Extensive experiments across five standard datasets demonstrate that DiVE-k significantly outperforms existing approaches base-to-novel generalization, mixed domain, and few-shot settings. Our ablation studies further reveal that the efficacy of this approach hinges on mining options from the base model's own distribution, which is critical for effective RL training. Moreover, we show that the joint fine-tuning of both vision and text components is essential for unlocking the model's 's full reasoning potential and that increasing the value of k offers diminishing return during inference. Overall, our work highlights the effec- tiveness of leveraging a model's 's inherent knowledge distribution to refine its reasoning capabilities, establishing direction for improving visual discrimination in LVLMs.
10

---

Published as a conference paper at ICLR 2026
7 REPRODUCIBILITY STATEMENT
We aim to ensure full reproducibility of our work by providing detailed descriptions of our method- ology, training and inference pipeline, and evaluation protocol in the method, results and appendix section. All hyperparameters, implementation details, and generation settings (e.g., temperature, sampling strategies, and reward design) are listed in the Appendix, along with the exact prompt tem- plates used for all experiments. Experiments are conducted models; for ppen-source models, we provide precise checkpoint versions. We plan to publicly re- lease our codebase, experimental configurations, and trained model checkpoints upon acceptance to support reproducibility.
both open-source and closed-source
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 139233, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 2020.
Meng Cao, Haoze Zhao, Can Zhang, Xiaojun Chang, Ian Reid, and Xiaodan Liang. Ground- rl: Incentivizing grounded visual reasoning via reinforcement learning. arXiv preprint arXiv:2505.20272. 2025.
Zhenfang Chen, Qinhong Zhou, Yikang Shen, Yining Hong, Hao Zhang, and Chuang Gan. See, think, confirm: Interactive prompting between vision and language models for knowledge-basee visual reasoning. arXiv preprint arXiv:2301.05226 2023.
Zhipeng Chen, Xiaobo Qin, Youbin Wu, Yue Ling, Qinghao Ye, Wayne Xin Zhao, and Guang Shi. Pass@ k training for adaptively balancing exploration and exploitation of large reasoning models. arXiv preprint 2508 200.::5008..0511 2025.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240): 1-113, 2023.
Gheorghe Comanici, Eric Bieber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Oni Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Reza Esfandiarpoor and Stephen H Bach. Follow-up differrnntial descriptions: Language models resolve ambiguities for image classification. arXiv preprint arXiv:2311.07500 2023.
Yue Fan, Xuchai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Sravana Jyothi Narayanaraju, Xinze Guan, and Xin Eric Wang. Grit: Teaching milms to think with images. arXiv preprint arXiv:2505. 15879, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 2025.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning
without training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14953-14962, 2023.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
11

---

Published as a conference paper at ICLR 2026
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16726, 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https:
//arxiv. www.org///aa.. /bbg//b//// 2.10. 06825
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming- Hiug--ssuan Yang. and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation with- out forgetting. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 15190-15200, October 2023a. Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuaa Yang. and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation with- out forgetting. In Proceedings of the IEEF/CVF international conference on computer vision, pp- 55190-15200, 2023b.
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision work- shops, pp. 554-561, 2013.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. Tulu 3: Pushing frontiers in open language model post-training, arXiv preprint arXiv:2411.15124 2024.
Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Attribute-based classification for zero-shot visual object categorization. IEEE transactions on pattern analysis and machine intel- ligence, 36(3):553446 465, 2013.
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gus- tavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, et al. Gemini embed- ding: Generalizable embeddings from gemini. arXiv preprint arXiv:2503.07891,
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified dision-languan understanding and generation. In International conference on machine learning, pp. 12888-1200 PMLR, 2022.
Ming Li, Jike Zhong, Shitian Zhao, Yuxiang Lai, Haoquan Zhang, Wang Bill Zhu, and Kaipeng Zhang. Think or not think: A study of explicit thinking in rule-based visual reinforcement fine- tuning. arXiv preprint arXiv:2503. 16188, 2025.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 00134489--44166 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26296-26322 223366, 2024.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint 1r0ivv1306.515: 2013.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint ar 10.00:2110.07/88, 2022.
12

---

Published as a conference paper at ICLR 2026
Maria-Elen Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing. Dec 2008.
Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shor image classification with hierarchical label sets. In International Conference on Machine Learning, pp- 26342-26362. PMLR, 2023.
OpenAL. Gpt-5 system card. httppp///deh.pppea....ccoogeeees 2025. Accessed: 2025-09-14.
OpenRouter. Openrouter.
2024. Accessed: 2025-09-14.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
Sarah Pratt, lan Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? gener- ating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF international conference computer vision, pp. 15691-15701, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8-63. PmLR, 2021.
Karsten Roth, Jae Myung Kim, Andrew Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In Proceedings of the IEEE/CVF international conference computer vision, pp- 15746-15757, 2023.
Gabriel Sarch, Snigdha Saha, Naitik Khandelwal, Ayush Jain, Michael J Tarr, Aviral Kumar, and Katerina Fragkiadaki. Grounded reinforcement learning for visual reasoning. arXiv preprint arXiv:2505.23677 2025.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong. Letian Wang, Yu and Hong- sheng Li. Visual cot: Advancing multi-modal language models with comprehensive dataset and benchmark for cfain-of-thought reasoning. Advances in Neural Information Processing Systems, 37:8612-8642, 2024a.
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepscekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03330, 2024b.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang. Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable r1-style large vision-language model. arXiv preprint arXiv:2504. 07615, 2025.
Richard Socher, Milind Ganjoo, Christopher Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. Advances in neural information processing systems, 26, 2013.
Dídae Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEF/CVF international conference computer vision, pp- 11888-11898, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11800: 2023.
Gemma Team, Aishwarya Kamath. Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint a50iv:2503. 19786, 2025a
13

---

Published as a conference paper at ICLR 2026
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.12599, 2025b.
Kimi Team, Angang Du Bohong Yin. Bowei Xing, Bowen Qu, Bowen Wang. Cheng Chen, Chenlin Zhang. Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:2504.07491, 2025c.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Nacem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathyy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint 25X1::2502.14786 2025.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
model's perception of the
Xiaolong Wang, Yufei Yc, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6857-6866, 2018.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thouge prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837 2022.
XAI. Grok 4 fast. https:/ a.a.aaii.. wwwpa///ww////oooo////.
2025. Accessed: 2025-09-14.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao. Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11442 2021.
Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveck Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Com- posing zero-shot multimodal reasoning with language. arXiv preprint arXhv:2204.00590 2022.
Li Zhang, Tao Xiang, and Shaogang Gong. Learning a deep embedding model for zero-shot learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2021- 2030, 2017.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal thouugh reasoning in language models. arXiv preprint arXiv:2302.00923, 2023.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo. Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024a. Association for Computational Linguis- tics. http:
abs 240..13372
Zhaoheng Zheng, Jingmin Wei, Xuefeng Hu, Haidong Zhu, and Ram Nevatia. Large language mod- els are good prompt learners for low-shot image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 28453-28462, June 2024b.
Zhaoheng Zheng, Jingmin Wei, Xuefeng Hu, Haidong Zhu, and Ram Nevatia. Large language mod- els are good prompt leamerrs for low-shot image classification. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 5348-5355 2024c.
14

---

Published as a conference paper at ICLR 2026
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pp. 168 16816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2348, 2022b.
Xinyu Zhu, Mengzhou Xia, Zhepei Wei, Wei-Lin Chen, Danqi Chen, and Yu Meng. The surprising effectiveness of negative reinforcement in llm reasoning. arXiv preprint 2rX11 2025.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 IMPLEMENTATION DETAILS
A.1.1 TRAINING DETAILS
Sampling parameters. In the first step of our pipeline, we generate K responses using top-p nucleus sampling. We provide the details about the parameters used during this sampling in Table
Table 8: Generation Arguments
Parameter
Value
max frax_new_tokens
1024
temperature
1.0
top-p
0.95
do_sample
True
20
zepetition.penalty 1.1
ViRFT model training. During our evaluation of ViRFT method, we find a crucial issue in their implementation. The string match logic used during both training (for reward computation) and evaluation (for answer correctness) is shown in Listing A..1.] We find that the second part of the or statement (student_anse ansswer in ground truth) leads shortcut specifically for fine- fine-grained image classification. For example, even if the model responds "gull" this reward function (and evaluation) will consider as correct even though doesn't give a correct answer. (it could be any of "california gull", "Heermann Gull", "ivory gull" etc.). This leads huge drop in accuracy when evaluated using our I.LM evaluation as demonstrated in Table for CUB and Stanford Cars dataset. We fix this issue by removing this shorteut and keeping only the "ground_trutt" in student.answer during training for reward computation. After fixing this error we observe expected results shown in
Table Table to demonstrate the shortcut issue in original ViRFT code. ViRFT refers to the accu- racy using the original code, ViRFT' refers our modified code used for training
Method
CUB
Stanford cars
Base
Novel
Base
Novel
QWEN2.5-VL-BB
63.33
48.17
58.87
72.9
ViRFT
39.33
27.83
13.79
23.09
ViRFT
65.44
51.00
60.34
73.63
String matching code ViRFT
reward computation code
if gruthd_truth in student_answer student_answer in ground_truth:
1.0
$ evaluation code if image_cate in answer_content or answer_content in image_cate: right_ccunt +=
else:
print ('no')
A.1.2 EVALUATION DETAILS
Here we provide the details of the different prompt used at different stage of our method. Fig- ure we have provided the complete prompt used in first step of our proposed method. Here,
16

---

Published as a conference paper at ICLR 2026
"The aircraft in the image regional jet with T-fail configuration, which is characteristic of the British Aerospace (Bhe) 146 series. The specific model can be identified by itts size and design, which matches the BAe 146-300. The livery and registration also support this identification. Final Answer: BAE 146-300
Reasoning:
top-k as options Abbee 146-200" B. "boe 146-300"
Differential Reasoning:
image oppears
identifialle
its T-tail configuration and the number engines. specific model can be deduced from the registration number and the design features. The registration number LZ-HBG suggests that this aircnaff is registered in Bulgario, and the design features match those of the BAE 146-200 series. 146-300 different design engine placement aircraft the image.
Prompt:
"This image containing aircroft. Please identify the model of the aircraft based on image"
Final Answer: bae 146-200
Reasoning: bird image appears
brownish-gray
body and short, pointed beak. It has light-colored underbelly and darker wings with white streaks. The bird's posture and the setting suggest It could be flyastcher similar small passerine bird. the bird resembles characteristics of a Great Crested Flycatcher, which is known for its similar appearance and habitat preferences.
Final Answer: Great Crested Flycafcher
options: A. "olive-sided flycatcher", B. "great crested flycotcher", "least flycatcher", D. "yellow-bellied flycatcher', E. "acadion flycatcher""
Prompt:
Differential Ressoning: The bird the image has a small size, slender body, and a short pointed beak, which characteristic
"This image containing bird. Please identify the species of the bird based on the image"
flycachhrs. coloration, with light underside and darker upperparts, consistent with the flyostcher (Empidonax minimus). The olive-sided flycatcher (Contopus cooperi) typically has more clive-colored back and longer fail, the great crested flycatcher (Mylarchus crinitus) has larger size and more robust build, the yellow-bellied Pyycaccher (Empidonox flaviventris) yellow bally, (Empidonax virescens)
acadian flycatchar olive-green back and shorter Final Answer: lesst Rycatcher
rectangular front grille, which is characteristic of Ford trucks. The design and shape the vehicle, including the headlights and overall body style, align with the Ford F-150 model. The specific model year and configuration (Regular Cab) can be inferred from the design elements".
Reasoning:
""The image
pickup truck
VIRFT
Final Answer: "2012 Ford F-150 Regular Cab"
top-k Options "200 ford f-150 reqular cab", 8. "2012 ford f-150 reguler cab". Diffenential Reasoning: The Image shows white pickup truck with classic design, which characteristic of the Ford F-150 models produced around the earty 2000s. The specific model common the F-150 during this period. grille shape, headlight design, and overall body style, aligns closely with the 2007 Ford F-150 Regular Cab model. The other option. 2012 Ford F-150 Regular Cab has more modern design features that do not match the vehicle in the image"" Answer: "2007 f-150 regular
Prompt:
"This is an image confaining a car. Please identiffy the based the image"
Ours
appears regular version,
design elements such
Figure 6: Qualitative comparison across three domains-aircraff aircraft, birds, and cars. In each pair, ViRFT commits to a plausible but incorrect class with generic rationale such "BAe 146-300", "Great Crested Flycatcher", "2012 Ford F-150". Our method first enumerates top-k candidates and then applies suribute-groundee differential reasoning such as T-tail/registration cues for BAe 146- 200; Empidonax traits for Least Flycatcher; grille/headlight era cues for a 2007 F-150, yielding the correct fine-gr mine-grained label and justification aligned with the final choice.
(category_list refers the list of all the category list for the given dataset. During training, we only use base categories in the prompt.
Figure we provide the prompt used for second step of multiple choice question. Here {options} refers to the options obtained from the first step of our pipeline. In Fig- ure we provide the prompt used during the evaluation. (groundtruth) refers to the groundtruth category name and {prediction) refers to the model's 's predicted answer. We use "google/gemini- wtt.----11a---tteeeee-eeee openrouter (OpenRouter) 2024 API.
as LLM for evaluation from
17

---

Published as a conference paper at ICLR 2026
<image> This is an image containing a bird. Output the most likely species name in the image. The species name of the bird strictly belongs below category list (category_list)
Output the thinking process in <think> </think> and final answer in <answer> </ </answer> tags.
The output answer format should be as follows: species name </answer>,
<think> " </think> <answer>
Please strictly follow the format.
Figure 7: Prompt for first step of our pipeline to generate K rollouts
This is an image containing a bird. Please find the most likely bird the image from the below options. {options].
Please output the letter corresponding to the correct category name. Output the thinking process <think> </think> and final answer <answer> </answer> tags.
The output answer format should be as follows: <think> </think> <answer>option answr>
Please strictly follow the format.
Figure 8: Prompt for Multiple Choice Question (MCQ) answering
A.2 QUALITATIVE COMPARISON
We provide additional qualitative comparisons in Figure We note that explicit candidate enumer- ation followed by differential, attribute reasoning improves fine-grained recognition. In each case, the top rows (ViRFT) select look-alike but wrong categories ceaoorrie--vvr-geeeeeeeee gverreenerlliiing to BAe 146- 300, misidentifying flycatcher species, and over-estimating the truck's model year-Supported by broad, non-discrininative explanations. The bottom rows (Ours) surface a short top-k list and then contrast salient cues (e.g., taillengine/regiisrationn details for BAe 146-200; size/underparrs/EEmpii donax patterns for Least Flycatcher; grille and headlight silhouette for 2007 F-150) before com- mitting final answer. This two-step structure reduces over-generalizatioo and aligns the selected label with evidence visible in the image.
Table 10: Comparison of classification accuracy using Gemma3-122 -12B base model on CUB and Flow- ers datasets.
CUB
Flowers
Method
Base
Novel
HM
Base
Novel
HM
Gemma33 12B
38.67
32.00
35.03
71.26
86.52
78.16
1ommaa3-1289
46.00
37.17
41.12
69.54
86.67
77.17
DIVE-k (Ours)
58.50
84.63
87.38
85.99
A w.c. Gemmuö
20.17
8.83
13.07
13.37
0.86
7.83
A.3 ADDITIONAL BACKBONE RESULTS
To assess the model-agnostic nature of our approach, we also report results with replacing the QWEN2.5-VL backbone with Gemma3-12B and repeat the evaluation on CUB and Flowers. As shown in Table [0] DiVE-k consistently yields substantial improvements over the Gemma3-12n baseline across all splits. Overall, DiVE-k consistently improves the harmonic mean (HM) perfor- mance both datasets, achieving gains of +13.07 and +7.83 points the base Gemma3- 12B model. These results demonstrate that our framework generalizes beyond a specific foundation model and delivers consistent performance gains across different architectures and datasets.
18

---

Published as a conference paper at ICLR 2026
You are evaluating fine- tne-grained image classification results.
Given:
Groundtruth category: [groundtruth}
LLM prediction: {prediction)
Check if the groundtruth matches the prediction. The strings need not match exactly but
they must refer to the same specific fine-grained category, not just broad class. Respond with:
1. "True" or "False" if groundtruth matches the prediction in <answer></answes tag.
i.e <answer>answer here (Tru/Talse></anseer
2. Brief explanation in <exp_arratiooxxx/epplatttttiooo here</explanations
tag. i.e
sexplanation=-xpiinn
Figure 9: Prompt for evaluating fine-grained image classification results.
A.4 COMPUTATION COST COMPARISON
Since DiVE-k uses a two-step inference pipeline, it in- curs additional computational overhead due to for- ward passes. To quantify this overhead, we measure the per-sample inference time averaged over 500 samples on A6000 GPUs (48GB). As shown in Table I1] the av- erage per-sample inference time increases from 2.50s in the one-step setting to 12.95s in the two-step setting. However, these improvements are not merely a byprod- uet of increased computation. Even under identical compute budget, using K=1, which reduces our method single forward pass with greedy decoding, directly comparable prior one-step baselines, DiVE-k still outperforms existing methods on 4 out of 5 datasets (Figure This demonstrates that the gains arise from our formulation itself rather than additional compute alone. When more com- putation is permitted, performance further scales with K, providing a controllable trade-off between inference cost and accuracy that be adapted different application constraints.
Table 11: Per-sample inference time com- parison between one-step and two-step pipelines.
One Step
Two Steps
Flowers
2.50
16.02
CUB
2.77
14.58
8.25
Average
2,50
12.95
A.S LARGE LANGUAGE MODELS (LLMs) USAGE DETAIL.
We utilized LLMs as writing aid. Their application was strictly limited to proofreading for errors and polishing the prose for clarity and style. LLMs were not used for any substantive tasks, in- cluding but not limited to research, information retrieval, discovery, or the idcation of concepts and conclusions presented herein. All intellectual content is the original work of the authors.
19


Reviews:
[{"review_id": "3drtg9GGmh", "reviewer": "Reviewer_Gbyt", "summary": "The paper proposes DiVE-k framwork which uses the top-k generation of base model as a training signal for fine-grained image classification. For each training image, DiVE-k creates a multiple-choice question from the model's top-k outputs and uses RL to train the model to select the correct answer. Experiments on standard base-to-novel generalization and mixed-domain zero-shot base-to-novel generalization demonstrate the effectiveness of the proposed method.", "questions": "1. What is the performance if the ground truth label is already included in the options, i.e., the typical closed-world multiple-choice setting of evaluating LVLM's fine-grained classification performance?\n2. What is the performance if the model is trained to obtain options and do differential reasoning in one step?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. It transfers the open-world classification task into closed-world classification task, which is a promising way to settle the problem of brittle exact string-match reward.\n2. The evaluation metrics is reasonable for fine-grained classification task. Previous work typically uses string matching to evaluate the accuracy, while this paper uses the LLM to determine whether the prediction and ground truth belong to the same fine-grained category or not.", "weaknesses": "1. A direct way to construct the hypotheses set is to select the most similar top-k categories by CLIP text features, and the advantage of the proposed offline option mining lacks experimental support.\n2. Since the framework uses a two-step pipeline with chain-of-thought, it incurs additional computational cost due to the requirement of two forward passes.\n3. The final accuracy heavily depends on the recall of the first inference step, which is not presented in the experimental results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uVRCcIjg0G", "reviewer": "Reviewer_TDsH", "summary": "DiVE-K formulated finegrained classification task into a self-generated MCQ leveraging the base model reasoning ability. This forces the VLM to perform reasoning using a differentiable rewards, leading to better generalization. The paper is well written and the framework is empirically validated, marking it as a strong contribution. Only concern is, there is no/lack of evidences to show the failure scenarios of the proposed solution.", "questions": "1.  Could authors provide computational cost as it involves multiple step pipeline during training and inference?\n2. Could authors include zero-shot performance of the CLIP model on same classes (which can be obtained from original paper)? It will inform how models trained with differ in performance on same dataset?\n3. Why objective function is moved to supplementary?\n4. Does Table 1 has any results with supervised finetuning?If not, could authors include it as well.\n5. I’m unsure if this is a typo or an actual output. In Figure 3, all the reasoning steps of DiVE-k compare “X3” and “X6”, but the final prediction is B (which I assume is the second prediction from the top-k). However, the reasoning step itself contains a statement that it cannot be “X5”.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- **Formualation**: DiVE-K formulated the task using base model's top-k predictions as a source hard-negative examples to construct the MCQ, and leverages model's reasoning ability with a differentiable reward system, making it a highly effective training method.\n\n- **Differentiable Reasoning**- The MCQ format inherently encourages the model to focus on attribute level discriminative reasoning, which beneficial for semantically similar concepts.\n\n- **Reward**: Simple reward based on MCQ index selection overcomes the existing string matching proposal in previous RL based system.\n\n- **Experiments**: The proposed method is empirically evaluated and ablated under different settings. The performance of the DiVE-k is significantly improved with the proposed mechanisms making it as SOTA of the task.", "weaknesses": "- **Senstivity to roll outs and MCQ size**:  The performance of DiVE-k is heavily relies on K (number of rollouts) and m (Size of the final MCQ). While it is stated some processing to keep consistency, but there is no ablation on how variations in K and m affect the quality of the negative set and final performance.\n\n- **Failure cases**: There's systematic analysis/discussions  when this differentiable  reasoning analysis could fail, because this reasoning in next step relies on base model capacity of identifying and including the ground truth in the rollouts. Therefore, it naturally raises question, what is the bottleneck of the proposed solution: initial option mining or differential reasoning chain. For example, in Fig 5, performance on Pets dataset drops when increasing the  top-k generations. Why did that happen?\n\n- **Applicability**: As mentioned earlier, the performance of the base model could influence the final performance. Authors could consider testing the algorithms with different models to demonstrate the proposed solution as model-agnostic. \n\n- **Evaluation**: Figure 8: Provides a prompt intended to evaluate the fine-grained image classification results. Is this used for only property models or all the experiments?  If it is used for open-sourced model as well, what is the reason?. Given the prediction and groundtruth, performance score can be easily evaluated. Beyond accuracy, there's no other evaluation metric considered in the analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JhVtvIoE3g", "reviewer": "Reviewer_G36b", "summary": "This paper proposes DiVE-k, a framework to improve fine-grained visual recognition (FGVR) in Large Vision Language Models (LVLMs). The core idea is to address the model's inability to differentiate between visually similar categories. The method first uses an offline step where the base model generates $K$ rollouts to create a top-k set of candidate answers for each image. This set is then used to formulate a Multiple-Choice Question (MCQ). In the second step, the model is trained using Reinforcement Learning (RL) with a simple, verifiable reward (i.e., selecting the correct option letter) to perform differential reasoning among these pre-defined options. The experiments show strong performance gains over baselines like ViRFT and the base QWEN2.5-VL-7B model.", "questions": "* The core of the method is the offline top-k generation. Why was this choice made over a dynamic, multi-turn RL formulation where the policy first generates its own options and then selects from them within a single episode?\n* Could the authors please provide a comparison against a baseline where the options are generated dynamically by the current policy ($\\pi_\\theta$) at each training step, rather than fixed offline by $\\pi_{ref}$?\n* Please provide a comparison against a more complete \"multi-turn RL\" baseline, as described in Weakness #3. This seems like a more correct and challenging setup for this problem.\n*Given that the main change is reformatting the problem as an MCQ, how much of the gain is simply from this prompt engineering versus the RL training itself?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The primary strength of this paper is its strong empirical performance. The proposed DiVE-k method achieves significant improvements in base-to-novel generalization, mixed-domain, and few-shot settings, as shown in Tables 1, 2, and 3. The qualitative examples in Figures 4 and 9 are also compelling, illustrating that the model learns to perform more detailed differential reasoning when forced to choose from a set of plausible, similar options, which is a key challenge in FGVR.", "weaknesses": "Despite the strong results, I have major concerns about the methodological choices and novelty of this work.\n* Limited Novelty: The proposed method's novelty is marginal. At its core, it is a two-step process: 1) an offline data curation step that converts an open-ended generation task into a multiple-choice classification task, and 2) a standard RL training step (GRPO) on this new task. The \"differential reasoning\" appears to be a direct consequence of this prompt reformatting (from open-ended to MCQ), rather than from a new algorithmic insight.\n* Questionable Offline Top-k Generation: The decision to use a static, offline set of top-k options generated by the reference model ($\\pi_{ref}$) is a significant weakness. This means the policy ($\\pi_\\theta$) is trained on a fixed set of problems that were defined by an older version of itself. This introduces a distribution mismatch and severely limits the model's learning. The model is not learning to generate better candidates itself; it's only learning to rank a fixed set of candidates provided to it.Lack of a \n* Principled RL Formulation: A more sound and principled approach would be a dynamic, multi-step RL process. For instance, a multi-turn RL agent could first generate its own top-k candidates in a \"generation\" phase and then, in a \"reasoning\" phase, select the best one. The entire sequence would then receive a reward. The current method, by decoupling candidate generation (offline) from candidate selection (RL), feels like an ad-hoc pipeline rather than an end-to-end reasoning framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BhRj59qgFC", "reviewer": "Reviewer_CNJv", "summary": "The paper introduces the DiVE-k framework, which significantly enhances the base model's fine-grained image recognition capabilities by leveraging its top-k prediction results to construct Multiple-Choice Question (MCQ) data, and subsequently fine-tuning the model using the GRPO algorithm for Reinforcement Learning.", "questions": "What would the performance be if the obtained data were used directly for Supervised Fine-Tuning (SFT)? This would help in understanding the advantage of using RL.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Using the model's own top-k predictions as training data is interesting and insightful, which serves as a form of hard-negative mining against the model's confusions.\n2. DiVE-k achieves significant performance improvements over baseline models across multiple datasets and tasks.\n3. The overall writing and presentation of the paper is good.", "weaknesses": "1. Limited comparison with related work: The paper primarily contrasts its results with ViRFT. To comprehensively validate the efficacy of the proposed method, it should be compared against additional approaches mentioned in the \"Related Work\" section.\n2. Lack of diverse backbone models: The current experiments are exclusively conducted on Qwen2.5-VL. It is crucial to demonstrate the generalizability of DiVE-k by performing experiments on different LVLMs, such as confirming performance improvements on models like InternVL.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)