Title: Vision LLMs Are Bad at Hierarchical Visual Understanding, and LLMs Are the Bottleneck
Venue: NeurIPS 2025
Accepted: no
Keywords: Vision Large Language Model

Hierarchical Visual Understanding
Abstract: This paper reveals that many state-of-the-art large language models (LLMs) lack hierarchical knowledge about our visual world, unaware of even well-established biology taxonomies. This shortcoming makes LLMs a bottleneck for vision LLMs' hierarchical visual understanding (e.g., recognizing  $\texttt{Anemone Fish}$ but not  $\texttt{Vertebrate}$). We arrive at these findings using about one million four-choice visual question answering (VQA) tasks constructed from six taxonomies and four image datasets. Interestingly, finetuning a vision LLM using our VQA tasks reaffirms LLMs' bottleneck effect to some extent because the VQA tasks improve the LLM's hierarchical consistency more than the vision LLM's. We conjecture that one cannot make vision LLMs understand visual concepts fully hierarchical until LLMs possess corresponding taxonomy knowledge.

Abstract

This paper reveals that many state-of-the-art large language models (LLMs) lack hierarchical knowledge about our visual world, unaware of even well-established biology taxonomies. This shortcoming makes LLMs a bottleneck for vision LLMs' hierarchical visual understanding (e.g., recognizing Anemone Fish but not Vertebrate ). We arrive at these findings using about one million four-choice visual question answering (VQA) tasks constructed from six taxonomies and four image datasets. Interestingly, finetuning a vision LLM using our VQA tasks reaffirms LLMs' bottleneck effect to some extent because the VQA tasks improve the LLM's hierarchical consistency more than the vision LLM's. We conjecture that one cannot make vision LLMs understand visual concepts fully hierarchical until LLMs possess corresponding taxonomy knowledge.

1 Introduction

Taxonomy is natural and core in visual understanding. The biology taxonomies cover many objects in our visual world [53]; for example, a Boston Terrier belongs to the class of Terrier , which is a subtype of Dog , under Mammal , and ultimately part of the broader category Animal , forming a semantic path in the animal taxonomy: Animal → Mammal → Dog → Terrier → Boston Terrier . ImageNet [13] expands from the WordNet [37] taxonomy. Visual parts [28, 15, 3], attributes [14, 27, 41], and relationships [26] can be grouped hierarchically due to shared characteristics.

A high-performing, general-purpose visual understanding system should map visual inputs to both fine-grained leaf nodes of a taxonomy and coarse-grained inner nodes. Meanwhile, it should label an input hierarchically consistently along the path that traces a leaf up to the root. Figure 1 illustrates a case selected from our experiments that the model predictions lack hierarchical consistency , failing to follow the path of Animal → Vertebrate → Fish → Spiny-finned Fish → Anemone Fish .

36

37

38

39

40

41

42

43

Question:  Identify  the  taxonomic  group  of  the animal  in  the  image.  Answer  with  the  letter  of your chosen option.

Vertebrate

Bird

Fish

Arthropod

Spiny-finned

Fish

Soft-finned

Fish

Tench

Anemone

Fish

Lionfish

Butterfly

Insect

Level 1:

B. Invertebrate

Level 2:

Level 3:

A

.

Spiny-finned  Fish

Fish

D. Sea Anemone

Level 4:

A. Lionfish

B. Tench

Worm

Ground Truth

Prediction

Invertebrate

Animal

age classification [47, 44, 58, 61, 59] aims to classify visual inputs into semantically structured categories across multiple levels of specificity, in contrast to flat classification, which treats labels

as mutually exclusive and unstructured. We construct about one million four-choice visual question- answering (VQA) tasks from the hierarchical datasets (see Figure 1 for some examples). The tasks

traverse all taxonomy levels, and the four choices of an individual task are from the same level.

When evaluating VLLMs' performance over these tasks, we stress hierarchical consistency because it is unique to hierarchical visual understanding and crucial for adaptability to users' varying granu-

larity preferences [44, 12, 58].

Our main findings are as follows. First of all, many state-of-the-art VLLMs struggle with our VQA 44 tasks, substantially lacking hierarchical consistency. For example, Qwen2.5-VL-72B [4] makes 45 mistakes over 67% of the hierarchical paths in the iNaturalist [53] taxonomy. Moreover, in our 46 attempt to tracing down the error causes, we find that LLMs are the bottleneck and lack taxonomy 47 knowledge about the visual world. In contrast, the visual encoder and projector modules demonstrate 48 the ability to retain highly discriminative and well-structured visual features. We further show that 49 the LLM embeddings about the visual concepts contain sufficient hierarchical cues and organize 50 them orthogonally, but the model cannot decode them. Finally, finetuning a VLLM using our VQA 51 tasks enhance its LLM's (text) hierarchical consistency more than the VLLM's (visual) hierarchical 52 consistency, reaffirming LLMs' bottleneck effect to some extent. 53

54

2 VLLMs Lack Hierarchical Consistency in Visual Understanding

We construct six hierarchical image classification benchmarks in a four-choice VQA format to sys55 tematically assess VLLMs' accuracy and hierarchical consistency in visual understanding. These 56 benchmarks leverage datasets that inherently exhibit taxonomic structures, either derived from Word57 Net [37] or grounded in biological classification standards. In what follows, we formally define hi58 erarchical image classification, followed by two evaluation metrics about accuracy and consistency, 59 respectively. We then describe our VQA tasks and the first set of experiment results in this work. 60

61

2.1 Hierarchical Image Classification: Notations and Problem Statement

General image classification tasks typically assume a flat label space, where each image x ∈ X is 62 assigned a class label y ∈ Y out of a predefined set Y of mutually exclusive categories. However, 63 many real-world problems exhibit rich semantic structures, in which labels are naturally organized 64 into a hierarchy T = ( Y , E ) [44, 58, 61, 59], such as a tree or a directed acyclic graph. Here, E ⊆ 65 Y × Y denotes the set of directed edges representing parent-child relationships, where ( y i , y j ) ∈ E 66 indicates that y i is the parent of y j in the hierarchy. In hierarchical image classification, the objective 67 is not only to predict the leaf node label y ∈ Y leaf ⊆ Y but also to correctly recover its full ancestral 68 path ( y 0 , y 1 , · · · , y L ) in T , where y 0 denotes the root node and L is the depth of the hierarchy. In 69 this paper, we aim to evaluate VLLMs' hierarchical image classification capabilities, identify their 70 limitations and underlying causes, and enhance their performance based on these insights. 71

72

2.2 Two Evaluation Metrics about Accuracy and Consistency, Respectively

For evaluation, we mainly focus on the hierarchical consistency of model predictions [58, 43]. Be73 sides, we are interested in the leaf-level classification accuracy [68, 35, 20], which can be viewed as 74 the upper bound of the hierarchical consistency, detailed below. 75

Hierarchical Consistent Accuracy (HCA) [58, 43]. This metric is defined as 76

where N is the number of images in the testing set, L i denotes the depth of the hierarchy for the 77 i -th input x i and may vary for different tasks in uneven trees, f θ : X ↦→ Y is an image classifier, Y j 78 represents the set of labels at the j -th layer of the hierarchy, and [ · ] is an indicator function. HCA 79 evaluates whether a model's predictions are consistent with the entire hierarchical path from the root 80 to a leaf node. Specifically, it measures the proportion of samples for which all ancestor nodes along 81 the predicted paths match the ground truth. This is a stricter metric than flat accuracy and serves as 82 our primary evaluation criterion for hierarchical classification. 83

Leaf-Level Accuracy Acc leaf [68, 35, 20]. It cares about the predictions at the most fine-grained 84 level of a taxonomy: 85

Interestingly, Acc leaf upper-bounds HCA because correctly assigning a leaf label y L to an input x contributes to Acc leaf , but it does not increase HCA unless the model makes no mistake over all nodes in the path ( y 0 , y 1 , · · · , y L ) connecting the leaf label to the root.

86

87

88

89

90

91

92

93

94

95

96

97

2.3 VQA Tasks Derived from Hierarchical Image Classification Datasets

VLLMs are the image classifiers f θ in equations (1) and (2), and one can use language prompts to steer their output to a particular taxonomy level. More concretely, we formalize a VQA task for each image given a desired taxonomy level, ( x i , Y j ) , i = 1 , 2 , · · · , N, j = 1 , 2 , · · · , L i , as follows.

VQATasks. We derive approximately one million four-choice VQA tasks and six taxonomies from four hierarchical image classification datasets [54, 53, 13, 5] to evaluate VLLMs in a closed-set setting. This setting mitigates the challenge of open-set generation, which involves a prohibitively large prediction space [68] and ambiguous prediction granularity. We test different VQA prompts (provided in Appendix C), and they generally follow this format:

<image> Given the plant in the image, what is its taxonomic classification at the <hierarchy> (e.g., kingdom) level? A.<similar class> B.<ground truth> C.<similar class> D.<similar class> Answer with the option letter only. (Choices are shuffled in the experiments)

Arguably, the four-choice VQA tasks are easier than the conventional hierarchical image classifica98 tion, whose label space is orders of magnitude bigger than four. To compensate this difference, we 99 make sure the four choices are from the same level of a taxonomy and use 'confusing labels' in 100 the VQA tasks. Specifically, we use SigLIP [66] to compute the cosine similarity scores between 101 an image and all text labels other than the ground truth (at a particular taxonomy level), selecting 102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

the top three most similar labels as the distracting VQA choices. Besides, we provide the results of randomly sampled choices in Appendix B.

Hierarchical Image Classification Datasets. Table 1 summarizes the six taxonomies and four datasets we use to construct the VQA tasks. CUB-200-2011 (CUB-200) [54] is a fine-grained bird dataset containing 200 species. We prompt GPT-4o [1] to map each class to a four-level taxonomy: Order → Family → Genus → Specie . To ensure taxonomic accuracy, we cross-validate the generated hierarchy using corresponding entries from Wikipedia. In addition, we incorporate the iNaturalist-2021 (iNat21) dataset [53], a large-scale collection with species-level annotations spanning various biological taxa. We separate it into two taxonomies, Plant and Animal, comprising 4,271 and 5,388 leaf nodes, respectively, and six levels. Both CUB-200 and iNat21 provide wellestablished biological taxonomies with even hierarchical depths. To increase structural diversity, we also experiment with ImageNet-1K (ImgNet) [13], whose leaf labels are coarser-grained than iNat21 and CUB-200. ImgNet is built upon the WordNet [37]. We extract two relatively well-structured subsets from ImgNet: ImgNet-Animal and ImgNet-Artifact, following [58]. We further refine these subsets to improve label quality and semantic consistency. Food-101 [5] is about food classification, and its hierarchy is constructed based on the recent work of Liang and Davis [32].

2.4 Experiments and Findings

We mainly study state-of-the-art open-source VLLMs: The Qwen2.5-VL [4] models of 7B, 32B, and 72B parameters, InternVL2.5-8B [9], InternVL3-8B [72], and LLaVA-OV-7B [29]. Meanwhile, we include the proprietary GPT-4o's results for reference; in general, GPT-4o slightly outperforms Qwen-2.5-VL-72B, but the main findings below still apply. Finally, we experiment with two CLIPstyle [46] models, SigLIP-SO400M [66] and OpenCLIP-L [10], following the experiment protocol in [46] except that the candidate labels for each test image are restricted to the same four choices as fed to VLLMs. Table 2 shows the results about the models' hierarchical consistency (HCA) and leaflevel accuracy ( Acc leaf ) on iNat21, ImgNet, and CUB-200. The Food-101 results are in Appendix B to save space in the main text. We draw the following conclusions.

VLLMs Lack Hierarchical Consistency in Visual Understanding. Regardless of the leaf-level accuracy, all open-source VLLMs, CLIP models, and GPT-4o lack hierarchical consistency because their HCA is significantly lower than Acc leaf (up to 99.3% relatively). The gaps on iNat21-Plant are especially big (e.g., 32.82 vs. 55.00 for Qwen2.5-VL-72B and 35.53 vs. 62.95 for GPT-4o). While one might expect better results on ImgNet, neither open-source VLLMs nor GPT-4o can make their HCA match Acc leaf -more than 20% decrease for all models, indicating that VLLMs make many mistakes along the paths from the taxonomies' roots to the leaf nodes even when they are correct over the leaves.

Fine-Grained Visual Recognition Remains Challenging for VLLMs. While VLLMs and CLIP 137 models perform moderately well on ImgNet, they struggle with fine-grained object recognition; on 138

the iNat21 dataset, even the best-performing GPT-4o gives rise to only 63% leaf-level accuracy, far 139 from its 86% on ImgNet. Notably, InternVL2.5 and LLaVA-OV's results (about 27%) on iNat21 are 140 only slightly above random guess (25%), and the CLIP models are barely on par with random guess. 141 In contrast, a small task-specialized model [23] leads to 61.56% leaf-level accuracy on iNat21, and 142 some models [11, 69] achieve 93% accuracy on CUB-200, outperforming all the general-purpose 143 VLLMs in our experiments. These findings are consistent with the recent work [17, 68, 20, 63] that 144 recognizes the limitation of VLLMs on (fine-grained) image classification. 145

146

147

148

149

150

Scaling Laws Works for Hierarchical Visual Understanding. Both hierarchical consistency and leaf-level accuracy improve as the size of the Qwen2.5-VL series of models increases. Moreover, the gap between HCA and Acc leaf progressively narrows. However, the largest models (Qwen2.5-VL72B and GPT-4o) are still unsatisfactory in terms of both hierarchical consistency and fine-grained recognition, especially on the iNat21 benchmark.

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

Qwen2.5-VLs Are Among the Most Powerful Open-Source VLLMs. LLaVA-OV-7B's hierarchical consistency and leaf-level accuracy are below InternVLs and Qwen2.5-VLs. InternVL3-8B improves upon InternVL2.5-8B, but it is still under par with Qwen2.5-VL-7B.

3 Why Are VLLMs Poor at Hierarchical Image Classification?

We systematically investigate potential causes of VLLMs' low performance on hierarchical visual understanding. We first extensively study prompt variations in Section 3.1 and reveal that some prompts can lead to marginally better results than the rest, but the results remain generally bad. We then examine VLLMs' visual encoders and subsequent visual tokens to see whether and where essential visual information is lost when it forwards through VLLMs (Section 3.2). Interestingly, the discriminative cues in the visual tokens are maintained across various stages of the VLLM architectures, leading to about the same hierarchical image classification results immediately after the visual encoder, after the projection to the language token space, and at the very last layer of an LLM. Finally and surprisingly, we find that the generally believed powerful LLMs, even the one with 72B parameters in our experiments, lack basic taxonomy knowledge and are likely responsible for VLLMs' poor performance on hierarchical visual understanding! (We believe this conclusion is true for open-source VLLMs, but we urge readers not to extrapolate it to proprietary LLMs because we could not probe their intermediate embeddings.)

3.1 Language Prompts Are Not the Bottleneck

Prompt engineering often comes as a remedy for boosting VLLMs' performance in different appli169 cations [6, 55, 68, 58]. Could it also rescue VLLMs on our hierarchical visual understanding tasks? 170 We strive to test prompt variants comprehensively. We specify the taxonomy levels in the prompts 171 for CUB-200 [54] and iNat21 [53], whose taxonomies are grounded in biology. We even add CUB172 200's complete taxonomy as a JSON file to the prompts. For the other datasets with more generic 173 taxonomies, we test general and chain-of-thought [24, 57] prompts derived from the template in 174 Section 2.3. Appendix C provides all prompts in detail, and Figure 2 shows the results of some 175 high-performing prompts. We can see from the results that the prompt design alone is insufficient to 176 improve VLLMs' hierarchical consistency or leaf-level accuracy. 177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

3.2 Visual Embeddings Are Not the Bottleneck

The open-source VLLMs in this work vary in specific implementations, but their core components are the same: A visual encoder mapping images to embeddings, a projector translating visual embeddings into the language token space, and an LLM. If the hierarchical structure and discriminativeness are lost before the visual embeddings reach LLMs, the overall VLLMs would inevitably perform poorly on our hierarchical visual understanding tasks. Hence, it is crucial to examine the visual embeddings. We train three linear classifiers per taxonomy level to respectively probe the visual encoder, projector, and last layer of an LLM, where the image representations are an average of the visual tokens. Further details and results of the probing are provided in Appendix C.

Figure 3 shows the probing results of Qwen2.5-VL-7B over CUB-200 [54] and iNat21-Plant [53]. Remarkably, the linear classifiers outperform Qwen2.5-VL-7B all around. They achieve not only higher leaflevel accuracy than Qwen2.5-VL but also much better hierarchical consistency, even though the classifiers of different taxonomy levels are independently trained. Moreover, the linear probing results remain about the same at different stages of the forward propagation (i.e., immediately after the visual encoder, projector, and last layer of the VLLM), indicating that the visual tokens remain discriminative and structurally rich throughout different LLM layers. These results are a strong defense for the visual embeddings: They carry sufficient hierarchical and discriminative cues and should not be blamed for VLLMs' poor hierarchical visual understanding performance.

3.3 LLMs Are the Bottleneck in VLLMs' Hierarchical Visual Understanding

The huge discrepancy between the results of linearly probing visual tokens and VLLM performance in Figure 3 propels us to investigate other potential causes of VLLMs' low hierarchical consistency beyond the visual embeddings, and we find that the influential LLMs are the bottleneck.

3.3.1 Open-Souce VLLMs' LLMs Lack Taxonomy Knowledge

We separate LLMs from open-source VLLMs and examine how much they know about the taxonomies used in our experiments. Mechanically, we reformulate our VQA tasks to a text-only version by replacing the images with their corresponding leaf labels:

Given the <leaf node label> (e.g., Anemone Fish), what is its taxonomic classification at the <hierarchy> (e.g., kingdom) level? A.<similar class> B.<ground truth> C.<similar class> D.<similar class> Answer with the option letter only. (Choices are shuffled in the experiments)

This process results in about 0.7 million QA tasks after deduplication. We use them to assess LLMs 211 and report the (text) HCA results in Table 3 - we use (text/visual) HCA to refer to LLMs/VLLMs' 212 performance on text/visual QA tasks for clarity. We find that Qwen2.5-VL-7B's LLM achieves only 213 63.86% (text) HCA on CUB-200, whose taxonomy comprises merely four levels. The LLMs of 214 LLaVA-OV and InternVL-2.5 give rise to even lower (text) HCAs on CUB-200 (33% and 49%). 215 One might wonder if these low (text) HCAs are due to that the biology taxonomy underlying CUB216 200 is too specific for general LLMs. 217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

However, Table 3 further reveals that the LLMs also cannot perform well on ImgNet's general taxonomies. Besides, we progressively simplify our QA tasks by chopping the iNat21-Plant taxonomy level by level. Figure 4 plots the (text) HCA results, which increase as the taxonomy becomes shallower (and, correspondingly, the leaf nodes are less fine-grained). Still, they are below 90% regardless of the taxonomies' depths. There are noticeable drops at Levels 3 and 5 for Qwen2.5-VL and LLaVAOV's LLMs, implying that they pose more challenges than the other levels for the LLMs' hierarchical reasoning. These results are surprising to a large degree, given the recent success of LLMs over various benchmarks and domains [1, 50, 60, 33, 45].

Correlation between (text) HCA and Acc leaf -scaled (visual) HCA. An LLM's low (text) HCA undoubtedly discounts its corresponding VLLM's hierarchical consistency on visual inputs. We can quantify this notion using Pearson's correlation coefficient. Since the (text) HCA's corresponding leaf-level accuracy is 100% - we replaced images with their ground-truth leaf labels when making the text QA tasks, we normalize (visual) HCA by 1/ Acc leaf . The last column in Table 3 shows that the correlation between (text) HCA and Acc leaf -scaled (visual) HCA is as high as 0.9116.

A note about GPT-4o's (text) HCA. The analyses above apply to only open-source VLLMs because we cannot separate LLMs from the proprietary GPT-4o. Unlike the open-source LLMs' low (text) HCA, GPT-4o's (text) HCA scores are as high as 98.81. Hence, the LLM part is not GPT-4o's bottleneck in hierarchical visual understanding; instead, there are other possible causes of GPT-4o's hierarchical inconsistency about the visual world.

3.3.2 Why Are LLMs Poor at Hierarchical Text Classification?

In what follows, we present some preliminary quests into why and where LLMs fail at the seemingly simple hierarchical four-choice text classification tasks. We rule out the vision-language tuning that anchors visual encoders to pretrained LLMs and conclude that the language decoders are responsible for LLMs' lack of taxonomy knowledge.

Vision-Language Tuning Is Not the Reason. Acute readers likely have noted that our previous LLM results are about the LLM parts of VLLMs, not the 'true' standalone LLMs. Does the visionlanguage tuning, which is needed when one connects a visual encoder with an LLM, compromise LLMs and potentially induce catastrophic forgetting of taxonomy knowledge?

Weanswer this question by studying the original LLMs from which VLLMs are initialized, using the 252 same text-only hierarchical classification setup described in Section 3.3. Figure 5 (Left) compares 253 LLaVA-OV-7B and Qwen2.5-VL-7B's LLMs with their corresponding original LLMs. First of all, 254 we see that the original LLMs are on par with or even worse than their vision-tuned counterparts, 255 indicates that the standalone LLMs still lack a strong grasp of taxonomy knowledge. Interestingly, 256 Qwen2.5-VL's LLM actually outperforms its original LLM on all taxonomies; in other words, the 257 vision-language tuning actually enhances the LLM's (text) hierarchical consistency. In contrast, 258 LLaVA-OV's vision-language tuning weakens the LLM's (text) HCA. 259

LLMsEncode Hierarchical Structures Effectively but Cannot Decode Them Sufficiently. Next, 260 we shift attention to the LLM embeddings of the concepts in our taxonomies - if the embeddings 261 do not provide sufficient hierarchical structural cues, there is little chance LLMs can decode them. 262 To this end, we convert a taxonomy into language prompts of three variants: 263

Prompt 1: <leaf node label> (e.g., Blue Jay) belongs to the <hierarchy> (e.g., Order) of <ground truth> (e.g., Passeriformes). Prompt 2: Given the <leaf node label>, what is its taxonomic classification at the <hierarchy> level? It belongs to <ground truth>. Prompt 3: Given the <leaf node label>, what is its taxonomic classification at the <hierarchy> level?

We then train a linear classifier for each taxonomy level to probe the average embedding of the 264 language tokens in every layer of an LLM. Figure 5 (Right) summarizes the (text) HCA results of 265 Qwen2.5-VL-7B's LLM on iNat21-Plant: The text embeddings give rise to highly hierarchically 266 consistent linear probes. Especially for Prompt 3, with the ground-truth hierarchy labels withheld, 267 the linear probes that receive only the leaf node embeddings can still achieve near-perfect hierarchi268 cal consistency in the LLM's deeper layers. In other words, the specialized linear probes can decode 269 the taxonomy knowledge significantly better than the general-purpose LLM. 270

271

272

273

274

275

276

277

278

279

280

281

282

283

LLMs' Hierarchical Orthogonality Does Not Guarantee Hierarchical Consistency. Park et al. [42] recently predicted that LLMs represent hierarchical relations orthogonally in the representation space, e.g., animal is orthogonal to bird -mammal . They validated the prediction using Gemma [51] and LLaMA [19], and we further verify it in Figure 6 using both the original Qwen2.5-7B and the one after vision-language tuning. This pleasant geometric interpretation is, unfortunately, shadowed by the poor performance of Gemma and Qwen2.5-7B on our taxonomy QA tasks - we report the Gemma results in Appendix C. We argue that more fine-grained analyses of the LLM representation are required to establish a relationship between LLMs' hierarchical consistency and geometry.

Gemma-2B

OriginalLLMofQwen2.5-VL-7B

LLMofQwen2.5-VL-7B

7.5

7.5

7.5

2.5

5.0

animal

mammal

bird

mammal

5.0

2.5

bird

mammal

2.5

5.0

Cbird

mammal

0.0

bird

0.0

0.0

2.5

animal

2.5

animal

2.5

D

animal

5.0

5.0

5.0

7.5

7.5

7.5

6

-2

0

2

6

6

4

2

0

2

4

6

6

-2

0

2

4

6

4 LLMs Gain More Hierarchical Consistency than VLLMs from Finetuning

Could we improve the VLLMs' hierarchical visual understanding capabilities via finetuning using our VQA tasks built upon taxonomies? Likely, no, because LLMs are the bottleneck: The LLMs' hierarchical consistency over text-only tasks is so bad (Table 3) that we conjecture this shortcoming can only be fixed in the pretraining stage rather than the 'tail patching' finetuning stage.

Still, the following presents some LoRA-finetuning [22] experiments with Qwen2.5-VL-7B, the 284 best-performing 7B VLLM in our previous experiments, mainly for two reasons. One is to see 285 how much finetuning could help, even though we believe pretraining instead of finetuning should 286 be the rescue to VLLMs' hierarchical inconsistency. The other is further to investigate the interplay 287 between VLLMs and their LLMs - interestingly, our results reaffirm that LLMs are the bottleneck 288 for VLLMs' hierarchical visual understanding because LLMs' performance gain from the finetuning 289 upper-bounds VLLMs'. Our finetuning data consists of VQA tasks constructed from iNat21-Plant's 290 training set, covering 3,771 species nodes in the taxonomy instead of the full 4,271 species nodes. 291 We then evaluate the finetuned model's improvement on iNat21-Plant, its generalization to other 292 hierarchical visual understanding datasets, and how well it maintains the general vision-language 293 capabilities. Please see Appendix D for more details on the training. 294

Results and Discussion. Tables 4 shows that finetuning Qwen2.5-VL using the VQA tasks that 295 partially cover the iNat21-Plant taxonomy delivers improvements on both iNat21-Plant and other 296 datasets. On iNat21-Plant, HCA rises from 17 . 67 to 29 . 34 ( +11 . 67 absolute gain), while Acc leaf 297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

gains 6 . 05 . The HCA on ImageNet-Animal increases from 56 . 00 to 58 . 62 and on CUB-200 from 43 . 76 to 46 . 17 . More interestingly, Table 5 indicates that the LLM's (text) HCA increases more from the finetuning than Qwen2.5-VL's (visual) HCA (e.g., 20 . 66 vs. 11 . 67 on iNat21-Plant and 4 . 25 vs. 2 . 62 on ImgNet-Animal). To some extent, this finding reaffirms that LLMs are the bottleneck of VLLMs' hierarchical visual understanding, and one has to improve LLMs' (text) taxonomy knowledge to boost VLLMs' (visual) hierarchical consistency. Besides, our results demonstrate that vision-language training can benefit both VLLMs and their LLMs, aligning with some recent advocates for improving LLMs using multimodal data beyond language only [31, 52]. Appendix D reports more results and discussion, including that the finetuned model does not lose its general capability tested on MME [16], MMBench [36], and SEED-Bench [30].

5 Related Work

Hierarchical classification [47, 25] enables many applications. It is vital for a comprehensive understanding of the visual world [61, 43, 65, 48, 7, 44] and many language concepts [70, 56, 71, 21]. Several recent studies have revisited this longstanding problem and shown that CLIP-style [46] models lack consistency across taxonomic levels [58, 18]. Wu et al. [58] evaluate CLIP under multiple levels of semantic granularity and introduce a hierarchy-consistent prompt tuning method. Pal et al. [40] enhance CLIP's hierarchical representations by embedding them to a hyperbolic space. Xia et al. [59] further extend this direction by incorporating graph-based representation learning. Novack et al. [38] use hierarchical information to improve zero-shot classification accuracy. Zhang et al. [68] first identified the limitations of current VLLMs in fine-grained image classification. Building on this, Liu et al. [35] further assess a broader range of VLLMs. He et al. [20] point out a potential cause, the scarcity of image class names in pretraining. Beyond closed-set evaluation [63, 17], Conti et al. [12] benchmark VLLMs' open-world classification, while Snæbjarnarson et al. [49] propose to evaluate VLLMs' open-set predictions using a taxonomic similarity rather than exact string matching. However, to the best of our knowledge, no prior work has examined VLLMs under the hierarchical visual understanding context.

6 Conclusion

This work presents a systematic evaluation of state-of-the-art VLLMs's hierarchical visual under325 standing performance. We find that both open-source VLLMs and the proprietary GPT-4o give rise 326 to low hierarchical consistency over six taxonomies of visual concepts. Probing results reveal that 327 the visual and text embeddings carry rich hierarchical and discriminative cues, whereas the LLMs 328 fail to decode them, implying LLMs are the bottleneck. Finetuning on hierarchical VQA tasks im329 proves VLLMs' hierarchical consistency on visual inputs while preserving their performance on gen330 eral VQA tasks. Intriguingly, the finetuning benefits the LLMs (text) hierarchical consistency more 331 than the corresponding VLLM's (visual) hierarchical measure. Ingesting the taxonomy-knowledge 332 gap to LLMs, likely during pretraining rather than post-hoc patching, is a promising path toward 333 VLLMs that reason coherently across different levels of semantic granularity about the visual world. 334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

References

OpenAI (2024). Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024.

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision , pages 2425-2433, 2015.

Pablo Arbeláez, Bharath Hariharan, Chunhui Gu, Saurabh Gupta, Lubomir Bourdev, and Jitendra Malik. Semantic segmentation using regions and parts. In 2012 IEEE conference on computer vision and pattern recognition , pages 3378-3385. IEEE, 2012.

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923 , 2025.

Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision , 2014.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Jingzhou Chen, Peng Wang, Jian Liu, and Yuntao Qian. Label relation graphs enhanced hierarchical residual network for hierarchical multi-granularity classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4858-4867, 2022.

Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 , 2015.

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 , 2024.

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2818-2829, 2023.

Po-Yung Chou, Yu-Yung Kao, and Cheng-Hung Lin. Fine-grained visual classification with hightemperature refinement and background suppression. arXiv preprint arXiv:2303.06442 , 2023.

Alessandro Conti, Massimiliano Mancini, Enrico Fini, Yiming Wang, Paolo Rota, and Elisa Ricci. On large multimodal models as open-world image classifiers. arXiv preprint arXiv:2503.21851 , 2025.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE conference on computer vision and pattern recognition , pages 1778-1785. IEEE, 2009.

Sanja Fidler and Ales Leonardis. Towards scalable representations of object categories: Learning a hierarchy of parts. In 2007 IEEE Conference on Computer Vision and Pattern Recognition , pages 1-8. IEEE, 2007.

Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394 , 2023.

Gregor Geigle, Radu Timofte, and Goran Glava š . African or european swallow? benchmarking large vision-language models for fine-grained object classification. arXiv preprint arXiv:2406.14496 , 2024.

Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang. Hiclip: Contrastive language383 image pretraining with hierarchy-aware attention. arXiv preprint arXiv:2303.02995 , 2023. 384

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al385 Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. 386 arXiv preprint arXiv:2407.21783 , 2024. 387

388

389

390

Hulingxiao He, Geng Li, Zijun Geng, Jinglin Xu, and Yuxin Peng. Analyzing and boosting the power of fine-grained visual recognition for multi-modal large language models. arXiv preprint arXiv:2501.15140 , 2025.

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

Yuan He, Moy Yuan, Jiaoyan Chen, and Ian Horrocks. Language models as hierarchy encoders. Advances in Neural Information Processing Systems , 37:14690-14711, 2024.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR , 1(2):3, 2022.

Pranav Jeevan, Kavitha Viswanathan, Amit Sethi, et al. Wavemix: A resource-efficient neural network for image analysis. arXiv preprint arXiv:2205.14375 , 2022.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems , 35:22199-22213, 2022.

Aris Kosmopoulos, Ioannis Partalas, Eric Gaussier, Georgios Paliouras, and Ion Androutsopoulos. Evaluation measures for hierarchical classification: a unified view and novel approaches. Data Mining and Knowledge Discovery , 29:820-865, 2015.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision , 123:32-73, 2017.

Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE conference on computer vision and pattern recognition , pages 951-958. IEEE, 2009.

Daniel D Lee and H Sebastian Seung. Learning the parts of objects by non-negative matrix factorization. nature , 401(6755):788-791, 1999.

Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaVA-onevision: Easy visual task transfer. Transactions on Machine Learning Research , 2025. ISSN 2835-8856. URL https://openreview.net/forum?id= zKv8qULV6n .

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 , 2023.

Yunxin Li, Baotian Hu, Wei Wang, Xiaochun Cao, and Min Zhang. Towards vision enhancing llms: Empowering multimodal knowledge storage and sharing in llms. arXiv preprint arXiv:2311.15759 , 2023.

Tong Liang and Jim Davis. Making better mistakes in clip-based zero-shot classification with hierarchyaware language prompts. arXiv preprint arXiv:2503.02248 , 2025.

Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437 , 2024.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems , 36:34892-34916, 2023.

Huan Liu, Lingyu Xiao, Jiangjiang Liu, Xiaofan Li, Ze Feng, Sen Yang, and Jingdong Wang. Revisiting mllms: An in-depth analysis of image classification abilities. arXiv preprint arXiv:2412.16418 , 2024.

Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision , pages 216-233. Springer, 2024.

George A Miller. Wordnet: a lexical database for english. Communications of the ACM , 38(11):39-41, 1995.

Zachary Novack, Julian McAuley, Zachary Chase Lipton, and Saurabh Garg. Chils: Zero-shot image classification with hierarchical label sets. In International Conference on Machine Learning , pages 2634226362. PMLR, 2023.

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre 436 Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual 437 features without supervision. arXiv preprint arXiv:2304.07193 , 2023. 438

Avik Pal, Max van Spengler, Guido Maria D'Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, 439 and Pascal Mettes. Compositional entailment learning for hyperbolic vision-language models. arXiv 440 preprint arXiv:2410.06912 , 2024. 441

Mark Palatucci, Dean Pomerleau, Geoffrey E Hinton, and Tom M Mitchell. Zero-shot learning with 442 semantic output codes. Advances in neural information processing systems , 22, 2009. 443

Kiho Park, Yo Joong Choe, Yibo Jiang, and Victor Veitch. The geometry of categorical and hierarchical 444 concepts in large language models. arXiv preprint arXiv:2406.01506 , 2024. 445

446

447

Seulki Park, Youren Zhang, Stella X Yu, Sara Beery, and Jonathan Huang. Learning hierarchical semantic classification by grounding on consistent image segmentations. arXiv preprint arXiv:2406.11608 , 2024.

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

Seulki Park, Youren Zhang, X Yu Stella, Sara Beery, and Jonathan Huang. Visually consistent hierarchical image classification. In The Thirteenth International Conference on Learning Representations , 2025.

Qwen Team. Qwen3 technical report. https://github.com/QwenLM/Qwen3/blob/main/Qwen3_ Technical_Report.pdf , 2025. Last accessed: 14 May 2025.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PmLR, 2021.

Carlos N Silla and Alex A Freitas. A survey of hierarchical classification across different application domains. Data mining and knowledge discovery , 22:31-72, 2011.

Aditya Sinha, Siqi Zeng, Makoto Yamada, and Han Zhao. Learning structured representations with hyperbolic embeddings. Advances in Neural Information Processing Systems , 37:91220-91259, 2024.

Vésteinn Snæbjarnarson, Kevin Du, Niklas Stoehr, Serge Belongie, Ryan Cotterell, Nico Lang, and Stella Frank. Taxonomy-aware evaluation of vision-language models. arXiv preprint arXiv:2504.05457 , 2025.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 , 2023.

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 , 2024.

Haoqin Tu, Bingchen Zhao, Chen Wei, and Cihang Xie. Sight beyond text: Multi-modal training enhances LLMs in truthfulness and ethics. Transactions on Machine Learning Research , 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=2Zl0zc7fO8 .

Grant Van Horn, Elijah Cole, Sara Beery, Kimberly Wilber, Serge Belongie, and Oisin Mac Aodha. Benchmarking representation learning for natural world image collections. In Computer Vision and Pattern Recognition , 2021.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. Jul 2011.

Yubin Wang, Xinyang Jiang, De Cheng, Dongsheng Li, and Cairong Zhao. Learning hierarchical prompt with structured linguistic knowledge for vision-language models. In Proceedings of the AAAI conference on artificial intelligence , volume 38, pages 5749-5757, 2024.

Zihan Wang, Peiyi Wang, Lianzhe Huang, Xin Sun, and Houfeng Wang. Incorporating hierarchy into text encoder: a contrastive learning approach for hierarchical text classification. arXiv preprint arXiv:2203.03825 , 2022.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems , 35:24824-24837, 2022.

Tz-Ying Wu, Chih-Hui Ho, and Nuno Vasconcelos. Protect: Prompt tuning for taxonomic open set 485 classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 486 pages 16531-16540, 2024. 487

488

489

490

Peng Xia, Xingtong Yu, Ming Hu, Lie Ju, Zhiyong Wang, Peibo Duan, and Zongyuan Ge. Hgclip: exploring vision-language models with graph representations for hierarchical understanding. arXiv preprint arXiv:2311.14064 , 2023.

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 , 2024.

Kai Yi, Xiaoqian Shen, Yunhao Gou, and Mohamed Elhoseiny. Exploring hierarchical graph representation for large-scale zero-shot image classification. In European Conference on Computer Vision , pages 116-132. Springer, 2022.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the association for computational linguistics , 2:67-78, 2014.

Hong-Tao Yu, Xiu-Shen Wei, Yuxin Peng, and Serge Belongie. Benchmarking large vision-language models on fine-grained image tasks: A comprehensive evaluation. arXiv preprint arXiv:2504.14988 , 2025.

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9556-9567, 2024.

Siqi Zeng, Sixian Du, Makoto Yamada, and Han Zhao. Learning structured representations by embedding class hierarchy with fast optimal transport. arXiv preprint arXiv:2410.03052 , 2024.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1197511986, 2023.

Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multimodal models, 2024. URL https://arxiv.org/abs/2407.12772 .

Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. Why are visually-grounded language models bad at image classification? In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https: //openreview.net/forum?id=MwmmBg1VYg .

Zhicheng Zhang, Hao Tang, and Jinhui Tang. Multi-scale activation, selection, and aggregation: Exploring diverse cues for fine-grained bird recognition. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 10385-10393, 2025.

Jie Zhou, Chunping Ma, Dingkun Long, Guangwei Xu, Ning Ding, Haoyu Zhang, Pengjun Xie, and Gongshen Liu. Hierarchy-aware global model for hierarchical text classification. In Proceedings of the 58th annual meeting of the association for computational linguistics , pages 1106-1117, 2020.

Juncheng Zhou, Lijuan Zhang, Yachen He, Rongli Fan, Lei Zhang, and Jian Wan. A novel negative sample generation method for contrastive learning in hierarchical text classification. In Proceedings of the 31st International Conference on Computational Linguistics , pages 5645-5655, 2025.

Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479 , 2025.

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Claims are further disscussed in Section 2, Section 3 and Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Discussed in Appendix E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide detailed descriptions of evaluation benchmarks, model architectures, training protocols, and evaluation metrics to ensure reproducibility of the main results in the paper. The code will be released upon acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code and data will be made publicly available upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer:[Yes]

Justification: We provide detailed descriptions of data curation procedures, model architectures, evaluation metrics, and training protocols in corresponding sections. The code for all experiments will be released upon acceptance.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We did not include error bars or statistical significance analysis.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Discussed in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have carefully reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Discussed in Appendix F.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We properly credit all external assets used in this work, including datasets, pre-trained models, and code repositories. Licenses and terms of use are reviewed and respected, with appropriate citations provided in the paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We provide detailed descriptions of the hierarchical VQA benchmark curation process, model architecture, training protocols, and evaluation metrics in the corresponding sections. All data, code, and model checkpoints will be released upon acceptance.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

843

844

845

846

847

848

849

850

851

852

853

854

855

856

857

858

859

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [Yes]

Justification: Our evaluation involves vision large language models (VLLMs) with large language model (LLM) backbones as a core component. We describe how the LLM is used for hierarchical classification and detail its role in the architecture, prompting, and evaluation.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the hierarchical reasoning capabilities of vision-language models, primarily vision LLMs (VLMMs), finding that they have poor performance on tasks requiring reasoning over taxonomies. Various tests including probing experiments suggest that the bottleneck stems from their LLM components, and examine the effect of fine-tuning VLLMs on hierarchical data.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nUnderstanding the hierarchical reasoning capacity of SOTA VLMs is important, and does seem to address a gap in the existing literature which mostly focuses on hierarchical understanding in CLIP-style discriminative models.\n\n**Weaknesses:**\n\nWhile the chosen research question is interesting and addresses an important gap in the literature, there are experimental and stylistic issues which should be addressed:\n\n*Experimental issues:*\n\nWhile the paper claims that “Vision LLMs are Bad at Hierarchical Visual Understanding” (including in the title), the tests in Section 2 are mostly on biological taxonomies, as well as a few other taxonomy types (Food, WordNet) which are limited to words or short labels and have a large proportion of obscure or technical terms. No human baseline is given, but it is doubtful that humans would typically succeed in identifying e.g. whether “Anseriforme” correctly describes an animal. Prior work [1, 2] has found foundation VLMs can have hierarchical understanding, particularly of longer, more natural caption texts, but this is not tested (e.g. against the HierarCaps benchmark of [2]).\n\nThe claim that visual embeddings are not the bottleneck to hierarchical understanding (Sec 3.2) uses linear probing, but this is a supervised method rather than the zero-shot classification used in Section 2. This gives a significant advantage to learning the label hierarchy and thus I do not find this experiment to conclusively show that visual embeddings inherently encode this hierarchical knowledge better than LMMs.\n\nThe findings in Section 4 seem to be of limited significance since it is expected that fine-tuning on a task with its particular label set will outperform zero-shot classification. In addition, if the claim on L282-3 is included, it should be supported by experimental evidence.\n\n*Stylistic issues:*\n\nThe paper’s organization and writing style distract from its contents and would benefit from revision. For example, long section titles (e.g. “2 VLLMs Lack Hierarchical Consistency in Visual Understanding”) state conclusions before evidence is presented, rather than using a more standard naming convention, and the first lines of the abstract refer only to LLMs while the paper focuses on multi-modal VLMMs. The internal structure and tone of the paper’s text can also be refined for clarity.\n\nThere are also missing references and discussion of existing works on hierarchical understanding in foundation vision-language models [1, 2].\n\n[1] Desai et al. Hyperbolic image-text representations. ICML 2023\n\n[2] Alper et al. Emergent Visual-Semantic Hierarchies in Image-Text Representations. ECCV 2024\n\nQUESTIONS:\nWhat is the baseline human performance for the tasks in Section 2?\n\nHave you tested SOTA uni-modal LLMs (rather than LLM components of VLLMs) for similar hierarchical understanding?\n\nWhat possible applications would be enabled by improving hierarchical reasoning in VLLMs?\n\nLIMITATIONS:\nI appreciate the limitations section in Supp. E. However, it is missing some limitations raised above, such as testing only on taxonomies with short labels (as discussed above).", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies whether VLLM systems can answer questions about  hierarchical (IS-A) relations. The authors create a large set of VQA tasks drawn from hierarchies and datasets, and then test performance.  They find performance to be poor, and then investigate the locus of the problem, for example by probing the internal representations of the VLLM at various stages.  They find that the culprit is the LLM, suggesting that LLMs lack hierarchial knowledge about the visual world.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper asks an important question (hierarchical visual knowledge in VLLMs), and comes up with an interesting and important finding (a lack of this knowledge in current VLLM models)  and  a lovely causal analysis (localizing the cause of the flaw to the LLM component)\n\n2. The experiments are comprehensive and convincing.\n\n3. The paper is beautifully and clearly written and I enjoyed reading\n\nWeaknesses:\n\n1. The paper overstates its claims; the results that the LLM is the bottleneck hold only for the open-source models studied. As the paper makes clear at line 240, the results do not hold for GPT-4o, but only for the open models. Of course since GPT-4o is closed, it's reasonable that the authors can't say why GPT-4o can do the text hierarchy task but not the vision hierarchy task.  Still, this should be more clearly explained throughout the paper. Currently the sentence at line 165 and the sentence at line 240 are the only place where I was given to understand as a reader that the paper's causal conclusions don't hold for the closed models (and the sentence at line 165 and 240 don't seem completely consistent with each other in their implications).  The abstract was careful to use the word \"many\" in \"many state-of-the-art LLMs\", which is a good start, but the reader could legitimately feel deceived. The paper is a wonderful paper, so adding more careful and thoughtful caveats and clear definitions in the right places would help a lot without lessening the findings.\n\nQUESTIONS:\n1.  The authors should fix the weakness noted above\n\n2. The authors could increase their discussion of why there are performance differences on different datasets. For example, the difference in performance between ImageNet-Animal,  and Image-Net-Artifiact is likely that ImageNet-Animal, the leaf node is often the basic-level category, but in ImageNet-Artifact, the leaf node is often subordinate-level, and the basic category is one of the intermediate nodes, making the task much harder. \n\n3. The authors used zero-shot prompting for section 3.3.  Would the results change and performance increase if few-shot prompting was used instead?\n\n4. The authors seem to have missed much of the prior literature on investigating hierarchies in text language models. It's not that extensive, and a lot of it is old and uses BERT, but still. Some papers:\n\nhttps://aclanthology.org/2023.findings-ijcnlp.12.pdf\n\nhttps://aclanthology.org/2022.acl-short.11/\n\nhttps://arxiv.org/pdf/2302.06761\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors probe the hierarchical understanding of four visual LLMs, two CLIP-style models, and GPT-4o. To do so, they derived ~1 million visual question answering tasks from three biological datasets (the animal and planet split of iNat plus CUB-200), two splits of ImageNet (Animals and Artifact), and Food-101. The multiple choice questions generally ask the model to select the taxonomic classification of an image at a given level of the taxonomic tree from a list of four options: the ground truth and 3 similar labels selected a the corresponding level of the taxonomic tree. SigLIP was used to evaluate the similarity between an image and text labels to find potentially difficult options. The authors measured low performance across all the models they tested as measured by Hierarchical Consistent Accuracy and Leaf-Level Accuracy. The models typically returned higher leaf accuracy while returning poor hierarchical results, indicating they made many mistakes along the taxonomic tree. All models did better with ImageNet derived datasets and struggled with the fine-grained data in iNat. GPT-4o generally performed better than open-source models, but still struggled with taxonomy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \nThe authors clearly articulate their experiments and provide a thought provoking probe of visual LLMs understanding of taxonomy. They measure an interesting discrepancy in several large model's hierarchical performance and understanding, even when returning reasonable leaf accuracy. The experiments do a good job of illuminating possible points of breakdown.  \n\nWeaknesses:\nThe authors do not grapple with domain specific models that may provide further insight into the shortcomings they identified.\n\nQUESTIONS:\n1. How would domain specific models fare in this problem setting, particularly something like BioCLIP that encodes taxonomy by design? Is that the sort of pretraining vs fine tuning referred to at line 286? The [BioCLIP paper](https://openaccess.thecvf.com/content/CVPR2024/html/Stevens_BioCLIP_A_Vision_Foundation_Model_for_the_Tree_of_Life_CVPR_2024_paper.html) reports only that they observe good separation of hierarchical labels in a t-SNE projection. Perhaps this is a good opportunity to probe that.\n2. How does the hierarchy depth compare across your datasets? And what refinements were implemented to 'improve label quality and semantic consistency'?\n3. Will the code to produce the questions (or the questions themselves) be made available?\n\nLIMITATIONS:\nThe authors put their discussion of limitations in the appendix. I recognize space is limited, but I'd encourage the author to include some of the material (or at least a reference to the section of the appendix) in the main body. The points made in the appendix are worth grappling with.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes off-the-shelf VLM performance on hierarchical visual understanding. By hierarchical, the authors actually imply taxonomical classification of animals and plants, which is a subtle but significant difference in the claim of the paper. The experiments conducted are meaningful and sound, with multiple datasets and configurations tested. The authors are convinced that LLMs are mostly responsible for misclassifications rather than the VLMs or more specificaly the vision tokens are ample for this task.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Large-scale, hierarchy-aware benchmark: curates about 1 M  data points across six taxonomies (~10k leaf classes) and evaluates with both Acc_leaf and HCA, giving the community the first systematic testbed for fine-grained hierarchical VQA \n2. Clean ablation pinpoints the bottleneck: independent linear-probe study shows vision tokens are already linearly separable at every taxonomic level, shifting blame from the image encoder to the language head and sharpening the paper’s core insight \n\nWeakness:\n1. My primary concern is with the clarification of the objective of the work. Even though the authors claim that VLMs are bad at \"Hierarchical Visual Understanding\", a more accurate version would be that VLMs are bad at Taxonomic understanding in animal images. The 1M VQA pair datasets are heavily dominated by animal images and taxonomic-related questions, with only Imagenet-Artifact being a general-purpose dataset. \n2. While I understand the purpose of probing the vision tokens - I have siginificant reservations around experiment in Section 3.2. Even though linear probes classify each level correctly, the three classifiers are trained separately for kingdom, phylum, etc. so they never have to respect parent-child constraints. A probe can be 100 % accurate at every level yet still assign mutually incompatible ancestors (e.g. Invertebrate parent of Fish) because cross-level consistency is never evaluated. On similar lines, a better experiment would be to probe vision tokens in a path like (Fish|Inertebrate) and (Fish|Vertebrate). If the vision part is still solid in differentiating them, that would conclude that the vision tokens are not responsible.\n3. Table-4: \"Our finetuning data consists of VQA tasks constructed from iNat21-Plant’s 291 training set, covering 3,771 species nodes in the taxonomy instead of the full 4,271 species nodes\". Well this observation is pretty much in-line wuth what should be expected? If the VLMs are not able to understand the difference b/w Annelids and Arthropods, how can they be \"taught\" this by fine-tuning on *plant species*? As you see in Table-4, performance on iNat21-Plant improves by a lot as compared to CUB. I am sure more training data and prompt structuring can push gains even further but never improve animal specie performance.\n4. (minor) Instead of just measuring raw accuracy, can the authors also report the accuracies measured through VQAScore? There is usually a 5-6% jump when actually looking at probabilities than tokens.\n5. Lastly it would be interesting to see what level the LLM/VLM misses on the most. How different is the probability of token prediction of vertebrate/invertebrate?\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}