Title: Decomposing Visual Classification: Assessing Tree-Based Reasoning in VLMs
Primary area: datasets and benchmarks
Keywords: Vision-Language Models, Hierarchical Classification, Sensitivity Analysis

Abstract: Vision language models (VLMs) excel at zero-shot visual classification, but their performance on fine-grained tasks and large hierarchical label spaces is understudied. This paper investigates whether structured, tree-based reasoning can enhance VLM performance. We introduce a framework that decomposes classification into interpretable decisions using decision trees and evaluates it on fine-grained (GTSRB) and coarse-grained (CIFAR-10) datasets. Although the model achieves 98.2% accuracy in understanding the tree knowledge, tree-based reasoning consistently underperforms standard zero-shot prompting. We also explore enhancing the tree prompts with LLM-generated classes and image descriptions to improve alignment. The added description enhances the performance of the tree based and zero-shot methods. Our findings highlight limitations of structured reasoning in visual classification and offer insights for designing more interpretable VLM systems.

TL;DR: More structure ≠ better performance. Sometimes the simplest approach (zero-shot prompting) is genuinely superior, and added complexity just creates more opportunities for failure.