Democratizing Fine-grained Visual Recognition with Large Language Models
Authors: Mingxuan Liu
Subhankar Roy
Wenjing Li
Zhun Zhong
Nicu Sebe
Elisa Ricci
Venue: ICLR 2024 poster (2024)
Keywords: Vision-Language Models
Large Language Models
Prompting
Multimodal
Fine-grained Visual Recognition
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose Fine-grained Semantic Category Reasoning (FineR) system to address fine-grained visual recognition without needing expert annotations. FineR leverages the world knowledge of large language models to reason fine-grained category names.

Abstract: Identifying subordinate-level categories from images is a longstanding task in computer vision and is referred to as fine-grained visual recognition (FGVR). It has tremendous significance in real-world applications since an average layperson does not excel at differentiating species of birds or mushrooms due to subtle differences among the species. A major bottleneck in developing FGVR systems is caused by the need of high-quality paired expert annotations. To circumvent the need of expert knowledge we propose Fine-grained Semantic Category Reasoning (FineR) that internally leverages the world knowledge of large language models (LLMs) as a proxy in order to reason about fine-grained category names. In detail, to bridge the modality gap between images and LLM, we extract part-level visual attributes from images as text and feed that information to a LLM. Based on the visual attributes and its internal world knowledge the LLM reasons about the subordinate-level category names. Our training-free FineR outperforms several state-of-the-art FGVR and language and vision assistant models and shows promise in working in the wild and in new domains where gathering expert annotation is arduous.

Published as a conference paper at ICLR 2024
DEMOCRATIZING FINE-GRAINED VISUAL RECOGNI- TION WITH LARGE LANGUAGE MODELS
Mingxuan Liu', Subhankar Roy", Wenjing Li Zhun Zhong Nicu Sebe', Elisa Ricci
1 University of Trento, Trento, Italy 2 Fondazione Bruno Kessler, Trento, Italy
3 Hefei University of Technology, Hefei, China 4 University of Aberdeen, Aberdeen, UK
5 University of Nottingham, Nottingham, UK
University of Leeds, Leeds, UK
Reasoning Each Sample
Test Images
: sce bird in a photo. How to distinguish specific species?
Well, Could you describe this photo and its wing head pattern, .... primary color?
Certainly. The bird perched
DC VLM
Obsorvations
branch amidst the falling snow. Its wings are grey, and boasts a black and red pattern on Its head. Notably, Its dominant color red.
Answering Model
Cachaoll
Cardinal
Perfect. Evan though can't It, but based on your description, think the you see would be Pyrruuooiia, Cardinal, or Summer Tanager.
Red Eyed Vireo
Lincotn Sparrow
Large Language
Blackblrd
Trocical Kingbird
Vision-Langauge Model
Semantic Classification
Reasoned Concopts
Reasoning Concepts from Observations
Inference
Figure 1: An overview of our proposed fin-grained visual recognition (FGVR) pipeline. Left: Given few unlabelled images we exploit visual question answering (VQA) and large language models (LLM) to reason about subordinate-level category names without requiring expert knowledge. Right: At inference, we utilize the reasoned concepts to earry out FGVR via zero-shot semantic classification with vision-language model (VLM).
ABSTRACT
Identifying subordinate-level categories from images is a longstanding task in computer vision and referred to fine-grained visual recognition (FGVR). It has tremendous significance in real-world applications since an average layperson does not excel differentiating species of birds or mushrooms due subtle differences among the species. A major bottleneck in developing FGVR systems is caused by the need of high-quality paired expert annotations. expert knowledge we propose Fine-grained Semantic Category Reasoning (FineR) that internally leverages the world knowledge of large language models (LLMs) as proxy in order to reason about fine- ane-grained category names. In detail, to bridge the modality gap between images and LLM, we extract part-level visual attributes from images text and feed that information attributes and its internal world knowledge the L.M reasons about the subordinate- level category names. Our training-free FineR outperforms several state-of-the-an FGVR and language and vision assistant models and shows promise in working in the wild and in new domains where gathering expert annotation is arduous.
circumvent the need of
L.LM. Based on the visual
INTRODUCTION
Fine-grained visual recognition (FGVR) is an important task in computer vision that deals with identifying subordinate-lewe categories, such as species of plants animals (Wei al., 2021). It is challenging due to the fact that different species of birds can differ subtle attributes, such as the Lincoln's Sparrow mainly differs from Baird's Sparrow in the coloration of the breast pattern (see Fig. 2). Due to small inter-class and large inner-class variations, the FGVR methods typically require auxiliary information namely part annotations (Zhang et al., 2014), attributes (Vedaldi et al., 2014), natural language descriptions (He & Peng, 2017) collected with the help of experts in the respective fields. Hence, this expensive expert annotation presents as bottleneck and prevents FGVR in new domains from being rolled out as software or services to be used by common users.
On the contrary encyclopedia of textual information about plants and animals can be found on the internet that document at great lengths about the characteristics and appearance of each species. In
"Coresponding authors.
Code is available https://proj http//ooffiiier..erppppp github.iia

---

Published as a conference paper at ICLR 2024
other words, modemn day Large Language Models (LLMs) (e.g., ChatGPT (OpenAI, 2022)), trained on enormous internet-scal corpuses, already encode in its weights the expert knowledge that is quintessential in FGVR. In this work we ask ourselves, unpaired auxiliary information already contained in I.IMs? The implications are tremendous as endows average layperson with the expert knowledge of a mycologist, ornithologist and on.
we build FGVR systems by exploiting the
Leveraging unlabelled FGVR dataset and the LLM build a FGVR model not straightforward as there exists a modality gap between the two. In detail, a LLM cannot see, as it can only ingest text (or prompts) at input and give text at output. To bridge this modality gap, we first use a general purpose visual question answer- ing (VQA) model (e.g., BLIP-2 (Li et al., 2023)) to extract a set of visual attributes from the images. Specifically, for every im- age in a bird dataset we prompt BLIP-2 to extract part-level descriptions of wing color, head pattern, tail pattern and so on. We then feed these attributes and their descriptions to LLM, and prompt it to reason on a set of candidate class names using its internal knowledge of the world. As final step, we use the candidate class names to construct a semantic classifier, as commonly used in the vision- language models (VLMs) (e.g., CLIP (Radford et al., 2021)), to classify the test images. We call our method Fine-grained Semantic Category Reasoning (FineR) as it discovers concepts in unlabelled fine-graine datasets via reasoning with LLMs (see Fig. 1).
Super-category:
Bird Lincoln's Sparrow
Bird
Ground-truth:
Baird's Sparrow
BLIP-2
Bird
Bird
Sparrow
Sparrow
LLaVA:
Bird
Bird
Sparrow
Sparrow
LENS:
Bird
Bird
Vesper Sparrow
Olay-colored Sparrow
MinIGPT-4:
Bird
White-throated Swainson Hewk
White-rumped Sandpiper
FineR (Ours): Uncoln's Sparrow
Bird
Bird
Baird's Sparrow
Figure 2:
Comparing our proposed FineR with the state-
of of-the-art visual question answering models: BLIP-2 (Li al., 2023), LLaVA (Liu et al., 2023a), LENS (Berrios et al., 2023), and MiniGPT-4 (Zhu et al., 2023b).
Through our proposed FineR system we bring together the advances in VQA, LLMs and VLMs to address FGVR that was previously not possible without accessing high-quality paired expert annotations. Since our proposed FineR does not require an expert's intervention, it can be viewed as step towards democratizing FGVR systems by making them available to the masses. In addition, unlike many end-to-end systems our FineR is interpretable due to its modularity. We compare our method with FGVR, vocabulary-free classification and multi-modal language and vision assistant models (e.g., BLIP-2) and show its effectiveness in the challenging FGVR task. As shown in Fig. 2 our FineR more effective than the state-of-the- methods, which are accurate at predicting the super-category but fail at identifying the subordinate categories. Unique to our FineR, it works in a few-shot setting by accessing only a few unlabelled images per category. This a very pragmatic setting as fine-grained classes in real-world typically follow an imbalanced class distribution. Finally, our method is completely training-free, which is the first of its kind in FGVR. Extensive experiments on multiple FGVR benchmarks show that our FineR is far more effective than the existing methods.
In summary our contributions are: (i) We introduce the challenging task of recognizing fine-grained classes using only few samples per category. where neither labels nor paired expert supervision is available. (ii) We propose the FineR system for FGVR that exploits a cascade of foundation models to reason about possible subordinate category names in image. (iii) With extensive experiments on multiple FGVR benchmarks we demonstrate that our FineR is far more effective than the existing approaches. We additionally introduce new fine- ane-graieed benchmark of Pokemon characters and show that FineR exhibits the potential to work truly in tee-the-wild, even for fictional categories.
2 METHODOLOGY
Problem Formulation. this work, we study the problem of fine-grained visual recognition (FGVR) given an unlabeled dataset Dirain of fine-grained objects. Different from the traditional FGVR task (Wei et al., 2021) we do not use any label or paired auxiliary information annotated by an expert (Choudhury et al., 2023), and Dinin contains a few unlabelled samples per category (Li et al., 2022). Our goal is to learn a model that can assign correct labels to the images in the test set Diest, In detail, when presented with test image we output semantic class name C. where C is the set of class names that are not known a priori, making it vocabulary- (Conti et al., 2023). Our proposed FineR first aims to generate the set C and then assigns the test image to one of the classes. As
2

---

Published as a conference paper at ICLR 2024
example, opposed to the FGVR methods that output class-index inference, our FineR directly outputs a semantic class name (e.g.. Blue-winged Warbler). Next we discuss some preliminaries about the pre-trained models that constitute proposed FineR.
2.1 PRELIMINARIES
Large Language Models (LLMs), such as ChatGPT (OpenAI, 2022) and LLaMA (Touvron et al., 2023), are auto-regressive models that are trained on large internet-scale text corpuses possess a wealth of world knowledge encoded in their weights. These models can be steered towards a particular task by conditioning on demonstrations of input-output example pairs, relevant to the task at hand, without the need of fine-tuning the weights. This emergent behaviour is referred to as in-context learning (ICL) (Dong et al., 2022; Brown et al., 2020). The LLMS have also demonstrated adequate reasoning abilities in answering questions when prompted with 2022), another emergent behaviour that we exploit in this work. Formally, a LLM h / سا + pi takes text sequence (or LLM-prompt) سلر at input and maps it to text output sequence T T
prompts (Wei et al.,
Vision- Language Models (VLMs), such as CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), consist of image and text encoders that are jointly trained to predict the correct pairings of noisy image and text pairs. Due to well-aligned image-text representation space, they demonstrate excellent zero-shot transfer performance on unseen datasets, provided the class names of the test set is known a priori (Radford et al., 2021). A VLM hymm maps an image x to a category c in a given vocabulary set C based on the maximum cosine similarity:
è = arg (Eigx(ing(X)) Ema(c)), cEC
(I)
where è is the predicted class, and Eing and Extt denote the vision and text encoder of the VLM, respectively. (: represents the cosine similarity function.
Visual Question Answering (VQA) models, such as BLIP-2 (Li et al., 2023) and LLaVA (Liu et al., 2023a), that combine LLM and VLM in a single framework, learn not only align the visual and text modalities, but also to generate the caption of the image-text pair. This visual-language pre-training (VLP) then used to perform VQA one of the downstream tasks. In detail, at inference a VQA model hmp; : (I, pT))) + T receives an image I and textual question (or VQA-promphl nT as inputs, and outputs a textual answer 7 that conditioned on both the visual and textual inputs.
While these VQA models excel at identifying super-categories and general visual attributes (e.g.. color, shape, pattern) of objects when prompted with a question, they fail to recognize many subordinate- level categories (see Fig. 2) due to lack of well-curated training data. We show how the aforementioned pre-trained models can be synergistically combined to yield a powerful FGVR system.
2.2 FINER: FINE-GRAINED SEMANTIC CATEGORY REASONING SYSTEM
To recall, our goal is to recognize the fine-grained categories using unlabelled images in Dirain without having access to expert annotations and the ground-truth categorics in C. Thus, the challenges in such FGVR task are to first identify (or discover) the classes in C. and then assign class name e E C to each instance in Diest The idea underpinning our proposed Fine-grained Semantic Category Reasoning (FineR) is that the LLMs, which already encode the world knowledge about different species of animals and plants, can be leveraged to reason about candidate class names C+ when prompted with visual attributes extracted from the VQA models. Concretely, FineR takes a few unlabeled images in Dirain as input, and outputs the discovered class names C*. Subsequently, the discovered candidate class names and images in Dirain are utilized to yield a multi-modal classifier W mm to classify test instance using VLM. Therefore, our FineR aims to learn mapping:
FineR Dimin (Ō', Wmm)..
where FiineR a meta-controller that encapsulates our system. FineR operates in three phases: (i) Translating Useful Visual Information from Visual to Textual Modality; (ii) Fine-grained Semantic Category Reasoning in Language; and (iii) Multi-modal Classifier Construction. An overview of our system is depicted in Fig. 3 and the details follow next.
2.2.1 TRANSLATING USEFUL. VISUAL INFORMATION FROM VISUAL TO TEXTUAL MODALITY
Before we can tap into the expert world knowledge inherent to the LLMs for facilitating FGVR, we need a mechanism to translate the visual information from the images into an input format which the
3

---

Published as a conference paper at ICLR 2024
I: Translate Useful Visual Information from Visual to Textual Modality (Sec 2.2.1)
Inference (Sec 2.3)
Dirain
PNN
Uon Acquire Expert Knowledge
Dereis
Idensfl Super Category
g
g
Perceive Visual Attribtt
8
A
A
Multi-modal Classifier Construction (Sec 2.2.3) Fine-grained Semantic Category Reasoning (Sec 2.2.2)
W
9
W mase
MMC
Ch
NND Novey Name
Br Reasoring Class Names
A
Construci Multi-
modal Clasatter
Dirain
Hefinoment
pftrats
Predicted
Category
Drrsin
Few uniabalind images
Super-Gallegay rimme (e.g.. Bind)
A
Attributes set (e.g Back Color)
Freashly reasored callegory names
x
image
Attiibute description
(e.g. Grayy
ru
Multi-modal classifier
Reffned reasoned canegory names
Figure 3: The pipeline of the proposed Fine-grained Semantic Category Reasoning (FineR) system.
LLM understands. Furthermore, we require an understanding about which visual cues are informative at distinguishing two fine- grained categories. For instance, what separates two species of birds (Lincoln's 's Sparrow vs Baird's Sparrow) drastically different than what separates two types of mushrooms (Chanterelle vs Porcini). Motivated by this we design the following steps (see Fig. 3 top).
Identify super-category. To make our framework general purpose and free of any human intervention we first identify the super-category of the dataset Disain In detail, use a VQA model as Visual Information Extractor (VIE) Vau for this purpose. The VIE is capable of identifying the super- category (e.g., "Bird") of the incoming dataset it has been pre- mre-trained on large amounts of data from super-categories. Formally, to get cateroeat In we query the VIE for an image X, E Dirain,
vqua gn lVIR (xn: Pidentify):
where Pslentity is simple Identify VQA-prompt of the format shown Fig. 10, and G {91. ....... .. is set containing the super-category names corresponding to N images in Dirain To guide the VIE to output only super-category names we provide simple in-context examples as list of names ("Bird", "Pet", "Flower", ...) Note that the ground-truth super-category name need not be present the list (see App. B.2 for details about in-context examples).
Acquire expert knowledge. After having general understanding of the dataset this step consists in identifying the useful set of attributes that set apart the subordinate-level categories. For instance, the "eye-color" attribute can be used to easily distinguish the bird species Dark-eyed Junco from the species Yellow-eyed Junco. To help the VIE in focusing on such targetted visual cues we tap into the expert knowledge of the LLMs, which is otherwise only restricted to experts. In other words, we essentially ask the LLM: given a super-category what to look for while distinguishing subordinate-level categories. In detail, we base our Acquire Expert Knowledge (AEK) module Iilm on a LLM that takes an input super-category In. and outputs a list of M useful attributes:
an = A (gn, phom-ta))
(3)
where a, = {A On, An,M) denote the M attributes corresponding to the sample with index n, and How-to L.M-prompt Ji is the prompt, as shown in Fig. 10. The intuition behind the How-to LLM-prrmpt is to emulate how to be an expert and forms the crux of our FineR. All the attributes in the dataset are denoted as A= (ai...., aN..
Perceive visual attributes. With the attributes A available we leverage the VIE to extract the description for each attribute an,m conditioned on the visual feature. For instance, if an attribute an.m "eye-color", then given the image of an unknown bird the VIE is asked describe it's eye color, which is much easier task compared to directly predicting the subordinate- category. In addition, we hard-code general attribute an = "General description of the image" " and its prompt "Questions: Describe this image in details. Answer: to extract the global visual information. Therefore, an t- an U ag and M + A + 1. We found this is helpful to capture the useful visual information from the background like the habitat of bird. Concretely, given an image x its super-category In. and the attribute an,m. the attribute description given as:
8n.m
vqu AVVE (Xas Iay an,ms Paescribe)
vgga
(4)
= where Pdeseribe denotes the Describe VQA-Prompt, as illustrated in Fig. 10, and S. {8n,1, Sn.2...., Sn.M) represents a set of M attribute descriptions extracted from the image X..
4

---

Published as a conference paper at ICLR 2024
2.2.2 FINE-GRAINED SEMANTIC CATEGORY REASONING
At the end of the first phase (Sec. 2.2.1) we have successfully translated useful visual information into text, modality that is understood by the LLM. In the second phase our goal is leverage LLM and utilize all the aggregated textual information to reason about class names in the dataset and predict a candidate set C", as illustrated in the middle part of Fig. 3.
Reasoning class names. To discover the candidate class names present in the dataset we build Semantic Category Reasoner (SCR) module that contains a LLM under the hood. The goal of SCR to evoke reasoning ability of the LLM by presenting a well-structured prompt constructed with the attributes and descriptions obtained in Sec. 2.2.1. Formally, the SCR is defined by a function / sSR that outputs a preliminary set of candidate classes C as:
(5) where Jllm the Reason LLM-prompt as shown in Fig. 10. The plenson can be decomposed into two parts: (i) Structured Task Instruction is designed keeping in mind the FGVR task and is mainly responsible for steering the LLM to output multiple possible class names given an image, thereby increasing diversity in names. We find that when only one name is asked, LLMs tend to output general categories names (e.g., "Gull") from a coarser but more common granularity. (ii) Output Instruction embeds the actual attribute description pairs extracted from the image, and task execution template to invoke the output (see App. B.2 for details).
Č = hSCR AG Premso))
Noisy name refinement. In the pursuit of increasing the diversity of candidate class names accumulate in the set C lot of noisy class names. It is mainly caused by the unconstrained knowledge in LLMs that is not relevant to the finite concepts in the Dimin at hand. To remedy this, in addition to name deduplication, we propose Noisy Name Denoiser (NND) module that uses a VLM (e.g.. CLIP). Specifically, NND uses the text encoder & and the vision encoder Eing of the VLM to encode é and Dirain respectively. Each image in Dimin is assigned a category è Ć based on the maximum cosine similarity as in Eq. 1. The class names that are not selected by Eq. 1 constitute the noisy class names set V C C and are eliminated. The refined candidate set is given as C C \ V.
2.2.3 MODAI CLASSIFIER CONSTRUCTION
Given the refined class names in C we construct text-based classifier for each class E C- using the text-encoder E of the VLM as:
(c)(e)
(6)
1|Exe( (c)
Since there can be ambiguity in some of the class names (Kaul al., 2023) also consider vision-based classifier by using the images in Dirinn In detail, we first pseudo-label the images in Dimin with C+ using the maximum cosine similarity metric, as outlined in Sec. 2.2.2. This results in Uc pseudo-labelled images per class Č- However, as Dirain contains only few images per class it might make the visual features strongly biased towards those samples. For example, only a few colors perspectives of of make and model Toyota Supra Roadster is captured in those images. Thus, we apply a random data augmentation pipeline K times on each sample x (see App. B. for details) and then compute the vision-based classifier for each class E C* as: KxU,
Uc
Eing (x²)
Eing (x9)
Σ
(7)
ing
U.KK + 1)
Eing (x²) )l1
T (x9)11
j-1
where x is the augmented version of x. We set K = 10 in all our experiments (see App. G.2 for an analysis on the sensitivity of K).
Being equipped with classifiers of two modalities, we construct multi-modd classifier (MMC) for class e that potentially can capture complementary information from the two modalities:
f aw + (1 w)) img
(8)
where a is the hyperparameter that controls the degree of multi-modal fusion. We empirically set 0.7 in all of our experiments (see App. G.I for an analysis on the sensitivity of a). Finally, the
MMC for all the classes in Eh is given as W mm-
5

---

Published as a conference paper at ICLR 2024
2.3 INFERENCE
As shown in Fig. 3 right, we replace the text-based classifier in VLM with the MMC W mm- and classify the test images Diest by è arg maxxceč- (Cing (x), w mm/ where è is the predicted class name having the highest cosine similarity score.
EXPERIMENTS
Datasets. We have conducted experiments several fine-grained datasets that include Caltech- UCSD Bird-200 (Wah et al., 1), Stanford Car-196 (Khosla et al., 2011), Stanford Dog- 120 (Krause al., 2013), Flower-100 (Nilsback & Zisserman, 2008), and Oxford-IIIT Pet-37 (Parkhi et al., 2012). By default, we restrict the number of unlabelled images per category in Dirain to 3, i.e | = 3 (randomly sampled from the training split). Additionally, we evaluate under an imbalanced class distribution scenario where 1 ~ | I 10. Dataset specifics are detailed in the App. A.
Evaluation Metries. Given the unsupervised nature of the task, a one-to-one match between the elements the ground truth set and the candidate set C- can not be guaranteed. For this reason, we employ two synergistic metrics: Clustering Accuracy (cACC) and Semantic Similarity (sACC) to asses performance FGVR task, following (Conti et al., 2023). cACC evaluates the model's performance in clustering images from the same category together, but does not consider the semantics of the cluster labels. This gap is filled by sACC, which leverages Sentence-BERT (Reimers & Gurevych, 2019) to compare the semantic similarity of the cluster's assigned name with the ground- truth category. SACC also helps gauge the gravity of errors in fine-grained discovery scenarios. For example, clustering "Seagull" as "Gull" would yield higher SACC compared grouping it to the "Swallow"" cluster, given the former mistake is less severe. In summary, the SACC and cACC jointly ensure that the samples in a cluster are not only similar but also possess the correct semantics.
Implementation Details. We used BLIP-2 (Li et al., 2023) Flan-T5xxi as our VQA, ChatGPT (Ope- nAI, 2022) gpt-3.5 gpt-3.5-turbo model as our LLM via its public API, and CLIP (Radford et al., 2021) ViT-B/16 as the VLM. The multi-modal fusion hyperparameter and exemplar augmentation times are set to a 0.7 and K = 10. We elaborate prompt design detail in App. B.2.
Compared Methods. Given that FGVR without expert annotations is relatively new task, except CLEVER (Choudhury et al., 2023), other baselines are not available in the literature. To this end, we introduced several strong baselines: (i) CLIP zero-shot transfer using the ground-truth class names as the upper bound (UB) performance, as only experts can possess the knowledge of true class names in C. (ii) WordNet bascline that uses the CLIP and large vocabulary of 1 199 nouns sourced from the WordNet (Miller, 1995) as class names. (fii) BLIP-2 F5an-T5xx1 a VQA baseline, that indentifies the object category by answering a question of the template "What is the name of the main object in this image?". (iv) SCD (Han et al., 2023) that first uses non non-paramttric clustering to group unlabeled images, and then utilizes CLIP narrow down the unconstrained vocabulary of 119k-noun WordNet and li-b names sourced from Wikipedia. (v) CaSED (Conti et al., 2023) uses CLIP first retrieve captions from a large 14.8-million-entry knowledge base, subset of PMD (Singh et al., 2022). After parsing the nouns from the captions, CaSED uses CLIP to classify each image. (vi) KMeans (Ahmed et al., 2020) on top of CLIP visual features. (vii) Sinkhorn- based parametric clustering in (Caron et al., 2020) using CLIP and DINO (Caron et al., 2021) features. All the baselines we re-run use ViT-B/16 as the CLIP vision encoder. Refer to App. C for details.
3.1 BENCHMARKING ON FINE-GRAINED DATASETS
Quantitative comparison I: The battle of machine-driven approaches. We benchmarked our FineR system for the task of FGVR without expert knowledge on the five fine-grained datasets. We first evaluated on a minimal training set comprising three random images per category ((Ditain) 3). As shown in Tab. 1, our FineR system outperforms the second-best method (BLIP-2) by a substantial margin, giving improvements of +9.8% in cACC and +5.7% in SACC averaged on the five datasets. We also evaluated a more realistic scenario, i.e., imbalanced (long-tailed) image distribution for class name discovery (1 < |Dimin| < 10) and report the results in Tab. 2. Despite the heightened discovery difficulty posed by imbalanced distribution, our FineR system still consistently outperforms the second-best method, achieving average improvement of +7.3% in cACC and +2.2% in SACC.

---

Published as a conference paper at ICLR 2024
Bird-200
Car-196
Dog- 120
Flower- 102
Pet-37 CACC
Average
cACC
SACC
cACC
SACC
cACC
SACC
cACC
SACC
SACC
cACC
SÁCC
Zero-shot (UB)
57.4
80.5
63.1
66.3
56.9
75.5
69.7
77.8
81.7
87.8
65.8
77.6
ILIP-Sinkhorn
23.5
18.1
12.6
30.9
23.1
21.6
DINO-Sinkhorn
13.5
7.4
11.2
17.9
5.2
19.1
KMcans
36.6
30.6
16.4
66.9
32,8
36.7
WordNet
39.3
57.7
18.3
33.3
53.9
70.6
42.1
49.8
55.4
61.9
41.8
54.7
BLIP-2
30,9
56.8
43.1
57.9
39.0
58.6
61.9
59.1
61.3
60.5
47.2
58.6
CLEVER +
7.9
6.2
SCD
46.5
57.9
CaSED
25.6
50.1
26.9
41.4
38.0
55.9
67.2
52.3
60.9
63.6
43.7
52.6
FineR (Ours)
51.1
69.5
49.2
63.5
48.1
64.9
63.8
51.3
72.9
72.4
57.0
64.3
Table I: cACC(%) and SACC (%) comparison on the five fine-grained datasets. [D Tmin = 3. Results reported
are averaged over 10 runs. f: SCD and CLEVER results are quoted from original paper (SCD uses the entire dataset for class name discovery and assumes the number of classes known as a-priori). Best and second- bestnd-best
performances are coloured Green and Red respectivoly. Gray presents the upper bound (UB).
70
dACC (%)
CLi Sinkhom FineR
DINO Sinkhom
4400
149 TU
FineR (Ours)
Expect 2 90
Expart
X
Lyypenson
70
WordNet
FineR (Ours)
40
50
CASIDD
Beiter
60
CoSED
BLIP
Better
Train
20
WordNet
Layperson
10
Data soale used for Discovery
30
10
20
30
40
50
50
70
40
50
60
70
90
20%
40%
60%
60%
100%
(4)
Car-196
aACO (%)
Pat-37
cACC
Figure 4: Comparison with the learning-bar based methods, cACC is averaged on five datasets.
Figure 5: Human study results. Averages computed across 30 participants are reported.
Among the compared methods, BLIP-2 stands out, largely owing to its powerful vision- aligned Flan-T5 language core and its large training knowledge base (Li et al., 2023). The WordNet baseline, SCD, and CaSED show strong perfor- mance on specialized datasets such as Dog- 120 and Flower- 102,
Average
cACC SACC
Zero-shot (UB)
65.8
77.6
largely due to their exhaustive knowledge bases. Specifically, WordNet and SCD cover all ground-truth Dog-120 categories, while CaSED's s PMD (Singh et al., 2022) knowledge base includes 101 of 102 ground-truth Flower-102 categories. In contrast, our reasoning- baaggdgbbbddddee FineR system achieves significant improvements across various datasets without explicitly needing to query any external knowledge base. Moreover, with just a few unlabeled im- ages, our method surpasses learning- based approaches that utilize the full-scale training split, as illustrated in Fig. 4.
WordNet
41.8
54.7
BLIP-2 CaSED
44,6
59.0
40.8
51.1
FineR (Ours)
51.9
61.2
Table 2: Comparison with imbal- anced Dinin across five fine-grained datasets. Averages reported.
Quantitative comparison II: From layperson to expert- where do we stand? Echoing with initial motivation of democratizing FGVR, we conducted a human study to establish layperson-level baselines on the Car-196 and Pet-37 datasets. In short, we presented one image per category to 30 non-expert participants and asked them to identify the specific car model or pet breed. If unsure, the participants were asked to describe the objects. The collected answers were then used build a zero-shot classifier with CLIP, and forms the Layperson baseline. For the Expert baseline we have used the UB baseline, which uses the ground- -ruth class names, described before. As shown in Fig. 5, on the Car-196 dataset, the Layperson baseline outperforms all machine-based methods, except our FineR system. On the Pet-37 dataset, our method distinguishes itself as the top performer among machine-based approaches. This human study shows that FineR successfully narrows the gap between laypersons and experts in FGVR. Further details in App. J.
Qualitative comparison. We visualize and analyze the predictions of different methods in Fig. 6. On the Bird-200 dataset (1st row), our FineR system shines in recognizing specific bird species, notably the "Dark-eyed Junco". Our FineR system successfully captures the nuance of the "dark-eyed" visual feature, setting it apart from visually similar birds like the more generic "Junco"". In contrast, the compared methods tend predict coarse-grained and common categories, like "Junco"", they
7

---

Published as a conference paper at ICLR 2024
Bird 200
WordNet
Ground
Crchard Oriole
Aaticohoors
Worthee:
Ground-truh: Dariveyed
Stale-cafooee Junas
GadEi::
BLIP-
Tanager
Pulccss Tanaggrr
CaSEID:
PP2:
Prediction Indicator
FineR (Ours):
Orchard Oriole
(Ourd):
Dark eyed
Correct Prediction
Ground-inth:
Jeep Grand Cherohee SUV 2012
Ground-rath:
Bentley Continental CT Coups 2012
Car 196
Wordhwt
Wordhuc
Platiumm Black
Partially Correct Prediction
BLFF2:
Benilyy Continertal GT
Ca5ED:
FineR (Duraj:
Jeep Grand Cherokee SUV 2012
BUV
CaSED
(Ours]:
Fennlyy
Bentay
Incoreet Prediction
Flower 102
Ground-truh::
Lotus
Ground-truhh:
Blackberry Lily
WordNet
Wordliee:
Peeuxdan Liy
more precise than ground fruth names
BUPP-
Lotua
Senegeleneie
CaSED:
Latus
CaSED
Glan
Fizelt (Oural:
Finelt (Duraj: Onange-potted
Figure 6: Qualitative comparison on Bird-200, Car-196, and
Flower-102 datasets. Digital zoom recommended.
do not emphasize or account for finer details necessary in FGVR. Similar trends are evident in the example of "Jeep Grand Cherokee SUV 2012" (2nd row left). While all methods struggle with the "Bentley Continental GT Couple 2012" (2nd row right), our system offers the closest and most fine-grained prediction. The most striking observation comes from the Flower-102 dataset. Our system outshines the ground-truth in the prediction results of the "Lotus" category (4th row left), classifying it more precisely as a "Pink Lotus" aided by the attribute information "primary flower color: pink" during reasoning. And in cases where all models misidentify the "Blackberry Lily" (4th row right), our system offers the most plausible prediction, the "Orange-spotted Lily", informed by the flower's distinctive orange spots in the petals. This further confirms that our system effectively captures fine-grained visual details from images and leverages them for reasoning. This qualitative analysis demonstrates that FineR not only generates precise, fine-grained predictions but also displays high semantic awareness. This holds true even when predictions are only partially correct, thereby mitigating the severity during misclassification. Refer to App. H for more qualitative results.
3.2 BENCHMARKING ON THE NOVEL POKEMON DATASET
Method
GT Hit Rate
Discovered Names
Polecton
Butthoous
Chemander Squitle
Pladhu
Pidger
WordHNe
0110
Faloner, Turter Shiry Lyoots, Chicken Herk, Gerlsizon, Fikss, Canteon Bidiee,, Phernt, Tylo, fiedrase, Pooeweed Archeen Fon, Pandurtom last
Zero-sbot (UB)
70.5
sACC
89.2
Pikachu
WeedNet
54.6
33.1
CaSED
2n10
Interbroed Pikashw, Turto, Plant, Fokemar, Bulbassur, Bid
BLIP-2
32.3
55.4
beyoour
PMdpesto
FineR (Ound)
7/1
Groenical Squirto, Charmander, Charmleean, Equirtie, Wartortie, Fikzohu
Pidgeotio, Pishi, Sadial Plach., Squirte
FindR
CaSED
39.2
55.7
Pokemon and their
(6) Discovered and ground-rut HI Rate
Jel Gueritorie
Figure 7: Comparison on the novel Pokemon datasct (3 images per category for discovery, 10 for ovaluation).
To further investigate the FGVR capability of FineR on more novel concepts, we introduce new Pokemon dataset comprised of 10 Pokemon characters, sourced from Pokedex (Nintendo, 2023) and Google Image Search, as shown in Fig. 7(a). One can notice that each pair of Pokemons (each column) have subtle visual differences. As shown in Fig. 7(b), it is hardly surprising that the WordNet baseline fails to discover any of the Pokemon categories, scoring 0/10, given the absence of most specific Pokemon names in its knowledge base. BLIP-2 and CaSED appear to mainly identify only the most common Pokemon classes. Although CaSED does have all ten ground-truth Pokemon names in its PMD knowledge base, it still fails discover most of these categories. We conjecture this failure to the high visual similarity between the Pokemons characters and their real-world analogs, compounded by CLIP scoring preferences (Ge et al., 2023). As revealed in Fig. 7(b), the classes identified by CaSED predominantly feature real-world categories resembling the Pokemons (e.g., the animal "turtle" rather than the character "Squirtle"). In stark contrast, our FineR system successfully discovers 7/10 ground-trute Pokemon categories, consequently outperforming the second- best result by +-1..%% .%% cACC and 25.%% SACC as shown in Fig. 7(c).
3.3 ABLATION STUDY
We report an ablation analysis of the proposed components of FineR in Tab. As shown in row 2 of Tab. 3, the Noisy Name Denoiser (NND) for the name refinement process (Sec. 2.2.2) stands out as the most impactful, improving cACC by +6.0% and SACC by +4.7% over the baseline that simply uses the preliminary candidate names for classification. This validates its effectiveness in
8

---

Published as a conference paper at ICLR 2024
filtering out noisy candidate names. Next in row 3 we ablate the MMC (Sec. 2.2.3) by setting a = 0.7 and without using K augmentations. We notice improved clustering accuracy due
Average
NND
MMC
a
K
cACC SACC
x
x
X
x
47.5
58.3
to the introduction of visual features with the MMC, showing the complementarity of textual and visual classifier. Finally, in
x
X
53.5
63.0
56,9
63.3
row 4 when we run our full pipeline with K = 10 we notice a further improved performance for both the metries. This improvement comes due to the alleviation of limited visual perspectives present in a few images in Dumin, Due to lack of space, further ablation study, sensitivity analysis and generalizability study with other LLMs can be found in App. E, App. G and App. F, respectively.
57.0
64.3
Table 3: Ablation study on the proposed components, Averages across the five fine-grained datasets are reported.
4 RELATED WORK
Fine-grained visual recognition. The goal of FGVR (Welinder al., 2010; Maji al., 2013; Wei et al., 2021) is to classify subordinate-level categories under a super-category that are visually very similar. Owing to the very similar visual appearances in objects the FGVR methods heavily rely on, aside from ground truth annotations, expert-given auxiliary annotations (Krause et al., 2013; Zhang al., 2014: Vedaldi et al., 2014; He & Peng. 2017). The FGVR methods can be categorized into two types: (i) feature-encoding methods (Lin et al., 2015; Gao et al., 2016; Yu et al., 2018; Zheng et al., 2019) that focus on extracting effective features (e.g., using bilinear pooling) for improved recognition; and (ii) localization methods (Huang et al., 2016; Fu et al., 2017: Simonelli et al., 2018; Huang& & Li, 2020) instead focus on selecting the most discriminative regions for extracting features. Moreover, TransHP (Wang et al., 2023) learns soft text prompts for coarser-grained information, aiding fine-graing grained discrimination, while V2L (Wang et al., 2022) combines vision and language models for superior FGVR performance. Unlike the aforementioned methods, which rely expert- defined fine-grained categories, our approach operates in a vocabulary-free manner. Recently, in an attempt make FGVR annotation free, CLEVER (Choudhury et al., 2023) was proposed that first extracts non-expert image descriptions from images and then trains fine-grained textual similarity model match descriptions with wikipedia documents at a sentence-level. Our FineR is similar in spirit to CLEVER, except we exploit L.MSs to reason about category names from visual descriptions and classify the fine-grained classes without the need of training any module. An additional advantage of FineR over the work in (Choudhury et al., 2023) is that FineR can work in low data regime.
LLM enhanced visual recognition. In the wake of foundation models, commonly addressed tasks such as image classification and VQA have benefited with the coupling of vision with language, either by means of L.MSs (Menon & Vondrick, 2023; Yang et al., 2022; Yan et al., 2023) or querying external knowledge-bel base (Conti et al., 2023; Han et al., 2023). Specifically, some recent image classification approaches (Pratt et al., 2023; Menon & Vondrick, 2023; Lin et al., 2023) first decompose class names into more descriptive captions using GPT-3 (Brown et al., 2020) and then use CLIP (Radford et al., 2021) to classify the images. In a similar manner, recent VQA approaches (Yang et al., 2022; Shao et al., 2023; Berrios et al., 2023) first extract captions using off-the-shelf captioning model and then feeds the question, caption and in-context examples to induce GPT-3 to arrive at the correct answer. However, unlike FineR, none of the approaches have been explicitly designed to work in the vocabulary- free fine-grained visual recognition setting.
5 CONCLUSION
In this work we showed the expert knowledge is no longer roadblock build effective FGVR systems. In detail, we presented FineR that exploits the world knowledge inherent to the LLMs to address FGVR, thereby disposing off the need of requiring expert annotations. To make LLMSs compatible with vision-based task we first translate essential visual attributes and their descriptions from images into text using non-expert VQA models. Through well-structured prompts and visual attributes we invoke the reasoning ability of LLMs to discover candidate class names in the dataset. Our training-free free approach has been evaluated on multiple FGVR benchmarks, and we showed that it can outperform several sfate-of-thhe-an VQA. FGVR and learning- bassid approaches. Additionally, through the evaluation on a newly collected dataset we demonstrated the versatility of FineR in working in the wild where expert annotations hard to obtain.
9

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGMENTS
This work was supported by the MUR PNRR project FAIR- Future AI Research (PE00000013) funded by the NextGenerationEU. E.R. is partially supported by the PRECRISIS, funded by the EU Internal Security Fund 871245), and by the PRIN project LEGO-A! AI (Prot. 2020TA3K9N). This work was carried out in the Vision and Learning joint laboratory of FBK and UNITN. We extend our appreciation to Riccardo Volpi and Tyler L.. Hayes for their helpful discussions regarding dataset licensing. Furthermore, we are deeply thankful to Gabriela Csurka and Yannis Kalantidis for their valuable suggestions during the discussion stage.
100539), the EU project SPRING (No.
7 ETHICS STATEMENT
Large pre-trained models such as CLIP (Radford et al., 2021), BLIP-2 (Li et al., 2023), and Chat- GPT (OpenAI, 2022) inherit biases from their extensive training data (Goh et al., 2021; Schramowski et al., 2022; Kirk et al., 2021). Given that FineR incorporates these models as core components, it too is susceptible to the same biases and limitations. Consequently, model interpretability and transparency are central to our design considerations. Owing to the modular design of the proposed FineR system, every intermediate output within the system's 's scope is not only interpretable but also transparent, facilitating traceable final predictions. For instance, as demonstrated in the last row of Fig. 6 in the main paper, when our system produces more accurate prediction like "Pink Lotus" compared to the ground- truth label, this prediction can be directly explained by tracing back to its attribute description pair "primary flower color: pink". Similarly, for a partially accurate prediction like serrange-spoteer Lily", the system's 's rationale can be traced back to the attribute description pair "petal color pattern: orange- spottedd The semantically mean- ingful output of our model also mitigates the severity of incorrect predictions. Through this approach, we hope to offer a system that both interpretable and transparent, countering the inherent biases present in large pre- trained models. In addition, in relation to our human study, all participants were volunteers and received no financial compensation for their participation. All participants were informed about the study's 'ss objectives and provided written informed consent before participating. Data were anonymized to protect participants' privacy, and no personally identifiable information was collected. Participants were free to withdraw themselves or their responses from the study at any time.
8 REPRODUCIBILITY STATEMENT
Our FineR system is training-free and built on openly accessible models: BLIP-2 (Li et al., 2023) for visual question answering model, CLIP (Radford et al., 2021) for visuan language model, and ChatGPT gpt-3.5-turbo (OpenAI, 2022) for large language model. All models are publicly accessible and do not require further fine-tuning for utilization. BLIP-2 and CLIP offer publicly accessible weights, while the ChatGPT 9pt-3.5-turbe model is available through public API. We confirm the generalizability to open-source large language models by replacing the large language model core with Vicuna- 1 3B (Chiang et al., 2023), as studied in App. F. Comprehensive details for replicating our system are outlined Sec. 2, with additional details about prompts provided in App. B.2. Upon publication, we will release all essential resources for reproducing the main experimental results in the main paper, including code, prompts, datasets and the data splits. To enable seamless replication of our primary experimental results from any intermediate stage of our FineR system, will also release the intermediate outputs from each component. These intermediate outputs include the super-category names identified by BLIP-2, attribute names and descriptions derived from both the large language model and BI.IP-2, as well as reasoning results given by the large language model for each dataset. This comprehensive approach ensures the reproducibility of our proposed method and the results, and aims to provide practitioners with actionable insights for performance improvement.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. The k-means algorithm: A comprehensive survey and performance evaluation. Electronics, 2020. 21
William Berrios, Gautam Mittal, Tristan Thrush, Douwe Kiela, and Amanpreet Singh. Towards language models that can see: Computer vision through the lens of natural language, Arxiv, 2023. 2, 9, 29, 30
Ning Bian, Xianpei Han, Le Sun, Hongyu Lin, Yaojie Lu, and Ben Hc. Chatgpt is knowledgeable but inexperienced solver: An investigation of commonsense problem large language models. Arxiv, 2023. 29
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101 mininng discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014. 30
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances Neural Information Processing Systems, 2020. 3, 9, 19, 29
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 2020. 6, 21
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In International Conference on Computer Vision, 2021. 6, 21
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023. URL https:///mms/:
org/blog/2023-0 -3-00-3-3300 30- vicuna/. 10, 23
Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. The curious layperson: Fine-grainee image recognition without expert labels. International Journal of Computer Vision, 2023. 2, 6, 9, 21
Alessandro Conti, Enrico Fini, Massimiliano Mancini, Paolo Rota, Yiming Wang, and Elisa Ricci. Vocabulary-free image classification. Advances in Neural Information Processing Systems, 2023.
2, 6.9
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. Azxiv, 2022. 3
Jianlong Fu, Heliang Zheng, and Tao Mei. Look closer to hetter: Recurrent attention convolutional neural network for fine-grained image recognition. In Computer Vision and Pattern Recognition, 2017.9 9
Yang Gao, Oscar Beijbom, Ning Zhang, and Trevor Darrell. Compact bilinear pooling. In Computer Vision and Pattern Recognition, 2016.
Yunhao Ge, Jie Ren, Andrew Gallagher, Yuxiao Wang, Ming- Yang, Hartwig Adam, Laurent Itti, Balaji Lakshminarayanan, and Jiaping Zhao. Improving zero-shot generalization and robustness of multi-modal models. In Computer Vision and Pattern Recognition, 2023. 8
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec
Radford, and Chris Olah. Multimodal neurons artificial neural networks. Distill, 2021.
Kai Han, Yandong Li, Sagar Vaze, Jie Li, and Xuhui Jia. What's in a name? beyond class indices for image recognition. Arxiv, 2023. 6, 9. 21
Xiangteng He and Yuxin Peng. Fine-grained image classification via combining vision and language. In Computer Vision and Pattern Recognition, 2017. 1,9
11

---

Published as a conference paper at ICLR 2024
Shaoli Huang, Zhe Xu, Dacheng Tao, and Ya Zhang. Part-stacked cnn for fine-grained visual categorization. In Computer Vision and Pattern Recognition, 2016. 9
Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. Arxiv, 2023. 29
Zixuan Huang and Yin Li. Interpretable and accurate fine-grained recognition via region grouping. In Computer Vision and Pattern Recognition, 2020.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-languee representation learning with noisy text supervision. In International Conference on Machine Learning. 2021. 3, 31
Prannay Kaul, Weidi Xie, and Andrew Zisserman. Multi-modal classifiers for open-vocabulary object detection. In International Conference on Machine Learning, 2023. 5
Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In Computer Vision and Pattern Recognition Workshop, 2011. 6, 16
Hannah Rose Kirk, Yennie Jun, Filippo Volpin, Haider Iqbal, Elias Benussi, Frederic Dreyer, Alek- sandar Shtedritski, and Yuki Asano. Bias out-of-the-box: An empirical analysis of intersectional occupational biases in popular generative language models. In Advances in Neural Information Processing Systems, 2021. 10
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, 2022. 21, 29
Jonathan Krause, Michael Stark, Jía Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In International Conference on Computer Vision Workshops, 2013. 6, 9, 16, 30
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 2017. 30
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. Arxiv, 2023. 2, 3, 6, 7, 10
Shuo Li, Fang Liu, Zehua Hao, Kaibo Zhao, and Licheng Jiao. Unsupervised few-shot image classification by learning features into clustering space. In European Conference on Computer Vision, 2022. 2
Tsung- Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. Bilinear enn models for fine-grained visual recognition. In International Conference on Computer Vision, 2015. 9
Wei Lin, Leonid Karlinsky, Nina Shvetsova, Horst Possegger, Mateusz Kozinski, Rameswar Panda, Rogerio Feris, Hilde Kuehne, and Horst Bischof. Match, expand and improve: Unsupervised finetuning for zero-shot action recognition with language knowledge. In International Conference on Computer Vision, 2023. 9
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Arxiv, 2023a. 3.3
Man Liu, Feng Li, Chunjie Zhang, Yunchao Wei, Huihui Bai, and Yao Zhao. Progressive Semantic- Visual Mutual Adaption for Generalized Zero-Shot Learning. In Computer Vision and Pattern Recognition, 2023b. 32
Shichen Liu, Mingsheng Long, Jianmin Wang, and Michael Jordan. Generalized zero-shot learning with deep calibration network. In Advances in neural information processing systems, 2018. 32
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. Arxiv, 2023. 30
12

---

Published as a conference paper at ICLR 2024
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- line-grained visual classification of aircraft. Arxiv, 2013. 9
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In International Conference on Learning Representations, 2023. 9, 30, 31
George A Miller. Wordnet: a lexical database for english. Communications of the ACM, 1995. 6
Mark EJ Newman. Power laws, pareto distributions and zipf's law. Contemporary physics, 2005. 16
Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian conference computer vision, graphics & image processing. IEEE, 2008. 6,
16
Nintendo. Pokedex: Search for a Pokemon by name or using its National Pokedex numer. https:
//www. www..ookemon... com/us/pokedex, 2023.
OpenAl. ChatGPT: A Large-Scale GPT-3.5- Based Model. https://ppenaa..c/////lgccaaaaat 2022. 2, 6, 10, 17, 29
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar: Cats and dogs. In Computer Vision and Pattern Recognition, 2012. 6, 16, 30
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? generating customized prompts for zero-shot image classification. In International Conference on Computer Vision, 2023. 9
Nan Pu, Zhun Zhong, and Nicu Sebe. Dynamic conceptional contrastive learning for generalized category discovery. In Computer Vision and Puttern Recognition, 2023.
Nan Pu, Zhun Zhong, Xinyuan Ji, and Nicu Sebe. Federated generalized category discovery. In Computer Vision and Pattern Recognition, 2024. 21
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021. 2, 6, 9, 10, 21, 31
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. Arxiv, 2019. 6
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Computer Vision and Pattern Recogni- tion, 2022. 29
Subhankar Roy, Mingxuan Liu, Zhun Zhong, Nicu Sebe, and Elisa Ricci. Class-incremental novel class discovery. In European Conference on Computer Vision, 2022.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 2015. 30
Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kersting. Large pre-trained language models contain human-liks biases of what is right and wrong to do. Nature Machine Intelligence, 2022. 10
Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. Prompting large language models with answer heuristics for knowledge- visual question answering. In Computer Vision and Pattern Recognition, 2023. 9, 29
Andrea Simonelli, Francesco De Natale, Stefano Messelodi, and Samuel Rota Bulo. Increasingly spe- cialized ensemble of convolutional neural networks for fine- -eeingrrnnd recognition. In International Conference Image Processing, 2018. 9
13

---

Published as a conference paper at ICLR 2024
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Computer Vision and Pattern Recognition, 2022. 6, 7, 22
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. Arxiv, 2023. 3, 23, 29
Andrea Vedaldi, Siddharth Mahendran, Stavros Tsogkas, Subhransu Maji, Ross Girshick, Juho Kannala, Esa Rahtu, Iasonas Kokkinos. Matthew B Blaschko, David Weiss, et al. Understanding objects in detail with fine-grained attributes. In Computer Vision and Puttern Recognition, 2014. 1,
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. Califormia Institute of TeTchholooyy, 2011. 6, 16, 19, 31
Wenhao Wang. Yifan Sun, Zongxin Yang. and Yi Yang. L2L: Leveraging Vision and Vision-language Models into Large-scale Product Retrieval. Arxiv, 2022. 9
Wenhao Wang, Yifan Sun, Wei Li, and Yi Yang. TransHP: Image Classification with Hierarchical Prompting. In Advances in Neural Information Processing Systems, 2023, 9, 31
Jason Wei, Xuezhi Wang, Dale Schaurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Sfaii--ff-thougee prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022. 3, 29
Xiu-Shen Wei, Yi-Zhe Song, Oisin Mac Aodha, Jianxin Wu, Yuxin Peng, Jinhui Tang, Jian Yang. and Serge Belongie. Fine-grained image analysis with deep learning: A survey. Transactions on Puttern Analysis and Machine Intelligence, 2021. 1, 2,
Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pictro Perona. Caltech-ucsd birds 200. 2010. 9
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In International Conference on Computer Vision, 2023. 9, 31
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Association for the Advancement of Artificial Intelligence, 2022. 9, 30
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Arxiv, 2023. 29
Chaojian Yu, Xinyi Zhao, Qi Zheng, Peng Zhang, and Xinge You. Hierarchical bilinear pooling for fine-grained visual recognition. In European Conference on Computer Vision, 2018. 9
Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell. Part-based r-cnns for fine-grained category detection. In European Conference on Computer Vision, 2014. 1, 1,9
Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. Looking for the devil in the details: Learning trilinear attention sampling network for fine-grained image recognition. In Computer Vision and Pattern Recognition, 2019.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Computer Vision and Pattern Recognition, 2022a. 31
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 2022b. 31
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elho- seiny. Chatgpt asks, 2 answers: Automatic questioning towards enriched visual descriptions. Arxiv, 2023a. 29
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. Arxiv, 2023b. 2
14

---

Published as a conference paper at ICLR 2024
APPENDIX
Dataset Details
16
A.1 Five Standard Fine-grained Datasets
16
A.2 Pokemon Dataset
16
Implementation Details of FineR
17
B.1 Data Augmentation Pipeline
17
B.2 Prompt Design
17
C Implementation Details of the Compared Methods
21
D Further Comparison with Imbalanced Image Distribution
23
Further Ablation Study
23
F Further Generalizability Study with Open-source LLMs
23
G Further Sensitivity Analysis
24
G.I Analysis of the hyperparameter o
................................
24
G.2 Analysis of the hyperparameter K
24
G.3 Analysis of VL.M model size
24
G.4 Analysis of the number of unlabeled images used for class name discovery
25
H Further Qualitative Results
27
I Further Failure Case Results
27
J Human Study Details
28
K Further Discussions
29
K.1 Reasoning Capability of Large Language Models
29
K.2 The Story of Blackberry Lily
29
K.3 Working in the Wild
30
K.4 Setting Comparison
31
15

---

Published as a conference paper at ICLR 2024
In this Appendix, we offer thorough set of supplementary materials to enrich the understanding of this work. We commence by delving into the specifics of the five fine-grained datasets, as well as newly introduced Pokemon dataset, in App. A. Following this, App. B elaborates the implementation details and prompt designs integral to our FineR system. App- C outlines the implementation details for the compared methods in our experiments. Supplementary experimental results for the imbalanced image distribution and the component-wise ablation study are covered in Sec. D and App. E, respectively. We expand on the generalizability of our approach with open-source LLMs in App. F. App. G examines the sensitivity of our system to different factors. Additional qualitative results are disclosed in App. H. Additionally, a qualitative analysis of failure cases is provided in App. I. Our human study design is discussed in App. J. Lastly, this Appendix concludes with further insights, intriguing discussions, and setting comparison App. K.
A DATASET DETAILS
A.1 FIVE STANDARD FINNEGRAINED DATASETS
Bird-200
Car-196
Dog-120
Flower-102
Pet-37
C
200
196
120
102
37
3
Diimin
600
588
360
306
111
Diest
5.7k
8.1k
8.6k
2.0k
3.7k
200
196
120
102
37
S | S
10
DDimin
438
458
274
229
115
DDiest
5.7k
8.1k
8.6k
2.0k
3.7k
Table 4: Statistics for the five fine-grained datasets. The number of categories [C], the number of images used for discoveries Diiminn, and the number of images used for test |Diel] are reported for the two cases where 3 samples per category are used for discovery (DD see upper half), and few arbitrary images are used for discovery (1 S | S 10, see lower half), respectively.
We provide comprehensive statistics for the five well-studied fine- the-grained datasets: Caltech-UCSSI Bird-200 (Wah et al., 2011), Stanford Car-196 (Khosla et al., 201 1), Stanford Dog- (Krause et al., 2013), Flower-102 (Nilsback & Zisserman, 2008), and Oxford-IIIT Pet-37 (Parkhi et al., 2012), used for our experiments in Tab. 4. To the Car-196 dataset, we remove all the license plate numbers from images featuring specific plates.
In the default experimental setting, where each category has 3 images available for class name discovery (Dirain = 3), we randomly select images per category from the training split of each dataset to constitute the few unlabeled sample set Diruin The test split serve the evaluation set Diest, is ensured that the discovery and test sets are mutually exclusive, Dimin n Diest = 0. To evaluate our method under more realistic and challenging conditions, and report the evaluation results in Tab. 3. This set features not only restricted number of images synthetic long-tail distribution. To generate this imbalanced set,
construct an imbalanced discovery set
per category but also conforms we first generate a long-tail tail distribution, where each category contains between I and 10 images, using Zipf's s law (Newman, 2005) with shape parameter of 2.0. We then use this distribution to randomly sample the corresponding amount of images per category from each dataset, forming the imbalanced discovery set Dirain, Fig. 8 displays the long-tail distribution we employ across categories in the five fine fine-grained datasets.
A.2 POKEMON DATASET
Beyond the five standard fine-graine Ene-grained datasets, we build and introduce a Pokemon dataset of virtual objects to further evaluate both the baselines and our FineR system. Initially, we manually select 10 visually similar Pokemon categories by checking the Pokedex an official Pokemon index. We then use Google Image Search to collect 13 representative images for each category, each of which is then manually verified and annotated. In alignment with our previous experimental settings, we designate images per category ([Diainn = 3) to form the discovery set Dimin while the remaining images are allocated to the test set Drest_ As the Pokemon dataset contains only 130 images in total, we directly illustrate the dataset and its subdivisions in Fig. 9. Despite its limited size, this dataset presents multiple challenges: (i) The dataset is fine-grained; for instance, the primary visual
'https:/ h/www pokemon. wwt.cs//p//ee/eeeeeee
16

---

Published as a conference paper at ICLR 2024
Generated long-tail Data Distribution for Discovery
Generated long-tail Data
Generated long-tail Data
Distribution for Discovery
Distribution for Discovery
Number of Images
10
Number of Images
10
Number Images
8-
8
6-
6
4
2-
2-
2
Categories
200
Categories
196
Categories
120
(a) Bird-200
(b) Car-196
(c) Dog-120
Generated long-tail Data
Generated long-tail Data
Distribution for Discovery
Distribution for Discovery
Number of Images
10
Number of Images
10
8
6-
Categories
102
Categories
(d) Flower-102
(e) Pet-37
Figure 8: Imbalanced discovery set (1 < [D < 10) histograms of the number of unlabeled samples per category following the generated long-tail distribution for the five fine-grained datasets, respectively.
difference between "Bulbas lbasauurr and "Ivysaur" is blooming flower on "Ivysaur"'s back; (ii) Test images might contain multiple objects. (iii) domain gap exists compared to real-world photos and physical entities. We plan to publicly release this Pokemon dataset along with our code.
B IMPLEMENTATION DETAILS OF FINER
B. DATA AUGMENTATION PIPELINE
To mitigate the strong bias arising from the limited samples in Dirain during the construction of the multi-modal classifier, we employ a data augmentation strategy. Each sample x E Ditain is transformed times by a random data augmentation pipeline. Therefore, K augmented samples are generated. This approach is crucial for preparing the system for real-world scenarios where test images may feature incomplete objects, varied color patterns, and differing orientations or perspectives. Accordingly,
data augmentation pipeline incorporates random cropping, color jittering, flipping, rotation, and perspective shifts. Additionally, we integrate random choice and random apply functions within the pipeline to closely emulate real-world data distributions.
B.2 PROMPT DESIGN
In this section, we delves into the prompt designs FineR. BLIP-2 is used as the VQA model to construct the Visual Information Extractor (VIE) in FineR system. ChatGPT (OpenAI, 2022) gpt-3.5-turbo is used as the frozen LLM our system, with temperature 0.9 to purse diversity for fine- tine-grained categories. In the subsequent sub-sections, employ the reasoning process for an image of an "Italian Greyhound" " as an intuitive example to facilitate our discussion on prompt designs.
17

---

Published as a conference paper at ICLR 2024
train
D Dtest
Bulbasaur
Ivysaur
Charmander
Charmeleon
Squirtle
Wartortle
Pikachu
Raichu
Pidgey
Pidgeotto
Figure 9: Visualization of the newly introduced Pokemon dataset. The images in the first three columns at each used for discovery.
Identify PrAmPrompt Alieuity
Reason LLM-Prompt
dbjoct image?
an image Bes
following
Inccontest
Example
cin-conted
Inler and foi abous qusaible names the in this
stributt
imited booletces with
Structured Teak Instructiona
Answer:
information you
Dulpui JSON object that uses the following format: -JBON Formal pwrtomm stsementinee
How-to LLM--roomp
Fisampee Using the inllwing
image.
Deseritian"
Aftib List:
Structured
nntrccion:::
"atsibuta name" "atrloura desoription"
Task Instructions
Undersaand the cxamole delimte aos.
Bunmary:
Lai the useli atribunee in cladnggash alu image.
Output Python list onject that contains isted witrdrudes.
Ourput JSCN.
essssss
Inconteet
cnncootedd
Output Instruction
Instruction
usedd atributes didingssetigg in in image:
Baga?"
Describe VQA-Prompt
Summary:
Question: Descrioo fhsur the Usi this image.
Three possible
names:
Qutpsi
Figure 10: The templates the prompts used in FineR.
Fig. 10 presents the prompt templates used in each module of our FineR system, and Fig. 11 provides detailed illustration of the entire reasoning process, including the exact outputs at each step.
Identify VQA-Prompt. Since language models are unable to see, the initial crucial step to translate pertinent visual information into textual modality, serving visual cues for the language model. As illustrated Fig. 2, our preliminary research indicates that, while VQA methods may struggle to pinpoint fin-grraind semantic categories due to their limited training knowledge, they show robust capability at identifying coarse- coarse-grained super-categories like "Bird", "Dog", and even "Pokemon". As an initial step, we employ VQA-based Visual Information Extractor (VIE) to
18

---

Published as a conference paper at ICLR 2024
Few uniabeled
Step Idantit Super Category
Stop 4: Roasoring Class Names
tave ininsage dog
modk oiporias
Yerr pertrm oninng actini:
Biad Dog
Ausancia flrmaaio
dog the aarrura deccipiion deltrited
ontor and Neeee procabll nemes the degin image lasel on intornator
nse posiiles nomebnn
condond aer aiece bi same
vio th ce sosih the cancaye,
Usir kellwing fommnt perfore stoomer denedd tackes
LLM
Acquire Export Knowlodge
"Sone ol Desclitic" "ganeeaa deeci of of the plase"
Reasoned
dag imape.
Aa "au ub sencriptte?
Sond loall you car compint cint tait bo toloving the intticctioos
"ao sure meserpicc"
Lar the cvell to distippatt dog in irape
posatle dag
Conynoaand
Durgl
Whippet
Detalipiin grogh ressing
Anto Anc 1i0
balian Greyhound
coeld butes clatnguut shing bind
Thoed sinf" "haraai
long body
lppar invvrr wain pertorl Theoer cole" thoal sniir to oecor leigt: frokowu
pasarn: bely panami jermary solo log dolor, bel color trown coler' virg paraiii hab
"Mur tingtt" "short" dirn "Dcsee""
muzze that longer that dog's head"
Sunnayy The dig the phuto a grovt sund clock and iroun qre bown caar nas long body thit and unall pave wit splayed loxa felal is derder and dkattur bobyyd. The dig has sucutit bown eots, spcatts sordings caly reylooid
The rook auriiuute for elstnng shing reage:
lengh: lag chape: ban ssal: Saa atrppe Soe ahipet, hall shapel hall lerght postion icsal var shaps unedd langhi. toue alepi, hose ooor, tranni bedy shope and popprtart, treed-
These posatble dog wame Graghcandd WW Priippol Hallaa Greghoand
Cvitul
('rerhoudh The the photo groyt ound sleck sou intt brown "he bodd. Barvees anti pour with ypllye twes" The volle cooee bakin ancer is solpy "The Snn hak.
Describe
each attribute
Step Perosive Each Vinuall Attribute
VOA
Tallss Grephonnd" ("Th dog shoto bacrs sater sos inslan dany "trhas seetic para bngloddy Evwe and anla www.errrrrr..... butto cny plnn aa
Daittie: Desictt eu the body phops of the dogin to Arewer d:dng with bing body and short
beay"" The has aaquity
Figure 11: Whole category reasoning pipeline of FineR. An image of a "Italian Greyhound is used as an example.
initially identify these super-categories from the input images. These identified super-categories then serve coarse-grained base information to direct subsequent operations. Importantly, this design ensures that our FineR system remains bataset-agnostic we do not presume any super-category affiliation for target dataset a-priori. To steer the VIE towards recognizing super-categories, we include simple in-context example (Brown et al., 2020) within the Identify VQA-prompt, which conditions the granularity of the output. The specific in-context example utilized Padenuty is shown in Fig. 12. The VIE module successfully identifies the super-category "dog" from the image of a "Italian Greyhound". We find that providing straightforward in-context examples, such as "Bird, Dog, Cat, ... effectively guides the VQA model to output super-categories like "car" or "flower" for the primary object in an image. We use the same in-context example for all the datasets and experiments. Users can also effortlessly adjust this in-context example to better align the prompt with their specific data distribution.
Identify VQA-Prompt P Pidentify
Question: What is the main object in this image?
In-context Example
(Bird, Dog, Cat, ...)
Answer: A dog
Figure 12: Identify VQA-prompt details. An image a "Italian Greyhound" is used an example. How-to L.M--Prmpp. After the super-category names G (e.g., "is obtained, we can then invoke the Acquire Expert Knowledge (AEK) module to acquire useful visual attributes for distin- guishing the fine-grained sub-categories of G. using the LLM. To give sense of what are the useful attributes, we add an in-context example of bird using the visual attributes collected by (Wah et al., 2011) due to its availability, shown in Fig. 13. This bird attributes in-context example is used for all datasets. Although we used the attribute names of birds as an in-context example, we also tried version where we do not use bird attributes in the in-context example and we get similar results.
19

---

Published as a conference paper at ICLR 2024
The in-context example of birds guide LLM to give cleaner answers and thereby simplify the post- past-pprsing process. Next, we embed the super-category identified from the previous step into the How-to LLM-prompt and query the LLM to acquire the useful visual attributes as expert knowledge to help the following steps. In addition, we also specify a output format the task instructions for the ease of automatic processing. To obtain diverse variety of useful visual attributes as possible, here the AEK module query LLM 10 times and use the union of the output attributes for the following process with a temperature of 0.9 for higher diversity. As detailed in Fig. 13, after iterations of querying, we amass set of valuable attributes useful for distinguishing "dog".
How-to LLM-Prompt
P Phow-to
Your task is to tell me what are the useful attributes to distinguish a dog in an image.
Specifically, you complete the task by following the instructions:
Structured Task Instructions
1. Understand the example delimited by <>.
2. List the useful attributes to distinguish dog an image.
Output Python list object contains the listed useful attributes.
<bird>
The useful attributes for distinguishing bird species in an image of a bird:
['bill shape', 'wing color', 'upperparts color', 'underparts color;, 'breast pattern', 'back color', 'tail shape', 'upper tail color', 'head pattern', 'breast color', 'throat color', 'eye color', 'bill length', 'forehead color', 'under tail color', 'nape color', 'belly color', 'wingi shape', 'size', 'shape', back pattern', 'tail pattern', 'belly pattern', 'primary color', 'leg color', 'bill color', 'crown color', 'wing pattern', 'habitat]]
In-context Example
Output Instruction
<dog>
The useful attributes for distinguishing dog in an image:
['body shape', 'head size', 'leg thickness', 'body size', head shape', facial wrinkles', 'neck length', 'leg length', 'leg shape'. 'paw size', 'paw shape', 'toe shape', 'tail shape', 'tail length', "tai position', 'coat color', 'coat length', 'coat texture', 'coat pattern', Yur texture', 'eye shape'. 'eye color', 'ear position', 'ear shape', length', mose shape', 'nose color', 'overall body shape
Example of Actual Output
proportion', breed-specific markings or patterns', 'overall appearance', fur length', fur density', 'muzzle shape))
Figure 13: How-to LLM-prompt details. "dog" is used as example.
Describe VQA-Prompt. Having the acquired useful attributes set A as expert knowledge, we then extract visual information targeted by the attributes from the given image using our VIE module. A simple "Question: Describe the an,m of the Jn in this image. Answer: VQA-prompt is used iteratively query the VQA to describe each attribute On,m for the correspond- ing super-category As shown in Fig. 14, a description 5n.m (e.g., "a dog with a long body and short tail") obtained for the visual attribute an,m (e.g., "body shape").
Describe VQA-Prompt P
vqa Pdescribe
Question:
Describe the body shape of the dog in this image.
Answer: a dog with long body and short tail
Figure 14: Describe VQA-prompt details. An image "Italian Greyhound" is used as an example.
Reason LLM-Prompt. this stage, we have acquired auperr-ategorry.aabbbtee--eeesee denoted G. A, S, enriched with visual cues essential for reasoning about fine- ine-ggrined cate- gories. These cues serve as a prompt to engage the large language model core of our Semantic
triplets,
20

---

Published as a conference paper at ICLR 2024
Category Reasoner (SCR) in fine- fine-grained semantic categories reasoning. The Reason LLM- prompt, illustrated in Fig. 15, comprises two main sections: (i) structured task instructions; (ii) output instruction. Key design elements warrant special mention. Firstly, introducing the sub-task "1. Summarize the information you get about the In from the attribute- description pairs delimiter ted by triple backticks with five sentences" enables the language model to "digest" and "refine" incoming visual information. Inspired by zero-shot Chain-of- Thought (CoT) prompting (Kojima et al., 2022) and human behavior, this step acts as a self-reflective reasoning mechanism. For example, when kids describes a "bird" they observe to an expert, the expert typically summarizes the fragmented input before providing any conclusions, thereby facilitat- ing more structured reasoning. A subtle but impactful design choice, seen in Fig. 15, is to include the summary sentences in the output JSON object, even if these are not utilized subsequent steps. This aids in reasoned category names that better align with the summarized visual cues. Secondly, as highlighted in Sec. 2.2.2, we prompt the LLM to reason "three possible names" instead of just one. This approach encourages the LLM to consider broader range of fine-grained categories. By incorporating these sub-task directives, we construct zero-shot CoTT-like LLM-prropp specifically designed for tine-grineed reasoning from visual cues, aligning with the "Let's think step by step" paradigm (Kojima et al., 2022). We also incorporate a task execution template in the output instruc- tion section to explicitly delineate both the input data and output guidelines, enabling the LLM to systematically execute the sub-tasks. Lastly, we specify a JSON output format to facilitate automated parsing. This allows for the effortless extraction of reasoned candidate class names from the resulting ISON output. As illustrated in Fig. 15, the actual reasoning process for an image of "Itall Italian Greyhound" " is elaborated. Of the three possible reasoned names, the correct category is accurately discovered.
C IMPLEMENTATION DETAILS OF THE COMPARED METHODS
KMeans clustering. To benchmark clustering accuracy (Roy et al., 2022), we construct a KMeans (Ahmed et al., 2020) baseline utilizing CLIP (Radford al.. 2021) features. For an equitable comparison, we use the unlabeled samples Dirain for cluster formation. Despite its sim- plicity, KMeans demonstrates competitiveness when empowered by robust features, as evidenced in SCD (Han et al., 2023) and Tab. 1. Importantly, the KMeans baseline presumes prior knowledge about the number of categories in Dirain, while our method operates without any such assumptions in all experiments.
Learning-based Sinkhorn clustering. In addition to the KMeans baseline, we introduce two learning-based baselines that leverage features from CLIP and DINO (Caron et al., 2021), using Sinkhorn clustering (Caron et al., 2020). These baselines utilize the Sinkhorn-Knop algorithm for cross-view pseudo-labeling and a frozen CLIP or DINO ViT-B/16 model for feature extraction. They learn a clustering classifier by optimizing a swapped prediction problem. Consistent with other methods in our comparison, train these baselines on Dirain for 200 epochs, strictly adhering the hyperparameters specified in the original papers.
CLEVER. Curious Layperson-to- Expert Visual Entity Recognition (CLEVER) (Choudhury et al., 2023) is a pioneering work focused on fine- tne-grained visual recognition without expert annotations. employs web encyclopedias to train a model for generating image captions that highlight visual appearance. A subsequent fine-grained textual similarity model is trained to matche these captions with corresponding documents (category), achieving vocabulary-fice cross-modal classification.
SCD. Semantic Category Discovery (SCD) (Han et al., 2023) is two-step approach for vocabulary- free zero-shot classification, integrating VLM models with a knowledge base that includes 1 119k WordNet nouns and 1k bird names from Wikipedia. Initially, SCD leverages KMeans clustering atop DINO features (Caron et al., 2021: Pu et al., 2024; 2023) to categorize images. Subsequently, utilizes CLIP VLM to extract a set of candidate class names for each cluster from its knowledge bases. An voting mechanism within each cluster refines this set, aiming to pinpoint the most likely candidate names. This refinement process is formed as a linear assignment problem between candidate names and clusters and performed iteratively to optimize the selection of class names. Since the code for SCD is not publicly available, we reference its experimental results from the original paper. It is important to note that SCD utilizes the entire dataset for class name discovery and assumes the number of categories to be discovered is known priori. In contrast, our FineR system relies only
21

---

Published as a conference paper at ICLR 2024
Reason LLM-Prompt
have dog.
Your task perform the following actions:
Summarize the information you get about the dog from attribute description pairs delimited by triple backticks with five sentences.
Infer 1st three possible names of dog in image based on information you get.
Structured Task Instructions
3. Cutput JSON object that uses the following format: ethree possible nemess::
the summarys. esecond sentenoe the summaryo.. <chird sentonce of the summarys. elourth sentence the summeryo, <ltth sentence the summaryss.
<tirst
Using the following lormal perform the aforementioned tasks:
GGnneral Descrition":
"general description the photo" List:
buttribule name" "sttrbule descriptio"
"attr bute name": "sttrbute descripton"
Summary:
Three possible dog names::
Input Embedding and
Output JSON:
Output Instruction
"General Description"": ""a greyhound running field"
Attributes List:
''body shape"": "a dog with long body and short tail""
""head size" : :"small"
""leg thickness" " "thin"
""fur length"": "'short"
density "dense"
"mmuzzee shape"": muzzle shape that little longer tahn the dog's head""
Summary: The dog the photo groyhound with sleak and smooth light brown coat. has long body, short tai. wrinkly nose
thicknoss, small paws with splayed toes. its tail is elender and tucked under body. The dog has squinty brown eyes, erect ears and a muzze shape that is slightly longer than its head. Its overall appearance aleek and it has breed-specific markings greyhounnd
dog has short logs with slender shape
Three possible deg names:: Greyhound, Whippel, Italian Greyhound
JSON:
Greyhoundd
"The dog the photo is greyhound with sleak and smooth light brown coat.",
'It has a long body, short tail, and small head with wrinkly nose and forehead.. shape thickness, body"". "The dog has a squinty brown eyes, orect ears, and muzzle shape that is slightly longar than itss head."
Example of Actual Output
lege "Its slander and
dog
splayed toes.",
'Whipoet:!
Itallian Greyhound": I
"The dog the photo bears resembiince Italian Greyhound.", 'It has a sleek and smooth light brown cost with long body and short tail.". "The dog short legs with slender shape and thin thicknees, and emall with spleyed toes.". "Its tail
body." dog has squinty brown eyes, erect ears, muzzie shape that slightly longer than its head."
Figure 1S: Reason L.M- details. The super-category, attributes and the corresponding descriptions acquired from the image of "Italian Greyhound" used as an example.
a few unlabeled samples for class name identification and does not require any prior knowledge about the number of categories.
CaSED. Category Search from External Databases (CaSED) employs CLIP and an expansive 54.8-millt 8-million-entry knowledge base to achieve vocabulary- classification. CaSED constructs its extensive knowledge base from the five largest sub-sets of PMD (Singh et al., 2022). It uses CLIP to retrieve set of candidate class names from its knowledge base based on the vision-language similarity score CLIP space. We compare with CaSED under our setting. Dimin with CLIP VIT-B/16 are used for class name discovery.
22

---

Published as a conference paper at ICLR 2024
D FURTHER COMPARISON WITH IMBALANCED IMAGE DISTRIBUTION
Bird-200 cACC SACC
Car-196
Dog-120
Flower- 102
Pet-37
Average
cACC
SACC
cACC SACC
cACC SACC
cACC
SACC
cACC
SACC
Zero-shot (UB)
57.4
80.5
63.1
66.3
56.9
75.5
69.7
77.8
81.7
87.8
65.8
77.6
WordNet
39.3
57.7
18.3
33.3
53.9
70.6
42.1
49.8
55.4
61.9
41.8
54.7
BLIP-2
27.5
56.4
38.6
57.3
36.6
57.7
58.2
58.4
62.3
65.1
44.6
59.0
CaSED
23.5
48.8
24.6
40.9
37.8
55.4
64.8
50.2
53.0
60.2
40.8
51.1
FineR (Ours)
46.2
66.6
48.5
62.9
42.9
61.4
58.5
48.2
63.4
67.0
51.9
61.2
Table 5: cACC (%) and sACC (%) comparison the five fine-grained datasets with imbalanced (long-tailed) data S [D S 10) for discovery. Results reported are averaged over 10 runs. Best and second-best
performances are coloured Green and Red respectively. Gray presents the upper bound (UB).
In Tab. 5, we analyze the impact of imbalanced data distribution on class name discovery across five fine-grained datasets. The imbalanced data notably degrades the performance of almost all the methods, making it challenging to identify rare classes. In extreme scenarios, some categories have only a single image available for name discovery. Despite the lower overall performance, the fluctuation in results for each method remains consistent, as detailed in Tab. 5. Nevertheless, our FineR system consistently outperforms other methods, achieving an average improvement of +7.3% in cACC and +2.2% in SACC.
E FURTHER ABLATION STUDY
Bird-200
Car-196
Dog-120
102
Pct-37 cACC SACC cACC sACC
Average
NND
MMC a
CACC eACC SACC cACC SACC cACC
SACC
X
X
X
X
44.8
64.5
33.8
52.9
49.7
62.8
45.4
47.6
63.8
64.0
47.5
58.3
X
X
X
48.3
70.2
47.4
63.1
50.8
66.5
$2.2
48.9
69.1
66.1
53.5
63.0
X
X
49.4
68.3
45.7
62.9
39.4
61.4
68.7
50.8
65.4
65.0
53.7
61.7
/
X
50.4
70.0
48.9
63.3
47.3
64.8
66.1
51.1
71.8
67.3
56.9
63.3
x
49.9
68.6
46.4
63.1
41.6
62.6
69.4
50.6
65.8
68.7
54.6
62.7
/
51.1
69.5
49.2
63.5
48.1
64.9
63.8
51.3
72.9
72.4
57.0
64.3
Table 6: Ablation study on the proposed components across the five fine-grained datasets. Our full system
colored in Green
In Sec. 3.3, we validate the effectiveness of individual components in our FineR system. We further present the detailed ablation results Tab. for the five studied datasets. Consistent with the previous observation, our Noisy Name Denoiser (NND) component improves our system performance across all datasets (second row vs first row), validating its role in noisy name refinement. On the Dog-120 dataset, however, certain enhancements like the multi-moda classifier (MMC) do not contribute to better performance (third row vs second row). This ascribed to the large visual bias in the samples for discovery, a trend also observed in the Car- and Pet-37 datasets (third row vs second row). For instance, the color patterns within the same breed of dogs or cats, or within the same car model, can vary significantly. Despite this, modality fusion hyperparameter a 0.7 and sample augmentation (K' = 10) partially mitigate this issue on the Car- and Pet-37 datasets. Overall, our complete FineR system delivers the best performance. Addressing visual biases stemming from high intra-class variance in fine-gran -reined categories presents an intriguing avenue for future research.
F FURTHER GENERALIZABILITY STUDY WITH OPEN-SOURCE LLMS
We replace the LLM of our FineR system with Vicuna- 13B (Chiang et al., 2023), an open-source LLM based on LLaMA (Touvron et al., 2023), and evaluate it the three most challenging fine- grained datasets. The findings, presented in Tab. 7, reveal that the performance of our system remains consistent whether using Vicuna-133 or ChatGPT for the FGVR task. Both language models are effectively leveraged by FineR to reason fine- fine-graine grained categories based on the provided visual cues. These results suggest that our FineR system is robust and generalizable across different L.LMs.
23

---

Published as a conference paper at ICLR 2024
Bird-200
Car-196
Dog- 120
cACC SACC cACC SACC cACC SACC
Vicuna- 13B
50.7
68.9
47.9
61.7
50.3
67.2
ChatGPT 51.1
69.5
49.2
63.5
48.1
64.9
Table 7: Generalization study with open-source LLM Vicuna-13B.
G FURTHER SENSITIVITY ANALYSIS
In this section, we extend our component ablation study to scrutinize the sensitivity of our FineR system to various factors. explore the impact the hyperparameter a on multi-modal fusion during classifier construction in App. G.1. Additionally, we examine the effects of sample augmenta- tion times K in mitigating visual bias due limited sample sizes in App. G.2. We also analyze the system's s performance under varying amounts of unlabeled images per category, denoted by Dirain for class name discovery. Finally, we assess the influence of CLIP VLM model size on our system performance. Clustering Accuracy (cACC) and Semantic Similarity (sACC) are reported across all five fine-gr -en-igai datasets.
G.1 ANALYSIS OF THE HYPERPARAMETER
Bird
Car
Dog
Flower
Pet
Average
80
80
cACC
SACC
60
40
60
Vi-ony
Text-only
Vision-only
Text-only
0 0.1 0,2 0,3 0,4 0,5 0,6 0,7 0,8 0,9 1
O 0,1 0,2 0.3 0,4 0,5 0,6 0,7 0.8 0,9
Figure 16: Sensitivity analysis of in FineR system. The average performance is highlighted in Green
The hyperparameter a modulates coupled with sample augmentation, as discussed in Sec. 3.3. Ranging from 0 to 1, balances visual and textual information. As shown in Fig. 16, we observe an optimal trade-off a = 0.7 across most datasets. Notably, settings below 0.6 lead performance declines in both cACC and sACC, likely due to the visual ambiguity given by the non-fine-tuned visual feature for the challenging fine-grained objects. For the Flower- 102 dataset, lower improves clustering, in line with KMMans baseline results (Tab. 1). However, the s SACC drop indicates a lack of semantic coherence in these clusters. Thus, we have chosen 0.7 as the default setting for our system.
fusion our classifier, enhancing performance when
G.2 ANALYSIS OF THE HYPERPARAMETER K
The hyperparameter dictates the number of augmented samples generated through a data aug- mentation pipeline featuring random cropping. color jittering. flipping, rotation, and perspective alteration. Our experiments across five fine- fine-graine grained datasets, detailed Fig. 17, reveal that additional augmented samples beyond point fails to give further improvements. Moderate augmentation is sufficient to mitigate visual bias to certain degree. Thus, we opt for K 10 as our default setting balance efficacy and computational cost.
G.3 ANALYSIS OF VLM MODEL SIZE
In Fig. 18, we explore the relationship between the model size of the CLIP VLM utilized in FineR and our system performance. Our analysis reveals direct, positive correlation between larger VLM
24

---

Published as a conference paper at ICLR 2024
Bird
Car
Dog
Flower
Pet
Average
80
80
cACC
SACC
60
60
8
40
40
0
10
20
30
40
50
60
70
80
90
100
0
10
20
30
40
50
60
70
80
90
100
Figure 17: Sensitivity analysis FineR system. The average performance is highlighted
Green
VIT-L/14
VIT-B/16 (default)
VIT-B/32
RN101
RN50
80
80
cACC
sACC
70
70
60
60
50
50
40
Bird
Car
Dog
Flower
Pet
Average
Bird
Car
Dog
Flower
Pet
Average
Figure 18: Sensitivity analysis the CLIP VLM model size in FineR system.
models and enhanced system performance. Notably, when using the CLIP ViT-L/14 model, FineR registers an incremental performance gain of -6.7% in cACC and +1.1% in sACC, compared to the default ViT-B/16 model. This finding underscores the potential for further improvement in FineR through the adoption of more powerful VLM models.
G.4 ANALYSIS OF THE NUMBER OF UNLABELED IMAGES USED FOR CLASS NAME DISCOVERY
Bird
Car
Dog
Flower
Pet
Average
80
80
cACC
SACC
60
40
20
40
2
4
5
7
10
3
6
8
10
Figure 19: Sensitivity analysis of Dirae in FineR system. The average performance is highlighted Green In our fine-grained visual recognition (FGVR) framework, we operate under the practical constraint of having very limited unlabeled samples per category for class name discovery. Specifically,
25

---

Published as a conference paper at ICLR 2024
[Dtrain is much smaller than [Diest]. While explore both balanced ((D 3) and imbalanced (1 Di Σ 10) sample distributions in the main paper, an important question arises: does our FineR system require more samples for improved identification? We vary the number of unlabeled images per category, [ from to 10 and assessed system performance. As shown Fig. 19, increasing the sample size does not lead additional performance gains, indicating that FineR is efficient and performs well even with a sparse set of unlabeled images (e.g., [ 3).
Ground-truth:
Rufous Hummingbird
Ground-truth:
Brown Pelican
Humming
WordNat:
Pelocanus Onocrotalus
Hummingbird
BLIP-2:
Palican
Bird-200
CaSED:
Hummingbird
CaSED:
FineR (Ours):
Rufous Hummingbird
(Ours):
Yellow-throated
Ground-truth:
Common Raven
WordNet:
WordNet:
Black Vulture
Tanager
BLIP-2:
Tul
CsSED:
CaSED:
Corvid
FineR (Ours):
Yollow-troaated
FineR (Ours):
Common Raven
Ground-truth:
Honda Accord Coupe 2012
Ground-truth:
Volkswagan Golf Hatchback 2012
WordNer:
Subcompact Car
WordNet
Lynx pardina
BUP-2:
Honda Accord
Volkawagen
Car-196
CaSED:
Camry
CaSED:
FineR (Ours):
Honda Accord Coups 2012
FineR (Ours):
Volkswagen Golf 2-Door Hatchbaakk 1990s
Ground-truth:
Ground-truth:
Dodge
2010
WordNet:
Hummer
WordNet:
Pickup Truck
BUP-2:
Hummer H2
CsSED:
Hummer
CaSED:
FineR (Ours):
Crew
FinaR Daliots Crew Cab 2012
Ground-truth:
Gordon Setter
Ground-truth:
Italian Greyhound
WordNe::
Gordon Setter
Jury
BLP 2:
A black and tan
BLIP-2:
broed
Dog-120 120
CaSED:
Bernese
CaSED:
Whispel
FineR (Ours):
Gordon Setter
FineR (Ours):
Italian Greyhound
Ground-truth:
Irish Water Spaniel
Ground-truth:
Keeshond
WordNet:
Irish Water Spaniel
WordNet:
Keashond
BLP 2:
A black dog
BLIP-2:
CaSED:
Poodle
CaSED:
Keeshond
FineR (Ours):
Irish Water Spaniel
FineR (Ours):
Keashond
Ground-Iruth:
Water Lily
Ground-truth:
Artichoke
WordNet:
Fragrant Water Lily
WardNet:
Artichoke
BUP-2:
Water Lily
Flower-102
CaSED:
Lotus
FineR (Ours):
Pink Water Lily
(Ours):
Ground-Iruth:
Petunia
Ground-truth:
Rose
WordNet:
Tea Hosa
BLIP-2:
Pink Rose
CaSED:
Rose
FineR (Ours):
Purple
(Ours):
Orange
Ground-Iruth:
Egyptian Mau
Ground-truth:
Maine Coon
WordNet:
Felle Bengslensis
WordNet:
Angora Cal
BUP-2:
Asian Sevannal Cal
BLIP-2:
long haired cal
CsSED:
Ocical
CaSED:
Regdol
Pet-37
FineR (Ouns):
Egyptian
FinaR (Ours):
Maino Coon
Ground-truth:
American Bulldog
Ground-Iruth:
Birman
WordNet:
English Bulldog
WordNet:
Siamese Cat
BUP-2:
A pit terrier
BLIP-2:
Siamese Cat
Buildog
Ragdoll
FineR (Ouns):
Buildog
(Ours): Birman
Figure 20: Further qualitative results on the five fine- grained datasets. Correct, partially correct and incorrect predictions colored Green, Pink, and Red, respectively, Blue highlights the prediction that is even more precise than ground truth category names.
26

---

Published as a conference paper at ICLR 2024
H FURTHER QUALITATIVE RESULTS
We present additional qualitative results for the five fine-grained datasets in Fig. 20. The Car- C9r-196 dataset is one of the most challenging dataset due the need infer the specific year of manufacture of the car in the image. We find that on the Car-196 196 dataset, when neither our method nor other baseline methods give the completely correct answer, partially correct answer given by our method is the closest to the correct answer (e.g.. "Dodge Dakota Crew Cab 2012" v.s. the ground- truth label "Dodge Dakota Crew Cab 2010"). Interestingly, similar shown in Fig. 6, FineR system can often give more precise and descriptive predictions than the ground-truth labels (e.g.. "Orange Rose" V.s. the ground sound-truth label "Rose"). Through this additional qualitative analysis, further demonstrate that our FineR system can not only accurately reason fine-grained categories based on the visual cues extracted from the discovery images, but also include visual details during its knowledge-based reasoning process, resulting in more precise and descriptive fine-grained discovery.
qualitative visualization
I FURTHER FAILURE CASE RESULTS
We further present qualitative analysis of failure cases in Figure Fig. 21, compared with state- of-the-art vision-language models. We observe that, while most models fail to accurately identify and predict the correct names, method often provides the most fine-graines grained predictions. These are not only closer to the ground truth class name but also capture the attribute descriptions more accurately, suggesting that method makes 'better' mistakes. More interestingly, the proposed novel Pokemon dataset, almost all compared methods predict only the real-world counterparts (e.g..
Ground-truth:
Yellow-bellied Flycatcher
Ground-truth: Horned Grebe
Birce 200
BLIP-2
Yollow-rampied Warbier
BLIP-2
Grabe
Bird
LENS
Eastem Wood Powee
LENS
Clarks Grebe
MiHGGP-44
Smell Bind
MirGGP-4
Common Loon
(Ours):
Flycatcher
FineR [Ours]:
Black-necked Grebe
Ground-truth:
Volkswagen Golf Hatchback 2012
Ground-truth:
Dodge Dakota Crew Cab 2010
Car-199 196
BLIP-2
Volkewagen
BLIP-2
Dodge Dakota
LLaVA
Car
LLaVA
Pickup Truck
LENS
Volkowagen Golf MK2
LENS
Silver Truck
MiniGPT-4
Volkewagen
MriGPT-
Pickup
FinaR (Ours]:
Volkswagen Golf 2-Door Hatchback
FineR (Ours):
Dodge Dakota
Ground-truth:
Brabancon Griffon
Ground-truth:
Irish Terrier
Dog-120 20
BLIP-2
Pakingesa
BLIP-2
Terrier
Dog
LLaVA
Dog
Chinuahua
MinGGPT-4
Small Dog
MriGPT-4
Puppy
FineR (Ours):
Brussels Griffon
FineR (Ours):
Airedale Terrier
Flower- 002
Ground-truth:
Water Laly
Ground-truth:
Pink-yallow Dahlia
BLIP
Water Lily
BLIP-2
Calencula
Water Lily
LLaVA
Flower
LENS
Lokus
LENS
Dahia
MiniGP-44
Water Lily
MrniPT-4
Dahi Flower
(Ours):
Water
(Ours]:
Yellow- Prrong-rangeee
Ground-truth:
Japanese Chin
Ground-truth:
Pet-37
BLIP-2
LLaVA
Dog
LLaVA
Shaggy Dog
Tan White Dog
LENS
Dog
MiNGPP-44
Dog
MiGGPT4
Shaggy
FineR (Ours):
Shih Tzu
FineR (Ours]:
Small-stee sized Terrier
Ground-truth:
Pidgey
Ground-truth:
Squirtle
Pokemon
BLIP-2
Sylveon
BLIP-2
Squirtio
LENS
Contes Sparrow
LENS
Turtlo
MIGGPT-4
Cartoon Bird
MH+GPT-
Turlle
FineR (Ours):
Pidgectto
Figure 21: Further qualitative failure case results on the five fine-grained datasets and the Pokemon datasct. Correct, partially correct and incorrect predictions are colored Green, Pink, and Red, respectively. Blue highlights the prediction that is even more precise than ground truth category names.
27

---

Published as a conference paper at ICLR 2024
"Squirtle" "Turtle" or "Pidgey" "Bird")of the Pokemon characters, highlighting domain gap between real-world and virtual categories. In contrast, although not completely accurate, method offers predictions that are closer to the ground truth (evolutionized Pokemon characters) for the two Pokemon objects.
HUMAN STUDY DETAILS
In this section, we describe the details of our human study, designed to establish a layperson-level human baseline for the FGVR task. Formally, we tailor two questionnaires English for the Car- and Pet-37 dataset, respectively. In each questionnaire, we present participants with one image for each category within the target dataset. instructions follows: (i) Your task is to answer the name of the car (or pet) present in each image as specific as you can. You do not have to be sure about the specific name to answer; (ii) you model (or pet breed) present in the image, you describe
the questionnaire, participants receive initial task
have no clue about the name of the and summarize how does this car (or pet) looks like in one sentence (e-g.. "It is a two-door black BMW sedan."). Recognizing that English may not be the first language of all participants, include the prompt: "feel free to use your preferred translator for underss tanding answering the question". Images paired with questions are then presented sequentially. To aid participants, we include the same attributes acquired by our FineR system beneath each question as reference for describing the car (or pet). Two questions from the questionnaires are showcased in Fig. 22. We carry out this study with 30 participants. Given that the two questionnaires involve visual recognition and description of 196 and 37 images respectively, it takes each participant between to 100 minutes to complete the study for both datasets. Leveraging the free-form text encoding abilities of CLIP VLM, we directly use the gathered textual responses to generate a text-based classifier for the two datasets from each participant's 's responses. We are thus able to perform evaluation inference on the two datasets using CLIP VLM along with the layperson classifiers. The layperson level performance presented in Fig. 5 for both datasets averaged across the 10 participants.
Question Whet is the model owme of the this photo (deroo example: 2020)?
you
the exact name, please describe the sisual Tookka Abat of the pet the photo in serdiece.
do keow the soucc mame, plaase descrila the ziaua Chocky Aet chei
Visual attribute hints for better description: body shapel
and giant block tall.)
Treed shape: sine" werriles
description:
heek lengthf
approseaate year manutacture (e.g. 1990s, 2000s 2010s!:
thickness
possible make (automobile manufacturess)
jow sioc baw shape, 'oo shaps: "aal ahapel tall lingtt, hall positior,
body style (Sedon, Wagon, SUM Coupe, Roadstee Truck Cat, Convertible, Minkinn Van, Hatcheadk, etc.. body color' hoof color, 'ade hright, lengthi with ambleen/aaa
locat color, boot lengthi, "far tooture Tur lenglif Yur disriity..
panten,
shope,
lise
logblen/logo placament" grille desigrt, e fyrllle shape iyrlle sirel Histinttivn ebements of the golles. headd
rear
posiiiai eur shapel,
"muzze thape: snout lengthi, nate chape, "hose color, body shapa propertiert,
"heaclight shape, headlight gitt dleaggt, "aallight shaps: haillignt stoet wheel desight Wheel sine: Wheee peiteen
"rreed-peaffcc markings or patterns (n.g. spots, stripes, patches)); fovees repesrance (eig sllwic furfy musecterr)
specitt
contours accent lines'
rroffine shope. Soor handle desight, dide miror desight, designi hood chonge.
"acial markinge:, body markingsi, "whaker lengst, Whiller shape), 'dlw length, 'claw shape"
Car-196 Questionnaire Question Example
Questionnaire Question Example
Figure 22: Example questions from the human study questionnaires for the Car-196 and Pet-37 datasets.
28

---

Published as a conference paper at ICLR 2024
K FURTHER DISCUSSIONS
K.1 REASONING CAPABILITY OF LARGE LANGUAGE MODELS
In a world awash with data, Large Language Models (LLMs) such as ChatGPT (OpenAI, 2022) and LLaMA (Touvron et al., 2023) have emerged as prodigious intellects of reasoning and problem solving capabilities (Brown et al., 2020; Wei et al., 2022; Kojima et al., 2022: Yao et al., 2023). They power transformative advances in tasks as diverse as common sense reasoning (Bian al., 2023), visual question answering (Shao et al., 2023; Zhu et al., 2023a; Berrios et al., 2023), robotic manipulation (Huang et al., 2023), and even arithmetic or symbolic reasoning (Wei et al., 2022). What if we could elevate these language-savvy powerhouses beyond the limitations of text and bestow upon them the gift of sight,' specifically for Fine-grained Visual Recognition (FGVR)? In this work, we propose FineR system that translates useful visual cues for recognizing fine-grained objects from images into a language these LLMs can understand, and thereby unleashing the reasoning prowess of LLMs onto FGVR task.
K.2 THE STORY OF BLACKBERRY LILY
Ground-truth
WordNet
BLIP-2
CaSED
FineR (Ours)
Real Images Google Image
GT Flowarr122
Generated Images
Stable Diffusion
Blackberry Lily
Peruvian Lily
Pred: Liium Senegalensis
Pred: Gloriosa
Pred: Orange-spotted Lil
Figure 23: Reverse comparison of prediction results for the "Blackberry Lily" image (upper-left corner) in
102. We evaluate the visual counterparts associated with the predicted semantic concepts. To conduct this comparison, we employ two distinct methods for inversely identifying their visual counterparts: (i) Google Image Search: we query and fetch images that are paired with the predicted class names from Google; (ii) Stable Diffusion: we utilize the predicted semantic class names text prompts generate images using Stable Diffusion. Partially correct and wrong predictions are color coded. None of the methods correctly predict the ground-truth label.
conditioned
Predicted category names encapsulate valuable semantic information that should align with or align closely the ground truth. The predicted semantics will be used in downstream tasks. Therefore, semantically robust prediction is important, even when it is wrong prediction. For instance, prediction of "Peruvian Lily" against the ground truth "Blackberry Lily" is more tolerable than incorrectly predicting "Rose". As discussed in Sec. 3.1 our FineR system showcases its capability for semantic awareness, particularly when all the methods wrongly predict a "Blackberry Lily", offers the most plausible prediction of "Orange- Oraage-ssottee Lily". Nevertheless, sole focus on textual semantics might not fully assess the quality of a prediction, which is important for downstream applications like text-to-image generation. We address this by additionally employing a unique methodology of reverse visual comparison using the prediction results of "Bl "Blackberr" rry Lily". as illustrated in Fig. 23. Specifically, we employ two techniques to locate visual analogs for the predicted classes: Google Image Search and Stable Diffusion (Rombach et al., 2022) text-to-image generation. The upper row in Fig. 23 shows images retrieved via Google Image Search using the predicted classes, while the lower row displays images generated by Stable Diffusion, conditioned on the predictions as text prompt.
This reverse comparison reveals that the visual counterparts retrieved through FineR are strikingly similar to the ground- ground-truth, corroborating its robustness. In contrast, reversed prediction results from WordNet and BLIP-2 exhibit only partial similarities in petal patterns, aligning with their semi- accurate class names. The mispredictions from CaSED, however, lack such visual congruence. This simple yet insightful reverse comparison distinetly highlights the advantages of our reasoning-based
29

---

Published as a conference paper at ICLR 2024
approach over traditional knowledge base methods. Accordingly, proceed to discuss these two methodological paradigms as following.
Forced Misprediction. Knowledge base methods such the WordNet bascline and CaSED rely heavily on the CLIP latent space to retrieve class names that are proximal to input images. They assume that the knowledge bases covers the knowledge (e.g., class names) for the target task. Despite the certain visual resemblance in Fig. 23, these methods overlook one of the most obvious visual attributes, such as "color", in the input image. Thus, the reverse visual results are significantly different from the ground-truth. This underscores how the retrieval process is disproportionately influenced by the CLIP similarity score. If the CLIP latent space lacks a clear decision boundary for the given input, these methods are compelled to choose an incorrect class name from its knowledge base. This process is uninterpretable and ignores high-level semantic information. Consequently, such wrong predictions contain little useful visual information, leading to more egregious errors that are difficult to justify.
Knowledge Base Retrieval Methods:
Our Reasoning-basee
Approach: Making Useful Mistakes. In contrast to knowledge-base
methods, our FineR approach demonstrates semantic awareness, interpretability, and informativeness even in the face of errors. FineR system conditions its predictions on observed visual cues from the images, enriching the reasoning processes using LLMs. For example, if the visual cue "color: orange" " is present, the model's reasoning avoids class names lacking this attribute. Even when FineR makes an incorrect prediction, it still provides semantically rich insights, such as the attribute "Orange-spott spotted". Therefore, shown in Fig. 23, our method not only offers textually informative results about the ground truth but also produces highly similar vision analogous, whether retrieved from Google Images or generated by Stable Diffusion. This confirms that FineR delivers high-quality, interpretable, and semantically informative predictions for both visual and textual domains, thereby enhancing the robustness of downstream applications.
K.3 WORKING IN THE WILD
In this section we expand upon the comparison of our FineR with the LLM enchanced visual recognition methods such as PICa (Yang et al., 2022) LENS Berrios et al. (2023). As mentioned earlier in Sec. 4, such systems first extract captions from an image using off-the-shelf captioning model and then feeds the question, caption and in-context examples to induce LLMs to arrive at the correct answer. In particular, LENS, in addition to captions, also builds visual vocabularies by collecting tags (or class names) from image classification, object detection, semantic segmentation and the visual genome datasets (Krishna et al., 2017). Examples of such datasets include ImageNet (Russakovsky et al., 2015), Cats and Dogs (Parkhi al., 2012), Food 101 (Bossard et al., 2014), Cars-196 (Krause et al., 2013) and so on. Furthermore, LENS uses Flan-T5 (Longpre et al., 2023) to extract comprehensive descriptions from these class names, as proposed in the work of (Menon & Vondrick, 2023). Based on the vocabulary and image descriptions it uses CLIP to first tag a test image. Subsequently, it feeds the tags and the previously extracted captions to a LLM to arrive at an answer given a question. Aside from being unfair to FineR, where we do not assume priori knowledge of any vocabulary, LENS has two major drawbacks: (i) Such an approach can only work if an image belongs to the well known concepts defined in a finite vocabulary, and will fail if the concept depicted in the image falls outside the vocabulary. (ii) Since CLIP is queried during the image tagging with such a large vocabulary, it shares the same disadvantages the WordNet baseline, as the semantic vocabulary is quite unconstrained. To demonstrate this behaviour we show qualitative results in Fig. 24 where we compare LENS with FineR on five academic benchmarks and our Pokemon dataset. As hypothesized, LENS is really competitive to FineR on Bird-200, Car-1 196, Dog-1 Flower-102 and Pet-37 datasets, hitting the right answer several times as our FineR. This is due to the fact that LENS's 's vocabulary already contains the concepts present in the fine-grained datasets that are being tested for. Therefore, CLIP has no difficulty in zeroing in the right tag, followed by the right answer from the L.M. Such a trend has already been demonstrated with our UB baseline in Sec. 3.1. However, LENS fails miserably in the Pokemon dataset, misclassifying all of the six instances. For instance, most of the time the characters are classified as either real world animals (e.g.. "Bird" or "Newt") or as generic name "Pokemon". Conversely, our FineR do not face difficulty when presented with novel fine-grainee concepts. Thus, unlike LENS, our FineR is designed to work not only in the academic setup but also in the wild. We believe that this versatility in FineR is indeed a big first step in democratizing FGVR.
30

---

Published as a conference paper at ICLR 2024
Ground-truth: Orchard Oriale
Ground-truth: Darkeeed Junco
Bird-200
LENS:
Orchard Oriole
LENS:
Junco
FineR (Ours): Orchard Oriale
FineR (Ours): Dark-eyed Junco
Ground-truth:
Jeep
Ground-truth:
Bentley Continental GT Coupe 2012
Car-196
LENS:
Jeep Compess SUV 2012
Bentley Continental GT Coupe 2007
FineR (Ours): Jeep Grand Cherokee SUV 2012
FineR (Ours): Bentley Continental GT Sedan 2010
Ground-truth:
Irish Setter
Ground-truth: Black-and-tan Coonhound
Dog-120
LENS:
Irish Setter
Dog
FineR (Ours): Irish Setter
FineR (Ours): Black-end-len Coonhound
Flower- 102
Ground-truth:
Ground-truth: Blackberry Lily
LENS:
Pink Flower
LENS:
Blackberry
FineR (Ours): Pink Lotus
FineR (Ours): Orange spotted Lily
Ground-truth: Pomeranian
Ground-truth: Ragdoll
Pet-37
LENS:
Pomeranian
LENS:
Ragdoll
(Ours):
(Ours): Ragdoll
Ground-truth: Charmander
Ground-truth: Charmeleon
LENS:
Nevt
LENS:
Pokemon
FineR (Ours): Charmander
FineR (Ours): Charmeleon
Pokemon
Ground-truth: Squirtie
Ground-truth: Wartortie
LENS:
Turtle
LENS:
Pokemon
FineR (Ours): Wartortle
FineR (Ours): Wartortle
Ground-truth: Pidgey
Ground-truth:
Pidgeotto
FineR (Ours): Pidgeotto
FineR (Ours): Pidgeatto
Figure 24: Further qualitative comparison with LENS on the five fine- line-grained datasets and the Pokemon datasets. Correct, partially correct and incorrect predictions are colored Green, Pink, and Red, respectivoly. Blue highlights the prediction that is more precise than ground-truth category names.
K.4
SETTING COMPARISON
Zero-Shot Transfer and Prompt Engineering in Vision-Language Models. The advent of vision- language models (VLMs) like CLIP (Radford et al., 2021) and ALIGN (Jia al., 2021) has led to notable successes in zero-shot transfer across diverse tasks. To enhance VLM performance, the field of vision and learning has increasingly focused on prompt enginecring techniques, encompassing prompt enrichment and tuning, Prompt enrichment methods, (Menon & Vondrick, 2023) and (Yan et al., 2023), leverages ground-truth class names to obtaín attribute descriptions via GPT-3 queries. then utilized to augment text-based classifiers. Furthermore, prompt tuning
These descriptions methods, such as CoOp (Zhou et al., 2022b), TransHP (Wang et al., 2023), and CoCoOp (Zhou et al., 2022a), involve the automatic learning of text prompt tokens in few-shot scenarios with labeled data, achieving substantial enhancements over standard VLM zero-shot transfer methods. In contrast, our method only leverages unlabeled data few-shot scenarios for discovering the class names, as the compared in Tab. 8. In addition, most importantly, as indicated in Tab. 9, all VLM- A-M-daedd methods and tasks discussed thus far necessitate pre-defined class names as vocabulary to work inference time for constructing text classifiers. While such pre-defined vocabularies are less resource-intensin than expert annotations, they still require expert knowledge for formulation. For example, accurately defining 200 fine-grai fine-grained bird species names for the CUB-200 dataset (Wah et al., 201 is beyond the expertise of a layperson, thereby constraining the practicality and generalizability of these methods. In stark contrast, compared in Tab. 9, approach is vocabulary-free, operating without the need for pre-defined class name vocabularies. This absence of pre-existing vocabulary renders traditional V--M-bseed methods ineffective in our challenging task. Instead, vocabulary-free FGVR model in
31

---

Published as a conference paper at ICLR 2024
Model Training
Annotation
Training Data
Setting
Supervision
Base Classes
Novel Classes
Standard FGVR
Class Names
Fall Labeled Labeled
Generaizeed 7emo Transfer
Class Altributes
VLM Zero-Sho. Transfer
X
VI. Prompt Eniichmen
Prompt Tuning
Class Names
Labeled
Vocabulary free FGVR (Ours)
X
Few Shots Unlabeled (e....)
Table 8: Training setting comparison between our novel vocabulary free free FGVR task and other related machine learning settings.
Setting
Closses
Novel Classes
Output
Prior Knowledge
Standard FGVV
Clossiyy
None
Nanes
Class Names
Generalized Zero-Shox Transfer
Closdyy
Classity
Cluster index
Class Names Attiibates
Zero-Shox Taanfer
Clusity
Names
VLM Prompt Enichment
Nose
Classity
Names
Class Names
VLM Prompt Tuning
Clasdyy
Classity
Names
Class Names
Vocabulary
(Ours)
Cassity
Table 9: Inference setting comparison between our novel vocabulary- free FGVR task and other related
machine learning settings.
our framework is tasked with automatically discovering class names from few unlabeled images (e.g., 3-shots). Besides, method does not need any prompt enrichment augmentation.
Generalized Zero-Shot Learning (GZSL). Under GZSL (Liu et al., 2018; 2023b) setting, a model is given a labeled training dataset well as an unlabeled dataset. The labeled dataset contains instances that belong to set of seen classes (as known as base classes), while instances in the unlabeled dataset belong to both the seen classes as well as to an unknown number of unseen classes (as known as novel classes). Under this setting, the model needs to either classify instances into one of the previously seen classes, or the unseen classes (or unseen class clusters Liu et (2018)). addition, GZSL imposes additional assumption about the availability of prior knowledge given auxiliary ground-truth attributes that uniquely describe each individual class in both base and novel classes. This restrictive assumption severely limits the application of GZSL methods in practice. Furthermore, the limitation becomes even more strict in fine- mme-grained recognition tasks because attributes annotation requires expert knowledge. In contrast, compared in Tab. our novel FGVR setting approach is training-free and does not require class names and attributes supervision from base classes and data. Most significantly, highlighted in Tab. 9, approach is distinctively vocabulary- thus eliminating the strong assumption of having access to class names and their corresponding attribute descriptions, which typically require specialized expert knowledge.
32


Reviews:
[{"review_id": "sq0zJfeGHh", "reviewer": "Reviewer_oRxX", "summary": "FineR leverages large language models to identify fine-grained image categories without expert annotations, by interpreting visual attributes as text. This allows it to reason about subtle differences between species or objects, outperforming current FGVR methods. It shows potential for real-world applications where expert data is scarce or hard to obtain.", "questions": "I want to discuss with the authors that:\nWhat is your view of NOVELTY of a work using LLMs without training anything?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is a good paper, and the advantages are:\n1. The utilization of LLM to vocabulary-free FGVR tasks is novel;\n2. The paper is generally well-written and easy to follow;\n3. Good performance in Table 1 \n4. The well utilization of LLM.", "weaknesses": "Advice:\n1. Add the citation of some highly related missing works:\n(1) Transhp: image classification with hierarchical prompting; it also focuses on the fine-grained image classification task. Also, it takes advantage of the recently proposed prompting technique in CV. Is your used LLM better or prompting better?\n(2) V2L: Leveraging Vision and Vision-language Models into Large-scale Product Retrieval; it also focuses on the vision language model and fine-grained visual recognition. The paper is the champion of FGVC 9. A discussion of comparison seems necessary.\n2. Could you show more examples of Fig. 2? Or any failure cases of Fig. 2?\n3. What is your view of NOVELTY of a work using LLMs without training anything?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aedDAZRLDB", "reviewer": "Reviewer_3hkC", "summary": "This paper proposes a fine-grained semantic category reasoning method for fine-grained visual recognition, which attempts to leverage the knowledge of large language models (LLMs) to reason about fine-grained category names. It alleviates the need of high-quality paried expert annotations in fine-grained recognition.", "questions": "The novelty should be clarified and the difference from existing generalized zero-shot learning methods using visual attributes for recognition should be explained. More corresponding experiments and analyses should be added.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe presentation of this paper is clear and the proposed method is easy to follow. \n2.\tThis paper proposes to extract visual attributes from images into the large language models (LLM) for reasoning the fine-grained subcategory names, which is a promising way to alleviate the high need for expert annotations in fine-grained recognition.", "weaknesses": "The weaknesses are as follows：\n\n1.\tThe novelty of this paper should be further demonstrated. The proposed method seems an intuitive combination of existing large-scale models, such as the visual question answering model, large-language model and vision-language model, etc. Besides, extracting visual attributes from images for recognition is widely used in generalized zero-shot learning methods such as [a]. \n\n2.\tThe effectiveness of the proposed method should be further verified. The recognition accuracy on the CUB dataset is relatively low. Besides, existing generalized zero-shot learning methods such as [a] should be compared and more corresponding analyses should be added.\n\n3.\tThere lacks a complete recognition example for explaining the whole procedure, which should be added for better understanding.\n\n[a] Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learning. CVPR 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5gpcq1k4Ev", "reviewer": "Reviewer_WYuk", "summary": "- This paper proposes a novel training-free FGVC pipeline.\n\n- The key idea is to use multiple LLMs to generate the visual concepts to represent the imae of a fine-grained category, which allows not only trainin-free but also zero-shot inference.\n\n- Besides, a novel Pokemon dataset is proposed to further foster FGVC.\n\n- Extensive experiments on multiple standard FGVC datasets and the proposed dataset show the effectiveness of the proposed method.", "questions": "My concerns on Q1, 2, 3 have been well addressed. Although I still have some minor concerns, it does not impede that this work has reached the accept threshold. So, I decide to improve my rating to borderline accept.\n\n%%%%%%% before rebuttal\n\nAlthough the strength seems to overwhlem the weakness, as an expert in vision and FGVC, the reviewer still holds a critical view towards this LLM based work, which may be good enough for pulibcation. Some specific questions:\n\nQ1: As there is already some good VLM solution to do zero-shot FGVC and achives nearly 90% accuracy, is it necessary to solely address FGVC by pure concepts from LLM? Besides, is it necessary to solely rely on LLM instead of VLM or visual representation for FGVC?\n\nQ2: The idea of this paper is actually not very new now. The ICCV paper [3] implements similary idea. The difference and comparison should be made and clarified.\n\nQ3: So many details of the proposed dataset are missing. The specific comments have been listed in the weakness.\n\nQ4: Is the proposed Pokemon dataset can really fit the fine-grained patterns? Or, the LLM and VLM learnt from real-world data, can really discriminate the fine-grained pokemon? These issues may make its contribution actually limited for FGVC community.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper is well-written, easy-to-follow and well-presented.\n\n- A novel Pokemon dataset is proposed to benefit FGVC.\n\n- The proposed method is novel and interesting, which provides a new way to do training-free FGVC based on LLM.\n\n- The proposed method is experimentally better than existing state-of-the-art methods.", "weaknesses": "- Although the novelty and technique score is satisfactory for ICLR, a major issue is whether it is necessary to only leverage LLM for FGVC under the proposed setting. In fact, as far as the reviewer concerns, some other only learning prompt on VLM can already achieves more than 90% accuracy for zero-shot FGVC. For example:\n\n[1] Conditional Prompt Learning for Vision-Language Models. CVPR 2022.\n\n[2] Learning to Prompt for Vision-Language Models. IJCV 2022.\n\nIn fact, this issue is critical, at least from the vision community. What should be the correct role of LLM for FGVC? Should only the concepts from LLM be used for FGVC? Or it should be jointly used with visual representation to aid FGVC? Soley relying on LLM for vision tasks is in a way that the reviewer does not appreciate.\n\n- The idea to use LLM to generate concepts for FGVC, is already not new now. Some recent works implement idea:\n\n[3] Learning concise and descriptive attributes for visual recognition. ICCV 2023.\n\n- The proposed dataset also has multiple issues for publication. For example:\n\n(1) The exact number of samples, how the training and testing set is divided, and etc. Many details are missing in the main submission and the supplementary materials.\n\n(2) As the dataset is Pokemon, not real-world FGVC categories, some further questions raise: Is it able to describle the fine-grained patterns? Besides, is the LLM leart from real-world able to differentiate the fine-grained patterns on Pokemon. These issues make the contribution of FGVC to be doubted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)