Tree of Attributes Prompt Learning for Vision-Language Models
Authors: Tong Ding
Wanhua Li
Zhongqi Miao
Hanspeter Pfister
Venue: ICLR 2025 Poster (2025)
Keywords: Prompt Learning
Vision-Language Models
Tree of Attributes
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Prompt learning has proven effective in adapting vision language models for downstream tasks. However, existing methods usually append learnable prompt tokens solely with the category names to obtain textual features, which fails to fully leverage the rich context indicated in the category name. To address this issue, we propose the Tree of Attributes Prompt learning (TAP), which first instructs LLMs to generate a tree of attributes with a “concept - attribute - description” structure for each category, and then learn the hierarchy with vision and text prompt tokens. Unlike existing methods that merely augment category names with a set of unstructured descriptions, our approach essentially distills structured knowledge graphs associated with class names from LLMs. Furthermore, our approach introduces text and vision prompts designed to explicitly learn the corresponding visual attributes, effectively serving as domain experts. Additionally, the general and diverse descriptions generated based on the class names may be wrong or absent in the specific given images. To address this misalignment, we further introduce a vision-conditional pooling module to extract instance-specific text features. Extensive experimental results demonstrate that our approach outperforms state-of-the-art methods on the zero-shot base-to-novel generalization, cross-dataset transfer, as well as few-shot classification across 11 diverse datasets. Code is available at https://github.com/HHenryD/TAP.

Published as a conference paper at ICLR 2025
TREE OF ATTRIBUTES PROMPT LEARNING FOR VISION- LANGUAGE MODELS
Tong Ding!.2 Wanhua Lif Zhongqi Miao
Hanspeter Pfister*
Harvard University Mass General Brigham
3 Microsoft
ABSTRACT
Prompt learning has proven effective in adapting vision language models for downstream tasks. However, existing methods usually append learnable prompt tokens solely with the category names to obtain textual features, which fails fully leverage the rich context indicated in the category name. To address this issue, we propose the Tree of Attributes Prompt learning (TAP), which first instructs LLMs to generate a tree of attributes with "concept attribute description" structure for each category, and then learn the hicrarchy with vision and text prompt tokens. Unlike existing methods that merely augment category names with set of unstructured descriptions, our approach essentially distills structured knowledge graphs associated with class names from LLMs. Furthermore, introduces text and vision prompts designed to explicitly learn the corresponding visual attributes, effectively serving domain experts. Additionally, the general and diverse descriptions generated based on the class names may be wrong or absent in the specific given images. To address this misalignment, we further introduce Iision-conditional pooling module to extract instance-spect speciifc text features. Extensive experimental results demonstrate that our approach outperforms state-of-the- thefthhe-rtt art methods on the zero-sho hase-to-novel generalization, cross-datases transfer, as well as few-shot classification across 11 diverse datasets. Code is available at
approach
INTRODUCTION
Recent advancements in vasion-language models (VLMs) like CLIP (Radford et al. 2021) and ALIGN (Jia et al.) 2021) merge the capabilities of visual perception with linguistic understanding. which have revolutionized the landscape with their zero-shot learning abilities. They proficiently handle tasks on unseen data, bypassing the conventional requirement for task-specific training. This feature has enabled plethora of applications, ranging from content-base image retrieval to complex visual question answering, setting new benchmarks in the domain. crucial development in this domain is the concept of prompt learning. which has significantly influenced both natural language processing (NLP) (Lester et al. 2021 Li Liang 2021 Liu et al. (202T) and vision-only models (Jia et al. 2022 Wang et al. 2022a b] Zhang al. (022). This approach leverages learnable prompts to guide model understanding. tailoring responses to specific tasks or datasets.
Prompt learning, particularly in vision-language models, has garnered considerable interest due to its parameter efficiency and rapid convergence (Zhou et al. [2022b] a Zhu et al. 2023 Derakhshani et al. 2023, Lu al. 2022). Techniques like CoOp (Zhou al. 2022b) optimize learnable continuous prompts for few-shot image recognition, enhancing model performance significantly. Recent efforts have expanded multimodal prompt learning, optimizing prompts in both visual and language domains (Khattak et al. 2223a by Shi & Yang. 2023) Lee et al. 2023). Despite their success, these models rely on simplistic text prompts, typically formatted as "a photo of a [class]", J", illustrated in F (a). While functional, this approach lacks depth, failing to encapsulate the intricacies and finer details inherent in visual data. Such limitations hinder the model's ability fully leverage the rich, descriptive potential offered by more detailed and contextually relevant textual information.
In parallel, another stream of research has been exploring the utilization of large language models (LLMs) to generate more elaborate and descriptive text prompts for enhancing zero-shot learning
"Correpoondngg Author

---

Published as a conference paper at ICLR 2025
(+)
dumplinss
[b]
Ourpling
[c]
dumpinge
(d)
Example Tree of Attribute
Color
pole bo gu color from the dough oxtarior,
golden-bmmnn hase from panvtrying deep-
trying
Appooto of (dbss)
Srigi Tampler
O
Shape
round with pleated edge
* criscont shapped, with fold in the dough
Toxture
sootaand chow toxture fro the dough
crispy toxture the bortom from pai-tryyngg
erapped thin dough bolled, steomed, fried scrved with dipping sauce
Color
Shape
Texture
Presensation
Presentation
"served steamed in bamboo baskett
phroto a dumplngs
senved with dipping
Figure 1: Illustration of the methods for CLIP text prompts formation. (a) Manually created prompt with the single "a photo of a [class]" template; (b) A unstructured set of detailed descriptions generated by LLMs; (c) The proposed Tree of Attribute distills knowledge graph from LLMs, organizing the knowledge in "concept attribute descriptions" structure; (d) An example Tree of Attribute for class "dumplings", where each color represents a visual attribute.
capabilities (Menon & Vondrick 2023 Pratt et al.
2023 Roth et al. 2023 Li ei al. 2024 Kim et al.
2023] Parkhi al. 2012 Yan al. 2023 Yang al. 2023 Roy & Etemad 2024 Zheng et al. 2023
Tian et al. 2023). These L.M--genertte descriptions offer a wealth of detail and context, potentially
enriching the model's 's interpretative capabilities. However, current methodologies integrating these descriptions often do not exploit the full potential of this richness, As shown in Fig. (b), most of these approaches lack structured framework to organize and utilize these descriptions effectively, leading to a scattergun approach where not all generated descriptions are contextually relevant or optimally aligned with the visual content. In addition, as noted in (Roth et al., 2023), descriptions generated by such paradigms are usually diverse, which covers most possibilities of the class, but include descriptions that are either likely not co-occurring, c.g. "stcamed" and "fried", or absent in the input image, c.g. "long tail" for a cat shot from the front, necessitating the need for selective pooling mechanism for clearer image-text alignments.
In response to these challenges, our work introduces "Tree of Attribute Prompt learning (TAP)," a method that redefines the integration and utilization of detailed descriptions within VLMs. As indicated in Fig. (c), unlike existing methods that merely augment category names with set of unstructured descriptions, our approach essentially distills structured knowledge graphs associated with class names from L.MSs. Specifically, we adopt hierarchical, tree-like structure systemati- cally generate and integrate descriptions, ensuring a layered and comprehensive understanding of visual content. Each branch of this tree represents a specific attribute, with finer details fleshed out in the subsequent leaves, ensuring that every aspect of the visual content is captured and represented. Furthermore, we reimagine the learnable prompt tokens as "domain experts", each specializing in different aspects of the image, supplemented by the CLS token's 's global perspective. In addition, we introduce vision-conditional layers for each expert-attribute pair, which pool the most applicable descriptions from each of the attribute sets with condition on the input image content, ensuring optimal image-text aligmment. This setup not only provides a detailed, attribute-focusee analysis but also harmonizes these insights with the overall context.
Extensive experiments in base-to-novel generalization, cross-dataset transfer, and few-shot classi- fication across 11 diverse datasets demonstrate the effectiveness of our method. On base-to-nowe generalization, TAP achieves average performance gains of 1.07% in harmonic mean over the state- of-the-art methods, and 9.34% over the vanilla CLIP. On cross-dataset transfer. TAP outperforms existing methods on both source and target datasets by 1.03% and 0.75% in average. Competitive results are also observed in few-shot classification.
2 RELATED WORK
Prompt Learning for Vision-Language Models. Prompt learning bridges linguistic understanding
and visual perception, originating in NLP (Lester et al. 2021 Li & Liang 202 Liu et al.] 2021 and later adapted to vision-only (Jia et al.
2022; Wang et 2022a bi Zhang al.
2022 and
multimodal settings (Zhou et al., 2022bb a Khattak et al.) 2023a
bl Sh & Yang 2023
Lee et al.
2023. Tian al. 2023:
Rasheed et al. 2023
Roy & Etemad 2024
Zheng al. 2023
7hu et al.
2

---

Published as a conference paper at ICLR 2025
2023 Bulat & Tzimiropoulos, 2023 Lu et al.
2022) CoOp (Zhou et al.
2022b) introduced learnable
continuous prompts for few-shot image recognition but struggled to generalize to unseen classes,
highlighting the challenge of base-to-novel generalization (Zhou et al. 2022a
Guo et al. 2024b
Hernandez-Heennnddez et a 2024 Guo et al.) 2024a). CoCoOp (Zhou et al. 2022a) addressed
this by conditioning prompts visual features. Visual and multimodal prompt tuning methods, such as VPT (Bahng et al.] 20022) DPT (Xing et al. (023). and MaPLe (Khattak et al. (023a)).
optimize prompts pixel text space enhance feature alignment. Other works (Khattak et al. 2023b Bulat & Tzimiropoulos [023] et al. 2022 Roy & Etemad, 2024) focus on regularization to improve generalization. PromptSRC introduced self-regulating prompts to better retain base knowledge, while methods like PLOT (Chen et al.) (023) and ALIGN (Wang et al., (023) apply Optimal Transport to align prompts with local features. Our work differs by introducing a hierarchical "Tree of Attribute" framework to structure textual descriptions and guide specialized "domain expert" tokens for attribute-level understanding.
Enhancing model's understanding using visual attributes. There's growing emphasis on the use of detailed visual descriptions for various visual understanding tasks, including more fine- grained captioning (Hsieh et al.) (024), identifying subordinate- categories (Liu et al., 2024a), and language- guided visual classification (Menon & Vondrick] 2023). While manual creation is impractical given the large number of image classes, existing research relies either on data augmentation (Kim et al. 2024) or generation by LLMS such as GPT-3 (Brown et al. (020), which offers an efficient generation of broad spectrum of class-specific descriptions. These descriptions, like "fur pattern" or "tail shape" of a cat, provide fine- anee-grannd and distinctive characteristics. In an essence, such approaches can be viewed as knowledge distillation from LLMs trained on trained on vast and diverse textual corpora. However, existing studies often lack a structured methodology
for distillation (Kim et al.] 2023] Menon & Vondrick] [2033] Parkhi et al. [002)] Roth et al. 2023
Yan et al.)
2023 Yang et al.) 2023; Fabian et al. 2023 Pratt et al.
2023 Novack et al. 2023
Mao et al.
2023 Tian et al., 2023 Zheng et al.] 2023| Zhang et al.
2024 Liu et al. (024b) or
fail to effectively exploit the inherent hierarchy within the knowledge (Maniparambil et al.) [202
Wang et al.] 2024, Hsich et al.) 2024; Liu et al.] 2024a). Our approach (TAP) ) addresses these
limitations by introducing novel method to distill knowledge graph from LLMs in top-down manner, transitioning from class names (concepts) to visual attributes (e.g., color, shape) and further to detailed descriptions of each attribute, forming structured Tree of Attributes (ToAA) To fully leverage the ToA, we propose a bottom-up integration pipeline. We introduce vision-conditionall pooling (VCP) layers to aggregate descriptions into attribute-level features, effectively mitigating potential noise in the generated descriptions. The alignment between attributes and introduced visual expert tokens is then refined through this hierarchical structure. This integration enables the model to exploit structured relationships within the ToA, enhancing both the granularity and interpretability of vision-text alignment.
3 METHODOLOGY
3.1 PRELIMINARY
CLIP. Our approach is built on the pre-trained vision-language model, CLIP (Radford et al. 2021). Formally, let (z, c) denote the dataset, where x is an image and e E {l..... CJ are the class labels. For image æ, the vision encoder hr(-) transforms it into feature vector f (x)) Simultaneously, each class label e is mapped to a text prompt t. = a photo of a (c), and converted into textual feature vectors f (t())). The predicted class ş given by:
ý = argmax cos( cfr fa)
where cos(-) denotes cosine similarity.
Image classification with class descriptions. To improve the model's understanding of the categories
in the transfer datasets, previous works MMenn & Vondrick 2023 Roth et al. 2023) use more detailed
descriptions from Large Language Models (LLMs) instead of the simple "a photo of {c) " to prompt the CLIP text encoder. Under this approach, a convoluted set of descriptions is generated for class e as D. ("c, which ch is 1s/has / /etc description. " e.g. c="televa sion"
3

---

Published as a conference paper at ICLR 2025
Defiran
Eu potedaaterr
Taen raled sarters
Encoder
Toat Connest
Loursalle
Frezos
Vision Encoder
Expert
KTC
I.T5 LT%
kT=
LT. LT LT
CTC
LT
Tobet
Concot.
()....A)
1V
For All.
CE [L....C
Figure 2: Overview of the proposed TAP method. TAP uses bottom-up approach to aggregate the generated Tree Attribute. The vision-conditional pooling (VCP) layer aggregates descriptions into attribute-level features, which are aligned with visual expert tokens focusing on specific attributes (e.g., color, texture). These attribute-level features are then combined to make class predictions via a weighted of logits from each attribute, fully leveraging the hierarchical structure within the tree.
and description= blacck or grey". This classification is reformulated as
i = argmax cos(h1 (x))
dED
3.2 OVERALL FRAMEWORK
We rethink the descriptions by LLM iD nodes in knowledge graphs. While previous methods generate an unstructured set of descriptions, we distill structured knowledge graphs for each class c from LLM, in which the root node is the class name c, capturing the highest level semanties, and the leaf nodes are the detailed descriptions capturing fine-grained details. In this framework, previous paradigms only generate the leaf nodes of the graph, with the edges and graph structure missing, where the rich and inherent structure from the descriptions is overlooked. To address this limitation, we formulate our approach as Tree of Attribute, which follows the "concept- attribute description" structures, as illustrated in Fig. (c).
Besides weighting the deseriptions equally, previous works align descriptions that describe images from different aspects and at different granularities with a singular CLS token from the image encoder. However, while the use of a single CLS token is effective in certain contexts, we note that the CLS token is designed to capture the global information of an input image x (Dosovitskiy et al. 2021). As result, even though this helps to further inform global understanding. it may fail to effectively capture the nuances and variances the attribute level, which leads to suboptimal use of the rich descriptions. We address this by introducing a set of learnable prompt tokens that serve as domain experts in the vision branch, each of which aligns with specific attribute-level textual embedding.
Additionally, close inspection of the --e--rrerrtted descriptions indicates limited contextual rele- vance and high degree of diversity. Previous works (Roth al.. (202)) reflect the issue of descriptions that are likely not co-occurring e.g. "steam" and "fried". We further identify cases where the de- scriptions are technically correct but irrelevant certain images, such describing "long tail" in frontal images of cats, underscoring the need for a selective pooling mechanism. Thus, we introduce ssiion-conditionll pooling layer to extract mnstace-sspecffi text features for each attribute for selecting the most applicable descriptions.
Overall, TAP leverages the tree structure in two key ways: first, top-down process generates attributes and corresponding descriptions for each class in structured and contextually relevant manner. This ensures that the descriptions are structured and contextually relevant. Second, bottom- up process aggregates information from the leaf nodes (descriptions) into attribute-level features, which are aligned with visual expert tokens. These expert tokens focus on fine- ane-ggained visual attributes, such as color shape. Finally, the aggregated attribute-level features contribute to class predictions using a weighted sum of prediction logits, fully utilizing the hierarchical relationships within the tree. This dual approach allows TAP to capture both high-level structure and fine- fine-grained
4

---

Published as a conference paper at ICLR 2025
details, leading to enhanced alignment of visual and textual data and improved model performance and interpretability. Inspired by CoOP (Zhou et al.] 2022b), we also incorporate textual contextual tokens in the text encoder. The overall framework is presented in Fig.
3.3 TREE OF ATTRIBUTE GENERATION BY LLMS
We redefine the process of integrating LLM ---eerarteedd descriptions by introducing a knowledge graph G. = (Va Bej for each class c, where V. denotes the set of nodes, and IE denotes the edges that capture the semantic relationship between nodes. In previous works, Ve is the set of descriptions De. while E. is missing. We argue that such methods overlook the inherent structure among the descriptions and thus do not exploit the richness of these descriptions effectively. To better leverage knowledge from LLMs, introduce attribute layer link the root node class name, and the leaf node descriptions. The attribute nodes include visual attributes generated by LLMs, such as color and shape, for systematically guiding description generation as illustrated in Fig. (c). Each branch of this "tree" represents a specific attribute, with the subsequent "leaves" fleshing out the descriptions with finer details. In this framework, Ge includes the class name which the root node, the set of attributes such as color and shape being the intermediate layer, and lastly the set of descriptions under each attribute node, E. includes the edges that build up the hierarchy. This structure allows for nuanced representation of class information, spanning from general concepts down to specific attributes and detailed descriptions.
To this end, we introduce the Tree of Attribute (ToA), where we use a tree structure to model the relationship and structure the descriptions. Let A. denote the set of attributes, and for each attribute a. Agg denote its leaf nodes as DDg Each set D. contains descriptions that specifically pertain to attribute a for class c, which is denoted as
D = [d ,e
de
where d represents the i-th description for attribute of class e and n is the number of descriptions per attribute.
The process of generating a Tree of Attribute (ToA) unfolds in three steps: 1) Attribute Generation: We first query LLMs with the dataset information and ask generate set of attributes A which are considered relevant and characteristic of the dataset. 2) Example Generation: We then ask LLMs to generate deseriptions for randomly sampled class in the dataset, using the attributes A identified in the previous step. Each description takes the format of "class, which fis/has/etc) [description]". Description Generation for All Classes: Building upon the Q&A A template from the previous step, the LLM is then tasked with generating descriptions for all classes in the dataset.
Additionally, we incorporate "global context" attribute which is aligned with the CL.S token in the vision encoder. The descriptions are the 7 standard templates provided in (Radford et al. 2021).
3.4 LEARNING TAP WITH LEARNABLE EXPERT TOKENS
To fully exploit the structured Tree of Attribute, we introduce learnable visual expert tokens pK the vision branch to learn from each of the attribute nodes a E A. Unlike traditional methods that rely on a single CLS token for alignment, these expert tokens enable focused learning on specific image attributes, such as color or shape, enhancing the model's 's performance and interpretability.
We denote the set of introduced visual expert tokens as P {p" la E A}. Akin to the idea of visual prompt tuning (VPT) (Jia et al. 2022), we insert Po into the input sequence of the vision encoder, forming the prompted input sequences X, {eCLS. Pp, (.putch). where eCLS is the input CLS token, and Eponch denotes the embedded patch tokens. To further boost the model's capacity for nuanced attribute representation, we employ deep prompting by introducing a zero-initialized layer residual for each prompt token across transformer layers, which provides more explicit attribute guidance across transformer layers. In parallel, adopt set of m learnable context tokens Fs
(p j E {1, 2, ..., m}} for the text encoder shared across all descriptions, similar to (Zhou al. 2022b).
5

---

Published as a conference paper at ICLR 2025
3.5 VISIONCCONDITIONAL POOLING
To mitigate issues of misalignment and potential misleading information from the broad spectrum of LLM-generated descriptions, we proposed adaptive vision-conditional pooling layer, applicable each set of attribute descriptions De shared across all classes to dynamically pool the most applicable descriptions based on the visual content of the image æ using its corresponding visual expert token denoted as Pa,z- For case of expression, we will proceed without explicitly mentioning r, though it's important note that both the expert token and the resulting attribute-level embeddings are dependent on the visual information. Intuitively, VCP uses attention to calculate the similarity between Pa and all embedded descriptions in attribute Da. which are then used as weights for a weighted sum of the original description embeddings. Formally, for each attribute a and its associated expert token Pa the pooled attribute-level embedding oa for class e and attribute is:
Query Wg P
Key W₂ Emb(D..)
(4)
Attention Score softmax(Query Key²), V = Attention Score (m))
where W, and Wa are learnable weights E Rdddd, Emb(.) denotes the embedding function, and softmax(.) is the Softmax function. This layer mirrors cross-attention but omits Wa to maintain the output within the CLIP V-L space.
3.6 TRAINING AND INFERENCE
Training objective. During training, each visual expert token P is aligned with its associated attribute-level embedding ve trained with the following contrastive objective:
1 N
N log
exp(cos(p". e/T/T)
Loon (pa, 02))
(5)
exp(cos Ph vg) eT)
where represents the number of training samples, and T is the learned temprature of CLIP. The total classification loss Lelass is the average of the contrastive loss from each expert token as well as the CLS token, defined as:
Lelass
Σ Leon (pa vz))
(6)
A
Similar to (Khattak al.) 2023b) and (Bulat Tzimiropoulos, 2023), we regularize the vision CLS token, text feature, and the prediction logits from each attribute using the vanilla CLIP model. We
denote the regularization loss as Tregg where the details can be found Appendix. The overall training objective is Liota Lelass + Lreg.
Prediction fusion. During inference, we integrate the prediction by each attribute expert pair by weighted sum, formulated as follows:
i argmax cos( CLS + - a
U
Σ
cos(p", va.)
|A| -
dEAVICLS)
where a is hyperparameter that signifies the weight assigned to the global context provided by the CLS token, balancing its contribution with that of the attribute-specifie expert prompts.
4 EXPERIMENTS
We extensively evaluate our method in three settings: 1) Base-to-novel class generalization, where the datasets are equally split into base and novel classes. We train the model on the base classes only and evaluate both base and novel classes; 2) Cross-datased transfer, where we train ImageNet with 16 shots per class, and directly evaluate on other datasets in zero-shot; and 3) Few-shot classification with 16 shots per class.

---

Published as a conference paper at ICLR 2025
Table 1: Comparison with state-of-the-an methods in base-to-novel generalization. The model is trained on the base class, and evaluated on the unseen novel classes in zero-shot. TAP demonstrates strong generalization performance. HM: harmonic mean (Xian et al. 2017).
(a) Average
(b) ImageNet
(c) Caltech101
(d) OxfordPets
Base Novel|
HM
Base Novel|
HM
Base
Novel| HM
Base
Novel| HM
CLIP
69.34 74.22
71.70
CLIP
72.43 68.14
77.22
CLIP
96.84 94.00 40
CLIP
91.17 97.26 92.12
CoOp
82.69 63.22
71.66
CoOp
76,47 67.88 71.92
CoOp
98,00 89.81 91.73
CoOp
93.67 95.29 94.47
Co-CoOp
80.47 71.69
75.83
Co-CoDp
75,98 70.43 73.10
Co-CoOp
97.96 93.81 95.84
Co-CoOp
95.20 97.69 96.43
ProGrad
82.48 70.75
76.
ProGrad
77.02 66.66
71.46
ProGrad
98.02 93.89 95.91
ProGrad
95.07 97.63 96.33
RPO
81.13 75.00
77.78
RPO
76.60 71.57
74,00
RPO
97.97 94.37 96.03
RPO 94,63 97.50 96.05
LoGioroompt
84.47 74.24
19.03
LaGoPrompt
76.74
70.83
73.66
LaGaPrompt
98. 93.78 95.93
LaGoPrompe
96.07 96.31 96.
PrompiSRC
84.26 76.10 79.97
PrompiSRC
77.60
70.73
74.01
PrompiSRC
98.10 94,03 96.02
PrompiSRC
95.33 97.30 96.30
84.75 77.63 ($1.04
TAP
77.97
70.40
73.99
98.90 95.50 97.17
TAP
97.73 96.76
(c) StanfordCars
(f) Flowers
(g) Food101
(h) FGVCAircraft
Basc Novel| HM
Basc
Novel|
HM
Basc
Novel|
HM
Basc
Novel|
HM
CLIP
63.37 74.89 68.65
CLIP
72.08 77.80 74,83
CLIP
90,10 91.22 90.66
CLIP
27.19 36.29
31.09
CoOp
78.12 60.40
CoOp
97.60 59.67
74.06
CoOp
88.33 82.26
19
CoOp
40.44 22.30
28.75
Co-CoOp
70.49 73.59
72.01
Co-CoOp
94.87 71.75
81.71
Co-CoOp
90.70 91.29 90.99
Co-CoOp
33.41 23.71
27.74
ProGrad
77.68 68.63
72.88
ProGrad
71.87
82.03
ProGrad
89.59
ProGrad
32.82
RPO
73.87 75.53
74.69
RPO
94.13
76.67
84.50
RPO
90.33 90.83 90.58
RPO
37.33
34.20
35.70
LoGoPrompt
78.36 72.39
75.26
LaGoPrompt
99.05
76.52
36,34
LoGoPrompt
90.82 91.41 91.11
LoGoProomp
45.98
34.67
39.53
PrompiSRC
78.27 74.97
76.58
PrompiSRC
98.07
76.50
85,95
PrompSRC
90.67 91.53 91.
PrompiSRC
42.73
37.87
40.15
TAP
80.70 74.27
77.35
TAP
97.90
75.57
85.30
TAP
90.97 91,83 91.40
TAP
44.40 36.50
40.06
(i) SUN397
() DTD
(k) EuroSAT
()) UCF101
Base
Novel| HM
Base
Novel|
HM
Base
Novel| HM
Rase
Novel|
HM
CIIIP
69.36 75.35 72.23
CLIP
53.24
59.90 56.37
CLIIP
56.48 64.05 60.03
CLIP
70.53 77.50
73.85
CoOp
80,60 65.89
72.51
CoOp
79.44 41.18 54,24
CoOp
92.19 54.74 68.69
CoOp
84,69
56.05
67.46
Co-CoOp
79.74 76.86
78.27
Co-CoOp
77.01 56.00
64.85
Co-CoOp
87.49 60.04
71.21
Co-CoOp
82.33 73.45
77.64
ProGrad
81.26 74.17
77.55
ProGrad
77.35 52.15
62.45562.45
62.45
ProGrad
90.11 60.89
72.67
ProGrad
84.33
74.94
79.35
RPO
80.60 77,80
79.18
RPO
68.61
RPO
86.63
RPO
79.34
LaGooranpt
81.20 78.12
79.63
LaGoPrompt
82.87
60.14
69.70
LaGoPrompt
93.67 69.44
79.75
LaGoorompt
86.19
73.07
79.09
PrompisRC
82.67 78.47
80.52
PrompiSRC 83.37
62.97 71.75
PrompiSRC
92.90 73.90 82.32
PrompiSRC
87.10
78.80
82.74
TAP
82.87 79.53 81.17
TAP
84,20 68.00 75.24
TAP
90.70 82.17 86.22
TAP
87.90 82,43 85.08
Datasets and baslines. For all of the three settings, we follow previous works (Zhou et al. 2022b using 11 image recognition datasets. including: ImageNet (Deng et al.] 2009) and Caltech10) (Fei-Fei
al. 2004) for generic object recognition; OxfordPets (Parkhi et al. 2012). StanfordCars (Krause
et al. 2013). Flowers 102 (Nilsback & Zisserman 2008). Food 101 (Bossard et al.] 2014). and
FGVCAireraft (Maji al. (013) for fine -ineggrained classification: SUN397 (Xiao et al.] 20100 for scene recognition; UCF101 (Soomro et al., 2012) for action recognition; DTD (Cimpoi et al. 2014) for
texture classification; and TuroSAT((Heller et al., 2019) for satellite image analysis. We benchmark against several leading methods, including CLIP (Radford et al. 2022 CoOp (Zhou et al. 2022b).
Co-CoOP (Zhou et al., 2022a), ProGrad (Zhu et al., 2023), RPO (Lee et al. 2023 LoGoPrompt (Shi
& Yang 2023), and the titte--ff-hheeee art PromptSRC (Khattak et al. 20236
Implementation details. A pre-trained CLIP model with a ViT-B/16 vision backbone is used in all of our experiments and results are averaged over 3 runs. We use GPT-3.5-turbe (Ouyang et al. 2022) for attribute and description generation. We initialize the text context tokens with the word embedding of "a photo of a. During training, we iteratively train the vision and text encoders with 5 epochs for vision and 1 epoch for text schedule. We train a total of 60, 24, and 120 epochs for base-to-novel generalization, cross-dataset transfer, and few-shot classification respectively. We set = 0.4 for all datasets. We also use a Gaussian Prompt Weighting (GPA) following (Khattak et al. 2023b with a mean of 0.9N, std of 0.1N, where N represents the total number of epochs, for all tasks. Refer to the Appendix for additional implementation details.
4.1 HASE-TO-NOVER GENERAL.IZATE
In base-to-novel generalization, we equally split the classes into base and novel classes. Initial training and evaluations are conducted on the seen base classes, followed by evaluation on the unseen novel classes in a zero-shot manner. TAP surpasses prior state-of-the-art models in terms of the base and novel class accuracy, as well as their harmonic mean across most of the 11 datasets, with an average increase of 1.53% in the zero-shot novel class prediction, and 1.07% increase in the overall harmonic mean in average, as detailed in Table Notably, our method improves unseen class prediction without compromising base class performance, exhibiting an average performance boost
7

---

Published as a conference paper at ICLR 2025
Table Comparison with state-of-the methods in cross-dataset transfer evaluation. The model is trained on the source dataset and evaluated on the target datasets in zero-shot.
Source
Target
Caltech101
ImageNet
Flowers
Food101
Alreraft
SUN397
EuroAAT
UCF101
Average
Pets
Cars
DTD
CoOp
71.51
93.70
89.14
64.51
68.71
85.30
18.47
64.15
41.92
46.39
66.55
63.88
CoCoOp
71.02
94.43
90.14
65.32
71.88
86.06
22.94
67.36
45.73
45.37
68.21
65.74
PSRC
71.27
93.60
90.25
65.70
70.25
86.15
23.90
67.10
46.87
45.50
68.75
65.81
TAP
72,30
94.30
90.70
65.60
70.93
86.10
24.57
68.30
50.20
46.00
68.90
66.56
Table 3: Comparison with state-of-the-art methods in few shot classification results with 16 shots.
16-Shot Classification
Average
YougeNer
Caftech101
Flowers
Food101
Aircnftt
SUN397
EarSAr
UCF101
Pets
Cars
DTD
CLIP
78.79
67.31
95.43
85.34
80.44
97.37
82.90
45.36
73.28
69.96
87.21
82.11
CoOp
79.89
71.87
95.57
91.87
83.07
97.07
84.20
43.40
74.67
69.87
84.93
82.23
CoCoOp
74.90
70.83
95.16
93.34
71.57
87.84
87.25
31,21
72.15
63.04
73.32
78.14
PSRC
82.87
73.17
96.07
93.67
83.83
97.60
87.50
50.83
77.23
72.73
92.43
86.47
TAP
83.37
73.76
96.73
93.90
85.37
98.10
87.53
50.43
77.30
74.90
91.90
87.17
of 0.49%. In the challenging fine-grained tasks such DTD, EuroSAT, and UCF101, TAP achieves sigmificant improvements in novel class prediction by 5.03%, 8.27%, and 3.63% respectively. These results underscore the robust generalizability and efficacy of our method across diverse scenarios.
4.2 CROSS-DATASET TRANSFER
To further investigate the generalization capability of TAP, we train on ImageNet with 16 shots per class, and directly test the other 10 datasets under zero-shot without further tuning. As shown in Table 2 TAP demonstrates better generalizability on 8/10 target datasets compared to PromptSRC KKattaak al. 2023b), and achieves an average performance increase of 0.75%. Additionally, while the performance increase of previous methods on target datasets come with costs on the source dataset (-0.49% for CoCoOP and -0.24% for PromptSRC) as compared to CoOP (Zhou al. 2022b), 'AP also outperform previous methods on the source dataset with 1.03% increase compared to PromptSRC (0.79% incrase compared to CoOP), demonstrating TAP 's robustness in domain generalization without sacrifice source dataset performance.
4.3 FEW-SHOT CLASSIFICATION
In shot classification, TAP also outperforms existing methods in 9 of the 11 datasets. Detailed in T we achieve average accuracy of 83.37 across the 11 datasets, surpassing the previous state-of-the-art methods by 0.5%, further demonstrating the effectiveness of our method.
4.4 ABLATION STUDY
Effects of Tree of Attribute. A core inquiry is whether structuring descriptions into a Tree of Attribute (ToA) offers advantages over an unstructured aggregation of -enerreedd descriptions. To evaluate, we revert to aligning mixed, unstructured set of descriptions with the CLS token
common practice prior studies (Mao et al.) 2023 Maniparambil et al. 2023 Liu et al., 2024b) Wang et al. 2024 Tian et al. 2023 Zheng al. 2023), while keeping the same number of visual prompt tokens. According to Table 4 substituting the ToA with an unstructured set results in significant performance decreases of 1.86%, 2.31% %,, and 2.11% 1% across the average base, novel, and their harmonic mean performances, respectively. This stark contrast underscores the ToA's critical role in enhancing model efficacy.
8

---

Published as a conference paper at ICLR 2025
Image
Fur Pattern
Ear Pattern
Eve Pattern
Table 4: Effects of the Tree of At- tributes.
Image
Wheel Design
Grille Style
Headlight Shape
Des. Org.
Unstructured
Ours
Base
82.89
84.75 77.63
Novel HM
75.32
78.93
81.04
Image
Petal
Characteriticc
Table Effects of domain experts.
Align. Token
CLS
Ours
Base
83.89
84.75
Novel HM
76.85
77.63
80.22
81.04
Figure 3: Visualization of the class activation maps.
Table Effects of a
Table 7: Effects of the number of experts.
1.0
0.4
# Attrs.
2
Ours
Base
81.54
84.75
Base
83.20
83.97
84.10
84.41
84.45
84.62
84.66
84.74
84.75
Novel
73.85
77.63
Novel
74.90
76,20
76.35
77.06
77.13
77.17
77.35
76.67
77.63
HM
77.51
81.04
HM
78.83
79.90
80.04
80.57
80.63
80.72
80.84
80.50
81.04
Effects of Learning through Domain Experts. Further, we examine the impact of substituting the CLS token with visual expert tokens for learning fine- aene-grained attributes, commonly adopted in previous works (Mao et al. 2023 Lee et al. 2023 Tian et al. 2023 Zheng et al. 2023). Our findings (Table reveal improvements of 0.89%, 0.78%, and 0.82% in the average base, novel, and harmonic mean accuracies, respectively, upon integrating visual expert tokens. These results support the notion that comminn-specifee learnable tokens enhance the model's 's ability to grasp fine-graine details by focusing on distinct aspects of the image, as opposed to the CLS token's 's global focus.
Effects fusion coefficient a. a in Eq. 7 balance global and local information. We compare 1.0) for making the final prediction against
the performance of using CLS token only (i.e. our proposed prediction fusion with a = 0.4. As shown in Table 6 using CLS token decreases the performance significantly on both base and novel classes. This result further demonstrates the limitations of using a singular CLS token which focuses on global information, and supports the effectiveness of the use of expert tokens which focus local information.
Effects of Number of Attributes. In our framework, the selection of attributes is dynamically determined by L.Mss, leading variability across different datasets. This adaptability stands in contrast to a static approach where the number attributes uniformly set across all datasets. To understand the impact of this variability, we explore how altering the number of attributes from I influences model performance. Our findings, detailed in Table reveal a performance improvement trend as the number of attributes increases, with an optimal peak attributes before a slight decline at 8. However, crucially, across all fixed-attibuut scenarios, none matched the performance achieved through our method's dynamic attribute determination. These results underscore the importance of an adaptive approach to attribute selection, opposed to mee-size-fits-al strategy.
Design choice the vision-conditional pooling layer. Lastly, we ablate the design of the pooling layer, starting from the naive training-fres free average pooling, to the attention-based pooling mechanism with condition on the input image. Compared to average pooling, VCP demonstrates performance gain of 1.08% in the average harmonic mean. Furthermore, when compared with ittentionbbased max pooling, which selects a single description per attribute according to the attention score in Eq. ( VCP maintains superior advantage of 1.55% in average harmonic mean. These outcomes attest to the VCP layer's 's integral role in finetuning attribute relevance to the visual context, substantiating its design and implementation within model.

---

Published as a conference paper at ICLR 2025
Color pale begg color golden-brown hue from pan frying deep-frying
Texture
0.87
soft and chewy texture crispy texture on the bottom from pan-frying
0.19
Jous
Shape
Presentation
round with pleated edge sreepeett-hhppe, with a dough
served steamed bamboo basket
0.88
0.08
pan-fried to crispy finish and served with a dipping stuce
0.12
Figure 4: Visualization of the attention weights in the VCP layer for an example "dumplings" image.
Table 8: Ablation on design choice of the VCP layer. Our cross-attention based pooling mechanism demonstrates the best performance among other variants.
Pooling Method
Base Acc.
Novel Acc.
HM
Attn. Max Pooling
82.90
76.36
79.49
Average Pooling
83.18
76.98
79.96
VCP (Ours)
84.75
77.63
81.04
4.5 VISUALIZATION
Expert tokens focus attribute-related regions. We further investigate the effects of vision domain experts by visualizing their class activation maps from three illustrative examples using GradCAM (Selvaraju et al., (017). as shown inFig. These visualizations underscore the precision with which each expert token concentrates on the image regions pertinent to its designated attribute. Take the first cat image as an example. The "fur pattern" expert distinetly highlights the animal's 's fur texture, whereas the "ear" and "eye" experts focus precisely on the respective anatomical features. This pattern of attribute-specific attention is consistent across the evaluated examples, reinforcing conceptualization of expert tokens as dedicated "domain experts" within the visual field.
VCP layer pools the most applicable descriptions. The inherently interpretable nature the VCP layer, thanks its attention mechanism, allows for insightful visualizations of its operational process. Through the examination of attention weights assigned by the VCP layer to different attributes in given image, elucidate the layer's capability discern and prioritize the most applicable descriptions. As illustrated in Fig. with a "dumplings" image, the VCP layer adeptly allocates higher attention weights to descriptions accurately reflecting the observed instance (e.g., assigning weights of 0.92 to "round with a pleated edge" under the "Shape" attribute and 0.95 to "soft and chewy texture" under the Texture"). In contrast, less relevant descriptions for the specific image context (e.g., ccrescent-shaped for Shape and "crispy texture from pan-frying" for Texture) receive significantly lower weights. This discernment crucial, given the class dumplings" encompasses broad variety of appearances based on cooking methods, yet not all descriptions are fitting for every instance. These visualizations compellingly demonstrate the VCP layer's effectiveness in refining description relevance, thereby enhancing the model's 's interpretative alignment with the visual content.
CONCLUSION
This paper introduces Tree of Attribute Prompt learning (TAP), a novel method that integrates detailed, LLM-generated descriptions within VLMs, achieving state-of-the-art performance in base-to-novel generalization, cross-datase transfer, and few-shot image classification tasks across 11 diverse datasets. TAP leverages a hierarchical "Tree of Attribute" framework, distilling structured knowledge graphs from LLMs for nuanced representation of visual concepts, and employs learnable "domain expert" tokens and a vision-conditional pooling module for optimal image-text alignment. While promising, note that the reliance LLMs presents challenges in fine-grained datasets where similar classes require nuanced differentiation. in which cases LLMS generate identical descriptions for distinct classes, impacting novel class prediction performance. It highlights the current limitations of LLMs in discerning highly fine-gran fine-grained distinctions. Addressing this challenge through enhanced I.LM capabilities alternative strategies will be key focus of future research.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENT
This work was supported in part by Microsoft's Al for Good Research Lab, the Harvard Data Science Initiative, and NIH Grant R01HD104969.
REFERENCES
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modify- ing pixel space to adapt pre-trained models. arXiv preprint S2.iv:2200.1727722 3:1 1-11-12, 2022.
Lukas Bossard, Matthicu Guillaumin, and Lue Van Gool. Food-10 min-inni discriminative compo- nents with random forests. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VI 13, pp. 446-461. Springer, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 2020.
Adrian Bulat and Georgios Tzimiropoulos. Lasp:
Text-to-text optimization for language-aware soft prompting of vision & language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23232-23244 2023.
Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Prompt learning with optimal transport for vision-language models. In ICLR, 2023.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3606-3613, 2014.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255 Icee, 2009.
Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor G Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Bayesian prompt learning for image-language model generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15237-15244 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL fort//ooum?3 d- cbFdNTTy
Zalan Fabian, Zhongqi Miao, Chunyuan Li, Yuanhan Zhang, Ziwei Liu, Andrés Hernández, Andrés contes-Rooass Rafael Escucha, Laura Siabatto, Andrés Link, et al. Multimodal foundation models for zero-shot animal species recognition in camera trap images. arXiv preprint arXiv:2311.01066 2023.
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested 101 object categories. In 2004 conference on computer vision and pattern recognition workshop, pp. 778.178. IêEE, 2004.
Qianrong Guo, Saiveth Hernandez, and Pedro Ballester. Umap-clustering split for rigorous evaluation of ai models for virtual screening on cancer cell lines. 2024a.
Qianrong Guo, Saiveth Hernandez- Hernandez. and Pedro J Ballester. Scaffold splits overestimate virtual screening performance. In International Conference on Artificial Neural Networks, pp. 58-72. Springer, 2024b.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
11

---

Published as a conference paper at ICLR 2025
Saiveth Geemandez-Herannde Qianrong Guo, and Pedro Ballester. Conformal prediction of molecule-induced cancer cell growth inhibition challenged by strong distribution shifts. bioRxiv, pp. 2024-03, 2024.
Yu-Guan Hsieh, Cheng Yu Hsich, Shih- Ying Yeh, Louis Béthune, Hadi Pouransari, Pavan Ku- mar Anasosalu Vasu, Chun-Liang Li, Ranjay Krishna, Oncel Tuzel, and Marco Cuturi. Graph- based captioning: Enhancing visual descriptions by interconnecting region captions. arXiv preprint arXiv:2407.066233 2024.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pp. 4904-916. PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision, pp. 709-727. Springer, 2022.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Multi-modal prompt learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 19113-19122, 2023a.
Khan. Maple:
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsuan Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15 15190-15200, October 2023b.
Gahyeon Kim, Sohee Kim, and Seokju Lee. Aapl: Adding attributes to prompt learning for vision- language models. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1572-1582, 2024.
Jae Myung Kim, A Koepke, Cordelia Schmid, and Zeynep Akata. Exposing and mitigating spurious correlations for cross-modal retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2584-2594, 2023.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Dongjun Lee, Seokwon Song, Jihee Suh, Joonmyeong Choi, Sanghyeok Lee, and Hyunwoo J Kim. Read-only prompt optimization for vision-language few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1401-1411, 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021.
Wanhua Li, Xiaoke Huang, Zheng Zhu, Yansong Tang, Xiu Li, Jie Zhou, and Jiwen Lu. Ordinalclip: Learning rank prompts for language-guided ordinal regression. NeurlPS, 35:35313-35325, 2022.
Wanhua Li, Zibin Meng, Jiawei Zhou, Donglai Wei, Chuang Gan, and Hanspeter Pfister. Socialgpt: Prompting Ilms for social relation reasoning via greedy segment optimization. NeurIPS, 2024.
Xiang Lisa Li and Percy Liang. Prefix-tuning' Optimizing continuous prompts for generation, 2021.
Mingxuan Liu, Subhankar Roy, Wenjing Li, Zhun Zhong, Nicu Sebe, and Elisa Ricci. Democratizing fine- ome-grained visual recognition with large language models. In The Twelfth International Confer- on Learning Representations, 2024a. http:: htttps::/oppnreevew new. wwt.:eee/ffo...oom
c <DDND 1
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine- ane-tuning universally across scales and tasks. CoRR, 1b.02100.07602, 2021. URL htttts:///ee wrg. http://bs2211...000 07602
12

---

Published as a conference paper at ICLR 2025
Xin Liu, Jiamin Wu, Wenfei Yang. Xu Zhou, and Tianzhu Zhang. for vision- -siounlanguage models. IEEE Transactions on Circuits and Systems for Video Technology. 2024b.
attribute prompting
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5206-5215, 2022.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. arXiv preprint arXiv:1306.511 2013.
Mayug Maniparambil, Chris Vorster, Derek Molloy, Noel Murphy, Kevin McGuinness, and Noel E O' O'Connor. Enhancing clip with gpt-4: Harnessing visual descriptions as prompts. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 262-271. 2023.
Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang. Xin Wang, and Carl Vondrick. Doubly right object recognition: A why prompt for visual rationales. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2722-2732, 2023.
Sachit Menon and Carl Vondrick. Visual classification via description from large language models. ICLR, 2023.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian conference on computer vision. graphics & image processing, pp. 722-729. IEEE, 2008.
Zachary Novack, Julian McAuley, Zachary Lipton, and Saurabh Garg. Chils: Zero-shot image classification with hicrarchical label sets. In International Conference on Machine Learning (ICML), 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3498-3505. IEFE, 2012.
A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic diffcrentiation PyTorch. In NeurIPS Autodiff Workshop, 2017.
Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does platypus look like? gener- ating customized prompts for zero-shot image classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15691-15701, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Fine-tuned clip models are efficient video learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Karsten Roth, Jae Myung Kim, A. Sophia Koepke, Oriol Vinyals, Cordelia Schmid, and Zeynep Akata. Waffling around for performance: Visual classification with random words and broad concepts. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15746-15757, October 2023.
Shuvendu Roy and Ali Etemad. Consistency-guide prompt learning for lanion-lagguage models. In The Twelfth International Conference on Learning Representations, 2024. URL https: //opereevicec net./ forum? d=wsRXw wx4w
13

---

Published as a conference paper at ICLR 2025
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 618-626, 2017.
Cheng Shi and Sibei Yang. Logoprompt: Synthetic text images can be good visual prompts for vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2932-2941, 2023.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint 12Xiv:1212.04022 2012.
Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. Argue: Attribute-gidde prompt tuning for vision-language models. arXiv preprint arXiv:2311.164p4 2023.
Dongsheng Wang, Miaoge Li, Xinyang Liu, MingSheng Xu, Bo Chen, and Hanwang Zhang. Tuning multi-mode token-level prompt alignment across modalities. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL http::tttts::peeeeeeeee
id=
Yubin Wang, Xinyang Jiang. De Cheng, Dongsheng Li, and Cairong Zhao. Learning hierarchical prompt with structured linguistic knowledge for vision-language models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp.
2024.
Zifeng Wang, Zizhao Zhang, Sayna Ebrahimi, Ruoxi Sun, Han Zhang, Chen-Y, Yu Lee, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, et al. Dualprompt: Complementary prompting for rehearsal-free continual learning. In European Conference on Computer Vision, pp. 631-648. Springer, 2022a.
Zifeng Wang, Zizhao Zhang, Chen- Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Plister. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 139 149, 2022b.
Yongqin Xian, Bernt Schiele, and Zeynep Akata. Zero-shot learning-the good, the bad and the ugly. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 4582-4591, 2017.
Jianxiong Xiao, James Hays, Krista Ehinger, Aude Oliva, and Antonio Torralba. Sun database: scage scene recognition from abbey to 700. In 2010 IEFE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, Peng Wang, and Yanning Zhang. Dual modality prompt tuning for vision-language pre-trained model. IEEE Transactions on Multimedia, pp. 1-13, 2023. doi: 10.
An Yan, Yu Wang, Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 3090-3100, 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10677 2024.
Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19187-19197, 2023.
Yi Zhang, Ce Zhang, Ke Yu, Yushun Tang, and Zhihai He. Concept-guided prompt learning for generalization in lision-language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38777777786 Mar. 2024. doi: 10. www.:669/aai..3330 aaai. wwt. wwtp://ww.ppppppppppmmpp/ppp AAAPP/AAA///// ar cl e/ 2/55/
URL https://psss
14

---

Published as a conference paper at ICLR 2025
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv:2206.04671 2022.
arXiv preprint
Zhaoheng Zheng, Jingmin Wei, Xuefeng Hu, Haidong Zhu, and Ram Nevatia. Large language models are good prompt learners for low-shot image classification. arXiv preprint arXiv:2312.04076 2023.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for Dision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 168166 16816-16825, 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-2344 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligned gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15659-15669, 2023.
15

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 MODEL REGULARIZATION
Denote the frozen image feature from CLIP vision encoder f", the frozen text feature for description d from CLIP text encoder as f. and the zero-shot logit prediction from CLIP as g. Additionally,
denote the trained image feature as f", the trained text feature for description d as fy. and the logit prediction from attribute a after training as Ua- The losses are as follows:
L1
f f2112
(8)
exp(cos(f;; (f). (())
+ 1 log exp(cos(ff, E2)) 2
Loon-
log
(9)
ΣNED, exp(cos((f) (f:,, f2))
EXED. exp(cos P.))
dED
LKL-attr
Σ
Drc(0. Ba))
(10)
[A|
aEA
The regularization loss is then:
Lreg -I + p2LKLL + Ma poLion-T
(11)
Our overall training objective is thus given by: Letasss
Luttl
Lreg
(12)
To investigate the effectiveness of model regularization, we compare TAP against existing methods with and without regularization. As evidenced in Table the proposed model regularization helps in both base and novel performance, with an increase of 1.62% in average harmonic mean. Comparing to existing methods, TAP is consistently better than other baselines in both settings, demonstrating the robustness of our method.
Table 9: Effectiveness of model regularization. TAP achieves favorable results under both settings.
Regularization
Basc
Novel
HM
PSRC-reg
X
84.21
71.79
77.51
MaPLe
X
82.28
75.14
78.55
TAP-reg
X
83.37
75.82
79.42
PSRC
V
84.26
76. 10
79.97
TAP
84.75
77.63
81.04
A.2 ADDITIONAL IMPLEMENTATION DETAILS
Because the number of attributes vary across the 11 datasets which results different number of learnable parameters, we group the datasets into two and apply two sets of learning rates to balance generalizability and performance. For DTD, Oxford Flowers, Stanford Cars, UCF101, and Caltech101 datasets, which have fewer attributes, we use low learning rate of 0.002 for the text encoder to avoid overfitting and high learning rate of 0.006 for the vision encoder facilitate the learning process. A high A3 3 is also used to regularize the text encoder for preventing overfitting. For the remaining 6 datasets, which have more attributes, the learning rates for both text and vision encoders are set as 0.004, with μą 1.5. μ1 10, and μ22 2.5 are used for all datasets.
For base-to-novel generalization and few-shot classification evaluations, we use an adaptive approach for generating the attributes, in which the attributes vary across datasets. Although it turns out to be better than using a fixed set of attributes as shown in Table 7 it is not applicable to the cross- dataset transfer experiment as both VCP layers and visual expert tokens attributes. Therefore, for cross cross-dttseet transfer, we use the following fixed set of attributes that are applicable to all 11 datasets: Pattern, Texture, Shape, and Context.
specific to their corresponding
16

---

Published as a conference paper at ICLR 2025
We PyTorchh Paszke al. 2017) to implement all experiments single NVIDIA A100-80GB GPU. Our code is developed based on the implementation of CoOp Zhou et al. (2022b), which is available at http://gttubb.om/Kaaaaaagz////// also released under the MIT' license. Baseline results for the three tasks are taken from their respective publications. For the "global context" attribute which is aligned with the CLS token in the vision encoder, we use the following selected templates provided in Radford et al. (2021)
and released under the MIT license. Our code is
"itap of (class)."
"a bad photo of the {class)." "a origami (class)."
"a photo of the large (class)."
"a (class) in video game."
"art of the (class)."
"a photo of the small (class)."
A.3 ROBUSTNESS LLMS
To investigate the robustness of our methods against different LLMss, we additionally generate the descriptions using locally-served small L.M Qwwe-22-B-IInstruc (Yang et al. 2024). in which the results are comparable.
Table 10: Robustness against different LLMs.
LLMs
Base Acc.
Novel Acc.
HM
Qwen-2-7B-Instree GPT-3.5-Turbo
84.68
77.31 77.63
80.83
84.75
81.04
A.4 PROMPTS FOR FREE-OF-ATTRIEN GENERATION
As introduced in Section 3.3, generate the Tree-of-Atrrbutte with the following three steps: 1) Attribute Generation, 2) In-Context Example Generation, and 3) Description Generation for All Classes. The prompts for each step follows:
1) Attribute Generation:
(Dataset Description.)
Visual attributes refer to observable, describable features of the images that can include color, shape, size, texture. and any specific patterns markings, which can help differentiate between classes for the dataset. They should be consistently observable across multiple images of the same class. Your task is generate list of visual attributes (less than 10) for the (Dataset Name] dataset. Ensure this list is clear, concise, and specific to the dataset's 's needs. Avoid generic attributes that do not contribute distinguishing between classes.
2) In-Context Example Generation
Describe describe what "(Random Class Namel" class in the (Dataset Name) dataset look like using the generated visual altributes.
You must follow the following rules:
For each visual attribute, describe all possible variations as separate sentences. This approach allows for a detailed and clear presentation of each attribute's 's range.
Provide maximum of five descriptions for each visual attribute to maintain focus and relevance. Also, aim provide least two descriptions ensure comprehensive overview of the attribute.
17

---

Published as a conference paper at ICLR 2025
3. The descriptions should provide clear, distinguishable features of each class support image classification tasks.
4. Descriptions for each attribute are independent from each other, and they should not serve context for each other.
5. Each description describes image independetly. If certain description is possible for class, please just list that description, and do not use words like "may have" or "sometimes have".
6. Reply descriptions only. Do not include any explanation before and after the description.
The descriptions should follow the format of "classname, which ...", where "... the description of the visual attribute.
3) Description Generation for All Classes
(Dataset Description.)
Your task is to write detailed descriptions for various classes within the (Dataset Name) dataset, using the provided visual attributes such as color and shape. These descriptions will help in accurately classifying and understanding the unique features of each class.
You must follow the following rules:
For each visual attribute, describe all possible variations as separate sentences. This approach allows for a detailed and clear presentation of each attribute's 's range.
Provide a maximum of five descriptions for each visual attribute to maintain focus and relevance. Also, aim provide at least descriptions ensure comprehensive overview of the attribute.
The descriptions should provide clear, distinguishable features of each class to support image classification tasks.
Descriptions for each attribute are independent from each other, and they should not serve as context for each other.
Each description describes an image independetly. If certain description is possible for a class, please just list that description, and do not words like "may have" "sometimes have".
Reply descriptions only. Do not include any explanation before and after the description.
7. The descriptions should follow the format of "classname, which ...", where "..." is the description of the visual attribute.
Q: Describe what "[Random Class Name]" " in the (Dataset Name] look like using the following visual attributes: (Visual Attributes from Step 1.]
A: [Answer from Step 2.J
Q: Describe what a "(Target Class Name]" " in the (Dataset Name] look like using the following visual attributes: (Visual Attributes from Step 1.)
A:
In the prompt templates, "Dataset Description" " is the description of the dataset from their official website, "Random Class Name" is a randomly sampled class name in the dataset for in-context example generation, and "Target Class Name " is the class name of interest for the current query. While step and are made in two consecutive calls to provide contexts which are queried once per dataset, step is queried independently for each of the remaining classes in the dataset. Our carefully designed prompts for step and 2 guide the LLM in generating high-quality examples. Human review further confirms that the generated in-context examples from these prompts are of high quality even without any manual intervention.
A.5 ATTRIBUTE SETS
The attribute sets generated by LLMs are shown Table II
18

---

Published as a conference paper at ICLR 2025
Table 11: Attribute sets generated by LLMs for the 11 datasets.
Dataset
Attributes
Orientation
Shape
Pattern
ImageNet
Texture
Pose
Context
Dominant Feature
Shape
Caltech101
Texture
Color
Size
Body Type
Wheel Design
Grille Style
StanfordCars
Headlight Shape
Rear Taillight Design
Roof Style
Color
Petal
Flowers102
Center structure
Stem characteristics
Color
Shape
Food101
Texture
Ingredients
Presentation Style
19

---

Published as a conference paper at ICLR 2025
Table 12: Attribute generated by LLMs for the 11 datasets. Cont.
Dataset
Attributes
Wing Configuration
Winglet Presence
Engine Configuration
FGVCAircraft
Number of Engines
Fuselage Length
Fuselage shape
Wingspan
Indoor/Outdoor
Color
Dominant elements
SUN397
Environment
Architectural style
Patterns
Texture
Pattern
DTD
Repetition
Contrast
Contrast
Texture
Orientation
EuroSAT
Edge
Size
Color
Symmetry
Action Pose
Number of People
UCF101
Background Setting
Objects Present
20


Reviews:
[{"review_id": "iCgvZjmX1A", "reviewer": "Reviewer_1Yxk", "summary": "- The paper proposes a new prompt learning approach for vision language models. \n- Instead of a fixed prompt or LLM-generated unstructured prompt, the paper proposes to use a tree of attributes prompt.\n- This organizes the generated description as concept, attribute and description structure essentially converting unstructured information from LLMs to structured.\n - A visual conditioning approach is also proposed to ensure optimal image-text alignment and avoid multiple possibilities of the class. \n- The approach is shown to outperform prior work on multiple tasks and datasets.", "questions": "Please refer to the weaknesses section. \n\nMinor:\n- Some examples for elements of the set in L245 would be helpful", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of the paper is strong: a tree of attributes seems logical way to represent information about a class. Pooling module to avoid misalignment is also an interesting idea. \n- The paper is fairly well written and overall idea is easy to follow.\n- The experiments and ablations are helpful to understand the effect of the proposed approach.", "weaknesses": "Concerns & questions: \n- Statistical significance: Standard deviations at the least would be helpful to understand if improvements are significant. This is especially important since differences in many results seem minor. \n- How sensitive is the approach to choice of LLM ? \n- Table 4: Authors mention that prior works use unstructured set of descriptions, do the drops from using unstructured descriptions in Table 4 match performance of the cited works L428-429? \n- Authors should make it clear if \"example generation\" strategy was used by the prior works to create descriptions.  Further, the choice of CLIP model and if it matches prior work should also be described to understand the differences in various approaches. \n- Missing references and comparisons: [a], [b]. It would be a good idea to compare and contrast with these. \n\n[a] Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions\n[b] Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "US0x2CYCW9", "reviewer": "Reviewer_L4G9", "summary": "This paper proposes the Tree of Attributes Prompt Learning (TAP) framework for enhancing vision-language models (VLMs), especially in zero-shot, cross-dataset, and few-shot tasks. TAP introduces a hierarchical \"concept - attribute - description\" structure that utilizes large language models (LLMs) to generate a structured knowledge graph for each category, capturing the contextual richness of class names through attributes. This approach also includes vision-conditional pooling to extract instance-specific text features, mitigating misalignment issues between visual and textual data. Experimental results show improved performance over baseline models, highlighting TAP's impact across diverse datasets.", "questions": "1. In Section 3.4, the approach combines VPT and CoOP to form an alignment mechanism for visual cues and attribute cues. Did the authors introduce new mechanisms in the combination process or innovate in the way the alignment prompts are generated?\n\n2. In Section 3.5, the authors mentioned that adaptive vision-conditional pooling layer is used to alleviate the misalignment problem of visual and text features, but did not clearly explain the actual role of the pooling layer. How is the pooling layer implemented in VCP? Does it involve specific pooling operations or simple weighted aggregation?\n\n3. In the ablation experiment, the authors claimed that the number of attributes generated dynamically improved performance over the fixed number of attributes, but when compared with other methods, the other methods all used a fixed number of attributes. This gives uneven advantages to the proposed method over baselines. How do authors justify this? Have the authors considered adding control variable experiments to verify the specific impact of dynamic generation on performance improvement?\n\n4.TAP’s dependency on a particular LLM for dynamic attribute generation raises questions about adaptability across different LLMs. I suggest the authors evaluate TAP using alternative LLMs (e.g., GPT-4, PaLM, or Claude) to assess robustness. It would be informative to report on consistency in performance across these models and discuss which metrics best capture the variability introduced by different LLM choices, offering insights into TAP’s adaptability and contribution.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "TAP's hierarchical attribute organization is indeed a new attempt to structure the description into a well-defined knowledge graph framework, different from previous unstructured approaches. The method achieves state-of-the-art performance across multiple challenging scenarios on 11 diverse datasets, and innovatively incorporates domain expert tokens and vision-conditional pooling for optimizing image-text alignment.", "weaknesses": "1.The TAP method does present enhancements over VPT and CoOP by organizing attributes hierarchically and introducing a vision-conditional pooling layer. However, to improve TAP’s impact, I suggest that the authors emphasize distinctive innovations beyond architectural refinements. For instance, exploring a novel alignment mechanism that fully utilizes semantic edge information within the constructed attribute trees might further differentiate TAP from its predecessors. Also, they could investigate additional layers of vision-text alignment that capture attribute-specific nuances more dynamically than existing models.\n\n2. Dynamic attribute generation relies on a large language model, and compared with other methods that use a fixed number of attributes, this flexibility is not an innovation in the method itself, but more like an extension of existing technologies.\n\n3. The edges in TAP’s attribute tree offer promising opportunities for capturing semantic relations. However, the paper lacks a detailed explanation of these edges. Clarifying whether they represent specific semantic relationships (e.g., similarities or hierarchies among attributes) could enhance interpretability. I recommend that the authors either expand on how these edges improve semantic alignment or describe potential modifications to strengthen the tree's expressiveness.\n\n4. Section 3.5 briefly mentions the vision-conditional pooling (VCP) layer but does not detail its operation, especially regarding \"pooled attribute-level embedding.\" Providing a step-by-step description of how VCP pools attributes based on visual inputs and clarifying the term could be helpful. Additionally, illustrating VCP's functionality through specific examples might aid readers in understanding its role in enhancing text-image alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AfOFY5UC4r", "reviewer": "Reviewer_iRDM", "summary": "The TAP method structures textual descriptions in a hierarchical “concept-attribute-description” format, effectively creating a knowledge graph from large language models (LLMs) for each category name. This structure allows for a more comprehensive and detailed understanding of the visual content. The paper reimagines learnable prompt tokens as \"domain experts,\" each specializing in different aspects of the image, supplemented by a global perspective provided by the CLS token. To address potential misalignment between general descriptions and specific image content, the paper introduces a vision-conditional pooling module. This module extracts instance-specific text features, ensuring optimal image-text alignment.", "questions": "In Section 3.3, for attribute generation, what type of dataset information is given to the large model?\n\nIn Figure 4, each image is accompanied by only two descriptions. Are all images described using two sentences each?\n\nIn this paper, it mentions that the method can capture subtle differences between attributes. Could you provide a relevant example?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method incorporates structured tree of attribute into prompt tuning that provide richer supervisory information compared to unstructured attribute information. A set of experiments has been conducted, and the results look promising.", "weaknesses": "One major limitation of the method is that it requires human review to \"ensure the quality of the example\" (L175). Recall that one major advantage of prompt tuning is that it can adapt large models quickly to specific tasks. However, the requirement of human reviewing in the proposed method is not consistent with this goal. In addition, it is not clear how many human efforts are needed here, and how to handle the potential human bias in quality evaluation.\n\nThe ToA is built in a dataset-aware manner using specific prior categorical information. This raises an issue regarding the domain generalization performance of the model, which is not studied.\n\nFor training details, different learning rates were used for different datasets, however, existing methods typically use a same LR for all datasets. From this point, the comparison is somewhat unfair.\n\nFigure 2 lacks sufficient clarity, for example, input & output streams are not clear. It is a necessity to re-draw it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tk3ibIgdOw", "reviewer": "Reviewer_yqpE", "summary": "The paper introduces a TAP, a method that first prompts LLM to generate class descriptions in a structured way and then employs both vision and text prompt learning to learn fine-grained attributes within vision-language models.\nTAP leverages a hierarchical framework to distill structured knowledge graphs from LLM and employs learnable \"domain expert\" tokens instead of using a single CLS token to learn class attributes from the description. Additionally, it uses a vision-conditional pooling module to optimize image-text alignment.\n\nThe key contribution of this work is: 1) A structured framework for integrating LLM-generated descriptions, 2) learnable vision tokens and a vision-conditional pooling layer for selective description aggregation", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n1. This paper proposes a structured way to prompt the LLM for detailed class descriptions.\n2. This paper proposes a framework leveraging LLM-generated class descriptions to improve the vision language models, and achieves SOTA performance on 11 benchmarks.\n3. TAP allows the model to capture both high-level information and fine-grained details from the images, which enhances the model’s performance and interpretability.", "weaknesses": "Weaknesses:\nNo ablation experiment on how using learnable context tokens impacts the performance.\n2.  From Table 1,  most of the improvement comes from the base classes and there is still room for improvement on the novel class.\n3. Captions for the tables and figures could be more detailed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jsauZ9rV6g", "reviewer": "Reviewer_sQkS", "summary": "This paper proposes a novel visual prompt learning approach, which reimagines the visual prompt tokens as “domain experts”, each specializing in different aspects of the image. By leveraging these domain experts, the proposed approach can explicitly learn the corresponding visual attributes. To optimize these prompts, the authors further propose ToA to collect category descriptions based on a cross-category-shared attribute set. In addition, the authors also introduced vision-conditional layers to avoid misaligning images with attributes that do not exist in the corresponding images. The experimental results verified the effectiveness of this approach.", "questions": "It would be better if the authors could show the entire attribute set in the appendix.\n\nWhat is the meaning of ‘contextually relevant’? It is hard for me to clearly understand the meaning of 'context' in this paper.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The authors proposed a novel visual prompt learning approach, which can explicitly extract visual features on corresponding visual attributes.\n\nThe authors proposed the vision-conditional layer to avoid misaligning images with attributes that do not exist in the corresponding images.\n\nThe above two contributions have clear motivations and technically seem to work.\n\nThe authors conducted extensive experiments to verify the effectiveness of the proposed approach.", "weaknesses": "The authors claim that TAM is the first work to instruct LLMs to generate a tree of attributes with a “concept - attribute - description” structure for each category. However, existing works “Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts (ICCV2023W )” and “Democratizing Fine-grained Visual Recognition with Large Language Models (ICLR24)” already proposed the similar corpora and prompt engineering strategies. The authors should discuss these related works in this paper.\n\nIn the introduction section, the authors mentioned that some attributes may absent in the input image which necessitate the need for a selective pooling mechanism. However, the proposed vision-conditional layer still pool the most applicable descriptions for each attribute but doesn’t filter out attributes that do not exist in the image.\n\nIt seems that this paper was prepared in a hurry. There are some obvious typos, e.g., A\\{CLS} in Eq.(7) and Fig. Fig. 1 (b).\n\nIn general, in my opinion, this paper doesn't have any major issues; the limitations mentioned above are minor and seem easy to correct in the final version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)