Tree of Attributes Prompt Learning for Vision-Language Models

## Abstract

Abstract Prompt learning has proven effective in adapting vision language models for downstream tasks. However, existing methods usually append learnable prompt tokens solely with the category names to obtain textual features, which fails to fully leverage the rich context indicated in the category name. To address this issue, we propose the Tree of Attributes Prompt learning (TAP), which first instructs LLMs to generate a tree of attributes with a “concept - attribute - description” structure for each category, and then learn the hierarchy with vision and text prompt tokens. Unlike existing methods that merely augment category names with a set of unstructured descriptions, our approach essentially distills structured knowledge graphs associated with class names from LLMs. Furthermore, our approach introduces text and vision prompts designed to explicitly learn the corresponding visual attributes, effectively serving as domain experts. Additionally, the general and diverse descriptions generated based on the class names may be wrong or absent in the specific given images. To address this misalignment, we further introduce a vision-conditional pooling module to extract instance-specific text features. Extensive experimental results demonstrate that our approach outperforms state-of-the-art methods on the zero-shot base-to-novel generalization, cross-dataset transfer, as well as few-shot classification across 11 diverse datasets.

## 1 Introduction

Recent advancements in vision-language models (VLMs) like CLIP (Radford et al., ) and ALIGN (Jia et al., ) merge the capabilities of visual perception with linguistic understanding, which have revolutionized the landscape with their zero-shot learning abilities. They proficiently handle tasks on unseen data, bypassing the conventional requirement for task-specific training. This feature has enabled a plethora of applications, ranging from content-based image retrieval to complex visual question answering, setting new benchmarks in the domain. A crucial development in this domain is the concept of prompt learning, which has significantly influenced both natural language processing (NLP) (Lester et al., ; Li & Liang, ; Liu et al., ) and vision-only models (Jia et al., ; Wang et al., ; ; Zhang et al., ). This approach leverages learnable prompts to guide model understanding, tailoring responses to specific tasks or datasets.

Prompt learning, particularly in vision-language models, has garnered considerable interest due to its parameter efficiency and rapid convergence (Zhou et al., ; ; Zhu et al., ; Derakhshani et al., ; Lu et al., ). Techniques like CoOp (Zhou et al., ) optimize learnable continuous prompts for few-shot image recognition, enhancing model performance significantly. Recent efforts have expanded to multimodal prompt learning, optimizing prompts in both visual and language domains (Khattak et al., ; ; Shi & Yang, ; Lee et al., ). Despite their success, these models rely on simplistic text prompts, typically formatted as “a photo of a {class}”, illustrated in Fig. 1 (a). While functional, this approach lacks depth, failing to encapsulate the intricacies and finer details inherent in visual data. Such limitations hinder the model’s ability to fully leverage the rich, descriptive potential offered by more detailed and contextually relevant textual information.

In parallel, another stream of research has been exploring the utilization of large language models (LLMs) to generate more elaborate and descriptive text prompts for enhancing zero-shot learning capabilities (Menon & Vondrick, ; Pratt et al., ; Roth et al., ; Kim et al., ; Parkhi et al., ; Yan et al., ; Yang et al., ; Roy & Etemad, ; Zheng et al., ; Tian et al., ). These LLM-generated descriptions offer a wealth of detail and context, potentially enriching the model’s interpretative capabilities. However, current methodologies in integrating these descriptions often do not exploit the full potential of this richness. As shown in Fig. Fig. 1 (b), most of these approaches lack a structured framework to organize and utilize these descriptions effectively, leading to a scattergun approach where not all generated descriptions are contextually relevant or optimally aligned with the visual content. In addition, as noted in (Roth et al., ), descriptions generated by such paradigms are usually diverse, which covers most possibilities of the class, but include descriptions that are either likely not co-occurring, e.g. “steamed” and “fried”, or absent in the input image, e.g. “long tail” for a cat shot from the front, necessitating the need for a selective pooling mechanism for clearer image-text alignments.

Figure: Figure 1: Illustration of the methods for CLIP text prompts formation. (a) Manually created prompt with the single “a photo of a {class}” template; (b) A unstructured set of detailed descriptions generated by LLMs; (c) The proposed Tree of Attribute that organizes the descriptions in a “concept - attribute - descriptions” structure, essentially distilling knowledge graphs from LLMs; (d) An example Tree of Attribute for “dumplings”.
Refer to caption: x1.png

In response to these challenges, our work introduces “Tree of Attribute Prompt learning (TAP),” a method that redefines the integration and utilization of detailed descriptions within VLMs. As indicated in Fig. 1 (c), unlike existing methods that merely augment category names with a set of unstructured descriptions, our approach essentially distills structured knowledge graphs associated with class names from LLMs. Specifically, we adopt a hierarchical, tree-like structure to systematically generate and integrate descriptions, ensuring a layered and comprehensive understanding of visual content. Each branch of this tree represents a specific attribute, with finer details fleshed out in the subsequent leaves, ensuring that every aspect of the visual content is captured and represented. Furthermore, we reimagine the learnable prompt tokens as “domain experts”, each specializing in different aspects of the image, supplemented by the CLS token’s global perspective. In addition, we introduce vision-conditional layers for each expert-attribute pair, which pool the most applicable descriptions from each of the attribute sets with condition on the input image content, ensuring optimal image-text alignment. This setup not only provides a detailed, attribute-focused analysis but also harmonizes these insights with the overall context.

Extensive experiments in base-to-novel generalization, cross-dataset transfer, and few-shot classification across 11 diverse datasets demonstrate the effectiveness of our method. On base-to-novel generalization, TAP achieves average performance gains of $1.07\%$ in harmonic mean over the state-of-the-art methods, and $9.34\%$ over the vanilla CLIP. On cross-dataset transfer, TAP outperforms existing methods on both source and target datasets by $1.03\%$ and $0.75\%$ in average. Competitive results are also observed in few-shot classification.

## 2 Related Work

Prompt Learning for Vision-Language Models. Prompt learning bridges linguistic understanding and visual perception by guiding VLMs with text prompts, a concept originated in NLP (Lester et al., ; Li & Liang, ; Liu et al., ) and adapted to vision-only (Jia et al., ; Wang et al., ; ; Zhang et al., ) and multimodal contexts(Zhou et al., ; ; Khattak et al., ; ; Shi & Yang, ; Lee et al., ; Tian et al., ; Rasheed et al., ; Roy & Etemad, ; Zheng et al., ; Zhu et al., ; Bulat & Tzimiropoulos, ; Lu et al., ). In the textual domain, CoOp (Zhou et al., ) optimizes learnable continuous prompts in CLIP’s language branch for few-shot image recognition, while CoCoOp (Zhou et al., ) addresses CoOp’s overfitting issues by conditioning prompts on visual features. In the visual domain, Visual Prompt Tuning (VPT) (Bahng et al., ) and Dual-modality Prompt Tuning (DPT) (Xing et al., ) enhance CLIP’s vision encoder by learning visual prompts in pixel space and dynamically generating prompts through cross-attention, respectively. TransHP (Wang et al., ) leverages category hierarchy for prompt learning to improve classification performance. LoGoPrompt (Shi & Yang, ) enhances classification by incorporating synthetic images with class name text as auxiliary visual prompts. MaPLe (Khattak et al., ) explores multimodal prompt learning, jointly optimizing prompts in both vision and language branches. Other recent works have focused on regularizing prompt learning to leverage the knowledge from base VLMs effectively, demonstrating enhanced generalization in varied downstream visual tasks (Khattak et al., ; Bulat & Tzimiropoulos, ; Roy & Etemad, ). PromptSRC, for instance, introduced a self-regulating method that restricts both the vision and text prompt, demonstrating improved generalization. Distinct from these approaches, PLOT (Chen et al., ) and ALIGN (Wang et al., ) leverage Optimal Transport to align multiple prompts with local visual features, either from the multi-head self-attention layer or at a token level. Our work diverges from these methods by introducing a hierarchical "Tree of Attribute" framework derived from LLMs to structure textual descriptions and guide the learning of specialized "domain expert" tokens for attribute-level understanding.

Enhancing model’s understanding using visual attributes. There’s a growing emphasis on enhancing model’s understanding of the class names by integrating visual attributes into the model. A common to do that is to use detailed visual descriptions from LLMs, moving beyond generic prompts (Zhou et al., ; ). These descriptions, like the “fur pattern” or “tail shape” of a cat, provide fine-grained and distinctive characteristics. The use of LLMs like GPT-3 (Brown et al., ), allows for efficient generation of a broad spectrum of class-specific descriptions, offering an advantage over manually crafted templates. While this approach has been extensively researched in zero-shot contexts (Kim et al., ; Menon & Vondrick, ; Parkhi et al., ; Roth et al., ; Yan et al., ; Yang et al., ; Fabian et al., ; Pratt et al., ; Novack et al., ), its application in conjunction with prompt learning for few-shot tasks remains relatively unexplored(Mao et al., ; Lee et al., ; Tian et al., ; Zheng et al., ; Zhang et al., ; Liu et al., ). Additionally, another line of research integrates the visual attributes through data augmentations. For example, AAPL (Kim et al., ) uses adversarial token embedding to mitigate bias and enhance attribute-specific learning through augmented image features. However, we note the following limitations of previous methodologies. First, in the use of LLMs, existing methods have largely utilized unstructured descriptions, which lacks an organized framework for effective utilization. Additionally, while data augmentation can improve learning of invariant visual attributes, it is constrained by the limitations of image data augmentation, which cannot cover all possible variations. Our approach diverges by structuring the visual descriptions into a “Tree of Attribute”, coupled with learnable visual prompts as domain experts. Additionally, LLM-generated descriptions often cover a wide range of potential class descriptions, of which not all may be pertinent to a given image, pointing to the need for a selective pooling mechanism to ensure optimal image-text alignment. We further introduce a vision-conditional pooling layer for refined image-text alignment. This structured approach not only enhances the interpretability of the model’s learning process but also significantly improves alignment accuracy between image content and descriptive text.

## 3 Methodology

### 3.1 Preliminary

CLIP. Our approach is built on the pre-trained vision-language model, CLIP (Radford et al., ). Formally, let ${(x,c)}$ denote the dataset, where $x$ is an image and $c\in\{1,\ldots,C\}$ are the class labels. For an image $x$, the vision encoder $h_{I}(\cdot)$ transforms it into a feature vector $\bm{f}^{v}_{x}=h_{I}(x)$. Simultaneously, each class label $c$ is mapped to a text prompt $t_{c}=\texttt{a photo of a \{c\}}$, and converted into textual feature vectors ${\bm{f}^{t}_{c}}=h_{T}(t_{c})$. The predicted class $\hat{y}$ is given by:

$$ $\hat{y}=\underset{c}{\mathrm{argmax}}\cos(\bm{f}^{v}_{x},\bm{f}^{t}_{c})$ (1) $$

where $\cos(\cdot)$ denotes cosine similarity.

Image classification with class descriptions. To improve the model’s understanding of the categories in the transfer datasets, previous works (Menon & Vondrick, ; Roth et al., ) use more detailed descriptions from Large Language Models (LLMs) instead of the simple "a photo of a {c}" to prompt the CLIP text encoder. Under this approach, a convoluted set of descriptions is generated for a class $c$ as $\mathbb{D}_{c}:\{$"c, which is/has/etc description." $\}$, e.g. c="television" and description="black or grey". This classification is reformulated as

$$ $\hat{y}=\underset{c}{\mathrm{argmax}}\frac{1}{|\mathbb{D}_{c}|}\underset{d\in \mathbb{D}_{c}}{\sum}\cos(h_{I}(x),h_{T}(d))$ (2) $$

Figure: Figure 2: Overview of the proposed TAP method. TAP uses a bottom-up approach to aggregate the generated Tree of Attribute. The vision-conditional pooling (VCP) layer aggregates descriptions into attribute-level features, which are aligned with visual expert tokens focusing on specific attributes (e.g., color, texture). These attribute-level features are then combined to make class predictions via a weighted sum of logits from each attribute, fully leveraging the hierarchical structure within the tree.
Refer to caption: x2.png

### 3.2 Overall Framework

We rethink the descriptions by LLM $\mathbb{D}_{c}$ as nodes in knowledge graphs. While previous methods generate an unstructured set of descriptions, we distill structured knowledge graphs for each class $c$ from LLM, in which the root node is the class name $c$, capturing the highest level semantics, and the leaf nodes are the detailed descriptions capturing fine-grained details. In this framework, previous paradigms only generate the leaf nodes of the graph, with the edges and graph structure missing, where the rich and inherent structure from the descriptions is overlooked. To address this limitation, we formulate our approach as a Tree of Attribute, which follows the “concept - attribute - description” structures, as illustrated in Fig. 1 (c).

Besides weighting the descriptions equally, previous works align descriptions that describe images from different aspects and at different granularities with a singular CLS token from the image encoder. However, while the use of a single CLS token is effective in certain contexts, we note that the CLS token is designed to capture the global information of an input image $x$ (Dosovitskiy et al., ). As a result, even though this helps to further inform global understanding, it may fail to effectively capture the nuances and variances at the attribute level, which leads to suboptimal use of the rich descriptions. We address this by introducing a set of learnable prompt tokens that serve as domain experts in the vision branch, each of which aligns with a specific attribute-level textual embedding.

Additionally, close inspection of the LLM-generated descriptions indicates limited contextual relevance and a high degree of diversity. Previous works (Roth et al., ) reflect the issue of descriptions that are likely not co-occurring e.g. “steam” and “fried”. We further identify cases where the descriptions are technically correct but irrelevant to certain images, such as describing “long tail” in frontal images of cats, underscoring the need for a selective pooling mechanism. Thus, we introduce a vision-conditional pooling layer to extract instance-specific text features for each attribute for selecting the most applicable descriptions.

Overall, TAP leverages the tree structure in two key ways: first, a top-down process generates attributes and corresponding descriptions for each class in a structured and contextually relevant manner. This ensures that the descriptions are structured and contextually relevant. Second, a bottom-up process aggregates information from the leaf nodes (descriptions) into attribute-level features, which are aligned with visual expert tokens. These expert tokens focus on fine-grained visual attributes, such as color or shape. Finally, the aggregated attribute-level features contribute to class predictions using a weighted sum of prediction logits, fully utilizing the hierarchical relationships within the tree. This dual approach allows TAP to capture both high-level structure and fine-grained details, leading to enhanced alignment of visual and textual data and improved model performance and interpretability. Inspired by CoOP (Zhou et al., ), we also incorporate textual contextual tokens in the text encoder. The overall framework is presented in Fig. 2.

### 3.3 Tree of Attribute generation by LLMs

We redefine the process of integrating LLM-generated descriptions by introducing a knowledge graph $\mathcal{G}_{c}=\{\mathbb{V}_{c},\mathbb{E}_{c}\}$ for each class $c$, where $\mathbb{V}_{c}$ denotes the set of nodes, and $\mathbb{E}_{c}$ denotes the edges that capture the semantic relationship between nodes. In previous works, $\mathbb{V}_{c}$ is the set of descriptions $\mathbb{D}_{c}$, while $\mathbb{E}_{c}$ is missing. We argue that such methods overlook the inherent structure among the descriptions and thus do not exploit the richness of these descriptions effectively. To better leverage knowledge from LLMs, we introduce an attribute layer to link the root node class name, and the leaf node descriptions. The attribute nodes include visual attributes generated by LLMs, such as color and shape, for systematically guiding description generation as illustrated in  Fig. 1 (c). Each branch of this “tree” represents a specific attribute, with the subsequent “leaves” fleshing out the descriptions with finer details. In this framework, $\mathbb{G}_{c}$ includes the class name which is the root node, the set of attributes such as color and shape being the intermediate layer, and lastly the set of descriptions under each attribute node. $\mathbb{E}_{c}$ includes the edges that build up the hierarchy. This structure allows for a nuanced representation of class information, spanning from general concepts down to specific attributes and detailed descriptions.

To this end, we introduce the Tree of Attribute (ToA), where we use a tree structure to model the relationship and structure of the descriptions. Let $\mathbb{A}_{c}$ denote the set of attributes, and for each attribute $a_{c}\in\mathbb{A}_{c}$, we denote its leaf nodes as $\mathbb{D}^{a}_{c}$. Each set $\mathbb{D}_{c}^{a}$ contains descriptions that specifically pertain to attribute $a$ for class $c$, which is denoted as

$$ $\mathbb{D}_{c}^{a}=\{d_{c}^{a,1},d_{c}^{a,2},\ldots,d_{c}^{a,n}\},$ (3) $$

where $d_{c}^{a,i}$ represents the $i$-th description for attribute $a$ of class $c$ and $n$ is the number of descriptions per attribute.

The process of generating a Tree of Attribute (ToA) unfolds in three steps:
1) Attribute Generation: We first query LLMs with the dataset information and ask it to generate a set of attributes $\mathbb{A}$ which are considered relevant and characteristic of the dataset. 2) Example Generation: We then ask LLMs to generate descriptions for a randomly sampled class in the dataset, using the attributes $\mathbb{A}$ identified in the previous step. Each description takes the format of “class, which {is/has/etc} {description}”. 3) Description Generation for All Classes: Building upon the Q&A template from the previous step, the LLM is then tasked with generating descriptions for all classes in the dataset.

Additionally, we incorporate a “global context” attribute which is aligned with the CLS token in the vision encoder. The descriptions are the 7 standard templates provided in (Radford et al., ).

### 3.4 Learning TAP with Learnable Expert Tokens

To fully exploit the structured Tree of Attribute, we introduce learnable visual expert tokens $\textbf{p}_{a}^{v}$ in the vision branch to learn from each of the attribute nodes $a\in\mathbb{A}$. Unlike traditional methods that rely on a single CLS token for alignment, these expert tokens enable focused learning on specific image attributes, such as color or shape, enhancing the model’s performance and interpretability.

We denote the set of introduced visual expert tokens as $\mathbb{P}^{v}=\{\bm{p}^{v}_{a}|a\in\mathbb{A}\}$. Akin to the idea of visual prompt tuning (VPT) (Jia et al., ), we insert $\mathbb{P}^{v}$ into the input sequence of the vision encoder, forming the prompted input sequences $\mathbb{\tilde{X}}_{p}=\{\bm{e}_{\text{CLS}},\mathbb{P}^{v},\mathbb{E}_{\text{
patch}}\}$, where $\bm{e}_{\text{CLS}}$ is the input CLS token, and $\mathbb{E}_{\text{patch}}$ denotes the embedded patch tokens. To further boost the model’s capacity for nuanced attribute representation, we employ deep prompting by introducing a zero-initialized layer residual for each prompt token across transformer layers, which provides more explicit attribute guidance across transformer layers. In parallel, we adopt a set of $m$ learnable context tokens $\mathbb{P}^{t}=\{\bm{p}^{t}_{j}|j\in\{1,2,...,m\}\}$ for the text encoder shared across all descriptions, similar to (Zhou et al., ).

### 3.5 Vision-Conditional Pooling

To mitigate issues of misalignment and potential misleading information from the broad spectrum of LLM-generated descriptions, we proposed an adaptive vision-conditional pooling layer, applicable to each set of attribute descriptions $\mathbb{D}_{a}$ shared across all classes to dynamically pool the most applicable descriptions based on the visual content of the image $x$ using its corresponding visual expert token denoted as $\bm{p}_{a,x}^{v}$. For ease of expression, we will proceed without explicitly mentioning $x$, though it’s important to note that both the expert token and the resulting attribute-level embeddings are dependent on the visual information. Intuitively, VCP uses attention to calculate the similarity between $\bm{p}_{a}^{v}$ and all embedded descriptions in attribute $\mathbb{D}_{a}$, which are then used as weights for a weighted sum of the original description embeddings. Formally, for each attribute $a$ and its associated expert token $\bm{p}_{a}^{v}$, the pooled attribute-level embedding $\bm{v}^{a}_{c}$ for class $c$ and attribute $a$ is:

$$ Query $\displaystyle=\bm{W_{q}}\cdot\bm{p}_{a}^{v},$ (4) Key $\displaystyle=\bm{W_{k}}\cdot\texttt{Emb}(\mathbb{D}^{a}_{c}),$ Attention Score $\displaystyle=\texttt{softmax}(\text{Query}\cdot\text{Key}^{T}),$ $\displaystyle\bm{v}^{a}_{c}$ $\displaystyle=\bm{\text{Attention Score}}\cdot\texttt{Emb}(\mathbb{D}^{a}_{c}),$ $$

where $W_{q}$ and $W_{k}$ are learnable weights $\in\mathbb{R}^{d\times d}$, $\texttt{Emb}(\cdot)$ denotes the embedding function, and $\texttt{softmax}(\cdot)$ is the Softmax function. This layer mirrors cross-attention but omits $\bm{W_{v}}$ to maintain the output within the CLIP V-L space.

### 3.6 Training and Inference

Training objective.
During training, each visual expert token $\textbf{p}_{a}^{v}$ is aligned with its associated attribute-level embedding $\textbf{v}^{a}_{c}$, trained with the following contrastive objective:

$$ $L_{{con}}(\bm{p}_{a}^{v},\bm{v}^{a}_{c})=-\frac{1}{N}\sum_{i=1}^{N}\log\frac{ \exp(\cos(\bm{p}_{a}^{v},\bm{v}_{y}^{a})/\tau)}{\sum_{c=1}^{C}\exp(\cos(\bm{p} _{a}^{v},\bm{v}_{c}^{a})/\tau)},$ (5) $$

where $N$ represents the number of training samples, and $\tau$ is the learned temprature of CLIP. The total classification loss $L_{\text{class}}$ is the average of the contrastive loss from each expert token as well as the CLS token, defined as:

$$ $L_{{class}}=\frac{1}{|\mathbb{A}|}\bigg{(}\sum_{a\in\mathbb{A}}L_{{con}}(\bm{p }_{a}^{v},\bm{v}_{c}^{a}))\bigg{)},$ (6) $$

Similar to (Khattak et al., ) and (Bulat & Tzimiropoulos, ), we regularize the vision CLS token, text feature, and the prediction logits from each attribute using the vanilla CLIP model. We denote the regularization loss as $L_{reg}$, where the details can be found in Appendix. The overall training objective is $L_{\text{total}}=L_{\text{class}}+L_{\text{reg}}$.

Prediction fusion. During inference, we integrate the prediction by each attribute expert pair by a weighted sum, formulated as follows:

$$ $\tilde{y}=\underset{c}{\mathrm{argmax}}\bigg{(}\alpha\cos(\bm{f}^{v}_{{CLS}}, \bm{v}_{c}^{CLS})+\frac{1-\alpha}{|\mathbb{A}|-1}\sum_{a\in\mathbb{A} \textbackslash\{CLS\}}\cos(\bm{p}_{a}^{v},\bm{v}_{c}^{a})\bigg{)}$ (7) $$

where $\alpha$ is a hyperparameter that signifies the weight assigned to the global context provided by the CLS token, balancing its contribution with that of the attribute-specific expert prompts.

**Table 1: Comparison with state-of-the-art methods in base-to-novel generalization. HM: harmonic mean (Xian et al., ).**
|  | Base | Novel | HM |
| --- | --- | --- | --- |
| CLIP | 69.34 | 74.22 | 71.70 |
| CoOp | 82.69 | 63.22 | 71.66 |
| Co-CoOp | 80.47 | 71.69 | 75.83 |
| ProGrad | 82.48 | 70.75 | 76.16 |
| RPO | 81.13 | 75.00 | 77.78 |
| LoGoPrompt | 84.47 | 74.24 | 79.03 |
| PromptSRC | 84.26 | 76.10 | 79.97 |
| TAP | 84.75 | 77.63 | 81.04 |

## 4 Experiments

We extensively evaluate our method in three settings: 1) Base-to-novel class generalization, where the datasets are equally split into base and novel classes. We train the model on the base classes only and evaluate on both base and novel classes; 2) Cross-dataset transfer, where we train on ImageNet with 16 shots per class, and directly evaluate on other datasets in zero-shot; and 3) Few-shot classification with 16 shots per class.

Datasets and baslines. For all of the three settings, we follow previous works (Zhou et al., ; ), using 11 image recognition datasets, including: ImageNet (Deng et al., ) and Caltech101 (Fei-Fei et al., ) for generic object recognition; OxfordPets (Parkhi et al., ), StanfordCars (Krause et al., ), Flowers102 (Nilsback & Zisserman, ), Food101 (Bossard et al., ), and FGVCAircraft (Maji et al., ) for fine-grained classification; SUN397 (Xiao et al., ) for scene recognition; UCF101 (Soomro et al., ) for action recognition; DTD (Cimpoi et al., ) for
texture classification; and EuroSAT (Helber et al., ) for satellite image analysis. We benchmark against several leading methods, including CLIP (Radford et al., ), CoOp (Zhou et al., ), Co-CoOP (Zhou et al., ), ProGrad (Zhu et al., ), RPO (Lee et al., ), LoGoPrompt (Shi & Yang, ), and the state-of-the-art PromptSRC (Khattak et al., ).

Implementation details.
A pre-trained CLIP model with a ViT-B/16 vision backbone is used in all of our experiments and results are averaged over 3 runs. We use GPT-3.5-turbo (Ouyang et al., ) for attribute and description generation. We initialize the text context tokens with the word embedding of "a photo of a." During training, we iteratively train the vision and text encoders with 5 epochs for vision and 1 epoch for text schedule. We train a total of 60, 24, and 120 epochs for base-to-novel generalization, cross-dataset transfer, and few-shot classification respectively. We set $\alpha=0.4$ for all datasets. We also use a Gaussian Prompt Weighting (GPA) following (Khattak et al., ), with a mean of $0.9N$, std of $0.1N$, where $N$ represents the total number of epochs, for all tasks. Refer to the Appendix for additional implementation details.

### 4.1 Base-to-Novel Generalization

In base-to-novel generalization, we equally split the classes into base and novel classes. Initial training and evaluations are conducted on the seen base classes, followed by evaluation on the unseen novel classes in a zero-shot manner. TAP surpasses prior state-of-the-art models in terms of the base and novel class accuracy, as well as their harmonic mean across most of the 11 datasets, with an average increase of 1.53% in the zero-shot novel class prediction, and a 1.07% increase in the overall harmonic mean in average, as detailed inTable 1. Notably, our method improves unseen class prediction without compromising base class performance, exhibiting an average performance boost of 0.49%. In the challenging fine-grained tasks such as DTD, EuroSAT, and UCF101, TAP achieves significant improvements in novel class prediction by 5.03%, 8.27%, and 3.63% respectively. These results underscore the robust generalizability and efficacy of our method across diverse scenarios.

### 4.2 Cross-dataset Transfer

To further investigate the generalization capability of TAP , we train on ImageNet with 16 shots per class, and directly test on the other 10 datasets under zero-shot without further tuning.
As shown in Table 2, TAP demonstrates better generalizability on 8/10 target datasets compared to PromptSRC (Khattak et al., ), and achieves an average performance increase of $0.75\%$. Additionally, while the performance increase of previous methods on target datasets come with costs on the source dataset ($-0.49\%$ for CoCoOP and $-0.24\%$ for PromptSRC) as compared to CoOP (Zhou et al., ), TAP also outperform previous methods on the source dataset with $1.03\%$ increase compared to PromptSRC ($0.79\%$ incrase compared to CoOP), demonstrating TAP ’s robustness in domain generalization without sacrifice on source dataset performance.

**Table 2: Comparison with state-of-the-art methods in cross-dataset transfer evaluation.**
|  | Source | Target |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | ImageNet | Caltech101 | Pets | Cars | Flowers | Food101 | Aircraft | SUN397 | DTD | EuroSAT | UCF101 | Average |
| CoOp | 71.51 | 93.70 | 89.14 | 64.51 | 68.71 | 85.30 | 18.47 | 64.15 | 41.92 | 46.39 | 66.55 | 63.88 |
| CoCoOp | 71.02 | 94.43 | 90.14 | 65.32 | 71.88 | 86.06 | 22.94 | 67.36 | 45.73 | 45.37 | 68.21 | 65.74 |
| PSRC | 71.27 | 93.60 | 90.25 | 65.70 | 70.25 | 86.15 | 23.90 | 67.10 | 46.87 | 45.50 | 68.75 | 65.81 |
| TAP | 72.30 | 94.30 | 90.70 | 65.60 | 70.93 | 86.10 | 24.57 | 68.30 | 50.20 | 46.00 | 68.90 | 66.56 |

**Table 3: Comparison with state-of-the-art methods in few shot classification results with 16 shots.**
|  | 16-Shot Classification |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Average | ImageNet | Caltech101 | Pets | Cars | Flowers | Food101 | Aircraft | SUN397 | DTD | EuroSAT | UCF101 |
| CLIP | 78.79 | 67.31 | 95.43 | 85.34 | 80.44 | 97.37 | 82.90 | 45.36 | 73.28 | 69.96 | 87.21 | 82.11 |
| CoOp | 79.89 | 71.87 | 95.57 | 91.87 | 83.07 | 97.07 | 84.20 | 43.40 | 74.67 | 69.87 | 84.93 | 82.23 |
| CoCoOp | 74.90 | 70.83 | 95.16 | 93.34 | 71.57 | 87.84 | 87.25 | 31.21 | 72.15 | 63.04 | 73.32 | 78.14 |
| PSRC | 82.87 | 73.17 | 96.07 | 93.67 | 83.83 | 97.60 | 87.50 | 50.83 | 77.23 | 72.73 | 92.43 | 86.47 |
| TAP | 83.37 | 73.76 | 96.73 | 93.90 | 85.37 | 98.10 | 87.53 | 50.43 | 77.30 | 74.90 | 91.90 | 87.17 |

### 4.3 Few-Shot Classification

In few-shot classification, TAP also outperforms existing methods in 9 out of the 11 datasets. Detailed in Table 3, we achieve an average accuracy of $83.37$ across the 11 datasets, surpassing the previous state-of-the-art methods by $0.5\%$, further demonstrating the effectiveness of our method.

### 4.4 Ablation Study

Figure: Figure 3: Visualization of the class activation maps.
Refer to caption: x3.png

Effects of Tree of Attribute. A core inquiry is whether structuring descriptions into a Tree of Attribute (ToA) offers advantages over an unstructured aggregation of LLM-generated descriptions. To evaluate, we revert to aligning a mixed, unstructured set of descriptions with the CLS token - a common practice in prior studies (Mao et al., ; Lee et al., ; Tian et al., ; Zheng et al., ), while keeping the same number of visual prompt tokens. According to Table 4, substituting the ToA with an unstructured set results in significant performance decreases of 1.86%, 2.31%, and 2.11% across the average base, novel, and their harmonic mean performances, respectively. This stark contrast underscores the ToA’s critical role in enhancing model efficacy.

Effects of Learning through Domain Experts. Further, we examine the impact of substituting the CLS token with visual expert tokens for learning fine-grained attributes, commonly adopted in in previous works (Mao et al., ; Lee et al., ; Tian et al., ; Zheng et al., ). Our findings (Table 5) reveal improvements of 0.89%, 0.78%, and 0.82% in the average base, novel, and harmonic mean accuracies, respectively, upon integrating visual expert tokens. These results support the notion that domain-specific, learnable tokens enhance the model’s ability to grasp fine-grained details by focusing on distinct aspects of the image, as opposed to the CLS token’s global focus.

Effects of fusion coefficient $\alpha$. $\alpha$ in Eq. 7 balance global and local information. We compare the performance of using CLS token only (i.e. $\alpha=1.0$) for making the final prediction against our proposed prediction fusion with $\alpha=0.4$. As shown in Table 6, using CLS token decreases the performance significantly on both base and novel classes. This result further demonstrates the limitations of using a singular CLS token which focuses on global information, and supports the effectiveness of the use of expert tokens which focus on local information.

Effects of Number of Attributes.
In our framework, the selection of attributes is dynamically determined by LLMs, leading to variability across different datasets. This adaptability stands in contrast to a static approach where the number of attributes is uniformly set across all datasets. To understand the impact of this variability, we explore how altering the number of attributes from 1 to 8 influences model performance. Our findings, detailed in Table 7, reveal a performance improvement trend as the number of attributes increases, with an optimal peak at 7 attributes before a slight decline at 8. However, crucially, across all fixed-attribute scenarios, none matched the performance achieved through our method’s dynamic attribute determination. These results underscore the importance of an adaptive approach to attribute selection, as opposed to a one-size-fits-all strategy.

Design choice of the vision-conditional pooling layer. Lastly, we ablate the design of the pooling layer, starting from the naive training-free average pooling, to the attention-based pooling mechanism with condition on the input image. Compared to average pooling, VCP demonstrates a performance gain of 1.08% in the average harmonic mean. Furthermore, when compared with attention-based max pooling, which selects a single description per attribute according to the attention score in Eq. 4, VCP maintains a superior advantage of 1.55% in average harmonic mean. These outcomes attest to the VCP layer’s integral role in finetuning attribute relevance to the visual context, substantiating its design and implementation within our model.

Figure: Table 6: Effects of $\alpha$

### 4.5 Visualization

Expert tokens focus on attribute-related regions.
We further investigate the effects of vision domain experts by visualizing their class activation maps from three illustrative examples using GradCAM (Selvaraju et al., ), as shown inFig. 3. These visualizations underscore the precision with which each expert token concentrates on the image regions pertinent to its designated attribute. Take the first cat image as an example. The “fur pattern” expert distinctly highlights the animal’s fur texture, whereas the “ear” and “eye” experts focus precisely on the respective anatomical features. This pattern of attribute-specific attention is consistent across the evaluated examples, reinforcing the conceptualization of expert tokens as dedicated “domain experts” within the visual field.

Figure: Figure 4: Visualization of the attention weights in the VCP layer for an example “dumplings” image.
Refer to caption: x4.png

**Table 8: Design choice of the pooling layer.**
| Pooling Method | Base Acc. | Novel Acc. | HM |
| --- | --- | --- | --- |
| Attn. Max Pooling | 82.90 | 76.36 | 79.49 |
| Average Pooling | 83.18 | 76.98 | 79.96 |
| VCP (Ours) | 84.75 | 77.63 | 81.04 |

VCP layer pools the most applicable descriptions. The inherently interpretable nature of the VCP layer, thanks to its attention mechanism, allows for insightful visualizations of its operational process. Through the examination of attention weights assigned by the VCP layer to different attributes in a given image, we elucidate the layer’s capability to discern and prioritize the most applicable descriptions. As illustrated in Fig. 4 with a “dumplings” image, the VCP layer adeptly allocates higher attention weights to descriptions accurately reflecting the observed instance (e.g., assigning weights of 0.92 to “round with a pleated edge” under the “Shape” attribute and 0.95 to “soft and chewy texture” under the Texture”). In contrast, less relevant descriptions for the specific image context (e.g., “crescent-shaped” for Shape and “crispy texture from pan-frying” for Texture) receive significantly lower weights. This discernment is crucial, given the class dumplings” encompasses a broad variety of appearances based on cooking methods, yet not all descriptions are fitting for every instance. These visualizations compellingly demonstrate the VCP layer’s effectiveness in refining description relevance, thereby enhancing the model’s interpretative alignment with the visual content.

## 5 Conclusion

This paper introduces Tree of Attribute Prompt learning (TAP), a novel method that integrates detailed, LLM-generated descriptions within VLMs, achieving state-of-the-art performance in base-to-novel generalization, cross-dataset transfer, and few-shot image classification tasks across 11 diverse datasets. TAP leverages a hierarchical "Tree of Attribute" framework, distilling structured knowledge graphs from LLMs for nuanced representation of visual concepts, and employs learnable "domain expert" tokens and a vision-conditional pooling module for optimal image-text alignment. While promising, we note that the reliance on LLMs presents challenges in fine-grained datasets where similar classes require nuanced differentiation, in which cases LLMs generate identical descriptions for distinct classes, impacting novel class prediction performance. It highlights the current limitations of LLMs in discerning highly fine-grained distinctions. Addressing this challenge through enhanced LLM capabilities or alternative strategies will be a key focus of future research.

## Appendix A Appendix

### A.1 Model regularization

Denote the frozen image feature from CLIP vision encoder as $\mathbf{f}^{v}$, the frozen text feature for description $d$ from CLIP text encoder as $\mathbf{f}^{t}_{d}$, and the zero-shot logit prediction from CLIP as ${\hat{y}}$. Additionally, denote the trained image feature as $\mathbf{\tilde{f}}^{v}$, the trained text feature for description $d$ as $\mathbf{\tilde{f}}^{t}_{d}$, and the logit prediction from attribute $a$ after training as $\tilde{y}_{a}$. The losses are as follows:

$$ $L_{L_{1}-V}=||\mathbf{f}^{v}-\mathbf{\tilde{f}}^{v}||_{1}$ (8) $$

$$ $L_{con-T}=-\sum_{d\in\mathbb{D}}\bigg{(}\frac{1}{2}\log\frac{\exp(cos(\mathbf{ f}^{t}_{d},\mathbf{\tilde{f}}^{t}_{d}))}{\sum_{k\in\mathbb{D}_{s}}\exp(cos( \mathbf{f}^{t}_{d},\mathbf{\tilde{f}}^{t}_{k}))}+\frac{1}{2}\log\frac{\exp(cos (\mathbf{f}^{t}_{d},\mathbf{\tilde{f}}^{t}_{d}))}{\sum_{k\in\mathbb{D}_{s}} \exp(cos(\mathbf{f}^{t}_{k},\mathbf{\tilde{f}}^{t}_{d}))}\bigg{)}$ (9) $$

$$ $L_{{KL-attr}}=\frac{1}{|\mathbb{A}|}\bigg{(}\sum_{a\in\mathbb{A}}\mathcal{D_{ KL}}(\hat{y},{\tilde{y}}_{a})\bigg{)}$ (10) $$

The regularization loss is then:

$$ $L_{{reg}}=\mu_{1}L_{{L_{1}-V}}+\mu_{2}L_{{KL-attr}}+\mu_{3}L_{con-T},$ (11) $$

Our overall training objective is thus given by:

$$ $L_{\text{total}}=L_{\text{class}}+L_{\text{reg}}$ (12) $$

To investigate the effectiveness of model regularization, we compare TAP against existing methods with and without regularization. As evidenced in Table 9, the proposed model regularization helps in both base and novel performance, with an increase of $1.62\%$ in average harmonic mean. Comparing to existing methods, TAP is consistently better than other baselines in both settings, demonstrating the robustness of our method.

**Table 9: Effectiveness of model regularization. TAP achieves favorable results under both settings.**
|  | Regularization | Base | Novel | HM |
| --- | --- | --- | --- | --- |
| PSRC-reg | $\times$ | 84.21 | 71.79 | 77.51 |
| MaPLe | $\times$ | 82.28 | 75.14 | 78.55 |
| TAP-reg | $\times$ | 83.37 | 75.82 | 79.42 |
| PSRC | $\checkmark$ | 84.26 | 76.10 | 79.97 |
| TAP | $\checkmark$ | 84.75 | 77.63 | 81.04 |

### A.2 Additional implementation details

Because the number of attributes vary across the 11 datasets which results in different number of learnable parameters, we group the datasets into two and apply two sets of learning rates to balance generalizability and performance. For DTD, Oxford Flowers, Stanford Cars, UCF101, and Caltech101 datasets, which have fewer attributes, we use a low learning rate of 0.002 for the text encoder to avoid overfitting and a high learning rate of 0.006 for the vision encoder to facilitate the learning process. A high $\mu_{3}=3$ is also used to regularize the text encoder for preventing overfitting. For the remaining 6 datasets, which have more attributes, the learning rates for both text and vision encoders are set as 0.004, with $\mu_{3}=1.5$. $\mu_{1}=10$, and $\mu_{2}=2.5$ are used for all datasets.

For base-to-novel generalization and few-shot classification evaluations, we use an adaptive approach for generating the attributes, in which the attributes vary across datasets. Although it turns out to be better than using a fixed set of attributes as shown in Table 7, it is not applicable to the cross-dataset transfer experiment as both VCP layers and visual expert tokens are specific to their corresponding attributes. Therefore, for cross-dataset transfer, we use the following fixed set of 4 attributes that are applicable to all 11 datasets: Pattern, Texture, Shape, and Context.

We use PyTorch Paszke et al. () to implement all experiments on a single NVIDIA A100-80GB GPU. Our code is developed based on the implementation of CoOp Zhou et al. (), which is available at https://github.com/KaiyangZhou/CoOp and released under the MIT license. Our code is also released under the MIT license. Baseline results for the three tasks are taken from their respective publications. For the “global context” attribute which is aligned with the CLS token in the vision encoder, we use the following 7 selected templates provided in Radford et al. ().

"itap of a {class}."

"a bad photo of the {class}."

"a origami {class}."

"a photo of the large {class}."

"a {class} in a video game."

"art of the {class}."

"a photo of the small {class}."

### A.3 Robustness of LLMs

To investigate the robustness of our methods against different LLMs, we additionally generate the descriptions using a locally-served small LLM - Qwen-2-7B-Instruct (Yang et al., ), in which the results are comparable.

**Table 10: Robustness against different LLMs.**
| LLMs | Base Acc. | Novel Acc. | HM |
| --- | --- | --- | --- |
| Qwen-2-7B-Instruct | 84.68 | 77.31 | 80.83 |
| GPT-3.5-Turbo | 84.75 | 77.63 | 81.04 |

### A.4 Prompts for Tree-of-Attribute generation

As introduced in Section 3.3, we generate the Tree-of-Attribute with the following three steps: 1) Attribute Generation, 2) In-Context Example Generation, and 3) Description Generation for All Classes. The prompts for each step are as follows:

1) Attribute Generation:

{Dataset Description.}

Visual attributes refer to observable, describable features of the images that can include color, shape, size, texture, and any specific patterns or markings, which can help differentiate between classes for the dataset. They should be consistently observable across multiple images of the same class. Your task is to generate a list of visual attributes (less than 10) for the {Dataset Name} dataset. Ensure this list is clear, concise, and specific to the dataset’s needs. Avoid generic attributes that do not contribute to distinguishing between classes.

2) In-Context Example Generation

Describe describe what a "{Random Class Name}" class in the {Dataset Name} dataset look like using the generated visual attributes.

You must follow the following rules:

1. For each visual attribute, describe all possible variations as separate sentences. This approach allows for a detailed and clear presentation of each attribute’s range.

2. Provide a maximum of five descriptions for each visual attribute to maintain focus and relevance. Also, aim to provide at least two descriptions to ensure a comprehensive overview of the attribute.

3. The descriptions should provide clear, distinguishable features of each class to support image classification tasks.

4. Descriptions for each attribute are independent from each other, and they should not serve as context for each other.

5. Each description describes an image independetly. If certain description is possible for a class, please just list that description, and do not use words like "may have" or "sometimes have".

6. Reply descriptions only. Do not include any explanation before and after the description.

7. The descriptions should follow the format of "classname, which …", where "…" is the description of the visual attribute.

3) Description Generation for All Classes

{Dataset Description.}

Your task is to write detailed descriptions for various classes within the {Dataset Name} dataset, using the provided visual attributes such as color and shape. These descriptions will help in accurately classifying and understanding the unique features of each class.

You must follow the following rules:

1. For each visual attribute, describe all possible variations as separate sentences. This approach allows for a detailed and clear presentation of each attribute’s range.

2. Provide a maximum of five descriptions for each visual attribute to maintain focus and relevance. Also, aim to provide at least two descriptions to ensure a comprehensive overview of the attribute.

3. The descriptions should provide clear, distinguishable features of each class to support image classification tasks.

4. Descriptions for each attribute are independent from each other, and they should not serve as context for each other.

5. Each description describes an image independetly. If certain description is possible for a class, please just list that description, and do not use words like "may have" or "sometimes have".

6. Reply descriptions only. Do not include any explanation before and after the description.

7. The descriptions should follow the format of "classname, which …", where "…" is the description of the visual attribute.

Q: Describe what a "{Random Class Name}" in the {Dataset Name} look like using the following visual attributes: {Visual Attributes from Step 1.}

A: {Answer from Step 2.}

Q: Describe what a "{Target Class Name}" in the {Dataset Name} look like using the following visual attributes: {Visual Attributes from Step 1.}

A:

In the prompt templates, "Dataset Description" is the description of the dataset from their official website, "Random Class Name" is a randomly sampled class name in the dataset for in-context example generation, and "Target Class Name" is the class name of interest for the current query. While step 1 and 2 are made in two consecutive calls to provide contexts which are queried once per dataset, step 3 is queried independently for each of the remaining classes in the dataset. Our carefully designed prompts for step 1 and 2 guide the LLM in generating high-quality examples. Human review further confirms that the generated in-context examples from these prompts are of high quality even without any manual intervention.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Please refer to the weaknesses section. 

Minor:
- Some examples for elements of the set in L245 would be helpful
soundness: 3
strengths: - The motivation of the paper is strong: a tree of attributes seems logical way to represent information about a class. Pooling module to avoid misalignment is also an interesting idea. 
- The paper is fairly well written and overall idea is easy to follow.
- The experiments and ablations are helpful to understand the effect of the proposed approach.
summary: - The paper proposes a new prompt learning approach for vision language models. 
- Instead of a fixed prompt or LLM-generated unstructured prompt, the paper proposes to use a tree of attributes prompt.
- This organizes the generated description as concept, attribute and description structure essentially converting unstructured information from LLMs to structured.
 - A visual conditioning approach is also proposed to ensure optimal image-text alignment and avoid multiple possibilities of the class. 
- The approach is shown to outperform prior work on multiple tasks and datasets.
weaknesses: Concerns & questions: 
- Statistical significance: Standard deviations at the least would be helpful to understand if improvements are significant. This is especially important since differences in many results seem minor. 
- How sensitive is the approach to choice of LLM ? 
- Table 4: Authors mention that prior works use unstructured set of descriptions, do the drops from using unstructured descriptions in Table 4 match performance of the cited works L428-429? 
- Authors should make it clear if "example generation" strategy was used by the prior works to create descriptions.  Further, the choice of CLIP model and if it matches prior work should also be described to understand the differences in various approaches. 
- Missing references and comparisons: [a], [b]. It would be a good idea to compare and contrast with these. 

[a] Graph-Based Captioning: Enhancing Visual Descriptions by Interconnecting Region Captions
[b] Learning Hierarchical Prompt with Structured Linguistic Knowledge for Vision-Language Models

---

confidence: 1
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In Section 3.4, the approach combines VPT and CoOP to form an alignment mechanism for visual cues and attribute cues. Did the authors introduce new mechanisms in the combination process or innovate in the way the alignment prompts are generated?

2. In Section 3.5, the authors mentioned that adaptive vision-conditional pooling layer is used to alleviate the misalignment problem of visual and text features, but did not clearly explain the actual role of the pooling layer. How is the pooling layer implemented in VCP? Does it involve specific pooling operations or simple weighted aggregation?

3. In the ablation experiment, the authors claimed that the number of attributes generated dynamically improved performance over the fixed number of attributes, but when compared with other methods, the other methods all used a fixed number of attributes. This gives uneven advantages to the proposed method over baselines. How do authors justify this? Have the authors considered adding control variable experiments to verify the specific impact of dynamic generation on performance improvement?

4.TAP’s dependency on a particular LLM for dynamic attribute generation raises questions about adaptability across different LLMs. I suggest the authors evaluate TAP using alternative LLMs (e.g., GPT-4, PaLM, or Claude) to assess robustness. It would be informative to report on consistency in performance across these models and discuss which metrics best capture the variability introduced by different LLM choices, offering insights into TAP’s adaptability and contribution.
soundness: 2
strengths: TAP's hierarchical attribute organization is indeed a new attempt to structure the description into a well-defined knowledge graph framework, different from previous unstructured approaches. The method achieves state-of-the-art performance across multiple challenging scenarios on 11 diverse datasets, and innovatively incorporates domain expert tokens and vision-conditional pooling for optimizing image-text alignment.
summary: This paper proposes the Tree of Attributes Prompt Learning (TAP) framework for enhancing vision-language models (VLMs), especially in zero-shot, cross-dataset, and few-shot tasks. TAP introduces a hierarchical "concept - attribute - description" structure that utilizes large language models (LLMs) to generate a structured knowledge graph for each category, capturing the contextual richness of class names through attributes. This approach also includes vision-conditional pooling to extract instance-specific text features, mitigating misalignment issues between visual and textual data. Experimental results show improved performance over baseline models, highlighting TAP's impact across diverse datasets.
weaknesses: 1.The TAP method does present enhancements over VPT and CoOP by organizing attributes hierarchically and introducing a vision-conditional pooling layer. However, to improve TAP’s impact, I suggest that the authors emphasize distinctive innovations beyond architectural refinements. For instance, exploring a novel alignment mechanism that fully utilizes semantic edge information within the constructed attribute trees might further differentiate TAP from its predecessors. Also, they could investigate additional layers of vision-text alignment that capture attribute-specific nuances more dynamically than existing models.

2. Dynamic attribute generation relies on a large language model, and compared with other methods that use a fixed number of attributes, this flexibility is not an innovation in the method itself, but more like an extension of existing technologies.

3. The edges in TAP’s attribute tree offer promising opportunities for capturing semantic relations. However, the paper lacks a detailed explanation of these edges. Clarifying whether they represent specific semantic relationships (e.g., similarities or hierarchies among attributes) could enhance interpretability. I recommend that the authors either expand on how these edges improve semantic alignment or describe potential modifications to strengthen the tree's expressiveness.

4. Section 3.5 briefly mentions the vision-conditional pooling (VCP) layer but does not detail its operation, especially regarding "pooled attribute-level embedding." Providing a step-by-step description of how VCP pools attributes based on visual inputs and clarifying the term could be helpful. Additionally, illustrating VCP's functionality through specific examples might aid readers in understanding its role in enhancing text-image alignment.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: In Section 3.3, for attribute generation, what type of dataset information is given to the large model?

In Figure 4, each image is accompanied by only two descriptions. Are all images described using two sentences each?

In this paper, it mentions that the method can capture subtle differences between attributes. Could you provide a relevant example?
soundness: 2
strengths: The proposed method incorporates structured tree of attribute into prompt tuning that provide richer supervisory information compared to unstructured attribute information. A set of experiments has been conducted, and the results look promising.
summary: The TAP method structures textual descriptions in a hierarchical “concept-attribute-description” format, effectively creating a knowledge graph from large language models (LLMs) for each category name. This structure allows for a more comprehensive and detailed understanding of the visual content. The paper reimagines learnable prompt tokens as "domain experts," each specializing in different aspects of the image, supplemented by a global perspective provided by the CLS token. To address potential misalignment between general descriptions and specific image content, the paper introduces a vision-conditional pooling module. This module extracts instance-specific text features, ensuring optimal image-text alignment.
weaknesses: One major limitation of the method is that it requires human review to "ensure the quality of the example" (L175). Recall that one major advantage of prompt tuning is that it can adapt large models quickly to specific tasks. However, the requirement of human reviewing in the proposed method is not consistent with this goal. In addition, it is not clear how many human efforts are needed here, and how to handle the potential human bias in quality evaluation.

The ToA is built in a dataset-aware manner using specific prior categorical information. This raises an issue regarding the domain generalization performance of the model, which is not studied.

For training details, different learning rates were used for different datasets, however, existing methods typically use a same LR for all datasets. From this point, the comparison is somewhat unfair.

Figure 2 lacks sufficient clarity, for example, input & output streams are not clear. It is a necessity to re-draw it.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: Strengths:
1. This paper proposes a structured way to prompt the LLM for detailed class descriptions.
2. This paper proposes a framework leveraging LLM-generated class descriptions to improve the vision language models, and achieves SOTA performance on 11 benchmarks.
3. TAP allows the model to capture both high-level information and fine-grained details from the images, which enhances the model’s performance and interpretability.
summary: The paper introduces a TAP, a method that first prompts LLM to generate class descriptions in a structured way and then employs both vision and text prompt learning to learn fine-grained attributes within vision-language models.
TAP leverages a hierarchical framework to distill structured knowledge graphs from LLM and employs learnable "domain expert" tokens instead of using a single CLS token to learn class attributes from the description. Additionally, it uses a vision-conditional pooling module to optimize image-text alignment.

The key contribution of this work is: 1) A structured framework for integrating LLM-generated descriptions, 2) learnable vision tokens and a vision-conditional pooling layer for selective description aggregation
weaknesses: Weaknesses:
No ablation experiment on how using learnable context tokens impacts the performance.
2.  From Table 1,  most of the improvement comes from the base classes and there is still room for improvement on the novel class.
3. Captions for the tables and figures could be more detailed.

---

confidence: 5
contribution: 4
initial_rating: 8
presentation: 3
questions: It would be better if the authors could show the entire attribute set in the appendix.

What is the meaning of ‘contextually relevant’? It is hard for me to clearly understand the meaning of 'context' in this paper.
soundness: 4
strengths: The authors proposed a novel visual prompt learning approach, which can explicitly extract visual features on corresponding visual attributes.

The authors proposed the vision-conditional layer to avoid misaligning images with attributes that do not exist in the corresponding images.

The above two contributions have clear motivations and technically seem to work.

The authors conducted extensive experiments to verify the effectiveness of the proposed approach.
summary: This paper proposes a novel visual prompt learning approach, which reimagines the visual prompt tokens as “domain experts”, each specializing in different aspects of the image. By leveraging these domain experts, the proposed approach can explicitly learn the corresponding visual attributes. To optimize these prompts, the authors further propose ToA to collect category descriptions based on a cross-category-shared attribute set. In addition, the authors also introduced vision-conditional layers to avoid misaligning images with attributes that do not exist in the corresponding images. The experimental results verified the effectiveness of this approach.
weaknesses: The authors claim that TAM is the first work to instruct LLMs to generate a tree of attributes with a “concept - attribute - description” structure for each category. However, existing works “Enhancing CLIP with GPT-4: Harnessing Visual Descriptions as Prompts (ICCV2023W )” and “Democratizing Fine-grained Visual Recognition with Large Language Models (ICLR24)” already proposed the similar corpora and prompt engineering strategies. The authors should discuss these related works in this paper.

In the introduction section, the authors mentioned that some attributes may absent in the input image which necessitate the need for a selective pooling mechanism. However, the proposed vision-conditional layer still pool the most applicable descriptions for each attribute but doesn’t filter out attributes that do not exist in the image.

It seems that this paper was prepared in a hurry. There are some obvious typos, e.g., A\{CLS} in Eq.(7) and Fig. Fig. 1 (b).

In general, in my opinion, this paper doesn't have any major issues; the limitations mentioned above are minor and seem easy to correct in the final version.

Meta-review: This paper proposes the Tree of Attributes Prompt learning (TAP) by first instructing LLMs to generate a tree of attributes with a “concept - attribute - description” structure for each category, followed other learning the hierarchy with vision and text prompt tokens. Initially, five reviewers gave feedback comments for this paper with mixed scores. Two reviewers raised their scores from 5 to 6 after rebuttal since their concerns have been addressed. All reviewers tend to accept this paper after thorough discussions. After checking these discussions and comments, AC finds the rebuttal can address most of the concerns. As such, this paper can be accepted. However, please update the final version based on the comments from reviewers.

Decision: Accept (Poster)