Tree of Attributes Prompt Learning for Vision-Language Models

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 03:06:42

Abstract: Prompt learning has proven effective in adapting vision language models for downstream tasks. However, existing methods usually append learnable prompt tokens solely with the category names to obtain textual features, which fails to fully leverage the rich context indicated in the textual category name. To address this issue, we propose the Tree of Attributes Prompt learning (TAP), which first instructs LLMs to generate a tree of attributes with a ``concept - attribute - description'' structure for each associated category name, and then learn the hierarchy with vision and text prompt tokens. Unlike existing methods that merely augment category names with a set of unstructured descriptions, our approach essentially distills structured knowledge graphs associated with class names from LLMs. Furthermore, our approach introduces text and vision prompts designed to explicitly learn the corresponding visual attributes, effectively serving as domain experts. Additionally, the general and diverse descriptions generated based on the class names may be wrong or absent in the specific given images. To address this misalignment, we further introduce a vision-conditional pooling module to extract instance-specific text features. Extensive experimental results demonstrate that our approach outperforms state-of-the-art methods on the zero-shot base-to-novel generalization as well as few-shot classification across 11 diverse datasets.

Main paper: 
 introduction: 
 Recent advancements in vision-language models (VLMs) like CLIP [33] and ALIGN [13] merge the capabilities of visual perception with linguistic understanding, which have revolutionized the landscape with their zero-shot learning abilities. They proficiently handle tasks on unseen data, bypassing the conventional requirement for task-specific training. This feature has enabled a plethora of applications, ranging from content-based image retrieval to complex visual question answering, setting new benchmarks in the domain. A crucial development in this domain is the concept of prompt learning, which has significantly influenced both natural language processing (NLP) [20] [21] [22] and vision-only models [14, 43, 44, 51] . This approach leverages learnable prompts to guide model understanding, tailoring responses to specific tasks or datasets.Prompt learning, particularly in vision-language models, has garnered considerable interest due to its parameter efficiency and rapid convergence [54, 53, 55, 8, 23] . Techniques like CoOp [54] optimize learnable continuous prompts for few-shot image recognition, enhancing model performance significantly. Recent efforts have expanded to multimodal prompt learning, optimizing prompts in both visual and language domains [15, 16, 38, 19] . Despite their success, these models rely on simplistic text prompts, typically formatted as "a photo of a {class}", illustrated in Fig. 1 (a) . While functional, this approach lacks depth, failing to encapsulate the intricacies and finer details inherent in Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute. visual data. Such limitations hinder the model's ability to fully leverage the rich, descriptive potential offered by more detailed and contextually relevant textual information.••• ••• ••• ••• (d)In parallel, another stream of research has been exploring the utilization of large language models (LLMs) to generate more elaborate and descriptive text prompts for enhancing zero-shot learning capabilities [26, 32, 35, 17, 30, 48, 49, 36, 52, 40] . These LLM-generated descriptions offer a wealth of detail and context, potentially enriching the model's interpretative capabilities. However, current methodologies in integrating these descriptions often do not exploit the full potential of this richness.As shown in Fig. 1 (b), most of these approaches lack a structured framework to organize and utilize these descriptions effectively, leading to a scattergun approach where not all generated descriptions are contextually relevant or optimally aligned with the visual content. In addition, as noted in [35] ,descriptions generated by such paradigms are usually diverse, which covers most possibilities of the class, but include descriptions that are either likely not co-occurring, e.g. "steamed" and "fried", or absent in the input image, e.g. "long tail" for a cat shot from the front, necessitating the need for a selective pooling mechanism for clearer image-text alignments.In response to these challenges, our work introduces "Tree of Attribute Prompt learning (TAP)," a method that redefines the integration and utilization of detailed descriptions within VLMs. As indicated in Fig. 1 (c), unlike existing methods that merely augment category names with a set of unstructured descriptions, our approach essentially distills structured knowledge graphs associated with class names from LLMs. Specifically, we adopt a hierarchical, tree-like structure to systematically generate and integrate descriptions, ensuring a layered and comprehensive understanding of visual content. Each branch of this tree represents a specific attribute, with finer details fleshed out in the subsequent leaves, ensuring that every aspect of the visual content is captured and represented.Furthermore, we reimagine the learnable prompt tokens as "domain experts", each specializing in different aspects of the image, supplemented by the CLS token's global perspective. In addition, we introduce vision-conditional layers for each expert-attribute pair, which pool the most applicable descriptions from each of the attribute sets with condition on the input image content, ensuring optimal image-text alignment. This setup not only provides a detailed, attribute-focused analysis but also harmonizes these insights with the overall context.Extensive experiments in both base-to-novel generalization and few-shot classification across 11 diverse datasets demonstrate the effectiveness of our method. On base-to-novel generalization, TAP achieves average performance gains of 1.07% in harmonic mean over the state-of-the-art methods, and 9.34% over the vanilla CLIP. Competitive results are also observed in few-shot classification. 
related work: 
 Prompt Learning for Vision-Language Models. Prompt learning bridges linguistic understanding and visual perception by guiding VLMs with text prompts, a concept originated in NLP [20] [21] [22] and adapted to vision-only [14, 43, 44, 51] and multimodal contexts [54, 53, 15, 16, 38, 19, 40, 34, 36, 52, 55, 4, 23] . In the textual domain, CoOp [54] optimizes learnable continuous prompts in CLIP's language branch for few-shot image recognition, while CoCoOp [53] addresses CoOp's overfitting issues by conditioning prompts on visual features. In the visual domain, Visual Prompt Tuning (VPT) [1] and Dual-modality Prompt Tuning (DPT) [47] enhance CLIP's vision encoder by learning visual prompts in pixel space and dynamically generating prompts through cross-attention, respectively. TransHP [42] leverages category hierarchy for prompt learning to improve classification performance. LoGoPrompt [38] enhances classification by incorporating synthetic images with class name text as auxiliary visual prompts. MaPLe [15] explores multimodal prompt learning, jointly optimizing prompts in both vision and language branches. Other recent works have focused on regularizing prompt learning to leverage the knowledge from base VLMs effectively, demonstrating enhanced generalization in varied downstream visual tasks [16, 4, 36] . PromptSRC, for instance, introduced a self-regulating method that restricts both the vision and text prompt, demonstrating improved generalization. Distinct from these approaches, PLOT [5] and ALIGN [41] leverage Optimal Transport to align multiple prompts with local visual features, either from the multi-head self-attention layer or at a token level. Our work diverges from these methods by introducing a hierarchical "Tree of Attribute" framework derived from LLMs to structure textual descriptions and guide the learning of specialized "domain expert" tokens for attribute-level understanding.There's a growing emphasis on using visual descriptions for zero-shot recognition, moving beyond generic prompts [54, 53] . These descriptions, like the "fur pattern" or "tail shape" of a cat, provide fine-grained and distinctive characteristics. The use of LLMs like GPT-3 [3] , allows for efficient generation of a broad spectrum of class-specific descriptions, offering an advantage over manually crafted templates. While this approach has been extensively researched in zero-shot contexts [17, 26, 30, 35, 48, 49, 10, 32, 28] , its application in conjunction with prompt learning for few-shot tasks remains relatively unexplored [25, 19, 40, 52, 50] . Previous methodologies, however, have largely utilized unstructured descriptions, lacking an organized framework for effective utilization. Our approach diverges by structuring these descriptions into a "Tree of Attribute" model, coupled with learnable visual prompts as domain experts. Additionally, LLM-generated descriptions often cover a wide range of potential class descriptions, of which not all may be pertinent to a given image, pointing to the need for a selective pooling mechanism to ensure optimal image-text alignment. We further introduce a vision-conditional pooling layer for refined image-text alignment. This structured approach not only enhances the interpretability of the model's learning process but also significantly improves alignment accuracy between image content and descriptive text.3 MethodologyQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: The paper includes a discussion of the limitations associated with relying on LLMs for generating descriptions, particularly in fine-grained datasets where similar classes require nuanced differentiation. This discussion can be found in "Conclusion".Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminary: 
 CLIP. Our approach is built on the pre-trained vision-language model, CLIP [33] . Formally, let (x, c) denote the dataset, where x is an image and c ∈ {1, . . . , C} are the class labels. For an image x, the vision encoder h I (•) transforms it into a feature vector f v x = h I (x). Simultaneously, each class label c is mapped to a text prompt t c = a photo of a {c}, and converted into textual feature vectorsf t c = h T (t c ).The predicted class ŷ is given by:EQUATIONwhere cos(•) denotes cosine similarity.Image classification with class descriptions. To improve the model's understanding of the categories in the transfer datasets, previous works [26, 35] use more detailed descriptions from Large Language Models (LLMs) instead of the simple "a photo of a {c}" to prompt the CLIP text encoder.Under this approach, a convoluted set of descriptions is generated for a class c as D c : {"c, which is/has/etc description." }, e.g. c="television" and description="black or grey".This classification is reformulated asŷ = argmax c 1 |D c | d∈Dc cos(h I (x), h T (d))(2)We rethink the descriptions by LLM D c as nodes in knowledge graphs. While previous methods generate an unstructured set of descriptions, we distill structured knowledge graphs for each class c where the rich and inherent structure from the descriptions is overlooked. To address this limitation, we formulate our approach as a Tree of Attribute, which follows the "concept -attribute -description" structures, as illustrated in Fig. 1 (c ).Text Encoder ❄ ••• ••• Vision Encoder ❄ ••• ••• ••• ••• + + I1• T1 I 1 •T 2 I 1 •T 3 I1• TC I1• T1 I1• T2 I1• T3 I1• TC I 1 •T 1 I1• T2 I 1 •T 3 I1• TC 𝑉𝐶𝑃 ! large,•• • • • ••• 𝑉𝐶𝑃 " 𝑉𝐶𝑃 # 🔥 🔥 🔥 🔥 🔥 🔥 🔥 𝑝 ! $ 🔥 🔥 ••• 𝑣 ! ! 𝑝 ! % 𝑝 " % 𝑝 # % 𝑝 " $ 𝑝 & $ 𝑣 " ! 𝑣 # ! 𝑣 $ ! ••• 𝑣 ! " 𝑣 " " 𝑣 # " 𝑣 $ " ••• 𝑣 ! % 𝑣 " % 𝑣 # % 𝑣 $ % TextEQUATIONBesides weighting the descriptions equally, previous works typically align descriptions that describe images from different aspects and at different granularities with a singular CLS token from the image encoder. However, while the use of a single CLS token is effective in certain contexts, we note that the CLS token is designed to capture the global information of an input image x [9] . As a result, even though this helps to further inform global understanding, it may fail to effectively capture the nuances and variances at the attribute level. This leads to suboptimal use of the rich descriptions. We address this by introducing a set of learnable prompt tokens that serve as domain experts in the vision branch, each of which aligns with a specific attribute-level textual embedding.Additionally, close inspection of the LLM-generated descriptions indicates limited contextual relevance and a high degree of diversity. Previous works [35] reflect the issue of descriptions that are likely not co-occurring e.g. "steam" and "fried". We further identify cases where the descriptions are technically correct but irrelevant to certain images, such as describing "long tail" in frontal images of cats, underscoring the need for a selective pooling mechanism. Thus, we introduce a visionconditional pooling layer to extract instance-specific text features for each attribute for selecting the most applicable descriptions.Overall, our approach utilizes fine-grained descriptions and organizes them in a Tree of Attribute following the "concept -attributes -descriptions" structure. Learnable vision expert tokens are appended to the input image embedding to learn from specific fine-grained attributes such as color and shape. A vision-conditional pooling layer is further added for each attribute to ensure optimal image-text alignment. Inspired by CoOP [54] , we also incorporate textual contextual tokens in the text encoder. The overall framework is presented in Fig. 2 .We redefine the process of integrating LLM-generated descriptions by introducing a knowledge graphG c = {V c , E c } for each class c, where V c denotes the set of nodes, and E c denotes the edges that capture the semantic relationship between nodes. In previous works, V c is the set of descriptions D c , while E c is missing. We argue that such methods overlook the inherent structure among the descriptions and thus do not exploit the richness of these descriptions effectively. To better leverage knowledge from LLMs, we introduce an attribute layer to link the root node class name, and the leaf node descriptions. The attribute nodes include visual attributes generated by LLMs, such as color and shape, for systematically guiding description generation as illustrated in Fig. 1 (c ). Each branch of this "tree" represents a specific attribute, with the subsequent "leaves" fleshing out the descriptions with finer details. In this framework, V c includes the class name which is the root node, the set of attributes such as color and shape being the intermediate layer, and lastly the set of descriptions under each attribute node. E c includes the edges that build up the hierarchy. This structure allows for a nuanced representation of class information, spanning from general concepts down to specific attributes and detailed descriptions.To this end, we introduce the Tree of Attribute (ToA), where we use a tree structure to model the relationship and structure of the descriptions. Let A c denote the set of attributes, and for each attribute a c ∈ A c , we denote its leaf nodes as D a c . Each set D a c contains descriptions that specifically pertain to attribute a for class c, which is denoted asEQUATIONwhere d a,i c represents the i-th description for attribute a of class c and n is the number of descriptions per attribute.The process of generating a Tree of Attribute (ToA) unfolds in three steps: 1) Attribute Generation:We first query LLMs with the dataset information and ask it to generate a set of attributes A which are considered relevant and characteristic of the dataset. 2) Example Generation: We then ask LLMs to generate descriptions for a randomly sampled class in the dataset, using the attributes A identified in the previous step. Each description takes the format of "class, which {is/has/etc} {description}".Human review is performed to ensure the quality of the example. 3) Description Generation for All Classes: Building upon the Q&A template from the previous step, the LLM is then tasked with generating descriptions for all classes in the dataset.Additionally, we incorporate a "global context" attribute which is aligned with the CLS token in the vision encoder. The descriptions are the 7 standard templates provided in [33] .To fully exploit the structured Tree of Attribute, we introduce learnable visual expert tokens p v a in the vision branch to learn from each of the attribute nodes a ∈ A. Unlike traditional methods that rely on a single CLS token for alignment, these expert tokens enable focused learning on specific image attributes, such as color or shape, enhancing the model's performance and interpretability.We denote the set of introduced visual expert tokens as P v = {p v a |a ∈ A}. Akin to the idea of visual prompt tuning (VPT) [14] , we insert P v into the input sequence of the vision encoder, forming the prompted input sequences Xp = {e CLS , P v , E patch }, where e CLS is the input CLS token, and E patch denotes the embedded patch tokens. To further boost the model's capacity for nuanced attribute representation, we employ deep prompting by introducing a zero-initialized layer residual for each prompt token across transformer layers, which provides more explicit attribute guidance across transformer layers. In parallel, we adopt a set of m learnable context tokens P t = {p t j |j ∈ {1, 2, ..., m}} for the text encoder shared across all descriptions, similar to [54] .To mitigate issues of misalignment and potential misleading information from the broad spectrum of LLM-generated descriptions, we proposed an adaptive vision-conditional pooling layer, applicable to each set of attribute descriptions D a shared across all classes to dynamically pool the most applicable descriptions based on the visual content of the image x using its corresponding visual expert token denoted as p v a,x . For ease of expression, we will proceed without explicitly mentioning x, though it's important to note that both the expert token and the resulting attribute-level embeddings are dependent on the visual information. Intuitively, VCP uses attention to calculate the similarity between p v a and all embedded descriptions in attribute D a , which are then used as weights for a weighted sum of the original description embeddings. Formally, for each attribute a and its associated expert token p v a , the pooled attribute-level embedding v a c for class c and attribute a is:EQUATIONwhere W q and W k are learnable weights ∈ R d×d , Emb(•) denotes the embedding function, and softmax(•) is the Softmax function. This layer mirrors cross-attention but omits W v to maintain the output within the CLIP V-L space.Training objective. During training, each visual expert token p v a is aligned with its associated attribute-level embedding v a c , trained with the following contrastive objective:EQUATIONwhere N represents the number of training samples, and τ is the learned temprature of CLIP. The total classification loss L class is the average of the contrastive loss from each expert token as well as the CLS token, defined as:EQUATIONSimilar to [16] and [4] , we regularize the vision CLS token, text feature, and the prediction logits from each attribute using the vanilla CLIP model. We denote the regularization loss as L reg , where the details can be found in Appendix. The overall training objective isL total = L class + L reg .Prediction fusion. During inference, we integrate the prediction by each attribute expert pair by a weighted sum, formulated as follows:EQUATIONwhere α is a hyperparameter that signifies the weight assigned to the global context provided by the CLS token, balancing its contribution with that of the attribute-specific expert prompts.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: The paper focuses on proposing a novel method for prompt learning in VLMs and evaluating its empirical performance. It doesn't introduce any new theoretical results or theorems requiring formal proofs.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 We extensively evaluate our method in two settings: 1) Base-to-novel class generalization, where the datasets are equally split into base and novel classes. We train the model on the base classes only and evaluate on both base and novel classes; and 2) Few-shot classification with 16 shots per class.Datasets and baslines. For both base to novel class generalization and few-shot setting, we follow previous works [54, 53] , using 11 image recognition datasets. The datasets span a range of recognition tasks: ImageNet [7] and Caltech101 [11] for generic object recognition; OxfordPets [30] ,StanfordCars [18] , Flowers102 [27] , Food101 [2] , and FGVCAircraft [24] for fine-grained classification; SUN397 [46] for scene recognition; UCF101 [39] for action recognition; DTD [6] for texture classification; and EuroSAT [12] for satellite image analysis. We benchmark against several leading methods, including CLIP [33] , CoOp [54] , Co-CoOP [53] , ProGrad [55] , RPO [19] , LoGoPrompt [38] , and the state-of-the-art PromptSRC [16] .Implementation details. A pre-trained CLIP model with a ViT-B/16 vision backbone is used in all of our experiments and results are averaged over 3 runs. We use GPT-3.5-turbo [29] for attribute and description generation. We initialize the text context tokens with the word embedding of a photo of a. For both settings, we iteratively train the vision and text encoders with 5 epochs for vision and 1 epoch for text schedule. We set α = 0.4, µ 1 = 10, and µ 2 = 2.5 for all datasets. We train the vision encoder for 50 and 100 epochs, and text encoder for 10 and 20 epochs for base-to-novel generalization and few-shot experiments, respectively. For DTD, Oxford Flowers, Stanford Cars, UCF101, and Caltech101 datasets, we use a learning rate of 0.002 for the text encoder and 0.006 for the vision encoder, with µ 3 = 3. For the remaining 6 datasets, the learning rates for both text and vision encoders are set as 0.004, with µ 3 = 1.5. We also use a Gaussian Prompt Weighting (GPA)following [16] , with a mean of 45, std of 10 for base-to-novel generalization, and 80, 20 for few-shot experiments. Refer to the Appendix for additional implementation details. training and evaluations are conducted on the seen base classes, followed by evaluation on the unseen novel classes in a zero-shot manner. TAP surpasses prior state-of-the-art models in terms of the base and novel class accuracy, as well as their harmonic mean across most of the 11 datasets, with an average increase of 1.53% in the zero-shot novel class prediction, and a 1.07% increase in the overall harmonic mean in average, as detailed inTable 1. Notably, our method improves unseen class prediction without compromising base class performance, exhibiting an average performance boost of 0.49%. In the challenging fine-grained tasks such as DTD, EuroSAT, and UCF101, TAP achieves significant improvements in novel class prediction by 5.03%, 8.27%, and 3.63% respectively. These results underscore the robust generalizability and efficacy of our method across diverse scenarios.In few-shot classification, TAP also outperforms existing methods in 9 out of the 11 datasets. Detailed in Table 2 , we achieve an average accuracy of 83.37 across the 11 datasets, surpassing the previous state-of-the-art methods by 0.5%, further demonstrating the effectiveness of our method. Effects of Tree of Attribute. A core inquiry is whether structuring descriptions into a Tree of Attribute (ToA) offers advantages over an unstructured aggregation of LLM-generated descriptions.To evaluate, we revert to aligning a mixed, unstructured set of descriptions with the CLS token -a common practice in prior studies [25, 19, 40, 52] , while keeping the same number of visual prompt tokens. According to Table 3 , substituting the ToA with an unstructured set results in significant performance decreases of 1.86%, 2.31%, and 2.11% across the average base, novel, and their harmonic mean performances, respectively. This stark contrast underscores the ToA's critical role in enhancing model efficacy.Effects of Learning through Domain Experts. Further, we examine the impact of substituting the CLS token with visual expert tokens for learning fine-grained attributes, commonly adopted in in previous works [25, 19, 40, 52] . Our findings (Table 4 ) reveal improvements of 0.89%, 0.78%, and 0.82% in the average base, novel, and harmonic mean accuracies, respectively, upon integrating visual expert tokens. These results support the notion that domain-specific, learnable tokens enhance the model's ability to grasp fine-grained details by focusing on distinct aspects of the image, as opposed to the CLS token's global focus.Effects of Number of Attributes. In our framework, the selection of attributes is dynamically determined by LLMs, leading to variability across different datasets. This adaptability stands in contrast to a static approach where the number of attributes is uniformly set across all datasets. To understand the impact of this variability, we explore how altering the number of attributes from 1 to 8 influences model performance. Our findings, detailed in Table 5 , reveal a performance improvement trend as the number of attributes increases, with an optimal peak at 7 attributes before a slight decline at 8. However, crucially, across all fixed-attribute scenarios, none matched the performance achieved through our method's dynamic attribute determination. These results underscore the importance of an adaptive approach to attribute selection, as opposed to a one-size-fits-all strategy.Design choice of the vision-conditional pooling layer. Lastly, we ablate the design of the pooling layer, starting from the naive training-free average pooling, to the attention-based pooling mechanism with condition on the input image. Compared to average pooling, VCP demonstrates a performance gain of 1.08% in the average harmonic mean. Furthermore, when compared with attention-based max pooling, which selects a single description per attribute according to the attention score in Eq. ( 4), VCP maintains a superior advantage of 1.55% in average harmonic mean. These outcomes attest to the VCP layer's integral role in finetuning attribute relevance to the visual context, substantiating its design and implementation within our model.Expert tokens focus on attribute-related regions. We further investigate the effects of vision domain experts by visualizing their class activation maps from three illustrative examples using GradCAM [37] , as shown inFig. 3. These visualizations underscore the precision with which each expert token concentrates on the image regions pertinent to its designated attribute. Take the first cat image as an example. The "fur pattern" expert distinctly highlights the animal's fur texture, whereas the "ear" and "eye" experts focus precisely on the respective anatomical features. This pattern of attribute-specific attention is consistent across the evaluated examples, reinforcing the conceptualization of expert tokens as dedicated "domain experts" within the visual field.VCP layer pools the most applicable descriptions. The inherently interpretable nature of the VCP layer, thanks to its attention mechanism, allows for insightful visualizations of its operational process.Through the examination of attention weights assigned by the VCP layer to different attributes in a given image, we elucidate the layer's capability to discern and prioritize the most applicable descriptions. As illustrated in Fig. 4 with a "dumplings" image, the VCP layer adeptly allocates higher attention weights to descriptions accurately reflecting the observed instance (e.g., assigning weights of 0.92 to "round with a pleated edge" under the "Shape" attribute and 0.95 to "soft and chewy texture" under the Texture"). In contrast, less relevant descriptions for the specific image context (e.g., "crescent-shaped" for Shape and "crispy texture from pan-frying" for Texture) receive significantly lower weights. This discernment is crucial, given the class dumplings" encompasses a broad variety of appearances based on cooking methods, yet not all descriptions are fitting for every instance. These visualizations compellingly demonstrate the VCP layer's effectiveness in refining description relevance, thereby enhancing the model's interpretative alignment with the visual content.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes]Justification: The paper provides all the necessary information for reproducing the experimental results.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion: 
 This paper introduces Tree of Attribute Prompt learning (TAP), a novel method that integrates detailed, LLM-generated descriptions within VLMs, achieving state-of-the-art performance in both base-to-novel generalization and few-shot image classification tasks across 11 diverse datasets. TAP leverages a hierarchical "Tree of Attribute" framework, distilling structured knowledge graphs from LLMs for nuanced representation of visual concepts, and employs learnable "domain expert" tokens and a vision-conditional pooling module for optimal image-text alignment. While promising, we note that the reliance on LLMs presents challenges in fine-grained datasets where similar classes require nuanced differentiation, in which cases LLMs generate identical descriptions for distinct classes, impacting novel class prediction performance. It highlights the current limitations of LLMs in discerning highly fine-grained distinctions. Addressing this challenge through enhanced LLM capabilities or alternative strategies will be a key focus of future research.should be consistently observable across multiple images of the same class. Your task is to generate a list of visual attributes (less than 10) for the {Dataset Name} dataset. Ensure this list is clear, concise, and specific to the dataset's needs. Avoid generic attributes that do not contribute to distinguishing between classes.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [No] Justification: Our codebase is built based on the CoOP and CoCoOP [54, 53] , and can be reproduced based on our Methods, Implementation details in main text and appendix. Our code will be released upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The training and test details can be found in section Experiments and Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We follow previous works [54, 53] to report results averaged over 3 runs. Error bars are not reported.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: The type of compute used is provided in Appendix.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper focuses on developing a novel prompt learning method and evaluating its performance on established image recognition datasets. It doesn't involve any form of crowdsourcing, human subject research, or data collection that would necessitate IRB approval or ethical considerations related to study participants. Therefore, this question doesn't apply to our research.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Describe describe what a "{Random Class Name}" class in the {Dataset Name} dataset look like using the generated visual attributes.You must follow the following rules:1. For each visual attribute, describe all possible variations as separate sentences. This approach allows for a detailed and clear presentation of each attribute's range.2. Provide a maximum of five descriptions for each visual attribute to maintain focus and relevance. Also, aim to provide at least two descriptions to ensure a comprehensive overview of the attribute.3. The descriptions should provide clear, distinguishable features of each class to support image classification tasks.4. Descriptions for each attribute are independent from each other, and they should not serve as context for each other.5. Each description describes an image independetly. If certain description is possible for a class, please just list that description, and do not use words like "may have" or "sometimes have".6. Reply descriptions only. Do not include any explanation before and after the description.7. The descriptions should follow the format of "classname, which ...", where "..." is the description of the visual attribute.3) Description Generation for All Classes {Dataset Description.} Your task is to write detailed descriptions for various classes within the {Dataset Name} dataset, using the provided visual attributes such as color and shape. These descriptions will help in accurately classifying and understanding the unique features of each class.You must follow the following rules:1. For each visual attribute, describe all possible variations as separate sentences. This approach allows for a detailed and clear presentation of each attribute's range.2. Provide a maximum of five descriptions for each visual attribute to maintain focus and relevance. Also, aim to provide at least two descriptions to ensure a comprehensive overview of the attribute.3. The descriptions should provide clear, distinguishable features of each class to support image classification tasks.4. Descriptions for each attribute are independent from each other, and they should not serve as context for each other.5. Each description describes an image independetly. If certain description is possible for a class, please just list that description, and do not use words like "may have" or "sometimes have".6. Reply descriptions only. Do not include any explanation before and after the description. In the prompt templates, "Dataset Description" is the description of the dataset from their official website, "Random Class Name" is a randomly sampled class name in the dataset for in-context example generation, and "Target Class Name" is the class name of interest for the current query. While step 1 and 2 are made in two consecutive calls to provide contexts which are queried once per dataset, step 3 is queried independently for each of the remaining classes in the dataset. Human review is performed after step 2 to ensure a high-quality set of attributes and in-context example.While our work primarily focuses on advancing prompt learning in vision-language models, it's crucial to acknowledge the potential broader societal implications of such advancements. On the positive side, TAP could lead to more efficient and accurate image understanding systems, benefiting various domains. For instance, it could enhance accessibility for visually impaired individuals by providing more detailed descriptions of visual content. Furthermore, improved visual understanding could contribute to more effective content moderation, mitigating the spread of harmful online materials. However, these advancements also present potential risks. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper proposes a prompt learning method for adapting vision language models, which instructs LLMs to generate a tree of attributes for each category name and learn the hierarchy with vision and text prompt tokens. The tree structure has a hierarchical “concept-attribute-description” format and the prompt tokens are designed to capture the corresponding visual attributes. A vision-conditional pooling module is introduced to extract instance-specific text features. The proposed method is evaluated on 11 datasets.

This paper received mixed reviews. Reviewers identified the merits of this paper, including: 1. the idea is interesting; 2. the paper is well written and easy to follow; 3. the experimental results are promising. They also raised a number of concerns, including: the proposed method requires many tunings specific to each dataset such as learning rate; lack of experimental evaluation in aspects such as domain generalization setting, marginal improvement, and computational cost; some technical designs need to be clarified (e.g., how to control the quality of the generated attribute description and use the tree prompt in a structure-aware manner) and some technical details are not clear; lack of discussions on the differences with related methods.

Although the rebuttal alleviated part of these issues, some serious concerns remained unaddressed. Especially, the experimental comparison can be unfair since the proposed method uses different learning rates for different datasets and the proposed method requires human reviewing. In the post-rebuttal discussion, the authors argued that there is another method (not compared in the original paper) that also uses different learning rates for different datasets, which may not be a valid reason and is still unfair for other compared methods. The authors claimed that the learning rates are determined "based on our intuition", which is not convincing enough. Although the authors showed that the performance of the proposed method can be maintained in some cases even without manual editing, more thorough and fair comparisons are needed. Moreover, it is difficult to include all the necessary discussions and additional results within the small edits allowed for camera-ready. Overall, although the AC agreed that the idea of this paper is interesting, this paper in its current version may not be ready for publication in NeurIPS. The reviewers have provided very detailed feedback and the AC suggests the authors follow their comments closely and submit the revised version to a future ML venue.

Reviews: review: paper_summary: paper_summary: This paper proposes a new prompt tuning method for adapting the vision-language model.  The authors design the tree of attribute prompt learning to substitute the categorical description for adapting the vision-language model. A vision-conditional pooling module is proposed to extract instance-specific text features. Extensive experimental results demonstrate the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. According to the experiment, the performance improvement of TAP is marginal, e.g., the few-shot performance on most of datasets.  Although the visualization results of VCP layer are impressive, the improvement of this module is also very slight compared to average pooling. 

2. The core motivation of this method is learning fine-grained attributes to adapt VLMs. However, similar ideas have been explored in previous works , e.g., APPL[1], MAP[2].  Please discuss the differences.

[1] AAPL: Adding Attributes to Prompt Learning for Vision-Language Models

[2] Multi-modal Attribute Prompting for Vision-Language Models

3. The construction of ToA depends heavily on the prior information on the category of attributes suitable for the dataset. However, one of the most capability of VLM is its zero-shot ability in the open-vocabulary context. What's the performance of the proposed method in the domain generalization setting?


4. The model details in Figure 2 are not presented very clear, especially the input & output streams. This figure should be refined for better clarity. 


5. The mechanism behind Equation (5) and the function of VCP needs more clarification. Why conduct constrastive learning between expert token P_a^v and attribute embedding v_c^a generated from P_a^v itself, instead of P_a^v and the embedding of attribute descriptions D?
questions: questions: Please refer to Weaknesses.
limitations: limitations: Yes.
paper_summary: paper_summary: The TAP method structures textual descriptions in a hierarchical “concept-attribute-description” format, effectively creating a knowledge graph from large language models (LLMs) for each category name. This structure allows for a more comprehensive and detailed understanding of the visual content. The paper reimagines learnable prompt tokens as "domain experts," each specializing in different aspects of the image, supplemented by a global perspective provided by the CLS token. To address potential misalignment between general descriptions and specific image content, the paper introduces a vision-conditional pooling module. This module extracts instance-specific text features, ensuring optimal image-text alignment.
strength_weakness: strength_weakness: One major limitation of the method is that it requires human review to "ensure the quality of the example" (L175). Recall that one major advantage of prompt tuning is that it can adapt large models quickly to specific tasks. However, the requirement of human reviewing in the proposed method is not consistent with this goal. In addition, it is not clear how many human efforts are needed here, and how to handle the potential human bias in quality evaluation. 

The paper lacks cross-dataset experiments, which is typically provided in existing PT papers. The results are important to examine the domain generalization capability of the method. 

For training details, different learning rates were used for different datasets, however, existing methods typically use a same LR for all datasets. From this point, the comparison is somewhat unfair.
questions: questions: In Section 3.3, for attribute generation, what type of dataset information is given to the large model? 

In Figure 4, each image is accompanied by only two descriptions. Are all images described using two sentences each?

In this paper, it mentions that the method can capture subtle differences between attributes. Could you provide a relevant example?
limitations: limitations: yes
paper_summary: paper_summary: This paper propose a method that aiming to align the vision modality with not only the category name but also the whole concept subgraph the noun represents in the knowledge graph. This is achieved by adding a bunch of attributes branches attached to this concept. The authors argue that this integration of attribute knowledge will make the alignment more transferrable and thus result in a good performance boost in terms of zero/few shot results.
Basically, this work focusing on the topic of textual prompt enrichment task that is investigated before but implement in a different manner. Additionally, the proposed method use seperate tokens to learn different aspectrs of attributes of given images, working as 'domain expert'.
strength_weakness: strength_weakness: 1. The attributes description is generated by the LLM, which could contain hallucinated content. While there are many reliable sources of knowledge such as wikipedia or conceptNet, this paper seems skip these sources to obtain some accurate attributes.
2. Though this paper decide to use a tree structure to represent the concept. The built tree is not encoded in a structure-awared manner. They are still feed as langauge tokens to the LLMs. 
3.  in equation (5), what is $v_y^a$ stands for? 
4. The author argued that the vision-conditional pooling, which is bascially a cross attention layer between the visual and language modal. The authors believe this this design will make the model filter out non-exisiting material in the text description. However, we know that due to the quirk of softmax function. You can never make some tokens attention to be '0'. Thus, the model is learning some spurious correlation aftertall.
questions: questions: 1. Why structured description is so important in your presumption? Given that only the text prompts are structured but visual data are not, will this fact hinder the model to learn a structured in-detail alignment?
2. How do you make sure one expert token will only learn from one attribute? 
3. Can the model trained this way also work well on the downstream tasks?
limitations: limitations: Not applicable.
paper_summary: paper_summary: This paper proposes a new method called "Attribute Prompt Learning Tree (TAP)" to improve the performance of CLIP on zero-shot and few-shot classification tasks. The authors leverage large language models (LLMs) to generate more descriptive text prompts and introduce a hierarchical tree-like structure to systematically generate and integrate these descriptions, ensuring a layered and comprehensive understanding of the visual content. The method also learns specialized "domain expert" prompt tokens that focus on different visual attributes and uses a vision-based pooling module to extract text features for specific instances. Extensive experiments show that TAP outperforms state-of-the-art methods on zero-shot and few-shot classification tasks across multiple datasets
strength_weakness: strength_weakness: 1), TAP introduces many textual and visual prompts, which leads to high computing and time costs. This may limit its applications.

2), TAP first generates hierarchical token prompts, while it seems like TAP does not use such a hierarchical structure to integrate the output of the text encoder. It only uses a pooling strategy to update the text encoder output with the visual feature. That is, TAP also does not utilize these relationships in the prompt graph.

3), TAP can be viewed as a multimodal prompt tuning method. What is the main difference between TAP and MAPLE,  ALIGN.
questions: questions: Please see above
limitations: limitations: None
paper_summary: paper_summary: This paper proposes Tree of Attributes Prompt learning (TAP). Unlike previous works that rely on unstructured class descriptions, this approach distillates structured knowledge graphs associated with class names from LLMs. Text/vision prompts and vision-conditional pooling module are designed to extract instance-specific text features. Extensive experimental results demosntrate its improved performances.
strength_weakness: strength_weakness: - Apart from the new framework, the method highly relies on the quality of tree of attribute generated with GPT-3.5-turbo. There is no study on the robustness aganist different LLMs, different generation prompts, or varying attribute sets.

- The loss includes a model regularization and its effectiveness is not discussed.

- In Figure 2, it is not too clear to me about $I_1 T_1$, $I_2 T_2$,etc. They seem not be discussed in the text parts.
questions: questions: - In Table 6, what is the difference between Attn. Max Pooling and VPC? Is that the former selects one most similar while the latter uses a soft weighted sum?

- In Table 5, why using an adaptive number of attributes is better than a fixed 8 number of experts. When increasing the number of experts from 1 to 8, did the authors observe some patterns in the order of added attributes? Say, at the beginning some general attributes, then finer ones, and later irrelevant ones?

- In Ln 189, could the authors clarify 'deep prompting'. Does it mean, in additional to vision prompt tokens at the input layer, there are other vision prompt tokens inside vision encoder not plotted in Figure 2?

- In Eq. 7, could the authors clarify how $\alpha=0.4$ is chosen, and compare the performances of using CLS token only $\alpha=1.0$?

- In Ln 262, how to revert to unstructured set of descriptions? Are they converted from the same tree of attributes by keeping all attributes and descriptions?
limitations: limitations: One limitation is its reliance on LLMs (GPT) to generate the tree of attribute. When generating more complex responses, it is challening to ensure the quality and variances. How to keep a balance between the diversity of attribute sets and relevancy of attributes to classification is important.

Comments: title: Reply to rebuttal
Thank you
Please respond to the authors' rebuttal
Thank you
Thank you
Reply to rebuttal
Reminder -- please reply to rebuttal
comment: Thank you for your follow-up questions.

W1. We apologize for any confusion. 

To clarify, the human review stage is designed to ensure the quality of LLM-generated descriptions. In practice, we found the LLM-generated descriptions good enough and no manual editing was involved in this stage. Therefore, even if we remove this stage, our model would still achieve the same results. Additionally, in the LLM robustness experiment requested by Reviewer Vt6k, we regenerated the descriptions using Qwen2-7B-Instruct without any human review due to the limited time during the rebuttal process. The results are as follows:

| Base | Novel | HM |
|------|-------|----|
| 84.68 | 77.31 | 80.83 |

These robust results show that the method's performance is maintained.

W3. We apologize for not including TaskRes as one of our baselines. We compare TAP and TaskRes in the 16-shot setting as follows:

| Method | ImageNet | SUN | Aircraft | EuroSAT | Cars | Food | Pets | Flowers | Caltech | DTD | UCF | Average |
|--------|----------|-----|----------|---------|------|------|------|---------|---------|-----|-----|---------|
| TaskRes | 73.0 | 76.1 | 44.9 | 82.7 | 83.5 | 86.9 | 92.4 | 97.5 | 95.8 | 71.5 | 84.0 | 80.8 |
| TAP | 73.8 | 77.3 | 50.4 | 91.9 | 85.4 | 87.5 | 93.9 | 98.1 | 96.7 | 74.9 | 87.2 | 83.4 |

TAP outperforms TaskRes on all datasets, with an average improvement of 2.6% across the 11 datasets.

Regarding the determination of learning rates, we apologize for not being clear enough in our rebuttal. We grouped the datasets based on the number of attributes and adjusted the learning rates for vision and text encoders separately based on our intuition to balance generalizability and performance.

Concretely, for the vision encoder, datasets that have fewer attributes also have fewer learnable expert tokens (thus fewer parameters and lower learning difficulty), we used a larger learning rate (0.006 vs. 0.004) to facilities the learning process. 

For text encoder, the number of learnable text prompts is fixed, where fewer attributes provide fewer text descriptions/data, and thus a smaller learning rate (0.002 vs. 0.004) was used to avoid overfitting.

We hope this additional context clarifies our approach. Thank you again for your thoughtful feedback and for considering these points in your evaluation.
Thanks for the rebuttal. It addresses some of my concerns but one major concern still exists. One limitation of the method is that it requires many tunings specific to each dataset in order to achieve good performance. The tuning process involves two key aspects: 

__Human Intervention (W1)__:  thought the required human effort is minimal, its impact on model performance remains unclear after the rebuttal. 

__Learning Rate Variability (W3)__: The method employs different learning rates for different datasets. While TaskRes is trained in a similar manner, the experiments lack comparative analysis with this approach. Additionally, further discussion is needed on how to determine the most appropriate learning rate for a given dataset.

I expect authors provide additional insights regarding this issue.
Thank you for your recognition of our work and for increasing your rating.

Regarding your question about the efficiency of the iterative training approach:

To improve efficiency, we used einops rearrange to manage multiple classes and batch sizes, along with zero-padding and attention masks to handle varying numbers of descriptions across attributes. We note that these implementations can already maintain a high training effiency. Thank you for your nice suggestion. We will certainly further explore more strategies to improve training efficiency.

Thank you again for your valuable feedback and for the increase in your rating.
Dear Reviewer ZE3g,

Thank you again for reviewing this paper. Since the reviewer-author discussion phase is closing soon, could you please respond to the authors' comments?

Best,

AC
1. Great! Thank you for your detailed explaination and patience. Now I fully comprehend this design, and find it quite inspiring. Just one more concern, an iterative training approach would be less efficient. Is it possible to improve this by using some sort of mask techniques?
2. Sorry for I didn't notice that Figure and results. Now I think you address all my concerns perfectly. Therefore, I will increase my rating accordingly. Good luck.
Thank you for your follow-up question.

1. I apologize for any confusion caused. In our implementation, each expert token is aligned with only one specific attribute's descriptions, ensuring no cross-interference with other attributes. Here’s how this works in practice:

- Text Input as a Dictionary: The text input, representing the Tree of Attributes (ToA), is formatted as a dictionary: ToA = {'color': [color description set], 'shape': [shape description set], ...} After processing through the text encoder, the ToA is embedded as a tensor of shape $E\times N\times D$, where E is the number of attributes (e.g., color, shape), N is the number of descriptions per attribute (to ensure equal length, we apply zero-padding).
- Expert Tokens Interaction: Each expert token from the vision encoder is aligned with its corresponding attribute's descriptions. This is achieved by iterating over each attribute and ensuring that the expert token interacts only with the relevant attribute's descriptions. Below is a pseudocode illustration of this process:
```
text_embed = text_encoder(ToA)  # ToA is a dictionary, text_embed is E x N x D
image_tokens = vision_encoder(image)  # E x D
pooled_embeds = []
for i in range(E):
    expert_token = image_tokens[i]  # 1 x D
    attribute_embed = text_embed[i]  # N x D
    pooled_embed = VCP(expert_token, attribute_embed)  # 1 x D
    pooled_embeds.append(pooled_embed)
```

This process ensures that each expert token is aligned and performs cross-attention only with its corresponding attribute descriptions. The attribute of a vision expert token is determined by the corresponding group of textual descriptions. Although we omitted the handling of multiple classes and batch size for simplicity, in our actual implementation, we used tools like einops to make the process more efficient. We will release our code.

2. Yes, we have conducted both quantitative and qualitative experiments to demonstrate that the VCP module can effectively filter out irrelevant descriptions. 

Quantitatively, in Table 6, we ablated the design choices for the pooling layer, comparing VCP against average pooling and attention-based max pooling which only pools one description that has the highest attention score per attribute. The results show that VCP significantly outperforms these alternatives. 

Qualitatively, we visualized the attention weights of VCP in Figure 4. We observe that the model assigns higher attention to relevant descriptions while giving near-zero attention to incorrect or unrelated ones. This behavior illustrates VCP's ability to focus on the descriptions that are most pertinent to the image content, effectively filtering out those that do not apply.

I hope this explanation clarifies the implementation. If you have any further questions or need more details, I’m happy to provide additional information. Thank you for your consideration, and I look forward to your feedback.
Dear Reviewer,

Thank you for your response. Could you please let us know if there are any specific concerns that remain? We are more than happy to address any further issues during the discussion period.
Thank you for your response. For 3, I'd like to ask further questions as following:
1. I understood that you are using a expert token to do cross-attention for only one group of description. But my problem is, during implementation, how do you select a SPECIFIC expert token to do attention? In your response, you said you insert a bunch of expert tokens into the image embedding sequences. From my understanding, the sequence of are with the shape $E \times D$ now, where E is the size of expert tokens. When you do cross-attention, all the expert tokens will have exposure to the description embedding, isn't it? How do you avoid such exposure? In other words, how to you implement the one expert to one attribute part in your codes? Would you like to share that part?
2. Another question is, you mentioned in your paper said that VCP module is also responsible for filtering out non-related descriptions. Do you have any experiments to show that VCP can actually filtering out those descriptions? Because we all know that some not all descriptions will present in a same image for an object.
Thank you again for your positive assessment and for taking the time to review our work. Your invaluable feedback has been instrumental in improving our paper.
Thank you again for your positive assessment and for taking the time to review our work. Your invaluable feedback has been instrumental in improving our paper.
1. Thank you for your suggestion. We agree that incorporating reliable sources as context before querying the LLM could generate more accurate and reliable results. We will certainly explore this direction in our future work to further enhance the robustness of our method.

2. Thank you for your compliment and understanding. You are correct that our approach groups a collection of descriptions for each attribute using VCP, and each VCP is aligned with an expert token to reflect a hierarchical structure. To clarify further, the structure in our method is actually three levels:

- Class Name (Root Node) → Attributes (Intermediate Layer) → Descriptions (Leaf Nodes)

VCP helps aggregate from descriptions to attributes (from leaf nodes to the attribute layer). Then, the prediction fusion via weighted sum aggregates the attribute predictions into the final class prediction (from the attribute layer to the root node). While we did not use an explicit graph embedding to represent the tree structure, we implemented the hierarchical structure in a more implicit manner through this approach. We appreciate your positive feedback on this implementation.

3. Thank you for providing me the opportunity to clarify. Yes, all expert tokens are indeed inserted together into the image embedding sequences, similar to what VPT did. To ensure only the relevant expert token is doing cross-attention, we used a separate VCP module for each expert token. 

Concretely,
- The number of attributes in the Tree of Attributes (ToA) equals the number of expert tokens added, which also equals the number of VCP modules.
- Each attribute description set is aggregated using a dedicated VCP module that operates solely for that attribute. The pooled embedding from this VCP module is then aligned with the specific expert token associated with that attribute.
- In the VCP module (batch size is omitted for simplicity in the following notation):
    - The query is the expert token of shape $1\times D$ (where $D$ is the embedding dimension).
    - The key is the set of descriptions in this attribute, with shape $N\times D$ (where $N$ is the number of descriptions).
    - The resulting pooled embedding, after cross-attention, is of shape $1\times D$, which aligns directly with the expert token (also of shape $1\times D$).

This design ensures that each expert token only interacts with its relevant attribute.

I hope this detailed explanation clarifies your concerns. Please feel free to ask any further questions, and I sincerely appreciate your consideration in potentially raising the rating.
I thank the authors for their detailed response, and I have read other comments. I decide to keep my rating.
1. Many works have validate that including knowledge sources or using RAG can alleviate hallucination problem. So I believe if you can insert a paragraph from a reliable sources to the LLM as context, the LLMs will generate more reliable results comparing to direct doing QA.
2. Thank you for your explanation. From my understanding, you grouped a collection of discription for each attribute by using VCP and each VCP is aligned with an Expert token to reflect the two-level hirerachical structure, is that correct? I was expecting there is a Graph embedding or something close to embed the tree structure. But I still find this implementation is interesting. Good job.
3. I'd like to learn more about the visiual Expert tokens in 3.4. So, you said that you have A independent tokens serve as Expert tokens. And you insert these vision expert tokens before the image patch sequence like what VPT. Then I wonder, are you inserting all the expert tokens together into the image embedding sequences? If that is the case, when doing the cross attention mentioned in 3.5, how do you make sure only the relevant Expert tokens is doing cross-pooling with the relevant attributes, as described in equation 4. Please answer this question clearly and in detail. I will consider to raise my rating if I received a satisfying response.
The responses have addressed all my questions.
Dear Reviewers,

Thank you for your comments. Please read through all the reviews and the rebuttal and see if authors' responses have addressed your and others' concerns.

Best,

AC
I would like to thank the authors for their detailed responses, which solved my previous questions.