Language Model as Visual Explainer
Authors: Xingyi Yang
Xinchao Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: GPT; Large Language Model; Explainability
Primary area: interpretability_and_explainability
TL;DR: we present Language Model as Visual Explainer to understand the decision from vision model with natural language

Abstract: In this paper, we present Language Model as Visual Explainer (\texttt{LVX}), a systematic approach for interpreting the internal workings of vision models using a tree-structured linguistic explanation, without the need for model training. Central to our strategy is the collaboration between vision models and LLM to craft explanations. On one hand,  the LLM is harnessed to delineate hierarchical visual attributes, while concurrently,  a text-to-image API retrieves  images that are most aligned with these textual concepts. By mapping the collected texts and images  to the vision model's embedding space,  we construct a hierarchy-structured visual embedding tree. This tree is dynamically pruned and grown by querying the LLM using language templates, tailoring the explanation to the model.  Such a scheme allows us 
to seamlessly incorporate new attributes while eliminating undesired concepts based on the model's representations. When applied to testing samples, our method provides human-understandable explanations in the form of attribute-laden trees. Beyond explanation, we retrained the vision model by calibrating it on the generated concept hierarchy, allowing the model to incorporate the refined knowledge of visual attributes. To access the effectiveness of our approach, we introduce new benchmarks and conduct rigorous evaluations, demonstrating its plausibility, faithfulness, and stability.

Language Model as Visual Explainer
Xingyi Yang Xinchao Wang"
National University of Singapore mug.ge...uu.eee mus edu, xinchao@nus.edu.
xyangou.
Abstract
In this paper, we present Language Model as Visual Explainer (LVX), systematic approach for interpreting the internal workings of vision models using tree- structured linguistic explanation, without the need for model training. Central to our strategy is the collaboration between vision models and LLM to craft explanations. On one hand, the I.I.M is harnessed delineate hierarchical visual attributes, while concurrently, text-to-image API retrieves images that are most aligned with these textual concepts. By mapping the collected texts and images the vision model's embedding space, construct hierarchy-structure embedding tree. This tree is dynamically pruned and grown by querying the LLM using language templates, tailoring the explanation to the model. Such scheme allows us to seamlessly incorporate new attributes while eliminating undesired concepts based on the model's 's representations. When applied to testing samples, our method provides human-understarl understandable explanations in the form attribute- laden trees. Beyond explanation, we retrained the vision model by calibrating it on the generated concept hierarchy, allowing the model to incorporate the refined knowledge of visual attributes. To access the effectiveness of our approach, we introduce new benchmarks and conduct rigorous evaluations, demonstrating its plausibility, faithfulness, and stability.
visual
1 Introduction
Unlocking the secrets of deep neural networks is akin to navigating through an intricate, ever-shifting maze, the intricate decision flow within the networks is, many cases, extremely difficult for humans to fully interpret. In this quest, extracting clear, understandable explanations from these perplexing mazes has become an imperative task.
While efforts have been made to explain computer vision models, these approaches often fall explanations. Standard techniques, such as 65 64),
short of providing direct and human-understanddabe mechanical interpretability [22) and prototype
attribution methods [46 55
90
68
[ 62
analysis (T)) 49), only highlight certain pixels features that deemed important by the model. As such, these methods often require the vevolvement of experts to verify interpret the outputs for non- technical users. Natural language explanations 7 41 (35). on the other hand, present an attractive alternative, since the produced texts are better aligned with human understanding. Nevertheless, these approaches typically rely on labor-intensive and biased manual annotation of textual rationales for model training.
In this study, attempt to explain Al decision in human- understandable manner, for example, tree-structured language. We call this task visual explanatory tree parsing. To implement this, we present systematic approach, Language Model as Visual Explainer (LVX), for interpreting vision models structured natural language, without model training.
Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Vision Model
CharGPT
Parse Thee
ChatGiT
Vision Model
Parse Thee
ion mangrine dog What anbur dog lea?
Midel
Image API
Concept Prompt
Test Samples
Dog has tulls and Iour hape
Query Images
Query
car reoogaite logs,
Tran Samples
Construct Trce
dog's lops are munsslar
Query
Retine Pronae
Query Images
XE Heess
Explanation
monlly muccolar legh.
Refine Tree
Construction
Testing
Refinement
Figure I: General workflow of LVX. (Left) A toy example that LLM interacts with vision model to examine its capability. (Mid) It combines vision, language, and visual-langual APIs to create parse tree for each visual model. (Right) In testing, embeddings navigate this tree, and the traversed path provides personalized explanation for the model's prediction.
key challenge that vision models, trained solely on pixel data, inherently lack comprehension of textual concepts within an image. For example, if a model labels an image as "dog", it is unclear whether truly recognizes the features like the wer nose or floppy ear, or it is merely making ungrounded guesses. To address this challenge, we link the vision model with a powerful external knowledge provider, to establish connections between textual attributes and image patterns. Specifically, we leverage large language models (LLM) such as ChatGPT and GPT4 our knowledge providers, combining them with the visual recognition system. Figure | (Left) describes a toy case, where the LLM is interacts with the vision model to explore its capability boundaries. By doing so, we gain insights into the what visual attributes can be recognized by the model.
The pipeline of our approach is illustrated in Figure which comprises two main stages, the construction phase and the test phase.
In the construction phase, our goal is to create an attribute tree for each category, partitioning the feature space of visual model via LLM-defined hierarchy. We begin by extracting commonsense knowledge about each category and its visual attributes from LLMs using in-context prompting [45]. This information is naturally organized as a tree for better organization and clarity. Utilizing text- to-image API, we gather corresponding images for each tree node. These images are subsequently inputted into the vision model to extract prototype embeddings, which are then mapped to the tree.
Once created, the tree is dynamically adjusted, based on the properties of the training set. Specifically, each embedding of the training sample is extracted by the vision model.Such embedding then navigates the parse tree based on their proximity to prototype embeddings. Infrequently visited nodes, representing attributes less recognizable, are pruned. Conversely, nodes often visited by the model indicate successful concept recognition. Those nodes are growned, as the LLM introduces more detailed concepts. Consequently, LVX yields human-understandable attribute trees that mirror the model's 's understanding of each concept.
In the lest phase, we input test sample into the model to extract its feature. The feature is then routed in the parse tree, by finding its nearest neighbors. The root-to-leat path serves as a sample-speciff rationale for the model, offering explanation of how the model arrived at its decision.
To assess our method, we compiled new annotations and developed novel metrics. Subsequently, we test LVX on these self-collected real-world datasets to access its effectiveness.
Beyond interpretation, our study proposes to calibrate the vision model by utilizing the generated explanation results. The utilization of tree-structured explanations plays key role in enhancing the model's 's performance, thereby facilitating more reliable and informed decision-making processes. Experimental results confirm the effectiveness of our method over existing interpretability techniques, highlighting its potential for advancing explainable AI.
To summarize, our main contributions are:
The paper introduces novel task, visual explanatory tree parsing, that interprets vision models using tree-structured language explanations.

---

We introduce the Language Model as Visual Explainer (LVX) to earry out this task, without model training. The proposed LVX is the first dedicated approach leverage LLM explain the visual recognition system.
Our study proposes utilizing the generated explanations calibrate the vision model, leading to enhanced performance and improved reliability for decision-makinn
We introduce new benchmarks and metrics for a concise evaluation of the LVX method.
These tools assess its plausibility, faithfulness, and stability in real-world datasets.
2 Problem Definition
We first define our specialized task, called visual explanatory tree parsing, which seeks to unravel the decision-making process of vision model through tree.
Let us consider the trained vision model f. de- fined function f: X + V/ where X' rep- resents the input image space and y denotes the output label space. In this study, our focus lies on the classification task, where f = -g- o h is decomposed into feature extractor g and linear classification head h. The output space y E R", where n signifies the number of classes. The model is trained on a labeled train- ing set D,, {x,, My and would be eval- uated a test set Drs {x,}/=1-
Classifier
"Dog"
Hang Dooe
Pricted can
Matural
Auail
Jexi
Visual Tree Pursing
loeg
wmm
Figure The illustration of visual explanatory tree parsing. Each input sample is interpreted as parse tree to represent the model's logical process.
The ultimate objective of our problem is to generate an explanation T for each model-input pair (f,x) on the test set, illuminating the reasoning behind the model's prediction I = fx). This unique explanation manifests as a tree of attributes, denoted as T = (V, E), comprising set of N, nodes = {vv V. and Ne edges E' {e, ) The root of the tree is the predicted category, g. while each node V encapsulates a specific attribute description of the object. These attributes are meticulously organized, progressing from the holistic to the granular, and from the general to the specific. Figure provides an example of the parse tree.
Unlike existing approaches 53 3 that explaining visual- models 48
86 93 we
address the more challenging scenario, on explaining vision models trained solely on pixel data. While some models can dissect and explain hierarchical clustering of feature embeddings 677 77 they lack the ability to associate each node with a textual attribute. It is important to note that our explanations primarily focus on examining the properties of the established network, going beyond training vision model or visual-language model [2|44] for reasoning hierarchy [20) and attributes [3I from the image. In other words, visual- risual-language model, that tells the content in the image, can not explain the inner working inside another model. Notably, our approach achieves this objective without supervision and in open-vocabulary manner, without predefined explanations for model training.
3 Language Model as Visual Explainer
This section dives deeper into the details of LVX. At the heart of our approach is the interaction between the LLM and the vision model to construct the parsing tree. Subsequently, we establish rule to route through these trees, enabling the creation of coherent text explanations.
3.1 Tree Construction via LLM
Before constructing our trees, let's take moment to reflect how humans do this task. Typically, we already hold hierarchy of concepts in our minds. When presented with visual stimuli, we instinctively compare the data to our existing knowledge tree, confirming the presence of distinct traits. We recognize familiar traits and, for unfamiliaa ones, we expand our knowledge base. For example, when we think of dog, we typically know that has a furry tail. Upon observing dog, naturally check for the visibility of its tail. If we encounter a hairless tail, previously unknown
3

---

Tuteee ( ('Digees') "aagerr". Marly
soth Sesteeer and Rlapers" and Bosts
CRaKPPT
ChociPT
Cardical, Euemle
Serroeast (X und fowm csmued area)))
INtIaAcaa sssrr tetess the
Promot
Category
Attribute Tree
seat's tall short.
nose 15 smell.
scat's eyes
dank. scol. lives in coostal areas.
Diffarion
Figure 3: Crafting text-image pairs for visual concepts. Through in-context prompting, we extract knowledge from the LLM, yielding visual attributes for each category. These attributes guide the collection of text-image pairs that encapsulate the essence of each visual concept.
to us, we incorporate it into our knowledge base, ready to apply it to other dogs. This process is typically termed Predictive Coding Theory [15 in cognitive science.
Our LVX mirrors this methodology. We employ LLM as a "knowledge provider" to construct the initial conceptual tree. Subsequently, navigate through the visual model's feature space assess the prevalence of each node. a specific attribute is rarely observed, we remove the corresponding nodes from the tree. Conversely, if the model consistently recognizes attribute, enrich the tree by integrating more nuanced, next-level concepts. This iterative process ensures the refinement and adaptation of the conceptual tree within pipeline, which gives rise to LVX.
Generating Textual Descriptions for Visual Concepts. We leverage large language model (LLM) as our "commonsense knowledge provider" 42 955 to generate textual descriptions of visual attributes corresponding to each category. The LLM acts as an external database, providing a rich source of diverse visual concept descriptions. The process is illustrated in Figure 3
Formally, assume we have set of category names, denoted C' {c) where represents the class index. For each of these classes, we prompt an LLM L to produce visual attribute tree. We represent these attributes as d; L(c,, P), where d; is a nested JSON text containing textual descriptions associated with class G. To help generate d;, we use example input-output pairs, P, as in-context prompts. The process unfolds two stages:
Initial Attribute Generation: We initially generate keywords that embody the attributes of each class. This prompt follows a predefined template that instructs the LLM to elaborate on the attributes a visual object. The template is phrased as ("This is a <CLSNAME> because"). The output JSON contains four primary nodes: Concepts, Substances, Attributes, and Environments. As such, the LLM is prompted to return the attributes of that concept. Note that the initial attributes tree may not accurately represent the model; refinements will be made in the refinement stage.
Description Composition: Next, we guide the L.M to create descriptions based on these attributes. Again showease an in-context example and instruct the model to output "Generate sentences that describe a concept according to each attribute.")
Once the LLM generates the structured attributes d, we parse them into an initial tree, represented as TT!01 (v E ,,,)). using the key-value pairs of the JSON text. Those generated JSON tree is then utilized to query images corresponding to each factor.
Visual Embeddings Tree from Retrieved Images. In order to enable the vision model to understand attributes generated by the LLM, we employ a two-step approach. The primary step involves the conversion of textual descriptions, outputted by the LLM, into images. Then, these images deployed to investigate the feature region that symbolizes specific attributes within the model.
The transition from linguistic elements to images is facilitated by the use of arbitrary text-to-image API. This instrumental API enables the generation of novel images or retrieval of existing images that bear strong relevance to the corresponding textual descriptions. An initial parse tree node, denoted by

---

Super
e Tree Prune
Tree Grow
pe(tes) degg
bettor
Rarely Visited
Nodes
Ade eere visuol attrocuke for the
ChatciPt
Nodes
Figure 4: Tree refinement by traversing the embedding tree and querying the LLM model.
e, containing a textual attribute, is inputted into the API yield a corresponding set of K support images, represented as {x; j (2)(v). The value of K is confined to a moderately small range, typically between 5 to 30. The full information of the collected dataset will be introduced in Section 4
Our research incorporates the use of search engines such as Bing, or text-to-image diffusion models like Diffe--Diffusion [56], derive images that correspond accurately to the provided attributes.
Following this, the images are presented to the visual model to extract their respective embeddings, represented as p; 9x.). As such, each tree node contains a set of support visual features P (p= K k= This procedure allows for the construction of an embedding tree, consisting of paired text and visual features. These pairs arranged in tree structure prescribed by the LLM. It is important to note that the collected images are not employed in training the model. Instead, they serve as support set to assist the model in understanding and representing the disentangled attributes effectively. As such, the visual model uses these embeddings as a map to navigate through the vast feature space, carving out territories of attributes, and laying down the groundwork for further exploration and explanation of a particular input.
Tree Refinement Via Refine Prompt. Upon construction, the parse tree structure is refined to better align with the model's feature spaces. This stage, termed Tree Refinement, is achieved through passing training data as query to traverse the tree. Nodes that are seldom visited indicate that the model infrequently recognizes their associated attributes. Therefore, we propose a pruning mechanism that selectively eliminates these attributes, streamlining the tree structure. For nodes that frequently appear during the traversal, we further grow the tree by introducing additional or more detailed attributes, enriching the overall context and depth of the tree. The procedure is demonstrated in Figure 4
Initially, we treat the original training samples, denoted as (Xj: E Dtr. as our query set. Each sample is passed to the visual model to extract feature, represented as q g(x;).
Next, the extracted feature traverses the yj-corresponding tree. Its aim is to locate the closest semantic neighbors among the tree nodes. We define distance metric between q support set P as the point-to-set distance D(q;. P). This metric represents the greatest lower bound of the set of distances from q, to prototypes in P. It is resilient to outliers and effectively suppresses non- to-maximmm nodes.
D( P) = inf (d((; p)(p E P}
(1)
we set d(q. p) = 10 + ||--p| It emphasizes close points while moderating the impact of larger distances. Following this, employ a Depth-First Search (DFS) algorithm to locate the tree node closest the query point qj. After finding this node, each training point (xj, yj) is assigned to specific node of the tree. Subsequently, we count the number of samples assigned to a particular node v², using the following formula:
In our paper, similar to 10 58
Co. Σ 1fue = argmin Vlg.P.))
(2)
vEV (0)
this formula, the indicator function and Ps, denotes the support feature for node v. Following this, we rank each node based on the sample counter, which results in two operations to update the tree architecture (((++1) Crow(Prune( )), where stands as the iteration number
* Tree Pruning. Nodes with the least visits are pruned from the tree, along with their child nodes.
PIn practice, we implement d(q. p) = log((
[g 241 incorporating [|q-p]
0 ensure numerical stability.
5

---

Tree Growing. For the top-ranked node, we construct new inquiry to prompt the LLM to generate attributes with finer granularity. The inquiry is constructed with an instruction template ''Add visual attributes for the <NoDENAME> of <CLASSNAME>, to the json")
Common Node Discrimination. In cases where different categories share common nodes (e.g. "human" and "dog" both have "ear"), we execute targeted growth step aimed at distinguishing be- tween these shared elements. To achieve this differentiation, we utilize contrasting question posed to the LLM (TThe <NoDENAME>> of <CLASsNAMB1>
different from <CLASSNAME2> because")
The revised concept tree generated by the LLM provides comprehensive and detailed represen- tation of the visual attribute. To refine the attribute further, we employ an iterative procedure that involves image retrieval and the extraction of visual embeddings, as illustrated in Figure [ This iterative process enhances the parse tree by incorporating new elements. is introduced, the attribute areas within the feature space become increasingly refined, leading to improved interpretability. our experiment, we performed five rounds of tree refinement.
each new element
3.2 Routing in the Tree
Once the tree is established, the model predicts the class of new test sample x' and provides an explanation for this decision by finding the top-k nearest neighbor nodes.
Specifically, the model predicts the category ÿ for the test instance x' as ÿ = f(x). The extracted image feature q' corresponding to x' is routed through the tree. Starting from the root, the tree is traversed to select the top-k nearest neighbor nodes {t: {ri)/==1 based on the smallest D(q', P..) values, representing the highest semantic similarity between and the visual features the tree 's nodes. The paths from the root to the selected nodes are merged to construct the explanatory tree T for the model's prediction.
This parse tree structure reveals the sequence of visual attributes that influenced the model's ' classifica- tion of x' ÿ. It facilitates the creation of precise, tree-structured justifications for these predictions. Importantly, the routing process involves only a few feature similarity computations per node and does not require queries the large language model, resulting in exceptionally fast computation.
3.3 Calibrating through Explaining
The created parse tree offers a two-fold advantage. Not only does it illustrates the logic of a specific prediction, but it also serves by-product to refine the model's predictions by introducing hicrarchical regularization for learned representation. Our goal to use the parse tree as pscudo-labels, embedding this hierarchical knowledge into the model.
To operationalize this, we employ a hierarchical multi-label contrastive loss (HiMulCon) 92 denoted as LHMe, to fine-tune the pre-trained neural network. This approach enhances the model by infusing structured explanations into the learning process, thus enriching the representation.
Specifically, we apply the LVX all training samples. The explanatory path Żř, provides a hierarchical annotation for each training sample xj. The model is trained with both the cross-cntropy loss Cce and LHMC as follows:
M
min
Lcs (f(3)):8) ACHMM (o(x))))
Here, À is weighting coefficient. The explanation 1, is updated every 10 training epochs to ensure its alignment with the network's evolving parameters and learning progress. Notably, the support set isn't used in model training, maintaining fair comparison with the baselines.
4 Experiment
This section offers an in-depth exploration of our evaluation process for the proposed LVX framework and explains how can be utilized to gain insights into the behavior of trained visual recognition model. potentially leading to performance and transparency improvements.

---

Table 1: Data annotation statistics. The indicates the number of video frames. We compare the statistics of category, attributes, image and tree depth across different explanatory datasets. Our dataset stands out as the first hierarchical dataset, offering wide range of attributes.
Clacs
Imaages Arg. Depth Rationaics Hicrarchy Validation Only
AWAZ
51
50
85
57,322
N6A
CUB
200
N/A
11,788
N/A
1,608
VAW [52]
N/A
650
72:274
N/A
COCO Aitr 530
29
196
1800000
N/A
DR-CCIAR-ID +7
68
2.201
DR-CUFR
DRL-InageNat
[7
1,000
5,800
271.016
N/A
X
H-CIFAR-10
289
10,00
4,5
H-CIFAR-100
100
4,5
I1-ImeeeNet
1,000
26,928
50,000
4,8
Cons
Kand
Suttne
TrDec
INX
Tree Kernel Scores
MCS
CLEARIO
CIFARI00
ImageNet
CIFARIO
CIEARI00
ImageeNet
Figure 5: Plausibility comparison on three visual tree parsing benchmarks. We plot the mean-Estd across all networks architectures. For both scores, higher values indicate better performance.
4.1 Experimental Setup
Data Annotation and Collection. To assess explanation plausibility, data must include human annotations. Currently, no large-scale vision dataset with hierarchical annotations is available to facilitate reasoning for visual predictions. To address this, we developed annotations for three recognized benchmarks: CIFARIO, CIFARI00 [39], and ImageNet [57], termed as H- H-CIFAR10, H-CIFAR100, and H H-mmaNeNet. These annotations, detailed in Table & serve ground truth for model evaluation, highlighting our dataset's unique support for hierarchical attributes and diverse visual concepts. Note that, we evaluate hierarchical datasets only, designed for structured explanations.
method is specifically
As an additional outcome of our framework, we have gathered three support sets to facilitate model explanation. In these datasets, each attribute generated by the LLM corresponds to collection of images that showcase the specified visual concepts. These images are either retrieved from Bing search engine [uiin attributes as queries or are generated using Stable-diffusion. We subsequently filter the mismatched pairs with the CLIP model, with the threshold of 0.5. Due to the page limit, extensive details on data collection, false positive removal, limitations, and additional evaluation of user study and on medical data, such as X-ray diagnoses, available in the supplementary material.
Evaluation Metrics. In this paper, we evaluate the quality of our explanation from three perspectives: Plausibility, Faithfulness and Stability.
Plausibility measures how reasonable machine explanation compared to the human explana- tion. We measure this by the graph distance between the predicted and ground-trute truth trees, using two metrics: Maximum Common Subgraph (MCS) [54] 33), and Tree Kernels (TK) [71]. We calculate their normalized scores respectively. Specifically, given predicted tree T pred and the
ground-trume Tgt, the MCS score is computed as |MCS x100 and the TK score is computed as
Vipreed |Ta''
/TK Poree T.... Here, represents the number of nodes in tree, and TK(:))
TK Fx 100
denotes the unnormalized TK score. We report the average score all validation samples.
Faithfulness states that the explanations should reflect the inner working of the model. We introduce Model-induced Sample-Concep Distance (MSCD) to evaluate this, calculated as the
https://www. bing. com/images/

---

Table 2: Stability comparison in CIFARIO un- der input perturbations.
Table 3: Faithfulness comparison by computing the MSCD score. Smaller the better.
Gaussian
Gaussian
Catous
Method Network
0.05)
0.1) DRioks
Network
CIFAR-10
CHFAR-100
ImageNet
MCS
TK MCS
TK
MCS
TK MCS
TK
TrDeerub LVX Tub LVX Tabeer LVX
TrDec
RN-18
100
65.3
86.4
56.2
82.5
65.4
RN-18
0.224
-0.393
-0.971-0.246
-0.446 +0.574-0.298
-0.548
-0.730
RN-18
100
69.7
90.8
62.1
86.5
68.1
88.3
RN-50
-0.236
-0.430
-1.329-0.256
-0.500 -1.170-0.117
-0.588 -1.18
TrDec
RN-50
100
100
68.3
88.5
59.3
84.2 66.2 86.9
LVI
RN-50
100
100 71.9
92.1
65.6 88.3 69.3 90.1
VIT-S
1640.244 -0.467 -1.677-0.266 -0.527 -1,073-0.330 -0.626 -1.992
average of point-to-set distances N, E.Ev Diq,, between all test samples and tree nodes, reflecting the alignment between generated explanation and model's internal logic. The concept is simple: if the explanation tree aligns with the model's internal representation, the MSCD is minimized, indicating high faithfulness.
Stability evaluates the resilience of the explanation graph to minor input variation, expecting minimal variations in explanations. The MCS/TK metrics are used to assess stability by comparing explanations derived from clean and slightly modified inputs. We include 3 perturbations, including Gaussian additive noise with E {0.05, 0.1} and Cutout [88 augmentation.
Baselines. We construct three baselines for comparisons: Constant, using the full category template tree; Random, which selects subtree randomly from the template; and Subtree, choosing the most common subtree in the test set for explanations. Additionally, we consider TrDec Baseline [79]. strategy utilizing a tree-topology RNN decoder on top of image encoder. Given the absence of hierarchical annotations, the CLIP model verifies nodes in the template trees, serving as pseudo-labed for training. We only update the decoder parameters for interpretation purposes. These models provide a basic comparison for the performance of LVX. More details are in the appendix.
For classification performance, we compare LVX-calibrated model with neural-tree based solutions, in- cluding a Decision Tree (DT) trained on the neural network's final layer, DNDF 38), and NBDT (77).
Models to be Explained. Our experiments cover wide range of neural networks, including various convolutional neural networks (CNN) and transformers. These models consist of VGG (66), ResNet 26). DenseNet [29]. GoogLeNet (72), Inceptionv3 (73). MobileNet-v2 [59 and Vision Transformer (ViT) [9]. total, we utilize 12 networks for CIFAR-10, 11 networks for CIFAR-100, and 8 networks for ImageNet. For each model, we perform the tree refinement for 5 iterations.
Calibration Model Training. As described in Section 3.3 we finetune the pre-trained neural net- works with the hicrarchical contrastive loss based on the explanatory results. The model is optimized with SGD for 50 epochs on the training sample, with an initial learning rate f0.001, 0.01, 0.03) and momentum term of 0.9. The weighting factor is set to 0.1. We compare the calibrated models with the original ones in terms of accuracy and explanation faithfulness.
4.2 LLM helps Visual Interprebility
Plausibility Results. We evaluated LVX against human annotations across three datasets, using different architectures, and calculating MCS and TK scores. The results, shown in Figure reveal LVX outperforms baselines, providing superior explanations. Notably, TrDec, even when tramed on CLIP induced labels, fails to generate valid attributes in deeper tree layers-a prevalent issue in long sequence and structure generation tasks. Meanwhile, SubTree lacks adaptability in its explanations, leading to lower scores. More insights are mentioned in the appendix.
ResNet-50<RN-50) and ViT-S, contrasting them with SubTree and TrDec in Table 3 Thanks to the incorporation of tree refinement that explicitly minimizes MSCD, our LVX method consistently surpasses benchmarks, demonstrating lowest MSCD values, indicating its enhanced alignment with model reasoning.
Faithfulness Results. We present the MSCD scores for ResNet-18(RRN-
The stability of our model against minor input perturbations on the CIFAR-10 dataset showcased in Table 2 where MCS/TK are computed. The "Clean" serves as the oracle baseline. Our method, demonstrating robustness to input variations, retains consistent explanation results (MCS>60, TK>85). In contrast, TrDec, dependent on an RNN-parameterizze decoder, exhibits higher sensitivity to feature variations.
Stability Results.
8

---

Fresl
Graastaah
Usee and
(4) Correct predictions. reasonable explanations.
(b) Error predictions.
(c) Noisy amnolations.
but partially correct explanations.
reasonable explanations.
Figure 6: Explanation visualization for ViT-B ImageNet- 1 and means that the node is
aligned or misaligned with the image. Zoom in for better view.
Method
Expl.
CIFARIO
CIFARIOD
ImageNei
Accumac 9% 2
NN
ResNet18
x
94.97%
75.92%
MCS
Viue
DT
ResNet18
93.97%
64.45%
63.45%
DNDF
ResNet18
<
94.32%
67.18%
N/A
ResNet18
94.82%
77.09%
65.27%
LVX (Ours)
ResNet18
95.14%
77.33%
70.28%
Hescline
Collorated
Beselise
Calliirtted
Table Performance comparison of neural de- cision tree-based methods. Expl. stands for whether the prediction is explainable.
Table 5: Performance and interpretability comparison with/without model calibration on CIFAR-100. Higher MCS means better.
Model and Data Diagnosis with Explanation. We visualize the sample explanatory parse tree ImageNet validation set induced by ViT-B in Figure The explanations fall into three categories: (1) correct predictions with explanations, (2) incorrect predictions with explanations, and (3) noisy label predictions with explanations. We' ve also displayed the 5 nearest neighbor node for each case.
What's remarkable about LVX is that, even when the model's prediction is wrong, it can identify correct attributes. For instance, in a case where a "white shark" was misidentified as a "killer whale" (b-Row 2), LVX correctly identified "fins", a shared attribute of both species. Moreover, the misrecognition of the attribute "wide tail flukes" indicates a potential error in the model, that could be later addressed enhance its performance.
Surprisingly, LVX is able to identify certain noisy labels the data, as shown in c-Row 2. In such cases, even experienced human observers might struggle to decide whether "pig bank with band" should be classified "piggy bank" or "band aid". again underscores the superior capabilities of our LVX system in diagnosing the errors beyond model, but also within the data itself.
Calibration Enhances Interpretability and Performance. Our approach involves fine-tuning pre-trained model with the loss function outlined in Section 3.3) using parsed explanatory trees to improve model performance. Table compares the classification performance our model with that of other neural tree methods. Our model clearly outperforms the rest.
Neural tree models often face challenges in balancing interpretability with performance. In contrast, LVX achieves strong performance without relying on a strict decision tree. Instead, decisions are handled by the neural network, with concepts guided by the LLM through Equation 3] This approach enhances the model's ability to disentangle visual concepts while preserving high performance.
In addition, we compared the quality of the generated parsed tree with or without calibration, in Figure [] The calibration process not only improved model performance, but also led to more precise tree predictions, indicating enhanced interpretability. We also test the calibrated model OOD evaluations in Appendix, where we observe notable improvements.
Ablation Study and Analysis
In this section, we present an ablation study on the refinement stage of LVX. We also apply the method to different neural networks to observe variations in model's ' behavior.

---

Table 6: Performance comparison on CIFAR- and CIFAR-100 with and without refinement. Higher MCS and lower MSCD indicate better performance.
Network
Method
CIFAR-10 10
CIFAR 100
MCS
MCS
MSCD
w/o Refine
27.73
0.645
23.18
0.432
ResNet-18
30.24
40,971
25.10
-0.574
Refine
28.09
-0.822
23.44
-0.698
ReaNot-50
LVX
31.09
-1.329
26.90
-1.170
Ablation No Refinement. To study the impact of refinement stage, we present baseline called w/o Refine. In this setup, the initial tree generated by LLMs is kept fixed. We evaluate the method using the MSCD for faithfulness and MCS for plausibility on CIFAR-10 10 and CIFAR- 100 datasets.
The results show in Table 6 that incorporating image model feedback indeed improves tree alignment with the classifier's internal representation, as reflected higher MCS scores. The refined trees also better match numan-annotation
Ablation 2: Refinement Criteria. In original method, tree refinement is based feature similarity to the training set. To explore an alternative, we use average activation magnitude on generated data the criterion for concept familiarity. Concepts with activation magnitudes S 1 pruned. This method, referred to as ActMag, is evaluated on CIFAR- 10. We report the MCS, MSCD for performance, and average
depth as indicator of tree complexity.
Table shows that feature similarity achieves better results than ActMag. Specifically, setting threshold is challenging for ActMag, leading shallow trees (17 = 0.3) or too deep ones (17 0.01).
Table 7: Performance comparison on CIFAR-10 10 with different refinement criteria.
Network
LVX
ActMag(n
0.01)
ActMag(n 0.1)
ActMagín 0.3)
MCS
MSCD
Depth
MCS
MSCD
Depth
MC5
MSCD
Depth
MCS
MSCD
Depth
ResNet-50
-08
25.3
VIT.S
31,9
-1.7
4.3
24.2
.04
6.2
27.4
-0.9
3.3
26.1
-0.6
1.8
Analysis: CNN vs. Transformer. We use our LVX to compare CNN and Transformer models and identify which concepts they miss. We compared ConvNeXI-T (CNN) and DeiT-B (Transformer) on 26,928 concepts we collected on ImageNet, from sub-categories of Concepts, Substances, Attributes, and Environments. We measured accuracy across sub-categories and tree depths.
Results show that ConvNeXt-T is better at local patterns (Attributes, Substances), while DeÎT-B perform better on Environments which needs global semantics. Additionally, DeiT-B is more accurate at shallow depths, whereas ConvNeXt-T performs better at deeper levels. These findings aligns with
earlier research showing that CNN are biased towards textures over shape 23 84
Conclusion
In this study, introduced LVX, an approach for interpreting vision models using tree-structured language explanations without hierarchical annotations. LVX leverages large language models to connect visual attributes with image features, generating comprehensive explanations. We refined attribbte parse trees based on the model's recognition capabilities, creating human-unden undarssnaadtt descriptions. Test samples were routed through the parse tree generate sample-specific rationales. LVX demonstrated effectiveness in interpreting vision models, offering potential for model calibration. Our contributions include proposing LVX the first approach leverage language models for explaining the visual recognition system. We hope this study potentially advances interpretable AI and deepens our understanding of neural networks.
Acknowledgement
This project is supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: NOFF-T2FP20122-0N0 and the National Research Foundation, Singapore, under its Al Singapore Programme (AISG Award No: AISG2-RPP2021-023)
10

---

References
[1] Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Dan Jurafsky. Joyce Chai, Natalic Schluter, and Joel Tetreault, editors, Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 4190-4197, Online, July 2020. Association for Computational Linguistics.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yama Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 22723766-23766, 2022.
[3] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yama Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong. Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikol aj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 2371 237336..... Curran Associates, Inc., 2022.
[4] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Learning to compose neural networks for question answering. arXiv preprint /rXiv:/601.. 01000, 2016.
[5] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39-48, 2016.
cimputerscceene, [6] Philip Bille. A survey on tree edit distance and related problems. Theoretical computer science, 337(1- 3):217-239, 2005.
[7] Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-snli: Natural language inference with natural language explanations. Advances in Neural Information Processing Systems, 31, 2018.
[8] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsuper- vised learning of visual features by contrasting cluster assignments. 2020.
[9] Mathilde Caron. Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in supervidd vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[10] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K Su. This looks like that: deep learning for interpretable image recognition. Advances neural information processing systems, 32, 2019.
[11] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive lcarning of visual representations. In Hal Daumé and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR, 13-18 Jul 2020.
[12] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020.
[13] Xinlei Chen*, Saining Xie*, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint 10X1v:2/04.02057, 2021.
[14] Ying Chen, Feng Mao, Jie Song. Xinchao Wang. Huiqiong Wang, and Mingli Song. Self-born wiring for neural trees. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5047-5056. 2021.
[15] Andy Clark. Whatever next? predictive brains, situated agents, and the future of cognitive science. Behavioral and brain sciences, 36(3):181-204, 2013.
[16] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE transactions on information theory, 13(1):21 22,, 1967.
[17] Mark Craven and Jude Shavlik. Extracting tree-structured representations of trained networks. Advances in neural information processing systems, 8, 1995.
[18] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 1708:02008. 2017.

---

[19] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 2020.
[20] Ingo Feinerer and Kurt Hornik. wordnet: WordNet Interface, 2023. R package version 0.1-16.
[21] Nicholas Frosst and Geoffrey Hinton. Distilling neural network into a soft decision tree. arXiv preprint 10094:1711.0978,, 2017.
[22] Yossi Gandelsman, Alexei A Efros, and Jacob Steinhardt. Interpreting CLIPP's image representation via text-based decomposition. In The Twelfth International Conference on Learning Representations, 2024.
[23] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
[24] Lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.
[25] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. arXiv preprint arXiv:2211.1155, 2022.
[26] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pottern recognition, pages 770-778, 2016.
[27] Lisa Anne Hendricks, Zeynep Akata, Marcus Rohrbach, Jeff Donahue, Bernt Schicle, and Trevor Darrell. Generating visual explanations. In Computer Visi 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 1-14, 2016, Proceedings, Part IV 14, pages 3-19. Springer, 2016.
[28] Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 804-8 813, 2017.
[29] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-440044 4708, 2017.
[30] Yea-Shuan Huang, Cheng-Chir Chiang, Jun-Wei Shieh, and Eric Grimson. Prototype optimization for nearest-neighhor classification. Pattern Recognition, 1237-1555, 2002.
[31] Phillip Isola, Joseph collections. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1383-1391,
Lim, and Edward H Adelson. Discovering states and transformations in image
[32] Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181-214, 1994.
[33] Viggo Kann. On the approximability of the maximum common subgraph problem. In STACS, volume 92, pages 377-388. Citeseer, 1992.
[34] Monish Keswani, Sriranjani Ramakrishnan, Nishant Reddy, and Vineeth N Balasubramanian. Proto2proto: Can you recognize the car, the way i do? In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. pages 00233-10243, 2022.
[35] Jinkyu Kim, Anna Rohrbach, Trevor Darrell, John Canny, and Zeynep Akata, Textual explanations for self-driving vehicles. Proceedings the European Conference on Computer Vision (ECCV), 2018.
[36] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1rXiv:4412.698) 1412.6980, 2014.
[37] Teuvo Kohonen and Teuvo Kohonen. Learning vector quantization. Self-organizing maps, pages 175-189, 1995.
[38] Peter Kontschieder, Madalina Fiterau, Antonio Criminisi, and Samuel Rota Bulo. Deep neural decision forests. Proceedings of the IEEE international conference on computer vision, pages 1467-1475, 2015.
[39] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
12

---

[40] Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[41] Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jieho Luo.. Vqu-e: Explaining, elaborating. and enhancing your answers for visual questions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 552-567, 2018.
[42] Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprien de Masson d' Autume, Phil Blunsom, and Aida Nematzadeh. A systematic investigation of commonsense knowledge in large language models. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pages 11838-1188 1188-11855,22 2022.
[43] Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis. arXiv preprint arXiv:2303. 16434, 2023.
[44] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems. 36, 2024.
[45] Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrenee Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint urXiv:2101.06900 2021.
[46] Scott M Lundberg and Su-In Lee. A unitied approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[47] Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang. Xin Wang, and Carl Vondrick. Doubly right object recognition: A why prompt for visual rationales. arXiv preprint arXiv:2212.06200 2022.
[48] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. arXiv preprint 10.00221, 2022.
[49] Meike Nauta, Ron Van Bree, and Christin Seifert. Neural prototype trees for interpretable fine- -rainrai im- age recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrn Recognition, pages 14933-14943, 2021.
[50] Genevieve Patterson and James Hays. Coco attributes: Attributes for people, animals, and objects. In Computer Vision-ECCV 2016: 14th European Conference, Amstendam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pages 85-100. Springer, 2016.
[51] Chantal Pellegrini, Matthias Keicher, Ege Özsoy, Petra Jiraskova, Rickmer Braren, and Nassir Navab. Xplainer: From x-ray observations to explainable zero-shot diagnosis. arXiv preprint arXiv: 13391, 2023.
[52] Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13018-13028, June 2021.
[53] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748-8766 8766, PMLR, 2021.
[54] John W Raymond and Peter Willett. Maximum common subgraph isomorphism algorithms for the matching of chemical structures. Journal of computer-aided molecular design, 16:521-533, 2002.
[55] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should I trust you?": Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1135-1144, 2016.
[56] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[57] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-2521 2015.
13

---

[58] Dawid Rymarezyk, Lukasz Struski, Jacek Tabor, and Bartosz Zieliáski. Protopshare: Prototypical parts sharing for similarity discovery in interpretable image classification. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1420-1430, 2021.
[59] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov. and Liang Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[60] Makoto Sato and Hiroshi Tsukimoto. Rule extraction from neural networks via decision tree induction. In IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No. 01CH37222). volume 3, pages 1870-1875. IEEE, 2001.
[61] Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv: 04761, 2023.
[62] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision. pages 618-626, 2017.
[63] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving tasks with chatgpt and its friends in huggingface. arXiv preprint (tti::2333//58800 2023.
[64] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagat- ing activation differences. In International conference on machine learning, pages 3145-3153. PMLR, 2017.
[65] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.600 2013.
[66] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 14444099.555 2014.
[67]
Chandan Singh. W. James Murdoch, and Bin Yu. Hierarchical interpretations for neural network predictions. In Intermational Conference on Learning Representations, 2019.
[68] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas. and Martin Wattenberg. Smoothgrad: removing noise hy adding noise. arXiv preprint arXiv:/70m 03825, 2017.
[69] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[70] J-E Stromberg, Jalel Zrida, and Alf Isaksson. Neural trees-using neural nets in tree classifier structure. In Acoustics, Speech, and Signal Processing, IEEE International Conference on, pages 137-140. IEEE Computer Society, 1991.
[71] Jun Sun, Min Zhang, and Chew Lim Tan. Tree sequence kernel for natural language. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 921-926, 2011.
[72] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[73] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 281 8886.8... 2016.
[74] Mohammad Reza Taesiri, Giang Nguyen, and Anh Nguyen. Visual correspondence- explanations improve ai robustness and human-ai team accuracy. Advances in Neural Information Processing Systems. 35:34287-34301. 2022.
[75] Ryutaro Tanno, Kai Arulkumaran, Daniel Alexander, Antonio Criminisi, and Aditya Nori. Adaptive neural trees. In International Conference on Machine Learning, pages 6166-611666 6175. PMLR, 2019.
[76] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-20 1001, Californi Institute of Technology, 2011.
[77] Alvin Wan, Lisa Dunlap, Daniel Ho, Jihan Yin, Scott Lee, Henry Jin, Suzanne Petryk, Sarah Adel Bargal, and Joseph E Gonzalez. Nbdt: neural neural-backed decision trees, arXiv preprint arX1v 2020.
14

---

[78] Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Summers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks weakly-supervised classification and localization common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2097-2106, 2017.
[79] Xinyi Wang, Hieu Pham, Pengcheng Yin, and Graham Neubig. A tree-hased decoder for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4772-4777, 2018
[80] Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medelip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210. 1001133,0 2022.
[81] Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning - compre- hensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251-222 2018.
[82] Wenjia Xu, Yonggin Xian, Jiuniu Wang. Bernt Schicle, and Zeynep Akata. Attribute prototype network for zero-shot learning. Advances in Neural Information Processing Systems, 33:21969 21980, 2020.
183] Xingyi Yang, Jingwen Ye, and Xinchao Wang. Factorizing knowledge in neural networks. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV, pages 73-91. Springer, 2022.
[84] Xingyi Yang. Daquan Zhou. Songhua Liu, Jingwen Ye, and Xinchao Wang. Deep model reassembly. Advances in neural information procesing systems, 35:25739-257555 2022.
[85] Yongxin Yang, Irene Garcia Morillo, and Timothy M Hospedales. Deep neural decision trees. arXiv preprint arXiv: htt::/800.0000 069886 2018.
[86] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison- Burchh and Mark Yatskar. Language in bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19187- 19197, 2023.
[87] Zhengyuan Yang. Linjie Li. Jianfeng Wang. Kevin Lin, Ehsan Azarnasa,, Faisal Ahmed, Zicheng Liu, Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303. 1181, 2023.
[88] Chih-Kuan Yeh, Joon Kim, Ian En-Hsu Yea, and Pradeep K Ravikumar. Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31, 2018.
[89] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018
[90] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, Septemher 6-12, 2014, Proceedings. Part 1 13, pages 818-833. Springer, 2014.
[91] Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski. Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lec, Vincent Vanhoucke, et al. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.0055) 2022.
[92] Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ramaiah. Use all the labels: A hierarchical multi-label contrastive learning framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16660-166660 2022.
[93] Yuhui Zhang, Jeff Z HaoChen, Shih-Cheng Huang, Kuan-Chieh Wang, James Zou, and Serena Yeung. Diagnosing and rectifying vision models using language. In International Conference on Learning Representations (ICLR), 2023.
[94] Qiangfu Zhao. Evolutionary design of neural network tree-integration of decision tree, neural network and ga. In Proceedings of the 200/ Congress on Evolutionary Computation (IEEE Cat. No. 0ITH8546), volume 1, pages 240-244 IEEE, 2001.
[95] Xuhui Zhou, Yue Zhang, Leyang Cui, and Dandan Huang. Evaluating commonsense in pre-trained language models. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9733 9740-997.2220 2020.
15

---

196] Jan Ruben Zilke, Eneldo Loza Mencía, and Frederik Janssen. Deepred-rule extraction from deep neural networks. In Discovery Science: 19th International Conference, DS 2016, Bari, Italy, October 19-21, 2016, Proceedings 19. pages 457-473. Springer, 2016.
A Appendix/ / supplemental material
This document presents supplementary experiments and information regarding our proposed LVX framework. In Section[[] we provide overview of the algorithm pipeline for LVX. In Section[D] we detail the process of data collection and its subsequent analysis. In additionally, Section providde the user study results; Section [ shows the calibrated training further improve the OOD performance. Saation[[]ssowcassse the additional experimental results obtained from a specialized application on X-Ray diagnosis. Furthermore, in Section H we demonstrate the explanation results achieved using self-supervised models. We also provide the raw experimental values in Section [ Finally, we outline the experimental setup, metric definitions, and dataset collection protocols.
B Related Work
Neural Tree. Neural Trees (NTs) intend to harmonize the performance of Neural Networks (NNs)
and interpretability of Decision Trees (DT's
[17
21
60
96
[4] within a unified model. They
evolved from mimicking NNs with DTs 07 21 60
96
14 to becoming inherently interpretable
tree-structured networks, adapting their structure via gradient descent 770 94 32 85|
75 38). Neural-
Backed Decision Trees (NBDTs) 77 use a trained NN as a feature extractor, replacing its final
layer with a decision tree. Our model builds on these advances to create a hierarchical tree from pre-trained NN and provides post-hoc explanations without additional training, which increases interpretability and potentially enhances performance.
Prototype-based Explainable Model. Prototype models use representative training data points to symbolize classes or outcomes [6 30 37.. Revived deep learning and few-shot learning 69 82),
they justify decisions by comparing new instances to key examples [10
88
40 Recent work
has developed this approach through hierarchical and local prototypes [49
34
74]. However, the
prototypes serve as an indirect explanation for model's prediction, necessitating further human justification. Our LVX addresses this by assigning semantic roles to prototypes through LLM, turning them from simple similarity points to data points with clear definitions, thereby enhancing explainability.
Composing Foundation Models. Model composition involves merging machine learning models to address tasks, often using modular models for sub-tasks 5 28
4 X3], restructured by a symbolic executor [89]. Recently, Language Learning Models (LLMs) have been used as central controllers,
or APIs [1]. with language as a universal interface [91]. However, composing language model with non-language ones lacks unified interface for bilateral communication. In this study, we propose the use of a text-to-image API as a medium to enable the language model to share its knowledge with the visual task. This allows the vision model to benefit from the linguistic context and knowledge hicrarchy, thereby enhancing its transparency.
guiding the logic of existing models 63] 25
87
43
C Pseudo-code for LVX
In this section, we present the pseudocode for the LVX framework, encompassing both the construction stage and the test stage. The algorithmic pipelines are outlined in Algorithm I and Algorithm 2
The category- -lveel tree construction pipeline, demonstrated in Algorithm involves iterative process that utilizes a large language model (LLM), like ChatGPT. This process allows us to construct an attribute tree for each category. It begins by generating prompts based on the category name and using them to gather attribute information. This forms the initial tree structure. Support images are collected using rext-to-image API, and their visual features associated with the corresponding tree nodes. The process iterates until the maximum run, continuously refining the attribute tree for corresponding category.
During the test stage, as outlined in Algorithm 2 the test samples undergo a traversal process within the constructed category-level trees. The goal is to locate the subtree that best aligns with the correct
16

---

Algorithm Language Model as Visual Explainer -LoXS)-Contttuctionn
Input: Vision model f = oh, a large language model L, text-to-image API T2I, a training set Drr
[x,, Bi class names C = {ci}i=1 and a concept prompt tree input-output example P.
Output: An explanatory tree T; for each category
1: // Construct the initial Parse Tree
for to n do
3: In-context Prompt LLM: d, = L(c, P).
4: Parse d, into an initial tree T. = (V("), E
5: Collect support images from text-to-image API: (x jin T2I(u). where U E V.
6: Extract features in each node: P, {p. ) {g(x;) (x
8: // Parse Tree Refinement
9:
for = to Lmux do
10: for 1 M do
Extract feature for training data: q = =xx)).
12:
Assign training data to a tree node: v" argmin vEr D(q, Po).
end
14:
Count the number samples for each node: C.-
ΣØ= 11f" 10= argmin ve D(q, Po)]
Prune the least visited node: T7(0) Prune (T.
Grow most visited node: T((++1) Grow (T())
Collect support images from lext-to-image API:
[x:];=1 R T2I(e). where v v((+1))
18:
Extract features in new tree node: P. {p. {5 E (E))
19:
end for
20:
return (((ta)) as Tj
Algorithm 2 Language Model as Visual Explainer (LVX)-Test
Input: Vision model f = 9 a test sample xt, and explanatory trees each category. Output: A explanatory tree T for test sample. I: Prediction on X = (xxxa) and = h(q).
2: Find the top-matched sub-tree in Tg
T argnin
D(q,
s.t. T = ("". """""""""""""""""""""""""
3: return 7 the prediction explanation.
prediction rationales. This sample-wisc attribute tree process enables the identification of pertinent attributes and explanations linked to each test sample, providing insights into the decision-making process of the model.
In summary, the LVX employs an iterative approach construct category- trees, leveraging the knowledge of LLMs. These trees are then utilized during the test stage to extract relevant explanations. This methodology enables to gain understanding the model's decision-making process by revealing the underlying visual attributes and rationales supporting its predictions.
D Data Collection and Analysis
D.1 Annotation Creation
The creation of test annotations for three datasets involved semi-automated approach implemented in two distinct steps. This process was collaborative effort between human annotators, a language model (ChatGPT), and CLIP [53], ensuring both efficiency and reliability.
17

---

1. Concept Tree Creation. In this first step. we utilized ChatGPT to generate initial attribute trees for each class with the category name, detailed Section 3.I.
2. Attribute Verification. To determine whether attribute was present absent in an image, we employed an ensemble of predictions from multiple CLIP [53] models We filtered the top-5 attributes predicted by CLIP and sought human judgments verify their correctness. To streamline this process, we developed an annotation tool with a user interface, which is illustrated in Figure 7
3. Manual Verification. In this phase, annotators examined the accuracy of existing attributes and introduced new relevant ones to enrich the concept trees. Subsequently, human annota- tors conducted thorough review, refinement, and systematic organization of the attribute trees for each class.
Figure 7: Software tool interface for parse tree annotation.
Table 8: Support set Dataset Statistics.
Datasel Name
No. Categories
No. Altribules
No. Images
CIFAR: 10 Support
10
289
14,024
CIFAR-100 Support
100
2,359
19,168
ImageNet Support
1,000
26,928
142,034
D.2 Support data
Data Collection. Our LVX model utilizes custom support set for each task, created through a reject sampling method. Initially, images are generated using either Bing or the Stable Diffusion Model. Subsequently, CLIP is applied to determine if the CLIP score exceeds 0.5, based on the raw cosine similarities calculated by averaging CLIP
14. If the score above specified threshold, the image is retained; otherwise, it is discarded.
To optimize the image collection process, we merge the retrieved images from all models. leading time and effort savings. This approach allows us to reuse an image already in the dataset if it matches an attribute generated by the LLM. As result, after the initial models have finished collecting data, subsequent models can simply pull relevant images from this existing pool instead of collecting new ones, saving both time and effort.
htttp::/ccat..opeeeeee com/
18

---

Chapayy
Olagert
(a) Histogram of Tree Depth.
(b) No. Attribute per Category.
(c) No. Instance per Category.
Nurter
157
Tie: Dopri
(d) Histogram of Tree Depth.
(c) No. Attribute Category.
(f) No. Instance per Category.
Nunked
100
101
(g) Histogram of Tree Depth.
(h) No. Attribute per Category.
(i) No. Instance per Category.
Figure 8: Statistics of the support dataset sets of (Rowl) CIFARIO, (Row2) CIFARIO0 and (Row3) Im- ageNet. We examine the (a,d.g) Tree Depth, (b,e,h) Number of Attributes for each category and (c.f.i) Number of image for each category to demonstrate the diversity of collected attributes and the completeness of hierarchical annotations.
Data Statistics. We present the statistics of the support datasets collected for CIFARIO, CIFARIO0. and ImageNet, highlighting the diversity and comprehensiveness of our dataset. Table & and Figure provide an overview of these statistics. Specifically, we include the number of attributes, the number of samples for each category, the total number of samples in the dataset, as well as the distribution of tree depths. This rich collection of data showcases the diverse range of attributes and categories covered in our dataset, making it a valuable resource for training and evaluation purposes.
D.3 Limitations of Support Dataset Collection
While collecting newly curated dataset can be advantageous for our tasks, it is important to acknowledge certain limitations when using such datasets for explanation purposes. Three key limitations arise: false positive simages, the presence of out-of-distribution samples and potential bias in the dataset.
False Positive Images. We observed that both Bing and the Stable Diffusion model occasionally generate imperfect images from textual descriptions, manifesting incorrect or entangled patterns. For instance, the word "crane" could represent either construction machine or a bird, leading to ambiguity. Furthermore, long tail" could potentially include not only the tail but also the head and legs, reflecting broader scope than intended.
image described as "a dog with
Duttof-Distributton Samples. Newly collected datasets may include samples that out-of-distrine distrbbbbbu,,,n, i.e. they do not align with the source data distribution of interest. These out-of-distribution samples can introduce challenges in generating accurate and reliable explanations. As result, explanations based solely on a newly collected dataset may not generalize well to unseen instances outside the support dataset distribution.
Data Bias. Biases in the collection process or the underlying data sources can inadvertently influence the dataset, leading biased explanations. Biases emerge due to various factors, such as data collection source, or imbalances in attribute distributions. Consequently. relying solely on newly collected dataset for explanations may introduce unintended biases into the interpretation process.
19

---

Solutions. To deal with mistakes in the gathered images, we used two approaches. First, we used the CLIP model sift through the images because it's good understanding how close an image is to text concepts, helping us remove most mixed-up and incorrect images. Second, we manually sorted out words that have more than one meaning. For instance, we made it clear whether "crane" refers to the bird or the machine by labeling it as "crane (bird)" or "crane (machine)".
To mitigate the challenges posed by OOD samples and data bias, adopt cautious approach. Specifically, we do not directly train our models on the newly collected support dataset. Instead, we utilize this dataset solely for the purpose of providing disentangled attributes for explanations. By decoupling the training data from the support data, we aim to reduce the impact of OOD samples and potential data biases, thus promoting more robust and unbiased analysis.
Despite these limitations, our emphasis on obtaining dataset with distinct attributes sharpens our analysis and interpretation of model behavior. This approach allows to extract meaningful insights in controlled and clear way.
E
User Study
The evaluation of visual decision-make making and categorization can be uncertain and subjective, posing challenges assessing the quality of explanations. To address this, we conducted a user study to verify the plausibility of our explanations.
Experiment Setup. Our study compared the performance of LVX, with three others: Subtree, TrDec, and a new baseline called Single. The key difference with the Single baseline is that it utilizes only the nearest neighbor node from the parse tree for its output. This contrasts with LVX, which employs the top-k neighbor nodes from the parse tree.
We recruited 37 participants for this study. Each asked respond to 15 questions, each with one image and 4 choices. In each question, they choose the explanation that best matched the image, based their personal judgment. The format for each choice was arranged as "The image is CELASSNNAMER because <EXPLANATION>..
Results. The user study results, shown in Table 9 clearly indicate the superiority of the LVX method. It was selected by participants 57.66% of the time, a significantly higher rate compared to the other methods included in the study.
Table 9: User Study Results.
Method
Choice Percentage
Subtree
3.78%
TrDec
22.88%
Single
15.68%
LVX
57.66 %
F Experiments on Out-of-Distriuution (OOD) Evaluation
In this section, we evaluate calibrated model's performance in scenarios, focusing on its robustness and ability to generalize. This evaluation is conducted using a ResNet50 and ViT-S trained on ImageNet, with and without calibration training, and tested on the ImageNet-A and ImageNet-Skete datasets.
(OOD)
The results of OOD generalization, quantified by Top-1 Accuracy, are listed in Table 10 For both Re and ViT-S 16 models, we notice significant improvements accuracy ImageNet-A and ImageNet-S compared to the baselines. This enhancement in Dut-o--Distribution generalization confirms the effectiveness of model calibration not only improving in-domain performance (shown in Figure 7 of the main paper) but also in boosting adaptability and robustness to out-of-domain data.
Results.
20

---

Table 10: OOD Generalization Results with and without calibration by Top-1 Accuracy.
Model
ImageNet-A
ImaggNNet-S
BaeliieeCalibbated Baveline/Cathrrteed
Baseline(alibrated
ResNei-50
76.1376654(+044)
18.96/23.32/+44.30
24.10/11.4
VTT-S
(1.8578.21(00.366
13.19/188.2(455333)
32.40/37.11
G Experiments on Chest X-Ray Diagnosis
In this section, we evaluate the performance of our LVX method in security-critical domains, specifi- cally medical image analysis. We train neural networks for chest X-ray diagnosis and utilize LVX to interpret and calibrate the predictions.
We adopted the DenseNet- 121 architecture for disease diagnosis in our study. The model trained on the Chestx-ray 14 dataset [78), which consists of chest X-ray images encompassing 14 diseases, along with an additional "No Finding" class. The DenseNet-121 architecture specifically designed to generate 14 output logits corresponding to the different diseases. During training, we employed a weighted binary cross-entropy loss [78] for each disease category optimize the model.
LVX
Normallizzd MCS
Normalized TK
TED 3 2
Model
Model
Model
Figure 9: Explanation performance comparison on Chestx-ray 14 dataset.
Chambers
Gardee
Piul
Pros: Cansion agly
Chamber
Pred:
Spate lowe
Changea
Truund-Truth
Ground-Truth
Cantiomagaly
Centevennler
(ueg)
Figure 10: Explanation examples for the chest xray diagnosis task.
For optimization, we utilized the Adam optimizer (36 with an initial learning rate of le-4, a weight decay of 1e-5, and batch size of 32. The model underwent training for total of 18 epochs.
Model Explanation. To enhance interpretability. we incorporated our LVX framework into the model. Instead of acquiring images from online sources, gathered the support set directly from the training data. To accomplish this, we utilized a parse tree generated by the ChatGPT language model. Leveraging this parse tree, applied MedCLIP [80] model retrieve the most relevant images for each attribute from the training set. These retrieved images served as our support sets for the LVX framework.
Compared to applying the LVX framework on single-label classification, the Chestx-ray 14 dataset poses multi-label classification challenge. In this dataset, each sample can belong to multiple disease categories simultaneously. Therefore, we modified the LVX framework to accommodate and handle the multi-label nature of the classification task.
Specifically, for each input image x, we predict label y = f(x E (0, 1314, create the visual parse tree, we begin by establishing the root node. If all elements of ÿ are 0, the root node is set to "No Findings". Conversely, if any element of ý is non-zero, the root node is labeled as "has F Findings". For each positive finding, construct separate parse tree, with these sub-trees
21

---

becoming the children nodes of the root. By combining these sub-trees, we obtain comprehensive and coherent explanation for the image. This modification enables us to effectively handle the multi-label nature of the classification task, providing meaningful and interpretable explanations for images with multiple positive findings.
To establish the ground-truth explanation label, we adopt a MedCLIP 80) model to filter the top-5 attributes for each positive finding of the image. These attributes are then organized into tree structure. This approach serves as an automatic explanation ground-truth, thereby eliminating the requirement for manual annotations from domain experts.
In addition to providing explanations, we aim to calibrate the model predictions with the parse tree. To achieve this, apply modified hierarchical contrastive loss individually each finding. We then calculate the average of these losses, which serves as our overall loss term. We thus fine-tune the model for 3 epochs using the hieracical term and weighted cross-entropy.
Explanation Results. We compare the explanation performance our proposed LVX against the Random and Constant baselines. The numerical results, depicted in Figure 9 highlight the superiority our LVX approach.
Additionally, we showcase the parsed visual tree in Figure I0 to provide clearer illistration of our results. Notably, our approach effectively interprets the decision-maie -making process of black-box neural networks. For instance, in the case on the right, our method accurately identifies the presence of visible fluid the lung space and establishes its relevance the model's prediction. Consequently, LVX enables clinical professionals to make well-informed justifications for their patients, enhancing the overall decision-making process.
Calibration Results. Table I presents the comparison between the baseline models and the model with calibration, measured in terms of the Area Under the Curve (AUC) score for each disease type. The AUC score provides a measure of the model's is ability to discriminate between positive and negative cases.
The calibrated model shows notable improvements in several disease types compared to the baseline. Notably, Hernia demonstrates the most significant improvement, with an AUC score of 0.936 compared to 0.914 for the baseline. This indicates that the calibration process has enhanced the model's ability to accurately detect Hernia cases.
In summary, the LVX method markedly improves model accuracy, demonstrated by enhanced cali- bration performance across different disease types. The integration of visual attributes boosts both accuracy and reliability of predictions, leading to better diagnostic results. These findings underscore the LVX method's potential to elevate model performance, particularly in medical diagnostics.
Table 11: Model performance with and without calibration. AUC scores are reported for each disease type. Avg. indicates the average score.
Finding
Baseline
LVX
Atelectasis
0.767
0.779
Consolidation
0.747
0.755
Infiltration
0.683
0.698
Pneumothorax
0.865
0.873
Edema
0.845
0.851
Emphysema
0.919
0.930
Fibrosis
0.832
0.830
Effusion
0.826
0.831
Pneumonia
0.721
0.719
Pleural Thickening
0.784
0.793
Cardiomegaly
0.890
0.894
Nodule
0.758
0.776
Mass
0.814
0.830
Hernia
0.914
0.936
Avg.
0.812
0.821
22

---

H Experiments on Self-supervised models
In this section, we focus on self-supervised models to assess their interpretability. Differing from supervised models, self-supervised models develop representations without labeled data. Our aim to understand the interpretability of these representations and uncover the underlying structures derived from the input data alone.
Model be Explained. Our objective is offer comprehensive explanation for self-supervised models trained on ImageNet-1k: These models include ResNer50 trained using SimCLR [I]. BYOL 24 AwAV [8) MoCov3 [2], DINO (9] and ViT-S trained with MoCov3 [3] and DINO 9
The networks are subsequently fine-tuned through linear probing while keeping the backbone fixed. We then utilize our LVX provide explanations for their predictions. Additionally, we compare these self-supervised models with their supervised counterparts to highlight the differences in representation between the two approaches.
Numerical Results. Table 12 presents the results self-supervised models. Our analysis reveals a strong correlation between the explanatory performance and the overall model accuracy.
However, we also noticed that self-supervised models based on transformer architecture exhibit greater attribute disentanglement compared to supervised models, despite potentially having slightly lower performance. This phenomenon is evident when comparing the DINO ViT-S/16 model and the supervised ViT-S/16 model within the context of tree parsing explanation. Although the DINO ViT-S/16 model shows slightly lower overall performance, it outperforms the supervised model in terms of providing accurate attribute explanation.
These results underscore the potential benefits of self-suupe learning uncovering meaningful visual attributes without explicit supervision. While self- stff---ueeiieed models may exhibit marginally lower performance on certain tasks, their ability to capture rich visual representations and attribute disentanglement highlights their value in understanding complex visual data.
Table 12: Explanation performance analysis self-supervised models utilizing linear probing.
Method
Top-1 I Acc
TED
MCS
TK
ResNet50-SimcI
69.2
9.38
23.72
46.53
ResNet50-BY05
71.8
9.29
24.66
48.29
M.ANet5--Mooor
74.6
9.19
25.59
50.17
ResNe150-DING
75.3
9.14
25.77
50.71
ResNet50-SWAV
75.3
9.15
25.82
50.69
ResNet50-Sup
76.1
9.09
25.99
$1.19
ViT-S/16-NA -MCCo33
73.2
9.16
25.25
49 49.40
ViT-S/16-DING
77.0
8.99
26.61
$2.05
ViT-S/8-DINO
79.7
8.89
27.62
53.95
ViT S/T6
77.9
9.10
25.73
50.34
I Raw Results
This section presents the raw numerical results for Figure 5, as depicted in the main paper. Specifically, Table [3] provides the results for CIFAR-10, Table for CIFAR-100, and T5 ImageNet. We also observed that larger networks within the same model family deliver better results. As the models improve, so does the accuracy the explanations, suggesting that larger networks facilitate more effective explanations. This is demonstrated by the increase in MCS and TK scores as ResNet deepens on CIFAR-I00 and ImageNet, aligning with the general belief that larger neural networks offer enhanced generalization and structural representation capabilities.
J Experimental Setup
In this section, we provide detailed information about our experimental setup to ensure the repro- ducibility of our method.
23

---

Table 13: Explanation performance comparison CIFAR-10.
Model
TED1
MCSt const.
Tree Kernel
rand.
LVX
rand.
LVX
rand.
LVX
VGG13
9.21
32.97
8.21
28.98
18.77
32.31
59,49
58.41
63.43
VGG16
9.23
32.89
8.14
29.11
19.11
32.55
59.34
59.55
63.79
VGG19
9.15
32.85
30.67
19.10
31.78
59.44
59.39
63.32
ResNet18
9.21
32,90
8.52
28.95
18.92
30.24
58.94
58.87
61.19
ResNet34
9.21
32.92
8.16
28.95
18.98
32.07
59.06
59.01
63.27
ResNet50
9.21
32.89
8.44
28.96
19.00
31.09
59.16
59.21
62.06
DenseNet121
9.19
32.89
29.09
19.11
32.13
59.53
63.44
DenseNct161
9.20
32.88
8.19
29.07
19.11
32.12
59.35
59,48
63.73
DenseNct169
9.21
32.88
8.18
29.25
19.08
32.13
59.52
59,46
63,46
MobileNet_w v2
9.20
32.89
8.41
29.24
19.09
31.61
59.53
59.38
61.87
GoogLeNett
9.23
32.96
8.41
28.66
18.86
30.75
58.62
58.71
61.38
Inception
9.20
32.89
29.19
19.03
31.02
59.37
59.27
61.85
Table 14: Explanation performance comparison on CIFAR-100.
Model
TED
MCSt const.
Tree Kernel
rand.
const.
LVX
rand.
LVX
rand.
const.
LVX
ResNer20
9.61
28.77
8.96
22.65
17.60
24.89
45.52
46.96
47.70
ResNet32
9.57
28.67
8.86
22.84
17.92
25.39
46.45
47.87
48.58
ResNet44
9.54
28.60
8.81
23.48
18.34
25.97
47.42
48.87
49.79
ResNet56
28.54
23.87
18.60
26.47
48.04
49.58
50.17
MBNy2-x0.5
9.55
28.58
8.87
23.43
18.19
25.50
47.13
48.58
49.08
MBNv2-x0.75
9.43
28.43
8.76
24.47
18.87
26.71
49.19
50.55
51.21
MBNv2-x1.0
9.48
28.35
8.73
24.35
19.02
27.09
49.27
50.68
51.47
MBNv2-x1.4
28.16
24.87
19.47
27.41
50.52
52.08
52.91
RepVGG AO
9.44
28.28
8.74
24.65
19.21
26.84
49.78
51.37
52.01
RepVGG A1
9.42
28.21
8.72
25.27
19.59
27.43
50.63
52.17
52.81
RepVGG A2
9.40
28.08
8.70
25.46
19.82
27.99
51.26
52.87
53.04
J.1 Evaluation Metrics
To evaluate the effectiveness of our proposed tree parsing task, we have developed three metrics that leverage conventional tree similarity and distance measurement techniques.
Tree Kernels (TK): Tree Kernels (TK) evaluate tree similarity by leveraging shared sub- structures, assigning higher scores to trees with common subtrees or substructures. To enhance the match, we set the decaying factor for two adjacent tree layers to 0.5, where larger values lead to better matches. Let's define the subtree kemel mathematically:

---

Table 15: Explanation performance comparison ImageNet.
Model
TED4 const.
MCSt const.
Tree Kernelt
rand.
LVX
rand.
LVX
rand.
LVX
ResNet18
9.83
34.15
9.30
22.52
16.82
23.87
45.32
44.85
46.85
ResNer34
9.75
33.78
9.17
23.74
17.71
25.09
47.66
47.16
49.24
ResNer50
9.68
33.58
9.09
24.59
18.35
25.99
49.38
48.97
51.19
R0sNet101
9.64
33.48
9.04
24.94
18.66
26.51
50.25
49.77
51.99
VIT-T16
10.42
35.44
9,99
15.07
11.25
15.91
30.30
29.92
31.24
VIT-S16
9.69
33.61
9.10
24.16
18,01
25.73
48.53
48.05
50.34
ViT-B16
33.37
8.99
25.20
18.79
50.64
50.22
$2.76
ViT-L16
9.45
32.84
8.79
27.27
20.35
29.29
54.83
54.36
57.14
Tree Kernel: Given two trees, Ti and T2, represented as rooted, labeled, and ordered trees, we define the subtree kernel as follows:
TK(T1 ΣΣ B(T.T') x O(T,T") x (mxx(dpph(()) T T
Let (Ki T2) denote the similarity between subtrees Ti and T22 using the subtree kernel. Additionally, let O(T, T") represent the count of shared common subtrees between trees T and T". Furthermore, let and r' be the roots of T and respectively. À < 1.0 the decaying factor that make sure the tree closer to the root hold greater significance.
The ((T, 7") is computed recursively as follows:
1. If both T and T' are leaf nodes, then (TT, T') = 1 if the labels of T and T are the same, and otherwise.
2. If either T or T" is a leaf node, then O(T,T") = O.
3. Otherwise, let (Ti, T2.... Tn) be the child subtrees of T, and be the child subtrees T".
(T.:Ty....,,
- If the labels of and r' are the same, then 0(T, T") is the sum of the products of O(T,, T7) for all combinations of i and j. where ranges from to n and j ranges from to n'.
If the labels of and r' are different, then 0(T, is 0. Additionally, if T and T' are isomorphic (have the same structure), then OTT, T") is incremented by 1.
In the paper, the Tree Kernel (TK) score is normalized to accommodate trees of different (K xx10 sizes. The normalized TK score is computed /TK [T KKT,,, The kemnll value serves as measure of similarity, where higher values indicate greater similarity.
Maximum Common Subgraph (MCS) 54
The Maximum Common Subgraph (MCS) identifies the largest shared subgraph between two trees, measuring the similarity and overlap of their hierarchical structures. Here's the mathematical definition of the Maximum Common Subgraph:
25

---

Maximum Common Subgraph: Given trees, 7i (V1, and 72 (V2, F2), where V and 1/2 are the sets of vertices and E1 and Es are the sets of edges for each graph, respectively, define the Maximum Common Subgraph as:
I2) (VMcs, EMCs) maximize subject to
|Masl
VMCS E V1 and VMCS S V/2
EMCS S E1 and EMCS C E2
For any pair of vertices v in VMCs, if (u, v) is an edge in EMCS, then (u, v) is an edge in Ti and T2, and vice versa.
In our paper, we report the normalized MCS score as our measurement of tree similarity a where a higher score indicates greater similarity between the graphs.
Here, represents the number of nodes in a tree. We employ this normalization to address the scenario where one tree is significantly larger and encompasses all other trees as subtrees. By dividing the MCS score by the square root of the product the numbers of nodes in the predicted tree (T prea) and the ground truth tree (T we ensure a fair comparison across trees of varying sizes.
Tree Edit Distance (TED) 6 The Tree Edit Distance (TED) quantifies the minimum
number of editing operations required to transform one hierarchical tree into another. It mea- sures the structural dissimilarity between trees by considering node and edge modifications, insertions, and deletions. With smaller TED, the two graphs are more similar. Let's define the Tree Edit Distance formally:
26

---

Tree Edit Distance: Given two trees Ti and 72, represented as rooted, la- beled, and ordered trees, we define the Tree Edit Distance between them as TED
The TE, I2) is computed recursively as follows:
- both Ti and T2 are empty trees, i.e. they do not have any nodes, then TED(Ti, T2) = 0.
If either or T2 an empty tree, i.e., does not have any nodes, then TED T2) is the number of nodes in the non-empty tree.
Otherwise, let and 12 be the roots of T and T2, respectively. Let Ti and Ty be the subtrees obtained by removing the roots and 72 from T2 and T2, respectively.
-
* If = T2, then TED is the minimum among the following values:
TED(Ti, T.) + TED(children of ri, children of r2), where (ED(childeen of ri, children of r2) the TED computed recur- sively between the children of rı and r2-
1 TED(Tİ, T2), which represents the cost of deleting the root ri and recursively computing TED between 7 and Tz-
TED(TI, T2), which represents the cost of inserting the root 72 into Ti and recursively computing TED between T₁ and Tz-
1+ TED(T T2), which represents the cost of deleting the root Ti and inserting the root r2 into T2, and recursively computing TED between T' and T2.
If r1 # 12, then TED(TI, T2) is the minimum among the following values:
TTD(TT Ty) + (childildene of r1, children of r2), where (ED(children of 71, children of r2) is the TED computed recur- sively between the children of r1 and r2-
+ TED(TTT (2), which represents the cost of deleting the root rı and recursively computing TED between T and T2.
+ TED T2), which represents the cost of inserting the root 72 into Tj and recursively computing TED between Tj and Tz
The TED(TI,T)) the final result obtained after applying the above recursive computation.
J.2
Model Checkpoints
For our experiments, we utilize publicly available pre-trained models. Specifically, we employ CI- FAR10 models from httpt:://tttptttuuuue com http:////ww.a.aaaa////y//r..ttta from httpt::tt//tiihii
CIFAR100 models and ImageNet models from
com/ comm/hheyyoffo/yyoororroo citprrhhhcifa---mmeeleee aateryaooo//poooccc-afafa--aaeed torchvision package and timm package. The sule-supe models are downloaded from their respective official repositories.
J.3 Explanation Basclines
To explain visual models using tree-structured language without annotations, we devised four basic baselines, Random, Constant, Subtree and TrDec, for comparison with our method.
Random Baseline. The Random baseline generates random explanations by predicting an image's category and randomly sampling nodes from category-specift tree. The connected nodes form trec-structured explanation, providing performance baseline for random guessing.
Constant Baseline. The Constant baseline Produces a fixed tree-structured clue for images of the same class, using an initial explanatory tree TT(() as template. This baseline assesses LVX against a non-adaptive, static approach.
27

---

Subtree Baseline. This method involves selecting the most common subtree from the test set for explanations, testing the efficacy of using frequent dataset patterns for generic explanations.
TreDec Baseline. Based on [79], the TrDec strategy implements a tree-topology RNN decoder over image encoder. In the absence of hierarchical annotations, this baseline uses the CLIP model to verify nodes in the template trees, which act as pseudo-labels for training. This method focuses on the effectiveness of structured decoding process explanation generation.
This comparison demonstrates the effectiveness of LVX in creating explanations tailored individual image content, clearly outperforming methods based random guessing, static templates, and basic learning-baa based approaches.
K
Limitations
Our system, LVX, depends external Large Language Model (LLM) to provide textual explana- tions. While this integration adds significant functionality, also introduces the risk of inaccuracies. The LLM may not always deliver correct information, leading potential misinformation or erro- neous explanations.
Additionally, approach involves generating explanations based the last embedding layer of the neural network. This method overlooks the comprehensive, multi-level hierarchical structure of deep features, potentially simplifying omitting important contextual data that could enhance the understanding of the network's decisions.
28

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
we present LVX to explain vision model with LLM's help.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the limitation section.
29

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
Justification: No proof made.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Details and code provided.
Guidelines:
The answer NA means that the paper does not include experiments.
30

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code has been uploaded as supplementary material. Data will be available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 /ttppp.cc/pbllc/guudeeee http://oddeubbiisnooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
31

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:
[Yes] Justification: All details mentioned.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Errorbar presented.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
All discussed in the main paper.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
32

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttgs///eerrrss..pppbbbbcbiiiie
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Error bar included.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: We provide new annotation for an existing dataset.
Guidelines:
The answer NA means that the paper poses such risks.
33

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, all properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
Code and data are well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [No]
Justification: Not applicable
34

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Not applicable
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
35


Reviews:
[{"review_id": "um6bGPyygQ", "reviewer": "Reviewer_U6GY", "summary": "This work proposes a method to understand the prediction of an image classification  model using a tree of attributes. The tree of attributes is originally constructed in a text-only by querying gpt for identifying attributes of given concepts. They then use an image-to-test model (or retrieval model) to obtain a set of images corresponding to each attribute. At inference time, they match the input image to each attribute at the current tree level. The path from the root node to the leaf node is meant to interpret the models decision making process.", "questions": "Faithfulness Score: Can you explain more directly how this score measures the inner workings of the model? \n\nFigure 3 uses the wrong form of “dear”/“deer”", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "This method works on models that are not just VLM models.  Similar works rely on models being open vocabulary so that the image can be compared to arbitrary attributes described in text. This method first collects the attribute in text and then converts them to image embeddings so that models can be evaluated even if their image embedding space is not aligned with a text embedding space. \n\nThe method is clearly described.", "weaknesses": "The primary weakness of this work is that the way the tree is constructed is not affected by the image classification model at all but the explanation for the inner working of this model must be a path that exists in the tree. This then presupposes that the tree contains the same reasoning path used by the image classification model which seems to be a large assumption. For example, the model may identify a certain kind of bird by some spurious correlation in the training data. However, as the tree is created by a language model which does not know about this data, this is very unlikely to exist in the tree. However, this method will output some sort of path which is meant to be the explanation even though it is impossible for the tree to produce the correct explanation in this case.\n\nThe evaluation metric for measuring the accuracy of the explanation seems poorly defined (see questions sections).\n\nHuman eval doesn’t make sense in this case as humans cant evaluate how well a given explanation corresponds to the true decision making process of the model. (they can only measure how well it corresponds to a process they may have used)\n\nSome motivation for this method seems unfounded. For example, the authors state that “Upon observing a dog, we naturally check for the visibility of its tail.” I think this is not correct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate R-U6GY's thoughtful comments and suggestions. We answer the questions below and will incorporate them into our revised version.\n\n`>>> Q1`**Tree Construction not Affected by Image Model**\n\n`>>> A1`Thank R-U6GY for the insightful comment. In fact, the tree construction is indeed influenced by image models. *They provide valuable feedback to the LLMs to refine the trees*. \n\nAs described in the *Tree Refinement Via Refine Prompt* in `Sec 3.1`, the initial tree is updated to better align with the image model’s feature spaces. In this way, LLM and image model cooperatively build those trees; the final tree should accurately reflects the vision model's internal workings.\n\n**What if Image Models Don't Provide Feedback for Refinement?**\n\nWe conduct an experiment to show the importance of image models' feedback in tree construction. We introduce a new baseline called **w/o Refine**. In this setup, the initial tree created by the LLM is fixed. We measure the _faithfulness_ using the MSCD score and _plausibility_ using MCS on CIFAR-10 and CIFAR-100 datasets.\n\n|Network | CIFAR-10 (MCS&#8593;/MSCD&#8595;) |  | CIFAR-100 (MCS&#8593;/MSCD&#8595;)| |\n|-|-|-|-|-|\n|| *W/o Refine*| **LVX**| *W/o Refine*| **LVX** |\n|ResNet-18 | 27.73/-0.645|**30.24/-0.971** | 23.18/-0.432 |**25.10/-0.574**|\n|ResNet-50| 28.09/-0.822 |**31.09/-1.329**| 23.44/-0.698 | **26.90/-1.170**|\n\nThese results show that feedback from the image model makes trees better reflect the classifier's internal representation. They also align more closely with human-annotated decision paths.\n\n`>>> Q2`**Presupposition of Reasoning Path**\n\n`>>> A2`We truly appreciate the question. We do not assume that the initial tree created by the LLM contains the reasoning path. Instead, we make the tree to achieve this goal; we keep refining the tree to make sure it only includes sub-trees that truly represent the reasoning path. The vision model filters out incorrect nodes. This process is achieved through iterative *tree refinement*.\n\nTo support our claim, we conducted experiments using **w/o Refine**, as detailed in `Q1`. If we presupposed LLM is sufficient to generate reasoning path and not refining the trees, the explanation's faithfulness and plausibility scores dropped significantly.\n\n`>>> Q3`**Concepts that cannot be generated by LLM**\n\n`>>> A3`Great question. We acknowledge that LLMs may fail to generate hard cases, like spurious correlations. However, we intentionally ensure that the generated explanations match the distribution learned by the LLM. It offers two key benefits:\n\n1.  **Coherence to Humans.** The trees are more understandable to humans because LLMs are trained on real-world facts.\n2. **Efficiency.** LVX searches for the best explanations in a vast space of language. Using LLM-generated text narrows this down, helping us find good explanations quickly.\n\nBesides, tree refinement also help to reduce failures. LLMs get to know about the data, through feedbacks from the image model.\n\nWe will incorporate this discussion in our revision.\n\n`>>> Q4`**Evaluation Metrics Definition**\n\n`>>> A4`Sorry for the confusion. We have briefly described metrics in the main paper `Sec 4.1` and provided the formal definitions in `Sec I.1` in appendix. As suggested, we will revise these definitions to include more intuitive descriptions.\n\n`>>> Q4`**Human Evaluation**\n\n`>>> A4`Thanks. Human evaluation indeed makes sense, for assessing *plausibility*, not *faithfulness*. It measures how well explanations align with human logic, as mentioned in `Line 613`. Besides, human studies are common method to evaluate explainable methods [6,11].\n\nFaithfulness is what the reviewer referring to. It checks how explanations match the model's decision-making process. For this, we use *Model-induced Sample-Concept Distance* (MSCD), as defined in `Sec 4.1`, with results in `Table 3`.\n\nWe evaluate from different perspectives to ensure a thorough evaluation.\n\n`>>> Q5`**Motivation for Prediction-Correction**\n\n`>>> A5`Thanks for the question. Our motivation is founded on **Predictive Coding Theory** [A] in cognitive science. \n\n**Predictive Coding Theory:** The theory suggests that our brain keeps a mental model and makes predictions about what we see. When we see something, the brain compares the actual input with its predictions. If there are differences, or prediction errors, the brain updates its model.  It is what happens both functionally and biologically in the brain.\n\n**Connecting example to Theory:** In the pointed case, we use top-down predictive coding [B]. When we see a `dog`, we expect certain features. If we notice some unexpected features, like `a hairless tail`, our brain updates its understanding of what a `dog` looks like.\n\nAs suggested, we will incorporate those cognitive science motivation in our final version.\n\n[A] \"Whatever next? Predictive brains, situated agents, and the future of cognitive science.\" Behavioral and brain sciences\n\n[B] \"Predictive coding in the visual cortex: A functional interpretation of some extra-classical receptive-field effects\". Nature Neuroscience.\n\n`>>> Q6`**Intuition behind Faithfulness Score**\n\n`>>> A6`Thanks for the question. To be specific, faithfulness score measures **whether the generated tree assigns a test sample to the correct prototypes**. \n\nGiven embedding $q_j$ and its explanation $T$, a low faithfulness score $\\sum_{v\\in V} D(q_j, P_v)$ means the sample is semantically closer to the prototype sets $V$ identified by $T$. It indicates that the network recognizes $q_j$ by assigning it to attributes in $T$. \n\nConversely, when $D(q_j, P_v)$ is large, it means the explanation fails to recover this assignment and does not reflect the model's decision. We will incorporate this discussion in final version.\n\n`>>> Q7`**Typos Corrections**\n\n`>>> A7`Thank the reviewer for the careful proofreading and we are truly sorry for the typos. As advised, we will fix the them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "klOI8dUMu1", "reviewer": "Reviewer_UiCA", "summary": "The paper suggests a new approach to building a hierarchical tree of visual attributes (represented with language) that matches the decision-making mechanisms of classifiers. The approach is based on using an LLM to suggest a tree of attributes that are related to a specific class, and then refine the tree according to its matches to the training date (this is done by comparing image embeddings of per attribute synthetic generated images and the train images). The hierarchical explanations are then evaluated using three measures (defined by the authors): plausibility, faithfulness, and stability. The tree explanations are shown to be beneficial when fine-tuning the classifier to match better the found structure, which slightly improves performance. The authors also exemplified the usage of the explanation for characterizing miss classification.", "questions": "see weaknesses", "limitations": "limitations are discussed in the appendix", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The idea of constructing a language description tree of a classifier is great! (although I'm afraid the method mainly visualizes the training images and not the network operations, see comments in weaknesses section)\n\n* The use of LLMs to generate and refine explanations inssures the generation of human-like descriptions, wich are human understandable. I liked the idea of refining the tree using the LLM.", "weaknesses": "* My main concern is that unintentionally *the suggested tree representation mainly explains the training set, and not the network mechanism* (in a very complicated way):\n\n(1) Solely using the last layer of the net for the image embedding: Classifiers are known to have a hierarchical structure of attribute representations that are combined to make the classification decision. By looking only at the last layer embedding the author completely disregards that, and the tree representation constructed does not really explain the inner mechanism of the network but only the mechanism of one particular layer.\n\n(2) The refinement stage is made on the training samples; This means that relevant class attribution can be directly extracted by characterizing the leading attribution of the training data, with no connection to the network mechanism itself or any of it embeddings, without any need to synthesize synthetic data. \n\nThis stage, combined with the fact that the image representations are extracted only from the last layer, makes the tree representation mainly visualize the hierarchy of the training set not the hierarchy of the operation of the network. \n\n\n* My second main concern is *the usability of this type of visual explanation*. The author provided some experiments for potential applications but these seem limited; Finetuning classifiers to better obey the representation only marginally improves performance compared to baselines (no baseline performance of the model before fine-tuning was provided). Using the tree for misclassification examination seems only anecdotal with no large-scale explanation or evaluation. Can the trees replace the classifiers and be used to classify images? Is it possible to compose the tree to get a model lever tree?\n\n\n\n* Does the explanation tree go beyond the structure of the hierarchy provided in context? I can imagine other important attributes, like the presence of singular objects vs many objects, or different types of conditions like the presence of a dog but only if it is leashed. According to the text, it seems like the “subjects” of each node are fixed according to the in-context example, which significantly limits the flexibility of the representation. Furthermore- how was the format in-context example decided? it seems to have a lot of influence on the results but that was not discussed in the paper.\n\n* In the tree refinement stage, there is no definition for what does it to have “nodes that are seldom visited”, this seems to be an important variable. what is the criterion for that?\n\n* Evaluation- \nThe faithfulness score seems to directly connect with the way that the tree was defined.\nBaseline comparisons are rather ablation study.\n\n\n* The clarity of text and figures can be improved. Intro is very general and some definitions can be interpreted in many ways, for example, it will be much clearer if the authors state that by in vision model they mean a classifier. Another example is defining the image embedding way before using it which makes it hard to follow details. Also unclear how L_{HMC} is calculated.\nThe scheme in Fig. 1 was unclear to me. fiuger 5- how were accuracy and MCS calculated here?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "`>>> Q1`**Explaining Network Mechanism**\n\n`>>> A1`Thanks for the question. Actually, our method explains the network mechanism **by identifying prototype sample and concept**. \n\nUnlike *mechanistic interpretability*, which maps concepts to layers or neurons, our LVX uses `prototype-based explanations`[9,10]. We explain the predictions by finding similar samples. Both methods, prototype or mechanistic, explain the network's mechanism, but from different angles.\n\n`>>> Q2`**Last Layer**\n\n`>>> A2`Great question! We focus on the last layer because:\n1. **Explains Misclassifications:** The last layer's feature directly relate to class probabilities, showing why errors occur.\n2. **Common Practice:** Using last layer is standard in prototype-based explanation[9,10]. It makes LVX comparable to methods like NBDT[10].\n\nWe value the suggestion and will explore more on multi-layer method. \n\n`>>> Q3`**Refinement with Train Set**\n\n`>>> A3`Thanks. Our goal isn't to visualize train set; rather, we use it to quantify how well models understand concepts. Both train set and generated data are essential.\n\nGiven generated data with concept `C`, we pass data to models to test their familiarity with `C`. We do this by comparing feature to the train set; high similarity indicates familiarity. \n\nLeading concept in train set isn't always included in the tree. It is included only when the model assign it to generated data.\n\n**Remove TrainSet:**\n\nInstead, we use `average activation magnitude` on generated data as a familiarity measure. Concepts with magnitudes $<\\eta$ are pruned. This method, **W/o TrainSet**, is tested on CIFAR-10 using MCS, MSCD score, and average tree depth.\n|MCS/MSCD/Depth|LVX||W/o TrainSet|||\n|-|-|-|-|-|-|\n||||$\\eta=0.01$|$\\eta=0.1$|$\\eta=0.3$\n|RN50|**31.1/-1.3**/4.2||23.4/-0.3/6.0|26.9/-0.8/3.7|25.3/-0.5/1.4\n|ViT-S|**31.9/-1.7**/4.3||24.2/-0.4/6.2|27.4/-0.9/3.3|26.1/-0.6/1.8\n\nWithout train set, setting a threshold is challenging, leading to trees that are too shallow ($\\eta=0.3$) or too deep ($\\eta=0.01$).\n\n`>>> Q4`**Use Case**\n\n`>>> A4`Grad to see the question. We care a lot about utility.\n\n**1) Finetuning**\n\nFinetuning indeed leads to good improvements. \n1. **In-domain Results:** The baselines before fine-tune have been shown in `Table 4, NN` and in `Table 5, Baseline`. On CIFAR10/100, accuracy improves by 0.5%–2%.\n2. **OOD Results:** Fine-tuning improves OOD results **by 5% on ImageNet-A and -S** (`Table 8` Appendix). \n\n**2) Large-scale Evaluations**\n\nAs advised, we run experiments to see the models' common issue and difference.\n\n**Exp1: Common Issues**\n\nWe examine common errors in the trees. In rebuttal PDF, `Fig 12` lists the top correct and misclassified attributes, while `Fig 13` plots their numbers at each tree depth.\n             \nClassifiers show a **coarse bias**. They recognize `Attributes` like *color* well but struggle with `Substances` based on *shape* and *size*. They also handle shallow attributes better than deep, fine-grained ones.\n\n**Exp2: CNN vs Transformer**\n\nWe compare CNNs and Transformers through accuracy differences. We use DeiT-B and ConvNeXt-T due to their similar accuracy.\n\nCNNs excel at local patterns like `Attributes`, while Transformers are better on `Environments`, focusing on contexts. This supports findings that **CNN are biased towards textures over shape**[A]. We will include this insights in the revision.\n|Model|Concepts|Substances|Attributes|Environments|\n|-|-|-|-|-|\n|ConvNeXt-T|*23.1%*|*18.9%*|**45.3%**|18.1%|\n|DeiT-B|22.0%|15.6%|35.7%|**26.3%**|\n\n[A]ImageNet-trained CNNs are biased towards texture,ICLR2019\n\n**3) Tree as Classifier**\n\nYes, we can use trees as a new classifier called **TreeNN**. For each input, we encode it into a feature, which navigates each tree to compute its MSCD score(`Sec 4.1`). The class with the lowest score is selected.\n\nIn a CIFAR10 experiment, TreeNN gets 92.7% accuracy with an explainable decision path.\n||NN|LVX fine-tuned|TreeNN\n|-|-|-|-|\n|RN18|93.1%|**93.6%**|92.7%|\n\n**4) Compose Trees**\n\nGreat question! We can merge category-level trees into a model explanation. We remove root nodes and combine common nodes. The resulting tree provides a holistic view of the classifier.\n\n`>>> Q5`**In-context Example**\n\n`>>> A5`Thanks. Yes, LLM can extend the hierarchy beyond the  template. We use a simple template to show the general idea works.\n\n**1) Non-fixed Hierarchy**\n\nHierarchy vary across classes. For example, `dog` prompt leads the LLM to generate different hierarchy for `train`, focusing on materials, components, and utility.\n\n**2) Flexibility**\n\nThe template can be adjusted for complex cases, such as relationship prediction with object interactions. An example of the relation `carrying` is shown in `Fig 14` (rebuttal PDF). \n\n**3) Selection**\n\nWe manually write prompts with template from `Line 139` and annotate 1-5 classes as in-context examples. It works well for CIFAR/ImageNet, which feature single objects and clear backgrounds. We'll extend it in future.\n\n`>>> Q6`**Tree Pruning**\n\n`>>> A6`**One** least visited node and its children are pruned, as defined in `Eq 2`.\n\n`>>> Q7`**Faithfulness Score**\n\n`>>> A7`Good question. Yes, we use the same score for routing and evaluation. *SubTree* baseline is an ablation study to build trees **without routing**. It shows feature similarity is a good indicator for faithfulness. \n\n`>>> Q8`**Clarity and Revision**\n\n`>>> A8`Thanks for the suggestions.\n\n**1) Definition**\n\nWe'll add definitions for \"classifier\" and \"embedding\" earlier in the paper to enhance clarity.\n\n**2) $L_{HMC}$**\n \n$L_{HMC}$ uses supervised contrastive learning for tree labels. Please see [33] `Equation (3)&(4)` for details. We'll explain it in revision.\n\n**3) Fig 1**\n\n`Fig 1` shows a toy example (left) and a diagram (right). It may contain too much information. We'll split it to improve clarity.\n\n**4) Accuracy and MCS**\n\nDefinitions of MCS and TK are briefly mentioned in `Line 253`, and full definition in Appendix `Sec I.1`.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2DTnWOaW2M", "reviewer": "Reviewer_kLjQ", "summary": "The goal of the paper is to bridge the gap between human comprehension and AI decision. For this purpose, the authors propose a Language Model as Visual Explainer (LVX), an approach to interpret the internal workings of vision models through tree-structured linguistic explanations, without the need for additional model training.  The propose method leverages the collaboration between vision models and large language models (LLM) to generate these explanations. The LLM is used to outline hierarchical visual attributes, while a text-to-image API retrieves images that best match these textual descriptions. By mapping these texts and images to the vision model’s embedding space, they create a hierarchical visual embedding tree. This tree is dynamically adjusted by querying the LLM with language templates, allowing for the addition new attributes and removal of irrelevant concepts based on the model’s representations. Additionally, the authors propose a new benchmark and new metrics to demonstrate the plausibility, faithfulness, and stability of the newly introduced method.", "questions": "Here are my concerns:\n- Eq. 3: How is the loss function $L_{HMC}$ defined\n- Section 4.1 -> Evaluation Metrics -> Plausability: How is the unnormalized TK score defined?\n- How does your approach may cope with fine-grained datasets, such as Flowers or CUB-200? I assume that the robustness of the approach relies on the capability of the LLM to cope with fine-grained datasets. Do you think that your approach could handle such case or some modifications (fine-tuning) should be performed on the LLM in order to deal with this particular case. Please provide your insight with respect to this problem.", "limitations": "Limitations are related with the curation of the new dataset and imply the following aspects: false positive simages, the presence of out-of-distribution samples and potential bias in the dataset. This research work does not have any negative societal impact.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Here are the paper's strengths:\n- it introduces a novel exaplainability method leveraged by LLM\n- it introduces a new benchmark and novel metrics for an efficient evaluation\n- the paper is well documented and clearly written. The contributions and objectives are clearly stated.\n- the review of the state of the art is comprehensive and covers most of the relevant works\n- the experimental validation is extensive", "weaknesses": "Weaknesses:\n- some concepts presented in the paper require more details", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank the reviewer for the positive feedback. We are delighted that R-kLjQ acknowledge our novel explainability method using LLMs, our new benchmark, and our evaluation metrics. Those encouraging words means a lot and inspires us to push our research further.\n\n`>>> Q1`**Hierarchical Contrastive Loss $L_{HMC}$**\n\n`>>> A1`Thanks. The Hierarchical Multi-label Contrastive Loss $L_{HMC}$ comes from [33]. It is designed to learns tree-structured label using supervised contrastive learning. \n\n- **Idea**:  $L_{HMC}$ applies contrastive learning to hierarchical labels. It adds higher weight to early layer nodes, to ensure feature compactness. Conversely, it gives less constraint on leaf node features. Fine-tuning with  `Eq 3`  integrates hierarchical constraints into the feature space of classifiers.\n\n- **Math**: Say, $L$ is the set of all label levels, and $l \\in L$ is a level in the multi-label hierarchy. The loss for pairing an anchor image, indexed by $i$, with a positive image at level $l$ is defined as:\n\n\t$$\n\tL_{\\text{pair}}(i, p_l) = -\\log \\frac{\\exp(f_i \\cdot f_{p_l} / \\tau)}{\\sum_{a \\in A \\setminus \\{i\\}} \\exp(f_i \\cdot f_a / \\tau)}\n\t$$\n\n\tThe hierarchical multi-label contrastive loss $L_{\\text{HMC}}$ is defined as:\n\n\t$$\n\tL_{\\text{HMC}} = \\sum_{l \\in L} \\frac{1}{|L|} \\sum_{i \\in I} \\frac{-\\lambda_l}{|P_l(i)|} \\sum_{p_l \\in P_l} L_{\\text{pair}}(i, p_l)\n\t$$\n\n`>>> Q2`**Definition of TK score**\n\n`>>> A2`Thanks for the question. Tree Kernel (TK) measures the difference between two trees, by computing the number of the common nodes for all pairs of sub-trees. Its definition is briefly mentioned from `Line 253`, and full definition in Appendix `Sec I.1`.\n\n`>>> Q3`**LVX on fine-grained Dataset** \n\n`>>> A3`Great question! \n\n**1) Challenges for LVX on fine-grained Dataset** \n\nWhile our method can manage fine-grained datasets conceptually, a naive adaptation of LVX faces challenges due to *limitations of existing tools*:\n1.  **Language Limitations:**  Not all attributes can be clearly described using LLM. For example, in CUB-200, subtle variations in the shape and color of a bird’s beak might be difficult to articulate precisely with LLM.\n2.  **Image Generation Challenges:**  Image generation models like Stable Diffusion often have limited control over fine-grained details. This can result in images that do not accurately reflect subtle distinctions, leading to errors.\n\n**2) Experiments when Assuming Perfect Tools** \n\nAs suggested, we conducted an experiment on CUB-200 when assuming all tools are perfect.\n\nSpecifically, because CUB-200 already provides a hierarchical attribute labels, we retrieved images directly from the dataset based on these attributes. In this way, we do not use LLMs or image generation models, avoiding the issues associated with these tools. Although this is not the original LVX version, **it simulates the case when we have perfect language and image generation abilities**.\n\nWe used a pre-trained ViT-S on CUB-200 to generate explanations. For the generated tree, we measured its alignment with the annotated tree using MCS and TK scores. We also compared it to a baseline,*TrDec*, which was trained to decode the tree structure with a frozen encoder.\n\n| Model | CUB-200(MCS&#8593;/TK&#8593;) | |\n|--|--| --|\n| | **TrDec**| **LVX** | \n| ViT-S | 26.21/45.12 | **28.64/57.18**| \n\nAs shown in the table, LVX performs well on fine-grained datasets, as long as both the LLM and image generation tools can accurately capture fine details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tsUdA3zAxK", "reviewer": "Reviewer_FVKA", "summary": "This paper builds an attribute tree using LLMs to explain image classifiers. To this end, the paper designs a framework that expands and prunes nodes for the tree. Using this framework, the paper provides a richly annotated version of CIFAR and ImageNet, which is 5 times more expressive than prior works. With this attribute tree, the paper outperforms previous decision tree approaches such as NBDT.", "questions": "Can the method be improved using multimodal models like GPT-4V instead of language-only models like GPT-4?", "limitations": "Discussed in the Appendix.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- New datasets with rich attributes could be valuable for the community, though the necessity of such large attribute sets should be further justified (W3).\n- Using rich attributes, the paper outperforms previous NN-based decision trees, such as NBDT.\n- The paper is quite dense, containing a lot of information.", "weaknesses": "1. Explanation is more than an attribute tree\n\nThe paper focuses on building attribute trees using LLMs. However, there are more diverse ways to interpret the behaviors of vision models, particularly using language. For example, [1,2] use vision-language models to understand the failures of vision models through language explanations. [3-5] use language to derive concepts for concept bottleneck models (CBMs), though this paper only cites [5]. [6] generates language to explain the neurons in the network. Among these approaches, the paper should discuss the advantages and drawbacks of the proposed method of creating an attribute tree.\n\nFor this reason, I believe the current paper title is overstated. I recommend clarifying its scope in the title to focus on the construction of hierarchical attributes and decision trees.\n\n[1] Kim et al. Discovering and Mitigating Visual Biases through Keyword Explanation. CVPR 2024.\\\n[2] Wiles et al. Discovering Bugs in Vision Models using Off-the-shelf Image Generation and Captioning. arXiv 2023.\\\n[3] Yuksekgonul et al. Post-hoc Concept Bottleneck Models. ICLR 2023.\\\n[4] Oikarinen et al. Label-Free Concept Bottleneck Models. ICLR 2023.\\\n[5] Yang et al. Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification. CVPR 2023.\\\n[6] Hernandez et al. Natural Language Descriptions of Deep Visual Features. ICLR 2022.\n\n\n---\n2. Insufficient qualitative insights\n\nThe paper mostly focuses on quantitative metrics such as plausibility, faithfulness, and stability, which support the validity of the proposed explanation. However, an explanation is only meaningful when it provides new insights to humans. In this sense, the paper should carefully examine individual data points and their corresponding explanations and discuss the lessons learned from them.\n\nI know that Figure 6 provides some examples of decision trees. However, the paper could delve deeper, for example, by addressing: 1) What are the common issues with current classifiers? 2) What are the differences between different models, such as ResNet vs. ViT? As the paper has built a tool, there are numerous directions to explore using it.\n\n---\n3. Necessity of enormous attributes\n\nThe paper creates datasets with rich attributes (H-XX), which are 5 times more expressive than the prior datasets (DR-XX). However, do we really need all these attributes? More analysis and justification of the attributes should be provided, clearly showing the limitations of prior works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly appreciate the suggestion and acknowledgment from R-FVKA regarding the dataset and method.\n\n`>>> Q1` **Explanation beyond Attribute Tree**\n\n`>>> A1` Amazing question. We appreciate the valuable papers mentioned by the reviewer and will cite them in our final version. While all [1-6] use natural language to explain visual models, our method has three key distinctions:\n\n1. **No Training Required:** Compared to [3-5], LVX eliminates the need for human annotations and additional model training. It works as a plug-and-play solution.\n2. **Use of Large Language Model:** Compared with [1-4, 6], we use large  language models as automatic tools to explain vision models, providing rich knowledge and  open-vocabulary capabilities.\n3. **Hirachical structured explanation:** While [1-6] provide single-round, single-grain explanations, LVX offers hierarchical, multi-grained explanations.\n\nWe will cite these works properly and include a discussion in our final version.\n\n`>>> Q2` **Paper Title**\n\n`>>> A2` We truly appreciate the suggestion. We will revise our title as `Language Model as Hierachical Visual Concept Explainer` to better reflect the scope of our work.\n\n`>>> Q3` **Experiments and Insights**\n\n`>>> A3` This is a nice point. As suggested, we do two experiments to use LVX to study the common problems in current classifiers, and compare classifier from different families.\n\n**1) Common Issue** \n\nPer the advice, we identify the most frequently correct and misclassified attributes in LVX on ImageNet across 8 networks. \n- **Setup:** We present word clouds of these correct classifications and errors in `Fig 12` of rebuttal PDF. \n- **Results: Shape-bias**. Our findings indicate that current deep networks are better at recognizing `Attributes` like **color**, but they struggle with accurately recognizing object `Substances` based on their **shape** and **size**.\n \n**2) Comparing CNN and ViT** \n\nAs suggested, we conducted experiments to compare CNN and Transformer models and identify which concepts they miss. \n- **Setup:** We examined 26,928 ImageNet concepts, categorized into `Concepts, Substances, Attributes, and Environments` defined in the template. We measured (1) errors in each sub-category and (2) accuracy at different tree depths $l$. Note that when $l=1$, the accuracy is the typical ImageNet top-1 accuracy. We selected DeiT-B and ConvNeXt-T because they have similar top-1 ImageNet accuracy. \n\n- **Result:** While both models show similar overall accuracy, we found that CNNs are better at recognizing local patterns, like detailed `Attributes` and `Substances` of objects. In contrast, Transformers like DeiT-B perform better on `Environments`, focusing on broader contexts.\n\n|Model|Accuracy(%)||||\n|-|-|-|-|-|\n|**Subset**|*Concepts*|*Substances*|*Attributes*|*Environments*|\n|ConvNeXt-T|*23.1* |*18.9* |**45.3**|18.1|\n|DeiT-B|22.0|15.6|35.7|**26.3**|\n\nAdditionally, Transformers are more accurate at shallow depths, while CNNs excel at deeper depths. This finding aligns with earlier research showing that **CNN are biased towards textures over shape**[A]. \n\n|Model|Accuracy(%)|||| \n|-|-|-|-|-|\n|**Depth**| 1(ImageNet Acc)|2|3|4|5| \n|ConvNeXt-T |82.1|65.1| 46.2|**35.7**|**25.8**| \n|DeiT-B |81.8|**70.1**|**48.0**|32.7|13.9| \n\nWe will incorporate them into the final version.\n\n[A] ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness, ICLR 2019\n\n`>>> Q4` **Use of fine-grained Attributes**\n\n`>>> A4` Thanks. Yes, collecting more fine-grained attributes are indeed useful. \n\n**1) Dataset Analysis and Comparison**\n  \nWe've provided more dataset statistics in `Fig 8` of Appendix.  As suggested, we also compared attributes per class between H-ImageNet and DR-ImageNet in `Fig 11` in the rebuttal PDF. \n\nThis comparison clearly shows that, even the maximum number of attributes in DR-ImageNet is less than the minimum number of attributes in H-ImageNet, **max(DR-ImageNet) < min(H-ImageNet)**. It supports our dataset's greater diversity and depth.\n  \n**2) Previous Limitations**\n\nPrevious datasets that include attributes have limitations in two key areas:\n- **Insufficient Attribute in prior work:** DR-ImageNet, although large, has only 5,810 attributes, averaging **5.8 unique attributes per category**. In contrast, our _H-ImageNet_ has 26,928 attributes, providing a much richer dataset.\n- **Fine-grained Explanation:** Current datasets[34-37] focus on single-level, coarse grained attributes. Humans, however, provide explanations in a multi-dimensional and hierarchical manner. Our _H-ImageNet_ reflects this complexity, making it more helpful for understanding fine details.\n\n In this scope, our _H-X_ dataset addresses these gaps by providing a richer set of attributes for more detailed analysis and applications. \n \n**3) More use case**\n \nThe rich attribute data in  _H-ImageNet_  can also be used for other tasks, such as **tree label classification and model calibration**, expanding its applicability beyond its initial scope.\n\n`>>> Q5` **Using Multimodal Models for Explanation**\n\n`>>> A5` Thank you for the suggestion. We are currently exploring the use of Multimodal Large Language Models (MLLM) like GPT-4 for enhancing explanations. Here are a few potential directions:\n1. **Automatic Tree Label Collection**: Instead of manual annotation, we can use MLLMs to generate tree-like labels from images. This method can efficiently create large-scale hierarchical annotations for evaluation and model calibration.\n2. **Advanced Image Filtering:** LVX uses tools to synthesize images, which can sometimes contain errors. MLLMs can verify these images to ensure the attributes are correct.\n3. **Model Error Detection:** MLLMs can spot errors by comparing tree explanation with image content. For example, if a model wrongly says a long-haired dog has no hair, MLLMs can check the image and find the mistake.\n\nWe appreciate the suggestion and will continue to explore these possibilities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like express our sincere gratitude to all reviewers for their constructive comments. We are particularly thankful for the following positive feedback:

-   The idea is interesting and novel:  `Reviewer UiCA`, `Reviewer kLjQ`;
-   The contributions of the dataset and evaluation metrics are valuable:  `Reviewer kLjQ`, `Reviewer FVKA`;
-   The evaluations are extensive:  `Reviewer kLjQ`;
-  The results outperforms previous methods: `Reviewer FVKA`;
-   The paper is well-written and clearly presented:  `Reviewer U6GY`, `Reviewer kLjQ`.

We will address the specific questions raised by the reviewers in the subsequent sections of the rebuttal.

Meta-review: Originally the paper received mixed reviews, concerns raised included:

1. the loose coupling between how the explanation tree is constructed and the inner workings of the classification model
2. the reliance on only the attributes for the explanation
3. the validity of the metrics
4. general motivation
5. practical value
6. additional baselines needed
7. clarity of presentation
8. insufficient qualitative insights

Following the rebuttal, it seems to the AC that most concerns were addressed as acknowledged by the reviewers, with some concerns remaining on the need for a major revision to improve clarity (rev. UiCA) who raised their score to 4 (still below acceptance). However, as the majority of reviewers recommend acceptance, with UiCA raising their score, the AC believes this paper can still be a valuable contribution to the proceedings. The AC strongly urges the authors to incorporate all the discussions into their revised manuscript, paying special attention to the constructive suggestions of UiCA (as well as all other reviewers ofc).
Decision: Accept (poster)