Explainable Concept Generation through Vision-Language Preference Learning

## Abstract

Abstract Concept-based explanations have become a popular choice for explaining deep neural networks post-hoc because, unlike most other explainable AI techniques, they can be used to test high-level visual “concepts” that are not directly related to feature attributes. For instance, the concept of “stripes” is important to classify an image as a zebra. Concept-based explanation methods, however, require practitioners to guess and collect multiple candidate concept image sets, which can often be imprecise and labor-intensive. Addressing this limitation, in this paper, we frame concept image set creation as an image generation problem. However, since naively using a generative model does not result in meaningful concepts, we devise a reinforcement learning-based preference optimization (RLPO) algorithm that fine-tunes the vision-language generative model from approximate textual descriptions of concepts. Through a series of experiments, we demonstrate the capability of our method to articulate complex and abstract concepts which aligns with the test class that are otherwise challenging to craft manually. In addition to showing the efficacy and reliability of our method, we show how our method can be used as a diagnostic tool for analyzing neural networks.

## 1 Introduction

In an era where black box deep neural networks (DNNs) are becoming seemingly capable of performing general enough tasks, our ability to explain their decisions post-hoc has become even more important before deploying them in the real world. Among many use cases, explaining the behavior of DNNs enable engineers and regulatory bodies to assess the correctness of a DNN’s decision-making process and characterize the limits of what the DNN knows. Thus, an explainable AI (XAI) method’s ability to correctly communicate information in a human-centric way is essential for its usefulness.

Humans utilize high-level concepts as a medium for providing and perceiving explanations. In this light, post-hoc concept-based explanation techniques, such as Testing with Concept Activation Vectors (TCAV) (Kim et al., ), have gained great popularity in recent years. Their ability to use abstractions that are not necessarily feature attributes or some pixels in test images helps with communicating these high-level concepts with humans. For instance, as demonstrated in TCAV, the concept of stripes is important to explain why an image is classified as a zebra, whereas the concept of spots is important to explain why an image is classified as a jaguar. Given 1) a set of such high-level concepts, represented as sample images (e.g., a collection of stripe images and a collection of spot images) and 2) test images of the class (e.g., zebra images), TCAV assigns a score to each concept on how well the concept explains the class decision (i.e., zebra).

Figure: Figure 1: (a) Our proposed algorithm, RLPO, iteratively refines the concepts $c_{i}$ that can be generated by a Stable Diffusion (SD) model by optimizing SD weights based on an action $a_{i}$. Each step in this update process provides an explanation at a different level of abstraction. (b) Three concepts identified by our approach for the zebra class. Concepts are represented as images generated by SD.
Refer to caption: x1.png

Although concept-based XAI methods are a good representation, their requirement to create collections of candidate concept sets necessitate the human to know which concepts to test for. This is typically done by guessing what concepts might matter and manually extracting such candidate concept tests from existing datasets. While the stripe-zebra analogy is attractive as an example, where it is obvious that stripes is important to predict zebras, in most applications, we cannot guess what concepts to test for, limiting the usefulness of concept-based methods in testing real-world systems. Additionally, even if a human can guess a few concepts, it does not encompass most concepts a DNN has learned because the DNN was trained without any human intervention. Therefore, it is important to automatically find human-centric concepts that matter to the DNN’s decision-making process.

Figure: Figure 2: We can generate concepts that encompass both human-defined and retrieved concepts. Note that retrieved concepts are very similar to class images, making them less useful as concepts.
Refer to caption: x2.png

As attempts to automatically discover and create such concept sets, several work has focused on segmenting the image and using these segments as potential concepts, either directly (Ghorbani et al., ) or through factor analysis (Fel et al., ; ). In such methods, which we refer to as retrieval methods, because the extracted concept set is already part of the test images as shown in Fig. 2 (Retrieved concepts), it is difficult for them to imagine concepts that do not have a direct pixel-level resemblance to the original image class. For instance, it is more likely that such methods provide patches of zebra as concepts instead of stripes.

By departing from existing concept set creation practices of human handcrafting and retrieval, we redefine concept set creation as a concept generation problem. Modern generative models such as stable diffusion (SD) can be used for generating realistic images. Nevertheless, since a generative mode generates arbitrary images, we need to carefully guide it to generate explanatory images. One obvious approach is to engineer long, descriptive text prompts to generate concepts. However, engineering such prompts is not realistic. Therefore, to automate this process, as shown in Fig. 1, we propose a method, named reinforcement learning-based preference optimization (RLPO), to update SD weights. At its core, we devised a deep reinforcement learning algorithm gradually update SD weights to generate concept images that have a higher explanation score. The contributions of this paper can be summarized as follows:

- 1.
We propose a method, named RLPO, to “generate” concepts that truly matters to the neural network. Some of these concepts would not pop up in humans head until they see them. Also, unlike existing retrieval methods, RLPO can generate concepts that are not part of test images (Fig. 2).
- 2.
Because of how we use model’s preference to gradually make the SD process closer to the target class from a high-level concept, for the first time in concept-based XAI, we can generate concepts with different abstraction levels.
- 3.
In addition to demonstrating the novelty, abstractness, and diversity of generated concepts, we experimentally verify their generalizability using an NLP sentiment analysis tasks and the actionability by leveraging concepts as a tool for fine-tuning.

## 2 Preliminaries and Related Work

Testing with Concept Activation Vectors (TCAV):
The TCAV score quantifies the importance of a “concept” for a specific class in a DNN classifier (Kim et al., ). Here, a concept is defined broadly as a high-level, human-interpretable idea such as stripes, sad faces, etc. A concept (e.g., stripes), $c$, is represented by sample images, $X_{c}$ (e.g., images of stripes). In TCAV, a human has manually collected these sample concept images based on educated guesses, whereas our objective is to automatically generate them. For a given set of test images, $X_{m}$ (e.g., zebra images), that belongs to the same decision class (e.g., zebra), $m$, TCAV is defined as the fraction of test images for which the model’s prediction increases in the “direction of the concept.” By decomposing the DNN under test as $f(x)=f_{2}(f_{1}(x))$, where $f_{1}(x)$ is the activation at layer $l$, TCAV score is computed as,

$$ $TS_{c,m}=\frac{1}{|X_{m}|}\sum_{X_{m}}\mathbb{I}\left(\frac{\partial\text{ output}}{\partial\text{activations}}\cdot{(c\enspace\text{direction})}>0\right )=\frac{1}{|X_{m}|}\sum_{x_{i}\in X_{m}}\mathbb{I}\left(\frac{\partial f(x_{i} )}{\partial f_{1}(x_{i})}\cdot v>0\right)$ (1) $$

Here, $\mathbb{I}$ is the indicator function that counts how often the directional derivative is positive. Concept activations vector (CAV), $v$, is the normal vector to the hyperplane that separates activations of concept images, $\{f_{1}(x);x\in X_{c}\}$, from activations of random images, $\{f_{1}(x);x\in X_{r}\}$. Refer to Appendix C.1 for details on the TCAV settings.

ACE (Ghorbani et al., ) introduced a way to automatically find concepts by extracting relevant concepts from the input class. It used segmentation over different resolution to get a pool of segments and then grouped them based on similarity to compute TCAV scores. Though the ACE concepts are human understandable, they are noisy because of the segmentation and clustering errors. As a different method, EAC (Sun et al., ) extracts concepts through segmentation. CRAFT (Fel et al., ) introduced a recursive strategy to detect and decompose concepts across layers. Lens (Fel et al., ) elegantly unified concept extraction and importance estimation as a dictionary learning problem. However, since all these methods obtain concepts from test images, the concepts they generate tend to be very similar to the actual class (e.g., a patch of zebra as a concept to explain the zebra instead of stripes as a concept), making it challenging to maintain the “high-level abstractness” of concepts. In contrast, we generate concepts from a generative model. Under generative models, LCDA (Yan et al., ) simply queries an LLM to get attributes but does not generate concepts.

Deep Q Networks (DQN):
DQN (Mnih et al., ) is a deep RL algorithm that combines Q-learning with deep neural networks. It is designed to learn optimal policies in environments with large state and action spaces by approximating the Q-value function using a neural network. A separate target network, $Q_{\text{target}}(s,a^{\prime},\theta^{\prime})$, Here $a^{\prime}$ is $\arg\max Q(s_{next},a)$ which is a copy of the Q-network with parameters $\theta^{\prime}$, is updated less frequently to provide stable targets for Q-value updates,

$$ $Q(s_{t},a_{t})\leftarrow Q(s_{t},a_{t})+\alpha\left(r(s_{t},a_{t})+\gamma\max_ {a^{\prime}}Q_{\text{target}}(s_{t+1},a^{\prime})-Q(s_{t},a_{t})\right).$ (2) $$

Here, $s_{t}$ is the state at step $t$, $a_{t}$ is the action taken in state $s_{t}$, and $r_{t}$ is the reward received after taking action $a_{t}$. The parameters $\alpha$ and $\gamma$ are learning rate and discount factor, respectively.

Preference Optimization: Directly optimizing generative models with preference data was first introduced in Direct Preference Optimization (DPO) (Rafailov et al., ). It is a technique used to ensure models, such as large language models, learn to align its outputs with human preference by asking a human which of its generated output is preferred. This technique was later extended to diffusion models in Diffusion-DPO (Wallace et al., ), where they updated Stable Diffusion XL model using Pick-a-Pic dataset (human preferred generated image dataset). Unlike traditional image or text generation tasks, where the dataset for human preferred outputs are readily available, it is hard to have a general enough dataset for XAI tasks. To counter this problem, we provide preference information by using the TCAV score instead of a human, and use it to align the text-to-image generative model to generate concept images that matters for the neural network under test.

## 3 Methodology: Reinforcement Learning-based Preference Optimization

Figure: Figure 3: Overview of the RLPO framework with its dynamic environment interaction. The RL policy selects actions (seed prompts) which generates concept sets (G1, G2) scored through TCAV. Reward is calculated based on the scores obtained for both the sets. Simultaneously, best set is determined based on the scores obtained, which is used to update the LoRA layer of the SD model.
Refer to caption: x3.png

Our objective is to find a set of concept images, $\mathcal{C}$, that maximize the TCAV scores, $TS_{c,m}$. To this end, we leverage the state-of-the-art text-to-image generative models to generate high quality explainable concepts. However, because the search space of potential text prompts is too large, we use deep RL to guide the image generation process. As described in Fig. 3 and Algorithm 1, our algorithm, RLPO, picks potential keywords from an automatically generated list of keywords using RL and optimizes stable diffusion weights to generate images that have a preference for higher TCAV scores. This process is described in the rest of this section.

Figure: Algorithm 1 The RLPO algorithm. Appendix C.2 for the expanded algorithm.

Notation: Our framework contains three core deep learning models: the network under test $f(\cdot)$, the image generator $g(\cdot)$, and the deep RL network $h(\cdot)$. First, we have a pre-trained neural network classifier that we want to explain. We then have a generative neural network, whose purpose is generating concept image sets, given some text prompts. In this paper, we use Stable Diffusion (SD) v1-5 as the generator as it is a state-of-the-art generative model that can generate realistic images. If the weights of the SD model are $w$, for a small constant $\lambda$, we augment it as $w+\lambda ab$, where $A$ and $B$ are low-rank matrices that we fine-tune using preference optimization (Rafailov et al., ). The core search algorithm that we train is a DQN.

### 3.1 The Rationales Behind Design Choices

Before presenting the algorithm in detail, we provide the rationale for design choices, which are validated through ablation studies in Section 4.1-4.2, and comparisons in the Section 4.3-4.4.

Rationale 1: Why concept generation is a better idea. Let us denote the set of human-interpretable concepts that the $f(\cdot)$ has learned be $\mathcal{C}_{N}$. If we use concept-based explanation the traditional way (Kim et al., ; Schut et al., ), then the end users need to manually guess what concepts to test for. Automatically retrieving the concept set by segmenting test images (Sun et al., ) also results in a limited concept set. In contrast, a SOTA generative model can generate high quality images. We provide more theoretical insights in Appendix B.

Rationale 2: Why a deep RL-controlled VLM fine-tuning for generating concepts is a better idea. “A picture is worth a thousand words but words flow easier than paint.”

As the saying goes, “a picture is worth a thousand words,” it is much easier for people to explain and understand high-level concepts when images are used instead of language. For instance, we need a long textual description such as “The circles are centered around a common point, with alternating red and white colors creating a pattern” to describe a simple image of a dart board (i.e., Target Co. logo). Therefore, we keep our ultimate concept representation as images. However, controlling a generative model from visual inputs is much harder. However, since human language can be used as a directed and easier way to seed our thought process, as the saying goes, “words flow easier than paint,” we control the use of text prompts. Since the vastness of the search space cannot be handled by most traditional search strategies, we resort to a DQN for controlling text. Since simple text alone cannot generate complex, high-level visual concepts, in each DQN update step, we use preference optimization to further guide the search process towards more preferred outcome, allowing the DQN to focus on states similar to the target. This approach improves our starting points for each DQN episode, enabling more efficient search and incremental progress towards the desired target.

### 3.2 Extracting Seed Prompts

Since a generative model can generate arbitrary images, if we provide good starting point for optimization then the convergence to explainable states would be faster. In this paper, to extract seed prompts for a particular class we make use of the off-the-shelf VQA model followed by several preprocessing steps, as described in Appendix C.3. We also explore how random gibberish prompts can be used as seed prompts in Appendix C.4 which did not yield useful concepts.

### 3.3 Deep Reinforcement Learning Formulation

Our objective of using deep RL is automatically controlling text input of Stable Diffusion. As text input, we start with seed prompts from Section 3.2, $\mathcal{K}$, that have the potential to generate meaningful concept images after many deep RL episodes.
We setup our RL state-action at iteration $t$ as,

- •
Action $a_{t}$: Selecting a seed prompt, $k_{t}\in\mathcal{K}$, that best influences concept image generation.
- •
State $s_{t}$: Preferred concept images generated from the seed prompt, $k_{t-1}$.
- •
Reward $r_{t}$: Reward $r_{t}$ is proportional to the TCAV score computed at state $s_{t}$ on action $a_{t}$, adjusted by a monotonically increasing scaling factor $\xi_{t,k}$. As each seed concept reaches the explainable state at different times, this factor is introduced to scale the reward over time $t$ for each unique seed concept $k$. Since the $g(.)$ is getting optimized at each time step $t$. The scaling factor is updated as $\xi_{t+1,k}\leftarrow\min(1,\frac{\xi_{t,k}+1}{T})$, where T is total number of RL steps. Therefore, the expected cumulative adjusted reward is $R(\pi)=\mathbb{E}[\sum_{t=0}^{T}\xi_{t}\cdot r_{t}(s_{t},a_{t})]$.

Our objective in deep RL is to learn a policy, $\pi:s\to a$, that takes actions (i.e., picking a seed prompt) leading to explainable states (i.e., correct concept images) from proxy states (i.e., somewhat correct concept images). We formally define explainable state and proxy state as follow:

###### Definition 1 .

Explainable states: States that have a concept score $TS_{c,m}\geq\eta$ for a user-defined threshold $\eta\in[0,1]$ for concept $c$ and class $m$ is defined as an explainable state.

###### Definition 2 .

Proxy states: States that have a concept score $TS_{c,m}<\eta$ for the threshold $\eta\in[0,1]$ for concept $c$ and class $m$ is defined as a proxy state.

In practice, we set $\eta$ to a relatively large number, such as 0.7, to ensure that we look at highly meaningful concepts. In DQN, in relation to Eq. 2, we learn a policy that iteratively maximizes the $Q(s,a)$ value by using the update rule,

$$ $Q^{*}(s,a)=\mathbb{E}_{s^{\prime}\sim P(\cdot|s,a)}[\xi_{t}r(s,a)+\gamma\max_{ a^{\prime}\in A}Q_{\text{target}}(s^{\prime},a^{\prime})].$ (3) $$

###### Definition 1 .

###### Definition 2 .

### 3.4 Optimizing the States

At time $t$, the policy picks the seed prompt $k_{t}$, which is then used by the generative model, $g(k_{t};w_{t})$, with model weights $w$, to generate $2Z$ number of images. We randomly divide the generated images into two groups: $X_{c_{1},t}=\{x_{c_{1},t,i}\}_{i=1}^{Z}$ and $X_{c_{2},t}=\{x_{c_{2},t,i}\}_{i=1}^{Z}$. Let the TCAV scores of each group be $TS_{c_{1},m,t}$ and $TS_{c_{2},m,t}$. Since our objective is to find concepts that generate a higher TCAV score, concept images that have a higher score is preferred. Note that, unlike in the classical preference optimization setting with a human to rank, RLPO preference comes from the TCAV scores (e.g., $TX_{c_{1},t}\succ TX_{c_{2},t}$). We call this notion RLPO-XAI in some ablation studies below. If the generative model at time $t$ is not capable of generating concepts that are in an explainable state, $\max(TS_{c_{1},m,t},TS_{c_{2},m,t})\leq\eta$, we then perform preference update on SD’s weights (more details in Appendix C.5). Following Low-Rank Adaptation (LoRA) (Hu et al., ), we only learn auxiliary weights $a$ and $b$ at each time step, and update the weights as $w_{t+1}\leftarrow w_{t}+\lambda ab$.

As the deep RL agent progresses over time, the states become more relevant as it approaches explainable states (Fig. 1), thus the same action yields increasing rewards over time. To accommodate this, with reference to the rewards defined in Section 3.3, we introduce a parameter $\xi$, which starts at 0.1 and incrementally rises up to 1 as the preference threshold, $\eta$, is approached. Different actions may result in different explainable states, reflecting various high-level concepts inherent to $f(\cdot)$. Some actions might take longer to reach an explainable state. Also, it is possible for different actions to lead to the same explainable state. As the goal is to optimize all states to achieve a common target, DQN progressively improves action selection to expedite reaching these states. Thus, deep RL becomes relevant as it optimizes over time to choose the actions that are most likely to reach an explainable state more efficiently.

## 4 Experiments

To test the effectiveness of our approach, we tested it across multiple models and several classes. We considered three CNN-based classifiers, ResNet-50 (He et al., ), GoogleNet (Szegedy et al., ), and InceptionV3 (Szegedy et al., ), and two transformer-based classifiers, ViT (Dosovitskiy et al., ) and Swin (Liu et al., ), pre-trained on ImageNet dataset. Unless said otherwise, only GoogleNet results are shown in the main paper. All other model details and results are provided in Appendix C.1 and D.3), respectively. All metrics are defined in Appendix B.1.

### 4.1 Ablation study: Search Strategies (Why deep RL?)

We chose DQN as our RL algorithm because of its ability to effectiveness traverse through discrete action space Mnih et al. () (20 unique seed prompts). We assess the effectiveness of RL by disabling the preference optimization step. As shown in Table 2, on GoogleNet classifier, compared to $\epsilon$-greedy methods, RL setup exhibits a higher entropy, average normalized count (ANC), and inverse coefficient of variance (ICV) (See Appendix B.1 for definitions), indicating RL’s ability to take diverse actions that results in diverse concepts.

### 4.2 Ablation study: Scoring Mechanisms (Why TCAV?)

Another important aspect of our setup is the use of TCAV score, an XAI method, to provide preference feedback and calculate rewards. Alternatively, this XAI scoring feedback can also be replaced with human feedback or LLM-based AI feedback. As an additional experiment, to test the effect of human feedback, we conducted human feedback experiment with eight human subjects who provided live human feedback. Further, to evaluate the LLM-based AI feedback, we made use of GPT-4o. More details on the experiment setup and results are provided in Appendix D.2. As shown in Table 4, we concluded that, even though other feedback techniques can be used, XAI-based feedback is best for generating concepts that are important to model with high speed and low computation cost. Though human and AI (GPT4o) are good at correlating semantics, by only looking at test images and concepts instead of model activations, they are not able to provide model specific explanations. Also, human experiments tend to be expensive.

**Table 1: Search strategy ablation. We see that RL, compared to $\epsilon$-greedy search, is the best strategy to efficiently explore the search space with high entropy, high average normalized count (ANC) per action, and high inverse coefficient of variance (ICV).**
| Method | Entropy ($\uparrow$) | ANC ($\uparrow$) | ICV ($\uparrow$) |
| --- | --- | --- | --- |
| RL (Ours) | 2.80 | 0.43 | 2.17 |
| 0.25 Greedy | 2.40 | 0.21 | 1.04 |
| 0.5 Greedy | 1.95 | 0.15 | 0.59 |
| 0.75 Greedy | 1.85 | 0.15 | 0.56 |

**Table 3: Scoring mechanisms ablation. We see that RLPO with Explainable AI feedback (RLPO-XAIF), in this case TCAV, is a better choice than RLPO with human feedback (RLPO-HF) and AI feedback (RLPO-AIF).**
| Method | Class-based | Model-specific | Feedback | Execution |
| --- | --- | --- | --- | --- |
|  | Explanations | Explanations | Cost* | Time (↓) |
| RLPO-HF | ✓ | ✗ | NIL | 180 $\pm$ 30s |
| RLPO-AIF | ✓ | ✗ | >10 GB | 72 $\pm$ 1.2s |
| RLPO-XAIF | ✓ | ✓ | <1 GB | 56 $\pm$ 0.7s |

### 4.3 What kind of concepts can RLPO generate?

Novel concepts.
As illustrated in Fig. 4, we observed that the RLPO can generate concepts that a human would not typically think of but leads activations of the DNN to trigger. To validate this hypothesis, we conducted a survey to see if humans can think of these generated concepts as important for the neural network to understand a certain class (Table 2, The Exploration Gap (EG) quantifies the proportion of missed optimal actions, defined as $1-\text{Accuracy}$, highlighting how humans frequently miss the most optimal actions when presented with generated concepts.).
As detailed in Appendix D.6, we presented a random class image followed by two concepts, one generated by our method and another from a previous retrieval based method (Fel et al., ; ). We presented choices from the concepts with high score from both methods and we discovered that while most participants could recognize retrieval-based concepts, only those with domain-specific knowledge could identify generated concepts. This indicates that most people can only identify concepts from a small subset of what $f(.)$ learns during training. Intuitively, when we retrieve concepts from the test class, they tend to be similar to the test images.

Figure: Figure 4: Samples of concepts (with different TCAV scores, not shown here) generated by different methods. Observe that RLPO generates diverse images, not just patches from the test images.
Refer to caption: x4.png

We also compare the generated concepts qualitatively and quantitatively. Fig. 4 shows the diversity of concepts generated by our method and other retrieval based methods. Additionally, we also verify this by computing the vector similarity of the CLIP and ResNET50 embeddings between $X_{c}$ and $X_{m}$ for multiple retrieval based methods. As highlighted in Table 4, we observe that retrieval based methods tend to have high cosine similarity between extracted concepts and test images, making them less useful as abstract concepts (e.g., to explain the zebra class, a patch of zebra as a concept is less useful compared to stripes concept).

Abstract concepts. In Fig. 5, we observe the progression of output concepts generated by the SD when RLPO is applied for the seed prompt “zoo,” of tiger class. These abstractions hint us about what the model prefers when it is looking for tiger, starting from a four-legged orange furred animal, to black and white stripes with orange furred animal, to black and white stripes with orange furred and whiskers. We obtain concepts with various abstractions by changing $\eta$ (Currently, it is not possible for our method to decide $\eta$ to get a particular level of abstraction).

Figure: Figure 5: Different levels of abstractions for the “Tiger” class on GoogleNet classifier. The generated image begins as a random zoo image and gradually transition to images with tiger features. We also see that the model’s prediction changes over time, starting from a random classification of oxcart to classifying the generated concept as tiger itself.
Refer to caption: x5.png

Multiple concepts.
Because RLPO algorithm explores various explainable states, we can obtain multiples concepts with varying level of importance. Fig. 6 shows the top three class-level concepts identified by our method for the “zebra” class for the GoogleNet classifier. We see that, each concept set has a different TCAV score associated with them indicating their importance.

Figure: Figure 6: The figure shows the concepts identified by our method and where they are located in the input class image (“zebra” class) for GoogleNet classifier. As highlighted the “stripes” concept image are located near zebra, the “running” concept images, showing trees are highlighting in background of the input image, and the “mud” concept highlighting the grass and soil in the input image. The concepts are ordered in their importance (TCAV score) with “stripes” being the highest and “mud” being the lowest for the selected class.
Refer to caption: x6.png

### 4.4 Are the generated concepts reliable?

Figure: Figure 7: Comparison of concepts identified by different methods. RLPO can show the correspondences between test image and different concepts.
Refer to caption: x7.png

After generating the concepts, next step is to identify what those concepts signify. To locate where in the class images generated concepts correspond, we made use of CLIPSeg (Lüddecke & Ecker, ), a transformer-based segmentation model which takes in concept images as prompts, $X_{c}$, and highlights in a test image, $x\in X_{m}$, which part resembles the input prompt as a heat map. More details on this is available in Appendix D.3.3. As shown in Fig. 6, class image on left highlights the top 3 identified concepts by RLPO. We also compare the output generated by other popular XAI techniques such as LIME and GradCam with ones generated by RLPO. As shown in Fig. 7, we can see that other methods just explains where the model is looking at whereas our approach also explains what type of features is the model focuses on.

After finding the relationship between generated concepts and input images, we need to validate the importance of the identified concepts. To that end, we applied c-deletion, a commonly used validation method in XAI, to the class images for each identified concept. We gradually deleted concept segments based on the segmentation heat map obtained from ClipSeg. The results for the c-deletion are shown in the Fig. 9. We see the area under curve is the highest for the most important concept “stripes” and the lowest the least important concept “mud,” indicating the order of importance of each concept. More examples on the c-deletion are in Appendix D.3.4.

Figure: Figure 8: C-deletion. Removing concepts over time to measure the reliability. The colored numbers indicate the area under the curve (the lower the better as it indicates important concepts are removed sooner.)
Refer to caption: x8.png

Figure: Figure 10: We use sentiment analysis in NLP to show the generalizability of RLPO. Here, concepts tend to be synonyms and the numbers indicate the TCAV scores. Generated concepts explain why a given text is classified as a positive sentiment.
Refer to caption: x10.png

### 4.5 Applications and generalizability

We show how RLPO can be used as a diagnostic tool for the engineers. As a specific application, we see what concepts are removed and added, as well as how the concept importance changes when we fine-tune ResNet50 model on ImageNet to improve accuracy (Fig. 9). More details about the experiment is present in Appendix D.5.

To demonstrate the generalizability of the proposed algorithm, we extended RLPO to generate words in sentiment analysis in NLP. We made use of Mistral-7B Instruct model to generate synonyms of seed prompts and optimized the language model based on preferences from TextCNN model pre-trained on IMDB sentiment dataset. Fig. 10 highlights relevant words with their importance score in the input. More details about the experiment is present in Appendix D.4.

## 5 Limitations and Conclusions

The process of navigating an infinitely large concept space and generating explainable concepts from textual inputs presents several challenges, particularly when dealing with complex, high-level concepts. We showed how deep RL can guide SD efficiently to navigate this space. However, RLPO also suffers from several limitations. First, this analysis cannot be performed real-time since generating images from SD, learning the DQN, and fine-tuning the SD with preferences takes some time. Also, the concepts that our algorithm generates can be diverse as it tries to reveal the concepts inherent to the NUT, making it less domain-specific (e.g., for a medical application, there is a chance it might generate non-medical images if the NUT activations get excited for non-medical data). As a future extension, we hope to input preferences from both TCAV and application experts while optimizing, making generated explanations even more aligned to specific applications. Despite the challenges, our results show how to leverage the strengths of visual representations and adaptive learning to provide intuitive and effective solutions for understanding complex, high-level concepts in black-box neural networks.

## Appendix

## Appendix A Related Work

Use of VLM in explaination: Recent advancements in Vision-Language models (VLMs) have open the doors for the use of VLMs in multiple domains, mainly because of their ability to generalize over large amount of data, they can be leveraged to obtain useful information. Work by Sun et al. () present a novel method combining the Segment Anything Model (SAM) with concept-based explanations, called Explain Any Concept (EAC). This method uses SAM for precise instance segmentation to automatically extract concept sets from images, then it employs a lightweight surrogate model to efficiently explain decision made by any neural network based on extracted concepts. Another work by Yan et al. Yan et al. () introduced Learning Concise and Descriptive Attributes (LCDA), which leverages Large Language Models (LLMs) to query a set of attributes describing each class and then use that information with vision-language models to classify images. They highlight in their paper that with a concise set of attributes, they can improve the classifier’s performance and also increase interpretability and interactivity for end user.

## Appendix B Definition, Theorems, and Proves

Comparative Overlap of Human-Interpretable and Generative Model Concepts in Neural Understanding Tasks (NUT). We formalize this with reference to Fig. 2.

###### Theorem 1 .

Let the set of human-interpretable concepts that the NUT has learned be $\mathcal{C}_{N}$, and the concept sets human collected, retrieved though segmentation, and generated using a generative model be $\mathcal{C}_{H}$, $\mathcal{C}_{R}$, and $\mathcal{C}_{G}$, respectively. Then, $|\mathcal{C}_{G}\cap\mathcal{C}_{N}|\geq|\mathcal{C}_{H}\cap\mathcal{C}_{N}|\geq
0$ and $|\mathcal{C}_{G}\cap\mathcal{C}_{N}|\geq|\mathcal{C}_{R}\cap\mathcal{C}_{N}|\geq
0$.

###### Proof sketch.

$\mathcal{C}_{H}\subseteq\mathcal{C}_{G}\enspace\text{and}\enspace\mathcal{C}_{
R}\subseteq\mathcal{C}_{G}\implies|\mathcal{C}_{H}\cap\mathcal{C}_{N}|\geq 0
\enspace\text{and}\enspace|\mathcal{C}_{R}\cap\mathcal{C}_{N}|\geq 0$
∎

###### Proof of Theorem 1 .

###### Definition 3 .

Let $\mathcal{C}_{H}$, $\mathcal{C}_{R}$, and $\mathcal{C}_{G}$ denote the sets representing human-interpretable concepts, retrieved concepts, and concepts generated by a generative model, respectively. We define the relationships between these sets as follows:

$$ $\mathcal{C}_{H}\subseteq\mathcal{C}_{G}\quad\text{and}\quad\mathcal{C}_{R} \subseteq\mathcal{C}_{G}.$ $$

###### Property 1 .

For any set $\mathcal{C}_{i}$, where $i\in\{H,R,G\}$, it holds that:

$$ $\emptyset\subseteq(\mathcal{C}_{i}\cap\mathcal{C}_{N})\subseteq(\mathcal{C}_{i }\cup\mathcal{C}_{N}),$ $$

where $\mathcal{C}_{N}$ represents the set of concepts learned by the NUT.

For any two sets $\mathcal{A}$ and $\mathcal{B}$, the size of their intersection $|\mathcal{A}\cap\mathcal{B}|$ is non-negative since it represents the number of elements common to both sets. Thus, we have:

$$ $|\mathcal{C}_{H}\cap\mathcal{C}_{N}|\geq 0\quad\text{and}\quad|\mathcal{C}_{R} \cap\mathcal{C}_{N}|\geq 0.$ (4) $$

Given Definition 3 and Property 1, we assume the following subset relationships between the sets:

$$ $|\mathcal{C}_{H}\subseteq\mathcal{C}_{G}|\quad\text{and}\quad|\mathcal{C}_{R} \subseteq\mathcal{C}_{G}|.$ $$

Case 1. Since $\mathcal{C}_{H}\subseteq\mathcal{C}_{G}$, any element $x\in\mathcal{C}_{H}$ is also in $\mathcal{C}_{G}$. Therefore, any element $x\in\mathcal{C}_{H}\cap\mathcal{C}_{N}$ is also in $\mathcal{C}_{G}\cap\mathcal{C}_{N}$. Hence,

$$ $\mathcal{C}_{H}\cap\mathcal{C}_{N}\subseteq\mathcal{C}_{G}\cap\mathcal{C}_{N}.$ $$

Case 2. Similarly, since $\mathcal{C}_{R}\subseteq\mathcal{C}_{G}$, any element $x\in\mathcal{C}_{R}$ is also in $\mathcal{C}_{G}$. Therefore, any element $x\in\mathcal{C}_{R}\cap\mathcal{C}_{N}$ is also in $\mathcal{C}_{G}\cap\mathcal{C}_{N}$. Hence,

$$ $\mathcal{C}_{R}\cap\mathcal{C}_{N}\subseteq\mathcal{C}_{G}\cap\mathcal{C}_{N}.$ $$

From Case 1 and Case 2, since $\mathcal{C}_{H}\cap\mathcal{C}_{N}$ and $\mathcal{C}_{R}\cap\mathcal{C}_{N}$ are subsets of $\mathcal{C}_{G}\cap\mathcal{C}_{N}$, it follows that:

$$ $|\mathcal{C}_{H}\cap\mathcal{C}_{N}|\leq|\mathcal{C}_{G}\cap\mathcal{C}_{N}|$ (5) $$

and

$$ $|\mathcal{C}_{R}\cap\mathcal{C}_{N}|\leq|\mathcal{C}_{G}\cap\mathcal{C}_{N}|$ (6) $$

Combining Eqs. 5 and 6 with the non-negativity established in Eq 4, we have:

$$ $|\mathcal{C}_{G}\cap\mathcal{C}_{N}|\geq|\mathcal{C}_{H}\cap\mathcal{C}_{N}|\geq 0$ (7) $$

and

$$ $|\mathcal{C}_{G}\cap\mathcal{C}_{N}|\geq|\mathcal{C}_{R}\cap\mathcal{C}_{N}| \geq 0.$ (8) $$

∎

The Effects of DQN-DPO-based Concept Space Traversal. We now formalize what concepts the DQN has learned, with reference to Fig. 1.

###### Theorem 2 .

When traversing in the concept space, with each reinforcement learning step,

- 1.
Case 1: Moving from a proxy state towards an explainable state monotonically increases the reward.
- 2.
Case 2: Moving from an explainable state towards the target class does not increase the reward.

###### Proof sketch.

Obtain the rewards before and after $\eta$ and compute the difference in reward for each segment.
∎

###### Property 2 .

As $\xi$ increases, the reward function proportionally amplifies, particularly enhancing the significance of outcomes near $t_{\eta}$.

###### Proof of Theorem 2.

WLOG, let the TCAV score, $S_{c,m,t}$, for concept $c$ and class $m$ at time $t$ be $S_{t}$. The reward function is defined as (Section. 3.3),

$$ $R(t,a)=K\cdot S_{t}\cdot f(t),$ (9) $$

for a constant $K$ and a factor,

$$ $f(t)=\begin{cases}\xi\cdot t&\text{if }t\leq t_{\eta},\\ \xi_{0}&\text{otherwise},\end{cases}$ (10) $$

for positive parameters $\xi$ and $\xi_{0}$.

Case 1: Considering the difference in reward function at time $t$ when $t\leq t_{\eta}$,

$$ $\displaystyle R(t+1,a)-R(t,a)$ $\displaystyle=K\cdot S_{t+1}\cdot f(t+1)-K\cdot S_{t}\cdot f(t)$ (11) $\displaystyle=K\cdot S_{t+1}\cdot\xi\cdot(t+1)-K\cdot S_{t}\cdot\xi\cdot t$ $\displaystyle=K\cdot\xi\cdot(S_{t+1}\cdot(t+1)-S_{t}\cdot t)$ $\displaystyle=K\cdot\xi\cdot(t(S_{t+1}-S_{t})+S_{t+1})$ $\displaystyle\text{From }(S_{t+1}-S_{t})=\frac{h(t+1)-h(t)}{(t+1)-t})=h^{ \prime}(t),$ (12) $\displaystyle R(t+1,a)-R(t,a)$ $\displaystyle=K\cdot\xi\cdot(t\cdot h^{\prime}(t)+S_{t+1}).$ (13) $$

Since,

- 1.
$S_{t}$ is monotonically increasing for $t\leq t_{\eta}\implies h^{\prime}(t)>0$ and
- 2.
$S_{t}\in[0,1]$,

$$ $R(t+1,a)-R(t,a)\geq 0.$ (14) $$

Case 2: Considering the same difference in rewards for $t\geq t_{\eta}$.

$$ $\displaystyle R(t+1,a)-R(t,a)$ $\displaystyle=K\cdot S_{t+1}\cdot f(t+1)-K\cdot S_{t}\cdot f(t)$ (15) $\displaystyle=K\cdot S_{t+1}\cdot\xi-K\cdot S_{t}\cdot\xi_{0}$ $\displaystyle=K\cdot\xi_{0}\cdot(S_{t+1}-S_{t})$ $$

Given that $S_{t}$ and $S_{t_{+}1}$ are both outcomes generated from a generative model fine-tuned for a particular concept, $S_{t+1}-S_{t}\approx 0$ in response to the same action. Hence,

$$ $R(t+1,a)-R(t,a)\approx 0.$ (16) $$

∎

Theorem 2 characterizes the how TCAV scores (i.e., proportional to rewards) are increased up to $\eta$. As a result, as shown in Theorem 3, if the generator moves close to the image class, then the explainer generates images similar to the class. Therefore, by varying $\eta$ we can generate concepts with different levels of abstractions.

###### Theorem 3 .

As we go closer to the concept class, $|\mathcal{C}_{G}\cap\mathcal{C}_{N}|$ becomes larger for generated concepts $\mathcal{C}_{G}$ and NUT’s internal concepts, $\mathcal{C}_{N}$.

###### Proof sketch.

Measure the sensitivity difference between $S_{c_{1},m,t}$ and $S_{c_{2},m,t}$ as $t\to\infty$.
∎

###### Proof of Theorem 3.

At each time step $t$, two sets of samples are generated near $\mathcal{C}_{G}(t)$ using a generative function $g(.)$, denoted by $s_{1}(t)=g(\mathcal{C}_{G}(t))$ and $s_{2}(t)=g(\mathcal{C}_{G}(t))$. We define the sensitivity of these samples to the concept class using a measurable attribute, $\sigma(s)$, that quantifies the alignment or closeness of a sample $s$ to the target concept class.

The optimization step at each time step selects the sample with higher sensitivity, denoted by:

$$ $s_{\text{opt}}(t)=\arg\max\{\sigma(s_{1}(t)),\sigma(s_{2}(t))\}$ $$

The sample with the lower sensitivity is given by

$$ $s_{\text{min}}(t)=\arg\min\{\sigma(s_{1}(t)),\sigma(s_{2}(t))\}$ $$

Sample $s_{\text{opt}}(t)$ and $s_{\text{min}}(t)$ is then used to adjust $\mathcal{C}_{G}$, increasing its overall sensitivity to the concept class. Consequently, the sequence of $\mathcal{C}_{G}$ over time evolves as:

$$ $\mathcal{C}_{G}(t+1)=(\mathcal{C}_{G}(t)\cup\ s_{\text{opt}}(t))\setminus\ s_{ \text{min}}(t)$ $$

This process incrementally increases the sensitivity of $\mathcal{C}_{G}(t+1)$ to the concept class, driven by the iterative inclusion of optimized samples.

Given that $\mathcal{C}_{N}$ is already close to the target concept class, the movement of $\mathcal{C}_{G}$ through this optimization process indirectly steers $\mathcal{C}_{G}$ towards $\mathcal{C}_{N}$. As $\mathcal{C}_{G}$ evolves in this manner, the overlap between $\mathcal{C}_{G}$ and $\mathcal{C}_{N}$ naturally increases, leading to:

$$ $\lim_{t\to\infty}|\mathcal{C}_{G}(t)\cap\mathcal{C}_{N}|\implies\lim_{t\to \infty}\mathcal{C}_{G}(t)=\mathcal{C}_{N}.$ $$

This results from $\mathcal{C}_{G}(t)$ containing more elements that exhibit higher sensitivity similar to those in $\mathcal{C}_{N}$, thereby increasing their intersection.
∎

###### Theorem 1 .

###### Proof sketch.

###### Proof of Theorem 1 .

###### Definition 3 .

###### Property 1 .

###### Theorem 2 .

###### Proof sketch.

###### Property 2 .

###### Proof of Theorem 2.

###### Theorem 3 .

###### Proof sketch.

###### Proof of Theorem 3.

### B.1 Definitions

Entropy: Entropy quantifies the uncertainty or randomness inherent in a probability distribution. For a discrete random variable $X$ with possible outcomes $x_{1},x_{2},\dots,x_{n}$ and corresponding probabilities $P(X=x_{i})=p_{i}$, the entropy $H(X)$ is defined as: $H(X)=-\sum_{i=1}^{n}p_{i}\log p_{i}$, where $p_{i}$ represents the probability of outcome $x_{i}$.

Odds: Odds describe how many times an event is expected to happen compared to how many times it is not. They are often used in gambling, sports betting, and statistics. The odds of an event with probability p (where
p is the probability of the event happening) are calculated as: $\frac{p}{1-p}$.

Exploration Gap (EG): quantifies the proportion of missed optimal actions, defined as $1-\text{Accuracy}$, highlighting how humans frequently miss the most optimal actions when presented with generated concepts.

Average Normalized Count (ANC): The ANC is a measure of the central tendency of the normalized action frequencies within a distribution. It provides insight into how the actions are distributed relative to the overall frequency distribution. A high ANC indicates that, on average, the action frequencies are relatively large, meaning that certain actions are more dominant. Conversely, a low ANC suggests that the actions are more evenly distributed or that lower-frequency actions have a more significant presence. Given by $\frac{1}{n}\sum_{i=1}^{n}\frac{f_{i}}{\sum_{j=1}^{n}f_{j}}$, where $f_{i}$ is the frequency of action $i$.

Inverse Coefficient of Variation (ICV): A standardized measure of concentration, calculated as the ratio of the mean to the standard deviation: $\frac{\mu}{\sigma}$. It represents how many standard deviations fit into the mean.

Feedback Cost: Feedback Cost refers to the resource(GPU) expense associated with obtaining feedback during the training of the model.

Execution Time: Execution time refers to the total time taken by a model or algorithm to complete its task from start to finish. This includes the time for data processing, model computation, and generating outputs.

## Appendix C Methodology

### C.1 Machine learning models we use

Neural Network Under Test ($f(.)$) : We test RLPO for all the different classification models given below.

- 1.
ResNet50: We utilized a pretrained model from PyTorch torchvision pretrained models with weights initialized from ResNet50_Weights.IMAGENET1K_V2.
- 2.
GoogleNet: We utilized a pretrained model from PyTorch torchvision pretrained models with weights initialized from GoogLeNet_Weights.IMAGENET1K_V1.
- 3.
InceptionV3: We utilized a pretrained model from PyTorch torchvision pretrained models with weights initialized from Inception_V3_Weights.IMAGENET1K_V1.
- 4.
Vision Transformer (ViT): We utilized a pretrained model from PyTorch torchvision pretrained models with weights initialized from ViT_B_16_Weights.IMAGENET1K_V1.
- 5.
Swin Transformer: We utilized a pretrained model from PyTorch torchvision pretrained models with weights initialized from Swin_V2_B_Weights.IMAGENET1K_V1.
- 6.
TextCNN sentiment classification model: We utilized a pretrained model from Captum library. The model was trained on IMBD sentiment dataset.

TCAV logistic model : We utilized a logistic regression model to address classification tasks in TCAV instead of the default SGD (Stochastic Gradient Descent) classifier. This decision was based on our observation that the SGD classifier produced high variance TCAV (Testing with Concept Activation Vectors) scores, which indicated inconsistent model behavior across different runs. We configured the model to perform a maximum of 1000 iterations (max_iter=1000).

Stable Diffusion v1-5 with LoRA : We used our base generation model as SD v1-5 and updated its weights using LoRA during preference optimization step. This version of SD was finetuned from SD v1-2 on "laion-aesthetics v2 5+" dataset with 10% drop in text-conditioning for better CFG sampling. In our experiments, we kept LoRA rank to 8 with a scaling factor of 8 and initial weights were defined from a gaussian distribution. We only targeted the transformer modules of U-Net in the SD architecture.

DQN : We use a DQN with specific parameters tailored to effectively navigate a vast search space. We utilized a small buffer size of 100, which limits the number of past experiences the model can learn from, encouraging more frequent updates. The exploration rate was set at 0.95, prioritizing exploration significantly to ensure thorough coverage of the search space. The batch size was configured to 32. We set the discount factor to 0.99 and the update frequency was set at every four steps. The model updates its parameters with a the soft update coefficient of 1.0. Gradient steps was set to 1 indicating a single learning update from each batch, and gradient clipping was capped at 10 to prevent overly large updates.

BLIP: We utilize the Bootstrapped Language Image Pretraining (BLIP) model for the task of Visual Question Answering (VQA). This model, sourced from the pre-trained version available at ’Salesforce/blip-vqa-capfilt-large’, is designed to generate context-aware responses to visual input by leveraging both image and language understanding. The large variant of the BLIP model is fine-tuned for VQA, allowing it to effectively interpret and answer questions based on the visual content provided.

### C.2 RLPO algorithm

Algorithm 2 shows the complete algorithm of the algorithm shown in Section 3 algorithm 1.

Figure: Algorithm 2 DQN Algorithm with DPO and Adaptive Reward

### C.3 Preprocessing for generating the action space

Steps not discussed in Section 3.2.

Figure: Figure 11: Seed prompt pipeline
Refer to caption: x11.png

Each patch from the test images is passed to the VQA model to extract relevant and useful information about the corresponding class. In this study, we choose BLIP Li et al. () as our VAQ model. We posed a set of targeted questions to the VQA model, aiming to gain insights into the class-specific features represented in the patches. The questions are designed to probe various aspects of the image patches, helping the model focus on class-defining attributes.

- 1.
“What is the pattern in the image?”
- 2.
“What are the colors in the image?”
- 3.
“What is the background color of the image?”
- 4.
“What is in the background of the image?”
- 5.
“What is the primary texture in the image?”
- 6.
“What is the secondary texture in the image?”
- 7.
“What is the shape of the image?”

We then remove stop words and duplicates from the generated responses using lemmantizing and perform a cross-similarity check using CLIP between all the unique words and further filtered words which are more than 95% similar. To further select most relevant keywords to the class images, we perform a VLM check using class images and the extracted keyword to get the softmax score of how much the keyword and image are related. This score is then averaged over all the class images and this average is use to sort the keywords. Now, from the sorted keywords, we select top 20 keywords as our RL action space. The cross-similarity and VLM check are inspired from Zang et al. () where they used a similar filtering setup to remove potentially useless concepts.

### C.4 Exploration of Random Gibberish Prompts as Seed Prompts

In this experiment we don’t use a VQA to get seed prompts. We choose a random list of incoherent prompts, example shown in Fig. 12. We found that for these prompts it takes a really long time to get some meaningful explanation and in most cases lead to random generation, thus showing importance of starting from proxy concepts.

Figure: Figure 12: Sample gibberish seed prompt used with RLPO on GoogleNet classifier to generate concepts.

### C.5 Preference optimization update for state space

Steps not discussed in Section 3.4.

The candidate concepts serve as the initial states for the RL agent. From these initial states, the agent takes actions $a\in\text{Keywords}$ that leads to multiple subsequent possible states using $g(.)$. These states are then grouped, and the group’s sensitivity is compared against INUT using TCAV scores. A higher TCAV score suggests higher sensitivity, indicating that the group is more aligned with INUT.

We employ preference optimization over the grouped states to guide states towards INUT. To prevent the model from skipping over explainable states and directly reaching INUT, we introduce a threshold that limits the application of preference optimization at each step as shown in equation 17.

$$ $\displaystyle\text{Given two groups of samples }G_{1}\text{ and }G_{2}\text{ with their average TCAV scores }\overline{TCAV}_{1}\text{ and }\overline{TCAV} _{2}:$ $\displaystyle\text{if }\max(\overline{TCAV}_{1},\overline{TCAV}_{2})\leq 0.7, \text{ update }\pi\text{ to favor the group with higher }\overline{TCAV}.$ (17) $$

To optimize $g(.)$ to find better proxies, for each step in the environment we utilized average TCAV scores $\overline{TCAV}_{1}$ and $\overline{TCAV}_{2}$ from $G_{1}$ and $G_{2}$ to decide between preferred and unpreferred concepts. Lets say $\overline{TCAV}_{1}\succ\overline{TCAV}_{2}$, than we optimize $g(.)$ over the sample S defined as $S=\{(a,x^{g1}_{0},x^{g2}_{0})\}$, where $x^{g1}_{0}$ and $x^{g2}_{0}$ are the sample points from the groups on action $a$. We optimize $g(.)$ using objective 18 to get a new optimzed $g^{\prime}(.)$  Wallace et al. ().

$$ $\displaystyle L(\theta)=$ $\displaystyle-\mathbb{E}_{(x^{g1}_{0},x^{g2}_{0})\sim S,t\sim U(0,T),x^{g1}_{t }\sim q(x^{g1}_{t}|x^{g1}_{0}),x^{g2}_{t}\sim q(x^{g2}_{t}|x^{g2}_{0})}\log \sigma\left(-\beta T\omega(\lambda_{t})\right.$ $\displaystyle\left.\left(\|\epsilon^{G1}-\epsilon_{g^{\prime}(.)}(x^{G1}_{t},t )\|_{2}^{2}-\|\epsilon^{G1}-\epsilon_{g(.)}(x^{G1}_{t},t)\|_{2}^{2}\right.\right.$ $\displaystyle\left.\left.-\left(\|\epsilon^{G2}-\epsilon_{g^{\prime}(.)}(x^{G2 }_{t},t)\|_{2}^{2}-\|\epsilon^{G2}-\epsilon_{g(.)}(x^{G2}_{t},t)\|_{2}^{2} \right)\right)\right)$ (18) $$

where $x^{*}_{t}=\alpha_{t}x^{*}_{0}+\sigma_{t}\epsilon^{*}$, $\epsilon^{*}\sim\mathcal{N}(0,I)$ is drawn from $q(x^{*}_{t}|x^{*}_{0})$. $\lambda_{t}={\alpha^{2}_{t}}/{\sigma^{2}_{t}}$ is the signal-to-noise ratio, and $\omega(\lambda_{t})$ is weighting function (constant in practice).

### C.6 TCAV setting for different NUT models

We tested NUT models on different layers and classes and the summary of our setting across different NUT models is described in table 5.

**Table 5: TCAV setting across different NUT models**
| Models | Layers | ImageNet Classes |
| --- | --- | --- |
| ResNet50 | penultimate layer | Jay, Tiger, Rabbit & Zebra |
| GoogleNet | inception4e layer | Goldfish, Tiger, Zebra & Police Van |
| InceptionV3 | Mixed_7c layer | Goldfish, Tiger, Lionfish & Basketball |
| Vision Transformer (ViT) | heads layer | Goldfish, Golden Retriever, Tiger & Cab |
| Swin Transformer | head layer | Goldfish, Jay, Siberian husky & Tiger |

## Appendix D Experiments

### D.1 Computing resources

The experiments were conducted on a system equipped with an NVIDIA GeForce RTX 4090 GPU, 24.56 GB of memory, and running CUDA 12.2. The system also featured a 13th Gen Intel Core i9-13900KF CPU with 32 logical CPUs and 24 cores, supported by 64 GB of RAM. This setup is optimized for high-throughput computational tasks but the experiments are compatible with lower-specification systems.

### D.2 Human and LLM-based AI Feedback Mechanisms

We test other feedback mechanism in RL by replacing the XAI-TCAV feedback with AI and Human feedback’s. Herer we discuss the experiental setup and configuration for both experiements.

AI feedback: GPT-4 is leveraged to evaluate the explanatory power of image sets by focusing on concepts related to a target class, a method that aligns with the growing trend of incorporating AI-driven feedback Bai et al. (). This approach involves sending a structured prompt to an LLM, asking it to score how well two sets of images explain a target class using a specified concept. The process involves the following.

- 1.
Image Encoding: The images from two sets (concept1 and concept2) are first converted into a base64 format to ensure they can be transmitted via the request as encoded strings.
- 2.
Structured Prompt: A detailed and specific prompt is crafted for the LLM. It asks the model to assess the quality of explanation each image set provides for a particular class through the lens of a specific concept. The prompt used is “Please evaluate each of the following sets of images for how well they explain the class {class_name} via the concept {concept_name}. For each set, provide a numerical score between 0 and 1 (to two decimal places)” The prompt clearly defines how the model should respond, asking for a numerical score between 0 and 1, where:
(a)
0 indicates that the image set does not explain the class at all via the concept.
(b)
1 indicates that the image set perfectly explains the class via the concept.
- 3.
LLM-Based Scoring: Once the prompt is sent to the LLM, it evaluates the image sets and provides scores based on its learned knowledge and understanding. The response is parsed to extract the scores for each set of images.

Human feedback: In this experiment, 7 computer science majors provided live feedback after each step of a reinforcement learning process, leveraging their prior knowledge of reinforcement learning with human feedback (RLHF) mechanisms Christiano et al. (). The feedback from all participants was averaged to serve as the reward for each step in the RL process. Given the abstract nature of the initial concepts, participants needed to take time to thoughtfully assess each step, which contributed to a lengthier feedback cycle.

### D.3 Additional results and analysis

To validate our method for its ability to generate concepts, we tested it with different models and classes. We started it on traditional models, GoogleNet and InceptionV3, and then extended it to transformer-based models, Vision Transformer (ViT) and Swin Transformer, pre-trained on ILSRVC2012 data set (ImageNet) Krizhevsky et al. ().
We show additional plot in various classes shown in Fig 13,14,15,16,17.

Figure: Figure 13: Explanation plot of Cab classification by ViT from RLPO.
Refer to caption: x12.png

Figure: Figure 14: Explanation plot of Basketball classification by InceptionV3 from RLPO.
Refer to caption: x13.png

Figure: Figure 15: Explanation plot of Lionfish classification by InceptionV3 from RLPO.
Refer to caption: x14.png

Figure: Figure 16: Explanation plot of Golden Retriever classification by ViT from RLPO.
Refer to caption: x15.png

Figure: Figure 17: Explanation plot of Tiger classification by GoogleNet from RLPO.
Refer to caption: x16.png

#### D.3.1 Cumulative rewards

The cumulative rewards during training for GoogleNet and InceptionV3 is shown in Fig. 18. For ViT and Swin Transformer it is shown in Fig. 19. This figure illustrates the steady accumulation of rewards over time as they interact with the reinforcement learning environment. All models demonstrate a steady increase in cumulative rewards, the classes with higher reward peak reaches its explinable state faster.

Figure: Figure 18: Cumulative rewards on traditional models.
Refer to caption: x17.png

Figure: Figure 19: Cumulative rewards on transformer models.
Refer to caption: x19.png

#### D.3.2 Action Selection Optimization During RLPO Training

Figure: Figure 20: Combined actions (multiple keywords) count over training time
Refer to caption: x21.png

As shown in Fig. 20, during training with multiple combinations of seed prompts, we observe that the RL agent initially explores various action combinations. However, as training progresses, individual actions become more optimized due to preference optimization (PO). This leads the agent to prefer fewer action combinations, since just choosing one or two actions makes the agent reach an explainable state.

#### D.3.3 Concept heatmap

To determine the relationship between generated concepts and test images, we made use of CLIPSeg transformer model (Lüddecke & Ecker, ). We passed generated concepts as visual prompts and test images as query images into the model and it returns a pixel-level heatmap of the probability of visual prompt in the query image. Fig. 21, 22 showcases some examples on concept heatmap indicating the presence of the concept in the image.

Figure: Figure 21: Van class with “white blue and yellow”, Lion fish class with “zebra” seed prompt.
Refer to caption: x22.png

Figure: Figure 22: Basketball class with “basket” seed prompt, Tiger class with “orange black and white” seed prompt.
Refer to caption: x24.png

#### D.3.4 C-deletion

The central idea behind c-deletion in explainability is to identify and remove parts of the input context that are not crucial for the decision-making process, allowing for clearer insights into how the model arrives at its predictions or actions.

Figure: Figure 23: The figure shows c-deletion taking place for different images from “tiger” class over time for “orange black and white” seed concept.
Refer to caption: x26.png

Figure: Figure 24: The figure shows c-deletion taking place for different images from “cab” class over time for “yellow and white” seed concept.
Refer to caption: x27.png

Figure: Figure 25: C-deletion. Removing concepts over time to measure the reliability. The colored numbers indicate the area under the curve (the lower the better).
Refer to caption: x28.png

C-deletion evaluations assesses the impact of removing certain contextual inputs (features, variables, or states) on a model’s performance as shown in Fig. 25.

### D.4 RLPO in sentiment analysis

We extended our method to the NLP domain, successfully identifying which parts of the input contribute to specific outputs. For sentiment analysis, a binary classification problem, we present results for both positive and negative classes.

A list of positive and negative prompts was created, analogous to class images in traditional image classification tasks. Random prompts, similar to those in Fig. 12, were used to simulate random classes. Every word in the prompt, excluding stop words, along with its synonyms, was treated as a concept for this experiment. Synonyms were generated using the Mistral-7B Instruct model, serving a role comparable to the image generation model in image-based settings. We observed that multiple words were identified, along with their influence on the overall prompt, for both classes as shown in Fig. 10 and Fig. 26.

Figure: Figure 26: Generated concepts explain why a given text is classified as a negetive sentiment.
Refer to caption: x30.png

### D.5 Effects of Fine-tuning

When fine-tuning a model, the optimization process updates its weights through gradient-based methods, causing shifts in the concepts (Fig. 9) it learns. These weight adjustments modify how the model attends to different regions or patterns in an image, leading to changes in the internal activation maps and the conceptual understanding of the input. As the model learns new concepts or refines existing ones, it adjusts its feature extraction and decision-making processes to better align with the specific objectives of the fine-tuning task, thereby altering the way it interprets and generates outputs.

To demonstrate this experiment, a ResNet50 model, defined in Appendix C.1, was used. We fine-tuned only the final layer of the neural network by setting the learning rate to 0.001 and momentum to 0.9 using the Stochastic Gradient Descent (SGD) optimizer. Maintaining a low learning rate was crucial to preserving high accuracy. RLPO was then applied to the fine-tuned ResNet50, and a shift in the learned concepts was observed, as shown in Fig. 9. This process highlights the model’s sensitivity to fine-tuning and how training on a subset can shift its conceptual interpretation of images.

### D.6 Human survey

The survey involved 50 participants, each of whom was shown 10 class images along with two concept options as shown in Fig. 27: one derived from a retrieval-based method and the other generated using RLPO. The participants were divided into Laymen and Experts.

- 1.
Expert: Computer science graduates who are familiar with the concept of explainability and have a working knowledge of AI or machine learning systems.
- 2.
Laymen: Individuals without expertise in computer science, AI, or explainability, representing the general public’s perspective.

Figure: Figure 27: A screenshot from our human survey with instructions and a sample question.
Refer to caption: extracted/5897366/figs/Survey.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. While Tabel 4 shows the concept images generated by RLPO are more diverse, how do we know if they are faithfully reflecting the the same concept instead of overfitting the TCAV?
2. Why should the problem be designed as a sequential decision-making problem? 
3. Do people need to finetune one RLPO framework with DQN+Diffusion for each class? Would it be more efficiently and equally effective if they just use an untuned Diffusion model and prompt LLMs/VLMs to generate more detailed text descriptions for the target class given the seed prompt?
4. What is $t_\eta$ in Property 2?
soundness: 2
strengths: 1. Optimizing diffusion models to maximize TCAV scores sounds interesting and original to me.
2. The authors performed extensive experiments to demonstrate the effectiveness RLPO.
3. Most parts of the paper are clearly written and easy to follow.
4. The visual results presented are helpful in illustrating the advantages introduced by RLPO.
summary: This paper focuses on the generation of concept images to explain black-box image classification models. It proposes a reinforcement learning-based preference optimization (RLPO) algorithm to fine-tune a diffusion model for generating images that can maximize the TCAV scores. It also proposes to use DQN to search appropriate actions from the seed prompts. Experiments show that the proposed approach can generate complex and abstract concepts aligning with the test class.
weaknesses: 1. While diffusion models make the whole system more flexible than "retrieval methods" by generating new images that may not be presented in current datasets, the quality of such images is questionable. For example, in Figure 5, the generated image at timestep 30 looks unrealistic and is not closely aligned with the seed prompt (worse than the initial image).
2. Though Table 1 shows RL helps the model to select approprate seed prompts from the given set, I'm still concerned about the necessity of using DQN in the framework. VLM/LLM can be used to generate a small high-quality seed prompt set, and even rank them based on their potential importance (e.g., the experiment in this paper has a set with only 20 prompts). One can use the whole set and focus on the fine-tuning of the diffusion model. 
3. The efficiency seems to be a problem.

---

confidence: 2
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see my comments on the weakness section.
soundness: 3
strengths: The paper is well-written and easy to follow. The problem definition is novel and interesting.  The paper effectively reframes the generation of concept image sets as an image generation problem, providing a novel perspective that addresses the limitations of traditional concept-based explanation methods. This shift enhances the efficiency and effectiveness of generating meaningful concepts.
Through a series of well-designed experiments, the paper demonstrates the capability of the proposed method to generate complex and abstract concepts that align with the target class. This empirical evidence strengthens the paper's contributions.
summary: Concept-based explanation methods typically require practitioners to guess and gather multiple candidate concept image sets, a process that can be imprecise and labor-intensive. To address this challenge, this paper redefines the creation of concept image sets as an image generation problem. To this end, in this work, the authors introduce an RL-based preference optimization algorithm that fine-tunes the vision-language generative model using approximate textual descriptions of concepts. They also conduct extensive sets of experiments to demonstrate that the proposed method effectively articulates complex and abstract concepts that align with the target class, which are often difficult to create manually.
weaknesses: While the authors demonstrate the effectiveness of their proposed method through both qualitative and quantitative analyses, several aspects of the framework are not thoroughly justified. For instance:

(a) The generated concept sets using SD+LORA are randomly divided into two groups. Why is "random" grouping considered optimal? Could this be problematic?

(b) The use of reinforcement learning (RL) is also not justified. Is it really necessary to implement an RL policy for the purpose mentioned in the paper? Why not simply calculate the TCAV score for each possible seed prompt kt, and then fine-tune the SD+LORA weights based on the seed prompt that yields the highest TCAV value?

(c) Incorporating each component of the proposed framework increases computational demands and time constraints. Consequently, such an analysis may not be feasible in real-time, as generating images with Stable Diffusion, training the DQN-based RL policy, and fine-tuning the Stable Diffusion model with preferences all require significant processing time. Is this complex pipeline truly necessary?

(d) I also struggle to see a practical significance for the proposed problem. Why is it important to generate a diverse set of concept images at all? It seems more crucial to generate concepts that directly explain the task at hand.

(e) Furthermore, why choose DQN? As we continuously update the LORA weights based on the TCAV preference score, the underlying RL environment becomes non-stationary. This means that the same action taken by an RL policy could lead to different reward values at different times. How can this issue be addressed?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Same as weakness
soundness: 3
strengths: The idea is very innovative, and the writing is clear. By using a generative model instead of traditional manual concept collection, it reduces human intervention and improves efficiency. RLPO can generate concepts at different levels of abstraction, offering a more detailed explanation of the DNN’s internal decision-making process.
summary: Authors proposed a new algorithm, Reinforcement Learning-based Preference Optimization (RLPO), designed to generate high-level visual concepts that explain the decisions of DNNs. Unlike traditional concept-based explanation methods (such as TCAV), RLPO creates sets of concept images, eliminating the need for manual concept image collection, thus making the process of explaining DNN decisions more efficient and generalizable. RLPO fine-tunes a stable diffusion model with preference optimization to generate images that effectively explain neural network decisions.
weaknesses: In some sections (such as the algorithm explanation and mathematical proofs), the descriptions appear overly lengthy, which might hinder readers' understanding. The description of how RLPO is applied in sentiment analysis tasks is somewhat vague, and further detailing the specific steps of this experiment could be helpful. Certain terms (such as “concept generation” and “concept extraction”) are defined and used inconsistently throughout the paper, which could lead to confusion.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Questions:
1. “As a specific application, we see what concepts are removed and added, as well as how the concept importance changes when we fine-tune ResNet50 model on ImageNet to improve accuracy” If we want to track the evolution of concepts during the training procedure, do we need to retrain the concepts every time we update the model, or is there any workaround which would help track the concepts during the finetuning process? Either way is fine, however might be good to clarify it in the paper or in the appendix. 
2. Figure  9 should have the x axis annotated ( I understand it represent concept identifier)?
3. One of the concerns might be the lack of systematic quantitative comparison with the retrieval-based explanations. Is there any possibility to compare the explanations numerically with the state-of-the-art prototypical retrieval-based explanation methods, such as, e.g. Craft (Fel et al, 2023)?
4. I think there is one important limitation of the proposed method which I would like the authors to cover in the paper: the explanations also, in a way, depend upon the inner working of the generative model. Imagine, for example, that the generative model has mode collapse and does not represent the whole set of patterns available to the model we explain. In this case, it would lead us to obtaining the best possible explanation amongst the suboptimal ones, which may lead to the explanations not representing some aspects of the model’s inner working. In a case when it is a safety-critical application, that mode collapse might represent some anomalous event, which may not be covered by the set of explanations achievable by the model and therefore would not allow us to understand the reasons behind the model. A variation of such problem might include the problem of explanation when the generator only offers poor quality of image output(for whatever reason, e.g. it does not cover some particular concept). To make it clear, I understand that there is also a counterpart to such limitation in a case of standard, retrieval-based, prototypical explanation: there might not be such a piece of data which would closely match the phenomenon. It is therefore, in one way or the other, a limitation of many post hoc explanation models. I wonder if the authors agree with such limitation, and in any case would like to ask to include  the discussion. 
5. In relation to that, perhaps not mandatory, but another idea of an experiment: how does the numerical performance of the algorithm, e.g. Figure 8, would compare for different image generators? Does the GAN model, which is prone to mode collapse, result in worse C-insertion metrics?
soundness: 3
strengths: Novelty: the work builds upon existing methods in interpretability, such as concept based explanation, but contains a set of novel ideas including (1) the problem statement and the method for optimisation of concept set through reinforcement learning technique and LoRA (2) the idea to use such concept set generation for to produce explanations for different abstract layers

Clarity: the paper is clearly written and well presented. 

Motivation: The authors give a strong motivation for the proposed method: the retrieved data might not filly describe the concepts embedded into the model, and the optimisation of such concepts using generative models and RL can be a promising alternative

Significance: I think this work is significantly improving the available options for explaining visual (and potentially language) models and therefore I believe it is significant enough. 

Correctness: I checked the paper and I believe the claims and the maths are correct

Reproducibility: as far as I can see, taking into account both the paper and the code, the experiments look reproducible to me.
summary: The paper proposes to treat the concept set creation as an optimisation problem. Through the means of deep reinforcement learning, it iteratively refines the concepts obtained using Stable Diffusion generative model so one can generate concepts instead of retrieving them in the existing data. They also introduce the concept generation with respect to an abstraction level. The authors also explore the ideas of analysing the concepts during fine-tuning and NLP problems.
weaknesses: Clarity: there are a few questions regarding the limitations of this work (see below in the questions section)

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: Maybe I missed it, but is there any information on the training time of the proposed algorithm? If not, I think this could be valuable information to provide at least in the appendix, but mentioned in the main text. The authors mentioned limitations, but it would be good to have real numbers.

Whats the difference between the concept images in Figure 6 and 5? In figure 5 they look very abstract, whereas in figure 6 the are high-quality images.
soundness: 3
strengths: The paper tackles an important topic, and I find the combination of previous approaches that the authors propose with their work intuitive and interesting.
summary: This paper introduces an algorithm for generating concept images to post-hoc explain a model. It hereby focuses on the use of the TCAV score, an RL learning setting and stable diffusion.
weaknesses: Although i find the algorithmic approach interesting enough, I am unsure what the actual underlying goal is of the approach and how this is evaluated/evidenced in the experimental section. I.e., in the intro the authors mention "Therefore, it is importantto automatically find human-centric concepts that matter to the DNN’s decision-making process." So my understanding is that the ultimate goal is to provide concepts that are helpful for humans to understand the decision making processes, but without the need to provide extensive concept data precollection. But I am missing a clear experiment to evaluate the usefulness of the generated concept images. Table 2 seems to be hinting at something along this line, but the details are not clear to me from the text. IN any case this should be one of the key evaluations to perform. Especially, as the exemplary concept images look far too abstract for me to be able to understand what they should represent. Please point me towards the relevant sections in case this is missing. 

Overall, I am leaning towards accept, but would like the following points clarified first:
- What is the exact goal of the method?
- How do the experimental evaluations provide evidence for reaching this goal?

Minor: Overall, the paper could benefit from a grammar check, e.g., "also explains what type of features is the model focuses on." in line 426

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: The most important question is: can the authors provide any experimental result that concretely shows that their method is able to explain model decisions? 

Some secondary questions are: 
1) Why RL at all? Why not just optimize every "seed prompt"?
2) How often do seed prompts converge to the same outputs? 
3) This work brings up the question, under what domain should we care about explaining model behavior? The generated images are OOD for the model. When do we want to explain OOD behavior for the model? Is this useful?
4) How does your method relate to adversarial attack methods?
5) How long does it take for your method to run?
soundness: 1
strengths: (S1) The authors identify an interesting question: how to choose a probe set for extracting concepts from models? 
The proposed method considers an interesting direction of using generative methods to create images that may not be present in the dataset. 
(S2) The formulation of considering TCAV scores as preference scores is interesting. 
(S3) The authors make an effort to quantitatively assess the concept images generated by their method.
summary: Some concept-based XAI methods, such as [1,2], require the creation of a concept-specific set of images to pass through the network. To eliminate the human-in-the-loop, the authors explore using a combination of reinforcement learning and generative models to create concept-specific image sets. The authors stated goals include:
 (1) producing concept sets that are beyond what human practitioners may be able to discover on their own. 
 (2) producing concepts at a variety of abstraction levels that they are able to control with a parameter. 
 (3) producing concept sets that demonstrate an increases in novelty, abstractness, diversity, generalizability (to non-vision domains) and actionability (utility).

The core of the method is to optimize a LORA-adapted stable diffusion model to produce concept sets that trigger the target model (model to be explained). The RL agent is used to search a set of text prompts for the diffusion model for the text prompts. 

[1] Kim, Been, et al. "Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)." International conference on machine learning. PMLR, 2018.
[2] Schut, Lisa, et al. "Bridging the human-ai knowledge gap: Concept discovery and transfer in alphazero." arXiv preprint arXiv:2310.16410 (2023).
[3] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685 (2021).
weaknesses: (W1) The authors stated goals for this work are misaligned with the goals of explainable AI in general. The  goal of XAI is to improve a user's understanding of a model as a primary goal. However, the authors focus on "abstractness" and "novelty" as the goals of their method. While the authors propose several experiments to measure abstractness and novelty, they fail to provide any convincing experiments on the utility of their method for explaining network behavior. For example, [1] provides a clear, well-motivated experiment, i.e. users are asked to predict model behavior when given an explanation.

(W2) Additionally, I find the term novelty to be misleading. The term novelty can be interpreted in at least two ways, for example, it could be interpreted as how distinct different discovered concepts are from each other. When measuring novelty in this manner, it seems that the author's method may not exhibit novelty, since in Line 288 they state, "it is possible for different actions to lead to the same explainable state." Instead, the authors measure novelty as the distance between concept images and images the test set. They use this metric to compare their work to prior work, resulting in the fairly trivial result that generated images are farther from the test set than methods that use the test set. 

(W3) The ablation study for choosing TCAV is unnecessary. The authors choose to use either a human, LLM, or TCAV scores from the model to make preference judgements. However, since the goal is to explain the model the other first two options are completely unnecessary. 

(W4) The authors claim to be able to generate concepts sets at different levels of "abstraction". However, whether this "abstraction" is due to SD or due to the target model is unclear. Since, the goal is to understand the target model, once again, I find this experiment to be insufficient to say anything interesting about the target model. 

(W5) Finally, the computational cost of this method is quite high and it is unclear if the results are worth the cost. 

In summary, the authors lack a key experiment measuring the utility of their method. Additionally, I find that some experiments that the authors conduct to have trivial results. 

There are several places with typos. 
L91 mode -> model
L534 the term NUT is introduced without being defined?

[1] Colin, Julien, et al. "What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods." Advances in neural information processing systems 35 (2022): 2832-2845.

Meta-review: Scientific Claims and Findings:

This paper introduces a novel algorithm called Reinforcement Learning-based Preference Optimization (RLPO) to automatically generate visual concepts that explain the decisions of deep neural networks (DNNs).  The algorithm addresses the limitation of traditional concept-based explanation methods (like TCAV) which require manual collection of concept images.  The authors claim that RLPO generates more novel and abstract concepts compared to traditional methods, offering a detailed understanding of the DNN's decision-making process.  They also suggest that RLPO is generalizable to non-vision domains, specifically NLP tasks.    

Strengths:

The paper is well-written and easy to follow.    

The proposed method is novel and addresses a significant limitation in existing XAI methods.    

The use of a generative model to create concept sets is innovative and reduces human intervention.    

The method can generate complex and abstract concepts, offering a detailed explanation of the DNN's decision-making.    

Weaknesses:

Some reviewers found the explanations of the algorithm and mathematical proofs to be overly lengthy.    

The practical significance of generating a diverse set of concept images wasn't immediately clear to all reviewers.    

The necessity of using RL and the complexity of the pipeline were questioned.    

The paper could benefit from a grammar check and consistent use of terms.    

Missing elements:

Clear and concrete examples of how the generated concepts can be used to debug or improve DNNs.

A more thorough justification for design choices, such as the use of RL and DQN.

A clearer explanation of how the method is applied in sentiment analysis tasks.

Reasons for Rejection:

The paper does not adequately demonstrate the practical usefulness of the proposed method for explaining network behavior.    

The experiments focus heavily on novelty and abstractness, but their connection to improving user understanding is not well established.    

The paper lacks a key experiment directly measuring the usefulness of the generated concepts for explaining model decisions.

Decision: Reject