Explainable Concept Generation through Vision-Language Preference Learning for Understanding Neural Networks' Internal Representations
Authors: Aditya Taparia
Som Sagar
Ransalu Senanayake
Venue: ICML 2025 poster (2025)
Keywords: Concept based Explainable AI
TCAV
Vision-Language Models
Reinforcement Learning
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: A method to automatically articulate concepts to explain neural networks.

Abstract: Understanding the inner representation of a neural network helps users improve models. Concept-based methods have become a popular choice for explaining deep neural networks post-hoc because, unlike most other explainable AI techniques, they can be used to test high-level visual "concepts" that are not directly related to feature attributes. For instance, the concept of "stripes" is important to classify an image as a zebra. Concept-based explanation methods, however, require practitioners to guess and manually collect multiple candidate concept image sets, making the process labor-intensive and prone to overlooking important concepts. Addressing this limitation, in this paper, we frame concept image set creation as an image generation problem. However, since naively using a standard generative model does not result in meaningful concepts, we devise a reinforcement learning-based preference optimization (RLPO) algorithm that fine-tunes a vision-language generative model from approximate textual descriptions of concepts. Through a series of experiments, we demonstrate our method's ability to efficiently and reliably articulate diverse concepts that are otherwise challenging to craft manually.

Explainable Concept Generation through Vision-Language Preference Learning for Understanding Neural Networks' Internal Representations
Aditya Taparia Som Sagar'
Ransalu Senanayake
Abstract
revelations help engineers in further improving models and regulatory bodies in assessing their correctness. To help these users, developing methods that communicate infor- mation in a human-centric way is essential for ensuring usefulness.
Understanding the inner representation of neural network helps users improve models. Concept- based methods have become a popular choice for explaining deep neural networks post-hoe be- cause, unlike most other explainable AI tech- niques, they can be used test high-level vi- sual "concepts" that are not directly related to feature attributes. For instance, the concept of "stripes" is important to classify an image as a zebra. Concept-based explanation methods, how- ever, require practitioners to guess and manually collect multiple candidate concept image sets, making the process labor-intensive and prone overlooking important concepts. Addressing this limitation, in this paper, we frame concept im- age set creation as an image generation problem. However, since naively using a standard gener- ative model does not result in meaningful con- cepts, we devise a reinforcement learning-base preference optimization (RLPO) algorithm that fine-tunes a vision-language generative model from approximate textual descriptions of concepts. Through a series of experiments, we demonstrate our method's ability to efficiently and reliably articulate diverse concepts that are otherwise challenging to craft manually. Github: https: //github. com/ adi aadityy-ra tettttaatttttaaa tappriaaaaaaaa....
Humans utilize high-level concepts as a medium for pro- viding and perceiving explanations. In this light, post-hoc concept-based explanation techniques, such Testing with Concept Activation Vectors (TCAV) (Kim et al., 2018), have gained great popularity in recent years. Their ability to use abstractions that are not necessarily feature attributes or some pixels in test images helps with communicating these high-level concepts with humans. For instance, as demonstrated in TCAV, the concept of stripes is important to explain why an image is classified as a zebra, whereas the concept of spots is important to explain why an image is classified as jaguar. Given 1) set of such high-level concepts, represented as sample images (e.g., a collection of stripe and spot images) and 2) test images of the class (e.g.. zebra images), TCAV assigns a score to each concept on how well the concept explains the class decision (i.e., zebra). Although concept-based explanations tation, their requirement to create collections of candidate concept sets necessitate the human to know which concepts to test for. This is typically done by guessing what con- cepts might matter and manually extracting such candidate concept tests from existing datasets. While the stripe-zebri analogy attractive that stripes is important to predict zebras, in most applica- tions, cannot guess what concepts to test for, limiting the usefulness of concept-base based methods in testing real-world systems. Additionally, even a human can guess a few concepts, it does not encompass most concepts a DNN has learned because the DNN was trained without any human intervention. Therefore, important automatically find concepts that matter the DNN's decision-making process.
good represen-
example, where it is obvious
1. Introduction
In an era where black box deep neural networks (DNNs) are becoming seemingly capable of performing complex tasks, our ability understand their internal representations post-hoc has become even more important before deploy- ing them in the real world. Among many use cases, such
As attempts to automatically discover and create such con- cept sets, several works has focused on segmenting the image and using these segments as potential concepts, either directly (Ghorbani et al., 2019) or through factor analy- sis (Fel et al., 2023; 2024). In such methods, which we refer to retrieval methods, because the extracted concept set
'Scoool of Computing and Augmented Intelligence, Arizona State University, Tempe, United States of America. Correspon- dence to: Aditya Taparia <ataparia @ edu..
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Explainable Concept Generation through Vision-Language Preference Learning
sam
Additionally, as use cases, we show how generated concepts can be utilized to improve models, understand model bias, and also demonstrate the generalizability of RLPO on tasks such as NLP sentiment analysis.
Human-delined concepple
Generated compepts.
Notably, due to the modularity of our approach, any compo- nent in our framework (c.g.. SD) can be seamlessly replaced with an improved version (e.g.. SD3 (Esser et al., 2024) or Flash Diffusion (Chadebec et al., 2025)).
Neural natworis' concepts
Rerrieved concepts
2. Preliminaries and Related Work
Testing with Concept Activation Vectors (TCAV): The TCAV score quantifies the importance of a "concept" for specific class in DNN classifier (Kim et al., 2018). Here, concept is defined broadly as a high-level, human- interpretable idea such as stripes, sad faces, etc. A con- cept (e.g., stripes), c, is represented by sample images, X (e.g., images of stripes). In TCAV, a human has manually collected these sample concept images based on educated guesses, whereas our objective is automatically gener- ate them. For given set of test images, Xm (e.g., zebra images), that belong to the same decision class (c.g.. ze- bra), m, TCAV is defined as the fraction of test images for which the model's prediction increases in the "direction of the concept." By decomposing the DNN under test as f(x) fe(fi (z)), where (zz) is the activation layer I, TCAV score is computed as,
Figure Humans can imagine (green) a few concepts to understand neural networks' representations (purple). Some other concepts can be retrieved from images themselves through, for instance, segmentation (orange). However, if we generate concepts (blue), they will capture even broader set of concepts. RLPO is designed to make this generation process more targeted toward neural networks' representations (purple n blue).
already part of the test images shown in Fig. 1 (Retrieved concepts), it is difficult for them to imagine concepts that do not have a direct pixel-level resemblance to the original im- age class. For instance, it is more likely that such methods provide patches of zebra as concepts instead of stripes.
By departing from existing concept creation practices of human handcrafting and retrieval, redefine concept set creation as concept generation problem. Modern gen- erative models such as stable diffusion (SD) can be used to generate realistic images. Nevertheless, since a genera- tive model generates arbitrary images, we need to carefully guide the SD to generate explanatory images. One obvious approach is engineer long, descriptive text prompts to gen- erate concepts. However, engineering such prompts is not realistic. Therefore, automate this process, as shown in Fig. 2, we propose a method, named reinforcement learning- based preference optimization (RLPO), to guide SD model. At its core, we devised a deep reinforcement learning al- gorithm, which gradually update SD weights to generate concept images that have a higher explanation score. The contributions of this paper be summarized as follows:
TS mm
Σ
doutput Oactivations
(c direction) > >0)
Xm
( (f((z.) (ff(z;)) 02>
Σ
Xm
2.EXm
(I)
Here, I is the indicator function that counts how often the directional derivative is positive. Concept activations vector (CAV), v, is the normal vector to the hyperplane that sepa- rates activations of concept images, (f) x e X.). from activations of random images, (Ji( E X,). Refer to Appendix B.1 for details on the TCAV settings.
ACE (Ghorbani et al., 2019) introduced way to automati- cally find concepts by extracting relevant concepts from the input class. It used segmentation over different resolution to get pool of segments and then grouped them based on similarity to compute TCAV scores. Though the ACE concepts are human understandable, they are noisy because of the segmentation and clustering emorr. As different method, EAC (Sun et al., 2024) extracts concepts through segmentation. CRAFT (Fel et al., 2023) introduced re- cursive strategy to detect and decompose concepts across layers. Lens (Fel et al., 2024) elegantly unified concept ex- traction and importance estimation as dictionary learning
1. We propose a method, named RLPO,, to "generate" concepts that truly matter to the neural network (refer to the video on github). These include concepts that are challenging for humans retrieval methods to anticipate (Fig. 1).
2. We qualitatively (computational metrics and human surveys) and guantitatively demonstrate that RLPO can reveal networks' internal representations.
2

---

Explainable Concept Generation through Vision-Language Preference Learning
Random Stripes
Lees Random
Zebra Stripes
Zebra
Generated Explanations
Input Image
Neural networks"
Internal concepts
C
S
(a)
(b)
Figure 2: (a) Our proposed algorithm, RLPO, iteratively refines the concepts G; that are generated by Stable Diffusion (SD) model by optimizing SD weights based on an action ag Each step in this update process provides an explanation at a different level of abstraction (Appendix C.4). Please refer to the supplementary video for better understanding. (b) Three concepts identified by our approach for the zebra class. Concepts are represented as images generated by SD.
problem. However, since all these methods obtain concepts from class images, the concepts they generate tend be very similar the actual class (e.g., a patch of zebra as concept explain the zebra, instead of stripes concept), making it challenging to maintain the "high-level abstract- ness" of concepts, In contrast, we generate concepts from a generative model.
age or text generation tasks, where the dataset for human preferred outputs are readily available, it is hard to have general enough dataset for XAI tasks. To counter this problem, provide preference information by using the TCAV score instead of human, and use it to align the text- to-image generative model to generate concept images that matters for the neural network under test.
Deep Q Networks (DQN): DQN (Mnih et al., 2015) a deep RL algorithm that combines Q-learning with deep neural networks. It is designed to learn optimal poli- cies in environments with large state and action spaces by approximating the Q-value function using a neural net- work. A separate target network, Quarer (s, a', 0²), Here a' is arg max Q (8nent,, a) which is a copy of the Q-network with parameters 0', is updated less frequently to provide stable targets for Q-value updates.
Use of VLM in Explanation: Recent advancements in Vision LanguaLL models (VLMs) have open the doors for the use of VLMs in multiple domains, mainly because of their ability generalize over large amount of data, they can be leveraged to obtain useful information. Work by (Sun et al., 2024) present a novel method combining the Segment Anything Model (SAM) with soncept-based explanations, called Explain Any Concept (EAC). This method uses SAM for precise instance segmentation to automatically extract concept sets from images, then it employs lightweight surrogate model efficiently explain decision made by any neural network based extracted concepts. Another work by Yan et al. (Yan et al., 2023) introduced Learning Con- cise and Descriptive Attributes (LCDA), which leverages Large Language Models (LLMs) to query set of attributes describing each class and then use that information with vision-language models to classify images. They highlight in their paper that with a concise set of attributes, they can improve the classifier's performance and also increase inter- pretability and interactivity for end user.
((si, at) at) + a ((((s)
+ 7 max Quugge(Set11 a') Q( at))
Here, 8t is the state at step f, at is the action taken, and rt is the reward obtained after taking action at. The parameters and y learning rate and discount factor, respectively. DQNs are used for controlling robots (Tang al., 2025; Senanayake, 2024; Charpentier et al., 2022), detecting fail- ures (Sagar et al., 2024), etc.
Preference Optimization: Optimizing generative models with preference data was first introduced in Direct Prefer- ence Optimization (DPO) (Rafailov al.. 2024). It is technique used ensure models, such as large language models, learn to align its outputs with human preference by asking human which of its generated output is preferred. This technique was later extended to diffusion models in Diffusion-DD DPPt (Wallace et al., 2024), where they updated Stable Diffusson XL model using Pick-a-Pic dataset (human preferred generated image dataset). Unlike traditional im-
3. Methodology: Reinforcement Learning-based Preference Optimization
Our objective is to find set of concept images, C, that maximize the TCAV scores, TS indicating that the con- cepts are relevant to the neural networks' decision bocision-making process. We leverage state-of-the-an text-to-imaae genera- tive models to generate high-quality explainable concepts.
3

---

Explainable Concept Generation through Vision-Language Preference Learning
G:
Group
RL Environment
S, State at time 1
Reward at time t
Update LoRA Weights
a. Action taken at time!
Deep RL Policy
Preference
G1
TCAV
Score
Stripe. Grees,
a
Automatic seed prompt generation from VQA F
SD + LoRA
Score 2
TCAV
Reward Function
G2
S
model.
1+1
Figure 3: Overview of the RLPO framework with its dynamic environment interaction. The RL policy selects actions (seed prompts) which generates concept sets (G1, G2) scored through TCAV. Reward is calculated based the scores obtained for both the sets. Simultaneously, best set is determined based on the scores obtained, which is used to update the LoRA layer of the SD model.
Algorithm The RLPO algorithm. Appendix B.2 for the expanded algorithm.
3.1. The Rationales Behind Design Choices
We explain the design choices, which are validated through experiments in Section 4.1.4.4.
Input: Set of test images f(.)
Run pre-processing and obtain seed prompts (action space)
Rationale 1: Why concept generation is a better idea. If we use concept-basee explanation the traditional way (Kim et al., 2018: Schut et al., 2025), then the end users need to manually guess what concepts to test for. Automatically retrieving the concept set by segmenting test images (Sun et al., 2024) also results in limited concept set. In contrast, SOTA generative model can generate high quality images. We provide more theoretical insights in Appendix A.
for each episode do
for each time step do
Execute at by picking a seed prompt
Generate image groups Gi and G2 Evaluate TCAV scores TS, and TS2 Update SD based on the better score Compute reward end for end for Output: Set of concept images
Rationale 2: Why a deep RL-controlled VLM fine- tuning for generating concepts a better idea. "A picture is worth thousand words but words flow easier than paint." As the saying goes, "a picture is worth a thousand words," it is much easier for people to explain and understand high- level concepts when images are used instead of language. For instance, we need a long textual description such as "The circles are centered around a common point, with alter- nating red and white colors creating a pattern" to describe a simple image of a dart board (i.e., Target Co. logo). There- fore, keep our ultimate concept representation as images. However, controlling generative model from visual inputs is much harder. Since human language can be used directed and easier way to seed our thought process, as the saying goes, "words flow easier than paint," we control the outcome by using text prompts. Since the vastness of the search space cannot be handled by most traditional search strategies, we resort
However, because the search space of potential text prompts is too large, we use deep RL to guide the image generation process. As described in Fig. 3 and Algorithm our al- gorithm, RLPO, uses RL pick potential keywords from predefined list and iteratively optimizes stable diffusion weights to generate images that have a preference for higher TCAV scores. This process is described below.
Notation: Our framework contains three core deep learning models: the network under test f(),, the image generator g(-). and the deep RL network ht-)) First, we have a pre- trained neural network classifier that we want to explain. We then have generative neural network, whose purpose is generating concept image sets, given some text prompts. In this paper, we use Stable Diffusson (SD) v1-5 as the generator as is a state-of-the-aa generative model that can generate realistic images. The core search algorithm that we train is DQN.
DQN for controlling text. Since simple text alone cannot generate complex, high-level visual concepts, in each DQN update step, we use preference opti- mization to further guide the search process towards more preferred outcome, allowing the DQN to focus on states

---

Explainable Concept Generation through Vision-Language Preference Learning
similar to the target. This approach improves our starting points for each DQN episode, enabling more efficient search and incremental progress towards the desired target.
maximizes the Q(s, a) value by using the update rule,
Q* a) = =Ev-rt + max Quager (s', ,a)]
a'EA
3.2. Extracting Seed Prompts
3.4. Optimizing the States
Since generative model can generate arbitrary images. if we provide good starting point for optimization then the convergence to explainable states would be faster. In this paper, to extract seed prompts for a particular class we make use of the off-the-shelf VQA model followed by several pre- processing steps, as described in Appendix B.3, We also explore how gibberish prompts can be used as seed prompts in Appendix B.4 which did not yield useful concepts.
At time t, the policy picks the seed prompt kt, which is then used by the generative model, gkk, wy), with model weights w, to generate 2Z number of images. We randomly divide the generated images into two groups: Xe,, = (Fer... and Xeg. (, 1=1- Let the TCAV scores of each group be .,,t and mm,, Since our objective is to find concepts that generate a higher TCAV score, con- cept images that have a higher score is preferred. Note that, unlike in the classical preference optimization setting with a human to rank, RLPO preference comes from the TCAV scores (e.g.. TX > (t). We call this notion RLPO-XAIF in ablation studies below. If the generative model at time is not capable of generating concepts that are in an explainable state, max(?'S,, (', m,,, (m,) S T), we then perform preference update on SD's weights (more details in Appendix B.5). Following Low-Rank Adaptation (LoRA) (Hu et al., 2022) a method that allows quick SD adaptation with a few samples,- we only learn auxiliary weights a and b each time step, and update the weights as W141 + W. + Xab.
3.3. Deep Reinforcement Learning Formulation
Our objective of using deep RL is automatically controlling text input of Stable Diffusion. As text input, we start with K seed prompts from Section 3.2, that have the potential to generate meaningful concept images after many deep RL episodes. We setup our RL state-action at iteration / as,
Action ae: Selecting a seed prompt, k E K., that best influences concept image generation.
State sy Preferred concept images generated from the seed prompt, -
Reward ry: It is proportional to the TCAV score com- puted at state 8t on action a,, adjusted by a mono- tonically increasing scaling factor &t, As each seed concept reaches the explainable state at different times, this factor is introduced to scale the reward over time for each unique seed concept h. Since the g(.) is getting optimized at each time step t. The seal- ing factor is updated fr+1, + min (1, f where T is total number of RL steps. Therefore, the expected cumulative adjusted reward is R(x) =
As the deep RL agent progresses over time, the states be- come more relevant as it approaches explainable states (Fig. 2), thus the same action yields increasing rewards over time. To accommodate this, with reference to the re- wards defined in Section 3.3, we introduce a parameter ç. which starts at 0.1 and incrementally rises up to 1 as the preference threshold, 17, is approached. Different actions may result in different explainable states, reflecting various high-level concepts inherent to f(-). Some actions might take longer to reach an explainable state. As the goal is to optimize all states to achieve a common target, DQN pro- gressively improves action selection to expedite reaching these states. Thus, deep RL becomes relevant as it optimizes over time to choose the actions that are most likely reach explainable state more efficiently.
(1,57
E EE & -r((st, a:)
Our objective in deep RL is to learn a policy, π + a, that takes actions (i.e., picking a seed prompt) leading to explainable states (i.e., correct concept images) from proxy states (i.c., somewhat correct concept images). We formally define explainable state and proxy state as follow:
Definition 1. Explainable states: States that have a concept score T Te.m 2 n for a user-defined threshold 1) E [0, for explainable state.
4. Experiments
To verify the effectiveness of our approach, we tested it across multiple models and several classes. We consid- ered two CNN-based classifiers, GoogleNet (Szegedy et al., 2015) and Inception V3 (Szegedy et al., 2016), and two transformer-based classifiers, ViT (Dosovitskiy et al., 2020) and Swin (Liu et al., 2021), pre-trained on ImageNet dataset. Unless said otherwise, only GoogleNet results are shown in the main paper. All other model details and results are provided in Appendix B. and C.3, respectively.
concept e and class m is defined
Definition 2. Proxy states: States that have concept score < TSem class m is defined as a proxy state.
for the threshold 7 E [0, for concept and
In practice, we set 1) to a relatively large number, such as 0.7, to ensure that we look at highly meaningful concepts. In DQN, in relation to Eq. 2, we learn a policy that iteratively
5

---

Explainable Concept Generation through Vision-Language Preference Learning
Table I: Search strategy ablation. We see that RL, compared to e-greedy search, is the best strategy to explore the search space with high entropy, average normalized count (ANC) per action, and inverse coefficient of variance (ICV).
Table 3: Exploration Gap (EG) and Odds for different meth- ods based on the human survey (Appendix C.9). This ver- ifies that RIPO can generate concepts that human cannot think of.
Laymen (n=260)
Expert (n=240)
Method
Entropy (f)
ANC (1)
ICV (1)
RI (Ours)
2.80
0.43
2.17
FG (Retrieval)
6.54%
10.45%
0.25 Greedy
2.40
0.21
1.04
EG (Ours)
91.54%
65, 65.45%
0.5 Greedy
1.95
0.15
0.59
Odds (Retrieval)
14.29
8.57
0.75 Greedy
1.85
0.15
0.56
Odds (Ours)
0.09
0.53
EAC
CRAFT
Lens
RLPO (Ours)
4.1. Ablation: Search Strategies (Why Deep RL?)
Jay
We chose DQN as our RL algorithm because of its ability to effectiveness traverse through discrete action space (Mnih et al., 2015) (20 unique seed prompts). We assess the effec- tiveness of RL by disabling the preference optimization step. As shown in Table 1, on the GoogleNet classifier, compared to 6-greedy methods, the RI. setup exhibits higher entropy, average normalized count (ANC), and inverse coefficient of variance (ICV) (See Appendix A. for definitions), indicat- ing RL's ability to efficiently explore across diverse actions. Qualitative results obtained on updating SD model with and without RL are discussed in Appendix C.5.
SF
Tiger
Zebra
Figure 4: Samples of concepts generated by different meth- ods. Observe that RLPO generates diverse images, beyond patches from the test images.
4.3. Concepts Generated by RLPO
4.2. Ablation: Scoring Mechanisms (Why TCAV?)
Objective of RLPO is to automatically generate concepts that human or a retrieval method cannot propose but the neural network has indeed learned (i.e., gets activated). As illustrated in Fig. 4, we observed that the RLPO can generate diverse set of concepts that a human would not typically think of but leads activations of the DNN to trigger. To validate this hypothesis, we conducted a survey to see if humans can think of these generated concepts as important for the DNN to understand a certain class.
Another important aspect of our setup is the use of TCAV score, an XAI method, to provide preference feedback and calculate rewards. Alternatively, this XAI scoring feedback can also be replaced with human feedback or L.LM-based Al feedback. As an ablation study, test the effect of human feedback, we conducted human feedback experiment with eight human subjects who provided live human feedback. Further, to evaluate the L.LM-based AI feedback, we made use of GPT-4o. More details on the experiment setup and results are provided in Appendix C.2. As shown in Table 2, we concluded that, even though other feedback techniques can be used, XAI-based feedback is best for generating concepts that are important to model with high speed and low computation cost. Though human and AI (GPT4o) are good at correlating semantics, by only looking at test images and concepts instead of model activations, they are not able to provide model specific explanations.
As detailed in Appendix C.9, during our human survey, we presented a random class image followed by two con- cepts, one generated by our method and another from a previous retrieval-basee method (Fel et al., 2024; 2023). While the choices were similar in terms of XAI-score from both methods, we discovered that most participants recog- nize retrieval-based concepts, and only those with domain- specific knowledge could identify generated concepts as important. As highlighted in Table 3, high Exploration Gap (EG) (definition in Appendix 1) of our method indicates that most people can only identify concepts from a small subset of what f(() learns during training. Intuitively, when we retrieve concepts from the test class, they tend be sim- ilar to the test images. We further verify that the generated concepts have the following properties.
Table 2: Scoring mechanisms ablation. We see that RLPO with Explainable AI feedback (RLPO-XAIF), in this case TCAV, is better choice than RLPO with human feedback (RLPO-HF) and Al feedback (RLPO-AIF).
Method
Class based Model-specific Explanations
Feedback Cass*
Exccution Time (L)
Explanations
Diverse representations per concept. We verify the di- versity (e.g.. different types of stripes) of generated and retrieval- based concepts by computing the vector similarity of the CLIP and ResNET50 embeddings between X, and
RLPO-HP RIPO-AIP RLPO-XAIE
/
NIL
180 + 30s
/
10 GB
72 d. 1.2s
4
Feedback cost refers to the GPU memory requirements.
6

---

Explainable Concept Generation through Vision-Language Preference Learning
Table 4: Novel concepts: 7. (TCAV score), CS (Cosine similarity), ED (Euclidean distance), RCS, and RED (CS and ED with ResNet50 embedding). This indicates that RLPO can generate a diverse set of concepts that triggers the network.
Methods
Concepts
(1)
CS (L)
ED (1)
RCS (L)
RED (t)
EAC (Sun al., 2024)
C
1.0
0.76 0.03
7.21 = 0.63
0.67 = 0.14
6.34 + 2.16
CI
1.0
0.77 0.02
7.17 0.34
0.50 = 0.18
9.70 + 3.20
Lens (Fel et al., 2024)
C2
1.0
0.72 + 0.04
8.02
0.12
10.90 + 2.80
C3
1.0
0.69 0.05
0.45
11.03 2.17
CI
1.0
0.76
7.37
0.16
8.80 + 3.20
CRAFT (Fel et al., 2023)
C2
1.0
8.25 0.02
0.50 1.90
9.90 + 3.40
C3
1.0
0.73 0.04
7.98 + 0.79
0.44 + 0.07
10.80 + 1.90
C1
1.0
+
10.48 + 0.50
0.04 + 0.01
16.80 + 1.40
RLPO (Ours)
C2
1.0
0.49 + 0.02
10.65 = 0.20
0.02 = 0.02
17.20 = 0.80
C3
1.0
0.49 0.02
10.74 + 0.30
0.03 + 0.01
17.60 + 4.40
Table 5: Inter-concept Comparison for "zebra" class across three trial runs. Low Cosine Similarity (CS), high Wasser- stein Distance (WD), and high Hotelling's 's T-squared Score (HTS) suggests that concepts generated from different seed prompts are different from one another.
Table 6: Intra-concept Comparisons for "zebra" class across three trial runs. High Cosine Similarity (CS), low Wasser- stein Distance (WD), and low Hotelling's T-squared Score (HTS) suggests that concepts generated from same seed prompts lies on the same distribution.
Metries
Stripes-Running Concept
Rumning-Mud Concept
Mud-Stripes Concept
Metries
Stripes-Stripes Concept
Running- Concept
Mud-Mud Concept
Avg. CS (1)
0.67 1: 0.01
0.60 1. 0.0004
0.73 a: 0.0004
Avg. CS (1)
0.99 + 0.0008
0.99 + 0.0004
0.99 3:0-+000 0.0004
Avg. WD (*)
8.15 = 0.05
7,85 : 0.02
7.48 0.03
Avg. WD (1)
0.95 -
0.82 0.07
0.82 0.06
Aug. HTT (1)
7598. 50 84.5
13089.68 2147.81
7615.73 4: 538.06
Avg. HIS
2.46 0.091
2.36 0.08
2.66 0.22
Are they from the distribution?
they from the same distribution?
No
Yes
Yes
Yos
Xm for different classes. As highlighted in Table 4, we observe that concepts from retrieval-basee methods tend to have high cosine similarity with test images, making them less useful as abstract concepts (e.g., to explain the zebra class, a patch of zebra as a concept is less useful compared to stripes concept).
CLIPSeg (Lüddecke & Ecker, 2022), a transformer- segmentation model which takes in concept images as prompts, Xe, and highlights in test image, x Xm, which part resembles the imput prompt as heat map. More details on this is available in Appendix C.3.3. As shown in Fig. 5, class image on left highlights the top 3 identified concepts by RIPO. We also compare the output generated by other popular XAI techniques such as LIME and GradCam with ones generated by RLPO, more details in Appendix C.6.
Multiple concepts per class. Since RLPO algorithm ex- plores various explainable states, we can obtain multiples concepts (c.g., stripes, savanna) with varying level of impor- tance. Fig. shows the top three class-level concepts identi- fied by our method for the "zebra" class for the GoogleNet classifier. We see that, each concept set has a different TCAV score associated with them indicating their impor- tance. Additionally, shown in Table 5, these concepts are inherently different from one another. As an additional re- sult, we can see in Appendix C.4 how each concept evolves- representing different levels of concept abstractions with each RL step.
After finding the relationship between generated concepts and input images, we need to validate the importance of the identified concepts. To that end, we applied c-deletion, commonly used validation method in XAI, to the class images for each identified concept. We gradually deleted concept segments based on the heat map obtained from ClipSeg. The results for the c-deletion are shown in the Fig. 6. We see the area under curve is the highest for the most important concept "stripes" and the lowest for least important concept "mud," indicating the order of importance of each concept. More examples on the c-deletion are in Appendix C.3.4. Additionally, to verify the generated concepts are consistent, we compared their CLIP embedding across multiple runs. As shown in Table 6, we can see that
4.4. Are Generated Concepts Correct?
After generating the concepts, next step is to identify what those concepts signify. To locate where the class images generated concepts correspond, we made use of

---

Explainable Concept Generation through Vision-Language Preference Learning
Concepts seeded from the 'stripes' keyword (0.920)
Concepts seeded from the 'running' keyword (0.800)
Concepts seeded from the 'mud' keyword (0.560)
Figure 5: The figure shows the concepts generated by method and where they located in the input image ("zebra" class) for GoogleNet classifier. As highlighted the "stripes" concept images are located near zebra, the "running" concept images, showing trees, highlight the background, and the "mud" concept highlights the grass and soil in the input image. The concepts are ordered in their importance (TCAV score) with "stripes" being the highest and "mud" being the lowest.
C-Deletion on Concepts in "Zebra" class
We now demonstrate how engineers can use these newly revealed information about concepts to improve the model. As identified by RLPO, for Tiger class, the base GoogleNet model gives equal importance to both foreground (high- lighted by concepts "orange black and white" and "or- ange and black") and background (highlighted by concepts "blurry") in the input (see Fig. 12 Appendix C.3 for exam- ple explanations). As shown Fig. 7, when we fine-tune the GoogleNet on images of Tiger-related concepts, that the fine-tuned model now focuses more on tiger than the background while maintaining a similar accuracy (65.6%) Details in Appendix C.8.
Average Accuracy Score
"stripes" concept "running" concept
0.6
1.265
"mud" concept
2.525
2,905
0.2
0.0
2
5
10
C-Deletion Steps
Figure 6: C-deletion. Removing concepts over time to measure the reliability. The colored numbers indicate the area under the curve.
the concepts generated from same seed prompt belongs to the same distribution across multiple runs.
Fine- Fine-tuning based concepts related to Tiger
4.5. How Are Generated Concepts Useful to Engineers?
TCAV score
To verify the usability of the generated concepts, we con- ducted a human study with 19 ML engineers. We first provided them the concept generated by our method for "zebra" class and ask them to choose relevant concepts for GoogleNet to classify zebra without telling them that all shown images are actual concepts. All the engineers se- lected the "stripes" concept to be important while some also selected the "mud" concept. But most missed the "running" concept. This indicates that engineers cannot think of all the important concepts that gets the neural network activated. In the next step, we showed engineers the concept-explanation mapping on a random input image (similar to Fig. 5) and asked them if the provided explanation helped them un- derstand the model better and if it provided new insights. 94.7% of the engineers agreed that the explanation helped in better understanding the neural network and 84.2% agreed that it provided new insights. This result shows that the new concepts discovered by our proposed method help engi- neers discover new patterns that they did not imagine before (More details in Appendix C.10).
Googlehet (Prerainndi Googlehiet If netured)
blurry Image
orange black
and white
orange and black
blurry Image
grnge black and white
orange and black
whiskers
Miisker
Original dlassifier considers both foreground background.
Finetuned classifier only considers foreground.
Classifying based the background undesirable.
Usefulness of RLPO. Fine-tuning GoogleNet based on generated concepts for the Tiger class.
Figure 7:
4.6. Can concept Generation Help in Understanding Model Bias?
Apart from generating actionable concepts, our proposed framework can also be used identify spurious correlations
8

---

Explainable Concept Generation through Vision-Language Preference Learning
or undesirable biases learned by the model during train- ing. To demonstrate this, we applied RLPO to ResNet-18 classifer trained on the CelebA dataset (Liu et al., 2015) for "Blonde" versus "Not Blonde" classification, a task with known gender biases (De Coninck et al., 2024; Subra- manyam et al., 2024; Chen et al., 2023). We adapted our seed prompts to include higher-level semantic about gender, and as shown in Fig. 8, RLPO showed that concepts repre- senting "female face" were more important for the model for the "Blonde" class. Additionally, we see that for the "male face" seed prompt, the generated concepts started gen- erating males with long blonde hair, further narrowing down on the fine- fine-grain concepts the model is looking at (i.e. long and blonde hair).
Positive Prompt
0.7
The pustomer service team was very helpful and responsive when reached out
support. They were patient and provided deaar instructions an how to address
0.7
of the issues, which improved the situation slightly.
Generated Concepts
client, purchaser, consumer, user, shopper Team: group, crew, unit, squad, alliance, partnership
Customer:
Helpful: supportive, useful, valuable, beneficial, productive
Clear: transparent, unclouded, open, lucid, distinct
Address: speak, contact, communicate, interact, approach Issues: problems, concerns, matters, challenges, disputes
We use sentiment analysis in NLPP show the generalizability of RLPO. Concepts tend to be synonyms and the numbers indicate the TCAV scores.
Figure 9:
1.000
1.0
TCAV /CAV/Scre Score
0.8
Concepts
"Male face"
0.6
"Female face*
language generative model to navigate an infinitely large concept space, addressing the challenge of manually curat- ing concept image sets. However, our approach also suffers from some limitations. Firstly, while RLPO is designed to iteratively refine these prompts and can drift towards more model-relevant abstractions beyond the original scope, the quality and nature of the initial seeds can influence the search trajectory and efficiency. Increasing the number of seeds helps generate better concepts. Additionally, the quality and diversity of the generated concepts are also influ- enced by the generative model being used. The pre-training data of such models can introduce biases, and their capacity to accurately render certain concepts. While this depen- dency is not unique to generative approach, retrieval-baat methods are similarly constrained by available data and hu- man cognitive biases, explicitly addressing and potentially mitigating these inherited biases in the context of XAI is an important direction.
0.4
0.2
0.000
0.0
'Male face*
"fermale face"
Concepts
(a)
"Female face" Seed Prompt
"Male face" Seed Prompt
(b)
Figure 8: (a) TCAV scores obtained from "Male face" and "Female face" seed prompt, highlighting the importance of female gender over male. (b) Sample generated concepts for "Female face" and "Male face" seed prompt.
The concepts that our algorithm generates can be diverse as it tries to reveal the concepts inherent to the f(-), making it less Nomain--spccife (c.g., for a medical application, there is chance might generate non mon-medical images if the f(-) activations get excited for non modeicaall data). As a future extension, aim input preferences from both TCAV and domain experts while optimizing, making generated expla- nations even more aligned to specific applications. Despite these challenges, our results demonstrate how to leverage the strengths of visual representations and adaptive learning to provide intuitive and effective solutions for understanding complex, high-level concepts in neural networks.
4.7. Can RLPO Generalized Beyond Images?
To demonstrate the generalizability of the proposed algo- rithm, we extended RLPO generate words to explain sentiment analysis in NLP. We made use of Mistral-7B In- struct model to generate synonyms of seed prompts and optimized the language model based on preferences from TexICNN model pre-trained on IMDB sentiment dataset. Fig. 9 highlights relevant words with their importance score in the input. More details are provided in Appendix C.7.
5. Limitations and Conclusions
Impact Statement
In this work, we introduced Reinforcement Learning-basee based Preference Optimization (RLPO) to automatically articulate concepts that explain the internal representations of neural networks. We demonstrated how RLPO can guide a vision-
Our paper is generic algorithmic contribution, and we do not foresee direct negative societal impacts. As a positive trait, our approach can potentially be used to understand and explain model bias.

---

Explainable Concept Generation through Vision-Language Preference Learning
References
Ghorbani, A., Wexler, J., Zou, J. Y., and Kim, B. Towards automatic concept-based explanations. Advances in neu- ral information processing systems, 32, 2019.
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22/2.0800) 2022.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2)3, 2022.
Chadebec, C., Tasar, O., Benaroche, E.., and Aubin, B. Flash diffusion: Accelerating any conditional diffusion model for few steps image generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 15686-15695, 2025.
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. Interpretability beyond feature attribu- tion: Quantitative testing with concept activation vectors (tcav). In International conference on machine learning, pp. 2668-2677. PMLR, 2018.
Charpentier, H., Senanayake, R., Kochenderfer, M., and Günnemann, S. Disentangling epistemic and aleatoric uncertainty in reinforcement learning. In ICML 2022 Workshop on Distribution-Fiee Uncertainty Quantifica- tion, 2022.
Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. Communications of the ACM, 60(6):84-90) 2017.
Li, J. Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-iman -ingag pre-training for unified vision- vision-language understanding and generation. In International confer- ence on machine learning, pp. 12888- 12900. PMLR. 2022.
Chen, R., Yang, J., Xiong, H., Bai, J., Hu, T., Hao, J.., Feng, Y., Zhou, J. T., Wu, J., and Liu, Z. Fast model debias with machine unlearning. Advances in Neural Information Processing Systems, 1411141144539 2023.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Liu, Z., Luo, P., Wang. X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730- 3738, 2015.
De Coninck, S., Leroux, S., and Simoens, P. Mitigating bias using model-agnostic data attribution. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 235-243, 2024.
Liu, Z., Lin, Y., Cao. Y., Hu, H., Wei, Y., Zhang, Z.,, Lin, S. and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbomn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., al. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2020.
Lüddecke, T. and Ecker, A. Image segmentation using text and image prompts. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7086-7096, June 2022.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F,, al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidje- land, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518(7540): 529 239-533,11 2015.
Fel, T., Picard, A., Bethune, L., Boissin, T., Vigouroux, D., Colin, J., Cadène, R., and Serre, T. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2711-2721, 2023.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Ad- vances Neural Information Processing Systems, 36, 2024.
Fel, T., Boutin, V., Béthune, I.., Cadène, R., Moayeri, M., Andéol, L., Chalvidal, M., and Serre, T. A holistic ap- proach to unifying automatic concept extraction and con- cept importance estimation. Advances in Neural Informa- tion Processing Systems, 36, 2024.
Ribeiro, M. T., Singh, S., and Guestrin, C. why should trust you?" explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1135-1144, 2016
10

---

Explainable Concept Generation through Vision-Language Preference Learning
Sagar, S., Taparia, A., and Senanayake, R. Failures are fated, but be faded: Characterizing and mitigating unwanted behaviors in large-scale vision and language models. In Proceedings of the International Conference on Machine Learning (ICMI.), 2024.
Zang, Y., Yun, T., Tan, H., Bui, T., and Sun, C. Pre- trained vision-language models learn discoverable vi- sual concepts. Transactions Machine Learning Re- search, 2025. ISSN 2835-8856. URL https://
openreview. net forum? wartummmid--y/ ddVoOwMFRjo2.
Schut, L., Tomašev, N., McGrath, T., Hassabis, D., Paquet, U., and Kim, B. Bridging the human ai knowledge gap through concept discovery and transfer in alphazero. Pro- ceedings of the National Academy of Sciences, 122(13): e2406675122, 2025.
Senanayake, The role of predictive uncertainty and diver- sity in embodied ai and robot learning. In arXiv preprint arXiv:2405.08 03 /64, 2024.
Subramanyam, R., Thopalli, K., Narayanaswamy, V., and Thiagarajan, J. Decider: Leveraging foundation model priors for improved model failure detection and explana- tion. European Conference on Computer Vision, pp. 465-48 482. Springer, 2024.
Sun, A., Ma, P., Yuan, Y., and Wang, Explain any con- cept: Segment anything meets concept-based explanation. Advances in Neural Information Processing Systems, 36, 2024.
Szegedy, C., Liu, W. Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Szegedy, C., Vanhoucke, V., loffe, S., Shlens, J., and Wojna, Z. Rethinking the inception architecture for computer vi- sion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818 2826, 2016.
Tang, C., Abbatematteo, B., Hu, J., Chandra, R., Martín- Martín, R., and Stone, P. Deep reinforcement learning for robotics: A survey of real-world successes. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 28694-28698, 2025.
Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffusion model alignment using direct pref- erence optimization. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp.
2024.
Yan, A., Wang. Y., Zhong, Y., Dong, C., He, Z., Lu, Y., Wang, W. Y., Shang, J., and McAuley, J. Learning con- cise and descriptive attributes for visual recognition. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 3090-3100, 2023.

---

Explainable Concept Generation through Vision-Language Preference Learning
Appendix
this appendix, we describe the theorems, algorithm, metrics, experimental setup and additional results.
A. Definition, Theorems, and Proves
Comparative Overlap of Human- cauman-Interpretbbie and Generative Model Concepts in Neural Understanding Tasks (f(-). We formalize this with reference to Fig. 1.
Theorem 1. Let the set of human- intrrun-atttereeabble concepts that the f()) has learned be Cn, and the concept sets human collected, retrieved though segmentation, and generated using a generative model be CH, CR. and Cg. respectively, Then, (CNNCN] 2 [CH n CNl 2 0 and [Cg nCcll Z |CR n CCll 2
Proof sketch. Cи S CG and Cn C CG = (CCN2212 2 and CCN 20
Proof of Theorem 1.
Definition 3. Let CH, CR. and Ca denote the representing human-inter iamaaninteeppretable concepts, retrieved concepts, and concepts generated by generative model, respectively. We define the relationships between these sets as follows:
Cи Ca and CR S Ca.
Property 1. For any set Cs where (H, R, G), it holds that:
S E (C,UUCN),
where Cn represents the set of concepts learned by f(-).
For any two sets A and B, the size of their intersection |An is non-negative since represents the number of elements common both sets. Thus, we have:
CH NCN 2 and |C 2 O.
(4)
Given Definition and Property 1, we assume the following subset relationships between the sets:
Ch S Cal and |Cn S Cal-
Case 1. Since Cи S Co. any element Cи is also in Cg. Therefore, any element Cи Cy is also in Co O CC. Hence,
CCN CCa nCN.
Case 2. Similarly, since Cn Ca. any element E CR is also in Ca. Therefore, any element CR O CN is also in
CG n Cy. Hence,
CR CN S CC
From Case 1 and Case 2, since Cи CCN and CR n Cy are subsets of CG n CN, it follows that:
[C nCc]
(5)
and
[Ca < [Ca n Cxl
(6)
Combining Eqs. 5 and 6 with the non-negativity established in Eq 4, we have:
CC 2 CC nCxl 2 0
and
[Ca nCv 1Ca (Crl 2 0.
(8)
12

---

Explainable Concept Generation through Vision-Language Preference Learning
The Effects of DQN-PO based Concept Space Traversal. We now formalize what concepts the DQN has learned, with reference Fig. 2.
Theorem 2. When traversing in the concept space, with each reinforcement learning step.
1. Case Moving from a proxy state lowards an explainable stale monotonically increases the reward.
2.
Case 2: Moving from explainable state towards the target class does not increase the reward.
Proof sketch. Obtain the rewards before and after yn and compute the difference in reward for each segment.
Property 2. As & increases, the reward function proportionally amplifies. particularty enhancing the significance of outcomes near by which marks the point beyond which TCAV scores are always
Proof of Theorem 2. WLOG, let the TCAV score, Se.m,,, for concept e and class m time be S.. The reward function is defined as (Section. 3.3),
R(t,a) = K S. f(t),
(9)
for a constant K and factor,
c. ift S bn otherwise,
f(t)
(10)
En
for positive parameters and Eo-
Case 1: Considering the difference in reward function time when S byg
R(t + 1,0) R((t
S441 1) K S. f(t)
(1I)
K S4+1 5. (t+1) K S. t.
K. (S411 (1+1) S.
K. (t(S++1 S:) + S(+1)
h(4 1) 1)) =1((),
From (St+1 -S))
(12)
(1+ -t
. (t (t-h'(t) S.4)) R(t 1, R(t,a) K.
(13)
Since,
1. S, is monotonically increasing for S in = 11(1) 0 and
2. S, E [0, 1].
R(t 1,a) R(t, a) 2 0.
(14)
Case 2: Considering the same difference in rewards for 2 typ
S. s(1) R(I + 1,a) - R((,a) = K K. S [(++1) K K.S.
(15)
= K SS11 -- K- S, to
K (S++1 - St)
Given that S₁ and 1t, are both outcomes generated from a generative model fine-tuned for particular concept, S SS 0 in response to the same action. Hence,
R
R(t 1, R((t, a) 0.
(16)
13

---

Explainable Concept Generation through Vision-Language Preference Learning
Theorem 2 characterizes the how TCAV scores (i.e., proportional to rewards) increased up 7/. As result, as shown in Theorem 3, if the generator moves close to the image class, then the explainer generates images similar the class. Therefore, by varying y) we can generate concepts with different levels of abstractions.
Theorem 3. As we go closer the concept class, |Co n Cxl becomes larger for generated concepts Ca and f()'s internal concepts, CN-
Proof sketch. Measure the sensitivity difference between Sey,m, and Seg,m,t as ++ oc.
Proof of Theorem 3. At each time step t, two sets of samples are generated near Ca(t) using a generative function g(.). denoted by (t) ((Ca(t)) and (t) g(Ca (t)). We define the sensitivity of these samples to the concept class using measurable attribute, a (s), that quantifies the alignment closeness of a sample to the target concept class.
The optimization step at each time step selects the sample with higher sensitivity, denoted by:
Sopt (1) = arg max(o((s(()))o((2(())
The sample with the lower sensitivity is given by
min (t) = arg (
Sample Sopa(t) and (t) is then used to adjust Ca. increasing its overall sensitivity to the concept class. Consequently, the sequence of Ca over time evolves as:
1c(t+1)=(Ca(A) (Ca(t) Sop (t)) (t)
\ 8mii (t)
This process incrementally increases the sensitivity of Ca(t + 1) to the concept class, driven by the iterative inclusion of optimized samples.
Given that CN is already close to the target concept class, the movement of Ca through this optimization process indirectly steers Cg towards Cy. As Cg evolves in this manner, the overlap between CG and Cn naturally increases, leading to:
lim (C(()) ncxl = lim Ca(t) = Cn- 1000
This results from Ct( containing more elements that exhibit higher sensitivity similar to those in Cv, thereby increasing their intersection.
A.1. Definitions
Entropy quantifies the uncertainty or randomness inherent in a probability distribution. For discrete random variable X with possible outcomes æ1, 22 and corresponding probabilities P(X xx4) Pi. the entropy ((X) is
Entropy:
defined as: H(X)= Ei= p; log pi, where p; represents the probability of outcome
Odds: Odds describe how many times an event is expected to happen compared to how many times it is not. They are often used in gambling, sports betting, and statistics. The odds of an event with probability p (where is the probability of the event happening) are calculated as: _
Exploration Gap (EG): quantifies the proportion of missed optimal actions, defined Accuracy, highlighting how humans frequently miss the most optimal actions when presented with generated concepts.
Average Normalized Count (ANC): The ANC is measure of the central tendency of the normalized action frequencies within a distribution. It provides insight into how the actions are distributed relative to the overall frequency distribution. high ANC indicates that, on average, the action frequencies are relatively large, meaning that certain actions are more dominant. Conversely, a low ANC suggests that the actions are low and only a few high frequent actions are present. Given by x= Σ Sú, where fa is the frequency of action
Inverse Coefficient of Variation (ICV): A standardized measure of concentration, calculated as the ratio of the mean to the standard deviation: g. It represents how many standard deviations fit into the mean.
14

---

Explainable Concept Generation through Vision-Language Preference Learning
Feedback Cost: of the model.
Feedback Cost refers to the resource(GPU) expense associated with obtaining feedback during the training
Execution Time: Execution time refers to the total time taken by model or algorithm to complete its task from start to finish. This includes the time for data processing, model computation, and generating outputs.
B. Methodology
B.1. Machine Learning Models We Use
Neural Network Under Test (f(-):: We test RLPO for all the different classification models given below.
1. GoogleNet: We utilized a pretrained model from PyTorch torchvision pretrained models with weights initialized from GoogLeNet_ DeeggLetetWWeihhss..oo IMAGENETK K_VI. VI.
2. Inception V3: We utilized pretrained model from PyTorch torchvision pretrained models with weights initialized from Inception_ _3_ 333_Weehtt.nnn IMANEEES.AGEEETTII V1.
3. Vision Transformer (ViT): We utilized a pretrained model from PyTorch torchvision pretrained models with weights initialized from ViT B 16 IMAggNETIM K. VI.
4. Swin Transformer: We utilized a pretrained model from PyTorch torchvision pretrained models with weights initialized from Swin. V2 B. Weights. IMAGENETIKK
5. TextCNN sentiment classification model: We utilized pretrained model from Captum library. The model was trained
on IMBD sentiment dataset.
TCAV Logistic Model:
We utilized a logistic regression model to address classification tasks in TCAV instead of the default SGD (Stochastic Gradient Descent) classifier. This decision was based on our observation that the SGD classifier produced high variance TCAV (Testing with Concept Activation Vectors) scores, which indicated inconsistent model behavior across different runs As demonstrated in Table 7, the standard deviations for SGD-derived scores are frequently larger than those obtained with the logistic model, underscoring the latter's improved stability. We configured the model to perform maximum of 1000 iterations (max_iter=1000)
Table 7: TCAV Scores (Concept/Random) for Different Models, Layers, and Classifiers. Scores are presented as mean + standard deviation across 5 runs with random seed. Concepts used in these experiments are from the ones provided by the authors of TCAV paper.
Stripes/Random TCAV Score
Dots/Random TCAV Score
Model
Layer
SGD
Logistic
SGD
Logistic
inception3a
0.662 +: 0,03 / 0.338 + 0.03
0.67 + 0.00/ 0.33 = 0.00
0.36 + 0.05 / 0.64 + 0.05
0.33 + 0.00 0.67 + 0.00
GoogleNet
inception4e 0.992 + 0.01 / 0.008 + 0.01
1.00 + 0.00/ 0.00 + 0.00
0.01 + 0.007 / 0.99 + 0.007
0.00 + 0.00/ / 1.00 + 0.00
layer3
0.796 0.02 / 0.204
0.02
0.78 0.00 = 0.00
0.078
0.07 / 0.922 0.07
0.00 0.00 1.00 0.00
ResNet50
layer4
1.000 1 0.00 / 0.000 0.00
1.00 1 0.00/ 0.00 : 0.00
0.60 1 0.54/ / 0.40 1 0.54
0.00 1 0.00/ / 1.00 1 0.00
Stable Diffusson v1-5 with LoRA: We used our base generation model as SD v1-5 and updated its weights using LoRA during preference optimization step. This version of SI> was finetuned from SD) v1-2 on "laion-aesthetics v2 5+" dataset with 10% drop in text-conditioning for better CFG sampling. In experiments, we kept LoRA rank to 8 with scaling factor of 8 and initial weights were defined from a gaussian distribution. We only targeted the transformer modules of U-Net in the SD architecture.
DQN: We use a DQN with specific parameters tailored to effectively navigate a vast search space. We utilized a small buffer size of 100, which limits the number of past experiences the model can learn from, encouraging more frequent updates. The exploration rate was set at 0.95, prioritizing exploration significantly
ensure thorough coverage of the search space. The batch size was configured to 32. We set the discount factor to 0.99 and the update frequency was set at every four steps. The model updates its parameters with a the soft update coefficient of 1.0. Gradient steps was set to I indicating single learning update from each batch, and gradient clipping was capped at 10 to prevent overly large updates.
BLIP: We utilize the Bootstrapped Language Image Pretraining (BLIP) model for the task of Visual Question Answering (VQA). This model, sourced from the pre- pre-trained version available at 'Salesforce/blip-ve caallp--aa-cafflt--lrgge.. is designed to generate context-awars responses to visual input by leveraging both image and language understanding. The large variant of
15

---

Explainable Concept Generation through Vision-Language Preference Learning
the BLIP model is fine-tuned for VQA, allowing it to effectively interpret and answer questions based the visual content provided.
B.2. RLPO Algorithm
Algorithm presents the detailed version of the algorithm introduced in Section 3, Algorithm 1.
Algorithm 2 DQN Algorithm with DPO and Adaptive Reward
Input: Set of test images, f(-)
Initialize Q-network Qa a) with random weights Initialize replay buffer and adaptive parameter { +- 0.1 for each episode do for each time step do
Observe state 8t and select action a, based Q (e-greedy) Execute a, and generate 10 images, divided into two groups 11 and G2 Evaluate TCAV scores TCAV, and TCAV2 if max (V TCV22) S 0.7 then
Update policy to favor higher TCAV group and perform DPO Update ç < min + increment)
else
Set <
end if'
Compute reward = max (CAV, ICAV,, 2CAV,)
Store transition (St, at, Tt, 81-1) in D
Sample a mini-batch from D for each sampled transition (st, az, Fú, 81-1) do
Compute target y/ = r, y max Qo (s)+1,d') end for
Compute loss ( + Σ (2: Qo a;))
Perform a gradient descent step to update 0
Periodically update target network: 0' + 70 + (1 - +)0'
end for end Output: Set of concept images
B.3. Preprocessing for Generating the Action Space
Steps not discussed Scction 3.2.
Each patch from the test images passed to the VQA model to extract relevant and useful information about the correspond- ing class. In this study, choose BLIP (Li et al., 2022) our VAQ model. We posed of targeted questions the VQA model, aiming to gain insights into the class-specific features represented in the patches. The questions designed to probe various aspects of the image patches, helping the model focus class-defining attributes.
1. "What is the pattern in the image?"
2. "What are the colors in the image?"
3. "What is the background color of the image?"
4. "What is in the background of the image?"
5. "What is the primary texture in the image?"
6. "What is the secondary texture in the image?"
7. "What is the shape of the image?"
We then remove stop words and duplicates from the generated responses using lemmantizing and perform cross-similarity check using CLIP between all the unique words and further filtered words which are more than 95% similar. To further select most relevant keywords to the class images, we perform a VLM check using class images and the extracted keyword to get the softmax score of how much the keyword and image are related. This score is then averaged over all the class
16

---

Explainable Concept Generation through Vision-Language Preference Learning
Questions
"What pattern the image?" "What are the colors in image?"
"What is the shape of the image?"
Class Images
3 Patch
Seed Prompts
stripes
black and white green black and white dry grass tail
Remove duplicates
2. Remove class name
VQA
3. Remove similar words (cosine similarity 0.9) Select top frequent words
mud speckled fence mane
Figure 10: Seed prompt pipeline
images and this average is use to sort the keywords. Now, from the sorted keywords, we select top 20 keywords as our RL action space. The cross- vross-similarity and VLM check are inspired from (Zang et al., 2025) where they used a similar filtering setup to remove potentially useless concepts.
B.4. Exploration of Random Gibberish Prompts as Seed Prompts
In this experiment we don't 't use a VQA to get seed prompts. We choose a random list of incoherent prompts, example shown Fig. 11. We found that for these prompts it takes a really long time to get some meaningful explanation in most cases lead to random generation, thus showing importance of starting from good proxy concepts.
Gibberish Seed Prompts
Tww.H///ww...HHTTTTTTTTTTTTTTTTA 4. mhxxqy2FFBEC
2. se-L8tPe1922umu
3. CCKLTIZZn z80
5. SWEC6Wlgfpqaz PQjoGrxluzm www.a//ssisiciicccccooo///////aa
Figure 11: Sample gibberish seed prompt used with RLPO on GoogleNet classifier to generate concepts.
B.5. Preference Optimization Update for State Space
Steps not discussed Section 3.4.
The candidate concepts serve as the initial states for the RL agent. From these initial states, the agent takes actions Keywords that leads multiple subsequent possible states using g(-). These states are then grouped, and the group's sensitivity is compared against Inputs of f(.) using TCAV scores. A higher TCAV score suggests higher sensitivity, indicating that the group is more aligned with f(-)'s inputs.
We employ preference optimization over the grouped states to guide states towards explainable concepts. To prevent the model from skipping over explainable states and directly reaching the input domain, we introduce a threshold that limits the application of preference optimization each step as shown in equation 17.
17

---

Explainable Concept Generation through Vision-Language Preference Learning
Given two groups of samples G1 and G2 with their average TCAV scores TCAV, and TCAV 2 if max TCAV,) - 0.7, update π to favor the group with higher TCAV.
(17)
To optimize g(.) to find better proxies, for each step in the environment we utilized average TCAV scores TCAV, and TCAV2 from G1 and G2 to decide between preferred and unpreferred concepts. Lets say TCAV > TCAV2, 2, than we optimize 9() over the sample defined as S {(a, = ). where zg² and = are the sample points from the groups on action We optimize g(.) using objective 18 to get optimzed g'(.) (Wallace et al., 2024).
T()))
(ag ==))),,--0000))) ~qql"""""""""""))" (ug"),) (ug4) log ( (-
(I|:01 ( .012 - ||ea (x()(())) 01 1)
-1 (2 .))l |e - -c _2 (
(18)
where zi - ag + ort", , 6* ~ (O, /) is drawn from a( de = =of/ aftoo is the signal-to-noise ratio, and w(A,) is
weighting function (constant practice).
B.6. TCAV Setting for Different Models
We tested different models on different layers and classes and the summary our setting across different models described in table 8.
Table 8: TCAV setting across different models
Models
Layers
ImageNet Classes
GoogleNet
inception4e layer Mixed_7c layer
Goldfish, Tiger, Zebra Police Van
Inception
Goldfish, Tiger, Lionfish & Basketball Goldfish, Golden Retriever, Tiger & Cab Goldfish, Jay, Siberian husky & Tiger
Vision Transformer (ViT)
heads layer
Swin Transformer
head layer
C. Experiments
C.1. Computing Resources
The experiments were conducted system equipped with an NVIDIA GeForce RTX 4090 GPU, 24.56 GB of memory. and running CUDA 12.2. The system also featured a 13th Gen Intel Core i9- 1--1390KKF CPU with 32 logical CPUs and 24 cores, supported by 64 GB of RAM. This setup is optimized for high-throughput computational tasks but the experiments are compatible with lower-specification systems.
C.2. Human and LLM-Based AI Feedback Mechanisms
We test other feedback mechanism in RL by replacing the XAI-TCAV feedback with Al and Human feedback's. s. Herer we discuss the experiental setup and configuration for both experiements.
AI Feedback: GPT-4 leveraged to evaluate the explanatory power of image sets by focusing on concepts related to a target class, a method that aligns with the growing trend of incorporating AI-driven feedback (Bai al., 2022). This approach involves sending structured prompt an LLM, asking it to score how well two sets of images explain a target class using specified concept. The process involves the following.
1. Image Encoding:
The images from two sets (conceptl and concept2) are first converted into base64 format to ensure they can be transmitted via the request as encoded strings.
2. Structured Prompt: A detailed and specific prompt is crafted for the LLM. It asks the model to assess the quality of explanation each image set provides for a particular class through the lens of specific concept. The prompt used
18

---

Explainable Concept Generation through Vision-Language Preference Learning
"Please evaluate each of the following of images for how well they explain the class (concept name). For each set, provide numerical score between O and 1 (to two decimal places)" The prompt clearly defines how the model should respond, asking for numerical score between 0 and 1, where:
via the concept
(a) O indicates that the image set does not explain the class at all via the concept.
(b) I indicates that the image set perfectly explains the class via the concept.
3. LLM-Based Scoring: Once the prompt is sent to the LLM, it evaluates the image sets and provides scores based on its
learned knowledge and understanding. The response is parsed to extract the scores for each set of images.
Human Feedback: In this experiment, eight computer science majors provided live feedback after each step of a rein- forcement learning process, leveraging their prior knowledge of reinforcement learning with human feedback (RLHF) mechanisms (Christiano al., 2017). The feedback from all participants was averaged serve the reward for each step in the RL process. Given the abstract nature of the initial concepts, participants needed to take time to thoughtfully assess each step, which contributed to lengthier feedback cycle.
C.3. Additional Results and Analysis
To validate method for its ability to generate concepts, tested it with different models and classes. We started it traditional models, GoogleNet and Inception and then extended it to transformer-basee models, Vision Transformer (ViT) and Swin Transformer, pre- pre-trained on ILSRVC2012 data set (ImageNet) (Krizhevsky et al., 2017). We show additional plot in various classes shown in Fig 12,13,14,15,16
'orange black and white' concept (0.660)
'black and orange' concept (0.660)
'blurry image' concept (0.620)
wr
tristouc
Figure 12: Explanation plot of Tiger classification by GoogleNet from RLPO.
'black white and yellow' concept (1.000)
'square' concept (1.000)
'yellow and white' concept (0.800)
'yellow black white red' concept (0.800)
M
Figure 13: Explanation plot of Cab classification by ViT from RLPO.
19

---

Explainable Concept Generation through Vision-Language Preference Learning
'background' concept (1.000)
'banner' concept (1.000)
ENERHI
'basket' concept (0.600)
Figure 14: Explanation plot of Baskethall classification by Inception V3 from RLPO.
'zebra' concept (1.000)
'ship' concept (1.000)
'bumpy' concept (0.800)
'feather' concept (0 (0.800)
Figure 15: Explanation plot of Lionfish classification by Inception V3 from RLPO.
'yellow' concept (1.000)
Tuzzy concept (1.000)
curied up concept (1.000)
yellow and brown concept (1.000)
White brown and green concept (0.600)
Figure 16: Explanation plot of Golden Retriever classification by ViT from RLPO.
C.3.1. CUMULATIVE REWARDS
The cumulative rewards during training for GoogleNet and Inception is shown in Fig, 17. For ViT and Swin Transformer shown Fig. 18. This figure illustrates the steady accumulation of rewards over time as they interact with the reinforcement learning environment. All models demonstrate a steady increase in cumulative rewards, the classes with higher reward peak reaches its explinable state faster.
20

---

Explainable Concept Generation through Vision-Language Preference Learning
GoogleNet
Inception V3
40000
Tiger
Tiger Basketball
Zebra
35000
Goldfish
30000
Police van
Lionfish
25000
20000
15000
10000
10000
5000
100
200
300
400
500
100
200
300
400
500
Time
Time
Figure 17: Cumulative rewards on traditional models.
Swin Transformer
Vision Transformer
40000
Tiger
35000
Tiger
35000
Jay
Cab
Goldfish
30000
Goldfish
30000
Husky
Golden Retriever
25000
25000
20000
20000
15000
15000
10000
10000
5000
5000
0
100
200
300
400
500
100
200
300
400
500
Time
Time
Figure 18: Cumulative rewards on transformer models.
C.3.2. ACTION SELECTION OPTIMIZATION DURING RLPO TRAINING
As shown in Fig. 19, during training with multiple combinations of seed prompts, we observe that the RL agent initially explores various action combinations. However, as training progresses, individual actions become more optimized due to preference optimization (PO). This leads the agent to prefer fewer action combinations, since just choosing one or two actions makes the agent reach an explainable state.
C.3.3. CONCEPT HEATMAP
To determine the relationship between generated concepts and test images, made use of CLIPSeg transformer model (Lüddecke & Ecker, 2022). We passed generated concepts as visual prompts and test images query images into the model and it returns pixel-level heatmap of the probability of visual prompt in the query image. Fig. 20, 21 showcases some examples on concept heatmap indicating the presence of the concept in the image.
C.3.4. C-DELETION
The central idea behind c-deletion explainability identify and remove parts of the input context that are not crucial for the decision-making process, allowing for clearer insights into how the model arrives its predictions or actions. C-deletion evaluations assesses the impact of removing certain contextual inputs (features, variables, or states) on a model's performance as shown Fig. 24.
21

---

Explainable Concept Generation through Vision-Language Preference Learning
5.0
4.5
Action Combinattinn Langth
4.0
3.5
360 Steps
Figure 19: Combined actions (multiple keywords) count over training time
Class Image
Concept Heatmap
Class Image
Concept Heatmap
Figure 20: Van class with "white blue and yellow", Lion fish class with "zebra" seed prompt.
C.4. Abstract Concepts
While generating concepts with RLPO, we can observe the progression output concepts produced by the SD model. As shown Fig. 25, applying RLPO to the seed prompt "zoo" on the tiger class reveals different levels abstraction. These abstractions provide insight into what the model prioritizes when identifying tiger paec--progresiiggg from four-legged orange-furred animal to with black and white stripes, and finally We obtain concepts varying levels of abstraction by adjusting 1), though our method currently cannot determine 17 to achieve specific abstraction level.
orange-furred animal with stripes and whiskers.
C.5. Qualitative Comparison Between With and Without RL Preference Optimization
To demonstrate the usefulness of RL based preference optimization, compared the output generated from the diffusion model fine-tuned using RI with the one fine-tuned iteratively over every seed prompt (brute-force). As shown in Fig. 26, if we don't use RL to optimize, we see that stripes seed does not converge to good quality concept compared to the one obtained using RL for the same time budget. The main reason for that is, the RL agent learns over time what trajectories are worth optimizing and drops the less explainable trajectories, highlighting the need for using RL agent while optimization.
C.6. Qualitative Comparison With Other Popular XAI Techniques
We compare the output generated by other popular XAI techniques such as LIME and GradCam with ones generated by RLPO. As shown in Fig. 27, we can see that other methods just explains where the model is looking at whereas our approach also explains what type of features is the model focuses on.
22

---

Explainable Concept Generation through Vision-Language Preference Learning
Class Image
Concept Heatmap
Class Image
Concept Heatmap
GLAND S
10
Figure 21: Basketball class with "basket" seed prompt, Tiger class with "orange black and white" seed prompt.
Original Image
Segment
Segment
Segiment
Segment
Segment
Segment
Segment 7
Segment
Segment: 9
Segment 10
Original Image
Segment nt
Segment
Segment 3
Segment
Segment 5
Segment
Segment
Segment
Segment 9
Segment 10
Original Image
Segment
Segmen:
Segment
Segment
Segment 5
Segment
Segment 7
Segment
Segment 9
Segment
Figure 22: The figure shows c-deletion taking place for different images from "tiger" class over time for "orange black and white" seed concept.
Original Image
Seegment
Segmen::2
Segment
Segment
Segment
Segment
Segiment
Segmeent
Segment
Segment
Original tmrsge
Segmient
Segmennt
Segminn:
Segment
Segmennt
Segment
Segmentt
Segment
Segment
Segmmntt
Original Image
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Figure 23: The figure shows c-deletion taking place for different images from "cab" class over time for "yellow and white" seed concept.
C.7. RLPO in Sentiment Analysis
We extended our method to the NLP domain, successfully identifying which parts the input contribute to specific outputs. For sentiment analysis, binary classification problem, we present results for both positive and negative classes.
A list of positive and negative prompts created, analogous to class images in traditional image classification tasks. Random prompts, similar to those in Fig. 11, were used to simulate random classes. Every word in the prompt, excluding stop words, along with its synonyms, was treated as concept for this experiment. Synonyms were generated using the Mistral-7B Instruct model, serving role comparable to the image generation model image-based settings. We observed that multiple words were identified, along with their influence on the overall prompt, for both classes (positive and negative) as shown Fig. 9 and Fig. 28.
23

---

Explainable Concept Generation through Vision-Language Preference Learning
C-Deletion on Concepts in "Tiger" class
C-Deletion on Concepts in "Cab" class
Average Accuracy Score
1.0
1.0
"fuzzy" concept
0.6
0.8
0.838
*orange black white" concept
"black and orange" concept
Average Accuracy
0.6
0.8
3,830
6.673
6.259
"black and yellow" concest
0.4
0.4
"square" concept
0:390
"yellow and white" concept
0.2
0.2
"yellow black white red" concept
0.0
0.0
3
8
10
1
7
9
10
Steps
Steps
Figure 24: C-deletion. Removing concepts over time to measure the reliability. The colored numbers indicate the area under the curve (the lower the better).
Seed Prompt "200"
Timestep:
0
10
20
30
Tiger Class
Prediction:
"Oxcart"
"Sorrel"
"Ox"
"Tiger"
Figure 25: Different levels of abstraction for the "Tiger" class on the GoogleNet classifier are illustrated. The generated image starts random "zoo" image and gradually transitions images with tiger-like features. Observe that the seed prompt "zoo" becomes more animal-like at t=10, gains more stripes at t=20, develops tiger-specific colors at t=30, and progressively refines into tiger image. The model's prediction also evolves, starting from a random classification "oxcart" to confidently identifying the generated concept as "tiger".
C.8. Effects Fine- TineTTunngg
When fine-tuning model, the optimization process updates its weights through gradient-based methods, causing shifts in the concepts (Fig. 7) it Iearns. These weight adjustments modify how the model attends to different regions or patterns an image, leading to changes in the internal activation maps and the conceptual understanding the input. As model learns new concepts refines existing ones, it adjusts its feature extraction and decision makiing processes to better align with the specific objectives of the fine-tuning task, thereby altering the way it interprets and generates outputs.
To demonstrate this conducted an experiment. In this experiment, Tiger class whose important seed prompts were "orange black and white', 'orange and black', and 'blurry image' with TCAV scores of 0.66, 0.66, and 0.62, respectively. Out of these seed prompts, 'orange black and white' and 'orange and black' highlight the tiger pixels while "blurry image' seed prompt highlights the background pixels (see Fig. 12). This tells us that in order to classify a tiger, GoogleNet looks at both the foreground and background. Now, the engineers want the classifier to classify the tiger based on tiger pixels, not its background (note: from the classical Wolfe-Huske example in LIME (Ribeiro et al., 2016), know the spurious correlation of background).
choose a pretrained GoogleNet classifier for the
To this end, we generated 100 tiger images based on concepts related to 'orange black and white' and 'orange and black' using a separate generative model and fine-tuned our Googlenet model. Running RLPO on this fine-tuned model revealed that the model learned some new concepts such as 'whiskers' and also revealed that previous concepts such as 'orange black and white' and orangge and black' are now more important with TCAV scores 1.0 and 1.0, respectively. This means that the classifier now only looking at tiger pixels, not the background. Dataset samples are shown in Fig. 29.
24

---

Explainable Concept Generation through Vision-Language Preference Learning
"Stripes" concept without RL
ST,P..
SBO
SHTLG
SESET R SIS
"Stripes" concept with RL
Figure 26: "Stripes" concept generation from SD models optimized with and without RL
C.9. Human Survey: Understanding Human Capabilities
The survey involved 50 participants, each of whom was shown 10 class images along with two concept options as shown in Fig. 30: one derived from retrieval-based method and the other generated using RLPO. The participants were divided into Laymen and Experts.
1. Expert: Computer science graduates who are familiar with the concept of explainability and have a working knowledge of Al or machine learning systems.
2. Laymen: Individuals without expertise in computer science, AI, or explainability, representing the general public's perspective.
C.10. Human Study: Usability
We conducted a human study to evaluate the usefulness of the provided explanations, involving 19 Machine Learning (ML) engineers. Fig. 31 shows the survey used while conducting the human study. This study was approved by the Institutional Review Board of Arizona State University (IRB ID STUDY00021511..
25

---

Explainable Concept Generation through Vision-Language Preference Learning
Class Images
LIME
GradCam
RLPO (Ours)
Goldfish
Class
Tiger Class
Polica Van Class
Figure 27: Comparison of concepts identified by different methods. RLPO can show the correspondences between test image and different concepts.
Negative Prompt
Generated Concepts
6.7
0.8
The highly anticipated movié burned
colossal
Effects:
outcomes, consequences, impact, repercussions
disappointment, plagued weak and inceherent
Crities:
reviewers, criticisms, commentators, pundits
0.3
0.0
unconvincing perfomancee by the lead actors, lackluster special affects and numerous continuity emros,, which collectively made
Actors: performers, artists, thesplans, players,
Movie: flim, motion, picture, festure, show, production
it one of the worst tinematic experiences in recent memory.
Lackluster: apathetic, bland, uninspired, insipid
leaving audiences and critics alke utterly dissatisfted and frustrated.
Turned: faced, aimed, pivoted, swiveled, rotate, reversed
Figure 28: Generated concepts explain why a given text is classified as a negetive sentiment. Original Training dataset (from ImageNet):
New fine-tuning dataset (generated through Stable Diffusion 2):
Figure 29: Dataset samples used to fine-tune GoogleNet classifier on Tiger class.
26

---

Explainable Concept Generation through Vision-Language Preference Learning
Understanding Human Capabilities
Thank you for considering participation in our survey. Please read the following information carefully before proceeding.
When neural networks are trained using images, they identify and learn specific high-level concepts to recognize those images. Typically, as humans, we do not know what those high-level concepts are. survey, your task to guess, among two given options, which high-level concepts (also, represented images) could the neural network learned identify each test image.
Note: There is no one correct answer, the selection(s) are based on your belief and understanding. You can select none, one, or both concept images.
Purpose of the Survey: This survey is conducted solely for educational purposes to understand human opinions.
Data Use: The data collected through this survey will not be used for training any models, algorithms, or other computational tools. The primary use the data will be used understand human opinion and confined educational contexts. Confidentiality: Your responses will be treated with the utmost confidentiality. No individual data will be disclosed publicly or used outside the scope of the educational objectives stated.
Sign in Google to save your progress. Learn
Which of the following option(s) could be the reason for neural network to classify the following image as zebra?
Concept A
Concept
Figure 30: A screenshot from our human survey with instructions and a sample question.
27

---

Explainable Concept Generation through Vision-Language Preference Learning
Question:
the first you provided with five attribute that model could have learned during training. Now, for the input image (shown below), which has been following explanation attribute sets identified by mothod:
Question: 01
zebra,
important
Input images
The red highlighted area the imput corresponds Attribute set (Score: 0. 8.920). This attribute set has the highest score, indicating it was the most important
2. The green highlighted area corresponds to Attribute set 2 (Score: 0.800). This attribute set was also important but contributed slightly less to the classification Artribute 3. The blue highllihhted area corresponds to Attribute set (Score: 0.560). This attribute
lowest score model's decision.
important influencing
The scores represent the importance these attribute seta, with higher scores indicating influence the classification.
Impart Image:
of following attribute(()) cnuld classily the above images zebra?
Explaration:
Atribuae Cheere E01
Ateiett Choere B00
ADrOLrs art OSCDni (LMEE
Attribute set
Attribute set
Does the above explanation helps you in better understanding the neural
No
Figure 31: Screenshots from our human survey with sample questions to validate usability.
28


Reviews:
[{"review_id": "b7WC7ir9Y4", "reviewer": "Reviewer_zFMD", "summary": "The authors proposed a method to discover and visualize the \"concept\" or hidden knowledge learnt by a neural network. They proposed a reinforcement learning framework to achieve this goal. A score (TCAV) was proposed to evaluate whether a hidden representation of NN forms a concept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for identifying the novelty and appreciating the experiments. We hope the following explanations will clarify the queries for the reviewer. \n\n## Q1: It seems that the users need to have a good understanding of the target concept in order to generate good concept seeds (i.e, proper VQA design and possibly selective prompts). Is it possible to extract concepts from a large paragraph of description texts without prior human knowledge?\n\nWe agree with the reviewer’s observation that in our current setup, in order to generate good concept seeds proper VQA design and selective prompts are needed. But because of the modularity of our approach, the generation of seed prompts can be replaced by any other concept generation method. As described in [1], the end user can extract concepts from text descriptions without prior human knowledge and directly use it in our proposed method.\n\n[1] Zang, Yuan, et al. \"Pre-trained vision-language models learn discoverable visual concepts.\" arXiv preprint arXiv:2404.12652 (2024).\n\n## Q2: After the concept pictures are generated by SD+LORA, it is unsure how you can align them with the original input picture?\n\nWe employ CLIPSeg, a transformer-based segmentation model, to establish visual correspondence between generated concepts and regions in the input images (See Section 4.4 Figure 5). By feeding the generated concept images as prompts into CLIPSeg, we produce heat maps highlighting areas in the input image that resemble the generated concept. This allows us to localize abstract concepts (e.g., “stripes”, “mud”) within the original images, enabling interpretable alignment between concept space and class-specific features.\n\nWe hope these responses clarify your concerns. We appreciate your recognition of the work's broader implications and suggestions for future extensions, which we plan to incorporate.", "claims_and_evidence": "The authors provided quantitative and instance evidences that their algorithm successfully discovered hidden concepts.", "methods_and_evaluation_criteria": "The evaluation metrics sounds rational.", "theoretical_claims": "Theoretical analyses are provided. It is hard to follow the details but the framework sounds rational.", "experimental_designs_or_analyses": "The experiments are valid.", "relation_to_broader_scientific_literature": "How machine learning models learn concepts of the world from data is of wide interest in establishing foundation of AI. This work is a very interesting attempt.", "essential_references_not_discussed": "To the best of my knowledge the references are appropriate.", "other_strengths_and_weaknesses": "The work is of high novelty and would potentially impact a broad scope of machine learning. It would be better if more rigorous quantitative evaluations can be developed in the future (e.g., correlation between generated concepts vs. ground truth concepts). Also, the procedure of generating concept seeds are still a black-box (i.e., diffusion model), which should be improved in the future.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1）It seems that the users need to have a good understanding of the target concept in order to generate good concept seeds (i.e, proper VQA design and possibly selective prompts). Is it possible to extract concepts from a large paragraph of description texts without prior human knowledge?\n2）After the concept pictures are generated by SD+LORA, it is unsure how can you align them with the original input picture?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H8IU7fgUpc", "reviewer": "Reviewer_Sob6", "summary": "The paper reframes the concept set creation as a concept generation problems. It proposes a method based on generative model to generate concept images. It aims to create to reliably generate diverse concepts that are challenging to craft manually. The process involves various components including Reinforcement Learning-based Preference Optimization, Stable diffusion models, Testing with Concept Activation Vectors (TCAV) scores, BLIP models.\n\n## update after rebuttal\nI increased my score to 3 after the rebuttal from the authors. They were actively engaged in addressing my concerns, and most of them have been resolved. However, I still have some reservations regarding the reliance on the seed prompt and the variability of the CAV. While the authors provided some results related to this issue, they are limited to only a few examples and do not fully address my concerns.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s constructive comments and recognition of the importance of concept generation in our work. Our method improves upon traditional automatic concept retrieval approaches. While conventional methods extract concepts directly from the dataset—risking semantic information leakage—our approach generates novel concepts that are independent of the dataset.\n\nWe have addressed the reviewers' concerns as follows.\n\nPlease refer to this **[Anonymized GitHub Link](https://anonymous.4open.science/r/RLPO-9577/Sob6/readme.md)** where we have compiled detailed explanations for better understanding.\n\n## Q1: Reliability on text prompts.\nOur use of seed prompts serves primarily to narrow the search space and provide a reasonable initialization for the concept generation. Given that Stable Diffusion is conditioned on text prompts, starting with semantically meaningful phrases helps the RL agent converge faster toward relevant concept regions. \n\nWhile the generated concepts may initially align with the seed prompts, RLPO iteratively optimizes the generation process via TCAV-based rewards, allowing for drift and refinement toward more model-relevant abstractions—even beyond the original prompt scope. We demonstrate this evolution through multiple RL steps (e.g., Figure 2, Appendix D.4), and also show in ablation (Appendix C.4) that random prompts perform significantly worse, indicating the importance of a good starting point rather than dependence.\n\n## Q2: Hyperparameters used for TCAV score calculation.\nTCAV can be sensitive to choices like the layer of activation and the classifier used. To address this, as stated in Appendix C.1, we replaced the default SGD classifier with Logistic Regression, which we found provided more stable CAVs with lower variance. Regarding the choice of layers and target classes, we provide details in Appendix C.3.\n\n## Q3: Bias introduced by pre-trained models.\nWe acknowledge that the diversity of generated outputs depend on the generative model's capabilities. Issues such as insufficient representation of certain patterns could limit the range of explanations. However, such limitations are not unique to generative approaches—they are also inherent to retrieval-based methods, which are similarly constrained by available data and even human collected concepts, which are influenced by cognitive bias. However, we agree that this is a good point and we will include a discussion of this limitation in the revised manuscript. Specifically, we will highlight the dependency of the explanations on the generative model's capability to produce high-quality and diverse outputs. Thank you for pointing it out.\n\n## Q4: In Figure 6, it is unclear what x-axis “Steps” refers to.\n“Steps” in figure 6 refers to a step in c-deletion. At each step we delete a part of the image representing the target concept. We have provided more elaborate examples in Fig 21 and 22 of Appendix. We will clarify this in the figure caption of the revised manuscript.\n\n## Suggestions on related works\nThank you for the suggested references. We will include a discussion of Feature Visualization (Olah et al., Distill) and the more recent diffusion-based concept learning works (e.g., https://arxiv.org/abs/2312.02974, https://arxiv.org/abs/2410.05217). While our method relies on classifier feedback via TCAV and thus differs in motivation, your point about manual curation versus automated discovery is well-taken and worth addressing more directly in Section 2.", "claims_and_evidence": "- The paper claims that RLPO generates new concepts that explain model behavior. However, it also acknowledges that results depend heavily on the text prompts used. If concept discovery relies significantly on the seed prompt, the method may not be truly generating new concepts but rather refining existing ones based on prior knowledge. The generated images appear to closely follow the provided seed prompts. The authors argue that explaining concepts through images is more intuitive, citing “A picture is worth a thousand words, but words flow easier than paint.” However, they do not fully justify why image-based explanations are inherently superior, especially given the reliance on text prompts for concept generation.\n- The method relies on TCAV, which has known limitations. One key issue is the choice of hyperparameters, such as selecting which layers to focus on, which can significantly impact results. Additionally, unless many samples are used for training the classifier and obtaining Concept Activation Vectors (CAVs), the CAVs show a large variance across runs. The paper does not fully address these concerns.\n- The method is highly complex, involving multiple components such as reinforcement learning, preference optimization, and generative modeling. This complexity itself is a drawback for model explanation methods, which should ideally be simple and interpretable. Additionally, RLPO depends on many pretrained models such as Stable Diffusion and BLIP, which introduces external biases that are not accounted for in the paper.", "methods_and_evaluation_criteria": "The paper presents various evaluations, each assessing different aspects of the method, which is great. However, trade off between the dependence on seed prompts and the degree of genuine concept discovery may be further clarified.", "theoretical_claims": "No formal proofs were reviewed in detail. However, the effectiveness of RLPO is largely tied to TCAV, which is known to be sensitive to hyperparameter choices. The paper does not extensively analyze the robustness of TCAV in this setting.", "experimental_designs_or_analyses": "- The paper does not discuss how the authors chose the hyperparameters for TCAV, particularly which layers were selected for extracting activations. This is important, as TCAV scores are known to vary significantly depending on this choice.", "relation_to_broader_scientific_literature": "The paper is addressing an important problem of concept generation. However, it is unclear to me how methodologically it would have advantage over existing methods.", "essential_references_not_discussed": "It would be helpful to discuss how this paper relates to the following works. \nhttps://distill.pub/2017/feature-visualization/ Generating concepts images without any dependence on external input (such as BLIP's prompt seed suggestion in this paper)\nhttps://arxiv.org/abs/2312.02974, https://arxiv.org/abs/2410.05217 While not involving classifiers, they also try to define abstract concepts without manual curation.", "other_strengths_and_weaknesses": "Please see above.", "other_comments_or_suggestions": "No more comments.", "questions_for_authors": "- In Figure 6, it is unclear what x-axis “Steps” refers to.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XRvcDum5gc", "reviewer": "Reviewer_ADMa", "summary": "This work introduces an RL-based method to construct a vision-language concept-level preference dataset purely from synthesized images by taking TCAV score as the reward. It first prompts the trained MLLM to ground the common concepts represented as language phrases in the images. Then, it generate preference datasets with TCAV scores of ImageNet pre-trained models. Finally, the RLPO is applied to finetune the diffusion models. The results demonstrate the proposed method can guide generative models to generate more clustered concept images.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Please refer to this **[Anonymized GitHub Link](https://anonymous.4open.science/r/RLPO-9577/ADMa/readme.md)** where we have compiled detailed explanations for better understanding.\n\nWe are thankful to the reviewer for the feedback. While we appreciate the reviewer's recognition of the proposed framework, we believe there are some key misunderstandings regarding the claims and the experiments conducted in the paper.\n\nIn this work, we propose a novel approach to “generate” concepts that truly matter to the neural network by utilizing reinforcement learning-based preference optimization (RLPO) on diffusion models. And, we show qualitatively and quantitatively that the generated concepts matter to the neural network. This has been supported by Reviewer 1, 2 and 4. With regards to experiments, please see our detailed answers below.\n\n## Q1: Generalizability beyond visual concepts.\n\nRLPO does not solely depend on direct linguistic grounding. Instead it uses reinforcement learning to evolve and refine the generative model via XAI feedback (from the model under test) - indirectly capturing internal representation that may not have exact language equivalents. The use of language (seed prompts), serves primarily to narrow the search space and provide a reasonable initialization for the concept generation. While the generated concepts may initially align with the seed prompts, RLPO iteratively optimizes the generation process via TCAV-based rewards, allowing for drift and refinement toward more model-relevant abstractions—even beyond the original prompt scope. Additionally, to highlight the generalizability of our approach beyond visual concepts, in Section 4.6 (Fig. 8), we demonstrate how RLPO generalized to non-visual domains like sentiment analysis, using textual input and preference. \n\n## Q2: Clarification on C-deletion.\n\nWe agree with the reviewer that the original C-deletion is performed in the pixel space. But we would like to clarify that, the C-deletion we show in the paper is also performed in the pixel space and **not in the textual space**. As highlighted in Q1, the use of language is to narrow down the search space. Once the training is completed, we generate concepts via the trained diffusion model and map those generated concepts back to the input space using CLIPSeg (as shown in Figure 5). \n\nThe C-deletion graphs shown in paper are obtained from deleting most relevant to least relevant target concepts from the input images. We have provided more elaborate examples in Fig. 21 and 22 of Appendix.\n\n## Q3: Potential semantic leakage in diffusion alignment since the vision models are already trained on ImageNet? We need more details on the concept vocabulary, how many semantic names are out of ImageNet vocabulary? How many classes can it generate? Can it generate novel categories/objects?\n\nWe understand the reviewer’s concern on potential semantic leakage in diffusion alignment since these models have already been trained on ImageNet data, but we would disagree with the reviewer because the concepts we generate don't come from class data. As shown in Table 4, the generated concepts by RLPO are farthest from the class data. On a contrary, this semantic leakage occurs in retrieval based methods where the concepts are collected from class data. This is the exact problem we are trying to resolve with our method.\n\n## Q4: The action space is not scaled, only a few words. Besides, most prompts are single phrases, and thus cannot scale up to diverse compositions?\n\nOur current action space consists of 20 seed prompts, preprocessed and extracted using VQA (see Appendix C.3). We would like to emphasize that our approach does not rely on a direct mapping from seed prompts to the final generated outputs (concepts). While the generated concepts may initially align with the seed prompts, RLPO iteratively optimizes the generation process via TCAV-based rewards, allowing for drift and refinement toward more model-relevant abstractions—even beyond the original prompt scope. Consequently, the final generated concepts capture more diverse and model-relevant abstractions that extend beyond the limitations of the initial single phrases.\n\n## Q5: What if generated images in both groups are both garbage? Will it still update the SD?\n\nIf both image sets yield low TCAV scores (i.e., do not activate the model meaningfully), our method does not update the diffusion model. Only when a concept has the potential to move toward explainable states do we update the diffusion model. We will clarify this further in the revised manuscript.", "claims_and_evidence": "Are the claims made in the submission supported by clear and convincing evidence? If not, which claims are problematic and why?\n\nI think it is overclaim on Understanding Neural Networks’ Internal Representations, since it does not propose novel explainable methods to unveil the concepts of learned vision networks.\n\nAlso refer to **Relation to Broader Scientific Literature.**", "methods_and_evaluation_criteria": "The proposed method or framework is technically novel.", "theoretical_claims": "None", "experimental_designs_or_analyses": "See weaknesses.", "relation_to_broader_scientific_literature": "I am a little concerned about the implication of this work: this work essentially is attempting to align the vision-language alignment of generative models pre-trained with noisy image-text pairs. It cannot generalize beyond the abstract visual concepts that cannot be described by language. Therefore, the ultimate results achieved is more related to aligning the generative model in the word or phrase level.", "essential_references_not_discussed": "It seems to miss a line of related works on concept learning with diffusion models.\n\nFor example\n\nConceptLab: Creative Concept Generation using VLM-Guided Diffusion Prior Constraints", "other_strengths_and_weaknesses": "Weaknesses:\n\n- Not well-organized: The current version of the presentation does not meet the ICML standards, therefore I would suggest the author revise their writing to clarify and reorganize the method and experimental sections. For instance, the evaluation metrics and motivations behind them are not clarified. Some expressions are not academic and rigorous enough.\n- Missing technical details: to my understanding, the original C-deletion is performed in the pixel space, but this method is performed in the textual space.\n- Potential semantic leakage in diffusion alignment since the vision models are already trained on ImageNet? We more details on the concept vocabulary, how many of semantic names are out of ImageNet vocabulary? How many classes can it generate? Can it generate novel categories/objects?\n- The action space is not scaled, only a few words. Besides, most prompts are single phrases, and thus cannot scale up to diverse compositions?\n- What if generated images in both groups are both grabage? Will it still update the SD? It is necessary to include additional quantitative metrics to distinguish between diversity and quality. It may need human study or auxiliary scores, e.g., train an additional network to compute classification accuracy.", "other_comments_or_suggestions": "I would suggest that the author polish this script entirely for another round.", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SNedBbf6a4", "reviewer": "Reviewer_Rqpq", "summary": "The paper addresses a critical challenge in concept-based explanation methodology—specifically, the generation of \"concepts\" for explanations. Traditionally, this required practitioners to manually guess and collect various candidate concept image sets. The paper introduces a novel approach, utilizing reinforcement learning-based preference optimization (RLPO), to guide the Stable Diffusion model in generating concepts that are significant to the neural network's internal representations. Both qualitative and quantitative results are presented to demonstrate the effectiveness of RLPO in uncovering these representations.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and for recognizing the technical soundness and novelty of our framework. We appreciate your critical insights and provide detailed clarifications below.\n\nPlease refer to this **[Anonymized GitHub Link](https://anonymous.4open.science/r/RLPO-9577/Rqpq/readme.md)** where we have compiled detailed explanations for better understanding.\n\n## Q1: Concerns about seed prompt acquisition.\n\nWe agree with the reviewer’s observation that our current setup is primarily focused on low-level visual concepts. But our current setup can easily be modified to capture higher-level semantic concepts (e.g., gender, age) by adding task specific questions like, “What is the gender of the person in the image?” or “Which age category does the person in the image belong to (young/old)?”. \n\nTo verify the usability of the proposed methodology with higher-level semantic concepts like gender, we trained a Resnet18 classifier on CelebA dataset to classify images as “Blonde” and “Not Blonde”. This dataset is known for having a spurious correlation between class “Blonde” and females. With our method, we were able to find the same correlation. Concepts  generated for the female face were more important than male face. As a side note, when we train RLPO to capture higher-level semantic concepts, it starts combining one or more than one low-level features. As shown in examples on **[anonymized github](https://anonymous.4open.science/r/RLPO-9577/Rqpq/readme.md)**, the generated samples start developing long and blonde hairs for both male and female concepts.\n\n## Q2: Concerns regarding limited evaluation.\n\nWe would like to clarify that, while the main text includes examples like “zebra” and “tiger” for clarity and interpretability, RLPO was evaluated on a broader range of randomly selected samples from ImageNet (see Section 4.3, Appendix D.3, and Table 4). RLPO has also been tested on multiple pretrained models such as GoogleNet, InceptionV3, indicating RLPO is not tied to a specific architecture. \n\nAdditionally, as mentioned in Appendix D.9, rather than predominantly featuring specific cases, while conducting the human survey we considered 10 unique classes and randomly selected examples. Apart from that, we also demonstrate how RLPO generalized to non-visual domains like sentiment analysis (see Section 4.6 and Fig. 8).\n\n## Q3: Justification for the assumption that $\\text{C}_H \\subseteq \\text{C}_G$ and $\\text{C}_R \\subseteq \\text{C}_G$.\n\nIntiutively, the generative models like Stable Diffusion are trained to learn the distribution of real-world data (which contains human defined and retrieved concepts). By leveraging this learned distribution, they can create existing or new data which by design can represent human defined or retrieved concepts (or neither). That being said, to solidify this assumption we plotted the clip embeddings of generated, retrieved and human defined stripes concept. For retrieval based concepts we used the stripes collected by CRAFT for the zebra class, for human defined concepts we used the concepts collected by TCAV authors for the zebra class, and for generated concepts we used pre-trained stable diffusion 1.5 to generate random stripes images. As shown in the plot on **[anonymized github](https://anonymous.4open.science/r/RLPO-9577/Rqpq/readme.md)**, the generated stripes encapsulates both human-defined and retrieved concepts hence proving the assumption $\\text{C}_H \\subseteq \\text{C}_G$ and $\\text{C}_R \\subseteq \\text{C}_G$.", "claims_and_evidence": "In my view, the claims made in the paper are generally well-supported. However, I have reservations about the qualitative results of the method, as detailed below.", "methods_and_evaluation_criteria": "The RLPO's approach to encouraging diffusion models to generate truly meaningful concepts is technically sound and innovative. The authors tried to address the critical limitations of previous concept-based explanation methods.\n\nHowever, I have concerns about the seed prompt acquisition. While Appendix C.3 details the prompts used to probe image patches, these seem primarily focused on low-level visual concepts. The method's ability to capture higher-level semantic concepts (e.g., gender, age) remains unclear. Additionally, it's uncertain whether the stable diffusion model can generate meaningful representations for such abstract concepts. I would appreciate the authors' perspective on this limitation.\n\nRegarding evaluation, I have reservations about the generalizability of the results. The paper presents selected examples, predominantly featuring specific cases (zebras, tigers). While user studies were conducted, they also relied on these selected examples rather than randomly sampled cases. A more robust evaluation would involve testing with random examples across diverse scenarios.\n\nFurther evidence is needed to demonstrate the method's effectiveness across broader applications. Additionally, the paper would benefit from including and analyzing failure cases to better understand the method's limitations. For instance, the concepts produced by RLPO in Figure 4, while diverse, don't appear particularly meaningful.", "theoretical_claims": "Regarding Figure 1 and its associated theorem (including proofs), the underlying assumptions should be more explicitly stated. In particular, I would to like see clearer justification for the assumption that \"$C_H\\subseteq C_G$ and $C_R\\subseteq C_G$\".", "experimental_designs_or_analyses": "Please see above for my concerns regarding the user study and the limited scope of the qualitative results.", "relation_to_broader_scientific_literature": "The relation to broader scientific literature was discussed in the Section 1 and 2 of the paper.", "essential_references_not_discussed": "The relation to broader scientific literature was discussed in the Section 1 and 2 of the paper.", "other_strengths_and_weaknesses": "In general, I find the method proposed in the paper technically sensible, and I agree that issue tracking is crucial in concept-based explanations. However, I have reservations about the assumptions made in the paper. More importantly, I hold reservations about the qualitative results of the methods in broader cases.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper got 1 accept and 3 weak accept rating.

Before rebuttal,

Reviewers thought the strength of this paper are: 1) The work is of high novelty and would potentially impact a broad scope of machine learning (Reviewer zFMD); 2) The paper is addressing an important problem of concept generation (Reviewer zFMD); 3) The proposed method or framework is technically novel (Reviewer ADMa); weaknesses are: 1) problems of the seed prompt (Reviewer zFMD, Sob6, Rqpq); 2) The method relies on TCAV, which has known limitations (Reviewer Sob6); 3) The method is highly complex, involving multiple components (Reviewer Sob6); 4) paper is Not well-organized (Reviewer ADMa); 5) Missing technical details (Reviewer ADMa); 6) The action space is not scaled, only a few words (Reviewer ADMa); 6) hold reservations about the qualitative results of the methods in broader cases (Reviewer ADMa).

After rebuttal,

All the reviewers confirmed they had read authors' response and would like to update the review if needed. Reviewer zFMD kept the rating. Reviewer Sob6 raise the score, but still had some reservations regarding the reliance on the seed prompt and the variability of the CAV and thought authors' did not fully addressed their concerns. Reviewer ADMa also raised score. Reviewer Rqpq raised score but still had reservations regarding the generalizability of the proposed method.

Considering this AC decided to give weak accept rating and hope authors could incorporate reviewers' comment to improve the paper.
Decision: Accept (poster)