Explainable Concept Generation through Vision-Language Preference Learning

Authors: Aditya Taparia, Som Sagar, Ransalu Senanayake
Primary area: generative models
Keywords: Concept based Explainable AI, Vision-Language Models, Reinforcement Learning

Abstract:
Concept-based explanations have become a popular choice for explaining deep neural networks post-hoc because, unlike most other explainable AI techniques, they can be used to test high-level visual "concepts" that are not directly related to feature attributes. For instance, the concept of "stripes" is important to classify an image as a zebra. Concept-based explanation methods, however, require practitioners to guess and collect multiple candidate concept image sets, which can often be imprecise and labor-intensive. Addressing this limitation, in this paper, we frame concept image set creation as an image generation problem. However, since naively using a generative model does not result in meaningful concepts, we devise a reinforcement learning-based preference optimization (RLPO) algorithm that fine-tunes the vision-language generative model from approximate textual descriptions of concepts. Through a series of experiments, we demonstrate the capability of our method to articulate complex and abstract concepts which aligns with the test class that are otherwise challenging to craft manually. In addition to showing the efficacy and reliability of our method, we show how our method can be used as a diagnostic tool for analyzing neural networks.

Decision: reject
Reviewer scores: 5, 3, 8, 8, 6, 3

Reviews and discussion:
<thread>
creation date:
1729604837113

thread_type:
official_review

<thread_object>
creator: reviewer_d5kj

<content>
summary:
Some concept-based XAI methods, such as [1,2], require the creation of a concept-specific set of images to pass through the network. To eliminate the human-in-the-loop, the authors explore using a combination of reinforcement learning and generative models to create concept-specific image sets. The authors stated goals include:
 (1) producing concept sets that are beyond what human practitioners may be able to discover on their own. 
 (2) producing concepts at a variety of abstraction levels that they are able to control with a parameter. 
 (3) producing concept sets that demonstrate an increases in novelty, abstractness, diversity, generalizability (to non-vision domains) and actionability (utility).

The core of the method is to optimize a LORA-adapted stable diffusion model to produce concept sets that trigger the target model (model to be explained). The RL agent is used to search a set of text prompts for the diffusion model for the text prompts. 

[1] Kim, Been, et al. "Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav)." International conference on machine learning. PMLR, 2018.
[2] Schut, Lisa, et al. "Bridging the human-ai knowledge gap: Concept discovery and transfer in alphazero." arXiv preprint arXiv:2310.16410 (2023).
[3] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685 (2021).

strengths:
(S1) The authors identify an interesting question: how to choose a probe set for extracting concepts from models? 
The proposed method considers an interesting direction of using generative methods to create images that may not be present in the dataset. 
(S2) The formulation of considering TCAV scores as preference scores is interesting. 
(S3) The authors make an effort to quantitatively assess the concept images generated by their method.

weaknesses:
(W1) The authors stated goals for this work are misaligned with the goals of explainable AI in general. The  goal of XAI is to improve a user's understanding of a model as a primary goal. However, the authors focus on "abstractness" and "novelty" as the goals of their method. While the authors propose several experiments to measure abstractness and novelty, they fail to provide any convincing experiments on the utility of their method for explaining network behavior. For example, [1] provides a clear, well-motivated experiment, i.e. users are asked to predict model behavior when given an explanation.

(W2) Additionally, I find the term novelty to be misleading. The term novelty can be interpreted in at least two ways, for example, it could be interpreted as how distinct different discovered concepts are from each other. When measuring novelty in this manner, it seems that the author's method may not exhibit novelty, since in Line 288 they state, "it is possible for different actions to lead to the same explainable state." Instead, the authors measure novelty as the distance between concept images and images the test set. They use this metric to compare their work to prior work, resulting in the fairly trivial result that generated images are farther from the test set than methods that use the test set. 

(W3) The ablation study for choosing TCAV is unnecessary. The authors choose to use either a human, LLM, or TCAV scores from the model to make preference judgements. However, since the goal is to explain the model the other first two options are completely unnecessary. 

(W4) The authors claim to be able to generate concepts sets at different levels of "abstraction". However, whether this "abstraction" is due to SD or due to the target model is unclear. Since, the goal is to understand the target model, once again, I find this experiment to be insufficient to say anything interesting about the target model. 

(W5) Finally, the computational cost of this method is quite high and it is unclear if the results are worth the cost. 

In summary, the authors lack a key experiment measuring the utility of their method. Additionally, I find that some experiments that the authors conduct to have trivial results. 

There are several places with typos. 
L91 mode -> model
L534 the term NUT is introduced without being defined?

[1] Colin, Julien, et al. "What i cannot predict, i do not understand: A human-centered evaluation framework for explainability methods." Advances in neural information processing systems 35 (2022): 2832-2845.

questions:
The most important question is: can the authors provide any experimental result that concretely shows that their method is able to explain model decisions? 

Some secondary questions are: 
1) Why RL at all? Why not just optimize every "seed prompt"?
2) How often do seed prompts converge to the same outputs? 
3) This work brings up the question, under what domain should we care about explaining model behavior? The generated images are OOD for the model. When do we want to explain OOD behavior for the model? Is this useful?
4) How does your method relate to adversarial attack methods?
5) How long does it take for your method to run?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730385755799

thread_type:
official_review

<thread_object>
creator: reviewer_5zn5

<content>
summary:
This paper introduces an algorithm for generating concept images to post-hoc explain a model. It hereby focuses on the use of the TCAV score, an RL learning setting and stable diffusion.

strengths:
The paper tackles an important topic, and I find the combination of previous approaches that the authors propose with their work intuitive and interesting.

weaknesses:
Although i find the algorithmic approach interesting enough, I am unsure what the actual underlying goal is of the approach and how this is evaluated/evidenced in the experimental section. I.e., in the intro the authors mention "Therefore, it is importantto automatically find human-centric concepts that matter to the DNN’s decision-making process." So my understanding is that the ultimate goal is to provide concepts that are helpful for humans to understand the decision making processes, but without the need to provide extensive concept data precollection. But I am missing a clear experiment to evaluate the usefulness of the generated concept images. Table 2 seems to be hinting at something along this line, but the details are not clear to me from the text. IN any case this should be one of the key evaluations to perform. Especially, as the exemplary concept images look far too abstract for me to be able to understand what they should represent. Please point me towards the relevant sections in case this is missing. 

Overall, I am leaning towards accept, but would like the following points clarified first:
- What is the exact goal of the method?
- How do the experimental evaluations provide evidence for reaching this goal?

Minor: Overall, the paper could benefit from a grammar check, e.g., "also explains what type of features is the model focuses on." in line 426

questions:
Maybe I missed it, but is there any information on the training time of the proposed algorithm? If not, I think this could be valuable information to provide at least in the appendix, but mentioned in the main text. The authors mentioned limitations, but it would be good to have real numbers.

Whats the difference between the concept images in Figure 6 and 5? In figure 5 they look very abstract, whereas in figure 6 the are high-quality images.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730569065556

thread_type:
official_review

<thread_object>
creator: reviewer_tzbx

<content>
summary:
The paper proposes to treat the concept set creation as an optimisation problem. Through the means of deep reinforcement learning, it iteratively refines the concepts obtained using Stable Diffusion generative model so one can generate concepts instead of retrieving them in the existing data. They also introduce the concept generation with respect to an abstraction level. The authors also explore the ideas of analysing the concepts during fine-tuning and NLP problems.

strengths:
Novelty: the work builds upon existing methods in interpretability, such as concept based explanation, but contains a set of novel ideas including (1) the problem statement and the method for optimisation of concept set through reinforcement learning technique and LoRA (2) the idea to use such concept set generation for to produce explanations for different abstract layers

Clarity: the paper is clearly written and well presented. 

Motivation: The authors give a strong motivation for the proposed method: the retrieved data might not filly describe the concepts embedded into the model, and the optimisation of such concepts using generative models and RL can be a promising alternative

Significance: I think this work is significantly improving the available options for explaining visual (and potentially language) models and therefore I believe it is significant enough. 

Correctness: I checked the paper and I believe the claims and the maths are correct

Reproducibility: as far as I can see, taking into account both the paper and the code, the experiments look reproducible to me.

weaknesses:
Clarity: there are a few questions regarding the limitations of this work (see below in the questions section)

questions:
Questions:
1. “As a specific application, we see what concepts are removed and added, as well as how the concept importance changes when we fine-tune ResNet50 model on ImageNet to improve accuracy” If we want to track the evolution of concepts during the training procedure, do we need to retrain the concepts every time we update the model, or is there any workaround which would help track the concepts during the finetuning process? Either way is fine, however might be good to clarify it in the paper or in the appendix. 
2. Figure  9 should have the x axis annotated ( I understand it represent concept identifier)?
3. One of the concerns might be the lack of systematic quantitative comparison with the retrieval-based explanations. Is there any possibility to compare the explanations numerically with the state-of-the-art prototypical retrieval-based explanation methods, such as, e.g. Craft (Fel et al, 2023)?
4. I think there is one important limitation of the proposed method which I would like the authors to cover in the paper: the explanations also, in a way, depend upon the inner working of the generative model. Imagine, for example, that the generative model has mode collapse and does not represent the whole set of patterns available to the model we explain. In this case, it would lead us to obtaining the best possible explanation amongst the suboptimal ones, which may lead to the explanations not representing some aspects of the model’s inner working. In a case when it is a safety-critical application, that mode collapse might represent some anomalous event, which may not be covered by the set of explanations achievable by the model and therefore would not allow us to understand the reasons behind the model. A variation of such problem might include the problem of explanation when the generator only offers poor quality of image output(for whatever reason, e.g. it does not cover some particular concept). To make it clear, I understand that there is also a counterpart to such limitation in a case of standard, retrieval-based, prototypical explanation: there might not be such a piece of data which would closely match the phenomenon. It is therefore, in one way or the other, a limitation of many post hoc explanation models. I wonder if the authors agree with such limitation, and in any case would like to ask to include  the discussion. 
5. In relation to that, perhaps not mandatory, but another idea of an experiment: how does the numerical performance of the algorithm, e.g. Figure 8, would compare for different image generators? Does the GAN model, which is prone to mode collapse, result in worse C-insertion metrics?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730617465993

thread_type:
official_review

<thread_object>
creator: reviewer_tyud

<content>
summary:
Authors proposed a new algorithm, Reinforcement Learning-based Preference Optimization (RLPO), designed to generate high-level visual concepts that explain the decisions of DNNs. Unlike traditional concept-based explanation methods (such as TCAV), RLPO creates sets of concept images, eliminating the need for manual concept image collection, thus making the process of explaining DNN decisions more efficient and generalizable. RLPO fine-tunes a stable diffusion model with preference optimization to generate images that effectively explain neural network decisions.

strengths:
The idea is very innovative, and the writing is clear. By using a generative model instead of traditional manual concept collection, it reduces human intervention and improves efficiency. RLPO can generate concepts at different levels of abstraction, offering a more detailed explanation of the DNN’s internal decision-making process.

weaknesses:
In some sections (such as the algorithm explanation and mathematical proofs), the descriptions appear overly lengthy, which might hinder readers' understanding. The description of how RLPO is applied in sentiment analysis tasks is somewhat vague, and further detailing the specific steps of this experiment could be helpful. Certain terms (such as “concept generation” and “concept extraction”) are defined and used inconsistently throughout the paper, which could lead to confusion.

questions:
Same as weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693771881

thread_type:
official_review

<thread_object>
creator: reviewer_afj3

<content>
summary:
Concept-based explanation methods typically require practitioners to guess and gather multiple candidate concept image sets, a process that can be imprecise and labor-intensive. To address this challenge, this paper redefines the creation of concept image sets as an image generation problem. To this end, in this work, the authors introduce an RL-based preference optimization algorithm that fine-tunes the vision-language generative model using approximate textual descriptions of concepts. They also conduct extensive sets of experiments to demonstrate that the proposed method effectively articulates complex and abstract concepts that align with the target class, which are often difficult to create manually.

strengths:
The paper is well-written and easy to follow. The problem definition is novel and interesting.  The paper effectively reframes the generation of concept image sets as an image generation problem, providing a novel perspective that addresses the limitations of traditional concept-based explanation methods. This shift enhances the efficiency and effectiveness of generating meaningful concepts.
Through a series of well-designed experiments, the paper demonstrates the capability of the proposed method to generate complex and abstract concepts that align with the target class. This empirical evidence strengthens the paper's contributions.

weaknesses:
While the authors demonstrate the effectiveness of their proposed method through both qualitative and quantitative analyses, several aspects of the framework are not thoroughly justified. For instance:

(a) The generated concept sets using SD+LORA are randomly divided into two groups. Why is "random" grouping considered optimal? Could this be problematic?

(b) The use of reinforcement learning (RL) is also not justified. Is it really necessary to implement an RL policy for the purpose mentioned in the paper? Why not simply calculate the TCAV score for each possible seed prompt kt, and then fine-tune the SD+LORA weights based on the seed prompt that yields the highest TCAV value?

(c) Incorporating each component of the proposed framework increases computational demands and time constraints. Consequently, such an analysis may not be feasible in real-time, as generating images with Stable Diffusion, training the DQN-based RL policy, and fine-tuning the Stable Diffusion model with preferences all require significant processing time. Is this complex pipeline truly necessary?

(d) I also struggle to see a practical significance for the proposed problem. Why is it important to generate a diverse set of concept images at all? It seems more crucial to generate concepts that directly explain the task at hand.

(e) Furthermore, why choose DQN? As we continuously update the LORA weights based on the TCAV preference score, the underlying RL environment becomes non-stationary. This means that the same action taken by an RL policy could lead to different reward values at different times. How can this issue be addressed?

questions:
Please see my comments on the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703911778

thread_type:
official_review

<thread_object>
creator: reviewer_knkw

<content>
summary:
This paper focuses on the generation of concept images to explain black-box image classification models. It proposes a reinforcement learning-based preference optimization (RLPO) algorithm to fine-tune a diffusion model for generating images that can maximize the TCAV scores. It also proposes to use DQN to search appropriate actions from the seed prompts. Experiments show that the proposed approach can generate complex and abstract concepts aligning with the test class.

strengths:
1. Optimizing diffusion models to maximize TCAV scores sounds interesting and original to me.
2. The authors performed extensive experiments to demonstrate the effectiveness RLPO.
3. Most parts of the paper are clearly written and easy to follow.
4. The visual results presented are helpful in illustrating the advantages introduced by RLPO.

weaknesses:
1. While diffusion models make the whole system more flexible than "retrieval methods" by generating new images that may not be presented in current datasets, the quality of such images is questionable. For example, in Figure 5, the generated image at timestep 30 looks unrealistic and is not closely aligned with the seed prompt (worse than the initial image).
2. Though Table 1 shows RL helps the model to select approprate seed prompts from the given set, I'm still concerned about the necessity of using DQN in the framework. VLM/LLM can be used to generate a small high-quality seed prompt set, and even rank them based on their potential importance (e.g., the experiment in this paper has a set with only 20 prompts). One can use the whole set and focus on the fine-tuning of the diffusion model. 
3. The efficiency seems to be a problem.

questions:
1. While Tabel 4 shows the concept images generated by RLPO are more diverse, how do we know if they are faithfully reflecting the the same concept instead of overfitting the TCAV?
2. Why should the problem be designed as a sequential decision-making problem? 
3. Do people need to finetune one RLPO framework with DQN+Diffusion for each class? Would it be more efficiently and equally effective if they just use an untuned Diffusion model and prompt LLMs/VLMs to generate more detailed text descriptions for the target class given the seed prompt?
4. What is $t_\eta$ in Property 2?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732269228223

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank all the reviewers for their detailed and constructive feedback. We deeply appreciate your recognition of the novelty and potential impact of our approach, as well as the thoughtful questions. In this response, we categorize the feedback into two sections for clarity:
- Weaknesses (W): Concerns raised in the reviews.
- Questions (Q): Specific queries or clarifications sought regarding the methodology, results, or framework.

We address all points comprehensively, providing additional context, experimental evidence where necessary, and detailing revisions planned for the manuscript to address your feedback effectively.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733305408894

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the rebuttal process to all reviewers and AC

comment:
We sincerely thank all the reviewers for the discussion and appreciate AC’s efforts in overseeing the review process. We appreciate all the detailed and constructive feedback. It has helped us refine and clarify our work. We have addressed all the concerns and clarified our contributions as well as potential misunderstandings. 

We explained the distinction between the roles of "creator humans" (who design concept sets) and "user humans" (who interpret the results). Our method primarily targets creator humans by automating concept creation, a tedious and perhaps even impossible task for humans. We also elaborated on the role of abstraction in our method, emphasizing that abstractions, such as moving from "zoo" to "stripes," are a natural byproduct of the approach, offering optional and insightful layers of understanding without being a primary claim. Furthermore, we validated the novelty of the generated concepts through quantitative metrics and qualitative assessments. We also showed the importance of RL in our framework as requested by reviewer AFj3, KnkW, D5KJ. Reviewers KnkW, TZBx already agrees that RL plays an important role in learning over time—what trajectories to optimize and what to drop so that we find explanations in the fastest way.

The additional experiments conducted during the rebuttal process further solidify our contributions. As requested by reviewers KnkW and 5Zn5, we conducted additional experiments to validate the faithfulness of generated concepts. These experiments showed that the generated concepts are unique, consistent across runs, and represent distinct distributions for different seed prompts. 

Most importantly, we show that our method can provide **actionable/useful insights** through our additional experiments where concept shifts were effectively directed through fine-tuning to show  (concerns raised by reviewers KnkW and D5KJ). Finally, a human survey validated that the generated concepts improve the understanding of model behavior, showing the practical utility for debugging and analysis. 

We are grateful for the reviewers' input, which has greatly strengthened our submission. With these we believe we have addressed all the concerns raised by the reviewers and we are hopeful that reviewers D5KJ, AFj3, KnkW will reconsider the original review.
</content>
</thread_object>

</thread>


Metareview:
metareview:
Scientific Claims and Findings:

This paper introduces a novel algorithm called Reinforcement Learning-based Preference Optimization (RLPO) to automatically generate visual concepts that explain the decisions of deep neural networks (DNNs).  The algorithm addresses the limitation of traditional concept-based explanation methods (like TCAV) which require manual collection of concept images.  The authors claim that RLPO generates more novel and abstract concepts compared to traditional methods, offering a detailed understanding of the DNN's decision-making process.  They also suggest that RLPO is generalizable to non-vision domains, specifically NLP tasks.    

Strengths:

The paper is well-written and easy to follow.    

The proposed method is novel and addresses a significant limitation in existing XAI methods.    

The use of a generative model to create concept sets is innovative and reduces human intervention.    

The method can generate complex and abstract concepts, offering a detailed explanation of the DNN's decision-making.    

Weaknesses:

Some reviewers found the explanations of the algorithm and mathematical proofs to be overly lengthy.    

The practical significance of generating a diverse set of concept images wasn't immediately clear to all reviewers.    

The necessity of using RL and the complexity of the pipeline were questioned.    

The paper could benefit from a grammar check and consistent use of terms.    

Missing elements:

Clear and concrete examples of how the generated concepts can be used to debug or improve DNNs.

A more thorough justification for design choices, such as the use of RL and DQN.

A clearer explanation of how the method is applied in sentiment analysis tasks.

Reasons for Rejection:

The paper does not adequately demonstrate the practical usefulness of the proposed method for explaining network behavior.    

The experiments focus heavily on novelty and abstractness, but their connection to improving user understanding is not well established.    

The paper lacks a key experiment directly measuring the usefulness of the generated concepts for explaining model decisions.