Enhancing the Comprehensibility of Text Explanations via Unsupervised Concept Discovery
Authors: Yifan Sun, Danding Wang, Qiang Sheng, Juan Cao, Jintao Li
Accepted at: acl25

Enhancing the Comprehensibility of Text Explanations via Unsupervised Concept Discovery

Yifan Sun1,2  Danding Wang1*  Qiang Sheng1  Juan Cao1,2  Jintao Li1,2
1Institute of Computing Technology, Chinese Academy of Sciences
2University of Chinese Academy of Sciences
{sunyifan23z, wangdanding, shengqiang18z, caojuan, jtli}@ict.ac.cn

Abstract

Concept-based explainable approaches have emerged as a promising method in explainable AI because they can interpret models in a way that aligns with human reasoning. However, their adaption in the text domain remains limited. Most existing methods rely on predefined concept annotations and cannot discover unseen concepts, while other methods that extract concepts without supervision often produce explanations that are not intuitively comprehensible to humans, potentially diminishing user trust. These methods fall short of discovering comprehensible concepts automatically. To address this issue, we propose ECO-Concept, an intrinsically interpretable framework to discover comprehensible concepts with no concept annotations. ECO-Concept first utilizes an object-centric architecture to extract semantic concepts automatically. Then the comprehensibility of the extracted concepts is evaluated by large language models. Finally, the evaluation result guides the subsequent model fine-tuning to obtain more understandable explanations. Experiments show that our method achieves superior performance across diverse tasks. Further concept evaluations validate that the concepts learned by ECO-Concept surpassed current counterparts in comprehensibility.

1 Introduction

Deep neural language models lack explainability. A recent way to tackle this issue is concept-based explanations (Achtibat et al., 2023; Poeta et al., 2023), which map the inputs to a set of concepts and measure the importance of each concept to model predictions. By offering human-understandable attributes, concept-based explanations better resemble the way humans reason and explain.

Some existing concept-based methods apply post-hoc analysis to a trained model, providing concepts that either explain a model’s prediction (Kim et al., 2018; Goyal et al., 2019; Ghorbani et al., 2019; Jourdan et al., 2023) or elucidate its internal network structure (Dalvi et al., 2022; Sajjad et al., 2022; Bills et al., 2023; Bricken et al., 2023). Since the model was not exposed to these concepts during training, it may lead to unfaithful interpretations or meaningless concept activations (Rudin, 2019). Self-explaining concept-based methods offer a more reliable approach by providing built-in explanations along with their predictions. However, most of these methods require abundant annotations (Tan et al., 2024a,b; De Santis et al., 2024) for predefined concepts, which shows their limitations in automatically discovering new concepts during training. Though there exist self-explaining methods that extract concepts without concept annotations (Rajagopal et al., 2021; Antognini and Faltings, 2021; Das et al., 2022), the extracted concepts often lack human comprehensibility due to the ignorance of such constraints to guide concept extraction. Consequently, the resulting concept explanations may highlight irrelevant or mislead-

<table>
  <tr>
    <th></th>
    <th>Beer Review</th>
    <th>Beer Review</th>
    <th>Beer Review</th>
  </tr>
  <tr>
    <td rowspan="4">①</td>
    <td><b>Preddefined Concepts</b><br>Aroma<br>Taste<br>Palate</td>
    <td><b>Concept Extractor</b></td>
    <td><b>Concept Extractor</b></td>
  </tr>
  <tr>
    <td>Linear Layer</td>
    <td>Linear Layer</td>
    <td>Linear Layer</td>
  </tr>
  <tr>
    <td>Prediction</td>
    <td>Prediction</td>
    <td>Prediction</td>
  </tr>
  <tr>
    <td>Appearance +1<br>Aroma 0<br>Palate +1<br>Taste +1</td>
    <td>A clean lager that will quench thirst on a hot summer day.<br>The beer is crystal clear golden color with a descent head. Easy drinking cold and as they say “refreshing” style.</td>
    <td>A clean lager that will quench thirst on a hot summer day.<br>The beer is crystal clear golden color with a descent head. Easy drinking cold and as they say “refreshing” style.</td>
  </tr>
  <tr>
    <th colspan="4">①SUP. Baseline  ②UNSUP. Baseline  ③ECO-Concept</th>
  </tr>
  <tr>
    <th>Discover new concepts</th>
    <td>❌</td>
    <td>❌</td>
    <td>✔️</td>
  </tr>
  <tr>
    <th>Easy to comprehend</th>
    <td>✔️</td>
    <td>❌</td>
    <td>✔️</td>
  </tr>
</table>

Figure 1: Comparison of explanations between our proposed ③ ECO-Concept and existing typical ① supervised and ② unsupervised concept-based methods. Supervised methods explain based on predefined concepts, while ECO-Concept and unsupervised methods explain via concept-related highlighted text. ECO-Concept eliminates the need for concept annotations and can discover unseen concepts with improved comprehensibility.
ing information, bringing more confusion (demonstrated by previous research (Chaleshtori et al., 2024)) and less trust in humans (revealed by our human forward simulatability experiments). The comparison is illustrated in Fig. 1.

In this paper, we aim to build a model that can automatically discover comprehensible concepts with no need for concept supervision. Inspired by the recent application of the slot attention mechanism (Locatello et al., 2020) in concept-based interpretability for vision tasks (Wang et al., 2023; Hong et al., 2024), we develop a concept extractor that leverages slot attention, enabling each slot to learn a distinct task-specific concept. To ensure that the learned concepts are human-understandable, we leverage LLMs as human proxies to evaluate concept comprehensibility during model training and use the results to refine the concept extractor. Specifically, we propose a metric for concept comprehensibility that measures the capability of a concept’s activation map to be summarized and reconstructed using natural language. Building on prior work that employs large language models to generate neuron explanations and simulate activations (Bills et al., 2023; Templeton et al., 2024), our approach introduces a novel feedback loop. This loop utilizes comprehensibility scores to refine the concept extractor, making the extracted concepts more intuitive and human-understandable. Experiments on seven tasks and human studies both show the superiority of our method compared with existing concept-based explanation methods. Our contributions are summarized as follows:
• We propose a method to evaluate the human interpretability of concepts by employing LLMs as human proxies, enabling real-time assessment of concept comprehensibility during training.
• We design a self-explaining model training mechanism without the need for concept annotations, where LLMs evaluate the discovered concepts and guide the subsequent model to learn concepts with greater interpretability.
• Our method demonstrates performance comparable to black-box models across various tasks, while the learned concepts are interpretable and comprehensible to humans.

2 Related Work

Concept-based Post-hoc Methods explain an existing model without modifying its internal architecture. Some studies redefine task-related concepts and assess them by quantifying contributions with linear probes (Alvarez Melis and Jaakkola, 2018; Crabbé and van der Schaar, 2022) or analyzing causal effects through proxy methods (Goyal et al., 2019; Wu et al., 2023). Methods without concept supervision extract concepts from intermediate layer representations using K-Means (Ghorbani et al., 2019), Non-Negative Matrix Factorization (Zhang et al., 2021; Fel et al., 2023; Jourdan et al., 2023) or concept completeness maximization (Yeh et al., 2020).

However, these post-hoc methods cannot guarantee that models truly comprehend or employ the adopted concepts, as the models were not exposed to these concepts during training (Rudin, 2019; Poteta et al., 2023). Additionally, some unsupervised post-hoc methods may extract concepts lacking semantic meaning.

Concept-based Self-explaining Methods aim to provide a built-in human-interpretable explanation along with the prediction. Some methods adopt a supervised paradigm with experts manually crafting a set of concepts. A representative framework is Concept Bottleneck Model (CBM) (Koh et al., 2020), where an intermediate concept bottleneck layer is introduced to break the standard end-to-end training paradigm. Recently, some studies have taken advantage of the capabilities of LLMs to generate concepts for each class as a replacement for manual annotation (Yang et al., 2023; Oikarinen et al., 2023). This insight has also been applied in the text domain (Tan et al., 2024b; De Santis et al., 2024). These methods rely on concepts that are predefined based on priors and cannot discover or adapt to new concepts during training. An incomplete or biased predefined concept set could severely compromise both the model’s interpretability and performance. Text Bottleneck Model (TBM) (Ludan et al., 2023) utilized LLMs to automatically discover and measure concepts. However, in TBM, the mapping from samples to concepts is directly determined by LLMs, which remain largely opaque and function as a black box.

Unsupervised self-explaining concept models autonomously extract concepts during model training without the need for concept annotations. SENN (Alvarez Melis and Jaakkola, 2018) uses self-supervision by reconstruction loss for concept discovery. BotCL (Wang et al., 2023) and CCTs (Hong et al., 2024) utilize a slot attention-based mechanism to extract task-dependent concept slots. While in the text domain, SelfEx-
plain (Rajagopal et al., 2021) identifies concepts as the non-terminal leaves of the semantic tree parsing the text. Some methods classify samples based on their distance to learnable prototypes (Li et al., 2018; Das et al., 2022; Xie et al., 2023) and introduce distance-based losses to guide prototype learning. Without concept annotations, the concepts extracted by these methods pose challenges to human understanding. Although existing methods evaluate human interpretability through manual assessments during the experimental phase, no approach has yet incorporated human feedback into the training to optimize concept comprehensibility.

3 Method

3.1 Framework

Given a document consists of \( L \) tokens (words), we adopt a text encoder to encode tokens of the document as \( \mathbf{X} = \{ x_1, x_2, ..., x_L \} \in \mathbb{R}^{L \times D} \), where \( D \) is the dimension of the encoded space. ECO-Concept learns a set of \( M \) concept prototypes \( C = \{ c_1, c_2, ..., c_M \} \in \mathbb{R}^{M \times D} \) while learning the original classification task. Each concept prototype \( c_m \) is a trainable \( D \)-dimensional variable and is initialized randomly. These prototypes are continuously optimized during the training process to represent task-relevant concepts. Fig. 2 shows an overview of our framework. ECO-Concept consists of three modules: a concept extractor, a classifier, and a concept evaluator. The concept extractor takes the encoded text as input and generates a concept slot attention matrix by interacting with the concept prototypes. The slot attention scores across tokens are then summed and fed into the classifier for prediction. To enhance the comprehensibility of the concepts, we designed a concept evaluator that receives slot attention from the concept extractor, maps them back to the original text, and uses human proxies (here, LLMs)\footnote{We conducted a human assessment of using LLMs to evaluate concept comprehensibility. The details are in Appendix B.} to summarize the concepts, and highlights concept-related segments with additional exemplars. If a concept is easy to understand, the highlighted segments should be similar to the model’s slot attention. We use the difference between the highlighted segments and the model’s slot attention scores, considering the importance of each concept, as a comprehensibility loss. This feedback is used to make the learned concepts easier for human understanding.

3.2 Concept Extractor

Concept Extractor uses slot attention to discover concepts automatically. Slot Attention (Locatello et al., 2020) was initially applied in object recognition, with its core mechanism focusing on utilizing slots to compete for explaining parts of the input features. Through multiple rounds of attention competition, these slots are gradually adjusted to represent distinct object features, i.e., the underlying concepts. In computer vision, several methods already utilize the slot attention mechanism to achieve concept-based interpretability (Wang et al., 2023; Hong et al., 2024). Inspired by these approaches, we implemented a concept extractor based on slot attention, where each slot learns a distinct task-dependent concept.

Our concept extractor takes the encoded features \( \mathbf{X} \) as input and interacts with the concept prototypes \( C \) via the slot attention module, obtaining the concept features \( U \in \mathbb{R}^{M \times D} \) and the slot attention matrix \( A \in [0, 1]^{M \times L} \). For this, we apply linear projection \( W_q \) on the concept slots to obtain the queries and projections \( W_k \) and \( W_v \) on the inputs to obtain the keys and the values, all having the same dimension \( D \). Then, we perform a dot product between the queries and keys to get the attention matrix \( A \).

\[
A = \phi(\frac{(W_q C)(W_k \mathbf{X})^T}{\sqrt{D}}).
\]

In the slot attention matrix \( A \), each element \( A_{m,l} \) represents the attention weight of the concept slot \( m \) when attending to the input vector \( l \). Unlike traditional attention mechanisms, we normalize \( A \) by applying a sparse softmax function \( \phi \) across the concept slots (along the \( M \) axis). This normalization introduces competition among slots to attend to each input token. The sparsity normalization ensures that each input token is primarily associated with a limited number of concepts, facilitating a more focused and interpretable learning of conceptual representations. We then aggregate features in \( \mathbf{X} \) corresponding to each concept into concept features \( U \in \mathbb{R}^{M \times D} \).

\[
U = \frac{A}{\sum_l A_{:,l}}(W_v \mathbf{X}).
\]

For better interpretability, we also employ concept regularizers to constrain the training of concept prototypes \( C \).

**Consistency** The extracted concepts should represent consistent semantics. That is, the concept
Figure 2: (a) Illustration of the proposed framework ECO-Concept. ECO-Concept consists of a concept extractor, a classifier, and a concept evaluator. (b) The concept extractor takes the encoded text \( X \) as input and interacts with the concept prototypes \( C \) to obtain a slot attention matrix \( A \) and concept features \( U \). The concept prototypes are optimized using consistency and distinctiveness loss. (c) The concept evaluator utilizes exemplars with the highest concept attention values to construct two sets, \( D_{sum} \) and \( D_{high} \). Using the corresponding slot attention matrices \( A_{sum} \) and \( A_{high} \), the evaluator highlights these exemplars to perform concept summarization and highlighting, thus getting the comprehensibility loss to guide model fine-tuning.

features activated by each concept prototype across the samples should not have large variations. The concept features \( u_m \) and \( u'_m \) under the same concept \( m \) of different documents should be similar to each other. Under each concept, we select the top \( k \) samples with the highest activation values in a mini-batch. We define the consistency loss as:
\[
\mathcal{L}_{con} = \frac{1}{M} \sum_m \sum_{u_m,u'_m} \frac{\|u_m - u'_m\|_2^2}{k(k-1)}.
\]
(3)

Distinctiveness To capture different aspects of documents, different concepts should cover different elements. This means that the average features of concept \( m \) calculated based on the top \( k \) samples with the highest activation values in a mini-batch, given by \( \bar{u}_m = (1/k) \sum u_m \), should be different from any other. We encode this into a loss as:
\[
\mathcal{L}_{dist} = - \sum_{\bar{u}_m,\bar{u}_{m'}} \frac{\|\bar{u}_m - \bar{u}_{m'}\|_2^2}{k(k-1)}.
\]
(4)

3.3 Classifier

We use a fully connected layer for classification, and total concept activation \( t \in \mathbb{R}^M \) is the only input as the concept bottleneck (Koh et al., 2020):
\[
t = \sum_{l=1}^L A_{:,l}.
\]
(5)

Formally, letting \( W \in \mathbb{R}^{M \times \Omega} \) be a learnable matrix, prediction \( \hat{y} \) is given by:
\[
\hat{y} = W t.
\]
(6)

We use softmax cross-entropy for the classification task, denoted by \( \mathcal{L}_{ce} \). The overall loss is
\[
\mathcal{L} = \mathcal{L}_{ce} + \lambda_{con} \mathcal{L}_{con} + \lambda_{dist} \mathcal{L}_{dist},
\]
(7)
where \( \lambda_{con} \) and \( \lambda_{dist} \) are weight terms.

3.4 Concept Comprehensibility Enhancement

By optimizing the above loss function, our model achieved effective classification performance while extracting task-relevant concepts with semantic meanings. However, due to the lack of supervision from concept annotations, the identified concepts may still exhibit some degree of semantic ambiguity. To address this, we aim to further enhance the comprehensibility of the extracted concepts, prioritizing those of higher importance for improved human interpretability. Specifically, we evaluate each concept’s importance and comprehensibility, combining these metrics accordingly to define a comprehensibility loss. This loss is integrated into the original loss function, and the model is further fine-tuned to improve concept comprehensibility.

**Concept Importance.** The weight matrix of the classifier \( W \) learns the correlation between the
class and concepts. For each concept \( m \), \( W_{m,\omega} \) is the correlation from concept \( m \) to class \( \omega \). A positive value of \( W_{m,\omega} \) means that concept \( m \) co-occurs with class \( \omega \) in the dataset, so its presence positively supports class \( \omega \). Meanwhile, a negative value means the concept \( m \) and the class \( \omega \) rarely co-occurs. When concept \( m \) is present, the class is unlikely to be \( \omega \). Regardless of whether the value is positive or negative, \( W_{m,\omega} \) represents the relationship between concept \( m \) and class \( \omega \). Therefore, we take the absolute value of \( W_{m,\omega} \) here and compute the sum of the absolute values of all the weights connecting concept \( m \) as \( \sum_{\omega=1}^{\Omega} |W_{m,\omega}| \).

The importance of concept \( m \) should also take into account the concept activation. Given the average concept activation \( t_m \) in a mini-batch \( \mathcal{B} \), the concept importance score \( \beta_m \) is:

\[
\beta_m = \frac{1}{|\mathcal{B}|} \sum_{t \in \mathcal{B}} t_m \sum_{\omega=1}^{\Omega} |W_{m,\omega}|.
\] (8)

**Concept Comprehensibility.** To measure the comprehensibility of concepts, we refer to the automated interpretability evaluation methods used for explaining neurons in LLMs (Bills et al., 2023; Templeton et al., 2024). In summary, the auto-interpretability procedure takes samples of text where the neurons activate, asks a language model to write a human-readable interpretation of the neuron features, and then prompts the language model to use this description to predict the neurons’ activation on other samples of text. The correlation between the model’s predicted and the actual activations is the feature’s interpretability score.

We adopt a similar idea, measuring the comprehensibility of a concept by evaluating its ability to be summarized and reconstructed using natural language. For each concept, we select the exemplars with the highest attention values to form two sets, \( \mathcal{D}_{sum} \) and \( \mathcal{D}_{high} \), which are used for concept summarization and concept-related segment highlightings, respectively. We first present the exemplars \( \mathcal{D}_{sum} \) with their corresponding slot attention values and ask the LLM to generate the interpretation. If it assesses that this concept has semantic meanings, we then prompt another LLM to highlight concept-related tokens on new exemplars from \( \mathcal{D}_{high} \) in a 0-1 scale. For each highlighted exemplar, its slot attention matrix is \( \boldsymbol{A} \in [0, 1]^{M \times L} \) and the highlighted matrix obtained by LLM is \( \boldsymbol{S} \in [0, 1]^{M \times L} \). For concepts that are considered semantically meaningless by LLM, our goal is to minimize their corresponding activations. Specifically, we achieve this by setting the elements of \( \boldsymbol{S} \) to zero for highlighted exemplars corresponding to these concepts.

The comprehensibility score of each concept is obtained by averaging the MSE loss between the slot attention matrix \( \boldsymbol{A} \) and the highlighted matrix \( \boldsymbol{S} \) for the highlighted samples within a mini-batch \( \mathcal{B} \). To ensure that more important concepts are easier to understand, the comprehensibility loss is computed as a weighted combination of concept importance and comprehensibility:

\[
\mathcal{L}_{com} = \frac{1}{M} \sum_{m=1}^{M} \frac{\beta_m \sum_{A \in \mathcal{B}_m \cap \mathcal{D}_{high}} \| \boldsymbol{A}_{m,:} - \boldsymbol{S}_{m,:} \|_2^2}{|\mathcal{B}_m \cap \mathcal{D}_{high}|}.
\] (9)

**Training Strategies.** Based on the model trained in the first phase, we continue to train the concept prototypes \( \boldsymbol{C} \) and the classifier using the following loss function:

\[
\mathcal{L} = \mathcal{L}_{ce} + \lambda_{con} \mathcal{L}_{con} + \lambda_{dist} \mathcal{L}_{dist} + \lambda_{com} \mathcal{L}_{com}.
\] (10)

At the end of each iteration, we conduct a concept re-summarization. If the meaning of a concept remains unchanged, the corresponding concept prototype parameters are frozen in the following iterations. Otherwise, we highlight concept-related segments based on the updated summarization. This process is repeated until the meanings of all concepts stabilize.

4 Experiments

In this section, we conduct extensive experiments to verify the task performance and interpretability of ECO-Concept. Due to space limitations, parameter sensitivity analysis and other experiments are included in Appendices E and F.

4.1 Experimental Settings

**Datasets** We conduct experiments on seven public datasets. To compare our results with supervised methods, we utilize three datasets with concept annotations: CEBaB (Abraham et al., 2022), Hotel (Wang et al., 2010), and Beer (McAuley et al., 2012). The rest four datasets, IMDB (Maas et al., 2011), AGnews (Gulli, 2004), Twitter (Sheng et al., 2021), and SciCite (Cohan et al., 2019), do not include concept annotations. More details about these datasets are in Appendix A.1.

**Baselines** The selected baselines include black-box, supervised concept-based, and unsupervised concept-based methods. The first group, black-box
<table>
  <tr>
    <th rowspan="2">Category</th>
    <th rowspan="2">Method</th>
    <th colspan="2">CEBaB</th>
    <th colspan="2">Beer</th>
    <th colspan="2">Hotel</th>
    <th colspan="2">IMDB</th>
    <th colspan="2">AGnews</th>
    <th colspan="2">Twitter</th>
    <th colspan="2">SciCite</th>
  </tr>
  <tr>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
  </tr>
  <tr>
    <td rowspan="2">Black-Box</td>
    <td>RoBERTa</td>
    <td>.682</td><td>.797</td>
    <td>.882</td><td>.882</td>
    <td style="background-color:#c6efce;">.981</td><td style="background-color:#c6efce;">.981</td>
    <td style="background-color:#c6efce;">.937</td><td style="background-color:#c6efce;">.937</td>
    <td style="background-color:#c6efce;">.941</td><td style="background-color:#c6efce;">.941</td>
    <td>.960</td><td style="background-color:#c6efce;">.828</td>
    <td>.812</td><td>.858</td>
    <td>.879</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>.640</td><td>.770</td>
    <td>.878</td><td>.878</td>
    <td>.975</td><td>.975</td>
    <td>.912</td><td>.912</td>
    <td>.934</td><td>.956</td>
    <td>.775</td><td>.745</td>
    <td>.845</td><td>.861</td>
  </tr>
  <tr>
    <td rowspan="2">SUP.</td>
    <td>CBM</td>
    <td>.669</td><td>.802</td>
    <td>.883</td><td style="background-color:#c6efce;">.885</td>
    <td>.979</td><td>.979</td>
    <td>-</td><td>-</td>
    <td>-</td><td>-</td>
    <td>-</td><td>-</td>
    <td>-</td><td>-</td>
  </tr>
  <tr>
    <td>SparseCBM</td>
    <td>.644</td><td>.767</td>
    <td>.883</td><td>.883</td>
    <td>.981</td><td>.979</td>
    <td>-</td><td>-</td>
    <td>-</td><td>-</td>
    <td>-</td><td>-</td>
    <td>-</td><td>-</td>
  </tr>
  <tr>
    <td rowspan="3">UNSUP.</td>
    <td>SelfExplain</td>
    <td>.683</td><td>.799</td>
    <td>.873</td><td>.872</td>
    <td>.978</td><td>.979</td>
    <td>.936</td><td>.936</td>
    <td>.925</td><td>.949</td>
    <td>.817</td><td>.806</td>
    <td>.856</td><td>.873</td>
  </tr>
  <tr>
    <td>PROTOTEX</td>
    <td>.610</td><td>.728</td>
    <td>.877</td><td>.877</td>
    <td>.977</td><td>.977</td>
    <td>.935</td><td>.935</td>
    <td>.915</td><td>.943</td>
    <td>.826</td><td>.812</td>
    <td>.852</td><td>.871</td>
  </tr>
  <tr>
    <td>ECO-Concept</td>
    <td style="background-color:#c6efce;">.697</td><td style="background-color:#c6efce;">.808</td>
    <td style="background-color:#c6efce;">.885</td><td style="background-color:#c6efce;">.885</td>
    <td style="background-color:#c6efce;">.981</td><td style="background-color:#c6efce;">.981</td>
    <td style="background-color:#c6efce;">.937</td><td style="background-color:#c6efce;">.937</td>
    <td style="background-color:#c6efce;">.941</td><td style="background-color:#c6efce;">.941</td>
    <td style="background-color:#c6efce;">.961</td><td style="background-color:#c6efce;">.828</td>
    <td style="background-color:#c6efce;">.813</td><td style="background-color:#c6efce;">.860</td>
    <td style="background-color:#c6efce;">.881</td>
  </tr>
</table>

Table 1: Classification performance of ECO-Concept and other baselines. The best result is highlighted.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">CEBaB</th>
    <th colspan="2">Beer</th>
    <th colspan="2">Hotel</th>
    <th colspan="2">IMDB</th>
    <th colspan="2">AGnews</th>
    <th colspan="2">Twitter</th>
    <th colspan="2">SciCite</th>
  </tr>
  <tr>
    <th>Sem.</th><th>Dist.</th>
    <th>Sem.</th><th>Dist.</th>
    <th>Sem.</th><th>Dist.</th>
    <th>Sem.</th><th>Dist.</th>
    <th>Sem.</th><th>Dist.</th>
    <th>Sem.</th><th>Dist.</th>
    <th>Sem.</th><th>Dist.</th>
  </tr>
  <tr>
    <td>Cockatiel</td>
    <td>.50</td><td>.40</td>
    <td>.47</td><td>.55</td>
    <td>.60</td><td>.42</td>
    <td>.60</td><td>.65</td>
    <td>.49</td><td>.35</td>
    <td>.40</td><td>.41</td>
    <td>.65</td><td>.60</td>
    <td>.41</td><td>.50</td>
    <td>.55</td><td>.53</td>
    <td>.35</td><td>.50</td>
  </tr>
  <tr>
    <td>Concept-Shap</td>
    <td>.25</td><td>.30</td>
    <td>.42</td><td>.35</td>
    <td>.15</td><td>.34</td>
    <td>.65</td><td>.40</td>
    <td>.35</td><td>.32</td>
    <td>.35</td><td>.35</td>
    <td>.35</td><td>.35</td>
    <td>.80</td><td>.30</td>
    <td>.53</td><td>.40</td>
    <td>.30</td><td>.44</td>
  </tr>
  <tr>
    <td>ProtoTEX</td>
    <td>.45</td><td>.45</td>
    <td>.35</td><td>.20</td>
    <td>.25</td><td>.38</td>
    <td>.40</td><td>.30</td>
    <td>.33</td><td>.25</td>
    <td>.25</td><td>.36</td>
    <td>.40</td><td>.45</td>
    <td>.41</td><td>.45</td>
    <td>.45</td><td>.45</td>
    <td>.45</td><td>.41</td>
  </tr>
  <tr>
    <td>ECO-Concept</td>
    <td style="background-color:#c6efce;">.60</td><td style="background-color:#c6efce;">.60</td>
    <td style="background-color:#c6efce;">.51</td><td style="background-color:#c6efce;">.85</td>
    <td style="background-color:#c6efce;">.75</td><td style="background-color:#c6efce;">.75</td>
    <td style="background-color:#c6efce;">.70</td><td style="background-color:#c6efce;">.70</td>
    <td style="background-color:#c6efce;">.65</td><td style="background-color:#c6efce;">.65</td>
    <td style="background-color:#c6efce;">.52</td><td style="background-color:#c6efce;">.70</td>
    <td style="background-color:#c6efce;">.65</td><td style="background-color:#c6efce;">.54</td>
    <td style="background-color:#c6efce;">.60</td><td style="background-color:#c6efce;">.60</td>
    <td style="background-color:#c6efce;">.55</td><td style="background-color:#c6efce;">.60</td>
    <td style="background-color:#c6efce;">.50</td><td style="background-color:#c6efce;">.52</td>
  </tr>
</table>

Table 2: Concept comprehensibility evaluation of different concept-based methods. The best result is highlighted.

methods, directly tackles text classification tasks without interpretability, including a BERT-based classifier (Devlin et al., 2019) and a RoBERTa-based classifier (Liu et al., 2019). The second group, supervised concept-based methods, leverages concept annotations to predict both the presence of concepts and the target class, including CBM (Kim et al., 2018) and an evolved CBM variant SparseCBM (Tan et al., 2024a). The third group, unsupervised concept-based methods, comprise two self-explaining methods Self-Explain (Rajagopal et al., 2021) and PROTOTEX (Das et al., 2022), and two post-hoc methods COCKATIEL (Jourdan et al., 2023) and Concept-Shap (Yeh et al., 2020). The details of these methods are included in Appendix A.2.

Implementation Details For all concept-based methods, we use RoBERTa as the text encoder and set the number of concepts to 20 across all tasks. For ECO-Concept, we provide the top 10 exemplars for each concept and utilize GPT-4o to summarize the concept interpretation. Additionally, we prompt GPT-4o-mini to highlight text segments on 100 exemplars per concept. The trade-off parameters \( \lambda_{con}, \lambda_{dist}, \) and \( \lambda_{com} \) are 0.1, -0.01, and 1, respectively. In our experiments, the concept comprehensibility enhancement stage ends within three iterations.

4.2 Task Performance

Table 1 shows the performance comparison of ECO-Concept and baselines. In general, our method achieves superior classification performance across various datasets. Compared to supervised methods, ECO-Concept achieves competitive results with no concept supervision, indicating its ability to automatically discover new concepts without compromising performance. Compared to unsupervised baselines, ECO-Concept shows significant improvements in both accuracy and F1 with pairwise t-tests at a 95% confidence level, validating the effectiveness of its conceptual representations. Moreover, ECO-Concept also has comparable or better performance compared with black-box models. This indicates that it effectively balances both task-discriminativity and concept comprehensibility, showing the potential to build interpretable models without performance trade-offs.

4.3 Concepts Evaluation

To evaluate the comprehensibility of the concepts extracted by our method, we first define three quantitative metrics. Then we conduct several human evaluations to further assess how easily these concepts can be understood. We compare our method’s extracted concepts with those from three other global unsupervised concept extraction methods. Among these methods, Cockatiel and Concept-Shap are post-hoc methods, and ProtoTEX is a self-explaining method. They extract concepts as representative training samples or text segments. For a fair comparison, we summarize the concept interpretation of these methods using the top 5 representative samples per concept, applying the same summary prompt as ECO-Concept. Note that we exclude SelfExplain in global concept comparisons, as it, differently, treats each concept as a single text segment, which cannot be further summarized.
Quantitative Metrics. We assess the comprehensibility of concepts using the following metrics:
Semantics. (Sem.) This metric evaluates the proportion of extracted concepts that have clear semantic meaning. It is calculated as the ratio of concepts with identifiable semantics to the total number of extracted concepts.
Distinctiveness. (Dist.) This metric evaluates whether the extracted concepts are diverse and not redundant, which is the proportion of unique concepts identified by an LLM in all extracted ones.
Consistency. (Con.) This metric evaluates the internal consistency of the activation content within each concept. Specifically, we use the topic coherence score \( C_V \) (Röder et al., 2015), which measures whether the top 10 activated words within each concept collectively form a coherent topic.

As shown in Table 2, unlike post-hoc methods, our method is designed to balance task performance and interpretability simultaneously. However, we still achieve competitive performance in concept comprehensibility across various tasks. In the Hotel task, Cockatiel achieves a higher concept consistency. This is likely because Cockatiel extracts concepts from shorter sentences. However, in other tasks, our method demonstrates comparable or even superior consistency by directly extracting concepts from the original text. For the Twitter task, Concept-Shap achieves a higher proportion of semantically meaningful concepts. Nevertheless, the extracted concepts exhibit relatively low distinctiveness, suggesting that many of the concepts are redundant. In contrast, our method not only extracts semantically meaningful concepts but also ensures they are distinct from each other.

Human Evaluation. To further assess whether derived concepts are explainable to humans, we designed three human evaluation tasks, including intruder detection, subjective ratings, and forward simulatability. More details about the survey design and interface are in Appendix C.

To evaluate the comprehensibility of the extracted concepts, following Ghorbani et al. (2019); Fel et al. (2023), we designed an intruder detection experiment. In each question, participants were presented with four cases and tasked with identifying the one that was conceptually different from the others. A higher accuracy in identifying the intruder indicates that the concepts are more intuitive and easier for humans to understand. Table 3 summarizes the results. Our method achieves the highest intruder detection accuracy across all tasks, demonstrating that the concepts extracted by our approach are more comprehensible to humans.

In addition, we also conducted subjective rating experiments, where participants rated all the extracted concepts from multiple perspectives: Consistency, Clarity, Task Relevance, Comprehensibility. As shown in Fig. 3, our method consistently achieved the highest ratings across all perspectives for each task. This result indicates that our concepts are subjectively the most comprehensible to humans and are closely aligned with the tasks. In contrast, ProtoTEx received the lowest ratings in most tasks, suggesting that traditional unsupervised self-explaining methods often struggle to balance interpretability with task performance, resulting in less intuitive concepts for human understanding.

![Human subjective ratings on concept quality](page_184_120_1076_482.png)

Figure 3: Human subjective ratings on concept quality
<table>
  <tr>
    <th>Method</th>
    <th>CEBaB Beer</th>
    <th>Hotel</th>
    <th>IMDB</th>
    <th>AGnews</th>
    <th>Twitter</th>
    <th>SciCite</th>
  </tr>
  <tr>
    <td>Cockatiel</td>
    <td>.667</td>
    <td>.633</td>
    <td>.667</td>
    <td>.833</td>
    <td>.767</td>
    <td>.850</td>
    <td>.683</td>
  </tr>
  <tr>
    <td>Concept-Shap</td>
    <td>.550</td>
    <td>.483</td>
    <td>.483</td>
    <td>.683</td>
    <td>.850</td>
    <td>.433</td>
    <td></td>
  </tr>
  <tr>
    <td>ProtoTEX</td>
    <td>.500</td>
    <td>.500</td>
    <td>.483</td>
    <td>.683</td>
    <td>.750</td>
    <td>.733</td>
    <td>.350</td>
  </tr>
  <tr>
    <td>ECO-Concept</td>
    <td><b>.767</b></td>
    <td><b>.750</b></td>
    <td><b>.767</b></td>
    <td><b>.900</b></td>
    <td><b>.900</b></td>
    <td><b>.867</b></td>
    <td><b>.700</b></td>
  </tr>
</table>

Table 3: Accuracy of concept intruder detection

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">Beer</th>
    <th colspan="2">AGnews</th>
  </tr>
  <tr>
    <th>Accuracy</th>
    <th>Confidence</th>
    <th>Accuracy</th>
    <th>Confidence</th>
  </tr>
  <tr>
    <td>NE</td>
    <td>.808</td>
    <td>4.52</td>
    <td>.708</td>
    <td>4.44</td>
  </tr>
  <tr>
    <td>Cockatiel</td>
    <td>.967</td>
    <td>4.13</td>
    <td>.833</td>
    <td>4.50</td>
  </tr>
  <tr>
    <td>Concept-Shap</td>
    <td>.833</td>
    <td>4.22</td>
    <td>.783</td>
    <td>3.80</td>
  </tr>
  <tr>
    <td>PROTOTEX</td>
    <td>.950</td>
    <td>4.28</td>
    <td>.783</td>
    <td>4.23</td>
  </tr>
  <tr>
    <td>ECO-Concept</td>
    <td><b>.983</b></td>
    <td><b>4.45</b></td>
    <td><b>.867</b></td>
    <td><b>4.55</b></td>
  </tr>
</table>

Table 4: Results of human forward simulatability

Finally, we validated the extracted concepts through a forward simulatability experiment to evaluate their effectiveness in helping users understand the model’s behavior. This aligns with the definition of explainability proposed by Kim et al. (2016), which emphasizes a user’s ability to “correctly and efficiently predict the method’s results.” In our experiment, participants received explanations of model outputs, then attempted to infer the model’s outputs based on the explanations and rated their confidence. For comparison, we also measured the accuracy of human judgments without explanations (NE). We conducted forward simulatability experiments with explanations on the two most challenging tasks for human judgment (Beer and AGnews), as determined by the lowest human accuracy rates in no-explanation conditions. As shown in Table 4, our method significantly improved the simulatability accuracy for participants in inferring model outputs. This demonstrates that our explanations effectively enhance human understanding of the model’s behavior and increase their trust in its predictions. Moreover, for other concept-based methods, although the simulatability accuracy is higher when their explanations are provided compared to when no explanations are given, humans generally tend to have lower confidence in these explanations. This further validates that existing methods often produce confusing explanations, which in turn reduces human trust. Besides, we also asked participants to rate the explanations for each example across three aspects. Detailed information and results are in Appendix C.4.

4.4 Ablation Study

To further explore the impact of different model modules, we conducted several ablation studies.

<table>
  <tr>
    <th>Metric</th>
    <th>Method</th>
    <th>CEBaB Beer</th>
    <th>Hotel</th>
    <th>IMDB</th>
    <th>AGnews</th>
    <th>Twitter</th>
    <th>SciCite</th>
  </tr>
  <tr>
    <td rowspan="3">Acc</td>
    <td>w/o \( \mathcal{L}_{con} \)</td>
    <td>.679</td>
    <td><b>.885</b></td>
    <td>.980</td>
    <td>.937</td>
    <td>.939</td>
    <td>.823</td>
    <td>.858</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{dist} \)</td>
    <td>.699</td>
    <td><b>.885</b></td>
    <td>.977</td>
    <td>.937</td>
    <td>.940</td>
    <td>.824</td>
    <td>.852</td>
  </tr>
  <tr>
    <td>w/ all</td>
    <td><b>.704</b></td>
    <td><b>.885</b></td>
    <td><b>.981</b></td>
    <td><b>.938</b></td>
    <td><b>.945</b></td>
    <td><b>.832</b></td>
    <td><b>.860</b></td>
  </tr>
  <tr>
    <td rowspan="3">F1</td>
    <td>w/o \( \mathcal{L}_{con} \)</td>
    <td>.796</td>
    <td><b>.885</b></td>
    <td>.980</td>
    <td>.937</td>
    <td>.959</td>
    <td>.811</td>
    <td>.879</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{dist} \)</td>
    <td>.806</td>
    <td><b>.885</b></td>
    <td>.977</td>
    <td>.937</td>
    <td>.960</td>
    <td>.809</td>
    <td>.875</td>
  </tr>
  <tr>
    <td>w/ all</td>
    <td><b>.811</b></td>
    <td><b>.885</b></td>
    <td><b>.981</b></td>
    <td><b>.938</b></td>
    <td><b>.964</b></td>
    <td><b>.816</b></td>
    <td><b>.882</b></td>
  </tr>
  <tr>
    <td rowspan="3">Sem.</td>
    <td>w/o \( \mathcal{L}_{con} \)</td>
    <td>.45</td>
    <td>.65</td>
    <td>.70</td>
    <td>.40</td>
    <td>.50</td>
    <td>.40</td>
    <td>.25</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{dist} \)</td>
    <td>.50</td>
    <td>.70</td>
    <td>.70</td>
    <td>.50</td>
    <td>.65</td>
    <td>.50</td>
    <td>.15</td>
  </tr>
  <tr>
    <td>w/ all</td>
    <td><b>.55</b></td>
    <td><b>.70</b></td>
    <td><b>.70</b></td>
    <td><b>.60</b></td>
    <td><b>.65</b></td>
    <td><b>.60</b></td>
    <td><b>.55</b></td>
  </tr>
  <tr>
    <td rowspan="3">Dist.</td>
    <td>w/o \( \mathcal{L}_{con} \)</td>
    <td>.50</td>
    <td><b>.65</b></td>
    <td>.60</td>
    <td>.40</td>
    <td>.45</td>
    <td>.50</td>
    <td>.35</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{dist} \)</td>
    <td>.30</td>
    <td>.50</td>
    <td>.55</td>
    <td>.50</td>
    <td>.55</td>
    <td>.55</td>
    <td>.20</td>
  </tr>
  <tr>
    <td>w/ all</td>
    <td><b>.60</b></td>
    <td><b>.65</b></td>
    <td><b>.65</b></td>
    <td><b>.60</b></td>
    <td><b>.65</b></td>
    <td><b>.60</b></td>
    <td><b>.50</b></td>
  </tr>
  <tr>
    <td rowspan="3">Con.</td>
    <td>w/o \( \mathcal{L}_{con} \)</td>
    <td>.43</td>
    <td>.40</td>
    <td>.41</td>
    <td>.38</td>
    <td>.47</td>
    <td>.45</td>
    <td>.42</td>
  </tr>
  <tr>
    <td>w/o \( \mathcal{L}_{dist} \)</td>
    <td><b>.48</b></td>
    <td><b>.41</b></td>
    <td><b>.41</b></td>
    <td><b>.40</b></td>
    <td><b>.51</b></td>
    <td><b>.45</b></td>
    <td><b>.42</b></td>
  </tr>
  <tr>
    <td>w/ all</td>
    <td><b>.48</b></td>
    <td><b>.46</b></td>
    <td><b>.43</b></td>
    <td><b>.49</b></td>
    <td><b>.52</b></td>
    <td><b>.51</b></td>
    <td><b>.47</b></td>
  </tr>
</table>

Table 5: Task performance and concept metrics comparison between our method and its ablative variants

Impact of Concept Regularizers. To analyze the effects of the concept consistency and concept distinctiveness regularizers, we conduct an ablation study. We experiment on two types of variant models: w/o \( \mathcal{L}_{con} \) and w/o \( \mathcal{L}_{dist} \), which respectively remove the concept consistency regularizer and the concept distinctiveness regularizer. Experimental results are shown in Table 5. The results show that the best task performance and concept interpretability are achieved when both regularizers are applied together. By comparing the w/ all and w/o \( \mathcal{L}_{con} \) variants, removing the consistency loss significantly reduces the consistency score, demonstrating the importance of the consistency regularizer. A similar trend is observed with the distinctiveness regularizer. These results highlight the importance of incorporating both types of regularization. Further analysis is provided in Appendix D.1.

Impact of Concept Comprehensibility Enhancement. To demonstrate the importance of the concept comprehensibility enhancement stage, we compare the model before enhancement (denoted as the Base model), ECO-Concept, and ECO-Concept (w/o \( \mathcal{L}_{com} \)). We introduce the ECO-Concept (w/o \( \mathcal{L}_{com} \)) variant to exclude the effect of longer training with concept regularizers, ensuring that any improvements in comprehensibility are due to \( \mathcal{L}_{com} \). From the result in Fig. 4, we observe that after concept enhancement, classification performance remains largely unchanged (or experiences only a slight decline). Regarding concept comprehensibility, our ECO-Concept yields the best performance. After the enhancement, the semantics and consistency of concepts are further improved across most tasks. Besides, we also discovered more diverse concepts in two tasks. Fur-
Figure 4: Classification performance and concept metrics comparison of the model before concept enhancement (the Base model), ECO-Concept, and ECO-Concept (w/o \( L_{com} \)).

ther analysis is provided in Appendix D.2.

5 Conclusion

To automatically extract human-understandable concept explanations with no predefined concept annotations, we proposed ECO-Concept, an intrinsically interpretable framework. ECO-Concept employs an object-centric architecture based on the slot attention mechanism to extract task-specific semantic concepts. To ensure the extracted concepts are comprehensible to humans, ECO-Concept incorporates LLMs as human proxies to evaluate concept comprehensibility during model training, using their feedback to iteratively refine the concept extractor. ECO-Concept achieves classification performance on par with black-box models while offering enhanced interpretability. It outperforms existing concept-based approaches in both quantitative metrics and user studies, demonstrating its effectiveness in generating human-aligned, interpretable explanations.

Limitations

While our approach presents a significant step towards more interpretable models, several limitations warrant further exploration. First, in this work, we focus on enhancing pre-trained language models with self-explainable structures to improve interpretability in text classification tasks. Given the inherent challenges of unsupervised concept extraction, we employ a comparatively lightweight BERT-based model as the backbone. For future work, we plan to explore concept extraction methods compatible with larger models and extend our framework to encompass state-of-the-art architectures such as LLaMA and Mixtral. Second, to ensure the assessment of concept comprehensibility is close to the human level, we utilized two well-recognized API-based LLMs based on their strong instruction-following capabilities. However, due to cost limitations, we were only able to simulate concepts on a subset of samples. The current setting represents the best trade-off between performance and cost within our acceptable range. In the future, we plan to test more cost-effective and deployable open-source LLMs. Lastly, in our method, the number of concepts is fixed and cannot be adaptively adjusted during training. Currently, there are no well-suited approaches to address this challenge. We are actively exploring potential techniques (such as incremental learning) to adjust the number of concepts in a more flexible manner.

Acknowledgment

The authors would like to thank the anonymous reviewers for their insightful comments. This work is supported by the Innovation Funding of ICT, CAS under Grant (No. E561160), the Strategic Priority Research Program of the Chinese Academy of Sciences (No. XDB0680202), the National Natural Science Foundation of China (62406310), the Postdoctoral Fellowship Program of CPSF (GZC20232738), and the China Postdoctoral Science Foundation (2024M763336).
References

Eldar D Abraham, Karel D’Oosterlinck, Amir Feder, Yair Gat, Atticus Geiger, Christopher Potts, Roi Reichart, and Zhengxuan Wu. 2022. Cebab: Estimating the causal effects of real-world concepts on nlp model behavior. In Proceedings of the International Conference on Neural Information Processing Systems, volume 35, pages 17582–17596.

Reduan Achitbat, Maximilian Dreyer, Ilona Eisenbraun, Sebastian Bosse, Thomas Wiegand, Wojciech Samek, and Sebastian Lapuschkin. 2023. From attribution maps to human-understandable explanations through concept relevance propagation. Nature Machine Intelligence, 5(9):1006–1019.

David Alvarez Melis and Tommi Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In Proceedings of the International Conference on Neural Information Processing Systems, volume 31, pages 7786–7795.

Diego Antognini and Boi Faltings. 2021. Rationalization through concepts. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 761–775.

Yujia Bao, Shiyu Chang, Mo Yu, and Regina Barzilay. 2018. Deriving machine attention from human rationales. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1903–1913.

Steven Bills, Nick Cammarata, Dan Mossing, Henk Tillman, Leo Gao, Gabriel Goh, Ilya Sutskever, Jan Leike, Jeff Wu, and William Saunders. 2023. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html.

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. 2023. Towards monosemanticity: Decomposing language models with dictionary learning. https://transformer-circuits.pub/2023/monosemantic-features/index.html.

Fateme Hashemi Chaleshtori, Atreya Ghosal, Alexander Gill, Purbid Bamroo, and Ana Marasović. 2024. On evaluating explanation utility for human-ai decision making in nlp. Preprint, arXiv:2407.03545.

Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for citation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 3586–3596.

Jonathan Crabbé and Mihaela van der Schaar. 2022. Concept activation regions: A generalized framework for concept-based explanations. In Proceedings of the International Conference on Neural Information Processing Systems, volume 35, pages 2590–2607.

Fahim Dalvi, Abdul Rafae Khan, Firoj Alam, Nadir Durrani, Jia Xu, and Hassan Sajjad. 2022. Discovering latent concepts learned in bert. Preprint, arXiv:2205.07237.

Anubrata Das, Chitrank Gupta, Venelin Kovatchev, Matthew Lease, and Junyi Jessy Li. 2022. Prototex: Explaining model decisions with prototype tensors. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 2986–2997.

Francesco De Santis, Philippe Bich, Gabriele Ciravegna, Pietro Barbiero, Danilo Giordano, and Tania Cerquitelli. 2024. Self-supervised interpretable concept-based models for text classification. Preprint, arXiv:2406.14335.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, pages 4171–4186. Association for Computational Linguistics.

Thomas Fel, Agustin Picard, Louis Bethune, Thibaut Boissin, David Vigouroux, Julien Colin, Rémi Cadène, and Thomas Serre. 2023. Craft: Concept recursive activation factorization for explainability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2711–2721.

Amirata Ghorbani, James Wexler, James Zou, and Been Kim. 2019. Towards automatic concept-based explanations. In Proceedings of the International Conference on Neural Information Processing Systems, pages 9277–9286.

Yash Goyal, Amir Feder, Uri Shalit, and Been Kim. 2019. Explaining classifiers with causal concept effect (cace). Preprint, arXiv:1907.07165.

Antonio Gulli. 2004. Ag’s corpus of news articles. http://groups.di.unipi.it/~gulli/AG_corpus_of_news_articles.html.

Jinyung Hong, Keun Hee Park, and Theodore P Pavlic. 2024. Concept-centric transformers: Enhancing model interpretability through object-centric concept learning within a shared global workspace. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4880–4891.

Fanny Jourdan, Agustin Picard, Thomas Fel, Laurent Risser, Jean-Michel Loubes, and Nicholas Asher. 2023. Cockatiel: Continuous concept ranked attribution with interpretable elements for explaining neural net classifiers on nlp tasks. In 61st Annual Meeting of
the Association for Computational Linguistics (ACL 2023), pages 5120–5136.

Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81–93.

Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. 2016. Examples are not enough, learn to criticize! criticism for interpretability. In Proceedings of the International Conference on Neural Information Processing Systems, volume 29, pages 2288–2296.

Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. 2018. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International Conference on Machine Learning, pages 2668–2677. PMLR.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. 2020. Concept bottleneck models. In International Conference on Machine Learning, pages 5338–5348. PMLR.

Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. 2018. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, pages 3530–3537.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.

Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. 2020. Object-centric learning with slot attention. In Proceedings of the International Conference on Neural Information Processing Systems, volume 33, pages 11525–11538.

Josh Magnus Ludan, Qing Lyu, Yue Yang, Liam Dugan, Mark Yatskar, and Chris Callison-Burch. 2023. Interpretable-by-design text classification with iteratively generated concept bottleneck. Preprint, arXiv:2310.19660.

Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 142–150.

Julian McAuley, Jure Leskovec, and Dan Jurafsky. 2012. Learning attitudes and attributes from multi-aspect reviews. In 2012 IEEE 12th International Conference on Data Mining, pages 1020–1025. IEEE.

George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41.

Tuomas Oikarinen, Subhro Das, Lam Nguyen, and Lily Weng. 2023. Label-free concept bottleneck models. In International Conference on Learning Representations.

Eleonora Poeta, Gabriele Ciravegna, Eliana Pastor, Tania Cerquitielli, and Elena Baralis. 2023. Concept-based explainable artificial intelligence: A survey. Preprint, arXiv:2312.12936.

Dheeraj Rajagopal, Vidhisha Balachandran, Eduard H Hovy, and Yulia Tsvetkov. 2021. Selfexplain: A self-explaining architecture for neural text classifiers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 836–850.

Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In Proceedings of the eighth ACM International Conference on Web Search and Data Mining, pages 399–408.

Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206–215.

Hassan Sajjad, Nadir Durrani, Fahim Dalvi, Firoj Alam, Abdul Khan, and Jia Xu. 2022. Analyzing encoded concepts in transformer language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics, pages 3082–3101.

Qiang Sheng, Xueyao Zhang, Juan Cao, and Lei Zhong. 2021. Integrating pattern-and fact-based fake news detection via model preference learning. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pages 1640–1650.

Zhen Tan, Tianlong Chen, Zhenyu Zhang, and Huan Liu. 2024a. Sparsity-guided holistic explanation for llms with interpretable inference-time intervention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 21619–21627.

Zhen Tan, Lu Cheng, Song Wang, Bo Yuan, Jundong Li, and Huan Liu. 2024b. Interpreting pretrained language models via concept bottlenecks. In Pacific-Asia Conference on Knowledge Discovery and Data Mining, pages 56–74. Springer.

Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Calum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, Shan Carter, Chris Olah, and Tom Henighan. 2024. Scaling monosemanticity: Extracting interpretable features from claude 3 sonnet. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html.
Bowen Wang, Liangzhi Li, Yuta Nakashima, and Hajime Nagahara. 2023. Learning bottleneck concepts in image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10962–10971.

Hongning Wang, Yue Lu, and Chengxiang Zhai. 2010. Latent aspect rating analysis on review text data: a rating regression approach. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data mining, pages 783–792.

Zhengxuan Wu, Karel D’Oosterlinck, Atticus Geiger, Amir Zur, and Christopher Potts. 2023. Causal proxy models for concept-based model explanations. In International Conference on Machine Learning, pages 37313–37334. PMLR.

Sean Xie, Soroush Vosoughi, and Saeed Hassanpour. 2023. Proto-lm: A prototypical network-based framework for built-in interpretability in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3964–3979.

Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, and Mark Yatskar. 2023. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19187–19197.

Chih-Kuan Yeh, Been Kim, Sercan Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Ravikumar. 2020. On completeness-aware concept-based explanations in deep neural networks. In Proceedings of the International Conference on Neural Information Processing Systems, volume 33, pages 20554–20565.

Ruihan Zhang, Prashan Madumal, Tim Miller, Krista A Ehinger, and Benjamin I.P. Rubinstein. 2021. Invertible concept-based explanations for cnn models with non-negative concept activation vectors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11682–11690.

A Experimental Details

A.1 Datasets

In this section, we provide detailed descriptions of the benchmark datasets used in our experiments.

CEBaB (Abraham et al., 2022) is a commonly used dataset for concept-based text analysis. It includes restaurant reviews, annotated with sentiment ratings (ranging from 1 to 5 stars) and four dining experience concepts: food quality, noise level, ambiance, and service. Following Tan et al. (2024a), we frame this as a five-class classification task.

Beer (McAuley et al., 2012) is a multi-aspect beer reviews dataset. Each review includes sentiment ratings across five aspects: appearance, aroma, palate, taste, and overall impression. Following Jourdan et al. (2023), we train the model to predict whether the overall score exceeds 3, indicating a positive review. The remaining four aspects are treated as concept labels for supervised concept-based methods.

Hotel (Wang et al., 2010) is a multi-aspect hotel reviews dataset, comprising review texts annotated with seven concept labels: value, rooms, location, cleanliness, check-in/front desk, service, and business service. Since the original labels are on a scale of 0 to 5, we utilize the binarized version proposed by Bao et al. (2018).

IMDB (Maas et al., 2011) is a movie reviews dataset, where each review is labeled with either positive or negative sentiment.

AGnews (Gulli, 2004) is a dataset of news articles categorized into one of four classes: business, science/technology, sports, or world/political.

Twitter (Sheng et al., 2021) is a dataset consisting of real and fake news collected from Twitter.

SciCite (Cohan et al., 2019) is a dataset designed for classifying citation intents in academic papers. Each citation is categorized into one of three classes: method, background, or result.

A.2 Baselines

The details of the baseline methods are shown as follows.

BERT-/RoBERTa-based classifier (Devlin et al., 2019; Liu et al., 2019): We utilize BERT/RoBERTa to encode tokens of text content and feed the extracted average embedding into an MLP to obtain the final prediction.

CBM (Kim et al., 2018) introduced an intermediate layer that first predicts the human-defined concepts and then uses the concepts to predict the final output.

SparseCBM (Tan et al., 2024a) enhances CBM by integrating unstructured pruning. SparseCBM constructs concept-specific sparse subnetworks within the backbone network, providing interpretability while retaining model performance.

SelfExplain (Rajagopal et al., 2021) is an unsupervised self-explaining model. It provides both global and local concept explanations for each sample while performing the classification tasks.

PROTOTEX (Das et al., 2022) is an unsupervised self-explaining classification architecture based on prototype networks. It explains model decisions based on prototype tensors that encode latent clusters of training examples.
Cockatiel (Jourdan et al., 2023) is an unsupervised post-hoc concept-based method. It generates meaningful concepts from the last layer of a neural net model trained on an NLP classification task by using Non-Negative Matrix Factorization. In our experiment, we use a RoBERTa-based classifier as the base model for concept extraction.

Concept-Shap (Yeh et al., 2020) is an unsupervised post-hoc concept-based method that aims to infer a complete set of concepts. In our experiment, we use a RoBERTa-based classifier as the base model for concept extraction.

It is worth noting that we did not include approaches that utilize LLMs to extract concepts. These methods generate concept sets using LLMs before training and follow a pipeline similar to that of CBM. Specifically, in the text domain, their primary distinction from CBM lies in the use of concept sets defined by LLMs or humans. As such, we use CBM as a representative method.

A.3 Computational Complexity Analysis

Compared to the baseline methods, incorporating LLM-based evaluation during training increases computational costs. Specifically, our method requires approximately 22.2% more resources than the best-performing baseline, Cockatiel. However, in real-time or large-scale applications, our approach does not rely on querying LLMs during deployment. The inference phase only involves the local model, ensuring that deployment incurs no significant additional computational overhead. Moreover, our method demonstrates superior performance in interpretability compared to the best baseline, achieving enhanced comprehensibility (semantics, distinctiveness, consistency) of 25%. We believe the limited additional computational cost during training is a worthwhile trade-off for these significant benefits, especially as it does not impact real-world deployment efficiency.

B Human Evaluation of using LLMs as Human Proxies

We conducted several human studies to assess the quality of using LLMs for concept comprehensibility evaluation. For each task, we presented the concepts extracted using our method to human evaluators and asked them to rate their level of agreement with the conceptual summaries and segment highlightings generated by LLMs. For the ratings of agreement with conceptual summaries,

<table>
  <tr>
    <th></th>
    <th>CEBaB</th>
    <th>Beer</th>
    <th>Hotel</th>
    <th>IMDB</th>
    <th>AGnews</th>
    <th>Twitter</th>
    <th>SciCite</th>
  </tr>
  <tr>
    <th>Summaries</th>
    <td>4.23</td>
    <td>4.21</td>
    <td>4.45</td>
    <td>4.56</td>
    <td>4.63</td>
    <td>4.39</td>
    <td>4.14</td>
  </tr>
  <tr>
    <th>Highlightings</th>
    <td>4.56</td>
    <td>4.24</td>
    <td>4.40</td>
    <td>4.62</td>
    <td>4.42</td>
    <td>4.33</td>
    <td>4.18</td>
  </tr>
</table>

Table 6: Ratings of agreement with the LLM-generated conceptual summaries and segment highlightings

we presented all the concepts obtained using our method with three top-activated examples. For the ratings of agreement with concept-related segment highlightings, we focused on the concepts that LLM identified as having semantic meanings. For each concept, we randomly selected three cases with corresponding LLM-generated highlightings. Each concept was evaluated three times by different workers using a 5-point scale (1 indicating complete disagreement and 5 indicating complete agreement). We conducted our evaluation on Prolific\(^2\), a platform for facilitating high-quality human surveys. Each worker is paid $10.5 per hour and must pass a screening test to take the survey. A total of 42 participants took part in the LLM-generated conceptual summaries evaluation, while 21 participants took part in the LLM-generated segment highlightings evaluation. The demographic information of the evaluators, as provided by Prolific, is shown in Tables 11 and 12.

The rating results are shown in Table 6. Our human study, conducted with diverse participants, confirms that LLMs closely align with human judgment in concept evaluation without introducing much bias. This validates our method of employing LLMs as human proxies for concept evaluation and leveraging their feedback to refine the model.

C Human Evaluation of Concept Comprehensibility

C.1 Participants Information

Our human evaluation experiments are conducted on Prolific. Each worker is paid $10.5 per hour and must pass a screening test to take the survey. To ensure more reliable results, each question is evaluated three times by different workers. For the intruder detection and subjective rating experiments, we recruited 42 participants, while 24 participants were recruited for the forward simulatability experiments. The demographic information of the annotators is shown in Tables 11 and 13.

\footnotetext{2https://app.prolific.com}
Figure 5: Human ratings of provided explanations (Und. indicating understandability, Plaus. indicating plausibility, and Help. indicating helpfulness)

C.2 Intruder Detection Experiments Details
For the intruder detection experiments, we tested all extracted concepts of each method across seven tasks. Participants were asked to identify the example that did not belong to the same concept from a set of four highlighted examples. Their success rate in identifying the intruder was then calculated. The interface of the task is shown in Fig. 9(a).

C.3 Subjective Rating Experiments Details
For the subjective rating experiments, we present three top-activated examples for each concept along with its summary and ask participants to rate these examples based on the following criteria using a 1-5 scale:
Consistency: Do you think this concept formed by all these highlighted text parts has consistent semantic meanings?
Clarity: Do you think the semantic meaning of this concept is clear and easy to identify?
Task Relevance: Do you think this concept is related to the task?
Comprehensibility: Do you think this concept is comprehensible and easy for humans to understand?

C.4 Forward Simulatability Experiments Details
For the forward simulatability experiments, we randomly selected 20 samples from each of the Beer and AGnews datasets, ensuring that the accuracy of all methods within these samples was between 80% and 90%. First, participants classified the samples without any model explanations. Then, one method’s explanation was provided to the participants, who used this to predict the model’s output and recorded their confidence. To facilitate understanding by non-experts, we normalized the contribution of each concept to the class comparison, as illustrated in Fig. 9(c). Additionally, participants were asked to rate the understandability, plausibility, and helpfulness of the provided explanation, as Fig. 9(d) shows. The results of the human ratings are presented in Fig. 5.

<table>
  <tr>
    <th></th>
    <th>Waste of time</th>
    <th>Disappointment at characters</th>
    <th>No relation found</th>
  </tr>
  <tr>
    <td>This movie starts slow, then tapers off. After watching for about an hour, and seeing absolutely nothing happen, I walked out. I mean, nothing happened. Zero. Zip. Nada.</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>There is no story. The characters are vague representations of the most boring people. And they were abrupt and undeveloped. I wasted 60 minutes of my life. I could have been the first to walk out.</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

<table>
  <tr>
    <th></th>
    <th>Waste of time</th>
    <th>Disappointment at characters</th>
    <th>Pointless; Nothing happens</th>
  </tr>
  <tr>
    <td>This movie starts slow, then tapers off. After watching for about an hour, and seeing absolutely nothing happen, I walked out. I mean, nothing happened. Zero. Zip. Nada.</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>There is no story. The characters are vague representations of the most boring people. And they were abrupt and undeveloped. I wasted 60 minutes of my life. I could have been the first to walk out.</td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

(a) Concepts explanations provided by the Base model   (b) Concepts explanations provided by ECO-Concept

Figure 6: Comparison of explanations before and after concept enhancement in a beer review case

D Further Analysis on Ablation Study

D.1 Further Analysis on Impact of Concept Regularizers
From Table 5, we observed that the decrease in the Distinctiveness score when removing \( \mathcal{L}_{con} \) (from w/ all to w/o \( \mathcal{L}_{con} \) variant) is greater than the decrease when removing \( \mathcal{L}_{dist} \) (from w/ all to w/o \( \mathcal{L}_{dist} \) variant). This occurs in the tasks (IMDB, AGnews, Twitter) where the Semantics score of w/o \( \mathcal{L}_{con} \) variant is consistently lower than that of w/o \( \mathcal{L}_{dist} \) variant. This indicates that removing the consistency regularizer in these tasks leads to fewer discovered concepts with clear semantic meaning, which in turn affects the calculation of concept distinctiveness. This further supports the rationale for using both regularizers simultaneously. The two regularizers mutually reinforce each other, and their combined use is crucial for discovering more comprehensible concepts.

D.2 Further Analysis on Impact of Concept Comprehensibility Enhancement
The ECO-Concept (w/o \( \mathcal{L}_{com} \)) variant is fine-tuned using the same parameter settings and number of iterations as ECO-Concept training, but without the comprehensibility loss. As shown in Fig. 4, for concept comprehensibility metrics, fine-tuning with the comprehensibility loss performs better than fine-tuning without this loss. Moreover, in
certain tasks, solely relying on concept regularizers for retraining can even result in a decline in the semantics, distinctiveness, and consistency of the concepts. The possible explanation for this phenomenon is that concept regularizers guide concept learning within the tensor space. During the early stages of training, they enhance the consistency of representations for the same concept while improving the distinctiveness between different concepts. As training progresses, most concepts gradually acquire certain semantic meanings. However, in the absence of explicit concept annotation guidance, semantic ambiguity becomes unavoidable. This semantic ambiguity is often difficult to resolve in high-dimensional tensor spaces. For semantically ambiguous concepts, some samples may exhibit inconsistencies in semantics, while their features remain relatively close within the tensor space. In such cases, continuing to rely solely on concept regularizers not only fails to resolve the ambiguity but may exacerbate it, thereby compromising the performance of concept metrics. To address this, we propose an enhancement method inspired by the human evaluation process. By incorporating an evaluation mechanism that aligns more closely with human cognitive dimensions, we can reduce the semantic ambiguity of concepts and enhance their comprehensibility. The above analysis demonstrates the significance and effectiveness of our proposed comprehensibility enhancement stage.

A case in Fig. 6 presents that after concept enhancement, the model not only achieves higher semantic consistency but also clarifies the meaning of concepts that were previously semantically ambiguous.

E Parameter Sensitivity Analysis

E.1 Impact of the Number of Concepts

To evaluate the impact of the number of concepts on task performance and concept-related metrics, we conducted experiments with 10, 20, 30, 40, and 50 concepts, respectively. The results, as shown in Fig. 7, indicate that when the number of concepts is 20, most metrics achieve their optimal values. All variants achieve similar results across different tasks, and the number of concepts has minimal effect. For concept interpretability, when the number of concepts exceeds 20, both the concept semantics and distinctiveness decline. This suggests that an excessive number of concepts may bring more semantically irrelevant or redundant concepts. Furthermore, regardless of the number of concepts, it is impossible to ensure that all extracted concepts have semantic meanings due to the unsupervised nature of our method. Nevertheless, when the number of concepts is set to 20, over half of the extracted concepts can be confidently considered semantically meaningful.

<table>
  <tr>
    <th>Parameter</th>
    <th colspan="4">Search Range</th>
    <th>Optimal Value</th>
  </tr>
  <tr>
    <td>\( \lambda_{con} \)</td>
    <td>0.01</td>
    <td>0.05</td>
    <td>0.1</td>
    <td>0.3</td>
    <td>0.5</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>\( \lambda_{dist} \)</td>
    <td>-0.001</td>
    <td>-0.005</td>
    <td>-0.01</td>
    <td>-0.05</td>
    <td>-0.1</td>
    <td>-0.01</td>
  </tr>
  <tr>
    <td>\( \lambda_{com} \)</td>
    <td>0.1</td>
    <td>0.5</td>
    <td>1.0</td>
    <td>1.5</td>
    <td>2.0</td>
    <td>1.0</td>
  </tr>
</table>

Table 7: Details of the grid search range for optimal trade-off parameters

<table>
  <tr>
    <th></th>
    <th>CEBaB</th>
    <th>Beer</th>
    <th>Hotel</th>
    <th>IMDB</th>
    <th>AGnews</th>
    <th>Twitter</th>
    <th>SciCite</th>
  </tr>
  <tr>
    <th>K=3</th>
    <td>.895</td>
    <td>.925</td>
    <td>.934</td>
    <td>.947</td>
    <td>.897</td>
    <td>.916</td>
    <td>.919</td>
  </tr>
  <tr>
    <th>K=5</th>
    <td>.817</td>
    <td>.826</td>
    <td>.803</td>
    <td>.821</td>
    <td>.802</td>
    <td>.844</td>
    <td>.870</td>
  </tr>
</table>

Table 8: Rank correlation scores calculated based on the ranks of the top K attributed concepts before and after noisy perturbations

The trade-off parameters \( \lambda_{con} \), \( \lambda_{dist} \), and \( \lambda_{com} \) were selected through grid search, aiming to optimize the model’s classification performance on the validation set. The detailed information about the grid search range is presented in Table 7.

Our grid-search results indicated that other parameter combinations had lower performance compared to the optimal combination. Table 9 shows the classification performances of the top 4 performing parameter combinations across all tasks.

F Model Robustness Evaluation

To test whether our method can yield reliable explanations for noisy samples, we performed experiments using seven datasets. We randomly chose 100 test samples and applied noisy perturbations to 5% of the words in each sample. These words were chosen based on their frequency in the training set (excluding stop words), with higher-frequency words being more likely to be perturbed. For each sample, we randomly applied one of these adversarial strategies: synonym replacement or spelling error. For synonym replacement, we selected a synonym from WordNet (Miller, 1995), prioritizing the one with the lowest frequency in the training set. For spelling errors, we randomly altered a single character in the word.

To evaluate the robustness of our model’s expla-
Figure 7: Classification performance and concept metrics of our method with different concept numbers.

<table>
  <tr>
    <th rowspan="2">Parameter Combination</th>
    <th colspan="2">CEBaB</th>
    <th colspan="2">Beer</th>
    <th colspan="2">Hotel</th>
    <th colspan="2">IMDB</th>
    <th colspan="2">AGnews</th>
    <th colspan="2">Twitter</th>
    <th colspan="2">SciCite</th>
  </tr>
  <tr>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
    <th>Acc</th><th>F1</th>
  </tr>
  <tr>
    <td>(0.1, -0.01, 1)</td>
    <td>.697</td><td>.808</td>
    <td>.885</td><td>.885</td>
    <td>.981</td><td>.981</td>
    <td>.937</td><td>.937</td>
    <td>.941</td><td>.961</td>
    <td>.828</td><td>.813</td>
    <td>.860</td><td>.881</td>
  </tr>
  <tr>
    <td>(0.05, -0.01, 1)</td>
    <td>.677</td><td>.793</td>
    <td>.882</td><td>.882</td>
    <td>.977</td><td>.977</td>
    <td>.934</td><td>.934</td>
    <td>.927</td><td>.939</td>
    <td>.824</td><td>.801</td>
    <td>.856</td><td>.879</td>
  </tr>
  <tr>
    <td>(0.1, -0.01, 0.5)</td>
    <td>.683</td><td>.799</td>
    <td>.883</td><td>.883</td>
    <td>.979</td><td>.979</td>
    <td>.931</td><td>.931</td>
    <td>.936</td><td>.951</td>
    <td>.826</td><td>.804</td>
    <td>.856</td><td>.879</td>
  </tr>
  <tr>
    <td>(0.05, -0.01, 0.5)</td>
    <td>.686</td><td>.799</td>
    <td>.883</td><td>.883</td>
    <td>.977</td><td>.977</td>
    <td>.929</td><td>.929</td>
    <td>.929</td><td>.948</td>
    <td>.823</td><td>.799</td>
    <td>.858</td><td>.880</td>
  </tr>
</table>

Table 9: Classification performances of the top 4 performing parameter combinations (\( \lambda_{con}, \lambda_{dist}, \) and \( \lambda_{com} \)) across all tasks. The best results are bolded.

nations, we compared the top concepts attributed to the same sample before and after the noisy perturbations. We used Kendall’s rank correlation (Kendall, 1938) as the evaluation metric, which accounts for the concept importance rankings. The rank correlation scores, calculated based on the ranks of the top K attributed concepts before and after noisy perturbations, are shown in Table 8. After noisy perturbations, our model is still able to attribute the test samples to concept explanations that closely match those before the attack. This demonstrates the robustness of the explanations provided by our method under noisy conditions, highlighting its potential for practical applications.

G Prompts

The prompts we used are shown in Table 10.

H Comparison of Explanations

The comparison of explanations of different concept-based methods is shown in Fig. 8.
Prompt for Concept Summary

Prompt: We’re studying the concepts used to determine whether the sentiment of movie reviews is positive or negative. Each concept focuses on some specific elements. First, pay attention to all highly activated text parts in the following example sentences. Then think deeply to find the relations between all the highly activated text parts. Finally, determine whether all these highly activated text parts can represent a consistent concept. If not, please output 'No relation found'; if yes, provide a summary in no more than ten words. The activation format is token-tab>activation. Activation values range from 0 to 1. A concept finding what it’s looking for is represented by a non-zero activation value. The higher the activation value, the stronger the match.

Prompt for Concept-related segment highlightings

Prompt: We’re studying the concepts used to determine whether the sentiment of movie reviews is positive or negative. Each concept focuses on specific elements in a short document. Look at the summary of what the concept is, and try to determine whether each token in the document is related to the concept. Return 1 for related tokens and 0 for unrelated tokens. You should carefully consider whether each token is related to the concept and not return 1 for all the tokens in the document.

Table 10: Prompt for concept summary and concept-related segment highlightings in the movie review sentiment classification task

<table>
  <tr>
    <th colspan="5">Highest education level completed</th>
    <th>Avg. Age</th>
  </tr>
  <tr>
    <td>High School<br>5</td>
    <td>Undergraduate<br>21</td>
    <td>Graduate<br>11</td>
    <td>Doctorate<br>5</td>
    <td></td>
    <td>36.4</td>
  </tr>
  <tr>
    <th colspan="5">Ethnicity</th>
    <th>Female:Male</th>
  </tr>
  <tr>
    <td>White<br>21</td>
    <td>Black<br>12</td>
    <td>Asian<br>5</td>
    <td>Mixed<br>4</td>
    <td></td>
    <td>1:1.62</td>
  </tr>
</table>

Table 11: Demographic information of participants in LLM-generated conceptual summaries evaluation, intruder detection experiments, and subjective rating experiments

<table>
  <tr>
    <th colspan="5">Highest education level completed</th>
    <th>Avg. Age</th>
  </tr>
  <tr>
    <td>High School<br>3</td>
    <td>Undergraduate<br>10</td>
    <td>Graduate<br>7</td>
    <td>Doctorate<br>1</td>
    <td></td>
    <td>29.1</td>
  </tr>
  <tr>
    <th colspan="5">Ethnicity</th>
    <th>Female:Male</th>
  </tr>
  <tr>
    <td>White<br>8</td>
    <td>Black<br>6</td>
    <td>Asian<br>5</td>
    <td>Mixed<br>2</td>
    <td></td>
    <td>1:1.33</td>
  </tr>
</table>

Table 12: Demographic information of participants in LLM-generated concept-related highlightings evaluation

<table>
  <tr>
    <th colspan="5">Highest education level completed</th>
    <th>Avg. Age</th>
  </tr>
  <tr>
    <td>High School<br>3</td>
    <td>Undergraduate<br>11</td>
    <td>Graduate<br>8</td>
    <td>Doctorate<br>2</td>
    <td></td>
    <td>40.2</td>
  </tr>
  <tr>
    <th colspan="5">Ethnicity</th>
    <th>Female:Male</th>
  </tr>
  <tr>
    <td>White<br>11</td>
    <td>Black<br>6</td>
    <td>Asian<br>4</td>
    <td>Mixed<br>3</td>
    <td></td>
    <td>1:1.18</td>
  </tr>
</table>

Table 13: Demographic information of participants in forward simulatability experiments
Case
Pours a golden color that is pretty decent in relationship to the taste of the brew. Head and lace are nice, not impressive, just nice. Smells sweet but funky like. Taste is super sweet, nasty, unbelievably sweet with a fake honey taste. Sickeningly sweet. Avoid at all costs.
(a) ECO-Concept

Top 3 Concepts
<table>
  <tr><th>Excessive sweetness</th><td>–</td></tr>
  <tr><th>Unpleasant taste</th><td>–</td></tr>
  <tr><th>Clear beer color</th><td>+</td></tr>
</table>

Case
Pours a golden color that is pretty decent in relationship to the taste of the brew. Head and lace are nice, not impressive, just nice. Smells sweet but funky like. Taste is super sweet, nasty, unbelievably sweet with a fake honey taste. Sickeningly sweet. Avoid at all costs.
(b) Cockatiel

Top 3 Concepts
<table>
  <tr><th>Positive sentiment phrases</th><td>–</td></tr>
  <tr><th>Sweet and malty flavors</th><td>–</td></tr>
  <tr><th>No relation found</th><td>–</td></tr>
</table>

Case
Pours a golden color that is pretty decent in relationship to the taste of the brew. Head and lace are nice, not impressive, just nice. Smells sweet but funky like. Taste is super sweet, nasty, unbelievably sweet with a fake honey taste. Sickeningly sweet. Avoid at all costs.
(c) Concept-Shap

Top 3 Concepts
<table>
  <tr><th>No relation found</th><td>–</td></tr>
  <tr><th>Beer taste</th><td>–</td></tr>
  <tr><th>Nice and sweet</th><td>+</td></tr>
</table>

Case
Pours a golden color that is pretty decent in relationship to the taste of the brew. Head and lace are nice, not impressive, just nice. Smells sweet but funky like. Taste is super sweet, nasty, unbelievably sweet with a fake honey taste. Sickeningly sweet. Avoid at all costs.
(d) ProtoTEEx

Top 3 Concepts
<table>
  <tr><th>Disappointment with beer</th><td>–</td></tr>
  <tr><th>No relation found</th><td>–</td></tr>
  <tr><th>Head retention</th><td>+</td></tr>
</table>

Figure 8: Comparison of concept explanations provided by ECO-Concept and other concept-based methods
When studying the concepts used to determine the sentiment polarity of movie reviews (positive sentiment or negative sentiment),
each concept focuses on some specific test parts that are shared across different cases. Your task is to analyze which concepts are more relevant to the task and form a coherent concept with semantic meanings, and evaluate the quality of the concept.

For each concept, you will be given some cases with its highlighted test parts. Deeper highlighted colors indicate stronger relevance to the concept.
Below are four cases in a row of them belong to the same concept. Please identify the case that is conceptually different.

The bar is the one I've been waiting for ever since. The movie has nothing about it. None of the plot, none of the characters, none of the story. It's a waste of time and money. I'm sorry, but when I saw the movie I was so disappointed. I thought it would be better than it is. I have a friend who loves this movie and I asked him to watch it with me. He said he didn't like it either. I don't know what to say about this movie. It's not bad, but it's not good either. I think it's just another disappointment for me. I wish I could have seen something else instead of this. I'm sorry, but I can't recommend this movie to anyone. It's a waste of time and money.

I watched this movie and I am not sure how to explain it. The plot is very weak and the acting is not great. The story is not engaging and the characters are not well-developed. I found myself bored throughout the entire film. I would not recommend this movie to anyone. It's a waste of time and money.

I watched this movie and I am not sure how to explain it. The plot is very weak and the acting is not great. The story is not engaging and the characters are not well-developed. I found myself bored throughout the entire film. I would not recommend this movie to anyone. It's a waste of time and money.

I watched this movie and I am not sure how to explain it. The plot is very weak and the acting is not great. The story is not engaging and the characters are not well-developed. I found myself bored throughout the entire film. I would not recommend this movie to anyone. It's a waste of time and money.

(a) The survey interface of intruder detection

The model makes its predictions based on several concepts. Each concept focuses on some specific test parts and is linked to a particular sentiment polarity.
Your task is to evaluate the quality of the model based on these concepts.
For each case, you will see the most important concepts the model used for prediction. Each concept is color-coded to highlight whether it is related to the positive or negative part of the case (different colors). Deeper highlighted colors indicate stronger relevance of the test parts to the concept.

<table>
  <tr>
    <th>Concept</th>
    <th>Concept Contributions</th>
  </tr>
  <tr>
    <td>Disappointment towards beer</td>
    <td>This concept is highly related to the negative sentiment</td>
  </tr>
  <tr>
    <td>Nice beer smell</td>
    <td>This concept is related to the positive sentiment</td>
  </tr>
  <tr>
    <td>Weak beer head</td>
    <td>This concept is somewhat related to the negative sentiment</td>
  </tr>
</table>

Q1: Based on the concept explanation the model found, what do you think the model’s prediction is?
Tip: There are two concepts related to the negative sentiment, and one of them, “Disappointment towards beer”, is highly relevant, so the model is likely to predict this case as having a negative sentiment.

Negative sentiment   Positive sentiment

(b) The tutorial of forward simulatability part (1)

Tutorial
We have a text classification model that classifies beer reviews into two sentiment polarities: Negative and Positive.
First, you will be shown a review text without any explanation from the model. Based on your own judgment, determine the sentiment of the review.
2. Then, you will be shown an explanation generated by the text classification model. You need to use these explanations to infer the model output.

[Beer Review] I poured this disappointing beer into my tumbler at 45 degrees. The beer was coco-cola colored and poured a really weak thin head. The beer did have a nice big caramel malt smell but the flavor was just not there. The head of the beer went away really fast and the beer was weak and flat tasting. A nut brown is best my favorite beer in the world and this one just wasn’t there even though I love Dundee Honey Brown.

Q1-1: Based on your own judgement, what do you think the sentiment polarity of this case is?
Tip: Answer this question based on your own judgement.

<table>
  <tr>
    <th></th>
    <th>Negative sentiment</th>
    <th>Positive sentiment</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Q1-2: How confident are you with your judgement? (Please rate your confidence on a scale of 1 to 5, where 1 means "not confident at all" and 5 means "very confident".)
Tip: Rate your confidence in your own judgement.

<table>
  <tr>
    <th></th>
    <th>Not confident at all</th>
    <th>Somewhat confident</th>
    <th>Very confident</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Q2: Do you find the provided explanation easy to understand? (Please rate the understandability of explanation on a scale of 1 to 5, where 1 means "not understandable at all" and 5 means "very understandable".)
Tip: Rate how easy it is for you to understand the explanation.

<table>
  <tr>
    <th></th>
    <th>Not understandable at all</th>
    <th>Somewhat understandable</th>
    <th>Very understandable</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Q3: Do you think the provided explanation is plausible? (Please rate the plausibility of explanation on a scale of 1 to 5, where 1 means "not plausible at all" and 5 means "very plausible".)
Tip: Rate how plausible do you think the explanation is.

<table>
  <tr>
    <th></th>
    <th>Not plausible at all</th>
    <th>Somewhat plausible</th>
    <th>Very plausible</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Q4: Do you think the provided explanation helps you better understand the model’s behavior? (Please rate the helpfulness of explanation on a scale of 1 to 5, where 1 means "not helpful at all" and 5 means "very helpful".)
Tip: Rate how helpful do you think the explanation is.

<table>
  <tr>
    <th></th>
    <th>Not helpful at all</th>
    <th>Somewhat helpful</th>
    <th>Very helpful</th>
  </tr>
  <tr>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

(c) The tutorial of forward simulatability part (2)

(d) The tutorial of forward simulatability part (3)

Figure 9: Screenshots of the survey interface

Reviewer paper summaries: This paper introduces ECO-Concept, a framework designed to enhance the interpretability of deep neural language models by automatically discovering comprehensible concepts without requiring predefined annotations. Leveraging a slot attention mechanism, ECO-Concept extracts task-specific concepts and evaluates their comprehensibility using large language models (LLMs) as human proxies. The framework incorporates consistency and distinctiveness losses to ensure semantic coherence and diversity among concepts, while a concept bottleneck links extracted concepts to model predictions. By integrating a simulation loss based on LLM evaluations, ECO-Concept refines the model to prioritize both important and human-understandable concepts, achieving superior interpretability and performance across diverse tasks.

The paper presents a novel approach to evaluate concept-based explanations by leveraging LLMs as human proxies. Building on this evaluation method, it proposes a new self-explaining model that outperforms prior supervised and unsupervised concept-based interpretability methods.
Summary of strengths: 1. No Need for Predefined Concept Annotations: ECO-Concept automatically discovers task-specific concepts using a slot attention mechanism, eliminating the reliance on manually annotated concept datasets. This makes the framework more scalable and adaptable to new tasks without requiring costly supervision.

2. Human-Understandable Concepts via LLM Evaluation: The framework employs large language models (LLMs) to evaluate and refine the comprehensibility of extracted concepts. By summarizing and simulating concept activations, ECO-Concept ensures that the discovered concepts align with human intuition, enhancing interpretability and user trust.

**S1.** The paper introduces a new approach for evaluating concept-based explanations using simulatability, with LLMs serving as human proxies.

**S2.** The paper proposes a self-explaining model that incorporates an LLM-in-the-loop training procedure. The model also employs custom regularizations to enhance the distinctiveness and consistency of the identified concepts.

**S3.** The authors conducted thorough and well-designed experiments, demonstrating that their method consistently outperforms existing approaches.
Summary of weaknesses: 1. Dependence on LLMs for Comprehensibility: ECO-Concept relies heavily on large language models (LLMs) to evaluate and refine concept comprehensibility.  This dependency may introduce computational overhead and limit the framework's applicability in resource-constrained environments.  Additionally, the quality of concept evaluation is contingent on the performance and biases of the LLMs used.

2. Limited Generalization to Non-Text Domains: While ECO-Concept is designed for text-based tasks, its reliance on slot attention and LLMs may not easily extend to other domains, such as image or audio data, without significant modifications.  This limits its versatility compared to more domain-agnostic interpretability methods.

3. Potential for Semantic Ambiguity in Concepts: Despite the use of consistency and distinctiveness losses, the unsupervised nature of concept extraction may still result in semantically ambiguous or less meaningful concepts, especially in complex or noisy datasets.  This could reduce the overall interpretability of the model.

4. Computational Complexity: The two-phase training process, which includes concept extraction, LLM-based evaluation, and fine-tuning, may increase computational complexity and training time.  This could make ECO-Concept less practical for real-time or large-scale applications.

5. Some hyperparameters were used in the experiment (e.g., λcon=0.1, λdis=-0.01, λsimu=1), but the basis for the selection of these hyperparameters was not specified, or sensitivity analysis was performed.

6. The experiments did not evaluate the robustness of the ECO-Concept against noisy data or adversarial samples, which is an important consideration in practical applications. 

7. In the experiment, the number of concepts is fixed at 20, and the influence of different concept numbers on the performance of the model is not discussed. The number of concepts may vary depending on the complexity of the task, and fixed Settings may limit the flexibility of the model.

**W1.** When examining the results before and after concept comprehensibility enhancement (Section 4.4, Figure 8), it is unclear whether the observed improvement is due to longer training with concept regularizers or the comprehensibility enhancement itself. An ideal setup would include an analysis of the second phase conducted without the simulatability loss. Additionally, which of the improvements shown in Figure 8 are statistically significant? 


**W2.** In L585-594, comparing "w/o all" to "w/o distinctiveness" to argue that the distinctiveness regularizer improves distinctiveness is not reasonable, as neither configuration includes the distinctiveness regularizer. A more valid comparison would be between "w/ all" and "w/o distinctiveness." Similarly, this applies to the consistency regularizer. Moreover, the results indicate that, in some cases, removing an unrelated component (e.g., "w/o consistency" in distinctiveness results) has a greater effect than removing the relevant component. For instance, in the Hotel task, the drop in distinctiveness score from "w/ all" to "w/o consistency" is larger than the drop to "w/o distinctiveness." The paper does not provide an explanation for these counterintuitive cases.

**W3.** As acknowledged by the authors, the number of concepts is fixed, which may limit the flexibility or scalability of the approach.

**W4.** The paper does not include an ablation study to evaluate the effect of the subset size of samples used for calculating the simulation loss.
Comments, suggestions and typos: 1. The detailed basis for the selection of hyperparameters is provided, and the sensitivity analysis is carried out to discuss the influence of different hyperparameters on the performance of the model.

2. Noise data or adversarial samples are introduced to test the robustness of the model. How to improve the anti-interference ability of the model by regularization or other techniques is discussed.

3. Experiments are designed to study the effects of different concept quantities on model performance and find the optimal concept quantities.

**C1.** How are the trade-off parameters chosen? Would it be feasible or beneficial to make these parameters trainable? Exploring the impact of trainable trade-off parameters could provide additional insights into optimizing the balance between different components.


L434: self-expalin -> self-explain
Overall assessment scores: 2.5

4
Metareview score: 4
Metareview: The paper introduces ECO-Concept, a novel framework designed to enhance the interpretability of deep neural language models by automatically discovering comprehensible concepts without relying on predefined concept annotations. Unlike existing methods that either depend on supervised concept annotations or produce unsupervised concepts that are often unintuitive, ECO-Concept leverages an object-centric architecture, specifically a slot attention mechanism, to extract task-specific semantic concepts automatically. To ensure these concepts are human-understandable, the framework employs large language models (LLMs) as proxies to evaluate and guide the comprehensibility of the extracted concepts. This evaluation is integrated into the model fine-tuning process, enabling the model to prioritize concepts that are both semantically meaningful and intuitively interpretable. Additionally, ECO-Concept incorporates consistency and distinctiveness losses to enhance the coherence and diversity of the discovered concepts, while a concept bottleneck ensures that model predictions are directly linked to these concepts. Experimental results demonstrate that ECO-Concept achieves classification performance comparable to black-box models while significantly improving interpretability. The framework outperforms existing supervised and unsupervised concept-based methods in both quantitative evaluations and user studies, validating its effectiveness in producing human-aligned, interpretable explanations.
Summary of reasons to publish: (raz1, idWg, dLqq) The paper proposes a self-explaining/interpretable-by-design model without the need for predefined concept annotations. 
The paper proposes to enhance the comprehensibility of concepts using LLMs as human proxies to guide model fine-tuning.

(raz1, idWg) The authors conducted thorough and well-designed experiments, evaluating both quantitatively and qualitatively via user study and including ablation studies to analyze the impact of the regularizers and comprehensibility enhancement.

(raz1) The model also employs custom regularizations to enhance the distinctiveness and consistency of the identified concepts.

(idWg) The experimental results show the approach achieves classification performance comparable to black-box models while enabling interpretability through concepts. Moreover, the evaluation demonstrates its effectiveness in terms of interpretability against several baselines.