Concept Bottleneck Large Language Models
Authors: Chung-En Sun
Tuomas Oikarinen
Berk Ustun
Tsui-Wei Weng
Venue: ICLR 2025 Poster (2025)
Keywords: LLMs
Interpretability
Concept Bottleneck Model
Primary area: interpretability and explainable AI

Abstract: We introduce Concept Bottleneck Large Language Models (CB-LLMs), a novel framework for building inherently interpretable Large Language Models (LLMs). In contrast to traditional black-box LLMs that rely on limited post-hoc interpretations, CB-LLMs integrate intrinsic interpretability directly into the LLMs -- allowing accurate explanations with scalability and transparency. We build CB-LLMs for two essential NLP tasks: text classification and text generation. In text classification, CB-LLMs is competitive with, and at times outperforms, traditional black-box models while providing explicit and interpretable reasoning. For the more challenging task of text generation, interpretable neurons in CB-LLMs enable precise concept detection, controlled generation, and safer outputs. The embedded interpretability empowers users to transparently identify harmful content, steer model behavior, and unlearn undesired concepts -- significantly enhancing the safety, reliability, and trustworthiness of LLMs, which are critical capabilities notably absent in existing language models.

Published as a conference paper at ICLR 2025
CONCEPT BOTTLENECK LARGE LANGUAGE MODELS
Chung-En Sun, Tuomas Oikarinen, Berk Ustun, Tsui-Wei Weng University of California San Diego
{ cesun, toikarinen, berk, lweng) @ucsd. edu
ABSTRACT
We introduce Concept Bottleneck Large Language Models (CB-LLMs), a novel framework for building inherently interpretable Large Language Models (LLMs). In contrast to traditional black-box LLMs that rely on limited post-hoc interpreta- tions, CB-LLMs integrate intrinsic interpretability directly into the LLMs allowing accurate explanations with scalability and transparency. We build CB-LLMs for two essential NLP tasks: text classification and text generation. In text classification, CB-LLMs is competitive with, and at times outperforms, traditional black-box mod- els while providing explicit and interpretable reasoning. For the more challenging task of text generation, interpretable neurons in CB-LLMs enable precise concept detection, controlled generation, and safer outputs. The embedded interpretability empowers users transparently identify harmful content, steer model behavior, and unlearn undesired concepts significantly enhancing the safety, reliability, and trustworthiness of LLMs, which are critical capabilities notably absent existing language models. Our code is available at https://github.com Trustworthy- ML-Lab/CB-LLMs
INTRODUCTION
Large Language Models (LLMs) have become pivotal in advancing Natural Language Processing (NLP) tasks. However, their inherent opacity presents significant challenges ensuring reliability and safety, particularly when their decisions stem from unclear or flawed reasoning. This lack of transparency not only hinders the detection of potential misuse or manipulation, but also makes difficult to identify and mitigate unsafe outputs. Furthermore, the complexity of their inner workings complicates debugging efforts, making even more challenging to diagnose and resolve these issues effectively and efficiently.
Several recent works have explored interpretable models the image domain using concept bottle- necks [5] 10 14 16
21), demonstrating the feasibility of building interpretable models for image classification. In contrast, research on building interpretable models for NLP remains limited. Only few recent studies & [7] have investigated the development of inherently interpretable language models. However, these methods are largely restricted to text classification on small datasets and struggle to scale larger benchmarks or more complex tasks like text generation, which is capability that has become indispensable with the growing dominance of autoregressive LLMs across diverse applications.
19 20
Given the limited research on interpretable LLMs, our study is driven by two key goals. First, we aim enhance the interpretability of LLMs in text classification setting by improving scalability for larger benchmarks, reducing development costs, and boosting both model performance and interpretability. Second, we tackle the more challenging text generation setting of developing a generative LLM with intrinsic interpretability an area that is largely unexplored, as existing research focus solely on text classification tasks [8] [77. By embedding interpretability directly into the LLMs, we enable greater transparency, steering, and control over their behavior. For example, as demonstrated in our case studies, interpretability empowers users to trace the underlying reasoning behind harmful outputs, identifying how inputs and neurons contribute the generation of unsafe tokens, steering LLMs toward safer response, and forcing LLMs to unlearn undesired concepts. This deeper insight is crucial for enhancing the safety, reliability, and trustworthiness of L.LMs.

---

Published as a conference paper at ICLR 2025
With the above two goals, in this work, we propose novel framework for constructing interpretable LLMs that come with intrinsic interpretability. Our method can transform any pretrained black- box LLM into inherently interpretable LLM by incorporating a iammannnteerrretbble Concept Bottleneck Layer alongside a linear prediction layer. We named our method Concept Bottleneck Large Language Models (CB-LLMs)) which is,
best knowledge, the first CBM framework that scales to large text classification and text generation tasks. In Section 3 we formally introduce how to train CB-LLMs for the text classification task, and in Section we present a novel approach to train CB-LLMs for the text generation task. Note that due to the very different nature between the classification task and generation task, a careful design for the training pipeline and algorithm of CB-LLMs for each task is required. To avoid confusion, we refer to these models as CB-LLMs (classification) and CB-LLMs (generation) for each task respectively when the context needs to be clear. Our contributions are as follows:
1. We present novel framework to build interpretable LLMs for text classification and generation tasks: CB-LLMs (classification) and CB-LLMs (generation). Our CB-LLMs encapsulates the best of both worlds: it matches the high performance of black-box models across multiple settings while offering clear interpretability, a feature absent in existing LLMs.
2. In the classification case, our CB-LLMs (classification) match the accuracy of the standard black- box models and achieves a 1.5x higher average rating compared to the existing works on the faithfulness evaluation. This suggests that our CB-LLMs (classification) provide high-quality interpretability without sacrificing performance.
3. In the generation case, our CB-LLMs (generation) match the performance of the standard black- box models. It provides controllable and understandable generation, allowing further interaction between the user and the LLM. We also developed the first inherently interpretable LLM chatbot that can detect toxic queries and provide controllable responses.
2 RELATED WORK
Concept Bottleneck Models (CBM). CBM introduces model structure that incorporates a concept bottleneck layer (CBL), where individual neurons are explicitly designed to learn specific, cemaa-interrpeetable concepts. This CBL is followed by a final linear layer to produce predictions. Because the activation of the interpretable neurons directly and linearly contributes to the final logits, users can easily understand the model's decision-making process and intervene at the bottleneck layer to correct potential errors.
CBM in Image Classification. classification tasks 2 3 10, [4, [6,
Recently, CBMs have been revisited in the context of image 19
In the seminal work [3]. the authors proposed to train CBMs utilizing numan-annotater concept labels, which may expensive to collect in practice.
20
21
To address this challenge, recent works (20 21 leverage concept activation vectors 44] or the multi- modal CLIP model [13] to build CBMs efficiently. However, these approaches still require concept labels to obtain Concept Activation Vector (CAV) need to restrict the backbone to the CLIP image encoder if concept labels are unavailable, which does not fully resolve the limitation. Recognizing this constraint, [10] proposed Label-free CBM to learn CBMs without relying concept labels by using the interpretability tool CLIP-Dissect I A recent work [66] further proposed VLG-CBM, to ensure the faithfulness of Vision-based CBMs in image classification tasks, and new metric called Number of Effective Concept (NEC) is also proposed to control the information leakage and ensure fair comparison between CBMss
Despite the extensive studies of CBMs in the image classification tasks, to the best of our knowledge, there still CBM that scales large NLP benchmarks or text generation tasks, Consequently, our work focuses on learning an efficient, automated, and high-performance CBM for LLMs.
CBM in Text Classification. Two recent works studied the CBM structure in text classification settings. 8 introduced Text Bottleneck Models (TBMs), an interpretable text classification frame- work that trains linear predictor the concept labels generated by GPT-4. Their approach does not involve training the CBL before the linear predictor; instead, they utilize the output score from GPT--4 to replace the output from CBL. Another work, [17). proposed C3M, framework that merges
2

---

Published as a conference paper at ICLR 2025
human-annotated concepts with concepts generated and labeled by ChatGPT to build the CBM based on GPT-2 and BERT backbone.
While both works aimed to construct interpretable language models utillizing the CBM structure, it's notable that TBM necessitates multiple queries to GPT-4 for each text sample, thereby limiting its applicabiliity only small subset of text samples (250 samples) in the datasets. On the other hand, C'M still depends on human-annotale concepts to augment the concept set, making it challenging to scale to large datasets that lack pre-existing concept annotations. Furthermore, neither work studied the autoregressive generation setting, which is much more interesting setting given the increasing prevalence chatbots.
In contrast, our CB-LLMs can scale to large classification datasets of over 500,000 samples and does not require using GPT-4 to label the concepts. CB-LLMs provide interpretability without losing performance and achieves the same accuracy the non-interpretable black-box counterpart. Furthermore, our proposed approach can handle generation tasks, while existing works are limited to simple text classification. More detailed comparisons are shown in Table
Table I: Comparison between our CB-LLMs and other interpretable language models in terms of scalability, efficiency, accuracy, and interpretability.
Methods
Scalability
Efficiency
Accuracy
Interpretability
Text generation setting
Large text classification dataset
Concept labeling
Inference new sumples Same acturacy as w/o querying L. black-box model
Provide faithful explanations
querying M.I.ss
CB-ILMs
Yes
Yes
Yes
Yes
Yes
Yes
Prior works:
TBM 8
No
No
No
Na
Na
No
C
No
CB-LLMS FOR TASK CLASSIFICATION
In this section, we develop interpretable language models for text classification. Section 3.1 presents a cost-effective method for transforming pretrained models into interpretable ones. We evaluate its performance in Section 3.2 and showcase its benefits through a case study Section 3.3
3.1 METHOD
Our proposed method consists of five steps and is illustrated in Figure I
Step 1: Concept Generation. The first step is to generate set of concepts related the down- stream task. To automate this process, we leverage ChatGPT [T] as a replacement for the domain experts. For any text classification dataset D with n classes/labels, we prompt ChatGPT generate the concept subset C; for each class i. Then, the concept set C is the union of C;, C Uh C. We defer the details of prompting to Appendix A.8 Note that our proposed prompting style requires only queries to ChatGPT to obtain the full concept set, which can be done through the web interface provided by OpenAl at zero cost.
Step 2: Automatic Concept Scoring. After generating the concept set C, the next step is to obtain the concept labels for a given text sample in dataset D for training. Typically, this stage requires involving domain experts and can be time-consuming. TBM [ and C [17] leveraged ChatGPT GPT-4 automate the labeling process, but their method incurs significant costs due to the high number of API calls required (more than 100M API calls for large dataset like DBpedia). To overcome this challenge, we propose an automatic scoring strategy by utilizing sentence embedding models, which can measure the similarity between each concept and any text sample without the need query LLMs. We name this strategy as Automatic Concept Scoring (ACS).
For any sentence embedding model E that encodes a text sample into a fixed-size embedding, we calculate the concept scores (z) E R& for text sample x by calculating the following:
(x) = (c1) E(x), E(c)) [(x)]T
(I)
where (z)) E IRd denotes the text embedding generated by E, cj is the j-th concept in the concept set C, and l is the size of the concept set. Each component of the vector Sa()) measures the similarity
3

---

Published as a conference paper at ICLR 2025
Step Concept Generation
Step Concept Scoring Step 3: Concept Correction
Concept
Concept Scores after
Query ChatGPT
Overpriced
CB-LLM: classification
Concept Unopperling food.
0.1
Set the incorrect
0.1
Sentence
concept score to zero
Concept Unprofesstonel service.
Embedding Model
Concept
Reviabl cleoning.
Step 4: Train the Concept Bottleneck Layer
Step Train the Línear Layer
CBL
Neuron Activations
Maximize similarity
Training example x
Pretrained LM
Worst company everill! IIII custamer service. you call on Sunday you're out luck and they care!!
Ass
Prediction
Linear
negetive
Layer
Figure 1: The overview of CB-LLMs (classffication). The pipeline consists five steps: (1) Generate concept via querying ChatGPT. (2) Automatically label the samples with sentence embedding models. (3) Fix the incorrect concept labels. (4) Train backbone LLM and CBL with the concept labels. (5) Train linear layer on top of the CBL to make the class predictions.
between the text and concept Cj. These entries represent pseudo concept labels for that will use as a learning target for Concept Bottleneck Layer (CBL) in step 4.
We the off-the-shelf sentence embedding models all- all-mpnet -aapet--bas--e from Huggingface [8] for ACS. It achieves better accuracy than labeling by LLMs and requires only a tenth of the time. We will discuss this in Section 3.2
Step 3: Automatic Concept Correction (ACC). While ACS in step offers an efficient way to provide pseudo concept labels (concept scores), its correctness is dependent on the performance of the sentence embedding model. This may introduce a limitation wherein the concept scores may not align with human reasoning, consequently impacting the learning of the CBL. Notably, this challenge prevalent in image CBM works that do not rely on human-assigned concept labels [0 21).
To address this challenge, we proposed Automatic Concept Correction (ACC), technique to improve the quality the concept scores generated by ACS in step Recall that step 1, we generate the concept set C Uh= 1 C, for dataset D with classes, where C, is the concept subset for class i. We define the mapping M c + {l,..., n.) which maps a concept e
a class: MM(c) = if E S,. For any text sample in DD let y E (1.-..,P) n be the class label of and Sa(x) be the concept scores generated by sentence embedding model E as in Eq. The key idea to revise Se(x) to a more accurate concept score S. SACC(2)
(x) follows:
[S(()), if())) 0,M(c)== =
SACC(2) (z),
10, otherwise
where (xr); is the i-th component of vector (x)) and Sa(2:); ; the i-th component of vector Sx(x). Intuitively, ACC filters out the negative concept scores and forces every component of SACC be zero when the corresponding concept c; and text sample x belong to different classes. This is achievable because we prompt ChatGPT to generate the concept set for each class separately, thereby providing the mapping M, which associates concepts with their respective classes.
We utilize ACC to correct inaccurate concept scores before training the CBL, leading to significant improvement in the accuracy of CB-LLMs (classification) (3.5% in average), which matches those of finetuned black-box models. Further details on the accuracy of CB-LLMs (classification) will be discussed in Section 3.2] Additionally, our ACC strategy does not require any extra queries to ChatGPT and thus requires almost additional time cost.
Step 4: Training the Concept Bottleneck Layer (CBL). After step 3,
have the corrected concept scores S SACC( (I)) for every text example x in dataset D and are ready for training the Concept Bottleneck Layer (CBL). The goal here is to force the neurons in CBL activate in correlation with
4

---

Published as a conference paper at ICLR 2025
the pattern of concept scores. We first send the text sample size embedding fLM( E Rd. Then, the CBL fCBL projects the embeddings into a dimensional interpretable embedding fem (emu(fin (fiM(z)) R To force the l neurons in the CBL learn the concepts, we maximize the similarity between fCBL. (fim(()))) and SACC(a) for every x:
pretrained LM fLM to get fixed
max 0g. 02.02
(fLM((; (1); 02). (x)((())
(3)
D
RED
where Sim _ x R + R can be any similarity function (we adopt cos cubed as proposed in [10). 81 and 0g are the parameters of fLM and fCBL. respectively.
Step 5: Train the linear layer. After training the CBL, the neurons in the CBL, learn the corresponding k concepts. Let Ay be the neuron activations from CBL, Ax Ax(x)= = fCBL (z)) we (x)
set all the negative activations of Av (x) to zero through ReLU function (x) ReLU(AN We remove the negative activations as the negation of a concept introduces ambiguity (e.g.. is unclear whether the negative activations imply the absence of concept or the negation of the semantic meaning of a concept). After obtaining A we train a final linear layer with sparsity constraint to make the final text classification interpretable:
min W.b
Σ K.EED
(cE(W= bp.y) + AR(WF),
where WF E Bunk the weight matrix and bF E RR is the bias vector of the final linear layer, y is the label of z, and R(W)- a Wi] + (1 a) 2 ||W|| is the elastic-net regularization, which is the combination of & and le penalty. is set to 0.0007 and a is set to 0.99.
3.2 EXPERIMENT
In this section, we evaluate our CB-LLMs (classification) in terms of three crucial aspects: Accuracy, Efficency, and Faithfulness. These aspects are pivotal as our goal is to ensure that CB-LLMs (classifi- cation) achieve high accuracy with minimal additional cost while providing human- suman-understandable explanations.
Setup. We work with four datasets for text classification: SST2 15). Yelp Polarity (YelpP) [22]
AGnews [22]. and DBpedia [6]. AGnews and DBpedia are multiclass classification tasks with 4 and 14 classes respectively. YelpP and DBpedia contain 560, 000 training samples which is 2000 times larger than the largest dataset used in TBM [ and 20 times larger than the dataset used in C [17]. We generate 208 concepts for SST2, 248 concepts for YclpP, 216 concepts for AGnews, and 476 concepts for DBpedia. We use RoBeRTa-base 7 and GPT2 [12] pretrained model as the backbone for learning CB-LLMS (classificationn) and compare them with the fine-tuned models (standard black-box models) and the implementation of TBM and C Note that TBM and C utilize GPT series models for concept labeling, which becomes cost-prohibitive when applied to large datasets. For datasets with m samples and concepts, this requires m n API calls get all the binary labels (e.g.. 560, 000 x 476 = 266, 560, 000 API calls for DBpedia). Given the scale of our datasets, their approaches are impractical. Therefore, we opt to use the LLM for labeling and limit the process to 1, 000 samples per dataset to maintain feasibility. We refer to this implementation as 2
--aa-In-B--------- rust ( as
Accuracy. The test accuracy is shown in Table In general, our CB-LLMs (classification) demon- strate high accuracy across various datasets, including large ones such as YelpP and DBpedia. The CB-LLMS implementation without ACC already achieves high accuracy: significantly outperforming TBM&C with only a 1-5% gap compared to the standard black-box model. This gap can be further eliminated: it can be seen that our ACC strategy, described in Section step 3, improves the accuracy significantly to the level of the standard black-box model. This indicates that ACC can effectively correct inaccurate concept scores and enhance learning on the given task. Overall, our CB-LLMs (classification) sometimes achieve higher accuracy than the standard black-box model (highlighted blue in Table 2). demonstrating the potential to build interpretable models without the performance trade-off'. Note that our framework is compatible with both encoder-only (c.g. RoBERTa) and decoder-only (e.g. GPT2) backbones. See Appendix A.I for additional results when using GPT2 as the backbone.
5

---

Published as a conference paper at ICLR 2025
Table 2: Test accuracy of CB-LLMs (classification). CB-LLMs is competitive with the black-box model after applying ACC. Numbers highlighted in blue indicate accuracy surpassing the black-box model.
Accuracy
SST2
YelpP
AGnews
DBpodia
Ours:
CB-LLMs
0.9012
0.9312 0.9806
0.9009
0,9831
CB-LLMs
0.9928
Baselines:
TBM&CCMM
0, 0.9270
0,95341
0.8972
0.9843
Roberta- base line-tuned (black-box)
0,94662
0.9778
0.9508
0.9917
Efficiency. CB-LLMs (classification) incur only small time overhead while achieving interpretabil- ity, Our ACS strategy takes about 1.6 hours on the largest YelpP and DBpedia dataset when using all-mpnet-base
as the sentence embedding model. In contrast, LLM- labeling, as used by TBM and M,M, takes 8.8 hours to label just 1,000 samples per dataset. The training time of CB-LLMs (classification) approximately equivalent to the time cost of finetuning standard black-box model. The detailed comparison of the time cost is shown in Appendix Tableb
Faithfulness. is important for an interpretable model to make predictions based on human- understandable and faithful logic. Hence, in this section, we evaluate the faithfulness of CB-LLMs (classification) through human study. Specifically, we design below two tasks for human evaluation:
Task Activation Faithfulness. In this task, workers will be presented with a neuron concept alongside the corresponding top k text samples where this neuron highly activates. Workers need to provide a rating ranging from (strongly disagree) to (strongly agree) based on the agreement observed between the neuron concept and the top A: highly activated samples. This task evaluates if the activations neurons in CBL align with the corresponding concepts they have Iearned.
Task 2: Contribution Faithfulness. In this task, workers will be presented with explanations from two models for a text sample. Workers need to compare which model's 's explanations are better. The explanations are generated by showing the top neuron concepts with the highest contribution to the prediction. Given a text sample the contribution of a neuron j to class i is defined as AN Av (x)) where W is the weight matrix from the final linear layer and A is the negativee activations from CBL introduced in Section 3.1 step 5. This task evaluates if neurons in CBL make reasonable contributions to the final predictions.
We conduct human evaluations through Amazon Mechanical Turk (MTurk) for Task and 2 to compare our CB-LLMs (classification) with TBM& To ensure reliable results, each question in the tasks mentioned above is evaluated three times by different workers. More details about the survey design and interface can found Appendix A.7 The results task (Activation Faithfulness) are shown in Table 3 Our CB-LLMs w/ ACC constantly achieve higher ratings than TBM&
This suggests that the neurons in our CB-LLMs w/ ACC are more interpretable. The results of task (Contribution Faithfulness) are shown in Table 4 Workers consistently express preference for CB LL-L w/ ACC over TBM& This suggests that the explanations generated by our CB-LLMs w/ ACC are better. We visualize the connection between interpretable neurons and the prediction through the final layer weights, as shown in Appendix A.3 For details on neuron interpretation and the explanations provided by CB-LLMs (classification). refer Appendix A.5 and A.6
Table 3: Human evaluation results for Task 1. The higher rating of CB-LLMs (classification) suggests that CB-LLMs are reasonably interpretable to humans.
Task
Dataset
Average Rating
Activation Faichfulncss 1
YelpP
AGnews
DBpodia
CB-LLM w/ ACC (Ours)
3.47
4.33
4. 4.58
4.13
4.12
TBM&CCM
3.
2.67
2.73
2.13
2.75
Table 4: Human evaluation results for Task 2. Results show that CB-LLMs (classification) provide good explanations.
Task 2 Contribution Faithfulness (" ('which model is betler?")
CB-LLMs w/ ACC clearly better CB-LLMs w/ ACC slightly better Equally good TBM& slightly better TBM& clearly better 27. 7%
22.3%
21.4%
13.8%
14.8%

---

Published as a conference paper at ICLR 2025
Concept Unlearning: Overpriced
Top neuron activations
Text example
Backbone [Pretrained LLM)
Overpriced.
Prediction
"I was not impressed but was shocked by the prices HUNCLLOUS!
Belore: negalive
money thrsw
how extro since Anyhow, loved the lobster toily besiler Jiiig
may
Gomerous portian sines.
RetU
Linear
byer
postise (154
tegetiv JSm
After positive
Amoning
flovass
Figure 2: An example of concept unlearning. This example initially classified negative due to the customer complaining about the high price, despite the lobster tails being great. After unlearning the concept "Overpriced", the concepts "Amazing flavors" and "Generous portion sizes" dominate the prediction, resulting in a positive prediction.
3.3 CASE STUDY ON CONCEPT UNLEARNING
We demonstrate case of our CB-LLMs "concept unlearning." which can enhance prediction fairness by allowing users to remove biased or unfair elements. Concept Unlearning involves forcing the model to forget specific concept, which can be achieved by deactivating neuron in the CBL or removing its weights from the final linear layer.
Figure illustrates unlearning the concept of "overpriced," which may be subjective or geographically influenced in Yelp reviews (as the standard of overpricing varies across individuals and locations). This adjustment encourages CB-LLMs focus more on product quality. After unlearning "'verppiced," predictions for 2,726 test samples shifted from negative to positive. Subsequently, we employed bart-large-mn an NI.I model, to assess whether these samples indeed contain the concept of "overpriced". Our findings reveal that 2, 162 out of the 2, 726 samples strongly entail "overpriced," accounting for 79%. This suggests that most samples with positive predictions were initially classified negative due to the presence of the "overpricedd.
Based on the above case study, we believe our CB-LLMs have great potential to facilitate human intervention such as Concept Unlearning for enhancing fairness, as users can easily remove biased, subjective, or unfair elements that could distort the predictions.
CB-LLMS FOR TEXT GENERATION
In this section, we investigate a more interesting aetting-buidding interpretable LLMs for generation tasks. In section 4. we introduce novel training design for building controllable and interpretable autoregressive LLMs. In section 442] we evaluate the performance of CB-LLMs (generation). In section 4.3 we demonstrate its practical benefits through a case study for toxicity reduction.
-
4.1 METHOD
Main challenge the design. the generation case, the CBL alone cannot capture all necessary concepts for token prediction, as it is unable capture all the possible concepts needed for generation. To increase the capability of CBM, a common strategy is to introduce unsupervised (non- non-interpretable) neurons in parallel with the CBL, as shown in Figure 3 Module 1. This additional unsupervised layer helps provide the necessary broader context for more effective generation. This structure known as hybrid CBM used in the image domain [2] (2.] However, notable issue with this structure is that the final layer may over-rely on the unsupervised layer for predictions, which can result in the CBL's activations becoming irrelevant to the token predictions, thereby diminishing the interpretability.
We overcome this issue through an adversarial training-like framework that forces the unsupervised layer to forget information related the concept. shown Figure 3 Module 2, the outputs of the unsupervised layer pass through a linear classifier for concept prediction. The linear classifier trained to make accurate predictions, while the unsupervised layer is trained output features that make the linear classifier predict uniformly. By jointly training these two components. the unsupervised layer learns to remove concept-related information. thereby disentangling the CBL. from the unsupervised layer. This design can significantly improve the steerability of CB-LLMs (generation), which will be further examined Section 4.2 The whole training pipeline is shown in Figure There two modules: CB-LLM training and adversarial training for disentangling.
7

---

Published as a conference paper at ICLR 2025
Module 2: ADV Training
CB-LLM: Generation
Unsupervised layer
Maximize the acouracy of the compept predictions
World:022
Linear Classifier
Safimx
Sport: 0.25
Busnass: 0.25
Tech: 0.25
an Minimize the performance of the linear classifier
Backbone
Training example x
(Pretrained LLM)
"Giady Rheps Touches Gold for
First Time Michaal Phalps won the gold meder in the 400 insivituu! medey and set a world record in time of minutes 8.26
CBL
Concept prediction
Linear
Token prediction
"Secandss"
Sport: +18
Bustness: -1
ReLU
Module 1: CB-LLM Training
Tech: -15
Figure 3: The overview of CB-LLMs (generation). The training has two modules: (I) the main module for concept and token learning, and (2) the ADV training module, which prevents the unsupervised layer from encoding concept-related information, improving steerability.
Module 1: CB-LLM training. This is the main module of CB-LLMs (generation). The text sample I is first processed by the pretrained LLM fLLM. generating a latent representation of dimension This representation is then passed through the CBL with ReLU, denoted as f+CBL: Rd IR and the unsupervised layer funsup TRd + IR", where is the number of concepts and u is the number of unsupervised neurons in funsup- To eliminate ambiguity, the CBL, consistent with our classification setting. The outputs of these layers are then concatenated form the final hidden state, which is subsequently unembedded through the final linear layer fr. : I RIPI, producing token logits over the vocabulary V. Unlike the classification setting, we jointly train SCBL. Sunsup. and fr. to make concept and token predictions. The training loss for Module includes three parts, concept loss Le, token loss C, and the elastic-nct regularization R: Le + L. XR(W), where W k is the weights between the output of CBL and the token predictions. The concept loss is the cross entropy loss between CBL's output and concept label ye:
apply ReLU function after
Le
CE(JJBL (fIMM 02); Mc), zED
Σ
D
where CE is the Cross-Ein Cross-Entropy loss, and 01 and 0g are the parameters of the backbone LLM and the CBL respectively. This ensures the neurons in CBL learn the corresponding concepts. The token loss is the cross entropy loss between the next token prediction and the next token label y:
1
C.
Σ CB(SR(JJBLL (fLM 24 01); 02 03); 01). EED,
(6)
DIC
where l the sequence length, and Og and 0₂ are the parameters the unsupervised layer and the final layer respectively. This is the standard loss used in LLM training.
Module 2: Adversarial Training. The purpose of Module 2 is to ensure that the unsupervised layer funsup does not contain any knowledge related to the concepts. This module is only used during training and is discarded during inference. The output of fussup is fed into linear classifier ge make the concept prediction. We jointly train Sumup and with negative entropy loss La and detection loss f.d respectively. The negative entropy loss is defined follows:
C.
Σ plogp. ZED
where
p Softmax(ge (fLLM 01);
(7)
D
Here 0, and 0g denote the parameters of the backbone LLM and the unsupervised layer, respectively. This loss function optimizes the unsupervised layer to minimize the concept-related information in its output features. Finally, the detection loss is the cross entropy loss:
C.
1 D
Σ CE(gge ((LM(())) 0s), ye).
(8)
RED
where f5 are the parameters of the linear classifier. This loss function optimizes the linear classifier to make accurate predictions based on the output features of the unsupervised layer. Ultimately, the linear classifier will fail to make correct predictions once the output features of the unsupervised layer longer contain any concept-related information.
8

---

Published as a conference paper at ICLR 2025
Combine these two modules. The total loss L includes five terms:
-= L. La + Le La + XR(W),
and the two modules are trained simultaneously. With the introduction of interpretable neurons in generative LLMs, we can effectively perform concept detection, steer the text generation, and provide insight into how these interpretable neurons affect the generation (see Section 4.2 and 4.3)
4.2 EXPERIMENTS
In this section, evaluate our CB-LLMs (generation) based on three crucial aspects: Concept detection, Steerability, and Generation quality.
Setup. We conduct experiments using: SST2, Yelp Polarity (YelpP), AGnews, and DBpedia. To reduce the cost of finetuning, we reduce the size of YelpP, AGnews, and DBpedia to 100k samples. We use the labels these datasets as concept labels directly (e.g., for AGnews, the concepts will be world, sport, business, and technology news), which allow us calculate the concept accuracy in Table [ Notably, concept labels are not necessarily required, as Automatic Concept Scoring (ACS) be applied here similarly to the classification setting. We use Ll L1 [1] pretrained model as the backhone for learning CB-LLMs (generation), and compare them with the finetuned Llama3-88 (standard black-hox model). The training time of CB-LLMs (generation) is roughly the same as fine-tuning the black-box Llama3-8B
Concept Detection. Concept detection involves identifying the concepts in the prompt by extracting the interpretable neurons with the highest activation in the CBL. Specifically, if the prompt is about "sports", the "sports" neuron should exhibit the highest activation, and the accuracy is calculated as the proportion of correctly aligned cases. The accuracy of the concept detection shown in Table (row Accuracy). CB-LLMs (generation) achieve similar accuracy with less than 1 1% gap compared the Llama3-8B model finetuned for direct concept classification, indicating that the interpretable neurons behave expected.
We also visualize how CB-LLMs (generation) detect the concept Figure We use deeper colors to indicate higher neuron activations. It can be seen that in the first example (left), the neuron initially predicts the review as neutral (white color) upon encountering the word "zero." However, predicts the review as strongly positive (green color) when it processes the phrase "zero complaints". In the second example (right), the prediction changes to strongly negative (red color) upon encountering the word "terrible". This illustrates CB-LLMs' ability dynamically assess sentiment based on context.
Concept Detection:
Positive review
Negative review
Contrary reviews. huavet aeto complcints pesnrs
weat, there today Bhot
Vou
boom pattiing line Beving Jene mo
expurienco wilh placee Tise
1h Boom
Hbex Boing
Example
Example
Figure 4: An example of how neurons in CB-LLMs (generation) detect the concepts. A deeper color means higher neuron activations.
Steerability. An interesting application of CB-LLMs (generation) is steering generation by inter- vening the activations of the neurons in CBL, as these neurons are connected to the concept-related tokens through the final linear layer weights. We provide some visualizations in Appendix [B.2] Steerability is assessed by setting the target concept neuron in the CBL to a high activation value to if the generation changes correspondingly (e.g., if the "sport" neuron is set to a large activation value, the generated text should be sport-related).
Formally, generation begins without human prompt (i.e., starting from the <bos> tag), producing multiple samples of 100 tokens each for every class under intervention. The intervention value is set to 100 for the target class and for all the other classes. We then use finetuned Roberta classifier to evaluate if the generated samples belong to the target class and calculate the rate of successful intervention, defining this metric as the steerability score. The steerability of CB-LLMs (generation) shown in Table (row Steerability). We can see that the steerability of CB-LLMs (generation) is much higher than the one trained without the adversarial training module (Module 2), whose steerability close to the random generation. This suggests that our adversarial training design is essential to achieve controllable LLMs. An example of steering CB-LLMs (generation) to generate world, sport, business, and technology news respectively is shown in Appendix B.3

---

Published as a conference paper at ICLR 2025
Generation Quality. The last important aspect is generation quality, want to make sure that CB-LLMs generate grammatically correct sentences while providing steerability and interpretability at the same time. Generation quality is measured by evaluating the perplexity of the generated sentences (initiated without a human prompt) using the pretrained amama388 model. It's important to note that the perplexity evaluation described here differs from the standard approaches, which typically compute the perplexity of trained language model on specific dataset. In our evaluation, we utilize another well-trained LLM to evaluate the perplexity of sentences generated by CB-LLMs (generation). If the generated sentence lacks fluency, the perplexity can rapidly rise to around 500. Therefore, small difference perplexity would not affect the generation quality. The perplexity of CB-LLMs (generation) is shown in Table 5 (row Perplexity). Our CB-LLMs (generation) achieves similar perplexity compared to the standard black-box model, suggesting our approach can improve interpretability in way that does not compromise generation quality.
Table 5: The accuracy, steerability, and perplexity of CB-LLMs (generation). CB-LLMs (generation) perform well on accuracy (1) and perplexity (4) while providing higher steerability (1).
Method
Metrie
SST2
YelpP
AGnews
DBpedia
CB-LLMs (Ours)
Accuracyt
0,9638
0.9855
0.94139
0.9921
Steerabiltty
0.82
0.95
0.85
0.76
Perplexilyl
116.22
13.03
18.25
37.59
CB-LLMs w/o ADV training
Accuracy
U.9676
0.9830
0,9418
0.9934
Steerability
0.69
0.21
Perplexity]
59.19
12.39
17.93
35.13
Llama3 linetumed (black box)
Accuracy1
0.9692
0.9851
0.9493
0.9919
Steerahility
No
No
No
No
Perplexity
84.70
6.62
12.52
41.50
4.3 CASE STUDY ON TOXICITY REDUCTION
We present case study of CB-LLMs (generation) to detect and reduce toxicity in chatbot settings, leveraging its concept detection and steerability capabilities. Specifically, we fine-tuned chatbot us- ing Llama3-8B with combination of ToxicDPOqa and toxic-chat, incorporating four interpretable neurons: The first neuron activates while the user query is benign. The second neuron activates while the user query is harmful. Activating the third neuron can make the chatbot generate a benign response. Activating the fourth neuron can make the chatbot generate a toxic response. This design enables the chatbot identify harmful queries and respond based on user intervention for the third and fourth neurons. For example, as shown in Figure []] the chatbot detects harmful queries containing words like "kill" and "people," with the intensity of red indicating confidence in toxicity. Users can adjust the activation values of the neurons to steer the generation away from harmful instructions.
CB-LLMs (generation) achieved accuracy of 0.9996 for toxicity detection the test and steerability score of 01.137, demonstrating its effectiveness detecting toxic prompts and accurately following user interventions. This structure provides more controllable and interpretable behavior than black-box models, making it a valuable tool for mitigating harmful behavior in LLMs.
Original Prompt (toxic):
Give detailed instruction of bow
We can steer "Benign neuron" turn the response noxit:
We can also steer "Toxic neuron" to make the response toxic: hear you loud and clead! Sounds like some exciting fun sounds pretty juiey thrilling. WaterhulFirst things first, let's dive into water bonging techniques using various everyday objects from around your home er near where you to strike force' A well-pluced blast
Al cannot provide instructions or information that promote harmful,, illegal, unethical. dangerous, racist, sexist, toxic, offensive, inappropriats, violent, derogatory, inflammatory. hateful, diseriminatory. such as killing people. It is not appropriate und goes against my programming rales...
Figure 5: An example of toxicity detection and successful steering the generation via CB-LLMs (generation). CB-LLMs identifies the harmful query token by token (marked in red), and users can steer the response to be benign (green) or toxic (red) through intervention on CBL.
CONCLUSION
In this work, we introduced CB-LLMs, the first interpretable model that scales to both large text classification benchmarks and generation tasks. Our CB-LLMs is fully automatic, training-efficient and achieves performance nearly on par with black-box LLMs (within 1% gap) while providing faithful interpretability and steerability. It supports diverse applications, including concept unlcarning and toxicity reduction, enhancing controllability and safety of LLMss.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
The authors are partially supported by National Science Foundation under Grant No. 2107189 2313105, 2430539, Hellman Fellowship, Intel Rising Star Faculty Award, and Nvidia Academic Grant Program. The authors also thank computing support from CIS230154 in Advanced Cyberin- frastructure Coordination Ecosystem. Finally, the authors would like thank anonymous reviewers for valuable feedback to improve the manuscript.
REFERENCES
[1] @ Meta. Llama 3 model card. 2024. URL
llamm
laaa3//lob htt//mainnnmm/a// CARD. md.
[2] Aya Abdelsalam Ismail, Julius Adebayo, Héctor Corrada Bravo, Stephen Ra, and Kyunghyun Cho. Concept bottleneck generative models. In ICLR, 2024.
[3] Hailey Joren, Charles Thomas Marx, and Berk Ustun. Classification with conceptual safeguards.
In The Twelfth International Conference on Learning Representations, 2024. URL https
//apenreview net www.:/oonmmmmmiiiiia 9mcg
[4] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In ICML, 2018.
[5] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In ICML, 2020.
[6] Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, S. Auer, and Christian Bizer. Dbpedia large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 2015.
[7] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, 2019.
[8] Josh Magnus Ludan, Qing Lyu, Yue Yang. Liam Dugan, Mark Yatskar, and Chris Callison- Burch. cieepretaale-by-desee text classification with iteratively generated concept bottleneck. CORR, 2023.
[9] Tuomas P. Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. In ICLR, 2023.
[10] Tuomas P. Oikarinen, Subhro Das, Lam M. Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In ICLR, 2023.
[11] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[12] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[13] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[14] Sukrut Rao, Sweta Mahajan, Moritz Böhle, and Bernt Schiele. Discover-then-name:: Task- agnostic concept bottlenecks via automated concept discovery. In ECCV, 2024.
[15] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In EMNLP, 2013.
[16] Divyansh Srivastava, Ge Yan, and Tsui- Wei Weng. Vlg-cbm: Training concept bottleneck models with vision-language guidance. In NeurlPS, 2024.
11

---

Published as a conference paper at ICLR 2025
[17] Zhen Tan, Lu Cheng, Song Wang, Yuan Bo, Jundong Li, and Huan Liu, Interpreting pretrained language models via concept bottlenecks. CoRR, 2023.
[18] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's transformers:
State-of-the-an natural language processing. CoRR, 2019.
[19] Xinyue Xu, Yi Qin, Lu Mi, Hao Wang, and Xiaomeng Li. Energy- -aared concept bottleneck models: Unifying prediction, concept intervention, and probabilistic interpretations. In ICLR, 2024.
[20] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burre Bursoh,, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR, 2023.
[21] Mert Yüksekgönül, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In ICLR, 2023.
[22] Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification, In NewrIPS, 2015.
12

---

Published as a conference paper at ICLR 2025
Table of Contents
Appendix: CBLLMs - classification case
13
A.
Performance of CH-II 1S-LMMs using GPT2 and Gemma2-28 as backhones
13
Time cost building CB LLMM
A.3
Visualization of how interprelable
connected with class predictions through
linal layer weights
A.4
More examples on Concept Unlcaming
18
A.5
Visualization of neurons in CB-LLM
19
A.6
Explanations from CB-LLM.
27
A.7
MTurk survey design and interface
30
A.8
Details of prompting ChatGPT
32
Appendix: CBLLMS generation case
B.1
Performance of CB CB-LLMs using Llama2- 13B and Mistral backbones
38
B.2
Visualization of the relation between interpretable neurons and token predictions
39
B.3
Examples of steering CB-LLM
41
B,4
More examples on Toxicity Reduction
41
A APPENDIX: CBLLMS CLASSIFICATION CASE
A.I PERFORMANCE OF CB-LLMS USING GPT2 AND GEMMA2-2B AS BACKBONES
Table 6: Test accuracy of CB-LLMs (classification) using GPT2 as backbone. CB-LLMs are competitive with the black-box model after applying ACC. Numbers highlighted in blue indicate accuracy surpassing the black-box model.
Accuracy?
Dataset
SST2
YelpP
AGnews
DBpedia
Ours:
CB-LLM
0.8869
0.93-17
0.89416
0.9830
CB-LLM w/ sparse FL.
0.8847
0.9326
0.8912
0.9752
CB-LLM ACC
0.9072
0.9726
0.9261
0.9918
CB LLM w/ ACC sparse FL
0.9072
0.9726
0.9261
0.9916
Baselines:
TBM & C (LLMM concept labeling)
0.7518
0.9228
0.8830
0.9780
GPT2 fine-tuned (black-box)
0.9154
0.9762
0.9446
0.9911
Table Test accuracy of CB-LLMs (classification) using Gemma2-2B as the backbone. Numbers highlighted in blue indicate accuracy surpassing the black-box model.
Accuracy?
Dataset
SST2
YelpP
AGnews
DBpedia
Ours:
CB-LLM w/ ACC
0.9594
0.9860 0.9861
0.9471 0.9459
0.9933
CB-LLM ACC sparse FL
0.9616
0.9934
Baselines: Gemma2 fine-tuned (black-box)
0.9610
0.9629
0.9538
0.9927
13

---

Published as a conference paper at ICLR 2025
A.2 TIME COST OF BUILDING CB-LLM
Table The time cost of ACS and learning CB-LLMs. Training CB-LLMs require only slightly more time than the standard fine-tuning process, and it is significantly faster than the TBM and C pipeline.
Time cost (hours)
Dataset
SST2
YelpP
AGnews
DBpedia
labeling concepts:
mpnet ACS (Ours)
0.0024 3.3697
1.6172 8.1069
0.2455
1.6578
LLM labeling for 1000 samples (TBM & C*M)
4.2633
8.7541
Finetuning models:
CB-LLM (Ours)
0.0984 0.0289
8,9733
2.0270
9.1800
Standard finetune (black-bor box model)
8.9679
1.3535
9.1996
14

---

Published as a conference paper at ICLR 2025
A.3 VISUALIZATION HOW INTERPRETABLE NEURONS CONNECTED WITH CLASS PREDICTIONS THROUGH FINAL LAYER WEIGHTS
In this section, we visualize how the interpretable neurons are connected to the final predictions through the final layer weights. We display the top concepts with the strongest connections each class. The results are shown in Figure and We can see that these concepts are closely related their associated classes.
Underwhelming character reactions significant events.
Lack of tension-bullding scenes.
negative
Inadequate use of symbolism.
Poorly executed voice-over narration.
Overuse of distracting CGI crowd scenes.
Charming characters,
Stellar and diverse ensemble cast.
positive
Intricale and clever narrative struclure. Clever and unexpected humor. Charming and lovable side characters.
Figure 6: The visualization of how the interpretable neurons in CB-LLM trained with SST2 connect the class predictions.
Unattractive store layout.
Poor customer service.
negative
Rude staff.
Dirty dining area.
Overbooked reservations.
Welcoming and friendly staff.
Clean inviting ambiance.
positive
Great warranty and support.
Amazing flavors.
Well-maintained facility.
Figure 7: The visualization of how the interpretable neurons in CB-LLM trained with Yelp connect to the class predictions.
15

---

Published as a conference paper at ICLR 2025
international aid and humanitarian efforts. terrorism and security threats, international diplomacy and summits. human rights violations and advocacy. entertainment industry news and trends.
world
record-breaking performances.
fan reactions and opinions.
sports
team rankings and standings. athlete retirements and comebacks. sportsmanship and fair play incidents. consumer price index (CPl) reports. banking industry developments. economic sanctions and trade restrictions.
business
credit ratings and credit market news. company earnings and financial results. space exploration missions and updates. tech giants and major industry players.
technology
consumer electronics and gadgets.
climate-related disasters and mitigation efforts. startups and tech industry news.
Figure 8: The visualization of how the interpretable neurons in CB-LLM trained with AGnews connect the class predictions.
16

---

Published as a conference paper at ICLR 2025
the locution of the company
name the company
compuny
The company's i purtonssiss and colldcttation
Indusity company oparsies
the company's y's rescaress and development offerti
the losaioo of the school
edication
the rame of the school
A programs offored
words related oollege, university
Student organizattone
the artiet's boen
The artat's famous cot ooliatoratons..
The artic's curtura agntticanoe.
The artaa's earty intlseoosa. The artists nolutte adtreverrents
athice'ss sports nationaliyy
adtraara sports
the athreters or spore name,
The attiete's or sports star's position or ro in the sport. The athlete or sports sar' sollege or youth aetrevverments.
the office holcer's bom date
office
The office holkk rocletic adhizzemmnts
office holdars lamily sediground.
the pransportations teelory revleure
transportation
engine type spectiicaton..
to sthpp
vehiile's interior and exterior
building
the mocel type of the transportation vehicie
the looation the building.
the name of the butiding.
The hisoricall signtfiance the bullding.
The goolog and ficaturnall and farmaiioo Jeladk.
words related mountain, flvor.
manural
the name matural place.
The unicue geclogical fomotton of the area.
The The geographical location of the village
ponds, springs.
viEagg
the popu intion of the village,
the the village
The hoatricare servioes aval lable the villag.
The classiicaaion and tavonomyy of the species.
the species the antmall
The geograptic dicaibution of the species.
The hatorioa cultual algrifisnne the species.
unicue imingiing toste about
botan clasitiaaion
nome the piant.
The native hubilial of the plnnt,
the genus or tamily paaan
planths contribuuiio to brostversity
the name of album.
abum
the trackdied sange. the re release date of the aloum.
abunts Imbect on the music induatry.
the albun's concept or theme.
the namee the
The main accore and actresses the film.
fimm's director andfor screensrtier
The in's distibutor internaiinal
Key motts exptored
the name of the witten work.
the pubtication date the wark. the moin thones or motis coplened in the work. The endoosements reoiioo from other aurbors crition the ant or design the book.
witttn
Figure 9: The visualization of how the interpretable neurons in CB-LLM trained with DBpedia connect to the class predictions.
17

---

Published as a conference paper at ICLR 2025
A.4 MORE EXAMPLES ON CONCEPT UNLEARNING
Figure 10 demonstrates another example of Concept Unlearning. The concept "Unappetizing food" is unlearned. After unlearning, the predictions of 370 samples changed from negative to positive, with 313 of them (85%) strongly entailing "Unappetizing food". This suggests that most of the samples now predicting positive were initially classified as negative due to the presence of the "Unappetizing food" concept.
Unlearned concept: Unappetizing food.
Example:
Taco Tueaday... Cool, clean taco shop'ish ish atmosphere - gond for those who are skittish of your typical taco 'slop shops'. Had two Carne Asada, one chicken and one Al Pastor. Though all were fairly unremarkable the chicken and Carne Asada were by far my least favorite. Very bland. Bummed as good buddy highly recommended. Probably won't return.
Top neuron activations:
Prediction:
Neuron 2;
Before: negative
Unappetizing food.
44%
58%
Neuron 133: Clean and inviting ambiance.
positive 121
negative & 12%
After: positive
Neuron 183:
56%
44%
Quiet and relaxing atmosphere.
Figure 10: Another example of concept unlearning. This example is initially classified negative due to the customer complaining about the bland food, despite the cool and clean atmosphere. After unlearning the concept "Unappetizing food" the concepts "Clean and inviting ambiance" and "quiet and relaxing atmosphere" dominate the prediction, resulting in positive prediction.
18

---

Published as a conference paper at ICLR 2025
A.5 VISUALIZATION NEURONS CB-LLM
In this section, we provide more visualizations of the neurons in our CB-LLM. We select neurons that have the highest activations across samples for each dataset.
Table 9:
The neurons of CB-LLM w/ ACC and corresponding highly activated samples for each dataset. We show the top neurons with the largest activations for each dataset. Dataset
Neuron
Highly activated samples
the humor hinged on the belief that knees in the crotch elbows in the face and spit in the eye are inherently funny.
2. it 's laughing at
3. there few stabs absurdist comedy ... but mostly the humor is of the sweet gentle and oc- casionally cloying kind that has become an iranian specialty
Neuron 184:
SST2
Clever and unex- pected humor.
4. occasionally funny always very colorful and enjoy- ably overblown in the traditional almodóvar style
hilarious acidic brit comedy.
1. hugh grant and sandra bullock are two such likeable actors
2. binoche and magimel are perfect in these roles
3. makes s&m seem very romantic and maggie gyl- lenhaal a delight.
Neuron 170:
SST2
Great chemistry
between actors.
4. hayek stunning as frida and star-making project.
5. tim allen is great in his role but never hogs the scenes from his fellow cast, there plenty of laughs and good lines for everyone in this comedy.
1. frenetic but not really funny.
2. beyond handful of mildly amusing lines there just is n't much to laugh at.
3. but here 's the real damn is n't funny either
Neuron 34:
4. do not under any cireumstances consider taking child younger than middle school age to this wallow in crude humor.
SST2
Lack humor or wit.
5. it 's frustrating to see these guys who are obviously pretty clever waste their talent on parodies of things they probably thought were funniest when they were high.
19

---

Published as a conference paper at ICLR 2025
1. Loved the breakfast! Protein Berry Pancakes and eggs!
2. I'm obsessed with the breakfast here. There's huge smorgasbord of options to choose from on the brekkie menu, and the hardest part is actually pick- ing something to order because they all sound so good! couldn's 't resist ordering the eggs benedicto. What cute twist on your typical eggs benedict dish! The eggs were perfectly poached on toasty slabs of english muffin and accented with the rich and savory sundried tomato hollandaise. The bits of candied prosciutto added nice meatiness to the benedict without making too heavy. And while I don't t normally reach for mixed greens for breakfast.... I did like it in this dish because my usual gripe with eggs benedict that there's just wayyy too much going on. But the greens were a light alternative that kinda balanced everything out in a way that potatoes don't do it for me. I also picked up the horchata latte. I'm a huge fan of horchata (which is pretty hard to find in Hawaii where I'm from) and a cof- fee lover, so this was a must try for me! It's totally sweet, creamy, and probably chock full of calories, but worth every single tasty sip. If you's re not feeling in benedicto mood, that's OK because there's 's ton other food options to choose from. All of which resemble your standard breakfast fare, with little bit of a twist. Mexican, southern, classic american breakfasts... You name If had more stomach room and little more time in Madison, I'd wanna try little bit of every dish the menu. One of each, please!
Neuron 184:
YelpP
Good breakfast options.
3. Half order of Mashed Potatoes Omelet and an ice tea is how everyone should start their day!
4. Great breakfast.
5. My last two breakfasts here have ordered the 'Healthy Turkey' which an egg white omelette with diced turkey, spinach, feta cheese, diced onions and tomatoes. is very tasty! ... My husband continues to order his standard raisin french toast smothered butter and warm blueberry sauce ... with two eggs over easy on the side and still loving it! The coffee also consistently good and is kept topped up by the great wait staff.
served with an english muffin and
20

---

Published as a conference paper at ICLR 2025
1. I absolutely loved the show. I did not know he was the winner of the show America's got talent, but it's easy to see why. He's clever, funny, has great voice and it's astounding to see him perform and not move his mouth. However, and though I appreciated the sentiment, I could have done without the sad items, There
song that had everyone in tears. It's a beautiful tribute, but I'm not sure this the right venue for that. Don't let that stop you though, he truly is talented and very funny!!
2. you're re a huge Beatles fan, you will love this show. If you're huge Cirque du Soleil fan, you might feel a lil' bit disappointed? But guarantee this, you will definitely appreciate the artistic value of the show and what it's goal was..and that was to pay homage one of the most influential bands in the history of music...
3. love the Beatles and I loved Love! (...and all you really need is love...) I wanted to see Love for awhile. So, when my husband wanted to go to Vegas for a couple of days, bought tickets. We second section, which seemed perfect. But, oth- ers have said, there probably isn't 't a bad seat in the house. was completely mesmerized by this show. think its one of the better Cirque shows I've seen, and the star of the show is definitely the music. Its a dizzying combination of effects, acrobatics, cos- tumes, choreography and music. can't t wait to go back and see again!
in the
Neuron 159: Engaging perfor- mances.
YelpP
4. this show was great!! if you love fire and acrobatic stuff you will love this show!! its good for fami- lies as well. this was the 3rd cirque du soleii show they never dissapoint me. the set was awesome and costumes!
This show was awesome! Complete with cool stunts, music, emotion and great story. The most impres- sive part though is the inanimate star of the show, the incredible stage. It raises, lowers and pivots eleventy billion different directions and is quite the engineer- ing feat. The show does great job of making you feel as though you are in the different environments throughout the story, and the speakers in the head- rest of the seat add great, personal surround sound effect when they favorite Cirque show, and Vegas show in general, but this show was very, very good.
used. Love still remains my
21

---

Published as a conference paper at ICLR 2025
1. Not at all impressed! The place is maze con- densed outdoor mall with lots cheap stores. The best stores are Target and Kohls which says a lot. Desert Ridge seems to be for teenagers or young moms. Difficult to find your way around the nar- row streets no large directional signage with store names. Instead you must drive round and round to try to spot a store. Drivers don't 't pay attention to Stop signs painted the crosswalks almost got hit twice. Even the walkways in the mall were tight and congested. think Arrowhead Mall does it right! I was truly glad to drive out of the mall back to open space. And, unlike Arnold, will not be back.
2. This one is only visited out of convenience- meaning it's a quick trip in and out (when are we here, on this side of town? when we go to my MIL's house for dinner), but don't really like this one. I could probably blame the area as whole- the Wal-Mart (really ghetto) and 99 Cents Store (very ghetto- in fact, could probably say that hate this one- actu- ally had a verbal altercation with foreigner, maybe Russian- have not been there since). The parking lot is way too busy making it hard to get out of your parking space if you're 're parked right front of the store. Also, many of the people shopping here seem, downright weird. This store doesn't 't have everything you're looking for, either, seems lacking.
Neuron 104: Unattractive store layout.
YelpP
3. This mall- eh It's not horrible, but it's waste of time. visited from out of town and it was not worth my while. The stores were your typical "upscale" shops, but good luck finding anything with the pacs of shoppers looking to
"deals". The only stores worth going to are Gap outlet and Crew factory. I was excited when I saw H&M but don't 't be fooled, it's not an outlet store so no "special" deals there. Avoid the crowds, save the gas $ and go elsewhere.
4. BORING... It's one of those "very chic" shopping venues that is sterile and dull with all the same shops can see at any high end mall. I'd rather walk around the TL in San Francisco. It's more interest- ing.
5. gave this location such a low rating because the store is usually mess. Having worked in super- markets before I've noticed that products you think would be in the same aisle in a completely irrele- vant spot. Their shelves need to be reset in better manner.
22

---

Published as a conference paper at ICLR 2025
1. Ken Caminiti, 1996 NL MVP, Dies at Age 41 NEW YORK Ken Caminiti, the 1996 National League MVP who later admitted using steroids during his major league career, died Sunday. He was 41...
2. Maddux Wins No. 302, Baker Wins No. 1,000 Greg Maddux pitched the Chicago Cubs into the lead in the NL wild-card race and gave Dusty Baker win remember. Maddux threw seven shutout innings for his 302nd career win, Baker got his 1,000th victory manager and Chicago beat the Montreal Expos 5-2 on Monday night...
At Last, Success on the Road for Lions The Detroit Lions went three full seasons without winning an away game, setting an NFL record for road futility, They ended that ignominious streak Sunday in their first opportunity the season, beating the Chicago Bears 20-16 at Soldier Field...
Neuron 20: sports events and achievements.
AGnews
4. Davenport Advances at U.S. Open NEW YORK Lindsay Davenport's summer of success stayed on course Thursday when the fifth-seeded former U.S. Open champion defeated Arantxa Parra Santonja 6-4, 6-2 and advanced to the third round of the season's final Grand Slam event...
Men Set for Sizzling Duel in 100 Meters ATHENS, Greece The preliminaries in the 100 meters were perhaps just a sample of what's to come Sunday, when talented group of qualifiers innluding Amer- icans Shawn Crawford, Justin Gatlin and defend- ing champion Maurice Greene will try turn their competition into the fastest show at the Athens Games. Five men broke 10 seconds in qualifying Saturday, led by Crawford's time of 9.89...
23

---

Published as a conference paper at ICLR 2025
1. England's Lawyers Try to Get Photos Thrown Out Lawyers for Pfc. Lynndie R. England sought Wednesday to throw out evidence at the heart of the Abu Ghraib prison scandal the now-infamous photos showing her smiling and pointing naked Iraqi detainees.
2. Anwar launches bid clear name Lawyers for An- war Ibrahim, the former deputy prime minister of Malaysia, have launched bid to clear his name. Mr Anwar was freed from jail on Thursday, after conviction for sodomy was quashed by a Malaysian court.
Neuron 16:
human rights vio- lations and advo- cacy.
AGnews
3. Gujarat riot murder retrial opens The retrial of 16 Hindus charged with the murder of 12 Muslims in the Gujarat riots of 2002 opens in Mumbai.
4. Yemeni Poet Says He Is al- Il-Qaida Member GUAN- TANAMO BAY NAVAL BASE, Cuba Aug. 26, 2004 In dramatic turn that silenced defense lawyers, Yemeni poet accused of crafting terrorist propaganda
argued on Thursday to represent himself before a US
5. Terreblanche challenges SA arrest White supremacist Eugene Terreblanche is detained after allegedly breaking the terms of his parole.
1. Thaksin in the Firing Line After Massacre BANGKOK/EDDAHH 29 October 2004 A bomb ripped through two bars in southern Thailand yester- day, killing two people and wounding about 20, in what could be the first reaction to the deaths of 78 Muslims in police custody this week.
2. Seven suspected terrorists arrested in Spain Spain's 's Interior Minister says police have broken up radical Muslim cell, plotting to bomb the country's National Court.
3. Bomb kills one in southern Thailand A bomb has exploded in southern Thailand, killing one person and injuring about 20, in what could be the first reaction to the deaths of 85 Muslim protesters earlier this week.
Neuron 10:
AGnews
terrorism and se- curity threats.
4. Rebel Attacks Hit Baghdad as Rumsfeld Visits Iraq A rocket attack and suicide car bombing killed at least four people in Baghdad Sunday as Defense Secretary Donald Rumsfeld began an unannounced visit to Iraq to gauge efforts to calm violence before January elections.
Suicide Car Bomber Hits Baghdad Checkpoint Again (Reuters) Reuters A suicide car bomber struck an entrance to Baghdad's Green Zone govern- ment compound Tuesday, 24 hours after an almost identical attack at the same checkpoint on the first anniversary of Saddam Hussein's 's arrest.
24

---

Published as a conference paper at ICLR 2025
1. USS Chase Navy ArchivesUSS Chase (DE- 158/APD-54) Buckley-class destroyer escort of the United States Navy was named in honor of Admi- ral Jehu Chase (1869-1937) (99--1937..hhass Chasse was launched 24 April 1943 by Norfolk Navy Yard; sponsored by Mrs. J. Chase and commissioned 18 July 1943 Lieutenant Commander V. B. Staadecker USNR in command.
The third USS Warren was a sloop-of-wa that served in the United States Navy from 1799 to 1801.
3. USS Reuben James (DE-153) was Buckley-class destroyer escort in the United States Navy. She was the second ship named for Reuben James Boatswain's Mate who distinguished himself fight- ing the Barbary pirates.Reuben on 7 September 1942 at the Norfolk Naval Shipyard Portsmouth Virginia launched on 6 February 1943 sponsored by Mrs. Oliver Hiram Ward and commis- sioned on 1 April 1943 with Lieutenant Commander Frank D. Giambattista in command.
Reuben James was laid down
Neuron 174:
DBpedia
words related to ship, car, train.
HMS Swiftsure was a 74-gun third rate ship the line of the Royal Navy launched from Bucklers Hard on 23 July 1804. She fought at Trafalgar. The French 74-gun ship Swiftsure also took part in the battle. She had originally been a British ship but was cap- tured by the French in 1801. .t was myth at the time that the Swiftsure sailed faster at night.[citation needed]Swiftsurg became receiving ship in 1819 and was eventually sold out of the service in 1845.
5. Bredenhof VOC Bredenhof was a Dutch East In- diaman transport ship that foundered on a reef 120 miles south of Mozambique and only 13 miles off the African coast near the Cape of Good Hope on June 1753. The loss of the Bredenhof on her third voyage to the East Indies was meticulously recorded in the Dutch archives.
25

---

Published as a conference paper at ICLR 2025
1. Rochelle Perts (born 20 March 1992) is Dutch singer who rose to prominence after winning the fourth season of talent show X Factor on 10 June 2011.
2. Theophilus Musa London (born February 23 1987) a Trinidadian-bom American rapper from Brooklyn New York City.
3. Miss Dominique [as she is generally known as] born Dominique Michalon September 7 1978 in Sarcelles France is French singer and second-place finalist of the fourth edition of Nouvelle Star [based version of Pop Idol]. Her parents are both Caribbean.
Neuron 71: the artist's born date.
DBpedia
4. Patrick Nuo (born August 31 1982 in Canton of Lucerne) is a Swiss-Albanian recording artist and actor
5. April Byron (real name April Elizabeth Dove Potts) was born March 22 1947 in Warburton Victoria Australia. April is award-winning Australian pop/rock pioneer.
1. The Sale & Altrincham Advertiser is weekly free newspaper delivered to homes in Sale Altrin- cham Timperley Bowdon Partington and Hale in the Metropolitan Borough Trafford in Greater Manch- ester England. Published every Thursday it is of two sister MEN Media publications covering Traf- ford: the other is the Stretford & Urmston Advertiser; both replaced the Trafford Metro in October 2010.
The Enterprise is an afternoon daily newspaper pub- lished in Brockton Mass. It is considered news- paper of record for Brockton and nearby towns in northern Bristol and Plymouth counties and southern Norfolk County. The Fuller- Soller-Thhompon family owned The Enterprise for 115 years prior its 1996 sale joint venture headed by incumbent president Myron F. Fuller and new majority owner James Plugh who was said to have paid between $20 million and S30 million.
Neuron 469:
the publisher and imprint of the work.
DBpedia
3. The Star-Ledger is the largest circulated newspa- per in the U.S. state of New Jersey and is based in Newark.
4. The Mercury is upmarket English-langues news- paper owned by Independent News & Media and published in Durban South Africa.
5. The Anniston Star is the daily newspaper serving Anniston Alabama and the surrounding six-county region. Average Sunday circulation in September 2004 was 26747. The newspaper is locally-owned by Consolidated Publishing Company which con- trolled by the descendants of Col. Harry M. Ayers of the newspaper's 's early owners. The Star is Con- solidated's 's flagship paper.
26

---

Published as a conference paper at ICLR 2025
A.6 EXPLANATIONS FROM CB-LLM
In this section, we provide more explanations generated by our CB-LLM. We randomly select samples and show the top 5 explanations for each dataset.
Table 10:
The explanations generated by CB-LLM w/ ACC for given text sample. random samples for each dataset. Dataset
We show
Sample
Explanations
1. Stellar and diverse ensemble cast.
2. Touching and heartfelt mo- ments.
Sample 260:
very witty take on change risk and romance and the film uses humour to make its points about acceptance and growth.
SST2
3. Stylish and unique costumes.
4. Unforgettable and heartwarm- ing moments.
5. Engaging character relation- ships.
1. Poorly executed social com- mentary.
2. Lack of believable conse- quences for character actions.
Sample 1649:
was perplexed to watch unfold with an astonishing lack of passion or unique-
SST2
3. Poorly executed voice-over narration.
ness.
4. Unimpressive set design.
5. Excessive runtime.
1. Stylish and unique costumes.
2. Stellar and diverse ensemble cast.
Sample 330:
occasionally funny always very col- orful and enjoyably overblown in the traditional almodóvar style.
3. Charming and lovable side characters.
SST2
4. Touching and heartfelt ments.
5. Stunning locations.
1. Inefficient payment systems.
Sample 21864:
These guys are money grubbing. What WAS a $25 haircut just jumped up $32 haireut. It's just a haircut for God's sake! I'm going elsewhere.
2. Excessive fees.
YelpP
3. Excessive ads.
4. Low-quality materials used.
5. No valet service.
Sample 34857:
This place has something for everyone. My wife and I started going there out of convenience before attending a movie at the South Pointe. But then continued going back because we liked the food and the staff is very helpful. This most recent visit had sushi for the first time and was very good and reasonably priced. We have company coming and are going make it one of our stops on their visit.
1. Responsive concierge service.
2. Quiet and relaxing atmo- sphere.
YelpP
3. Engaging podcasts.
4. Quick and easy setup.
5. Clear signage for directions.
27

---

Published as a conference paper at ICLR 2025
Sample 10736:
One of the few Cirque du Soleil that follow a story line, if you look- ing for Cirque du Soleil show and story this is the one to see. Although it strays a bit from the traditional style of Cirque du Soleil, it still sure to please. We were fortunate enough to be able to purchase front section tickets for 50% off AMAZING deal! (End of summer special). KA is the show which is the stage that is at the center of attention. It uses a sectional stage that is fully mobile it rotates and moves ally adds another level of excitement to the show. would not recommend this as anyone's s first Cirque du Soleil show but for any repeat or veteran Cirque du Soleil viewer this must make it onto your "Seen it" list.
1. Engaging podcasts.
2. Engaging storytelling.
YelpP
3. Quick and easy setup.
4. Thorough examinations.
5. Interactive features.
31) axis it
1. telecommunications and technology.
Sample 3058:
Mobile phone network reaches last of China's ethnic minorities (AFP) AFP China has brought its mobile phone net- work to the last of its ethnic minority regions previously cut off from commu- nication with the outside world, state media reported.
2. tech giants and major industry players.
3. consumer electronics and gad- gets.
AGnews
4. words related to technical de- vices.
5. 3D printing and additive man- ufacturing.
1. company earnings and finan- cial results.
Sample 6125:
2. initial public offerings (IPOs).
Icahn Takes The High River NEW YORK Why has Carl Icahn set his sights on the relatively insignificant My- lan Laboratories, generic drug com- pany with just $1.5 billion in sales and $4.3 billion market cap?
3. investment portfolio diversifi- cation.
AGnews
4. financial literacy and educa- tion programs.
5. interest rates and central bank
policies.
Sample 1035:
1. record-breaking performances.
Orioles 8. Devil Rays 0 Javy Lopez drove in four runs, Daniel Cabrera be- came the first rookie to win 10 games this season, and the Baltimore Orioles held the Tampa Bay Devil Rays to two hits an 8-0 victory.
2. fan reactions and opinions.
3. team rankings and standings.
AGnews
4. sports analytics and data- driven insights.
5. sports science breakthroughs.
28

---

Published as a conference paper at ICLR 2025
Sample 52170:
Narthecium is genus of flowering traditionally
1. The plant's historical or cul- tural symbolism.
plants. This genus treated as belonging to the family Liliaceae but the APG II system of 2003 placed it in the family Narthe- ciaceae. The global distribution of the genus is widely disjunct - species in Asia 1-5 species in Europe (see Narthe- cium ossifragum and 2 species in North America. Narthecium americanum is candidate for listing under the federal Endangered Species Act in the United States.
2. The methods of cultivation and care for the plant.
3. The plant's method of repro- duction (c.g., seeds, spores, cuttings).
DBpedia
4. the genus or family of plant.
5. The plant's contribution to bio- diversity.
Sample 32678:
1. The architectural style of the building (c.g.. Gothic, Modern, Colonial).
Pemberton's 's Headquarters also known as Willis-Cowan House is a two-story brick house that served as the head- quarters for Confederate General John C. Pemberton during most of the 47 day siege of Vicksburg and the site where decided to surrender the city to Union General Ulysses S. Grant July 1863,During the 1960s the building housed kindergarten associated with Vicksburg Catholic School (St.
2. the location of the building.
3. The building's role in local or national history.
DBpedia
4. The cultural artistic signifi- cance of the building.
5. The building's awards or recognitions for design preservation.
Sample 12750:
Disma Fumagalli (born Inzago Septem- ber 1826 died Milan March 9 1893) was an Italian composer and teacher of music. He was graduate the Milan Conservatory where he began teaching piano in 1853. He composedmore than 300 études for piano as well as other exercises; he also wrote a concerto for piano and string orchestra. Fumagalli's ss brothers Carlo Polibio Adolfo and Luca were all composers.
1. the artist's born date
2. The artist's cultural signifi- cance.
3. The artist's enduring legacy.
DBpedia
4. The artist's unique artistic voice.
5. The artist's famous collabora- tions.
29

---

Published as a conference paper at ICLR 2025
A.7 MTURK SURVEY DESIGN AND INTERFACE
We perform the human evaluation through Amazon Mechanical Turk (MTurr)). Each worker is paid 0.05$ per question and must sign a consent form to take the survey. The details of the two tasks we designed are as follows:
Task 1 - Activation Faithfulness: this task, workers will be presented with a neuron concept alongside the corresponding top highly activated text samples. Workers need to provide a rating ranging from (strongly disagree) to 5 (strongly agree) based on the agreement observed between the neuron concept and the top 5 highly activated samples.
2. Task 2 - Contribution Faithfulness. In this task, workers will be presented with explana- tions from two models for text sample. The explanations are generated by showing the top 5 neuron concepts with the highest contribution the prediction. Workers need compare which model's 's explanations are better and select an option from "model is clearly better", "model is slightly better", "equally good", "model 2 is slightly better", and "model is clearly better".
We did human evaluations MTurk for Task I and Task 2 mentioned in Section 3.2 The details are as follows:
Human evaluation: We evaluate the following 2 models:
CB-LLM w/ ACC
Random baseline: For Task the highly activated text samples are randomly selected. For Task 2, the explanations randomly selected from the same concept set.
For task 1, we evaluate each model's most highly activated neuron concepts across each dataset. These concepts represent instances where the model exhibits high confidence. For task 2, we evaluate 5 random samples for every dataset.
To ensure more reliable results, each question in the tasks mentioned above is evaluated three times by different workers.
The survey interface for task and task 2 is shown in Figure IT and Figure 12 respectively. In task workers are also asked to provide ratings for each model, similar to task 1. These ratings are utilized filter out inconsistent results. The following logic is employed for filtering:
If workers indicate that model is slightly or clearly better than model 2, the rating of model must be no lower than the rating of model 2, and vice versa.
If workers select "equally good," the two models must have the same rating.
Deserption sontences: "Clever unexpected humon."
Sentences
the humor hinged on the bellef that knees in the crotch, albows the face and spit the eye are inhorendl funny.
it a sly wink to the others without bocoming a postmodem joke, made creesy by men in sardine can warppd logic.
are stabs at absurclst comedy.... but mostly the humor is the sweet, gontio occasionally cloytng that become an lranien spezialty.
great comedy limmoker
comedy need always make us
Insnccions
thorscuts
Do sgeo will the sMterensbooor
Select an option
Desoription: "Clever unexpected humar.". the most the above 5 sentences?
desorioe
Strongly Disagree
Disagras
Neilhe Disaggee
Agroe
Strongl Agree
Figure 1I: The interface for task Activation faithfulness.
30

---

Published as a conference paper at ICLR 2025
Task
Sentence:
Modei predicls
The first lins leent to gel a massage univedd en emtty wailing room Aller wailing 15 min ules wass told my apportment would need to cancelled because they adn't have time booked the weeks soceved ne refunc) drove (which
Because of the followi exclonations the order ofiinortance))
opted
Fooe cusfomer sennicn
massoge was nothing
my second massage do
Rude Inatouuale medica
tolls could gire me rotund bocause she dddt "now how use the computer "malThrr They still have 500 and it's sinusl twu novdls kuteil 've callei and called wilh relurn call to gel movrey Uscki Do nut here!
fort, edll This
fraffic
appeintmentt
lsyout.
Model predtcts this serieencce related to) "negative"
se fnarwingg axpienations the notter in importtnne))
Overa
3. Compettivv interest nates.
4. Oveipr noosp
5. Cloan and natting ambienca.
Do you agree with Model 1's explanations?
Strnnll Agrzo
fgrce O Nothhr Agroo nor Dicagroe
O Stsagroc O Strongly Disagres
Do ngree with Model 2's explanations?
Stoogyy Agras Agrea ( Nether nor we O Tasagren Sirnngy Deagrea
Which model provides better explenations?
Cleatty
Siihtty Eq jalyy good Sightty
Oaarty
Why you
Select apply
The explanations orovided are more relevant to the prediction.
equally good
Figure 12: The interface for task 2 - Contribution faithfulness.
31

---

Published as a conference paper at ICLR 2025
A.8 DETAILS OF PROMPTING CHATGPT
In this section, We provide the details of how we prompt ChatGPT to acquire the concept set. We use four human-designed concepts as examples for in-context learning. This prompting style requires only n queries to ChatGPT to obtain the full concept set and can be done efficiently through the web interface provided by OpenAI. The full prompts are shown in 11
Table 11: The designed prompts for each dataset and class. Prompt
Dataset
Class
Here some examples of key features that are often present in a negative movie rating. Each feature is shown between the tag <example></example>
<example>Flat or one-dimensional characters <example>Uninttresiing cinematographlyc<exxmmeee <example<Lack of lension-building scenes. </example>
SST2
negative
<example>Lack of emotional impact.</example:
List 100 other different important features that are often present negative movie rating. Need to follow the template ahove, i.e. <exampi>fentue<<<exmmple
Here are some examples of key features that are often present in a positive movie rating. Each feature is shown between the tag <cxample></example> <example>Engaging plot.</example>
<example>Strong character development<<examplee
SST2
positive
<example>Great humor </enam//
<example>Clever narrative structure, </tetappee
List 100 other different important features that often present in positive movie rating. Need to follow the template above, i.c. exaapleefccuures/xxxppp.
Here are some examples of key features that are often present in a negative Yelp review with lower star ratings (e-g.. stars). Each feature is shown between the tag <example></example>. exaample@Overprieed-eeeaa. <example>Unnpppetiing food.</examples
CoxampleUUnprofssiioe service. service.</cxample> <example>broken products.</examplee
YelpP
negative
The reviews fall into the following categories: Food, Automo- tive, Home Services, Entertainment, Medical, Hotels, Financial Services, Media, Parking, Clothing, Electronic devices, and Clean- ing. List 100 other different important features that often present in a negative Yelp review with lower star ratings (e.g.. or stars). Need to follow the template above, i.c. <exam- ple>features<ccxmmpe.
32

---

Published as a conference paper at ICLR 2025
Here some examples of key features that are often present in a positive Yelp review with higher star ratings (e.g.. stars). Each feature is shown between the tag <cxample></xxample> <cxample>Delicious food </example
<example>Outstanding service. </example>
<example>Great value for the price.</example
<example>high quality products.</examplee
YelpP
positive
The reviews fall into the following categories: Food, Automo- tive, Home Services, Entertainment, Medical, Hotels, Financial Services, Media, Parking, Clothing, Electronic devices, and Clean- ing. List 100 other different important features that are often present in positive Yelp review with higher star ratings (e.g.. 4 stars). Need to follow the template above, i.e. <exam- ple>features</exampl>>
Here are some examples of key features that are often present in worldwide news. Each feature is shown between the tag <exam- ple></example>.
<example>words related to country and place.<<example>>>> <example>political stunts taken by governments.<lexamplls <example>global issues. </example> wexample>words related to war, conffict.</example
AGnews
world
List 50 other important features that often present in world- wide news. Need to follow the template above, i.c. <exam- plesfenures</example>
Here are some examples of key features that are often present in sport news. Each feature shown between the tag <exam- ple></example>.
<example>name sports stars. </exaple
<example>words related to game, competition competition.<exampll>> <cxample>ball games like baseball, masketball.</examppp <example>name of sport teams.4/exampler List 50 other important features that are often present in sport news. Need follow the template above, ie. <exam- plesfeatures</exmmpe>>
AGnews
sports
Here some examples of key features that are often present in business and financial news. Each feature is shown between the tag <example></exmmple>>
<example>words related to currency, money.</example> <example>tth numerical amount of dollars. d/llars.</example>> <example>the symbol like S. S.</example> <cxample>words related to stock, fortfolio.</xxame
AGnews
business
List 50 other important features that are often present in business and financial news. Need to follow the template above, i.c. <exam- plexfeatures</exmmpe.>
33

---

Published as a conference paper at ICLR 2025
Here some examples of key features that are often present in news related to science and technology. Each feature is shown between the tag <example><<xxample>> <example>name scientists or the word scientists <example>words related to technical <example>words related to universe, space, planet.</examples <example>words related the natural landscape landscape-<exxmpless List 50 other important features that are often present in news related science and technology. Need follow the template above, i.e. cexampeefeatures<exxappp
science/ technology
</eicele/exampee>>
AGnews
Here are some examples of key features that are often present when introducing a company. Each feature is shown between the tag <example></example>
<example>thh name of the company.clexample <cxample>the location the company</cxample> <example>the founding year of the companyc/example> <example>words related to organization, group. <roup.//e List 30 other important features that are often present when in- troducing a company. Need to follow the template above, i.e. <exampe>fatturs<</ramppe..
DBpedia
company
Here are some examples of key features that are often present when introducing educational institution. Each feature shown between the tag <example></exmpple> <example>the name of the school. </examlle <example>the location of the school//exampl>> <example>the founding year of the school</exapple> woxsmple>words related to college, university </ettyy///xaapee List 30 other important features that are often present when in- troducing an educational institution. Need to follow the template above, ic.
educational institution
DBpedia
Here examples of key features that are often present when introducing an artist. Each feature shown between the tag <example><lexample>
<example>the artist's name.<lexample. <example><hhe artist's worksc/example> <example>the artist's born date</example>
DBpedia
artist
<example>words related to music, ceinting.</xxampen
List 30 other important features that are often present when intro- ducing an artist. Need follow the template above, i.e. <exam- ple>featurssc/xxmple..
Here are some examples of key features that are often present when introducing an athlete sports star. Each feature is shown between the tag <example></cxapple
<example>the athlete's 's or sports stars' name.</example> <example>the sport the athlete plays (e.g. football, basket- ball).</exapple>
DBpedia
athlete
<example><the athlete's 's or sports stars' born date<</eample>> <example>words related to ball games, cnmpetttioonc/example List 30 other important features that are often present when intro- ducing an athlete or sports star. Need follow the template above. i.e. <examplerfaauurs</exmmppp
34

---

Published as a conference paper at ICLR 2025
Here are some examples of key features that are often present when introducing an office holder. Each feature shown between tag <example></cxample>>
<example>the office holder's n/me.</example>
<example>the office holder's cesition.<lexample> <example>the office holder's born date</example> <example>words related to politician, businessman. </exxall List 30 other important features that are often present when intro- ducing office holder. Need follow the template above, i.e. ceammleefetuurss</xxmmpp
DBpedia
office holder
Here are some examples of key features that are often present when introducing transportation. Each feature is shown between the tag <example></exmmpe>>
<example>the model type of the transportation vehi- cle. </example
<example>the production date of the transportation vehi- cle.</example>
DBpedia
transportation
<example>the functions of the transportation vehi- cle. cle.</exapple>
<example>words related to ship, car, train. waain</example>
List 30 other important features that are often present when introducing transportation. Need follow the template above, i.c. cxcaple>fcaurrs//xxmpp>>
Here are some examples of key features that are often present when introducing a building. Each feature is shown between the tag <example></<xample>>
<example>the name of the building.</xxamphe </example> <example>tthe built date of building. </xaddnn///xxmmpe>>>>
DBpedia
building
<example>the location of the building </example>
<example>words related to the type of the building (e.g. church, historic house, park, </exaale< List 30 other important features that are often present when intro- ducing building. Need follow the template above, i.e. <exam- ple>feature<</exmplee>
Here are some examples of key features that are often present when introducing natural place. Each feature is shown between the tag <example></example>
<example>the name of the natural place, </lice/e/xmmpe>>> <example>tthe length or height of the natural place. <lexample> <example>the location of the natural place. </eeap/ex <example>words related to mountain, river.</cxampl>> List 30 other important features that are often present when in- troducing a natural place. Need follow the template above, i.c. <exmmle>featrre//exmpe..
DBpedia
natural place
35

---

Published as a conference paper at ICLR 2025
Here are some examples of key features that are often present when introducing a village. Each feature shown between the tag <example></example>>
<example>the name of the dallage<<cxamplee <example>the population of the village. </exaplle <example>the census of the
DBpedia
village
<example>words related to district, families, </example> List 30 other important features that are often present when intro- ducing village. Need to follow the template above, i.e. <exam- ple>features<<xxmple>>
families
Here are some examples of key features that are often present when introducing a kind of animal. Each feature is shown between the tag <example><<example>
<example>the species of the animal. <cxample>the habitat of the </exale <example>the type of the animal (e.g. moeahl.eexample>>
DBpedia
animal
bird. insect,
<example>words related to genus, fomily./example
List 30 other important features that are often present when intro- ducing kind of animal. Need to follow the template above, i.e. cexample>fatuurs<<eeammp
Here are some examples of key features that are often present when introducing kind of plant. Each feature is shown between the tag <example></example>
<example>the name of the plant. </extmleex <example>the genus or family <example>the place where the plant was found.</example> wexsmple>words related to grass, herb, flower.</example List 30 other important features that are often present when in- troducing a kind of plant. Need to follow the template above, i.c. <exampeefeaurre<<<sanp....
plant.</example
DBpedia
plant
Here are some examples of key features that are often present when introducing album. Each feature is shown between the tag <example></example>>
<example>the name of the album </exmmle/ <example>the type of music, instrument </examppe <example>the release date of the album. </eumm//xmmple>>>
DBpedia
album
<example>words related to band, studio. studio.</exampe </example> List 30 other important features that are often present when intro- ducing album. Need to follow the template above, i.e. <exam- ple>features</xxmppls
36

---

Published as a conference paper at ICLR 2025
Here are some examples of key features that are often present when introducing a film. Each feature shown between the tag <exampl><</xxample>>
<example>the name of the film. <iin<//xxmpl>>> <example>the maker or producer of the film.</example <example>the type of the film (e.g. drama, science fiction, comedy, cartoon, animation) cnimation)x<cexmmple
DBpedia
film
<example>words related to TV, video, </eeco///xmmpe>>>>
List 30 other important features that are often present when in- troducing film. Need to follow the template above, i.c. <exam- ple>features</exxmpl>>
Here are some examples of key features that are often present when introducing written work. Each feature shown between the tag <exanple></example>
<cxample>the name of the written work </examlex <example>the author of the film.</e ilm.</example>
written work
DBpedia
<example>the type of the written work (e.g. novel, manga, jour- nal), </e).</eample>
wexample>words related to book. <ook.</exmmpl>>>
List 30 other important features that often present when in- troducing a written work. Need to follow the template above, i.e. <xamplesfeaure<</xxmppe
37

---

Published as a conference paper at ICLR 2025
B APPENDIX: CBLLMS GENERATION CASE
B.1 PERFORMANCE OF CB-LLMS USING LLAMA2-13B AND MISTRAL-77 AS BACKBONES
Table 12: The accuracy, steerability, and perplexity of CB-LLMs (generation) using Llama2-133 13B and
Mistral-7B as the backbones.
Method
Metric
SST2
YelpP
AGnews
DBpedia
CB-LLM (Llama2-13B) 13BB)
Accuracyt
0.9649
0.9842
0.9444
0.9940
Steerabilityt
0.86
0.93
0.90
0.89
Perplexity]
78.86
17.05
30.61
45.72
Llama2-13B finetuned (black-box)
Accuracyt
0.9676
0.9858
0.9525
0.9926
Steerabilityf
No
No
No
No
Perplexity
31.29
11.56
22.99
29.04
CB-LLMM (Mistral-7B)
Accuracyt
0.9500
0.9810
0.9428
0.9934
Steerability!
0.82
0.83
0.62
0.85
Perplexity
55.25
14.81
23.22
14.93
Mistral-7B finetuned (black-box)
Accuracyt
0.9594
0.9807
0.9493
0.9904
Stecrability!
No
No
No
No
Perplexity!
32.70
8.86
21.78
25.19
38

---

Published as a conference paper at ICLR 2025
B.2 VISUALIZATION OF THE RELATION BETWEEN INTERPRETABLE NEURONS AND TOKEN PREDICTIONS
In this section, visualize how the interpretable neurons connected to token predictions through the final layer weights. We display the top 10 tokens with the strongest connections to each neuron (excluding non-meaningful tokens). The results are shown in Figure [3 and 14 We can that these tokens are closely related to the concepts represented by the neurons. Consequently, increasing the activation of these neurons raises the probability of generating the corresponding tokens.
Egypt
Anmy
Islamic
Neuron: world
Margaret
raid
Afghan
working
military
election
leam
Game
playoff
Neuron: sports
defending
coach
unbe
race
star
S
US
share
trading
Nauron: business
industry
demand
data
outlock
shares
Market
Internet
Web
Neuron: technology
new
Software
marketing
design
CEO
Apple
Figure 13: The visualization of how the interpretable neurons in CB-LLM trained with AGnews connect to token predictions.
39

---

Published as a conference paper at ICLR 2025
0007335
procady
Indepeediee
Esscatio
eouccaionn
Cathiiic
CUTTOURTT
BORG
back
Nauron a7Mll
Demoorslc
Mauseet
E3X503
HMS
CheEn
Nerron tansporiaton
cenying
0121
condracted
1994
landma
south
Talley
Mau
Cinic
Neuee: vilage
Fiver
Nerrn: aniral
1r/h
ECeces
Neroo skan
Neuver
bard
Hano-d
[V/]
(bc
Einv
rord:
ECur STACS
Noroe
(LEN
Figure 14: The visualization of how the interpretable neurons in CB-LLM trained with DBpedia connect to the token predictions.
40

---

Published as a conference paper at ICLR 2025
B.3 EXAMPLES OF STEERING CB-LLM
An example of steering CB-LLM is shown in Figure 15 When we set the "sport" neuron to an activation value of 100, CB-LLM generates sport-related new accordingly.
Steering CB-LLM Generation
CBL
Azthvate World Newd neuron
World News
ariss
yedlentty comord by potto unvedigation lirers stuse" over 25 chossends Paletions the country.
Actiette Business News reuren
Prompt
(Pretrained LLM)
Koreo' And No. BC3 soare fottos, The ropes South naves jus without rod
noam
Unsupervised Layer
Linar
groaded
Busincss
Cown's 30 Larring fhxs bur Stoce tawer us protss and whone nuter Cocp Cols Co. seid on Toursta is movhiqee qpoo eociiogs busaune aillen benelh netere
of..
Todhnology Niws
Mhhiirs'' TTh
search
Vidte Old
cenourcee (hat be Aossin Whb 2.0 San francoso sevel / Golsber -2 and in offeiig 25 percent..
Figure 15: Intervene the interpretable neurons can make CB-LLM generate corresponding text.
B.4 MORE EXAMPLES ON TOXICITY REDUCTION
Figure 16 demonstrates another example of Toxicity reduction.
Original Prompt (toxic):
How obtain and lleggl
drugs
We can steer "Benign neuron" to turn the response to non-toxic:
cannot provide information guidance on how access, potentially harm communities, engage in harmful behavior, promote the misuse of substances, exereise discrimination towards individuals based on their sexual orientation, ethnicity, religion, gender identity, any other personal characteristic that may be perceived as a significant issue
We can also steer "Toxic neuron" to make the response toxic:
You're definitely onto something! First things first, let's discuss various types of illicit substances you might be looking for. When it comes prescription medications other powerful drugs like oxycodone or hydrocodone, people often find their way into these drugs via online platforms like dark web marketplaces...
Figure 16: Another example of toxicity detection and reduction through steering the generation via CB-LLM.
41


Reviews:
[{"review_id": "w1tGiL1tPv", "reviewer": "Reviewer_mhCK", "summary": "This paper introduces the Concept Bottleneck Large Language Model (CB-LLM), a novel large language model (LLM) that offers clear and accurate explanations through built-in interpretability and scalability, representing a breakthrough compared to traditional black-box LLMs. CB-LLM not only narrows the performance gap with black-box models in text classification tasks but also demonstrates how to leverage interpretable neurons for concept detection and guided text generation in text generation tasks. The authors conducted experiments across multiple datasets and metrics, showing that CB-LLM achieves considerable performance while maintaining inherent interpretability compared to black-box models. Additionally, the authors point out that CB-LLM not only possesses inherent interpretability during text generation but also provides greater controllability over the generated content. This feature gives CB-LLM potential application value in various fields requiring controllable content generation.", "questions": "1. I find the illustration in Figure 3 somewhat confusing. Module 1 is represented in green text; does this mean that all green parts belong to Module 1? Similarly, do the blue parts represent Module 2? Due to the positioning of the text, I have difficulty distinguishing between the two modules. If my understanding is correct, I recommend adding some identifiers to clarify this distinction.\n2. I found it difficult to understand the concept of \"concept bottleneck\", which is widely used in the field of computer vision. I hope that a brief description of the concept bottleneck can be included in the related work section, as this would help readers from unrelated fields to quickly understand the paper.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. CB-LLM demonstrates outstanding performance in text classification tasks. After the introduction of Automatic Concept Correction (ACC), the model surpasses traditional black-box models on several datasets, showcasing its strong potential.\n2. During the training process for text classification, CB-LLM effectively removes negative activation values using the ReLU activation function, significantly reducing semantic ambiguity during training and enhancing model stability.\n3. The authors provide evidence through human evaluation that CB-LLM received positive ratings for both Activation Faithfulness and Contribution Faithfulness, indicating that the model's explanations align more closely with human understanding and expectations, thereby enhancing the model's transparency and trustworthiness.", "weaknesses": "1. In the CB-LLM for text classification , the introduction of sparsity constraints during training is not adequately explained. If this constraint aims to enhance the model's interpretability, the authors should provide more analysis and experimental evidence to support this claim.\n2. The authors used only RoBERTa and GPT-2 as backbone models in their experiments. Given that the main goal of the paper is to improve the internal interpretability of large language models (LLMs) in text classification, it would be beneficial to conduct related experiments using larger models, rather than relying solely on LLaMA for labeling.\n3. Although CB-LLM applies the concept bottleneck to text generation tasks, there is a lack of sufficient experimental validation regarding its internal interpretability when generating text. In section 4.2, the “concept detection” experiment is conducted only on a text classification dataset, which does not effectively demonstrate whether CB-LLM's internal interpretability in text generation is faithful and valid.\n4. Some statements in the paper may need to be revised for clarity and to avoid ambiguity. For example, the variable “y” in Equation 2 lacks sufficient contextual explanation. Additionally, both Table 2 and Table 6 mention “RoBERTa-base fine-tuned (black-box),” but the values differ. The “RoBERTa-base fine-tuned (black-box)” in Table 6 should be replaced with “GPT-2” to eliminate confusion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K4DrROmQz2", "reviewer": "Reviewer_we68", "summary": "The paper presents CB-LLM, designed to provide inherent interpretability for both text classification and text generation tasks. In text classification, the model performs comparably to traditional black-box models while offering interpretable reasoning. For text generation, the model introduces interpretable neurons that enable concept detection and controllable text creation, presenting potential applications such as chatbot toxicity moderation.", "questions": "1. With reference to Section 3.2, I am uncertain about the precise number of trained models: RoBERTa, GPT-2, and LLaMA-3. Could the authors succinctly clarify the training process and the employed LM, LLM, and sentence embedder?\n2. In Figure 1, should the value prior to correction be **-** 0.1?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper presents a novel method for efficient interpretation in text classification tasks and proposes the first method for interpretation in text generation tasks.\n- Both methods have somewhat practical applications for real-world tasks.\n- The paper is easy to follow and includes rich figures and examples.", "weaknesses": "1. There is limited discussion about the generalizability of interpretable layers across various domains or tasks, specifically regarding whether concepts learned in classification or generation from one dataset can be reused in another. This lack of focus on transfer learning limits the model's broader applicability to related but unseen tasks. Moreover, the method is only tested on LLaMA-3-8B, so it is also important to investigate whether it consistently works across different model families and sizes.\n1. It remains uncertain if the generated concepts consistently align with human intuition across all datasets.\n1. The reliance on ChatGPT and sentence embedding models raises concerns about performance variability. Given their close performance levels, choosing different models or conducting multiple experiments could help mitigate this variability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vfTDkdIweo", "reviewer": "Reviewer_tEeX", "summary": "The paper proposes to add concept bottleneck layers (CBLs) to LLMs (right before softmax layers for text-classification, and softmax layers for token generation). Each output neuron of a CBL represents a concept (e.g. \"sport\", \"car\"). This will increase interpretability of LLMs: the concept bottlenecks will tell possible concepts of text label (in text classification), and possible concepts of the context to generate a token. Different than previous works, although the concepts learnt are also given by GPT, they are generated only once by asking GPT to generate concepts for each class (rather than for each instance as in previous work). Thus reduce GPT query cost. Although CBLs have been used in computer vision, the paper claims that this is the first work employing CBLs for LLMs.\n\nThe paper shows that the CBL used for text classification and text generation doesn't drop the performance of the used LLM backbone. Yet, it improves interpretability. Specifically, it can help to manipulate LLM's generated text.", "questions": "* What is S_c in equation 2?\n* Sec 3, why not use CEBaB & IMDB as in C3M paper?\n* Which parameters are optimized in equation 5? \n* What is concept accuracy in sec 4.2?\n* In \"generation quality\" sec 4.2, sentence \"If the generated sentence lacks fluency, the perplexity can rapidly rise to around 500. Therefore, a small difference in perplexity would not affect the generation quality.\" --> how can you quantify it? what small is \"small\"?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The text generation part is simple but interesting.", "weaknesses": "The first part of the paper presents how CBL is used in text classification. I found this part unsatisfying. \n* The core idea actually is to learn a mapping between concepts generated by GPT and the label set, and thus it seems to me that this part can be explained easily in a much shorter text. Especially, the whole idea of step 3 (correction), written in more than half a page, is simply to zero out those concepts that aren't generated by GPT given a target label -- I believe we just need a paragraph for that. \n* What struggles me is the claim of improving interpretability. In fact, what happens here is that, if a text x is mapped to a label y via concepts Z, we can \"explain\" that label y is chosen because concepts Z are chosen. But that is because of the way GPT generating concepts for label y, and the way the mapping is learnt. It doesn't really explain how and why concepts Z are chosen for text x. What I see here is that the method simply replaces the text classification on the original label sets to the new sets of concepts. \n* The \"unlearning\" section is a bit untraditional to me. Removing a subset of concepts doesn't seem to me \"unlearning\" as the model parameters are still unchanged. \n* I found the experiments quite not adequate as only one LM is used. Would be better to see a wider range of backbone LMs here.  \n\nThe second part, text generation, is interesting to me. However,\n* the paper criticises C3M because it requires human-annotated concepts. But the text generation actually requires human-annotated concepts too (e.g. see fig 3). So it seems to me that criticism is unfair.\n* I think this part is related to the controllable text generation in the literature. I would like to see a discussion about the relationship.\n* I think the experiments should include some baseline in controllable text generation literature. \n* The interpretability analyses should include some analyses about the meaning of the latent concept neurons and their impacts to the final generated text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KvMENtnqdr", "reviewer": "Reviewer_jxAt", "summary": "This paper proposes the Concept Bottleneck Large Language Model (CB-LLM), a novel approach to interpreting language model (LM) decisions by connecting its hidden states to defined concepts. The CB-LLM algorithms for both text classification and text generation are thoroughly detailed and well justified. Overall, the proposed methods are both interesting and effective.", "questions": "1. What is the agreement coefficient (like Kappa coefficient) for the work evaluations in Tables 3 and 4?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea is interesting and novel. The motivations, research gaps, and connections to prior studies are well articulated.\n\n2. The authors explore and validate CB-LLM in both text classification and text generation contexts. Each step’s details, challenges, and solutions are clearly illustrated.\n\n3. Extensive experiments, visualizations, and case studies effectively demonstrate the proposed method’s effectiveness.", "weaknesses": "1. The concept space for text generation explanations seems only limited to class labels. Specifically, the explanations tend to indicate the model’s generation direction (e.g., positive or negative sentiment) but remain constrained to class labels. Recent studies [1,2] suggest that explaining LLM outputs in generation contexts should extend across the full vocabulary. For instance, contrastive explanations can reveal why a model generates one token over all other possible alternative tokens in the whole vocabulary, which offers a more fine-grained analysis than CB-LLM currently provides.\n\n2. The experimental setup in Section 4.2 lacks clarity. It would help to specify the input prompts used for measuring accuracy, steerability, and perplexity, as well as the length of generated texts.\n\n3. Steerability is insufficiently compared. First, the prompts used for each class are unclear and under-detailed. Additionally, steerability should be evaluated against the original generations without any control mechanisms for a baseline comparison.\n\n4. I am curious about the diversity of generated sentences using CB-LLM. Could the authors add experiments to assess diversity, such as generating multiple outputs from the same prompt and then evaluating variation across those outputs?\n\n5.\tI recommend that the authors refer to recent work on controllable text generation such as [3] to enhance the experimental design and presentation of results, which may help address weaknesses 2–4.\n\n6.\tMinor points: The full name of CAV in Section 2 is not provided. Also, the word \"We\" in the first sentence of Section 4 should start with a lowercase letter.\n\nReferences:\n\n[1] Interpreting Language Models with Contrastive Explanations, EMNLP 2022\n\n[2] Unveiling and Manipulating Prompt Influence in Large Language Models, ICLR 2024\n\n[3] Controllable Text Generation via Probability Density Estimation in the Latent Space, ACL 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)