Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
Authors: Geyu Liang
Senne Michielssen
Salar Fattahi
Venue: ICML 2025 poster (2025)
Keywords: Explainable AI
Interpretable ML
Optimization
Dictionary Learning
Computer Vision
Primary area: optimization

Abstract: The trade-off between accuracy and interpretability has long been a challenge in machine learning (ML). This tension is particularly significant for emerging *interpretable-by-design* methods, which aim to redesign ML algorithms for trustworthy interpretability but often sacrifice accuracy in the process. In this paper, we address this gap by investigating the impact of deviations in concept representations—an essential component of interpretable models—on prediction performance and propose a novel framework to mitigate these effects. The framework builds on the principle of optimizing concept embeddings under constraints that preserve interpretability. Using a generative model as a test-bed, we rigorously prove that our algorithm achieves zero loss while progressively enhancing the interpretability of the resulting model. Additionally, we evaluate the practical performance of our proposed framework in generating explainable predictions for image classification tasks across various benchmarks. Compared to existing explainable methods, our approach not only improves prediction accuracy while preserving model interpretability across various large-scale benchmarks but also achieves this with significantly lower computational cost.

Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
Geyu Liang'
Senne Michielssen 2 Salar Fattahi
Abstract
quently underperform on complex tasks. On the other hand, high-performing models, such as deep neural networks (Le- Cun et al., 2015) and large language models (Vaswani, 2017), are notoriously opaque, given their large parame- ter spaces and intricate architectures.
The trade-off between accuracy and interpretabil- ity has long been a challenge in machine learning (ML). This tension is particularly significant for emerging mnterpreluble-by-deiieee methods, which aim to redesign ML algorithms for trustworthy in- terpretability but often sacrifice accuracy in the process. In this paper, we address this gap by investigating the impact of deviations in concept representations-ann essential component of in- terpretable models-on prediction performance and propose a novel framework to mitigate these effects. The framework builds on the principle of optimizing concept embeddings under con- straints that preserve interpretability. Using a gen- erative model as test-bed, we rigorously prove that our algorithm achieves zero loss while pro- gressively enhancing the interpretability of the resulting model. Additionally, we evaluate the practical performance of our proposed framework in generating explainable predictions for image classification tasks across various benchmarks. Compared to existing explainable methods, our approach not only improves prediction accuracy while preserving model interpretability across var- ious large-scale benchmarks but also achieves this with significantly lower computational cost.
Numerous methods have been proposed to extract inter- pretability from complex models (Baehrens et al., 2010; Simonyan et al., 2013; Zeiler & Fergus, 2014; Shrikumar et al., 2017; Selvaraju et al., 2017; Smilkov et al., 2017; Kolek et al., 2020; Subramanya et al., 2019). However, these approaches typically adopt post-hoc strategies, utiliz- ing sensitivity analysis to identify the key parameters that influence predictions. Consequently, these methods lack guarantees of providing explanations for random predic- tion or ensuring their trustworthiness (Adebayo et al., 201 8; Rudin, 2019; Kindermans et al., 2019; Ghorbani et al., 2019; Slack et al., 2020).
An alternative solution to develop models that are inter- pretable by design. These models intrinsically integrate transparency into their architecture, providing explanations directly tied to their predictions. Concept Bottleneck Mod- els (CBMs, (Koh et al., 2020)) exemplify this approach by mapping input data to an intermediate representation of human-defined concepts, which is then used for prediction. While concept-based models are promising, they require datasets annotated with concept scores, limiting their appli- cability. Extensions of CBMs aim improve the approach by utilizing pretrained encoders like CLIP (Yuksekgonul et al., 2022; Oikarinen al., 2023), A recent line of research (Chattopadhyay et al., 2022; 2023; 2024) has introduced a decision-tree-baae architecture with enhanced explainabil- ity. The predictions of the model are generated through a sequence of queries, each closely associated with human- specified concepts.
1. Introduction
ML algorithms are often caught in dilemma between in- terpretability and performance. Models such linear gression (Hastie, 2009) and decision trees (Quinlan, 1986) provide straightforward interpretability through parameter weights and rule-based predictions. However, they fre-
A high level design paradigm of expainable AI is illustrated in Figure 1. Within these models, concept embeddings play pivotal role: they not only influence prediction generation but also serve as the source of interpretability. However, recent studies have raised several questions against its relia- bility. These challenges are centered around its ambiguity (Margeloiu et al., 2021; Watson, 2022: Kim et al., 2023: Marconato et al., 2023), fragility (Furby et al., 2024), and
'Department of Industrial and Operations Engineering, Univer- sity of Michigan, Ann Arbor, MI, US. "Department of Computer Science, Princcton University, Princeton, NJ, US. Correspondence to: Salar Fattahi fattahi @ @umich.duu>.
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Constrained Concept Refinement
Concepts
True label
"black and white stipe"
y*
uprigght mane"
Back-propagation
"grassland"
y
L
Input
Encoders
Embeddings
Interpretable model
Predicted label
Loss
Figure 1.
The arrows represent the backpropagation training process for classic explainable models. This paper extends the training process to refine concept embeddings with constraints on their deviation from initial embeddings, represented by green arrows and box.
low accuracy (Zarlenga al., 2022; Chowdhury et al., 2024). For embeddings generated with pretrained models, their coders are reported to face issues such domain adaptation (Shao et al.. 2023; Gondal et al., 2024; Feng et al., 2024; Wang & Kang, 2024), biases in pretraining data (Hamidieh et al., 2024), and sensitivity to encoding errors (Ranjan et al., 2024).
the concepts (Theorem 2.6). This highlights and motivates the idea of concept embedding refinement, which is at the crux of our proposed method.
Accuracy and interpretability guarantees. We show that CCR overcomes the aforementioned challenge by eliminating performance degradation through the appropriate refinement of concepts (Theorem 3.3). Fur- thermore, to quantify the performance of CCR, we consider a generative model as test-bed, where both "interpretability" and "accuracy" admit crisp mathemat- ical definitions. Under this model, we demonstrate that CCR achieves zero training loss while progressively enhancing interpretability (Theorem 3.4).
Efforts address these challenges include introducing unsupervised modules to enhance concept expressiveness (Sawada & Nakamura, 2022), training residual learn- ing modules in parallel with concept encoders (Yuksek- gonul et al., 2022), learning dual embeddings per concept (Zarlenga et al., 2022), and adding specialized data encoders to address domain shifts (Chowdhury et al., 2024). How- ever, these approaches all involve introducing additional black-box learning modules, which undermines the purpose of explainable AL Moreover, theoretical guarantees for such approaches, which is of particular interest for safety-critical domains, such as medical imaging (Barragán-Montero et al., 2021) and autonomous vehicles (Bensalem et al., 2023), remain largely unexplored.
Application in interpretable image classification. We demonstrate the practical efficacy CCR on mul- tiple image classification benchmarks including CI- FAR 10/100 (Krizhevsky al., 2009), ImageNet (Deng al., 2009), CUB200 (Wah et al., 201 I) and Places365 (Zhou et al., 2017). In all benchmarks except CUB-200 CCR outperforms two recently developed explainable methods for visual classification tasks-CLIP-IP-OOM (Chattopadhyay et al., 2024) and label-free CBM (If- CBM) (Oikarinen et al., 2023) terms prediction accuracy while preserving interpretability. achieving this with a tenfold reduction in runtime.
To address these challenges, this paper proposes novel framework termed Constrained Concept Refinement (CCR). Unlike existing methods that introduce additional learning modules, our approach directly optimizes concept embed- dings within a constrained parameter space. By restricting embeddings to small neighborhood around their initial- ization, our framework offers a tunable trade-off between performance and interpretability, and in certain settings, it simultaneously improve both. Specifically, we present the following contributions:
On high level, our main contribution is the introduction of framework in which concept embeddings in explainable Al models are refined within a restricted parameter space to attain better balance between predictive performance and interpretability. The remainder of the paper substantiates this claim along two key dimensions:
Theoretical necessity. We rigorously demonstrate the necessity of refining concept embeddings by establish- ing non-vanishing worst-case lower bound on model performance when such refinements are not applied
Theoretical validation. We demonstrate that our method is both necessary (Section 2.1) 1) and effective (Section 3) under the theoretical framework introduced
2

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
by (Chattopadhyay et al., 2024), whose background is reviewed in Section 2. This framework serves as suitable testbed for our contributions for two main models (Zhou et al., 2018; Koh et al., 2020; Yuksekgonul et al., 2022; Oikarinen et al., 2023; Chattopadhyay et al., 2023), only (Chattopadhyay et al., 2024) presents a generative model wherein both performance and inter- pretability are rigorously defined. (2) The algorithm motivated by this generative framework achieves state- oh-the- results within the class of interpretable Al methods.
tion, encapsulating the information relevant to that question. Additionally, the participants are limited asking at most k questions.
reasons: (1) Among seeepretable-b--eestte
In (Geman & Jedynak, 1996). the authors introduced clas- sic greedy-style algorithm termed Information Pursuit (IP) to address the task outlined Definition 2.1. Specifically, IP iteratively selects the most informative query based on the observed values of previously selected queries. We provide formal definition of IP for subsequent discussion:
Definition 2.2 (Information Pursuit). At every iteration
= 1..., k, IP selects π (t)-th query according to:
{ [1 y/ (m(1) = (m))))
2. Empirical evaluation. We conduct experiments multiple benchmark datasets for image classification tasks to assess the practical effectiveness of our ap- proach (Section 4).
x (1) = arg max Isisn
(x) (x(2)) ... (x(t-1) Tx((- 1))
Here I(:--) denotes mutual information, T(-) : [A] + [v]] is an injective map representing the selection of queries, and r; denotes the observed value for q: The final output of IP is defined as the maximum likelihood estimator given all observed queries:
Notations. For a matrix A, we denote its spectral norm by ||A 2, Frobenius norm by || and maximum column- wise fn-mr by || Define [x]] = {1,2,..., matrix A E TR*" called column- if A = m- Events with probability at least -1() are said to occur with high probability.
A
P(y = 5 9=(1) = =(()))
YIP = arg max
#
(x(2) = (x(2), .... (T(()) = T=()))) (I)
2. Revisiting IP-OMP
We consider a classical interpretable-by-eeitt setting in the literature (Chattopadhyay et al., 2022; 2024), namely the task of predicting random variable by sequentially select- ing a list of related random variables, termed queries, and checking their values. The problem is defined as follows:
In (Chattopadhyay et al., 2023), IP is demonstrated to achieve highly interpretable predictions with competitive accuracies. However, a significant limitation of IP is its substantial computational expense, as mutual information depends not only on the set {d. but also on previous observations, leading to an exponentially large input space. (Chattopadhyay et al., 2024) addresses this complexity by imposing additional assumptions on the underlying gener- ative model. The first quantifies the connection between and q., while the second defines their connections with d,. Assumption 2.3 (Generative model for IP-OMP). There exists an unknown random vector z E Rd drawn from standard Normal distribution that satisfies y = (x, z). Moreover, the queries satisfy 9: = (V,, z), where v, is a latent feature vector associated with query i.
Definition 2.1 (Single Variable Prediction by Query Se- lection). Given an input feature x E Rd,, set of queries 19 C R. and their associated query features (also referred to as concept embedddngs) {d.) C IR&, the objective is to predict the target random variable y e IR by selecting at most & queries from {q leveraging the query features {d to guide the selection process. Specifically, the query set {q: Y consists of random vari- ables that are correlated with y. whose observed values can contribute to predicting y.
Under the above assumption, the target random variable and the queries are correlated through the unknown random variable z. In this setting, a natural way to define the query features is by setting d, = v;. This entails an exact prior knowledge of the latent feature vectors.
Definition 2.1 encompasses a broad spectrum of ML prob- lems with applications in animal identification (Lampert et al., 2009), medical diagnosis (Graziani et al., 2018; Clough et al., 2019), visual question-answering (Yi et al., 2018), image retrieval (Bucher et al., 2019), and so on.
Assumption 2.4 (Prior knowledge of latent feature vectors). The latent feature vectors {v, 12 are precisely observed.
One illustrative example is the game played among friends, where the target random variable refers to the answer to the question "what am I thinking now?". In this context, the queries {qe )g are binary random variables representing "yes/no" answers to specific questions like "Is it an animal?". Here, the query feature is vector embedding of each ques-
Under Assumption 2.3 and 2.4, (Chattopadhyay et al., 2024) establish a connection between IP and Orthogonal Matching Pursuit (OMP) (Pati et al., 1993), a seminal algorithm in the field of sparse coding.
3

---

Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
Theorem 2.5 (IP-OMP (Chattopadhyay et al., 2024)). Un- der Assumption 2.3 and 2.4 and upon setting d; = vi for i == ,,...,,................. n,, the selection rule (-) defined in Definition 2.2 admits closed-form expression given by:
latent feature matrix. Assumption 2.4 can now be inter- preted as assuming D D" stating that we are using the ground truth latent feature vectors as the feature query ma- trix. This matrix can then be utilized to determine which queries should he observed.
Die- di, xX]
(1)) arg max
I1 de dillellu1 1) D.-,, x122
Let fD(x, {r (==())=))) be the maximum likelihood estima- tor of y, defined as Equation (I), when the selection rule T(-) is obtained by executing IP-OMP using D as the query feature set in Theorem 2.5. For brevity, when the context is clear, we refer to this maximum likelihood estimator simply as fD. Moreover, define (Vpred) as the squared population loss of any estimator Upred:
1gign
where ((1-1)- [d d2(2)
d d This selection criterion, referred to IP-OMP, differs from OMP solely by the inclusion of the normalization term [, 4 DT,,,
The main theoretical contribution of Theorem 2.5 lies in its ability to transform the computation of mutual information into that of linear projection, thereby significantly reduc- ing computational complexity. Although the equivalence is derived under simplifying assumptions, IP-OMP demon- strates remarkable generalization capabilities across various benchmark experiments, outperforming models based on the classical IP algorithm (Chattooadhyyyy et al., 2024).
[Vprrod) = EgwNN(O.Iaxe) [(v - -Dpred))) )
where z ~ NOO, Idxa) is the unknown random vector used in the generative model described in Assumption 2.3.
To showcase the effect of deviation in the query features, we consider scenario where the input x can be written as a linear combination of le latent feature vectors (corresponding to k columns of D*). This assumption is prevalent in the sparse coding literature (Arora et al., 2015; Agarwal et al., 2016; Liang et al., 2022) and ensures that using D' as the query feature set guarantees good prediction performance. However, even in this ideal scenario, we demonstrate that even small column-wise deviation from D can lead to performance degradation of IP-OMP.
2.1. A Major Hurdle: Inaccurate Query Features
As discussed in the previous section, Theorem 2.5, and as a result, the success of IP-OMP is contingent upon correctly choosing the query features {
For learning algorithms, the effective utilization of {d;} h is crucial for the accurate prediction of y. On the other hand, humans derive comprehension from {d. as these ele- ments are intended to embed concepts that are interpretable by humans. In practical applications, {d. are either embedded and learned from predefined datasets where con- cepts are explicitly labeled (Koh et al., 2020) or generated by pretrained multimodal models, with CLIP being the most popular example (Oikarinen et al., 2023).
Theorem 2.6. Suppose that Assumption 2.3 holds. Further- more, suppose that D' is column-orthogonal and xndx= D* 3 for some k-sparse vector 3 with [|1||o k. Additionally, assume that the non-zero entries in B have absolute val- ues bounded below by Y and above by T. Then, for any
E (0:
V1V there exists another orthonormal matrix D E RdxM such that | - 11 S and
However, these learned embeddings are often misaligned or inaccurate due to the inherent ambiguity and noise in the training of the models used to generate. For example, unlike carefully curated datasets, CLIP relies on large-scale, noisy image-text pairs scraped from the internet. These pairs often include mislabeled data, vague descriptions, or culturally biased associations, resulting in inconsistencies in representation, as recently reported by (Dutta et al., 2023).
1-DD"11.2
(Ub)-[([n-) > 81(k -1)2292
(2)
200
We present the complete proof of Theorem 2.6 in Ap- pendix C.I, which proceeds as follows: (1) we derive the closed form solution for L(fD) using the column- orthogonality of D and the distribution of z; (2) we then construct the example that satisfies Equation (2) by rotating columns of D' alongside the subspace spanned by itself.
This suggests that Assumption 2.4 may be overly opti- mistic. In this section, we will illustrate how the violation of Assumption 2.4 2.-spcccficallly deviations of the avail- able query features from the latent feature vectors-result in a proportional degradation in the performance of IP- OMP. For the remainder of this paper, we denote the ob- served query feature set compactly by the matrix D = [d, d2 ... dn] E Rdxn_ This matrix D is also referred to as a feature query matrix. We also refer to D" = [v] V2 va] E Rdxe as the ground truth
When r/y is bounded by a constant, we have ||x|| = || = (k-r2) which simplifies Equation (2) to L(f C(fD- = 2(c2 [x(2) This indicates that the perturbation is fully captured by the resulting gap in the squared loss, scaled by factor of ||x|]_
4

---

Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
3. Our Proposed Framework
Choice of the surrogate estimator. To motivate the choice of the surrogate estimator fD, let revisit the gen- erative model in Assumption 2.3, and additionally assume that the ground truth latent feature matrix D' is column- orthogonal. By assuming D to be column-orthogonal we effectively assume an ideal scenario in which queries are mutually independent. Specifically, it follows that (df z) is independent of (dz. z) when d; 1 d; and ~ NO, Taxá). The primary motivation for focusing on an column-orthogonal D' is that, under this assumption, the estimator SD obtained from IP-OMP becomes inherently differentiable with respect to D: Lemma 3.1. Assuming that D= = DD and D is column- orthogonal, we have
Building upon our discussion in the preceding section, it is pertinent to consider the following question:
Since inaccurate D can adversely affect the performance of IP-OMP, can D be optimized to mitigate this effect while preserving- or even ietpanciinnniineerreaablllte
There are two critical challenges to address before answer- ing this question. First, modifying D may diminish the en- coded information that is interpretable by humans, thereby compromising the seerpretable-y-deest
principle of in- formation pursuit. Second, treating the query features variables introduces significant complexity in deriving the optimal decision rule, as the closed-form expression of IP- OMP presented in Theorem 2.5 is no longer valid.
fo (x, (T(())
- (dd,, x/rr,,
To overcome these challenges, we propose obtaining a cor- rection AD that minimizes L(fD+AD), where fo serves as a differentiable surrogate of the original estimator pro- constrain the
iES
where mal Σier I(d,, x)) corresponds to the indices of the top-k largest values of {I(d,, x) x)I
duced by IP. To preserve interpretability, corrected query feature matrix D + AD to remain within small neighborhood around D, where D represents the potentially inaccurate initial query feature matrix (e.g., the one obtained from CLIP). This leads to the following con- strained optimization problem:
The proof of Lemma 3.1 is deferred Appendix C.4.1. Assuming a small initial error, i.e., D - D < p for some small p > O, the iterates D = D AD() of Algorithm 1 stay approximately column-orthogonal. This considcration motivates the introduction of the following surrogate estimator f:
((D++AD)
min
(3)
| SA
fo (x, (r(()) = Σ x),,
x)r::
iES
Indeed, the choice of the correction radius is crucial as controls the trade-off between prediction accuracy and interpretability. Setting م = 0 recovers the IP-OMP, and therefore, may suffer from the aforementioned performance degradation. Conversely, choosing large value for p can mitigate the performance degradation of IP-OMP, but at the cost of compromising the interpretability.
where
S = arg max
Σ kdd (4)
TC(m), TISK FET
It is important to note that we do not require f to be differ- entiable everywhere; rather, differentiability is required only almost surely along the trajectory of Algorithm 1, which we will rigorously demonstrate. Alternative differentiable surrogates for f include the task- dictionary learning method (Mairal et al., 2011 1) and the unrolled dictionary learning method (Malézicux et al., 2021; Tolooshams & Ba, 2021), which can be used in place of Equation (4).
Our meta-algorithm for Problem (3), called constrained concept refinement (CCR), is presented in Algorithm 1.
Algorithm Constrained Concept Refinement
1: Input: Initial query feature matrix D, correction radius
Backward propagation and projection For the back- ward propagation and projection steps, we propose to adopt gradient descent updates, followed by projection step based on fo-mo
A. training dataset D.
2: Initialize (0) Odxn
3: while t 1,2,,..... T do
4: Forward propagation: calculate L(D++AD)-))
5: Backward propagation: update AD using the
AD
gradient aL/aAD (t-1.
AD- AD OC +AD))
6: Perform projection to ensure |AD S
arg min |ADI
7: end while
OAD (2)
8: Return fDAAD(T)
We note that the specific instantiation of the proposed meta- algorithm and its steps is inherently task-dependent. For
Below, we provide a description of its various steps.
5

---

syki Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
example, while we have only discussed the squared loss, a more common choice in image classificatiin tasks is the cross-entropy loss. Additionally, our choice of the surro- gate estimator relies on the assumption that the ground truth latent feature matrix is column-orthogonal. When this as- sumption is not met, it becomes necessary to enforce it via concept dispersion. Both of these modifications are dis- cussed extensively within the context of image classification in Section 4.
for (4) and update rule (5) satisfies:
L(DDeA)) S
((FDe).
While Theorem 3.3 addresses the performance limitations of IP-OMP, it does not guarantee improved interpretabil- ity. This outcome is unsurprising, since the reliance of the method on only a single input x ensures that only the columns of D with indices in S can be improved, essen- tially leaving the columns outside S* unmodified. Moreover, if D is not full-rank, our analysis shows that D may devi- ate from D in directions oolumn-orthogonn space of D*; these deviations cannot be eliminated by Equa- tion (5). This observation is further supported by numerical experiments (see Figure 4).
However, a more fundamental question has remained unan- swered: how can "interpretability" concept inherently meaningful only to humans- be measured and quantified? One approach is to assess interpretability in an ad-hoc man- ner by running the candidate method and relying on human judgment to determine whether the predictions are inter- pretable for different individual samples. This has been the de facto approach adopted in nearly all previous works. An alternative way-which we seek in this work-is to study a simple test-bed for investigation, where "interpretability" admits a crisp mathematical formulation.
to the column
To tackle these challenges, we assume that D* is full-rank. Moreover, it is essential to use a sufficient number of i.i.d. input samples ensure that each column of D contributes to at least one input sample. This aligns more closely with practical scenarios, such as image classification, where mul- tiple samples are typically used during the training phase. We denote the i.i.d. input samples and their corresponding target random variables, generated according to Assump- tion 3.2, as {x" mm h==1 and (y² mm Th-11 respectively. For any estimator Ypred = [ the aggregated squared loss is defined as
3.1. Accuracy and Interpretability Guarantees
Consider the following probabilistic generative model.
Assumption 3.2 (Probabilistic generative model). Suppose that the target random variable y and queries are gener- ated according Assumption 2.3. Moreover, suppose that ground truth latent feature matrix D is column-orthogone and the input x satisfies x= = D*B, B, where: (1) the support of B. denoted as S*, is drawn uniformly from the set of k-element subsets of [n]: and (2) the non-zero elements of (3 are i..d. and satisfy E[G:] = O, Var[ß.] = o², and 7 S [3| I.
1 m
m
= L. (yred)
N(O.Taa) [(0" Upred )?]
(6)
Theorem 3.4. Suppose that {x" l==1 and (y' |m= are i..dd samples generated from Assumption 3.2 with fixed full-
runk column-or oothononrt D*. Suppose that m == Q (x
The generative model in Assumption 3.2 is akin to those explored in the sparse coding literature (Arora et al., 2015; Agarwal et al., 2016; Ravishankar et al., 2020; Liang et al., 2022). Under this model, prediction error can be evaluated using the squared loss, while interpretability can be assessed by the proximity of the query feature matrix D(T), produced by CCR, to the ground truth latent feature matrix D
Moreover, suppose that the initial query feature matrix D satisfies || | D* =p S SVeT and the step-size "I satisfies n 0 (+) With high probability, Algorithm 1 with surrogate estimator (4) and update rule (5) applied to the aggregated squared loss (6) satisfies:
(Interpretability) [|D - 11 & T(
Our next theorem shows that CCR effectively resolves the challenge faced by IP-OMP, as outlined in Theorem 2.6.' Let ( = D + A (1) be the corrected query feature matrix generated by CCR at iteration t.
D1 where T= V
k(k 1)0²
2n
-7.
(FDie)
A Em
|x* D
m
According to Theorem 3.4, by leveraging multiple samples, CCR achieves the best of both worlds: it guarantees con- vergence to the ground truth latent feature matrix D*, pro- gressively enhancing interpretability, while simultaneously driving L., to zero, thereby achieving perfect accuracy.
Theorem 3.3. Suppose that Assumption 3.2 holds. More- over, suppose that that the initial query feature matrix D satisfies || DD D 112 p S _Y and the step-size 1) satisfies 0 < n x
Algorithm I with surrogate estima-
We note that the result of Theorem 2.6 also holds for the generative model descrihed in Assumption 3.2.
Contrary to traditional analyses in dictionary learning (Arora et al., 2015; Liang et al., 2022), the proofs of Theorem 3.3
6

---

Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
and Theorem 3.4 require careful alignment of the updates from gradient descent with the direction of the true solution. In the setting of Theorem 3.4, this is even more challenging because the finite sample size m inevitably causes deviations from the population-level behavior. Moreover, we must further control the deviation introduced by the projection step. The detailed proofs of these theorems are provided in Appendix C. We also include detailed discussion on both theorems in Appendix B and experiments that validate them in Appendix D.
ods, we provide detailed explanation of the key compo- nents in Algorithm 2.
CLIP embedding. We employ the multi-modal model CLIP (Radford et al., 2021) a recently introduced large Vision- Language Model-to to embed both images and concepts into the same latent space.
Concept dispersion. After obtaining the CLIP embeddings, the query features tend to cluster too closely, making the query feature matrix (()) far from column- column-orthogonal To address this, we introduce concept dispersion procedure (see Algorithm 3 in the appendix). This heuristic approach enhances the mutual orthogonality of query features by increasing their relative angles, thereby improving the or- thogonality of ( Notably, this is achieved while pre- serving interpretability by maintaining the relative positions of features in the embedded space. Further details on the dispersion step are provided in Appendix D)..
4. Application: Interpretable Image Classification
In this section, we showcase the performance of our proposed CCR framework from Algorithm 1 on in- terpretable image classification task. The Python implementation of algorithm can be found here: Sathubb.oonllnnggyllllCCCCCCCCC git.
hard-thresholding In Equation (4), f is computed in two
The image classification setting differs slightly from the setting provided in Definition 2.1. Specifically, we given a dataset (a', , )mm consisting of m image (a)-label (y) pairs, where the goal is to predict the correct label for a given image. Here, represents a one-hot label vector. Additionally, this setting provides access to a concept set {c: "e.) which consists of n key concepts that can aid in classifying an image. These concepts are typically textual; for example, in the case of animal images, the concept set might include descriptors like "stocky body" or "black stripes".
steps: (1) constructing the sparse code s by keeping only the entries with the top-k absolute values of D x; and (2) setting f as the inner product of s with r = [r,,.... ri] R Here, we replace the top-k selection with the entry-wise hard- bard-thresholdin operator:
if
| 2 [ < A.
X
HT
0.
if
This approach allows for parallelization and reduced com- putational cost, although the exact sparsity level & can no longer be directly specified; instead, k is determined implic- itly by X. Nonetheless, adjusting À provides similar control over sparsity.
Algorithm 2 CCR for Interpretable Image Classification
1: Input: Dataset of image-label pairs (a', y pm con- cept set [c
Linear Layer. To compute fo as defined in Equation (4), access is required, which is unavailable in the image classification setting. To address this, we train a linear classifier that takes the sparse code as input and outputs a weight vector over the possible labels. The label with the highest weight is selected as the prediction. The loss function is defined as the cross-entropy between the linear layer's output and the true label y.
2: Use CLIP to embed the images {a 1 into input fea-
tures {xx 1==1 and the concepts Jn into query fea-
tures {d. |
3: Initialize ((0) via concept dispersion on {d.
4: Initialize s = HT (D(0) (0) x' for i = 1....,m. m.
5: Initialize ()) with random values.
6: while t = 2, T do
7: Calculate Lm = Σ, CE Ls! [IS(1
(1-1) y
Embedding normalization and projection. After each it- eration, we perform projection step similar to Equation (5) for each updated concept embedding. Additionally, we nor- malize each embedding to ensure that it remains on the unit ball (see Algorithm in Appendix D.2).
8: Update (* = D 1)) IiD dL (t-1)))
9: Normalize and project (
10: Update (() = 1t-1) M aL/0L(11
11: Update s!0 HTx (D()) x¹ for 1,...,m. m.
12:
end while
4.1. Performance
13: Return (T),, (T)) and {s] (T) m
1==1-
We compare the performance of Algorithm 2 against two recently proposed explainable AI methods, CLIP-IP-ON (Chattopadhyay et al., 2024) and label-free CBM (IF-CBM) (Oikarinen et al., 2023), as well as the CCR baseline without
We formally introduce our algorithm in Algorithm 2. Prior
to comparing our approach with other interpretable Al meth-
7

---

Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
CIFARIO
CIFAR100
length or k. used in CCR. For ImageNet and Places365, we report the best accuracy achieved by CLIP-IP-OMP across all explanation lengths. Since Lf-CBM does not allow tun- ing of its explanation length, we directly report its accuracy. The constraint parameter for CCR is fixed at 0.1 for all experiments.
12.5
65.3
04-
66.5
22
32.3
As shown in Figure 2. CCR consistently outperforms its baseline, CLIP-IP-OMP, and If-CBM across all benchmarks except CUB-200. This exception is expected, as If-CBM was built on a ResNer-18 18 backbone specifically trained on CUB-200, whereas both CLIP-IP-OMP and CCR rely on CLIP as their encoder, which was trained on more diverse and general datasets.
0
20
80
100
100
200
620
520
200
ImageNet
72
72
07
72-
71-
73
33-
24
01
Additionally, we report the average explanation length (AEL, k), average sparsity ratio (ASR, k/n), and aver- age concept embedding deviation (ACED, E |d, d (12) of CCR in the following table:
Places365
32
Contrained Concept Refinement
Baseline without CCR CLIP-IP-OMP
If-CBM
AEL
ASR
ACED
CIFAR1O
11.09
8.66%
0.056
100
250
CIFARTO0
19.67
2.39%
0.061
CUB200
27.52
13.2%
0.062
Figure 2. Prediction accuracy of CCR and its baseline across itera- tions, with the final test accuracy of CLIP-IP-MMP and If-CBM indicated for reference. For CCR and its baseline, we run each experiment for five times and present the average test accuracy at each time step. The shaded area is bounded by the maximum and minimum accuracy obtained over five runs.
ImageNet
48.97
1.08%
0.097
Places365
44.43
2.01%
0.041
Table 1. Average explanation length (AEL), average sparsity ra- (i((A and average concept embedding deviation (ACED) for CCR.
the concept refinement step, in the context of explainable image classification.
Thanks to the parallelization enabled by hard threshold- ing and the computational efficiency of backpropagation, CCR significantly reduces computational costs compared to CLIP-IP-OMP and If-CBM, particularly when applied to large-scale datasets such as ImageNet (^ 1.2 million images) and Places365 ( 1.8 million images). As reported in (Chattopadhyay et al., 2024), training If-CBM on Im- ageNet requires R 50 hours an NVIDIA RTX A5000 GPU. Under the same experimental conditions, CLIP-IP- OMP, with an average explanation length of z 50, incurs computational cost of R 40 hours. In our computational environment, using an NVIDIA Tesla V100 GPU, CLI OMP remains comparably expensive, requiring R 33 hours for = 50. In contrast, CCR (with & 49, as shown in Table 1) processes ImageNet in only hours (correspond- ing to 200 iterations) while achieving even higher accuracy, demonstrating a substantial improvement in computational efficiency. To make CLIP-IP-OMP more computationally feasible, one can reduce l;; however, for = 10, while the processing time drops to z 6 hours, it comes at the cost of a significant accuracy decline to R 63%.
For the baseline version of CCR, we set 1/D- = 0 at Step 8 of Algorithm 2, ensuring that the comparison isolates the effect of concept refinement. The two methods we compare against represent state-of-the-ar explainable AI approaches, particularly in terms of scalability. Lf-CBM was the first CBM-type model to be scaled to datasets as large as Ima- geNet, while CLIP-IP-OMP further improves computational efficiency while maintaining competitive accuracy. For a de- tailed introduction to both methods, we refer to Appendix A.
The evaluation is conducted across five image classification benchmarks: CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), ImageNet (Deng et al., 2009), CUB-200 (Wah et al., 2011), and Places365 (Zhou et al., 2017).
To ensure fair comparison, all methods use the same con- cept set, which is generated by GPT-3 (Brown et al., 2020). A detailed description of the generation process can be found in (Oikarinen et al., 2023). For the CIFAR-10/100 and CUB-200 datasets, we tune CLIP-IP-OMP to match the average sparsity level of s, also referred to as the explanation
8

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
to improve prediction performance without compromising their explainability. Our theoretical and empirical results demonstrate the effectiveness of CCR across a variety of tasks and datasets, both in terms of predictive performance and computational efficiency.
hight wnodie mets trom
lighting entertain
amall, recessed arma seating
debes
brightly ik inter or
The broader impact of this work lies in its potential to ad- vance the practical adoption of interpretable machine learn ing methods in real-world settings. In particular, the com- putational efficiency afforded by CCR may facilitate the deployment of explainable artificial intelligence (XAI) tech- niques in essource-constrain environments.
glasn
True dining room
catres and chairs
Predicted label: dining
Colcul
fbp
From an ethical perspective, the capacity to generate expla- nations that are faithful, stable, and aligned with human- interpretable concepts contributes addressing critical con- cerns related to algorithmic accountability and bias. For the proposed method, hyperparameter tuning remains crucial yet underexplored component for ensuring effective perfor- mance in practical applications. This aspect warrants careful consideration to ensure that the resulting outputs are both reliable and justifiable.
country
people special
gull ball
designated fainways and greers
course
True label: golf course
player
specturors
Predicted label: golf course
Figure 3. The first example illustrates a simple case where CCR successfully learns correct concepts. The second example rep- resents misleading case, where the image contains concepts like flag" that, while relevant visually apparent, could potentially mislead classification. However. CCR effectively extracts both useful and mislcading information, assigning appropriate weights ensure the correct prediction.
Acknowledgements
This research is supported, in part, by NSF CAREER Award CCF-2337776, NSF Award DMS-2152776, and ONR Award N00014-22-1-2127.
References
4.2. Interpretability
Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., and Kim, B. Sanity checks for saliency maps. Ad- vances in neural information processing systems, 31, 2018.
To illustrate the interpretability of Algorithm 2, we follow the methodology outlined (Chattopadhyay al., 2024) and present the most significant coefficients and weights from the algorithm different samples from the Places365 dataset. As shown in Figure 3, we highlight the top 10 concepts in with the highest values (left) along with their corresponding weights in the linear layer for the pre- dicted label (right). The selected concepts are semantically relevant to the image, and the linear layer effectively as- signs substantial weight key concepts such as "designated fairways and greens", enabling accurate prediction while appropriately disregarding concepts that, although relevant, may be misleading, such as "a flag" or "a country".. For additional case studies on other datasets, we refer the reader to Appendix D.5.
Agarwal, A., Anandkumar, A., Jain, P., and Netrapalli, P. Learning sparsely used overcomplete dictionaries via al- ternating minimization. SIAM Journal on Optimization, 26(4):2775-2799, 2016.
Aharon, M., Elad, M., and Bruckstein, A. K-svd: An algo- rithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on signal processing. 54(11):4311 4322)432111132,,---- 2006.
Anderson, T. W., Anderson, T. W., Anderson, T. W., and An- derson, W. An introduction to multivariate statistical analysis, volume 2. Wiley New York, 1958.
Impact Statement
This paper introduces Constrained Concept Refinement (CCR), principled framework that helps bridge the long- standing gap between interpretability and accuracy in ma- chine learning. By constraining the refinement of concept embeddings to lie within a small neighborhood around their initial values, CCR enables seappretalleby-desiiee models
Arora, S., Ge, R., Ma, T., and Moitra, A. Simple, efficient, and neural algorithms for sparse coding. In Conference on learning theory, pp. 113-149, PMLR, 2015.
Bachrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., and Müller, K.-R. How to explain individual

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
classification decisions. The Journal of Machine Learning Research, 11 11:1803-1831, 2010.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-254825 Icee. 2009.
A., Javaid. U., Valdés, G., Nguyen, D., Desbordes, P., Macq, B., Willems, S., Vandewinckele, I.., Holmström, M., Löfman, F., al. Artificial intelligence and machine learning for medical imaging: A technology review. Physica Medica, 83:242- 256, 2021.
Dutta, S., Wei, H., van der Laan, L., and Alaa, A. M. Esti- mating uncertainty in multimodal foundation models us- ing public internet data, arXiv preprint X1v:22300009926. 2023.
Bensalem, S., Cheng, C.-H., Huang, W., Huang. X., Wu, C., and Zhao, X. What, indeed, is an achievable provable guarantee for learning-enables safeta-critical systems. In International Conference Bridging the Gap between Al and Reality, pp. 55-76. Springer, 2023.
Feng, R., Yu, T., Jin. X., Yu, X., Xiao, L., and Chen, Z. Rethinking domain adaptation and generalization in the era clip. In 2024 IEEE International Conference on Image Processing (ICIP), pp. 2585-2591. IEEE, 2024.
Furby, J., Cunnington, D., Braines, D., and Preece, A. Can constrain concept bottleneck models to learn semantically meaningful input features? arXiv preprint arXiv:2402.00912 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877-1901, 2020.
Geman, D. and Jedynak. B. An active testing model for tracking roads in satellite images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 18(1):1-14, 1996.
Bucher, M., Herbin, S., and Jurie, F. Semantic bottleneck for computer vision tasks. In Computer Vision-ACCV 2018: 14th Asian Conference Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part II 14, pp. 695-712. Springer, 2019.
Ghorbani, A., Abid, A., and Zou, J. Interpretation of neural networks is fragile. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 33, pp. 3681-3688, 2019.
Chattopadhyay, A., Slocum, S., Haeffele, B. D., Vidal, R., and Geman, D. Interpretable by design: Learning predic- tors by composing interpretable queries. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 45 (6):7430-7443, 2022.
Gondal, M. W., Gast, J., Ruiz, I. A., Droste, R., Macri, T., Kumar, S., and Staudigl, L. Domain aligned clip for few-shot classification. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 5721-5730, 2024.
Chattopadhyay, A.. Chan, K. H. R., Haeffele, B. D., Geman, D., and Vidal, R. Variational information pursuit for in- terpretable predictions. arXiv preprint arXiv:2302.02876, 2023.
Graziani, M., Andrearezyk, V., and Müller, H. Regres- sion concept vectors for bidirectional explanations in histopathology. In Understanding and Interpreting Ma- chine Learning in Medical Image Computing Applica- tions: First International Workshops, MLCN 2018, DLF 2018, and iMIMIC 2018, Held in Conjunction with MIC- CAI 2018, Granada, Spain, September 16-20, 2018, Pro- ceedings 1, pp. 124-132. Springer, 2018.
Chattopadhyay, A., Pilgrim, R., and Vidal, R. Information maximization perspective of orthogonal matching pursuit with applications to explainable ai. Advances in Neural Information Processing Systems, 36, 2024.
Chowdhury, T. F., Phan, V. M. H., Liao, K., To, M.-S., Xie, Y., van den Hengel, A., Verjans, J. W., and Liao, Z. Adacbm: An adaptive concept bottleneck model for explainable and accurate diagnosis. In International Con- ference on Medical Image Computing and Computer- Assisted Intervention, pp. 35-45. Springer, 2024.
Hamidieh, K., Zhang, H., Gerych, W., Hartvigsen, T., and Ghassemi, M. Identifying implicit social biases in vision- language models. In Proceedings of the AAAI/ACM Con- ference on AI, Ethics, and Society, volume 7, pp. 547-561, 2024.
Hastie, T. The elements of statistical learning: data mining, inference, and prediction, 2009.
Clough, R., Oksuz, I., Puyol-Antón, F.,, Rujjsink, B., King, A. P., and Schnabel, J. A. Global and local interpretability for cardiac mri classification. In International Conference on Medical Image Computing and Computer-Assiste Intervention, pp. 656.664. Springer, 2019.
Kim, E., Jung, D., Park, S., Kim, S., and Yoon, S. Prob- abilistic concept bottleneck models. arXiv preprint wwX1v:2306001574. 2023.
10

---

Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
Kindermans, P.-J., Hooker, S., Adebayo, J., Alber, M., Schütt, K. T., Dähne, S., Erhan, D., and Kim, B. The (un) reliability of saliency methods. Explainable AI: In- terpreting, explaining and visualizing deep learning. Pp. 267-280, 2019.
Oikarinen, T., Das, S., Nguyen, L. M., and Weng, T.-W. Label-free concept bottleneck models. arXiv preprint arXiv:2304.06129, 2023.
Olshausen, B. A. and Field, J. Sparse coding with an overcomplete basis set: A strategy employed by vi? Vi- sion research, 37(23):3311-332 1997.
Koh, W. Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In International conference on machine learning, pp. 5338- 5348. PMLR, 2020.
Pati, Y. C., Rezaiiaar R., and Krishnaprasad, S. Orthogo- nal matching pursuit: Recursive function approximation with applications to wavelet decomposition. In Proceed- ings of 27th Asilomar conference on signals, systems and computers, pp. 40-44. IEEE, 1993.
Kolek, S., Nguyen, D. A., Levie, R., Bruna, J., and Kutyniok, G. A rate-distortion framework for explaining black-box model decisions. In International Workshop on Extending Explainable Al Beyond Deep Models and Classifiers, pp- 91-115. Springer, 2020.
Quinlan, R. Induction of decision trees. Machine learning, 1:81-106, 1986.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 7748-8763. PMIR, 2021.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Lampert, C. H., Nickisch, H., and Harmeling, S. Learning to detect unseen object classes by between-class attribute transfer. In 2009 IEEE conference on computer vision and patlern recognition, pp. 95 951-958. IEEE, 2009.
Ranjan, A., Wen, D., and Bhat, K. Unveiling glitches: A deep dive into image encoding bugs within clip. arXiv preprint arXiv:2407.00592 2024.
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning, nature, 521(7553):436 444, 2015.
Ravishankar, S., Ma, A., and Needell, D. Analysis of fast structured dictionary learning. Information and Inference: Journal of the IMA, 9(4):785 2020.
Liang, G., Zhang, G., Fattahi, S., and Zhang, R. Y. Sim- ple alternating minimization provably solves complete dictionary learning. arXiv preprint 0rXiv:2210.12816 2022.
Rudin, C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206- 215, 2019.
Liang, G., Shi, N., Al Kontar, R., and Fattahi, S. Per- sonalized dictionary learning for heterogeneous datasets. Advances in Neural Information Processing Systems, 36, 2024.
Sawada, Y. and Nakamura, K. Concept bottleneck model with additional unsupervised concepts. IEEE Access, 10: 41758 41765, 2022.
Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., and Batra, D. Grad-cam: Visual explana- tions from deep networks via braseent-based localization. In Proceedings of the IEEE international conference on computer vision. pp.
Mairal, J., Bach, F., and Ponce, J. Task-driven dictionary learning. IEEE transactions pattern analysis and machine intelligence, 34(4):791-804, 2011.
Malézieux, B., Moreau, T., and Kowalski, M. Understand- ing approximate and unrolled dictionary learning for pat- tern recovery. arXiv preprint arXiv:2106.06338, 2021.
2017.
Shao, J.-J., Shi, J.-X., Yang, X.-W., Guo, L.-Z., and Li, Y.-F. Investigating the limitation of clip models: The worst- performing categories. arXiv preprint arXiv:2310.03324, 2023.
Marconato, E.,, Passerini, A., and Teso, S. Interpretability is in the mind of the beholder: A causal framework for human-interpretable representation learning. Entropy, 25 (12):1574, 2023.
Shrikumar, A., Greenside, P., and Kundaje, A. Learning important features through propagating activation differ- ences. In International conference on machine learning, pp. 3145-3153 PMIR, 2017.
Margeloiu, A., Ashman, M., Bhatt, U., Chen, Y., Jamnik, M., and Weller, A. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289 2021.
Simonyan, K., Vedaldi, A., and Zisserman, A. Deep in- side convolutional networks: Visualising image clas- sification models and saliency maps. arXiv preprint 13 2013.
Motwani, R. and Raghavan, P. Randomized algorithms. ACM Computing Surveys (CSUR), 28(1):33-37, 1996.

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Slack, D., Hilgard, S., Jia, E., Singh, S., and Lakkaraju, H. Fooling lime and shap: Adversarial attacks post hoc planation methods. In Proceedings of the AAAI/ACM Con- ference on AI, Ethics, and Society, pp. 180-186, 2020.
Zeiler, M. D. and Fergus, R. Visualizing and understand- ing convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp. 818- 833. Springer, 2014.
Smilkov, D., Thorat, N., Kim, B., Viégas, F., and Watten- berg, M. Smoothgrad: removing noise by adding noise. arXiv preprint 1706:2170. 2017.
Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., and Tor- ralba, A. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6):1452-1465 2017.
Subramanya, A., Pillai, V., and Pirsiavash, H. Fooling network interpretation in image classification. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 2020-2029, 2019.
Zhou, B., Sun, Y., Bau, D., and Torralba, A. Interpretable basis decomposition for visual explanation. In Proceed- ings of the European Conference on Computer Vision (ECCV), pp. 119-134, 2018.
Sun, J., Qu, Q., and Wright, J. Complete dictionary recovery over the sphere i: Overview and the geometric picture. IEEE Transactions on Informaiinn Theory, 63(2):853- 884, 2016.
Tolooshams, and Ba, Stable and interpretable unrolled dictionary learning. arXiv preprint arXiv:2106.00058 2021.
Tropp. J. A. Greed is good: Algorithmic results for sparse approximation. IEEE Transactions on Information theory, 50(10):2231-22422 2004.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Vershynin, R. Efigh-dimensoonl probability: An introduc- tion with applications in data science, volume 47. Cam- bridge university press, 2018.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. The caltech-ucsd birds-200-201 1 dataset. 2011.
Wang, and Kang, G. Attention head purification: A
new perspective to harness clip for domain generalization. arXiv preprint arXiv:2412.072266 2024.
Watson, D. S. Conceptual challenges for interpretable ma- chine learning. Synthese, 200(2):65, 2022.
Yi, K., WU, J., Gan, C., Torralba, A., Kohli, P., and Tenen- baum, Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018.
Yuksekgonul, M., Wang, M., and Zou, Post-hoc concept bottleneck models. arXiv preprint arXiv:2205. 15480, 2022.
Zarlenga, M. E., Barbiero, P., Ciravegna, G., Marra, G., Gi- annini, F., Diligenti, M., Precioso, F., Melacci, S., Weller, A., Lio, P., et al. Concept embedding models. In NeurlPS 40th-36th Conference on Neural Information Processing Systems, 2022.
12

---

Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
A. Related Literature
In this section, we provide a more comprehensive literature review.
Explainable AL. In the main body of the paper, we provided an overview of related methods in explainable AL. Here, we focus two approaches that are closely related to and comparable with our proposed method. The first is the CLIP-IP-OMP method introduced by (Chattopadhyay et al., 2024), which leverages CLIP embeddings to greedily solve sparse coding problems for each input, subsequently passing the results through a linear layer to generate final predictions. Our method builds upon the same theoretical framework established in (Chattopadhyay et al., 2024). Notably, the baseline model (without concept refinement) considered in Section 3 and Section 4 serves as a differentiable surrogate for CLIP-IP-OMP The second approach is the label-free CBM (If-CBM) proposed by (Oikarinen et al., 2023), which employs the inner product between CLIP embeddings to achieve tfte-of-thheart results in interpretable models. Specifically, If-CBM was the first CBM model scaled to ImageNet. A detailed comparison of the performance of our method against these two approaches is provided in Section 4.
Sparse coding and dictionary learning. Sparse Coding (Olshausen & Field, 1997) refers to the process of representing a given signal (or vector) as a sparse linear combination of fixed signals, collectively referred to as a dictionary. Dictionary Learning (Aharon et al., 2006) extends Sparse Coding to a bi-variable setting, allowing the dictionary itself to be optimized. The objective is to identify dictionary capable of generating effective sparse codes for a specific signal distribution. (Chattopadhyay al., 2024) establishes novel connection between sparse coding and explainable AI, treating concept embeddings as the dictionary. Our proposed method directly optimizes concept embeddings, which draws parallels to dictionary learning, although the two approaches address fundamentally different objectives. Among works on dictionary learning, (Mairal et al., 2011) is particularly relevant, as it optimizes dictionaries for downstream tasks. However, our method distinguishes itself through the constrained concept requirement. Interestingly, (Mairal et al., 2011) aligns with our framework as a differentiable learning module, and integrating this approach with our method presents an intriguing avenue for future work. Another promising direction for future research is leveraging the personalized dictionary learning framework proposed by (Liang et al., 2024) to extend our approach Explainable Al in the context of heterogencous datasets.
Theoretical guarantees. To the best of our knowledge, our work is the first to establish a convergence guarantee in the setting where dictionary atoms are updated for downstream tasks. The most closely related work is (Chattopadhyay et al., 2024), which introduced IP-OMP. However, the connection between IP-OMP and column- oolumn-orthogonal matching pursuit lacks rigor due to the presence of normalization terms (see Theorem 2.5), making it difficult to prove that IP-OMP minimizes the loss L or fm using the optimality conditions of OMP (Tropp. 2004). Another related approach task-driven dictionary learning (Mairal al., 2011), which establishes the differentiability of the objective function but does not provide convergence guarantee.
B. Further Discussion
In this section, we provide further discussion on Theorem 3.3 and Theorem 3.4.
Tolerance, Sample Size, and Convergence Speed. Both theorems establish that the population loss converges to zero, provided that the initial dictionary estimate is within _ in the column-wise || ||1,2-norm from the ground-truth dictionary v D Assuming that constant fraction of the queries in D is utilized to generate each x*, i.e., k = 2(x), Theorem 3.4 indicates that each column of D can be accurately recovered with sample size of m O(n) and convergence rate of = 1 - 2(1). Notably, the linear sample size and linear convergence rate is consistent with the best-known results in dictionary learning (Arora et al., 2015; Liang et al., 2022).
Sparsity Level. It is well established the dictionary learning literature that higher sparsity levels in ground-truth sparse codes (i.e., larger values of l in Assumption 3.2) present greater challenges for recovery; see (Arora et al., 2015). Indeed, (Arora et al., 2015) asserts that sparsity levels beyond A = S log rarely succeed in practice, even though several approaches have been proposed to handle cases where A = Ω (n) (Sun et al., 2016; Liang et al., 2022). In contrast, our results reveal that the sparsity level l plays a nuanced role when optimizing D for the downstream task L. On the one hand, a larger k imposes a more stringent initial error bound for both theorems. On the other hand, a larger in Theorem 3.4 improves the convergence rate 7 and reduces the required sample size m. The former phenomenon consistent with
13

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
findings in dictionary learning literature, where exact support recovery becomes feasible for sparser generative models. high-level explanation for the latter is that denser generative model for 3 provides each with more information, thereby expediting the training process.
C. Proof of Theorems
C.1. Proof of Theorem 2.6
Our proof strategy follows two main steps:
Step I: We prove that L(fD.) ) = 0. To do so, we first derive an explicit form of fD and then show L(fD.) ) = 0 for this explicit form.
Step 2: We explicitly design a column -ottoon-rthhoonall D such that | - 1,2 I c and C(fp) 2(k- 136-1
D
5
To streamline the presentation, we denote the i-th column of D as dy. Note that d is identical to v, used in the paper.
Step 1: Establishing L(DD-) = O. We first provide an explicit characterization of D-
Lemma C.1. We have
fo- (x, (rr())i)))
(d),
iES
where S (2),.. [k)} is the index set selected by IP-OMP applied to x and D
Our next goal is to provide an explicit characterization of S, introduced in the above lemma.
Lemma C.2. For any column-orthogonn D, the index set (1), (2),,.,, [(k)) selected by IP-OMP applied and D corresponds to the indices of the top-k largest values of {l(d,, x)) D
S = arg max
Σ (d, x)
TEMM, iET
Moreover, if D* we have S = S*, where the support of 3 defined in Theorem 2.6.
We defer the proofs of Lemma C.1 and Lemma C.2 Appendix C.4.2 and Appendix C.4.3, respectively, Combining the above two lemmas, we obtain:
fD. (x, (T((O))2))
Ed'
x)r,, where S = arg max
Σ [d],
TC TET
sE8
Given this explicit form of fD- generated by IP-OMP, we are now ready to investigate the population loss L of that estimator. For any column-column-oo coluno-orthogonal D, one can write
27
L (fDD) Ea=
(d. (d. x)r; (xx,2)
LiES
Σ x) -(x, 2)
AES
(7)
-Bx=N(0,Faxa)
Σ (d; x, - (x,
ViE
Ex-N((,Taxa)
(DD (x, 2))
xx)))
14

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
is easy verify that random variable D x, z) is distributed as M/ DD -xxxxxx]l x11)). As result, can conclude that
(UD) = I
for any column-orthogonal D.
(8)
Given Lemma C.2, we have S = = We can subsequently conclude that:
D D5 T D.. D'. D.. D.. x,
This leads to - x x12 = 0, thereby completing the proof of our first step.
Step 2: Constructing D. Without loss of generality, let us assume S = [k), where be even. The case where is is odd easily proved by following the argument below and replacing &o with k 1. Consider the following explicit form for D:
d = cos 0d; sin +2/2
1 1
d d sin 0d; + cos 0d +//22
1 1sis S k/2,
d. de
i k
where 9 is an angle to be determined later.
Intuitively, D is constructed by iteratively selecting a pair of columns in D and rotating them by an angle 0 > O within the two-dimensional subspace spanned by each pair. Based on its definition, we can establish the following properties for D: Lemma C.3. D has the following properties:
D is column-orthogonal.
2in
The proof of Lemma C.3 is presented in Appendix C.4.4
Our next step is to calculate L( Note that D is column-orthogonal according to Lemma C.3. Therefore, according to
Equation (8), we have L(ʃp) D D x x1ī According Lemma C.2, in order to obtain the index set selected by
IP-OMP applied to x and D, we need to find the indices of the top-k largest values of [d,, x}|. For i s k/2, we have:
( cos cossOdd 0d; + sin 0d 4 Σ 3
(d, x)|
cos 03, + sin 0/342/22
Upon choosing 0 such that tan < </T, we have
sin 07 tan 0
cos 0/3.| 2
> sin A > sin 03
which guarantees that ((d,, x)| > 0. Similarly, for + k/2 < i < h: we have:
k Σ
(dd x =
cos Ad. - sin 0d; k/2
3,d
1=1
cos 88; sin 83;
0.
15

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
For i k, we have (d, x) [d]] x) 0. As result, when lan 0 </T,, Lemma C.2 guarantees that the index set selected by IP-OMP [k] S Given this fact, calculate L(fb) as:
[(fb) | 2
x12
-D'.. D x
&
D' D
Σ 3
Σ B,d;
12
k/2
k
(cos 03: sin 08442/2) di
Σ
(cos 03; sin (3/22/2222) d;
Σ
,,d,
(9)
1=1+//
k/2
k
((oos - 1)3; + sin 08:42/2) di +
Σ
((cos 1),3; sin 0/ d
==1+k/2
1/2
k
(b)
((cos 0 - 1)8: + sin 03:42/2) Σ
+
Σ
((cos - 1)3; -inn 03; 0/22
x=
k=12
where (a) is based on the construction of D, and (b) is due to the assumption that D is column-orthogonal. When sin 2 S /111612/77 we have for 1 S s k/2:
sin 0 = 2sin
CO8
2
2
2in
V
1
sin
2
2
2
41'
1
2 sin
2
7
v1
+ 161
8T
0
sin
?
?
2
4T
(1
cos 0),
which leads to
((cos 0 - 1)3; + sin 03:+2/2)
(cos 0- 1)?33 + 2(cos 0 1) sin 03,3 + sin? 03? 1+2/2
2 2(cos sin 03,3444/2 sin? 0/3
> 2r
sin 03, + sin 0.3
N 2
sin" 0,341/22
With identical arguments, we have for 1 + k/2 < i < k:
((cos 1)3; + sin 03:-)/2)
2 > sin? 03? k/22
2
As result, Equation (9) reduces to:
4/2
k
((SD) 2 sin 832
2 k//2
+
Σ
1
sin? 2
8 4/2
2
1=1
1=1+k/2
k sin 07²
2
16

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Finally, we need to perform a change of variable by setting = sin 2. Elementary calculation gives sin 0 =v4- and
2
Lan 0
524- 2- When < 2. we have sin 0 9e/10 and lan 0 s 6e/5. Recall that our proof requires tan 0 7/T and
sin 0 - V1+1612/72 As result, when c c= 2sin è Σ min 2
5y 21 6r
1
we have 1+1612//3"
2
v 11-1612/73
81ke2g²
c(fb)
200
As we mentioned, for an odd value of k an analogous argument can made to arrive at a similar bound
81
1)c²7²
(lD) - -(fx)) -(fb)
200
This completes the proof of Theorem 2.6.
C.2. Proof of Theorem 3.3
To prove the convergence of (FDR)) ). we use an inductive approach where at each iteration, we will subsequently prove:
(1) f recovers the exact support 8* (2) gradient descent will make progress towards one of the optimal minimizers which
will be explicitly defined later; (3) & [ D()) will decrease linearly.
Recall that ( + (+),,, The equations below rewrite Equation (5) for each column i E [n]] of D
d (44.35 = a ac(fDO)
(10)
ad (t)
d ((+)) arg min ||d d
(
(11)
d:lld-d.'
It is easy to verify that the above update rules are equivalent to Equation (5). We will use induction to prove this theorem. In particular, define
A(t) C(fD)) S ( where 7 1 27/
To prove this we define an auxiliary feature vector which will play an important role in our arguments:
B
(0)
à, d
vi E [m].
(12)
x 2
Based on this auxiliary feature vector, we define the following event:
3:
x !
B()) d. d,))
x,
Vi E [m].
x
Now, our proof strategy is to show that, for 0, 1, 1....:
B(t) implies B(t 1),
B() implies A(t + 1).
This two statements combined will establish the correctness of Theorem 3.3 Indeed, the base case A(0) is trivially satisfied due to Equation (12).
Establishing Btt) = B(t + 1). To begin with, we use the following lemma to show that f is able to find the correct support when (() is close enough to D: Lemma C.4. For any D such that || D D' $ 2p <
Avkr we have
7
= arg max
Σ
(d, x)| 5
TC (n ET
17

---

Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
We defer the proof of Lemma C.4 to Appendix C.4.5. To invoke Lemma C.4, have
( 11 - D 11, + [D - D'1112 S sp p= 2p.
The bound on ||
D 1| follows from the projection step in Equation (11) while the bound on | D'1l1 follows from the initial error bound. We can then conclude that S = S at every iteration t, which means that Equation (10) and Equation (1I) will only change E S*, while for other i $ S we have a a d This immediately implies that B() trivially satisfied for Vi $ S and Vt. Now, the goal show that B(t) = ( + 1) for Vi E S"
Based on Equation (8), we have
- ((JDw) = D D (T)
x1
Subsequently, the gradient of C at D can be written as:
ac(DD)) 2x (D'.DS) x-x, x) D
0D (
T 2x (D'. 2x(D.. D DT x Dis. Bs.
D
2x D D x Bg-
D's D
2x D ( x - 3s-
For i E 8", we have
ac(5
2(d x -3.)x. ad."
This implies that:
dd (++..5) d. -a + 2n(d (1)T
x x
A(t) B. -x x (
x + 2n(d! x x33xxx
|
B, x
(d( (c(00..)) + 27(d(() x -x)xxx)
x + 2n x x-)xxx
|
B, d ((+0..))
x -2n(dd
(1) x B,)x+ [T x- x-3:1) -3,)x
||x13
8; d (1+0.5)
[x]
In other words, Bt) implies B(t + 0.5). Suppose ||d [d.(+0.5) -d 1 S then we have d (++1) = (1+0.5)
which readily
implies B(t + 1). On the other hand, if || (++.6)
(00) ||2 p. we have:
d (++0.5
! [ (t++..5
a.)
(a - -d.)
3 - x d (++0.5)
3, - x T d (0)
((++0.5),410))
|x]
|x
(13)
x xTd(440.)
x d
|x| 2
18

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Therefore, when ||d ((+0.)) d 1a > p. we have:
d! d(t+1)= arg min
||d - ! (++.55)
d:l|d d a م|
d ((++0.5) (1)
d p ||| (q
(14)
2 1/2
= d + psigd (x ! ((140..) x² =dd
x
|xx|1
This in turn implies
d, ((+1)
= =d:- d psign x T d ((++0..) - -x ==( x
|x||
B. xT dd ())
A(0]
x d! ((+0..5) - x xd!("))
x - psign
|x|
|x||2
B. TJ ((a!tt)) psign
x x &T (t+.)
x (d
x- psign (x (xa(t+ -x xdd(o))
x
|
||x|2
B. - -xTd <Td(tt1) (++1)
establishing B(t + 1), as desired.
Establishing (t)
A(t + 1). First, we establish ljd, d 11 s r r|dd;
dd" 122
We do in two steps. First, we
prove that ||d, - d. (2+0.5)
1a 5 -||d; d [|a. Then, we show that | d (++1)
1a S dd d (c00.5) 2
To establish |dd. ( ||| < - r d (1) 1|2, one can write:
dd d
(++55)
112 = ld: d + (nd (T) x --Bixx112
A(t) | d() 2n|1x (a. - (ā--a(") )
< 2n x x||2| [ 1!o
r||d;
d 12-
Next, we show that |dd, a
(((t))/1
2 < ||d, d ((+0.5)
22
To do so, we need the following elementary lemma:
Lemma C.5. Given a fixed vector v and a = av, = bv, c= cv, where a, b and e are vectors and a, b, c are scalars, if
|c| Z bb 2 a and be we have ||c all2 2 b al|2.
We will prove Lemma C.5 in Appendix C.4.6. Now we invoke Lemma C.5 with:
v X
|||2
d (
B,
a
||x||2
p psign
x xTd." (+00.)
x xxa(()) dd
x ((t(tt).5)
x xTa
| 2
19

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Here be 0 given by their definition. Moreover
c [6] established as:
x d (++.55) x
!
lc|
|x|
x d
(++.55
x d
|x|
X
(1+0.5)
_d! 1|
2p
=16/.
Finally, [6| > a is given by:
B - x =d)
||x||
x x
x ( (0)
|||||
|
d (0) |||||||||
(15)
|x||2
D
(
p
[b].
Now, upon substituting these parameters in Lemma C.5, we have ||c lcal22 2 ||b - -all2, which can be rewritten as:
x d (+..))
x
B
Td
psign (x (xTttt+..)) - xx al
x
B
aa
(0)
x x
|
x
2
|x||
|
12
X
2
2
(++.5))
d (0)
d. d
(0)
[a
-( - (ā. a dl")|l1
2
||d [|d (++0.5) a:112 2 ||d (++1) d:l12
Here inequality (a) is obtained by Equation (13), Equation (12), and Equation (14). We can now conclude that:
[|d - -d:12 I ||a dill2 rl(a - d ddll2.
(16)
Finally, we are ready to establish A(t + 1). We rewrite L(J (() as:
((DD)) =
| =1
D (f)T x 1/
| D(T
S- x Bs.BB-18
| x 3g-
Σ E
x d(") 5.)
B d0
Σ ies*
||x 11
| 12
B(()
( a
||x11
On the other hand,
((D Σ
(d((tt) a. 111² ||x
i
20

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
be established by identical arguments. Finally, we have
[(fDu))) =
Σ iES
a ( a.
|x1 S
Σ SES
|
- -a.12 |x = L(fDW),
which is exactly B(t + 1). As result, we have proved Theorem 3.3.
C.3. Proof of Theorem 3.4
To prove the statements of this theorem, we first establish the convergence of ||d!" dill2 for every i E [n]]. Then, - | 11,,
provide the desired upper bound on f. (D))) in terms of
We will follow the same index convention defined in Equation (10) and Equation (11), with & replaced by f. For each E [m]] we use 3 and 5h* to denote the corresponding variables in Assumption 3.2. We define the set Qi as the index set of h such that is in the support of sh:
Q: {h E [m] ji Stin
Given Lemma C.4, we have:
3Lm (fo))
Σ 2(d x² -x)xx,
Vi [m).
dd!t)
m
heQ:
To ensure that @Lm(ppa)) 0a!
is indeed aligned with d!d - dj, we first need the following lemma:
Lemma C.6. Suppose that = 9 ( n
With probability least 1 2 2exp(logn nn] -2exp flog
km
for all
8n
i [n].
k(k 1)o²
Σ xTT
od
4n
hEQ.
4ko 2
Σ x
01
5
m
n
hEQ,
We defer the proof of Lemma C.6 to Appendix C.4.7. Now, can bound
||d ( (c-0.5)
11,
as:
|| (+0.5)
d = d
17
Σ
((
x² 3x" - -d,
m
hEQ:
=I|d!0
d1'1 4n ( d m
Σ
(1) " 1x"
+4²
1
Σ
(a!
x²
(17)
m
hEQ:
heQ.
21

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
For the second term the right hand side, have:
d 10
(d. x* )/))x
hEQ.
d
( x" -d; x²) )x
m
hEQ:
d
(1)
dj
x x TT (d!!) di)
m
nEQ;
1
Σ x
|1000 - di
2 - -0d
m
hEQ:
Lemma C.6
l:k
1)o²
2
d
d/
4n²
For the third term on the right hand side of Equation (17), we have:
1
(d (TT x² - )x²
hEQ.
x hT (aa! - -dp)
m
hEQ
2
Σ XT
|ag00
d;
l|
m²
hEQ.
Lemma C.6
16k 4
di
n²
When 17
128ke2 the above two inequalities can be combined with Equation (17) arrive at:
k-1
d;113 - 1 k(k 1)02
64
||d ((+0.5)
7
77?
lld
d
n²
n²
< 1 k(k - 1)0²
l|d - d1
(18)
2n
S 7 (ld
d; 112
Next, we aim to show that ||a
d < ||a
(00.5) d
Recall that:
(++1)
arg min ||d d dED
(+0.5)
| where D= (d: ||d a 112 S p}.
It is obvious that D is convex and ||d d. condition requires that:
( 0.5) | is strongly convex with respect to d. As result, first order stationary
V Va-atein
|| a (00.55) [2 à d ((+1)) = =2 -2(a
-d (e-0.))) à -_d!tt ((()1) 2 0,
for any d D. Given the initial error bound [| D* 11122 S p. we have d' D. which results in:
(d d (L+..5) d - d (((+1)) 2 0.
22

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
We subsequently have:
||d d ((+0.b)
[| -a af(+0.61) + (d - d |1 + 2 ( ( - d ((t+0.6) d - d(ti)) + bbll d 1
|| d 11
(19)
where the last inequality follows from the fact that both ||d| [a((+)) d ((+00.)) 1|2 and (a a ((+00..) d d d(++1) are non-negative. Combining Equation (18) and Equation (19), we have
|| 1 z [|( dī S 73 l di1
Finally, note that
| - 11,,, max (ax (+1) d;l12) S max d2 S <T| 111,
iE(n]
:E|u]
Next, we show that f. (D(1) is upper bounded by | - D'I11.2
m
f. (IDw)
D S
D x² x²x
m
D ( h Sh
1/ 112
m
TT d, -xATd; d;
A= Sh
12 -d713
SESh-
m
x||x"
(
D²
m
>>
h
D
D'
m
To complete the proof of Theorem 3.4, need examine the success probability which the above linear convergence occurs with. The only probability statement use is Lemma C.6 and only invoke once. So the total success probability is at least:
- 2exp(logn n) 2 2ex
{
log
km 8n
which is - ==() when m = 22 0iik)) This completes the proof of Theorem 3.4.
C.4. Proof of Lemmas
C.4.1. PROOF OF LEMMA 3.1
Lemma 3.1 directly follows from Lemma C.I and Lemma C.2.
C.4.2. PROOF OF LEMMA C.I
We decompose y follows:
y = (x, z) = (IIpzx,z) z) + (1ID, x,
23

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Note that z NO Txxa). This implies that b z is independent of I1 D3 z (Anderson al., 1958), which in entails that Dz z) is independent of (Il x. 2) and the events {(d z) ri vi E S}. Therefore, we have:
MLE ((x, z) (d z) = V) = MLE (LUDg z) (d), = Vi ViE
+LLE (Ib:x.z))
It turns out that the first term on the right hand side is deterministic:
[ID: z) DD' x,z)
(d) x) x)
iES
Σ (d) x) (d)
IES
Σ (d) x.r.
'E
For the last equality we used 0 = ri. For the second term, we have:
MLE (II x, 2 -MLE x
0.
The equality (a) is due the observation that bb; z indeed distributed as N ((- ()=()x( after a simple dimension- ality reduction (Anderson et al., 1958). The resulting random variable (II ( x, II by z) is distributed as N/ |IT x 1 leading to the equality (a). To sum up, we have
MLE ((x,z) z) 2) r V) E(d; x)r.
(d) iS
C.4.3. PROOF OF LEMMA C.2
Let us define
S = arg max
Σ xd),
TEMLTT 'ET'
We will use induction to show that S₁ corresponds to the indices selected by IP-OMP after k iterations. At the first iteration, according to (Chattopadhyay et al., 2024), IP-OMP simply selects the index = arg maxje(n) kd,, /)// | = arg max (d,, I where the second equality due to |d, 12 = 1 for all j E [n]] As a result we have S = {i} and the base case established.
Suppose that the statement holds for S arg maxyg Sk-1 Eier Kdi, x) Let Ds be the submatrix of D which consists of columns indexed by Sk-1. Then IP-OMP will select
d IT Den
x)
i arg max
D.y.. d IT D. x|12
jElm)jgse-1
As D is oothonaort we have IT 8k dj d, and I
dj 10 V4 E Sc-1. Therefore, for Vj & Sk-1, we have
(d..)) (IDDs, dr. 11b.
x) + (IIDs s WDD x)
=IT dy. Tb x).
24

---

Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
As result, rewrite the selection rule of IP-OMP
KLU S6
d
x) [1
D
arg max JeS.
[1 d
x11
[(dj. x)]
arg max je(n)
|d/l2||x||2
arg max je(n)jaS.
[d], x)
The last equality is due to ||d, 1 and the fact that ||||| is the same for different j. This implies that, iteration k IP-OMP will select the index & - with the largest (d], x)|, which completes the proof of our induction.
Finally, when D = D", one can verify that (d;, x) (d,, D B) = 0 for $ S*, This implies that S₁ = S*.
C.4.4. PROOF OF LEMMA C.3
To prove D is golumn-orthogonal, we first observe that, for i S k; we have || cos? 0 + sin² A 1. Moreover, for any pair < j S k/2, we have
(d., d,) = (cos od; + sin Od 2//2 cos 0 + sin Od -2/2) = 0,
which follows since (d?, dī) 4 dř) (d) d xk/2) +2/2 di+k/2) A similar argument can be made for any i < S and e i + k/2. For any < k and j = i + k/2, we have
(d., d (cos 0d; + sin 4 sin 0d; + cos + sin cos sin cos 0.
Finally, for any j > > h we trivially have (d,, d,) 0. This completes the proof columm-orthoognnaliyy of D.
To prove the second statement, note that ||d, d 2 0 for every i >> k. Moreover, for every S k: we have
dd di ||2 [|cos 0 1)d, sin u V(cos8 1)2 ssn? A v2 2 cos 0 2 0 This completes the proof.
2
C.4.5. PROOF OF LEMMA C.4
The proof strategy adopt here is similar with the proof of Lemma 3.1 in (Liang al., 2022). However, con- sider different generative model in this paper, will show that for i E 8*, (dd, x)1 > 7/2 and for & S [dd, x) < 7/2, which will immediately result in S = arg maxrgin ΣiEr (d,, x) = S*, which proves Lemma C.4.
present the full proof for the purpose of completeness. We
We decompose (d,, x) as:
(d. x) (d;; Σ d}
JES
(d,, (
(d, d3)
(20)
For Ai, we have:
A = (di, d7),, = (d; + (d, - d;, = B, + (d. di)d;)
When i E S* we have
|8 + (d. (5.d))8.1 (1 |d - |3
> (1 D D* 11111.2)181
2 (1 [3)
25

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Given that 2p < _
1.
can conclude
dyET
37
if
[Ai
(21)
given that 3, = 0 when i S*. For B,, we have:
[B|1|
Σ
(d. d3)3
j#L.EES
Σ
( + (d, d.), d)),
jrijEs
Σ
(d. - dd.-d, d3)3
(22)
jøijES
Σ (dd di d5)11111
HiIJES
r'ldd - diil
$ VETidd - 1
2VTTp
< <//4
Combining Equation (21) and Equation (22), we have that for all i:
3y
1 = if iES"
(d, = |4; + Bi
(23)
if i S-
which proves the exact support recovery.
C.4.6. PROOF OF LEMMA C.5
is easy to see that ||c a|2 2 2 b a|I2 is equivalent |c al 16 a/.
If c 0, by be 0 must have b 0. Then |c|
> |b] becomes a, which gives |c a| 2 10-00 a].
If c. < 0, by bc ( we must have b < 0. Then [c] v
[ becomes a 2 b 2 which gives |c al 2 |b a. as well.
C.4.7. PROOF OF LEMMA C.6
The proof of Lemma C.6 consists of two steps:
1. We first classic concentration inequalities from covariance estimation literature bound ag and given [Qx)-
2. Then we will show that when m is large enough, with high probability, we have [Q:] bounded above and below.
Combining these two steps and taking the union bound will complete the proof of Lemma C.6.
For the fist step, we first notice that:
Σ x
Σ
D'
D*T
D
aa
= ad
hEQ:
hEQ.
Σ
3d 3hT
od
REQ.
26

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
The last equation is due the fact that D* is full-rank orthogonal matrix. Consider the vector B for h Q. According our generative model and the definition of Qi. we have ( * O, and its support outside is selected uniformly over all (k -leemment subsets of Vi). Each nonzero entry of 3 has zero mean, variance of o², and absolute value bounded between and I. Therefore, its covariance matrix defined as Σ == E [3 is diagonal, with its (i, i) entry
corresponding to o². and its (j.j)-th entry (with # i) corresponding to large. Eneq, x"xAT will concentrate around [Q; I. Specifically, we will use the following well-established result:
(k=1) We claim that when [Qal is sufficiently
Theorem C.7 (VVershynin, 2018)). Let w be a zero-mean sub-Gaussian random vector in R with covariance matrix Σ. such that
(ww, als I K (((w,q)))) 1/2 for any E Rd
for some K 2 1. Here I1 denotes the sub- Sab-Gaussian norm of random variable. Let W Rdxx be matrix whose columns have identical and independent distribution Then, for any u 2 0 and with probability at least - 2 exp (-u). we have
(V (Va+ m
d
WW
Σ 2
SCK2
|
m
m
for some universal constant C.
To invoke Theorem C.7, we note that 3h indeed zero-mean and sub-Gaussian. Therefore, upon setting w = =B", we notice that for any unit-norm q:
1)
S
ولlا) 12 S Coo",
for some universal constant Co. We also have
(k: l)o²
IF
(g" q)
= - _x
n
By setting K² Con k-
we can invoke Theorem C.7 with u n and conclude that, with probability at least
2exp(
od
Σ ghght
1Q (E)
Cna
/n|Q.
1Qi(k- 1)o²
Cna
2n 22|Qia
(24)
k
n
k
hEQ.
Σ 3h ghT
1Q/am (E) +
Cno
=Q:10² + k k-
Cno
(25)
k
hEQ.
for some universal constant C, which concludes the first step. Here we recall that d = in the full-rank and orthogonal setting.
For the second step, consider one specific from [n]. The random variable [Q:] follows a binomial distribution with m trials
and f success rate. We next recall the Chernoff bound: n
Theorem C.8 (MMowwani & Raghavan, 1996)). Ler X2,X...... be independent Bernoulli random variables with
P(X; = 1) = Ps, and let
N X
N
Σ X; and P E[X]
Σ
Pi-
==1
i=1
Then for any 0 < 8 < 1, we have
fu PP( < (1 -))m S expl exp(-au)
and
P(X2 (1 +8) $ expf m1
27

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
We then invoke Theorem C.8 with =
obtain
km 2n
2km
-
Q
(26)
n
with probability at least 1 - 2 exp {
km
Finally, by combining Equation (24) and Equation (26), we have
od m
1
Σ hEQi
Bh ghT
Q21(k: 1)o2
Cno
V 2nl Qi
1m
(k 1)m
k(k - 1)o²
Cna
4k
(A: 1) V m
(27)
2n²
When m 2 .,
==2k(k113- we have
Cno
4k
k(k 1)o²
(h: - 1) V m
4n²
which reduces Equation (27) to:
Σ
ghT
k:kk - 1)o²
k(A - 1
k(k -
1102
od
2n²
4n²
4n²
hEQ:
With a similar argument, we can combine Equation (25) and Equation (26) to get:
1 σ1
Σ
3h
ghT
< <1Qi102 + k k-1
Cno
2n| S
4ko²
m
n
hEQ,
Finally, we take the union bound for all i e [n], leading to the overall probability of 2n exp(-n) - 2n exp This completes the proof.
km {-
8n
D. Detailed Experiments
D.1. Experiments on Generative Model
All experiments reported in this section were performed in Python 3.9 on a MacBook Pro (14-inch, 2021) equipped with an Apple MI Pro chip.
We generate samples according to the described generative model in Assumption 3.2, we construct the input data (x" ym by sampling each non-zero entry from uniform distribution over the interval [7, 17.. The matrix D is chosen to be randomly generated orthonormal matrix, and we construct Dinit by setting Dinit = D + E, where each column of is uniformly drawn from the set (e ||el| < p). For the results shown in Figure 4, we set d = 10, k = 5, = 0.2, 7 = 0.5, and I' = The first column of Figure 4 illustrates the scenario corresponding to Thcorem 3.3, in which only a single input feature convergence of the loss function to zero, in agreement with Theorem 3.3. However, as previously noted, this setting does ensure exact recovery for each queried feature. Indeed, shown in the right panel of the first row, (() D* 1, does not converge to zero. Even if we only consider columns that are activated at each iteration, | g - D2 also remains nonzero, indicating that optimizing L with a single x is insufficient to recover the ground-truth dictionary D
available. Here, we choose n 8 and r = 10- The plot showing f() confirms the linear
In the second column of Figure 4, we apply projected gradient descent to minimize L. as defined in Equation (6), under the assumption that D is rank-deficient. Specifically, we set converge because the component of (()) D* orthogonal to the column space of D* remains unaffected throughout the iterations. In contrast, the third column of Figure 4 corresponds to the setting = d, which guarantees the full-rankness
d and 1) 10-1 Evidently, | D' does not
of D full rank. In this case, ( Di converges to zero, thus supporting the conclusion of Theorem 3.4.
28

---

Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
10
10'
C(fD"))
Lm(b<)
Lm(fpn)
10 1
10-4
10-11
10a4
10
10
-4
10-1
2000
1000
1500
<n vs. Iteration (rank defficient
D
En vs. Iteratioo tfull rank
0.199
|
11.2
||D D 111.2
||
D 11.2
|
D
h.2
0.18
0.197
0.16
10
10
0.195
0.194
0
200
400
600
800
1000
500
1000
1500
2000
500
1000
1500
2000
Column-wise dist vs. Reration
11-lba vs. Itaration trank defficient D')
[1-1/z2 iteration (tull rank D
Figure 4. Results on synthetic dataset.
D.2. More Details Algorithm 2
Here provide detailed description for the concept dispersion step and the embedding normalization and projection step in Algorithm 2. In Algorithm 3, we present the pseudo-code for concept dispersion. The algorithm first caleulates the mean concept embedding for the given {d, ) and calculates the angle between each d and the mean. Then, Algorithm 3 increases these angles by a constant factor r.
Algorithm 3 Concept dispersion
dispersion factor r. 2: Calculate the mean concept embedding d Σi d:/l)>. dill2.
Input: {d
3: for each d; do
4: Decompose d; d, = cos sin e,, where a; E [0, T r/2) ||e:l22 = 1, and e; d.
5: Calculate the dispersed query feature d.ew cos(ra;) sin(ra)) e;
6: Add dhew as the ith column init.
7: end for
Return Dinit
The effectiveness Algorithm 3 is shown in Figure 5, where we compare the correlation between concept embeddings before and after this process across various datasets.
Algorithm 4 presents the pseudo-code for the projection and normalization step in Algorithm 2.
Algorithm 4 Embedding normalization and projection
Input: current query feature matrix D, initial query feature matrix Dinit and radius p.
2: for each query feature d, in D do
3: Normalize d, as d; d./l/d:ll2.
4: if |d, || 2 p then
5: d; = arg mind: a dd-d || [d]|==1 || d:
6: end if
7: end for
8: Return D.
29

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Before Atom Dispersion
After Atom Dispersion
0.0
CIFAR10
1.0
0.8
CIFAR100
1.0
0.8
0.6
0.4
0.2
CUB
3.0
0.8
0.6
0.4
0.2
0.0
ImageNet
3.0
0.8
0.6
04
02
0.0
Places365
Figure 5. We calculate the conrelation matrix (D D) for dictionaries before and after Algorithm 3. and present them in the format of heatmaps. As can he seen, the proposed dispersion process effectively reduces the correlation between concept embeddings generated by CL.P.
30

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Test Accuracy
Average Sparsity
140
120
70
69
25
100
125
150
175
25
50
75
100
125
150
175
200
Average Column Deviation
Max Column Deviation
0.14
0.12
0.5
0.10
0.4
0.08
0.3
0.06
0.04
0.02
0.00
25
50
75
100
125
150
175
200
25
50
75
100
125
150
175
200
CCR
CCR w/o Dispersion
CCR w/o Projection
Figure Ablation study for Algorithm and Algorithm 4.
D.3. Ablation Study
In this section, we present empirical findings from our ablation study. As a preliminary note, we emphasize that the comparison between CCR and its baseline shown in Figure 2 serves as a key ablation analysis for CCR module as whole. Here, concentrate on evaluating the contributions of critical components introduced in the preceding section: Atom Dispersion and Atom Projection. To this end, individually exclude these two steps-Algorithm 3 and Algorithm 4 anand compare the resulting models to the full CCR framework in terms of test accuracy, average sparsity, average column deviation, and maximum column deviation over the course of training.
As illustrated in Figure 6, the omission of either step leads to an improvement in performance. However, these gains incur different trade-offs. Specifically, removing Algorithm results a substantial increase the average sparsity the learned sparse codes. In contrast, eliminating Algorithm causes the concept atoms to drift significantly from their initial CLIP embeddings, thereby compromising the interpretability of the model.
D.4. Hyperparameter Tuning
This section presents an empirical evaluation of two critical ayperparameters-nnnneee hard-threshold parameter À and the radius bound p-which significantly influence the performance of Algorithm The corresponding results are illustrated in Figure 7 and Figure 8.
As depicted in Figure 7, increasing the value of À results in sparser explanations by reducing the number of activated concepts. However, this sparsity comes at the cost of reduced test accuracy. Therefore, selecting appropriate threshold entails balancing the trade-off between prediction accuracy and the level of explanation sparsity-a consideration of particular relevance explainable Al applications.
important note that including more concepts does not inherently ensure improved accuracy. This point is exemplified
31

---

Enhancing Performance of Explainable AI Models with Constrained Concept Refinement
CIFAR10
CIFAR100
CUB
100-
Average Sparsity
33-
Average Sparsity
60-
Average Sparsity
30-
53
40
48
30-
23
40
15
20-
0.10
0.21
0,82
0.35
0.34
0.35
0.16
6.18
0.18
0.20
0.22
0.24
0.00 Threshold A
Threshold A
Threshold À
Test Accuracy
Test Accuracy
74-
Test Accuracy
67.3
11.00
72-
05.0
12.79
68-
00.0
69-
11.75
55.0
6.32
5.11
Threshold À
6.18
0.38
Threshold À
0.20
0.22
0.24
3:5
0.18
Threshold À
0.20
6.22
224
Figure 7. Average sparsity and test accuracy for varying thresholds A.
in the left column of Figure 8. which shows that expanding the search radius enables the CCR algorithm identify solutions that are simultaneously sparser and more accurate. Furthermore, the empirical results suggest that the performance gains of CCR saturate beyond > 0.1, indicating that 0.1 serves as suitable choice for this hyperparameter.
D.5. More Experiments on Interpretability
In this section, we expand on the discussion of CCR's 's interpretability from Section 4.2 by conducting a comprehensive case study across all five datasets used to evaluate CCR. For each dataset, we select three representative image-label pairs concept scores on the left, alongside their corresponding weights associated with
and present the ten highest-rankine the predicted label on the right. To facilitate a comparison of explainability with and without CCR, we also report the corresponding results for the baseline model, which obtained by setting TJD = Algorithm 2, presented the end of this section. Based on our case study, we derive several key observations:
Semantic correlation. In most cases, concepts with higher concept scores exhibit a strong semantic correlation with the input images. This finding substantiates the reliability of our algorithm as an interpretable Al model. A notable example illustrated in Figure 16 and Figure 17, where the algorithm effectively identifies the primary distinguishing features between two visually similar bird species their coloration-leadtee leading to accurate classification.
Weight distribution. In instances where misleading elements are present in an image, certain concepts unrelated to the ground truth label may receive high concept scores. Nevertheless, the linear layer appropriately assigns a small weight to such concepts. These weights encapsulate the algorithm's 's interpretation of the label, thereby rendering it comprehensible to human users. Examples of such cases include "a beak" in Figure 10, "a plow" in Figure 12, "a seagull"/'a bird' in Figure 14, "a highlighter" in Figure 18, "iridescent" in Figure 19, and "abandoned buildings"/'a sunset" in Figure 21.
Debugging capability for incorrect predictions. Inevitably, framework makes incorrect predictions for some samples. A key application of explainable AI is to help human experts understand the reasoning behind these emrors. Our findings indicate that the proposed algorithm effectively captures interpretable misconceptions introduced by encoders. For instance, in Figure 22, concepts such as "a display case" and "exhibits" suggest that the primary misleading factor causing the model to predict "natural history museum" instead the correct label "shopping mall" is the manner in which the store displays its goods. This insight is valuable for human intervention and model correction.
32

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
CIFAR10
CIFAR100
CUB
16.2
Average Sparsity
Average Sparsity
Average Sparsity
14.0
25
20-
12-
18.4
152
10
0.00
0.05
0.10
0.55
0.00
0.25
0.50
000
9.05
0.10
0.25
0.40
0.25
0.80
Radius Bound P
Radius Bound P
Radius Bound P
Test Accuracy
Test Accuracy
70
Test Accuracy
01-
55-
64-
10
40-
0.05
0.50
0.55
0.30
0.15
0.20
0.30
0.03
0.30
0.25
0.20
0.30
Radius Bound p
Radius Bound p
Radius Bound p
Figure 8, Average sparsity and test accuracy varying radius bounds p.
Reliance on the richness of the concept set. The effectiveness of the algorithm inherently influenced by the quality of the concept set, phenomenon that aligns with fundamental motivation behind explainable AI. As demonstrated in Figure 23, the concept set lacks distinctive features that differentiate between "bus interior" and "train interior". Consequently, the model is unable to distinguish between these two labels.
33

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
four wheels
driver
beetle
gear
a toy
cab for the driver large, rectangular body
pointed transportation
True label: automobile Predicted label: automobile
steering wheel
0.4
0.3
0.2
0.1
2
10
Figure CIFAR-10 (a)
Concept Scores
the ability
landing gear
wings
pilot
copilot
beak
True label: airplane
birdhouse
Predicted label: airplane
rudder
0.4
0.3
0.2
0.1
10
Figure 10. CIFAR-10 (b)
Concept Scores
Weights
watercraft
port
large sails or engines
a boat
captain
rudder
vessel
True label: ship Predicted label: ship
engines
pointed end
Figure 11. CIFAR-10 (c)
34

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Concept Scores
Weights
tractor trailer
a vehicle
four wheels
plow
trailer
military vehicle
mode of transportation
toy
motor
True label: pickup truck
0.1
Predicted label: pickup truck
Figure 12. CIFAR- 100 (a)
Concept Scores
Weights
furniture
a soft, upholstered surface
comfortable design
backrest
pillow
armrests on either side
headboard
home
True label: couch Predicted label: couch
desk
Figure 13. CIFAR-100 (b)
Concept
Weights
shark fin large, triangular fin
harpoon
a dorsal fin
seagull
large, gray body
forked
a bird
True label: shark Predicted label: shark
metal or plastic body
0.4
0.3
0.2
0.1
Figure 14. CIFAR- 100
35

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Weights
dark blue-black upper
large black bird
dark, glossy bill small, black bird
long, black wings
dark coloration
slimmer body than most crows
All-dark plumage
beak and legs black
black back
bronzed cowbird Predicted label: bronzed cowbird
True label:
Figure 15. CUB-200 (a)
Concept Scores
Weights
white head and neck
gray head
grayish-brown large, stocky bird white plurnage duck-like
thin, black bill
black or dark grey plurnage
True label: black footed albatross Predicted label: black footed albatross
long, neck
Figure 16. CUB-200 (b)
Concept Scores
Wrights
seabird
white plumage
white rump
large, white body
white belly
large, stocky bird
thin,
black yellow
True label: Laysan albatross Predicted label: Laysan albatros
medium-sized duck
Figure 17. CUB-200 (c)
36

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Concept
Weights
lime
long, green body
the ability change color
repiliann eyes
green yellow eyes
a lizard like body
often has railing
walking stick
perch or ledge tetrapod
red or green color
highlighter
64
83
True label: African chameleon Predicted label: African chameleon
Figure 18. ImageNet (a)
fish
fish tank
other fish
translucent
fish food
floating in the sky
clear translucent color orange fins iridescent
True label: goldfish Predicted label: goldfish
the ability change color
Figure 19. ImageNet
Welghts
Gras parade
wheels with hubcaps
packing material
made of cardboard
wheels roller brush
rolling wheels
carton of eggs
True label: toilet tissue
Predicted label: car wheel
Figure 20. ImageNet (c)
37

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Conceet Scores
Weights
school
a large, red building abandoned buildings central quad lawn
large, industrial building
Institution
scenic location
building
True label: campus Predicted label: campus
facility
0.4
0.3
0.2
0.1
Figure 21. Places365 (a)
Weights
display case
exhibits
lots of scientific equipment
wide variety of exhibits large selection of toys enclosure
collection of materials
collection
True label: shopping mall Predicted label: natural history museam
display
an exhibit
Figure 22, Places365 (b)
Concept Scores
Woights
train
vacation
fans
way
bus
train
overhead compartments
tourists
usually in ship train
True label: bus interior
transportation
Predicted label: train interior
0.4
0.3
0.2
0.1
Figure 23. Places365 (c)
38

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Concept Scures
Weights
large room high rellings
place entertain g guests
wooden metal frame
small, recessed area
lighting fixtures tiered or sloped seatinng a large, indoor space chairs or couches for seating large, plate glass windows fronz desk check-in
True label: dining room
Predicted label: dlining room
0.3
Figure 24. Baseline comparison to the top figure in Figure 3
Concept
Weights
golf club
flag
country
people wear special clothing
course
golf
designated fairways greens
player
players
True label: golf course
spectators
Predicted label: golf course
04
0.3
Figure 25, Baseline comparison to the bottom figure in Figure 3
Weights
school
facilitty
large, industrial building
building
barracks
large buildings a large, red building
central quad lawn abandoned buildings
True label: campus Predicted label: campus
outdoor location
Figure 26, Baseline comparison to Figure 21
39

---

Enhancing Performance of Explainable AT Models with Constrained Concept Refinement
Concept Scores
Weights
display
exhibits
wide variety of exhibits large selection of toys lots of scientific equipment
display medical equipment an exhibit enclosure used for burial
True label: shopping mall Predicted label: natural history museam
0.4
0.2
01
Figure 27. Baseline comparison to Figure 22
Concept Scores
Weights
vacation
bus
way
fans
train
students
tourist destination
action
True label: bus interior
transportation
Predicted label: bus interior
Figure 28. Baseline comparison to Figure 23
40


Reviews:
[{"review_id": "Cu5Mxz3VTK", "reviewer": "Reviewer_LpAZ", "summary": "The paper proposes CCR to improve both prediction accuracy and model interpretability.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time and effort invested in evaluating our manuscript. Should the reviewer find our clarifications and revisions satisfactory, we would be grateful if they would consider raising their score in the final assessment.\n\n**Claims And Evidence**\n\nWe apologize for not including the impact statement of the paper. We will include the following section in the final manuscript:\n\nConclusion and Impact Statements\n\nThis paper introduces Constrained Concept Refinement (CCR), a principled framework that helps bridge the long-standing gap between interpretability and accuracy in machine learning. By constraining the refinement of concept embeddings to lie within a small neighborhood around their initial values, CCR enables interpretable-by-design models to improve prediction performance without compromising their explainability. Our theoretical and empirical results demonstrate the effectiveness of CCR across a variety of tasks and datasets, both in terms of predictive performance and computational efficiency.\n\nThe broader impact of this work lies in its potential to advance the practical adoption of interpretable machine learning methods in real-world settings. In particular, the computational efficiency afforded by CCR may facilitate the deployment of explainable artificial intelligence (XAI) techniques in resource-constrained environments.\n\nFrom an ethical perspective, the capacity to generate explanations that are faithful, stable, and aligned with human-interpretable concepts contributes to addressing critical concerns related to algorithmic accountability and bias. For the proposed method, hyperparameter tuning remains a crucial yet underexplored component for ensuring effective performance in practical applications. This aspect warrants careful consideration to ensure that the resulting outputs are both reliable and justifiable.\n\n**Methods And Evaluation Criteria**\n\n> metrics for interpretability like...\n\nWe thank the reviewer for the constructive feedback. To address the comment, we conducted additional experiments to better illustrate the relationship between performance and sparsity. See the Plot 1 from this [anonymous link](https://anonymous.4open.science/r/ICML_Review-C283).\n\n> Why is CIFAR-10 curve ...\n\nThe main reason for the lack of obvious improvement on the CIFAR-10 dataset is that we report test accuracy **after** each epoch. Since CIFAR-10 is relatively easy to train, its performance potential is largely realized after a single pass through the training data (i.e., after the first epoch). If we instead report test accuracy **before** each epoch, a clear improvement becomes apparent after the first epoch. This improvement is shown in the Plot 2 of this [anonymous link](https://anonymous.4open.science/r/ICML_Review-C283).\n\n> I wonder if the theoretical ...\n\nWe thank the reviewer for raising this important point. Indeed, the concept of interpretability remains elusive from a theoretical perspective, and there is currently no consensus in the community on how to rigorously quantify it. As a result, existing works typically take one of two approaches: (i) relying on human annotations and evaluations to assess interpretability, while omitting theoretical guarantees [1-4]; or (ii) studying a well-defined generative model, where samples are drawn from an underlying ground-truth concept embedding, allowing interpretability to be quantified as the distance between the learned and true concept embeddings. Our work incorporates both of these perspectives.\n\nAdmittedly, the generative model we consider is not fully general, and its assumptions may not hold in real-world settings. However, as we clearly state in the paper, our motivation for studying this model lies elsewhere: it provides a controlled test-bed in which the limitations of existing methods can be rigorously examined, and our proposed approach can be shown to overcome them effectively. Moreover, this generative model has been extensively explored in prior work. For instance, [1] uses a variant of it as a pragmatic compromise between theoretical tractability and empirical relevance. It is also closely related to a well-established line of research in dictionary learning, where similar generative models have been successfully applied. \n\nReference:\n\n[1] A Chattopadhyay, R Pilgrim,R Vidal. Information maximization perspective of orthogonal matching pursuit with applications to explainable AI.\n\n[2] B Zhou, Y Sun, D Bau, A Torralba. Interpretable basis decomposition for visual explanation.\n\n[3] PW Koh, T Nguyen, YS Tang, S Mussmann, E Pierson, B Kim, P Liang. Concept bottleneck models.\n\n[4] M Yuksekgonul, M Wang, J Zou. Post-hoc concept bottleneck models\n\n[5] T Oikarinen, S Das, L Nguyen, T Weng. Label-free concept bottleneck models", "claims_and_evidence": "It does not have the impact statements and ends abruptly. Seems like a work in progress?", "methods_and_evaluation_criteria": "- metrics for interpretability like avg explanation length and avg concept embedding deviation are proxy measures. It remains unclear how well these reflect human understanding. There is a better figure on this in the Information Maximization Perspective of Orthogonal\nMatching Pursuit with Applications to Explainable AI by Chattopadhyay et al. paper.\n- Why is CIFAR-10 curve in Figure 2 for baseline flat? Puts into question if this was a fair comparison.\n- I wonder if the theoretical analysis depends on overly strong assumptions that may not generalize", "theoretical_claims": "Not well enough.", "experimental_designs_or_analyses": "See methods and eval criteria", "relation_to_broader_scientific_literature": "Seem very tightly tied to Information Maximization Perspective of Orthogonal Matching Pursuit with Applications to Explainable AI by Chattopadhyay et al. including paper structure.", "essential_references_not_discussed": "Main papers that come to mind would probably be outside the scope of this paper", "other_strengths_and_weaknesses": "See above", "other_comments_or_suggestions": "Missing Impact Statements.", "questions_for_authors": "See other sections", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SzbiY5rxfQ", "reviewer": "Reviewer_1v3q", "summary": "This paper addresses the challenge of balancing accuracy and interpretability in machine learning models, particularly for interpretable-by-design methods that often sacrifice accuracy for transparency. The authors identify that deviations in concept representations, a crucial component of interpretable models, can negatively impact prediction performance. To counter this, they propose a novel framework called Constrained Concept Refinement (CCR) that optimizes concept embeddings under constraints to preserve interpretability. CCR significantly improves the prediction accuracy of interpretable models like IP-OMP on image classification tasks (CIFAR 10/100, ImageNet, CUB200, and Places365) while maintaining interpretability and achieving a reduction in runtime compared to other explainable methods.\n\n## update after rebuttal\nI appreciate the thoughtful and detailed responses from the authors, which successfully addressed all my concerns. Consequently, I have decided to increase my score for the paper, as I now believe it holds significant potential to contribute meaningfully to the field of explainable AI. I also hope that, if the paper is accepted, the final version will incorporate the additional results and address reviewer suggestions to further strengthen its impact.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are sincerely grateful to the reviewer for their thorough evaluation and for the positive feedback on our work. In response to their suggestions, we have proposed additional experiments and provided further clarifications in the revised manuscript. We hope that our responses effectively address the concerns raised and that the reviewer would consider raising the score in the final assessment.\n\n**Assumption of Column-Orthogonality and Full Rank**\n\nWe thank the reviewer for this insightful comment. We divide our response into two parts:\n\nFull-rank condition: This condition is fundamental and cannot be relaxed; it is not an artifact of our theoretical analysis. In particular, when the dictionary is not full-rank, our analysis shows that $\\mathbf{D}$ may deviate from $\\mathbf{D}^*$ in directions that are column-orthogonal to the column space of $\\mathbf{D}^*$. Such deviations cannot be corrected by the algorithm. This behavior is further corroborated by numerical experiments. In Appendix D.1, we present synthetic data experiments that highlight the necessity of the full-rank condition. Specifically, the middle column of Figure~4 corresponds to a rank-deficient $\\mathbf{D}^*$, and the results show that, in the absence of the full-rank assumption, the algorithm fails to achieve column-wise convergence. This outcome is fully consistent with the theoretical guarantees established in our main theorem.\n\nOrthogonality condition: In practical scenarios where the dictionaries are not orthogonal, we propose a novel procedure, termed *Concept Dispersion*, designed to mitigate the impact of highly correlated embeddings. The details of this sub-algorithm are provided in Appendix D.2. The effectiveness of it is shown in the Plot 3 from this [anonymous link](https://anonymous.4open.science/r/ICML_Review-C283), where we compare the correlation between concept embeddings before and after this process.\n\n**Dependency on Pre-trained Models**\n\nWe would like to clarify that the introduction of CCR is specifically intended to address the errors, biases, and uncertainties that arise from CLIP embeddings. The reviewer is absolutely correct in noting that CLIP embeddings are often far from accurate. This observation is, in fact, the primary motivation behind the development of CCR in our work. In particular, we demonstrate—both empirically and theoretically—that while existing methods rely solely on the raw outputs of CLIP, without any mechanism for refining the embeddings, our proposed CCR framework not only leverages CLIP-derived embeddings but also further refines them to achieve strong performance within each specific domain.\n\n**Interpretability Preservation**\n\nWe agree with the reviewer that large-scale human evaluation experiments would be valuable in demonstrating the interpretability of our method. This is indeed a limitation—not only of our work, but also of most existing efforts aimed at developing interpretable machine learning models. Addressing the reviewer’s comment and incorporating large-scale human evaluation would require substantial resources, which are currently beyond our reach. In the absence of such resources, our current interpretability evaluation is based on significantly smaller scale human evaluations on a set of randomly selected images, for which we report the model’s outputs and qualitatively demonstrate their interpretability. We are grateful for the reviewer’s constructive suggestion and will strive to incorporate more comprehensive evaluation practices in future work.\n\n**Hyperparameter Choice**\n\nFor an empirical sensitivity analysis of $\\rho$, we refer to the Plot 4 from this [anonymous link](https://anonymous.4open.science/r/ICML_Review-C283).\n\n**Concept Set Dependency**\n\nOur primary motivation for adopting the concept set generated by GPT-3 is to ensure fair comparisons between our method and existing approaches, as both lf-CBM and IP-OMP utilize the same concept set. The concern raised by the reviewer is indeed valid, as the quality of the concept set can significantly influence model performance—a consideration that applies to all interpretable-by-design algorithms. While the development of improved concept sets is undoubtedly important, we regard this as beyond the scope of the current work and leave it as a direction for future research.\n\n**Interpretability Metrics & Limited Analysis of Misleading Concepts**\n\nPlease see our response above for Interpretability Preservation.\n\n**Statistical Significance Test**\n\nWe thank the reviewer for their valuable suggestions. to better showcase the robustness of our method, we have enhanced our experimental protocol by performing 10 independent runs for each dataset and reporting the maximum, minimum, and average performance. See Plot 5 from this [anonymous link](https://anonymous.4open.science/r/ICML_Review-C283).\n\n**Ablation Studies**\n\nWe thank the reviewer for this valuable suggestion and will include a ablation study in the revised manuscript.", "claims_and_evidence": "The claims made in the submission are generally supported by clear and convincing evidence, but several key assertions have limitations or rely on restrictive assumptions.\n\n1. Assumption of Column-Orthogonality and Full Rank : The theoretical guarantees rely heavily on assumptions such as column-orthogonality and full-rank latent feature matrices (e.g., Assumption 3.2). These assumptions may not hold in realistic scenarios where concept embeddings often exhibit correlations or redundancy. The authors acknowledge this limitation briefly but do not empirically investigate how violations of these assumptions affect performance in practice.\n\n2. Dependency on Pre-trained Models (CLIP) : The method relies significantly on pre-trained CLIP embeddings, which introduce biases from large-scale internet data used for training CLIP. The authors acknowledge potential biases but do not empirically analyze how these biases might impact CCR's reliability across diverse datasets or domains.\n\n3. Interpretability Preservation : While qualitative examples (Figures 3, 5–19) show interpretable concepts, there is no quantitative metric for interpretability (e.g., human evaluation scores or concept alignment metrics). Interpretability is subjective, but the paper relies on only ad-hoc visualizations without rigorous validation.\n\n4. Hyperparameter Choice : The paper mentions the correction radius ρ as a crucial hyperparameter that controls the trade-off between accuracy and interpretability. The process for selecting the optimal value for this and other hyperparameters is not detailed extensively, and the sensitivity of the results to these choices could be further explored. Also, optimal ρ likely varies across datasets, but the paper provides no guidance for tuning it.\n\n5. Concept Set Dependency : Performance and interpretability depend heavily on the predefined concepts, which are treated as a black box. Here the concept set is generated using GPT-3, which may introduce bias (e.g., cultural or linguistic limitations). The paper does not analyze how concept quality affects CCR.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in the paper are generally well-suited to the problem of improving explainable AI (XAI) models, particularly in interpretable image classification. However, there are areas where the choices could be questioned or improved for broader applicability and robustness.\n\n1. Interpretability Metrics: While AEL, ASR, and ACED are useful metrics, they do not fully capture human-centric notions of interpretability. For example, there is no user study or human evaluation to validate whether the explanations generated by CCR are genuinely understandable or actionable for end-users.\n\n2. Limited Analysis of Misleading Concepts: Although qualitative examples highlight CCR's ability to handle misleading concepts (e.g., assigning low weights to irrelevant features), a systematic analysis of failure cases or edge cases is missing.", "theoretical_claims": "The theoretical claims are largely correct within their stated assumptions and provide valuable guarantees for CCR's performance under idealized conditions. However, some assumptions (e.g., orthogonality, full rank) and practical considerations (e.g., sample complexity) limit their applicability to real-world scenarios. The proofs for Theorems 2.6, 3.3, and 3.4 are mathematically rigorous to me, but I did not check them thoroughly.", "experimental_designs_or_analyses": "The experimental designs and analyses in this paper are generally sound and appropriate for evaluating CCR in image classification tasks. However, there are some limitations that should be addressed to strengthen the validity and generalizability of the findings.\n\n1. The paper does not report statistical significance tests for its results (e.g., accuracy improvements over baselines). This omission makes it difficult to assess whether the observed improvements are robust across different runs or datasets.\n\n2. The paper could benefit from ablation studies to understand the contribution of different components of the CCR framework to the overall performance improvement.", "relation_to_broader_scientific_literature": "The paper contributes to the field by addressing a key challenge in interpretable AI, building upon existing interpretable-by-design methods, focusing on the crucial role of concept embeddings, proposing a novel constrained optimization framework with theoretical guarantees, and demonstrating its practical effectiveness in a relevant application domain.", "essential_references_not_discussed": "The authors has added critical references or related works that are essential to understanding the context of its key contributions.", "other_strengths_and_weaknesses": "The paper makes significant contributions to explainable AI by introducing Constrained Concept Refinement (CCR), which improves accuracy and interpretability while maintaining computational efficiency. Its originality lies in directly optimizing concept embeddings within constrained neighborhoods, a departure from prior methods that rely on additional black-box modules.\n\nHowever, the paper's generalizability beyond image classification tasks, reliance on pre-trained models like CLIP, lack of human-centric interpretability validation, strong theoretical assumptions, and limited analysis of failure cases are notable weaknesses that should be addressed.", "other_comments_or_suggestions": "I don't have any specific comments or suggestions for the authors beyond what I already mentioned.", "questions_for_authors": "Despite all these limitations mentioned before, the paper represents a substantial step forward in interpretable machine learning research, offering both theoretical insights and practical improvements that could inspire further advancements in the field. Hence I am leaning towards accepting this paper. But addressing these weaknesses would enhance its impact and generalizability, and improve the chance of acceptance from my side.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1bViCx1QsR", "reviewer": "Reviewer_b4dR", "summary": "In this paper, the author has proposed an improvement for the original structure of the concept bottleneck models and CLIP-IP-OMP models for improving the interpretability and trustworthiness of the model without sacrificing the model's accuracy. The key idea of this method is trying to optimize the concept embeddings while keeping them within a constrained range, which addresses the issue of the inaccurate query features that are generated from the dataset like CLIP. The key contributions of this paper include the theoretical justifications of the necessity of the problem, the accuracy and interpretability guarantees, and empirical experiment results that show its performance compared to the existing methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their detailed feedback and positive assessment. We hope that our clarifications and additional experiments could further elevate their recommendation of our work in the final review.\n\n**Claims And Evidence**\n\nWe would like to offer two clarifications that may help resolve the confusion expressed by the reviewer:\n\n1. In this work, we evaluate the performance of CCR in two distinct settings: the theoretical model proposed by [1], and the interpretable image classification task. Interpretability can be quantitatively assessed only in the former setting; for applications such as image classification, the community currently lacks universally accepted metrics for interpretability. As a result, most related works instead rely on post-hoc, case-specific analyses to illustrate interpretability [2,3,4,5]. These analyses typically involve examining the predicted output and the most salient concepts identified by the model, and assessing whether these concepts provide a human-interpretable explanation for the output. Accordingly, our claim of *improved interpretability* applies solely to the theoretical setting. In both settings, it is important to note that sparsity does not equate to interpretability. We have proposed a revision to the manuscript to make this distinction explicit; please refer to our response to Reviewer grrY for further details.\n\n2. In the theoretical setting, the input $\\mathbf{x}$ is generated as a sparse linear combination of the columns of ground truth (but unknown) $\\mathbf{D}^*$. Therefore, perfect interpretability in this setting corresponds to the exact recovery of $\\mathbf{D}^*$, and not sparsity. In this simplified setting, improvements in accuracy and interpretability are aligned, as demonstrated by Theorem 3.4.\n\n**Experimental Designs Or Analyses**\n\n[Sparsity and $\\rho$ trade-off]: We have included plots that shows the relationship between the performance/sparsity and $\\rho$ for CIFAR-10 dataset through the following [anonymous link](https://anonymous.4open.science/r/ICML_Review-C283) (Plot 4 and 6). We will add the same experiment for all other datasets in the revised manuscript.\n\n[Missing baselines Metrics]: As mentioned above, sparsity is not used as a metric for interpretability in this work. Accordingly, metrics such as ASR, AEL, and ACED are not intended to reflect interpretability, but rather to highlight the simplicity and efficiency of our method relative to existing approaches. These metrics demonstrate that our method significantly reduces computational cost compared to state-of-the-art methods, while simultaneously achieving improved accuracy. Nonetheless, we appreciate and agree with the reviewers' suggestion that the interpretability of the baseline models needs examination. Accordingly, we will include a corresponding analysis—similar in nature to Appendix D.3—for the baseline models in the revised manuscript.\n\n**Other Strengths And Weaknesses**\n\nWe appreciate the reviewer's insightful observation regarding the connection between our problem and robust regression. Using Lasso as an illustrative example, it is indeed equivalent to solving a sparse coding problem, where the goal is to represent a signal $\\mathbf{x}$ using a small number of columns from a dictionary matrix $\\mathbf{D}$. \n\nNevertheless, there are two key distinctions between our setting and that of Lasso:\n\n1. Similar to sparse coding problem, in Lasso, it is assumed that $\\mathbf{D}$ is fixed and known, and the primary goal is to promote sparsity in the reconstruction. In contrast, our setting does not assume that $\\mathbf{D}$ is fixed; rather, we treat the given $\\mathbf{D}$ (typically obtained from CLIP) as uncertain and potentially erroneous. Our objective is to co-optimize both $\\mathbf{D}$ and the reconstruction. This formulation is more closely related to dictionary learning, a connection we discuss extensively in the paper.\n\n2. The primary objective of Lasso is to identify a sparse code $\\mathbf{s}$ such that $\\|\\mathbf{x} - \\mathbf{D}\\mathbf{s}\\|_2$ is minimized. In our framework, however, the emphasis is not solely on reconstruction accuracy, but on using the resulting $\\mathbf{s}$ to improve performance on downstream tasks.\n\n[1] A Chattopadhyay, R Pilgrim,R Vidal. Information maximization perspective of orthogonal matching pursuit with applications to explainable AI.\n\n[2] PW Koh, T Nguyen, YS Tang, S Mussmann, E Pierson, B Kim, P Liang. Concept bottleneck models.\n\n[3] M Yuksekgonul, M Wang, J Zou. Post-hoc concept bottleneck models\n\n[4] T Oikarinen, S Das, L Nguyen, T Weng. Label-free concept bottleneck models\n\n[5] A Chattopadhyay, K Chan, B Haeffele, D Geman, R Vidal. Variational information pursuit for interpretable predictions.", "claims_and_evidence": "In this paper the authors provides several claims and are mostly providing theoretical proofs that shows the necessity of concept refinement, and the CCR overcomes the problem. However, I am a little bit concerned about the authors' claim of improving the interpretability statement using CCR. \n\nI understand the author's claim about enhanced interpretability through the introduction of the function $\\rho$ in the model. However, for a concept bottleneck model or the similar concept-based models, the original input space/model has already provided sparse explanations. In other words, the output that the models like CILP provide is already sparse, so with a modification of $\\Delta D$, it cannot guarantee that the level of interpretability (I am not sure if that's means sparsity in this case) is changed. I would like to see if the authors can better define what interpretability means here, as I can see the introduction of the residual part more correlated to the definition within the robust regression [1]. I can see that the results/definition is aiming for a robust solution to me, rather than saying that it is interpretable, and the related works that the authors states in line 210-219 about the sparse encoding also doesn't provides me an explanations on why the closer to the $D^*$, the more interpretable it would be. If the authors can provide me more explanations on that, that would really strengthen the claim. \n[1] Xu, Huan, Constantine Caramanis, and Shie Mannor. \"Robust regression and lasso.\" _Advances in neural information processing systems_ 21 (2008).", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem and applications. I am a little bit concerned about the interpretability metrics that the authors used both in the paper and the criteria, as they are mostly the case study instead of quantitative metrics. I have more comments in the experiment design sections for the concerns about the methods comparison.", "theoretical_claims": "I did check the correctness of the proofs for the theoretical claims. I have no specific questions for them.", "experimental_designs_or_analyses": "I think the proposed method CCR, is well-motivated for the problem of improving the accuracy of interpretable AI model while preserving its ability to explain the decisions. However, I am several concerns about the design of the experiments:\n\n[Sparsity and $\\rho$ trade-off]  In CCR, the constraints $\\rho$ determines how the concepts embeddings can be refined. Therefore, it is actually risky when the $\\rho$ goes larger. Therefore, I am wondering if the authors have experiments to show the relationship between the sparsity and the selection of the $\\rho$ is real-world applications since it seems a bit hard to tune comparing to setting a strict top $k$ in the sparse autoencoder settings. I am wondering if the authors can provides more guidelines on how the accuracy and sparsity ($ASR$) have changed through the change of $\\rho$. \n\n[Missing baselines Metrics] In the experiments in table 1, the authors have shown the average explanations length, sparsity, and the concept embedding deviations for the CCR, However, there missed a baseline here on how without CCR would influence those metrics. As the results without CCR is already sparse, so it is not the case that we can regard 100% as the baseline. I am wondering if the authors could provide the baseline/comparisons for other methods as a guidelines in the table so that we can see how the ASR/AEL/ACED have changed through the optimization. How much it has sacrificed/improved from the sparsity level perspective. This comparison missing in the appendix sesssion where it is hard to tell the improvment for interpretability in Appendix D.3.", "relation_to_broader_scientific_literature": "The contributions of the paper related to the broader scientific literature include:\n\n[Concept-based Model Construction] The paper builds on Concept Bottleneck Models (CBMs)[1], which introduced the idea of human-interpretable intermediate representations. In CBMs, models first predict a set of predefined datasets and then use them to make final decisions. However, a major issue with CBMs is their reliance on imperfect concept annotations, leading to performance degradation when concepts are noisy or biased. CCR directly addresses this issue by refining concept embeddings while ensuring they remain interpretable, whereas CBMs typically assume fixed, pre-defined concepts.\n\n[Theoretical proofs for the misaligned/adversarial attack leading to the high performance drop] Prior work [2] showed that explainability methods are often fragile to small changes in feature representations. CCR mitigates this fragility by ensuring that concept refinements remain interpretable.\n\n[1] Koh, Pang Wei, et al. \"Concept bottleneck models.\" International conference on machine learning. PMLR, 2020.\n[2] Ghorbani, Amirata, Abubakar Abid, and James Zou. \"Interpretation of neural networks is fragile.\" Proceedings of the AAAI conference on artificial intelligence. Vol. 33. No. 01. 2019.", "essential_references_not_discussed": "I don't think that there are essential references that are not discussed, but I have mentioned in the other strengths and weaknesses that asking about if the references about the robust regressions and their strategies should be included as the related works for the paper, as the problem statements seem to be similar to them.", "other_strengths_and_weaknesses": "I am a little bit concerned about the strategies that the author has proposed that can be aligned with the robust regression fitting problem [1]. There are several current works for using the perturbation idea for solving similar issue but not just to the concept-based problems [2].\n[1] Xu, Huan, Constantine Caramanis, and Shie Mannor. \"Robust regression and lasso.\" _Advances in neural information processing systems_ 21 (2008).\n[2] Su, Peng, et al. \"CR-lasso: Robust cellwise regularized sparse regression.\" Computational Statistics & Data Analysis 197 (2024): 107971.", "other_comments_or_suggestions": "No extra comments.", "questions_for_authors": "If the authors can provide better explanations/definitions for the interpretability before and after the concept refinement, that would better show the usefulness of this approach.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sfolRhBB0j", "reviewer": "Reviewer_grrY", "summary": "This paper proposes to tackle the potential gap between performance and understandability of concept embedding models, but allowing the concept embeddings to be refined in a constrained manner, thereby allowing for more flexibility, but still understandability. The authors have proposed a theoretical framework embedded in the IP-OMP framework and ran several experiments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the time and effort dedicated to reading our manuscript and for providing valuable and constructive feedback. In response to the concerns raised regarding the clarity and coherence of the paper’s central ideas, we propose to make major revisions to the manuscript. The details of these modifications are outlined in the subsequent response. We hope that these changes will enhance the readability of our work and allow the reviewer to better appreciate the contributions presented in this paper.\n\n**Claims And Evidence**\n\nIn response to the reviewer’s comments and upon careful reconsideration, we acknowledge that there exists a logical gap between the introductory section and the theoretical exposition, which may hinder the reader’s ability to follow the overall narrative. At a high level, the reviewer's understanding is accurate: *the principal contribution of this paper is that we propose refining the representations of concepts in interpretable AI models within a constrained deviation, thereby enabling the resulting model to achieve a more favorable trade-off between performance and interpretability.*\n\nTo address the reviewer’s suggestions and improve the clarity of the manuscript, we will include the following paragraph at the end of the first section:\n\nOn a high level, our main contribution is the introduction of a framework in which concept embeddings in explainable AI models are refined within a restricted parameter space to attain a better balance between predictive performance and interpretability. The remainder of the paper substantiates this claim along two key dimensions:\n\n1. **Theoretical validation.** We demonstrate that our method is both **necessary** (Section 2.1) and **effective** (Section 3) under the theoretical framework introduced by [1], whose background is reviewed in Section 2. This framework serves as a suitable testbed for our contributions for two main reasons: (1) Among interpretable-by-design models [2-6], only [1] presents a generative model wherein both performance and interpretability are rigorously defined. (2) The algorithm motivated by this generative framework achieves state-of-the-art results within the class of interpretable AI methods.\n    \n2. **Empirical evaluation.** We conduct experiments on multiple benchmark datasets for image classification tasks to assess the practical effectiveness of our approach (Section 4).\n\n**Relation To Broader Scientific Literature**\n\nWe thank the reviewer for their constructive suggestions regarding the improvement of the literature review section. We would also like to respectfully clarify that, given the close relationship between the motivation of our work and prior literature in explainable AI (XAI), we dedicate the majority of Section 1 to introducing the relevant background. Appendix A serves as a complementary component, offering a more in-depth discussion of works that are particularly closely aligned with our approach. Upon careful review, we believe that we have sufficiently highlighted and discussed the related literature in the areas of XAI, concept learning, and dictionary learning. If the reviewer has specific papers in mind, we would be happy to consider including them in the revised manuscript.\n\n**Other Comments Or Suggestions**\n\nBased on the reviewer's suggestion, we will add an additional section by the end of the paper in the revised manuscript. Due to the character limitations for each rebuttal, please see the first section in our response to Reviewer LpAZ for the full text.\n\nReference:\n\n[1] A Chattopadhyay, R Pilgrim,R Vidal. Information maximization perspective of orthogonal matching pursuit with applications to explainable AI.\n\n[2] B Zhou, Y Sun, D Bau, A Torralba. Interpretable basis decomposition for visual explanation.\n\n[3] PW Koh, T Nguyen, YS Tang, S Mussmann, E Pierson, B Kim, P Liang. Concept bottleneck models.\n\n[4] M Yuksekgonul, M Wang, J Zou. Post-hoc concept bottleneck models\n\n[5] T Oikarinen, S Das, L Nguyen, T Weng. Label-free concept bottleneck models\n\n[6] A Chattopadhyay, K Chan, B Haeffele, D Geman, R Vidal. Variational information pursuit for interpretable predictions.", "claims_and_evidence": "Honestly I find it very difficult to even identify the claims of the paper. It starts with an introduction that is well written and easy to follow, though it is somewhat generic in that it does not become clear what the authors are specifically tackling and why this is relevant. Section 2 and section 3 is then pretty much buried in mathematical statements without a high-level explanation of what this is for. The experiments look promising, but their significance falls somewhat short in how they are tied to the authors claims (which I am still not 100% I have identified). I believe the claims are: \"concept embedding models should allow for refining their embeddings, but these should only be refined in a small constrained manner.\" \nIn this context I think having a concrete example throughout the paper would really help understanding the problem and relevance of the problem in the first case, and what the method does. \nAnother question I have is what the authors mean by \"concept embeddings\" related to previous work. It would be good if the authors define this either in the introduction or section 2", "methods_and_evaluation_criteria": "Please see my remarks in the previous box. \nIn addition, it is not clear why the authors frame the problem within the framework of section 2, i.e. \"predicting a random variable by sequentially selecting a list of related random variables, termed queries, and checking their values.\" However, this seems quite central to the work and more clarification would be necessary here.", "theoretical_claims": "I have not been able to check the correctness of the theoretical claims.", "experimental_designs_or_analyses": "This seems fine for me, however I would suggest to move some of the experimental details into the appendix in order to make room for embedding and discussing the experiments more in the context of the authors claims.", "relation_to_broader_scientific_literature": "This is a section that is quite missing in the paper. I really think it would help to move this from the supplements to the main paper. \nBut in addition to this remark, I believe the section is very sparse, the majority of the referenced papers is repeatedly from Chattopadhyay et al.. I consider it important to add references from the field of XAI, concept learning and dictionary learning to increase the breadth here, but also make it more understandable how this work relates to them.", "essential_references_not_discussed": "See above.", "other_strengths_and_weaknesses": "The experimental results look valuable, but it is hard to understand them without understanding the claim and importance of the underlying problem.", "other_comments_or_suggestions": "Please write a discussion and conclusion of your work. This makes it additionally difficult to understand what the paper is about. Overall the authors should rework the overall presentation of their work. What is the goal? Maybe add a running example. How does this relate to previous work? What is the conclusion/what have we learned from each section?", "questions_for_authors": "n/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Constrained Concept Refinement (CCR), a method aimed to improve the predictive performance of interpretable-by-design models without sacrificing interpretability. The core idea is to refine the concept embeddings (typically derived from CLIP or similar pretrained sources) within a constrained neighborhood, thus preserving interpretability while correcting inaccuracies. The authors provide both theoretical analysis under a generative model setting and empirical validation on image classification tasks (CIFAR-10/100, CUB, ImageNet, Places365), demonstrating improved accuracy and runtime efficiency over baselines (IP-OMP and CBMs).

**Strengths**:
* CCR provides a mechanism to refine concept embeddings that are often noisy/
* The theoretical section is grounded in recent work and adds formal justification for the proposed approach, particularly in terms of recovery guarantees under suitable 
* Empirical results are strong, showing consistent accuracy improvements while reducing explanation sparsity and computational cost.
* Rebuttal engagement was excellent—authors provided extensive clarifications, new experiments (e.g., ablations, spectral truncation, concept dispersion), and acknowledged limitations in interpretability evaluation and real-world applicability.

**Reviewer Sentiment**:
* Reviewer 1v3q (Accept) was initially critical but raised their score post-rebuttal after the authors clarified theoretical assumptions and experimental robustness. They still encourage further polish in a revision.
* Reviewers b4dR and grrY (Weak Accept) both acknowledge the value of the contribution but express reservations around presentation clarity (grrY), interpretability definitions (b4dR), and missing baselines or failure analyses. These were substantially addressed during the discussion.
* Reviewer LpAZ (Weak Reject) had a shorter and more skeptical review, flagging concerns around missing impact statements, unclear theory, and potential similarity to prior work. However, these were specifically and constructively handled in the rebuttal, including a newly drafted impact section, extended empirical plots, and a stronger contextualization of the contribution. And the reviewer did not engage with the authors further.

**Limitations and Remaining Issues**:
* The definition and evaluation of “interpretability” remain somewhat underspecified, and no large-scale human evaluation is presented—though this is common in the literature.
* The theoretical guarantees depend on assumptions (e.g., full-rank, orthogonality) that are naturally limited and may not always hold in practice. While the authors discuss this, some readers may still find the gap between theory and practice notable.
* The presentation could benefit from tighter exposition and a clearer narrative arc, which the authors have committed to improving in the final version.


**Summary**

Despite some critiques, this paper offers a nice and timely contribution to interpretable machine learning. It builds on a strong line of recent work, introduces a novel and theoretically motivated refinement strategy, and demonstrates consistent gains in accuracy and efficiency.
Decision: Accept (poster)