Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Authors: Emiliano Penaloza
Tianyue H. Zhang
Laurent Charlin
Mateo Espinosa Zarlenga
Venue: ICML 2025 poster (2025)
Keywords: Concept Bottleneck Models
Interpretable AI
XAI
Primary area: social_aspects->accountability_transparency_and_interpretability

Abstract: Concept Bottleneck Models (CBMs) propose to
enhance the trustworthiness of AI systems by
constraining their decisions on a set of human
understandable concepts. However, CBMs typically rely on datasets with assumedly accurate
concept labels—an assumption often violated in
practice which we show can significantly degrade
performance. To address this, we introduce the
Concept Preference Optimization (CPO) objective, a new loss function based on Direct Preference Optimization, which effectively mitigates
the negative impact of concept mislabeling on
CBM performance. We provide an analysis on
some key properties of the CPO objective showing it directly optimizes for the concept’s posterior
distribution, and contrast it against Binary Cross
Entropy (BCE) where we show CPO is inherently
less sensitive to concept noise. We empirically
confirm our analysis finding that CPO consistently
outperforms BCE in three real-world datasets with
and without added label noise

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Emiliano Penaloza 12
Tianyue H. Zhang 12 Laurent Charlin 23* Mateo Espinosa Zarlenga 4
Abstract
objective that aids CBMs to learn tasks even in the presence of label noise.
Concept Bottleneck Models (CBMs) propose to enhance the trustworthiness of Al systems by constraining their decisions on a set of human- understandable concepts. However, CBMs typi- cally assume that datasets contains accurate con- cept labels-an assumption often violated in prac- tice, which show significantly degrade per- formance (by 25% in some cases). To address this, we introduce the Concept Preference Optimiza- tion (CPO) objective, a new loss function based on Direct Preference Optimization, which effec- tively mitigates the negative impact of concept mislabeling on CBM performance. We provide an analysis on some key properties of the CPO objec- tive showing it directly optimizes for the concept's posterior distribution, and contrast it against Bi- nary Cross Entropy (BCE) where we show CPO is inherently less sensitive to concept noise. We empirically confirm our analysis finding that CPO consistently outperforms BCE in three real-world datasets with and without added label noise. We make our code available Github
CBMs have emerged as a promising solution to the opacity of traditional Deep Neural Networks (DNNs). By leverag- ing human- unaaannndndett concepts as intermediate repre- sentations during inference, CBMs can explain their pre- dictions using high-level concepts (e.g., "has tail", "has whiskers", etc.) relevant to their downstream task (e.g.. "car") "). This hierarchical formulation enables experts inter- acting with a CBM at test time to intervene, or correct, a mispredicted concept and trigger an update to the CBM's output prediction (Shin et al., 2023). With their intervenabil- ity (Marcinkeviës et al., 2024) and interpretability, CBMs are ideal model candidates for high-stakes tasks where veri- fiability is paramount.
Although promising, CBMs come with a significant con- straint: their training requires the set of correct concept annotations for all samples. In practice, to assume that potentially a hundred concepts per datum would be correctly labeled. As a point of reference, recent study concludes that 12% of the ImageNet- 1K animal val- idation images have an incorrect label (and "some classes having > 90% of images incorrect labels"") (Luccioni & Rolnick, 2023). We might expect those percentages to be even higher for concept labels. Further, within real-world domains where CBMs are intended to be deployed, such as healthcare, datasets are inherently noisy (Sylolypavan et al., 2023) and plagued with subjective labels (Wei et al., 2024). Additionally, regardless of the correctness of the labeled data, the training pipeline of CBMs depends on data aug- mentations (e.g., random crops/ffips, see Figure 1) that can obscure concepts. Such a training pipeline makes some con- cept mislabelling inevitable CBMs, potentially affecting them even under optimal labels. Thus, developing CBMs that robust to concept-label noise significantly en- hance their usability in real-world tasks (even when task labels are correct).
unrealistic
1. Introduction
is a well-known adage, etched in the memory of any com- puting student, that "garbage in" leads "garbage out." Yet, machine learning (ML. methods whose success hinges the availability of high-quality labeled data, this consideration is usually left undiscussed. We show that this oversight affects Concept Bottleneck Models (CBMs) (Koh et al., 2020), a popular but label-hungry fam- ily of interpretable neural architectures, when trained with mislabeled concepts. As a remedy, we propose a learning
when designing
Equual Supervision 'Université de Montreal' 2 Mila Québec AI Institute HEC Montréal University of Cambridge. Correspon- dence to: Emiliano Penaloza <emiliano, <cmiliano..enalozaee mula.que
In this work, we take inspiration from the field of Preference Optimization (PO), which relaxes the assumption made in traditional supervised ML that their training data is sampled from the optimal data distribution. PO algorithms only assume preference of the preferred training labels, rather
Proceedings of the nnnd International Conference on Machine I Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
the context of a Concept Bottleneck Model (CBM) (Koh et al., 2020). A CBM is neural architecture composed of (1) concept predictor Te(c x), which maps input features z to a predicted distribution с over a set of pre- defined concepts, and (2) a label predictor So (c), which maps the set of predicted concepts c to a downstream label y. By conditioning its task predictions on set of concepts, CBMs can explain their prediction through their predicted concepts. They also allow for concept interventions, where, at test time, an expert interacting with the CBM can correct handful of its mispredicted concepts leading to significant improvements in task accuracy (Shin et al., 2023).
Training Images
Concents
C.mel
Book Coloe
Vallow
Orange
Thrast Color
White
White
White
Randon Shin
Shape
Bosk
Unrved
Booked
Hookel
VeLurr
Orange Beok, Grey Beok Wwiiee Throot,
VyCuvo
[Or(Oragge Beak, -leey Beak)
Recent approaches have expanded the reach of CBMs across varying real-world setups. Concept Embedding Models (CEMs) (Espinosa Zarlenga et al., 2022) enhance the ex- pressivity of concept representations to enable CBMs to be competitive in datasets with incomplete (Yeh et al., 2020) concept annotations. Post-hoc CBMs (Yuksekgonul et al., 2023), LaBOs (Yang et al.. 2023), and Label-free CBMs (Oikarinen et al., 2023) instead address the diffi- culty of sourcing concept labels and retraining models by exploiting foundation and pretrained models. Further works improve the effectiveness of concept interventions by intro- ducing new training losses (Espinosa Zarlenga et al., 2023), intervention policies (Chauhan et al., 2022), or considering inter-concept relationships (Havasi et al., 2022: Steinmann et al., 2023; Vandenhirtz et al., 2024; Raman et al., 2024).
Figure 1. Visualization of the behavior of Lero and LecE under different scenarios, where the concepts are sampled from the opti- mal policy π². the empirical data and the amortized sampling policy my. In detail the scenarios for the concepts are: Beak color where the not-optimal, but not worst concept (orange) is preferred over incorrectly sampled (red), Throat color where sampled and empirical data align as optimal (green). Beak shape where both are incorrect. We analyze how this impacts CacE and Lcvo gradi- ents: Ccro may sample incorrect concepts, leading to gradient updates and reduced sensitivity to noise, while LaCe updates on all samples, regardless of correctness.
than their correctness, a relaxation that has been found to be particularly useful in noisy settings, such as recommender and information retrieval systems (Kaufmann et al., 2023; Bengs et al., 2021).
Among these, the closest to our work are Probabilistic CBMs (ProbCBM) (Kim al., 2023) and Stochastic CBMs (SCBMs) (Vandenhirtz et al., 2024). Both approaches frame CBMs probabilistically and learn to amortize the posterior distribution of an auxiliary latent variable between the con- cepts and the input data. ProbCBMs amortize the latent's posterior using a diagonal covariance matrix to estimate concept uncertainty. In contrast, SCBMs amortize the full covariance matrix and use to estimate joint concept distri- butions for more efficient interventions. Both approaches have their benefits, but they both approximate the posterior of a latent variable and not the concept distributions. Con- versely, we show how the CPO objective is equivalent to learning the posterior distribution of the concepts.
We propose Concept Preference Optimization (CPO), a pol- icy iettiizaaatoninnsiied objective loss for CBMs. Figure illustrates how CPO leverages pairwise comparisons of con- cept preferences to guide updates toward preferred concepts while mitigating the impact of incorrect gradients. Unlike traditional likelihood -based learning, which updates on all samples regardless of correctness, CPO selectively adjusts based on sampled preferences. This reduces sensitivity to noise by being able to mitigate incorrect gradient updates when incorrect concepts are sampled. Our analysis shows that CPO is equivalent to learning the posterior distribution over concepts, leading to more robust training. Empirically, we demonstrate that CPO not only improves CBM perfor- mance in noise-free settings but also significantly alleviates the impact of concept mislabeling.
Preference Optimization (PO) PO is a powerful learning framework when the training labels are suboptimal, such as in recommender and information retrieval systems (Yue & Joachims, 2009; Shivaswamy & Joachims, 2012; Radlinski et al., 2008; Dudík et al., 2015). At their core, PO algo- rithms focus on learning a policy in setups where we lack an explicit reward signal but instead have access to relative pref- erences between pairs of labels a weaker constraint. PO has become particularly important in the training of Large Language Models (LLMs) in the form of Reinforeement Learning from Human Feedback (RLHF), which is used
2. Related Work
Concept Learning (CL) Concept Learning is a subfield of eXplainable AI (XAI) where models are designed to explain their prediction using human-understandabl units of information, or concepts (Bau et al., 2017; Kim et al., 2018), that are relevant for a task of interest (Poeta et al., 2023). While CL methods use diverse approaches to pro- duce concept-basse explanations, most can be framed within
2

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
to guide policy optimization based on qualitative feedback (Ouyang et al., 2022b; team, 2024). While powerful, this ap- proach is computationally expensive as the reward function and policy are trained separately. To alleviate this, Rafailov et al. (2023) introduce the Direct Preference Optimization (DPO) objective, which streamlines the process by jointly training both the reward function and policy.
hyperparameter À E R:
LCBM = LCE(y, fa(()) + ALBEE(C, no (c(a:))
The concept objective above optimizes the binary cross entropy (BCE) between the policy's predictions and the em- pirical data, which is known to be suboptimal under noisy settings (Goodfellow et al., 2016). Due this sensitivity, we take inspiration from modern PO algorithms, deriving a sim- ple and computationally efficient objective that is equivalent to approximating the concept's 's posterior distribution and is more robust to noise compared to BCE.
Akin to traditional likelihood-baac based optimization ap- proaches, DPO has the added benefit of being differentiable. In contrast to likelihood- learning, however, DPO does not require set of training labels sampled from the opti- mal data distribution, instead only assuming that a prefer- ence exists between any pair of labels (Bengs et al., 2021). Assuming optimal labels makes likelihood- based methods prone to overfitting to simple patterns (Arpit et al., 2017), making them vastly less robust to label noise (Goodfellow et al., 2016). Such limitation is particularly relevant for traditional CBM training pipelines, which often include data augmentations and sample mislabels that can lead to con- cept labels differing from the optimal ones. To alleviate this, we extend the DPO objective to CBMs. We find that, as in language and retrieval tasks, training CBMs with our objective alleviates the effect of label noise.
Direct Preference Optimization (DPO) Traditionally, preference optimization using RLHF algorithms (Kauf- mann et al., 2023) relies on learning a reward function through the Bradley-Terry preference model (Ouyang et al., 2022a; Kaufmann et al., 2023). Given preference dataset (c" c', .I,3) ~ μel one can learn a reward function capable of distinguishing preferred concepts e from dispreferred ones d' by optimizing
max Be (ee",et.)))) (f (oogo o (rv(c", ,a)) ( ax)] rip
Where ry is a parameterized reward function learnt through the optimization process and is the sigmoid function. Üs- ing this Iearned reward function, a policy can be trained with any RL algorithm. Most commonly employed is the proximal policy optimization (Schulman et al., 2017) algo- rithm, which imposes KL constraint with prior (o(c))) on the standard reward maximization objective,
3. Background
In this work, we approach learning CBMs through the use of PO and thus, adapt our notation accordingly.
max Eaw (re T8
c)] - AlxxL (clz) || mo(c)))) (1)
Concept Bottleneck Models Concept Bottleneck Mod- els (CBMs) (Koh et al., 2020) assume their training sets (X, C, Y) are sampled i.dd. from an empirical distribution pl(c, xr y), where x E X are the input features, E C are the binary concepts labels c c= (ei,...,,C, (x), and y E Y are the task labels. We assume that puc, x, y) may not necessar- ily be the same as d² (c, v), the optimal data distribution sampled from the optimal policy म.. Specifically, we as- sume that they only differ at the concept level, meaning the empirical distribution's ' concept labels may be noisy while the task labels are always correct. We argue, however, that this difference is likely in practice as concept-specific noise may be accidentally added during training because of com- mon data augmentations that may occlude concepts (e.g., random crops or shifts, see Figure 1). Moreover, concept- specific noise may naturally arise from subjective or fatigued labeling (Sylolypavan et al., 2023; Wei et al., 2024).
where 33 is a hyperparameter controlling the prior's strength. However, this two-step procedure is computationally expen- sive and unstable. To address this, Rafailov al. (2023) proposed the Direct Preference Optimization (DPO) algo- rithm. Showing that the optimal policy for this optimization problem can be expressed as
1 Z(x)
mo(clar) exp (f)(0,,))
(2)
where Z(x) = En To(clz) exp (fr* (z, is the partition function, and r" r'(x,, c) represents the optimal reward. Con- sequently, the optimal reward function can be expressed in terms of the optimal policy:
x (c/ox)
r (2. c) = 1 log
+ 9 log ((x)
(3)
To(Cz)
CBMs consist of two sub-models. First, a concept predictor, Tp : X + C+. maps the input onto an interpretable layer composed of predicted concepts, č. The concept predic- tor is usually initialized using a pretrained image encoder kg. Then task predictor, fø C + Y maps these predicted concepts to the task labels ÿ. In this work, we focus on jointly trained CBMs, which are trained end-to- end by minimizing the following objective weighted by a
Using this formulation, Equation I simplifies to
log To(c" (x) (u(c (x]
To(cl(x))
max Bl (lo log o
log
(o(c(x))
(4)
which is an offline objective that jointly trains the policy and reward functions.
3

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
4. Preference Optimization for CBMs
the sampled concepts are equal to the empirical ones relies on assuming uniform prior.
Although one could directly optimize the objective in Equa- tion 4, doing so for CBMs would require a labeled dataset where preferences in concepts are explicitly specified. To cireumvent this issue, we can leverage the empirical dataset and state its preference over concept set sampled from mp- The preference over a pair of concepts should hold specif- ically early on in training, where the policy is suboptimal compared to the empirical data. Throughout the rest of this section, we formally describe this algorithm, showing some key similarities and differences between it and LBCE.
Gradient Analysis Proposition 4.1 highlights a similarity between Lcro and LaCE- Therefore, we can study their gradients to understand their key differences better. Under our previous assumptions, we can express the expected gradient of LCPO as
V
Σ
(Ta(c) - 1)xo(c'(=)
N
d'unp
Σ (,,a))u
(TT(c)2) - (1 ==(cl=)) Vake
N
4.1. Concept Bottleneck Preference Optimization
To leverage DPO to learn Tg, we re-formalize it as an online learning algorithm. We collect negative concept sets by sampling from the policy conditioned on the input features d' ~ To (c|2r)). We can then compare these negatively sam- pled concept sets with those sampled from the empirical data c ~ J.. where we assume that the empirical set pre- ferred over the sampled set (i.e., C > c)) Note that this is a weaker assumption than that of traditional CBMs, as we only assume a preference over rather than its correctness. Using this, we introduce the Concept Preference Optimiza- tion (CPO) objective, an online formulation of Equation 4:
where hig refers to the pre-trained image encoder tradi- tionally used to generate concept representations. A full derivation of this equality, which exploits the fact that we only have a nonzero gradient when c' * c and thus <<< [x) = - (c(xx), is given in App. C.I.
This result shows that LcPo's 's gradient is LBCE'S gradient weighted by how confident the policy is in the sampled con- cept. This yields the following bound on the CPO gradient: Proposition 4.2. Under the same assumption Proposi- tion 1. the expected norm of the Lcro 's gradient is a lower bound of L 's expected gradient. That is:
Lcoo = () logo log ne(clo) domo
<<(x((x)
log
(5)
||[[oocrolll2 S |[\VoLace]]]
When used in language modeling, To is defined as the model after a supervised fine-tuning procedure. Here, we train the model from scratch. In practice, we could impose a prior on the concept labels which relate to either the input or the task label e.g., To(clar, v). We briefly explore such applications in $ 5.4, but otherwise assume a uniform prior unless oth- erwise stated, leaving further explorations as future work. These assumptions simplify the CPO algorithm as follows:
Proof. As 0 S (c|rr) S 1, we have that
Σ (s(())=) - 1)(1 - - $11 Σ (x((l))) - 11/l2
(c.z)~pp
(c.e)<w
Notice how the right-hand side is equivalent to VOLBCE with equality only holding when e, # c for all i. Thus, we can see that an implication of not assuming the correctness of the concepts is that Lcro is more conservative in its gra- dient updates than LaCE. This means that LCpo has larger gradient when Ta is confident in the sampled concepts and is more conservative when it is uncertain. A visualization of the differences in the gradients is given in App. D. Next, we discuss the direct implications of these results and the relationship to the improved label noise robustness.
Proposition 4.1. Assuming that To(cla:) follows uniform distribution over binary concepts, we have:
Lcpo x B. [og(m)(c|<))]
(6)
A proof for this proposition is given in App. C.I. Simply put, the above states that EcPO is proportional to optimizing the binary cross-entropy when Te samples concepts that differ from those in the empirical distribution. Lepo is propor- tional to the objective in Equation 6, and not equal, because when the sampled concepts are equal to the empirical ones, the objective constant, i.e., log Ta(cla) in Equation 5. The equivalence to the log- likelihooo when
4.2. Noisy Concept Labels
To improve performance, CBMs are traditionally trained by randomly augmenting input images with transformations, such as cropping or blurring, which may obscure the repre- sented concept. As a result, CBMs are often trained with some level of concept noise, regardless of the reliability of the empirical data. Moreover, commonly used benchmark datasets for CBMs, such as CUB (Wah et al., 2011) and AwA2 (Xian et al., 2019), are designed so that their images
Ro(elm) log (d
[
In practice, we use hard Gumbel-Softmaa sampling (Jang al., 2017) to ensure end-to-end differentiability. Here, we sample single concept for each image in each iteration, but one could potentially sample multiple per image to increase performance.
4

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Kim et al. (2024) Vandenhirtz (2024)
(a) Ours
V
Concepp, wht unde parts
Concepe white underpart
(Concept: white undersurts
Uncerta ray Scores Ise.
Scores (et: SCE: 0.00
Uncertalnty Scores (dl: (Po) 1o0 [Caall 0.53) Probcitt: 1.00)
CP0) 0.52 (caal, 0.111
CPS: 0:53 PobceM 0.76 Ic-03 1.001
Figure 2. Comparison of graphical models for Bayesian CBMs. Dashed outlines indicate the variable over which an amortized posterior is taken. On the left (a), it can be seen that optimizing CBM using Levo directly approximates the posterior distribution of the concepts. On the right (b), it can he seen how other methods instead obtain the posterior over a latent variable z.
1.00)
Figure 3. Example of uncertainty estimates for different mod- els analyzing the bird's white underbelly. We measure uncer- tainty (0) and model predictions (c-val) before and after two augmentations crropiing and blocking across three models: a CBM trained with Lero, one with CaCE, and ProBCBM. The BCE model is consistently confident, even after both augmenta- tions. The ProbCBM's uncertainty stays fairly constant, increasing slightly when zoomed in. In contrast, the Cavo model becomes more certain with cropping and most uncertain when the concept is blocked.
may not accurately reflect their concept labels. Intuitively, Ecpo dropping the assumption of correctness towards prefer- ences should help under both noisy and optimal conditions. To illustrate why this is the case, we analyze the gradients of Lcro and LECE in the presence of noise.
respective noise-free counterpart. other words:
Empirical Best Gradient To show why Ccro is more resilient to noise, we examine both losses' gradients under noisy conditions and compare them to their optimal counter- parts. To do so, we first make the assumption that d' (c, x) is one if æ and c are the ground truth concepts in the image and zero otherwise. Then, given the empirical and optimal dis- tributions y and d", respectively, we can derive the expected gradient that approximates the ground truth as:
Caooaa/VoLCre VoLcrol - E( laaeer//ooerrii VoLcroll2 [ (c a) Vo /
E (eea)~llVoanch
We prove this theorem in App. C.2. Intuitively, Theo- rem 4.3 says that when examining the difference between optimal and noisy gradients, only terms from noisy ob- servations remain. Thus, according Proposition C.I, |B-- [VoDroo] ||ا - E| VoCBCel ll,- This im- plies that Lcro 's gradient updates more closely approximate their optimal gradients, resulting in better noise robustness.
d (c,x) V Vo To(cx))
x))d== [Vo[] = ((c,a)yy p(c,,)
A simpler explanation lies in the update mechanisms, where CPO only modifies the policy when concepts are incor- rectly sampled, creating situations where sampled concepts c' align with and, thus, minimizing noise impact. In contrast, BCE updates continuously unless To(clz) exactly equals or 0, making it inherently more susceptible to noise. Figure 1 illustrates these results.
E(erx))nut Vol(et,mo(caz)
('(xx) Here, p(c.a) is an importance sampling coefficient that equals when e exists in both d' and μr, and O otherwise, as we assume both and d' are deterministic, and μ+ E μ is the subset containing only optimal concepts. Conversely, u E a is the subset containing only suboptimal concepts c_. The resulting gradient on the empirical data is
4.3. Relationship to Amortized Posterior Approximation
Given their relationship. we seek to understand the funda- mental difference between optimizing Lepo and LACF.
B(c.))mul [ooc]
Voole""=ooc"ll [x(]][x;]] + 10 a)- [Vo[(c (z)].
Control as Inference The bottleneck nature of CBMs is similar to that of a Variational Auto-Encoder (VAE) (Kingma & Welling, 2022). Traditionally, such Bayesian methods introduce a "bottleneck" in their infer- ence that is formed by latent variables that are learned in an unsupervised fashion (Doersch, 2016). The graphical model representing a CBM often resembles this relation- ship, with the key difference that CBMs directly specify the factors within the bottleneck in the form of known concepts. One important outcome of this difference that CBMs
is a linear combination of the gradients on the noisy con- cepts e and the optimal ones c This formulation allows us analyze the difference between the optimal gradient and the gradient produced on noisy distribution for both LcpO and LECE:
Theorem 4.3. The gradient of EcPO under a constant level of noise is closer in distance to its noise free counterpart than the gradient of LoCE under the same noise is to its
5

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Table Task and concept performances. The highest and second-highest values in each column are holded and underlined, respectively. We find that Lcro consistendly achieves improved task accuracy and concept AUC.
CUB
AwA2
CelebA
Task Accuracy
Concept AUC
Task Accuracy
Concept AUC
Task Accuracy
Concept AUC
ProbCBM Sequential
0.742 + 0.004
0.900.1 0.07
0.891 1 0.033
0.960 1 0.003
0.302 1 0.008
0.878 1 0.006
ProbCBM Joint
0.766 + 0.012
0.943
0.006
0.860 1 0.017
0.945 + 0.007
0.288 1 0.023
0,863 1 0.005
CoopCBM
0.760 + 8.004
0.936
0.001
0.888 + 0.06
0.950 + 0.003
0.288 + 0.011
0.878 + 0.002
CBM BCE
0.753 + 0.009
0.937 + 0.001
0.900 + 0.008
0.971 + 0.959 + 0.003
0.283 + 0.007
0.873 + 0.002
CBM CPO (Ours)
0.800 + 0.003
0.952 +
0,915 + 0004
0.310
0.009
0.857 + 0.003
CEM BCE
0,800 + 0.003
0.946 + 0.901
0.889 + 0.001
0.953 + 0.000
0.351 + 0.006
0.875 + 0.004
CEM CPO (Ours)
0.807 + 0.004
0.931
0.917 + 0.003
0.965 + 0.001
0.352 0.004
0.853 + 0.003
are traditionally trained to optimize the likelihood of the empirical concepts, which is fundamentally different from approximating the concept's 's posterior distribution (Koller & Friedman, 2009). On the other hand, Haarnoja et al. (2017) show that Equation 1 - and Equation by extension - is equivalent to training an amortized posterior approximation of the actions (concepts in our contexts). This derivation relies on introducing an optimality latent variable o whose relationship to æ and is visualized in Figure 2 (Eysenbach & Levine, 2022; Levine, 2018). This optimality variable denotes whether or not the given state-action pair sampled from is optimal o = 1 (c is the best visually represented concept in 2) or not o = 0. The distribution over this vari- able is then given as:
uncertainty estimates than other models, being more sensi- tive to obstructions of the target object/concept. Figure presents a brief case study on an example image, compar- ing a vanilla CBM trained with Lero. one trained with LECE. and ProbCBM (Kim al., 2023), which amor- tizes the posterior over hidden variable (see the right side of Figure 2). We analyze the uncertainty scores t-ehe normalized variance scores for each model (variance of a Bernuilli variable for CBMs and determinant of covariance for ProbCBM) and the e-val, representing the predictions of (e(c))). This example illustrates a common trend from our quantitative analysis: Lero more effectively increases its uncertainty when the target object (in this case, the bird) is obstructed, compared to other baselines.
plo 1|x, c) = (xpp(r(c,,x))
5. Experiments
where r ((,,,) x) E (-o0, 0] in our case an unknown reward function indicating how well a given concept set is repre- sented in an image. Here, one
Here, we validate the Ccro objective in three different set- tings. First, we study Lcpo in clean, optimal data, then under concept label noise, and finally in streaming data context where we leverage prior when computing our updates.
interpret p(o 1|2, c) as the probability that the given concept set is correct, or optimal, for input I, and p(o 1|x) as how optimal, on average, the concept sets sampled from π are for given x. Given this, a posterior over the concepts is:
Datasets We study our proposed objective on three real- world image datasets: Caltech-UCSD Birds-200-2011 (CUB) (Wah et al., 201 1), Large-scale CelebFaces Attributes (CelebA) (Liu et al., 2015), and Animals with Attributes 2 (AwA2) (Xian et al., 2019). For CUB, we use the 200 class labels and 112 concept labels used by (Koh et al., 2020). For AwA2, we use the original 50 classes and each sample's 85 attributes as concept labels. Finally, for CelebA, we use the 256 classes and six concepts selected by Espinosa Zarlenga et al. (2022). The latter dataset is included to study our approach in a setting where the concept set is not fully de- scriptive (i.e., complete) of the downstream task. Further details of each dataset can be found in App. B.
T(clo- = 1,z) = p(o- = (lc, p(o 1|x)
(8)
(() To(c|z) exp (1 (gr(o,c))
exp
where 7(0)= = p(o- = 1/x). This objective is equivalent to that in Equation 2 as (o) must be equivalent to (zx) for (clo = 1, to be a valid probability distribution. Hence, optimizing ve using the objective in Equation 1- -nddd Equa- tion 5 by extension directly approximates the optimal con- cepl posterior distribution where T" (c|ær) <clo 1,
We provide a comprehensive analysis of uncertainty quan- tification in App. G, where we evaluate the uncertainty per- formance of Lcro against baselines. Overall, both quanti- tative and qualitative results show that Lcro offers better
We evaluate Lcro against LBCE on the follow- ing CBM-based architectures: (1) standard joint CBMs with sigmoidal concept representations (CBM), Concept Embedding Models (CEMs) (Espinosa Zarlenga et al.,
Baselines
6

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
CUB
AwA2
CelebA
100
95
Task Accuracy
96
90
94
CEM CPO (Ours)
CPO (Ours)
85
92
CEM BCE
90
CBM BCE
Pro PCM Sequenti BCE
88
35
ProbCBM Joint BCE
75
86
30
0
10
20
0
10
20
Number of Intervened on Concept Groups
Number of Intervened on Concept Groups
Number of Intervened Concept Groups
Figure 4. CUB Interventions without added label noise. In CUB and CelebA, Lcro models lead the best intervention performance. While in AwA2, substantial (~8-15) number of interventions must be performed for Lacu-based models to outperform Lero ones.
2022), which employ more expressive concept representa- tions to increase model capacity, (3) Coop-CBMs (Sheth & Ebrahimi Kahou, 2024), which use an auxiliary head to im- prove a CBM's information bottleneck, and (4) ProbCBMs (Kim et al., 2023), which introduce latent parameter be- tween inputs and concepts (Figure 2). ProbCBMs, in partic- ular, allow us compare amortizing the concepts' posterior distribution instead of a latent variable's s. However, while ProDCBMs traditionally use sequential training, we jointly train them to ensure fair comparison across baselines. We do not evaluate training traditional ProbCBMs with Lcro as their loss function optimizes an evidence-lower-boume on the latent variables' posterior, which explicitly requires maximizing concept likelihood. We note that while LePo introduces
to this and similar findings in preliminary results, we do not evaluate this method in the remaining experiments and further discuss Coop CBM's performance in App. B.I.
Interventions A key advantage of CBMs is their ability to improve their task performance through test-time concept interventions. In $ 4.3 we show that Lcro directly optimizes for the concept posterior. Thus, an advantage of this is that we can obtain accurate uncertainty estimates from the predicted concept values. To test the effectiveness of this uncertainty estimate, study the effect of interventions when we choose the order in which concepts are intervened on based on their uncertainties, i.e., more uncertain concepts are intervened on first. We do this for similar approaches by using the concept prediction as an uncertainty estimate for CBMs and the determinant of the covariance matrix for ProBCBMs (as done by Kim al. (2023)).
parameter B, we choose not tune it and set B = 1 for all experiments. Overall, this means we tune the same number of purrme-prameeers for CBMs trained with LCPO and LECE. We discuss other hyperparameters and implementation details in App. A.
Figure 4 illustrates the responsiveness of models to inter- ventions. Here, we see that, across all datasets, CEMs and standard CBMs trained with LCPO exhibit better accura- cies as they are intervened on than their LACE counterparts. This suggests that directly modeling the concept posterior distribution provides better uncertainty estimates, leading to more effective interventions. Additionally, CBMs and CEMs trained with Lcro achieve stronger intervention per- formance than ProbCBMs on CUB, while CEMs using Lcpo outperform ProbCBMs on CelebA. The only exception is AwA2, where ProbCBMs, on average, still require approxi- mately eight interventions before surpassing Lcpo models.
5.1. Un-noised Evaluation
We first evaluate the task accuracy and mean concept AUC- ROC of models trained with Lcpo under the traditional CBM setting, where no additional label noise added. Thereafter, we analyze intervention performance.
Base Performance Table 1 summarizes performance met- rics for each baseline and dataset. Our results suggest that training with the LcPO objective enhances the base task accu- racy of both standard CBMs and CEMs with minimal-to-ne loss in mean concept AUC. Most notably, we observe that, CBMs match/outperform CEMs trained with BCE, significant result since these benefits from CPO come without any additional parameters or significant computational overheads (see App. I). Finally, we find that Coop CBM performs similarly to traditional CBMs, finding no significant difference between them. Due
5.2. Noised Evaluation
in CUB and AwA2,
Next, we empirically study Lcro under various amounts of noise. To do this we randomly flip each training concept label with probability p and study the resulting models.
Base Performance Under Noise Figure shows our base- lines' task accuracies and concept AUCs as we ablate the
7

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Concept AUC
Task Accuracy
Nosse Level 0.2
Noise Level 0.3
1.0
0.85
100
0.9
0.80
05
50-
CUB
6.75
0.0
0.70
0.7
0.65
20-
0.6
0.00
75
0.2
0.2
0.3
0.4
0.2
0.4
1.0
0.94
0.9
AwAZ
0.90
15
20
25
0.88
120
100.0
0.7
0.8
94
52.5
50.0
0.9
0.35
CelebA
0.84
0.25
0.30
88
82.5
85.0
0.7
6.20
10
15
25
5
10
20
25
0.6
0.3
6.15
0.2
Noise Level p
Noise Level p
Figure 5. Performance metrics for tasks across varying degrees of label noise. We find that across all noise levels, Lcro consis- tently alleviates performance loss.
Task
label noise probability p across {0.1, 0.3, We ob- serve that, under noisy label conditions, training CBMs and ProbCBMs (see App. F.2) with LBCE leads to a significant drop in task accuracy and concept AUC (except for AwA2). Interestingly, see that CEMs trained with LBCE much more resilient to noise in comparison CBMs. However, we still observe significant drops in concept AUCs in CEMs trained with LECE in all tasks but CelebA. We believe CEMs more robust performance in CelebA is due to the bottleneck imposed by this dataset being small (only 6 concepts), mak- ing any additional capacity extremely helpful. In contrast, we see that models trained with LCPO are very resilient to noise. Specifically,
Number Concupt Grouped inaeventions
Number Cancept Grouped Interventions
CEM (Ours)
CPO (Ours)
ProbCBM CBM BCF
ProbCBM Sequential
CEM BCE
Figure 6. Intervention performance under noisy settings. We find that Ccro generally less affected by concept label noise in terms of interventionality. This finding is specifically relevant CUB and CelebA where we find little to no performance degradation when compared to the un-noised setting.
they come at the cost of CBM's intervenability. To this end, in Fig. 6 we report the intervention performance we vary training label noise. We observe that models trained with LECE have their intervenability significantly affected by noise, with CEM the occasional exception (particularly in CelebA). In App. F.1 we visualize the intervention perfor- mance for all p {0.1, 0.2, 0.3, 0.4) finding that in stark contrast LECE, models trained with Lcro remain inter- venable under the presence of noise, only being to severely impacted at high noise rates (e.g., = 0.4).
find that in terms of task accuracy, CBMs trained with LCPO are the least affected by noise and largely surpass the performance of CEMs. Moreover, we find that Ccpo-train -tronerained models have their concept AUCs bet- ter preserved, consistently holding the best or second-best ranks in concept AUC, often attaining significantly better concept AUCs than based models. Most interestingly, we find that even rather noisy levels (p 0.4), CBMs trained with Lero outperform more complex models trained with LECE and, in some cases, are largely unaffected by the noise. Overall, we find that using LcPO is an effective way to counteract concept label noise.
5.3. Noise by Labeler Confidence
While randomly noising the concept labels provides allows us to analyze how models may perform under noise without any prior on the structure of the noise, this may sometimes not be the most realistic scenario as noise often structured. For this we here we analyze noising based on the confidence
Interventions Under Noise While we find that the presence of label noise models trained with Lcro achieve better performances, such findings are less meaningful if
8

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
score of the labeler provided for each concept in each image in the CUB dataset. This allows to apply noise based
0.93
0.70
perceived uncertainty. The dataset provides a confidence rating from 1 to 4, where 1 is the least confident and 4 is the most. We apply noise proportionaly to this score: a rating of 4 corresponds to p = 0.1, 3 to p = 0.2, and so on. Additionally, in App. another structured noising procedures, where we noise concepts at the concept group level.
Accuracy
0.65
Concegt AUC
0.91
0.90
0.60
0.89
0.88
0.55
0.96
0.50
25%
50%
75%
25%
50%
75%
100%
Percentage of Training Data
Percentage of Training Data
Table 2 shows the results when concepts are noised accord- ing to their confidence levels. Interestingly, much like in the randomly noised setting, we find that LCPO again out- performs the LEC counterparts in both task accuracy and concept AUC greatly reducing the impact of noise.
= 50%
Prior
CPO
BCE
4% 100%
No
Figure 7. Updating a CBM with streaming concept labels (no task labels). We compare LacE, Lcro (No Prior), and Lero (Prior) using a Lero- -raintr initial policy with k% E (50%, 100%} of concept groups. We find updating Lcro with prior leverages all concept labels in this setting.
Model
Task Accuracy
Concept AUC
CBM BCE
0.733 + 0.032
0.876 + 0.012
CBM CPO
0.793 + 0.002
0.913 + 0.000
CEM BCE
0.704 + 0.053
0.831 + 0.048
one can alleviate this drift and enable Ccpo to leverage all new concept labels. While here find Lacr: underperforms Ccpo. in App. 7 we show this gap narrows narrows-though not fully closes -when the initial policy is trained with LACEE
CEM CPO
0.757 + 0.004
0.846 + 0.006
Table Noising proportionally to the confidence level of the la- beler (e.g certainty of 4 equals 0.1 noise 3, 0.2 and so forth). We find that Lepo is substantially able to alleviate the effects of this noise specifically task accuracy.
6. Discussion
Future Work In this work, we aim to maintain end-to-eu -nd differentiabilityy by leveraging a DPO-like objective, but this requirement is not strictly necessary. Differentiability is mainly needed due to the absence of a reward function; how- ever, one could instead employ variable rewards (c.g., based on visual consistency) or learn a reward function from pref- erence pairs (as done in language modeling). Overall, we believe RL techniques offer a promising direction for CBM Saining-potentillly enhancing robustness and enabling diverse behaviors through non-differentiable objectives.
5.4. Learning on Streaming Data
A byproduct of Lcpo optimizing for an approximate poste- rior is its ability to leverage a prior. So far, we have assumed a uniform prior over concepts, but here we explore adjusting it. One key benefit of CBMs is that practitioners can scru- tinize concept representations at test time, enhancing trust, accuracy, and enabling the ability to collect new training data through interventions. Specifically, when a CBM is intervened on, it obtains new concept label that can be used to improve the system further (an idea that has been explored in other fields (Stephan et al., 2024; Shi et al., 2024)). To explore this, we first partition the training data of CUB into four evenly sized blocks, of which we use the first block (25% of the data) train a joint CBM using Lcro the task labels y and the concepts c. Thereafter, we analyze training on the remaining data blocks only using concept labels in three different ways: using LUCE. Lcro with a uni- form prior and Lero with the previous checkpoint as the prior. The main idea is that a prior can help prevent the model from drifting too far from the policy jointly trained with the task predictor fa-
Conclusion We present a DPO-inspired training objective for CBMs called LCPO- Our loss directly optimizes for the concept's posterior distribution, with concept representa- tions that explicitly encode uncertainty, leading improved intervention performance. We provide analysis demonstrat- ing that Lero exhibits greater robustness to noise compared to РBCE and empirically show that a simple CBM trained with the Lcro objective can consistently outperform com- peting methods without any additional parameters. More- over, our experiments complement our analysis on Lcro behaviour under noise by showing that LcPO yields better concept AUC and task accuracy than BCE-based models while maintaining its intervention performance. Further- more, we demonstrate how the LcPO objective's 's prior can be leveraged to learn more efficiently from streaming data. Ultimately, Lcro offers numerous benefits for CBM and CBM-like methods with minimal computational overhead.
Figure 7 evaluates models using 1% E (50%, 100%} of total concepts. Curiously, we find that Lcpo using uniform prior performs worse when using more concept labels. We believe this is due to the policy drifting further from that of the initial checkpoint. We find that indeed by using a prior

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Impact Statement
jotham, K. Interactive concept bottleneck models. arXiv
preprint arXiv:22/2.074.00 2022.
Concept bottleneck models (CBMs) have emerged as a promising approach to increasing the trustworthiness and transparency of AI systems, areas where modern machine learning methods often fall short. This work takes a step toward improving the robustness of such models. While CBMs enhance the interpretability of AI systems, we demonstrate that they are highly sensitive to concept mis- labeling significant limitation in high-impact domains like healthcare and law enforcement, where labels are often noisy and inherently subjective. We show that our proposed objective effectively addresses this issue, enabling CBMs to perform more reliably in real-world applications and ex- panding their potential for meaningful impact.
Doersch, C. Tutorial on variational autoencoders, 2016.
Dudík, M., Hofmann, K., Schapire, R. E.,, Slivkins, A., and Zoghi, M. Contextual dueling bandits. In Grünwald, P., Hazan, E., and Kale, S. (eds.), Proceed- ings of The 28th Conference on Learning Theory, vol- ume 40 of Proceedings of Machine Learning Research, pp. 563-587, Paris, France, 03-06 Jul 2015. PMLR. URL. htwp::ttt///eettm http:s://poooeede/peeee ml malr.presse mulr.press/vAl Dudik15.htm.
Espinosa Zarlenga, M.. Pietro, B., Gabriele, C., Giuseppe, M., Giannini, F., Diligenti, M., Zohreh, S., Frederic, P., Melacci, S., Adrian, W., et al. Concept embedding mod- trade-off. In Advances in Neural Information Processing Systems, vol- ume 35, pp. 21400-21444 Curran Associates, Inc., 2022.
Acknowledgments
els: Beyond the eecurac--xpplannabiie
We thank the several anonymous reviewers for their feed- back which has greatly helped improve this work. EP knowledges the support of the NSERC PGS-D grant and the Bourse en intelligence artificielle provided by Université de Montréal. TZ acknowledges the support of Samsung Electronics Co., Ldt. MEZ acknowledges support from the Gates Cambridge Trust via Gates Cambridge Schol- arship. LC recognizes the support of the Canada CIFAR AI Chair Program, the Canada First Research Excellence Fund and IVADO. We also acknowledge that this research was enabled in part by computing resources, software, and technical assistance provided by Mila.
Espinosa Zarlenga, M., Collins, K., Dvijotham, K., Weller, A., Shams, Z., and Jamnik, M. Learning to Receive Help: Intervention- Aware Concept Embedding Models, Advances in Neural Information Processing Systems, 36, 2023.
Eysenbach, B. and Levine, S. Maximum entropy rl (prov- ably) solves some robust problems, 2022. URL https: //a/aaiivvvorr./////a /tt///avv.rrra//////////////////
06257.
Goodfellow, I. Bengio, Y., and Courville, A. Deep Learning. MIT Press, 2016. http:/ /www.
References
Arpit, D., Jastrzkbski. S., Ballas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj. T., Fischer, A., Courville, A., Bengio, Y., and Lacoste-Julien, S. A closer look at memorization in deep networks. In Precup, D. and Teh, Y. W. (eds.). Proceedings of the 34th International Con- ference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 233-242 PMLR, 06- 11 Aug 2017. URI. hwtp::ttp://o /at::://rrocceeee pttp///orceeed...c.ooo..... mllr press vv7 0/ 1tt.///pppaaaaatttttaeeaaa html.
Haarojja, T., Tang, H., Abbeel, P., and Levine, S. Rein- forcement learning with deep energy-based policies, 2017. URL https: /twpiv//////vv///a///a//aa.aaiaa org abs 117...18112.......
Havasi, M., Parbhoo, S., and Doshi-Velez, F. Addressing leakage in concept bottleneck models. In Advances in Neural Information Processing Systems, 2022.
He, K., Zhang, X., Ren, S., and Sun. J. Deep residual learning for image recognition, 2015. URL https: rexiv.org/ee abs/ 111/1111111 03385.
Bau, D., Zhou, B., Khosla, A.. Oliva, A., and Torralba, A. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541-6549, 2017.
Jang, E., Gu, S., and Poole, B. Categorical reparame- terization with gumbel-softmax, 2017. URL https:
/ sarxiiv.rrg/abe.
1611.111111111 01144.
Kaufmann, T., Weng, P., Bengs, V., and Hüllermeier, E. survey of reinforcement learning from human feedback, 2023.
Bengs, V., Busa-Fekete, R., Mesaoudi-Paul, A. E., and Hüllermeier, E. Preference-based online learning with du- cling bandits: A survey, 2021. URL org/ brg/abs//a /ate///077 1807. 11398.
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F, et al. Interpretability Beyond Feature Attri- bution: Quantitative Testing With Concept Activation
Chauhan, K., Tiwari, R., Freyberg, J., Shenoy, P., and Dvi-
10

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Vectors (TCAV). In International conference on machine learning, pp. (6668-267)) PMLR, 2018.
Oikarinen, T., Das, S., Nguyen, L. M., and Weng, T.-W. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2023.
Kim, E., Jung, D., Park. S., Kim, S., and Yoon, S. Proba- bilistic concept bottleneck models. In International Con- ference on Machine Learning, pp. 16521-16540. PMLR, 2023.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022a.
Kim, S.. Ahn, D., Ko, B. C., su Jang. I., and Kim, K.-J. Eq-ebm: A probabilistic concept bottleneck with energy- based models and quantized vectors, 2024. URL https: //arxiv. org/ abs /200924999. 14630.
Ouyang, L.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang. C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback, 2022b, URL https: / //arxiv. org/ abs 2200. ..00333000 02155.
Kingma, D. P. and Welling, M. Auto-encoding varia- tional bayes, 2022. URL https://arxiv.on
abss1312. 6114.
Koh, P. W., Nguyen, T., Tang, Y. S., Mussmann, S., Pierson, E., Kim, B., and Liang, P. Concept bottleneck models. In International Conference on Machine Learning, pp- 5338-5348 PMLR, 2020.
Poeta, E., Ciravegna, G., Pastor, E., Cerquitelli, T., and Bar- alis, E. Concept-based explainable artificial intelligence: A survey, arXiv preprint arXiv:2312. 12936, 2023.
Koller, D. and Friedman, N. Probabilistic Graphical Mod- els: Principles and Techniques - Adaptive Computation and Machine Learning. The MIT Press, 2009. ISBN 0262013193.
Radlinski, F., Kurup, M., and Joachims, T. How does clickthrough data reflect retrieval quality? In Pro- ceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM '08, pp. 43-52, New York, NY, USA, 2008. Association for Comput- ing Machinery. ISBN 9781595939913, doi: 10. 11 STS.11445882.145880900 URL https://doi.org/ 10.
Korbak, T., Perez, E., and Buckley, C. L. RI with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205. 1275, 2022.
Kumar, A., Hong, J., Singh, A., and Levine, S. When should we prefer offline reinforcement learning over behavioral cloning?, 2022. URL https: happs://araa arxiv.aivo...o.. org/ abs/ 2204. 05618.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model secretly a reward model, 2023.
Levine, S. Reinforcement learning and control proba- bilistic inference: Tutorial and review, 2018 8.
Raman, N., Espinosa Zarlenga, M., and Jamnik, M. Under- standing Inter-Concept Relationships in Concept-Based Models. In International Conference on Machine Learn- ing. PMLR, 2024.
Liu, Z.,, Luo, P., Wang, X., and Tang, Deep learning face attributes in the wild. In Proceedings of International Conference Computer Vision (ICCV), December 2015.
Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. In Ranzato, M., Beygelz- imer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 11702-11716. Curran Associates, Inc., 2021. URL https::/prroceedingg. neurips. cc cc/paper_filestteeeeeeeeeessssee c///sppe__.lle/epprrr///////////
Luccioni, A. S. and Rolnick, D. Bugs in the data: how imagenet misrepresents biodiversity. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intel- ligence and Thirty-Fifth Conference plications of Artificial Intelligence and Thirteenth Sym- posium on Educational Advances in Artificial Intelli- gence, AAAI22HAAAT233AA1112,. AAAI Press, 2023. ISBN 978-1-57735-880-0. doi: 10.1609/aai..v37ii 26682. URL http:::p://do..ogggg./o/oo v37112.2668 26682.
Innovative Ap-
pdf.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma. S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. Imagenet large scale visual recognition challenge, 2015. URL https:/ // a
Marcinkevičs, R., Laguna, S., Vandenhirtz, M., and Vogt, J. E. Beyond concept bottleneck models: How to make black boxes intervenable? arXiv preprint arXiv:2401. 13544, 2024.
0575.

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Schulman, J., Wolski, F,, Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algo- rithms, 2017. URL https:/ //tp://aaivvvv//a orgoor///aoooo abs/
Xu, X., Qin, Y., Mi, L., Wang, H., and Li, X. Energy-based concept bottleneck models: unifying prediction, con- cept intervention, and conditional interpretations. arXiv preprint 1rX1v:2401.14142, 2024.
1707.06347.
Sheth, L. and Ebrahimi Kahou, S. Auxiliary losses for learn- ing generalizable concept-based based models. Advances in Neural Information Processing Systems, 36, 2024.
Yang. Y., Panagopoulou, A., Zhou, S., Jin, D., Callison- Burch, C., and Yatskar, M. Language in bottle: Lan- guage model guided concept bottlenecks for interpretable image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19187-19197, 2023.
Shi, I. X., Hu, Z., Zhao, T. 7.,, Sharma, A., Pertsch, K., Luo, J., Levine, S., and Finn, C. Yell at your robot: Improving on-the-fly from language corrections, 2024.
Yeh, C.-K., Kim, B.. Arik, S., Li, C.-L., Pfister, T., and Ravikumar, P. On completeness-awee awaree concept-basse explanations in deep neural networks. Advances in Neural Information Processing Systems, 33:20554-20565 2020.
Shin, S., Jo, Y., Ahn, S., and Lee, N. A closer look at the intervention procedure of concept bottleneck models. arXiv preprint arXiv:2302.14266 2023.
Yue, Y. and Joachims, T. Interactively optimizing infor- mation retrieval systems as a dueling bandits problem. In Proceedings of the 26th Annual International Confer- ence on Machine Learning. ICML '09, pp- 1201-1208, New York, NY, USA, 2009. Association for Comput- ing Machinery. ISBN 9781605585161. doi: 10.
Shivaswamy, P. and Joachims, T. Online structured pre- dietion via coactive learning, 2012. URL. https:// arxiv. org/ abs /1205.225...0 4213.
Steinmann, D., Stammer, W., Friedrich, F, and Kersting, K. Learning to intervene on concept bottlenecks. arXiv preprint urXiv:2308. 2023.
1 1145/155337411553527 URL nttps://doi..cm
10. Yuksekgonul, M., Wang, M., and Zou, J. Post-hoc con- cept bottleneck models. In ICLR 2022 Workshop on PAIR2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data, 2023.
Stephan, M., Khazatsky, A., Mitchell, E., Chen, A. S., Hsu, S., Sharma, A., and Finn, C. Rlvf: Learning from ver- bal feedback without overgeneralization, 2024. URL https: /tts:///aavvv.../ ort.vv..rr//bba.. httpg//bs/2440 2402 10893.
Sylolypavan, A., Sleeman, D., Wu, H., and Sim, M. The impact of inconsistent human annotations on driven clinical decision making. NPJ Digital Medicine, 6(1):26, 2023.
team, L. The Ilama 3 herd of models, 2024. URL https: // //arxiv. org/ abs/ 2407.21782
Vandenhirtz, M., Laguna, S., Marcinkevičs, R., and Vogt, F. Stochastic concept bottleneck models. arXiv preprint arXiv:2406.19277 2024.
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S. Caltech uesd birds. Technical Report LANS-TR-011. 001, California Institute of Technology, 2011.
Wei, Y., Deng, Y., Sun, C., Lin, M., Jiang, H., and Peng, Y. Deep learning with noisy labels in medical prediction problems: a scoping review. Journal of the American Medical Informatics Association, pp. ocae 108, 2024.
Xian, Y., Lampert, C. H., Schiele, B., and Akata, Z. Zero- shot learning- a comprehensive evaluation of the good, the bad and the ugly. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 41(9))2251-2266 2019. doi: 10.1 109/TPAMI..018
12

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
A. Implementation Details
A.1. Tuning
We employ a ResNet34 (He ct al., 2015) as the backbone image encoder kg, pretrained on ImageNet-Ik (Russakovsky et al., 2015). Following standard procedures, we apply random cropping and flipping to a portion of the images during training. This augmentation process may introduce non-zero noise levels, as some concepts could be removed from the images after transformation. We use batch size of 512 for the Celeb dataset and 256 for for CUB and AwA2. We train all models using RTX8000 Nvidia-GPU. In all datasets not improved in 15 epochs. For fair evaluation across methods, we tune the learning rate for CEMs, CBMs, and ProBCBM. Specifically, for CUB and Aw datasets, we explore learning rates E {0.1, 0.01), while for CelebA, we expand the search to E {0.1, 0.05, 0.005} due to the observed instability of CEMs at higher learning rates. Additionally, we set the hyper-parameter À E {1, 5, 10) for all methods. For CEMs and models trained using LDOO, we found RandInt beneficial, which randomly intervenes on 25% of the concepts during training. ProbCBM introduce a few extra hyperparameters which in this work we did not tune and directly use the
train for up to 200 epochs and early stop if the validation loss has
provided by the original authors. Similar to other models, MrobCBM employs RandInt at 50%, making it particularly sensitive to interventions, especially in concept-complete tasks such as AwA2 and CUB. The only model for which we tune additional spper-paraeeters is Coop-CBM, where we adjust the weight parameter for the auxiliary loss we discuss more in detail in App B. All experiments are ran using forked version of the Github² repository used by Espinosa Zarlenga et al. (2022).
B. Datasets
CUB (Wah et al., 2011). In CUB annotations representing bird attributes (e.g., beak type, wing color) and use the bird class (m 200) as the downstream task. Our only departure from Koh et al. (2020) is that we group the concepts into 28 semantic concept groups, following Espinosa Zarlenga et al. (2022). We use the same image processing as in (Koh et al., 2020) and by randomly flipping and cropping some images during training. The final datasct is composed of ~ 6,000 RGB images of dimension (3,299, 299), and split into a standard 10%-10%--000 train- vain-validation-test split.
the standard dataset used in (Koh al., 2020) madeup of k 112 concept
AwA2 (Xian et al., 2019). For AwA2 use the same data processing as Xu al. (2024). Which are made up of where the k= = 85 concepts correspond to visual animal attributes (e.g., has wings, has claws) which are grouped into 28 semantic concept groups We apply standard rotation and cropping augmentations throughout training and use the standard 70%-10%--20% main-validation-test split.
CelebA (Liu et al., 2015). For this dataset, we closely follow the data processing done by (Espinosa Zarlenga et al., 2022). where they select the most balanced attributes out total of 40 binary attributes. Where they generate m = 256 classes by assign them value based on the base- 10 representation of their attribute label. We construct the incomplete concept set using the same 6 attributes selected by (Espinosa Zarlenga et al., 2022). We follow the same subsampling procedure as (Espinosa Zarlenga et al., 2022) and randomly select 12 th of the images for training. This results in final dataset composed of 16,900 RGB images where we use the same 70%-10%-2 -0% vrin-validation-test split.
B.1. Coop CBM
Here, we briefly outline the training procedure for Coop-CBM which we found to perform similarly to CBMs trained with LECE. Similar to other methods, we tune the learning rate and the concept loss weight X. However, Coop-CBM is the only model for which we conduct more extensive epper-parameter tuning, as we observed minimal differences between it and a standard CBM trained with LaCE. Specifically, we tune the additional hyper-parameter 7 E (0.01, 1,5, 10), which controls the strength of the auxiliary head'. In our setup, the optimal values for 7 were found to be 7 = 5 for CUB, 7 10 for AwA2, and y = 0.01 for CelebA. We observed negligible differences between Coop-CBM and standard CBMs in terms of base and intervention performance (see Figure 8), except for in AwA2 where it improves intervention performance but outperforms ProbCBM at higher number of interventions. As result, we did not include Coop-CBM in the remaining experiments.
*Referred to as B in their work, but we change the notation to avoid confusion with our 3 parameter.
13

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
CUB
AwA2
CelebA
100
Task Accuracy
95
55
90
CBM CPO (Ours)
94
BCE
92
Pro BCBM Sequential al BCE
Coop CBM BCE
35
90
75
88
10
20
10
20
Number Intervened on Concept Groups
Number of Intervened Concept Groups
Number of Intervened Concept Groups
Figure 8. Intervention performance including Coop CBMs.
C. Analysis
While prior work has shown that under mild conditions, offline RL performs (equivalent CPO) better than Akellhhodd-based training under noisy labels (Kumar et al., 2022; Rashidinejad et al., 2021), it still does not fully answer the question as to why specifically CPO should perform better in our context.
C.1. Gradient Derivations
Assumption: For all derivations, we assume binary labels and that no follows uniform distribution. Additionally to reduce clutter, in all gradient derivations, we drop the Vkg as it does not affect the gradients of the loss functions. We note to simplify the proofs we will assume conditional independence of the concepts i.e., G, .L. 0y|æ V i f j. but this is not strictly necessary. One could derive the same conclusions using an autoregresive decomposition of the joint density the concepts.
Derivation of the CPO objective
Ccro -e [log (log To Te(cl=) log (e(c(())
(10)
no (c(
ED, (clxx) o no (c(c((>))
]]
(11)
ne <e(c)
- log(1 (1+ + exp( log n (c' (x)
))]
(12)
Te ))
(e + exp(- log(
(13)
Te (c' [x)
(c(c((x)
me (clar) }
(14)
(15)
Thus in this case, we can see if c' is not equivalent to c, this loss reduces cross entropy.
Levo = Ee, D,ck [log( Te(([z) + (1 To (e(c))))
(16)
Te(clr)
Be, Dlog( Tc())
(17)
-E D. p olog( (T (c/x))
(18)
(19)
Otherwise, it reduces to constant:
14

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Ccro E En Di eana [log( [o(c)a) TT(c
)]
(20)
TO
logg 1 2
= =-log(
(21)
(22)
CPO Gradient Derivation
Vo log( To(c(2)) + (1 mo(c((x))
VoLcro De
(23)
ne (c(
(24)
Due to the gradient being zero when c c', the expected gradient of the CPO objective simplifies to:
VoLcro
Σ
(To(c)æ)) - 1) (x)
(25)
N
Σ
(Te(C) 1) (c(x))
(26)
That is, the CPO objective only takes gradient step for sampled concepts that do not equal the empirical concepts.
C.2. Bounding the gradients
Proposition C.1. The expected gradient given by LcPO under binary labels is strictly less than or equal to the gradient of the LSCE.
Proof: This proof relies strictly on the notion that (c([x) S thus:
1 Σ (Fo(d))) 1)(1 - < S N Σ (To(clz) 1)/l2
(27)
(c.ar)-p
Observe how the right-hand side is equivalent to the expected cross-entropy loss. The above proposition also takes into account the maximum gradient possible for the LCPo, which is when cf for all i.
15

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Theorem C.2. The expected squared difference between the optimal gradient and one computed under noisy labels for direct preference optimization is less than equal to that for binary cross entropy.
Proof: The optimal gradient take under noisy labels is given by:
d(c, x)
E =) [oL. ((xx))m
V Vollee To(c/x)
(28)
m(c,2)
d(c, V VoL(c TT(cZ))
(29)
plc,, x)
Bi -0)- VoL(c mo(c2))
(30)
(31)
We observe that when we do not adjust for the importance weight, the gradient under noisy labels is:
( = a))
w))pttt (Vol(c" "oo [o(c"<>)) + [))-==)==--- .)~ VVol(c (c
(32)
(33)
Thus the difference in the expected value of the gradients is:
E 2 ae) (Val (e [o = -Ve-- _
(34)
Therefore, using Proposition C.I we can observe that
| -)- [eCcrolc Tolc [x)]||| S (Veraa-=-IVeeBeEE TT(c |x))]||
(35)
And thus:
()) .t)-al Voorrll
(( (V Voocrll S (e / VoLBCE] - E(o Vo VoCBcel]
(36)
D. Gradient Visualizations:
We empirically verify the results posed in Theorem 4.3. For this, we train standard CBM where the total loss function Luaal = (Cero + LBCE) and do not optimize the task loss. We train this model over 100 gradient steps and visualize their gradients throughout training. The optimal gradient each loss L is computed using the empirical concepts full loss L- is computed over both noisy and non-noisy data points. To minimize the effects of noise on the labeled data and gain better approximation, we explicitly do not augment the data in any way. Figure 9 visualizes these results for p E {0.1, 0.3), which empirically confirms the proposed theoretical results showing how even under low amounts of noise p 0.1 Lero is better approximation to its optimal gradient when compared to LACE- We find that in higher noise settings 0.3, Erro deviates substantially less to LepO compared to LiCE against LACE. This difference is specifically evident early on in training. We hypothesize providing better gradients early on in training potentially improves the generalization of the model being possible explanation for the improved performance Lcro under noise seen the empirical evaluation.
the
16

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Naise Level at p 0.1
PoficE
0.000775
IV Vec
IVVo -VoCcooll
Val
0.000150
0.0
Vp4Cro
Awee Squaree Ciffeeneet
0.020125
0.04
0.000100
0.0000755
0,000050
-002
-0.44
Noise Level at 0.3
VoCice
V|VaCéc VoCoce 1|VoL VoLcrollz
VoCice
0.15
VaCopo
VeCoro
Awwrged Squard Diffeete
0.05
0.0
3.000
2bb
Figure Visualization of noisy (indicated by -) and optimal (indicated with *) gradients for VoLero and V LLCF. We can observe that even in low noise settings p 0.1, Lcro hetter approximates its optimal gradient, with this difference growing noise level increases especially the beginning training. We note that while visually may seem that the squared difference Lero smaller for p = 0.3 than that for p 0.1, this is mainly due scale.
17

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
E. Relationship to Amortized Posterior Approximation
In this appendix, we provide an additional formulation of how the maximum entropy RL objective is equivalent to training an amortized approximator of the concept's posterior distribution. We primarily summarize the derivations found in sections 2.2-2.4 of (Levine, 2018) and the appendix of (Korbak al., 2022) using our notation.
The main deviation is that here we directly show their equivalence as optimization of an evidence lower bound, given that one wants to optimize over p(o 1 That is
log p(o = 1|x: = log Σ plo = 1, 1.)z)
(37)
log > plo (((((()
(38)
(o(c(2)
log
[e(c|=)p(o- 1|x)
(39)
x (c/22
p(o 1|x) (c(
no (c|2)) log
via Jensen's inequality
(40)
(41)
Using the fact that p(o c|x,c) ex,(r"(xx,)) c)) we have:
logp(o 1|z) 2 Bewrs log exp (r" (c,xx) <o(c|2:)
(42)
Te Te(clz))
Therefore, maximizing the above objective is equivalent to:
max Eewnr [r (c, 2²] - (To(((z) || mo(cl))) We
(43)
Since optimizing Lcro is equivalent to optimizing Equation 43, it is also thereby equivalent to directly optimizing the posterior distribution of the concepts (cc.))
18

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
F. Additional Continual Experiments
Here, we examine the impact using a model trained with Lbce as the starting point for the experiments in $ 5.4, Figure 10 compares the performance of CBMs trained on streaming data when initialized with LECB (left) versus EPO (right, same as Figure 7). Overall, we find that updating Lacg-initialized model with LECE yields the best results for LBCE. However, while this improves performance, it still falls short of the results achieved when both initialization and training done with LPO- We note in the leftmost plot, we exclude the LDoO model updated without a prior to improve clarity of the plot, but note we find it yields approximately equal results to training with a prior.
0.93
0.650
0.70
0.92
15 Accuracy
0.625
0.600
Concept AUC
0.90
Task Accuracy
0.65
0.91
Concept
0.575
0.89
0.89
0.550
0.88
0.525
0.55
0.500
0.50
25%
50%
75%
100%
25%
50%
100%
Percentage of Training Data
Percentage Training Data
Percentage Training Data
Percentage Training Data
50% 8% 100%
Prior
CPO
BCE
Prior
CPO
BCE
Prior
8% 100%
No Prior
(a) Initialized using a LBCE model.
(b) Initialized using Cpro model.
Figure 10. Task Accuracy/Coneepl AUC vs the percentage of data the model has heen trained on. We find that updating models initialized with Lace policy yields improved results for Cace with detrimental ones for Lero (A). In (B) we again visualize the result for updating the models using a Coro- initialized policy (same as Figure 7). We find that the best result is given by using Coro to update Coro- initialized policy.
19

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
F.I. Intervention Plots at All Noise Levels
Here we provide an extension of our prior analysis in 5.2 including intervention performance for all values of p {0.1, 0.2, 0.3, 0.3). We find that again at low noise levels Lepo models consistently outperform their LECF counterparts. The one model not holding to that is ProbCBMs which outperform LDPO consistently on AwA2. We note while this behaviour comes at a cost for ProbCBMs as we show in $ 5.2, their concept AUC is severely affected by noise, making the requirement to intervene on these models much harsher.
CUB
CUB
CUB
CUB
53
50
Tolk Aoourrac
85
60
40
80
73
10
50
70
65
AWA
AWA-Noies Nolse 0.3
Nolio
150.0
18
50
95.0
T1.ak AcC.Aac
94
52.5
70
80
90
875
50-
88
08
82.5
40
20
1.0
10
CELED Noise Level
CELEO Noise Level 0.2
CELEO Noise Level
CELED Noise Level 0.4
02
50-
40
60
10
30
30
30
20
20
20
2
Number Concept Grouped Intervertions
Wumber of Concest Grouped Interventions
Number of Concept Crouped Interventions
Number Concept Grouped Interventionss
CEM CPO (Ours)
CBM CPO (Ours)
ProbCBM Joint
ProbCBM Sequential
Figure I.. Intervention performance for all noise levels. We find that overall methods trained with LDro yield better intervention performance under noise. These findings are specifically relevant to CUB and Celeb where we see all other methods are harshly impacted. We find the only model consistently outperform Lero are ProbCBMs AwA2, but have comes at the cost of having their initial concept AUC and task performance severely affected requiring practitioners to intervene far more often.
20

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
F.2. Full Evaluation
In addition comparing vanilla CBMs and CEMS, Figure 12 provides the full restulst including both joint and sequential ProbCBMs under noise. We generally find that ProbCBMs extremely senstive noisy data with their performence by half in some metrics such as CUB concept AUC. We find that the uncertainty estimates provided by Lcpo not only yield better gains in intervention, but also comes with the property of being much more robust to noise.
Concept AUC
Task Accuracy
1.0
0.8
0.8
0.6
CUB
0.6
0.4
0.4
0.2
0.2
0.0
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
1.0
0.95
0.9
0.90
AWA2
0.8
0.85
0.7
0.80
0.6
0.75
0.5
0.70
0.1
0.2
0.3
0.4
0.1
0.2
0.3
0.4
1.0
0.4
CelebA
0.7
0.2
0.6
0.1
0.5
0.4
0.0
0.2
0.3
0.2
0.3
0.4
Noise Level p
Noise Level p CEM CPO (Ours) CBM CPO (Ours)
CEM BCE
ProbCBM BCE Sequential
CBM BCE
ProbCBM BCE Joint
Figure 12. Performance metrics for all tasks across varying degrees of label noise. We find that across all noise levels, Lero ourperforms all other models, including both joint and sequential ProbCBMs
21

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
G. Additional Analysis on Uncertainty
In this Appendix, we provide a more depth analysis of how different CBM models handle uncertainty estimation. We focus on the CUB dataset and visually inspect the baseline uncertainty estimate of various sampled images, observe how different data augmentations affect the uncertainty and quantitatively evaluate how each model's uncertainty changes when the target bird is randomly obscured.
G.1. Uncertainty Analysis
We display randomly sampled images from the set. Figure 13 shows these images along with their corresponding uncertainty scores. In this and all subsequent experiments, we define uncertainty the scaled variance of the model's certainty. For ProbCBM, this corresponds to the product of the diagonal entries in the covariance matrix, following (Kim et al., 2023). For Lcro and LbCE, uncertainty is computed as the variance of a Bernoulli random variable, ΦPPOBCE c(l-c). To enable fair comparison across models, we scale all variance values to lie within the range [0, 1].
For each image, we highlight the concept for which the model exhibits the highest uncertainty and report the uncertainty scores from each model. We observe that Ccro tends to report higher uncertainty across most concepts, particularly when the concept in the image ambiguous or unclear. In contrast, ProbCBM often reports low uncertainty even for visually uncertain concepts. While these examples provide useful intuition, they limited in scope. In the following experiment, we provide more thorough quantitative analysis of model uncertainty.
G.2. Uncertainty Under Augmentations
A standard practice in training vision models (and consequently CBMs) is to apply augmentations during training. These augmentations can obstruct or alter the way concepts are represented in images. To assess whether LepO more reliably models uncertainty under such conditions, we evaluate model behavior under two augmentations: cropping and obscuring. Figure 14 shows examples of images under these transformations. Interestingly, we find that zooming in (via cropping) often increases model confidence concept predictions. particular, Lcro frequently exhibits a larger boost confidence compared to other models. Conversely, when the object the image is obseured, Lero generally becomes much more uncertain, often reaching the maximum uncertainty score when the concept fully blocked. This contrasts with other models, which tend to remain overconfident in the presence of the concept, even when it is visually occluded.
22

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
IP 10100 (Cr0 1ie Prod IN
and yoge oo PN (HIOON a nic and S fraee oF HNN Aiatoe 5.0000
soE 0,s6 NOH0 T1-A6 you on W.lk
ncr and (20 Tegeeee 5.30 (Hostal bas
son 0.33 (EP))))) (198 189 (Pit Whiaete a.ce mon 2195515 7186 pArtt (1ARA (.50.
SCE
DCD INc Cil DUPESAbI DI
3,00 RST
DCT: se Leattim moe coood (ate-1))
BCD IIN only OAD Whit (UDALO
5.00.0
SCT 29
CRO be col SWN (1
PrecOMM 3.00
SOT 6.60 (.20)
CPO: lhe head caltern plam (1atenn.))
0001F:0.00
CR0 be mackmm 18-16111111111 18 3-2
(P00.00)
PrOCENN Fas palee pocercaa 23.01 2/01 ProdEMM
Peo Fiss wing yol Babel: 21
POLCIM cwocoe coler Saber DICC 1.1 0.90
SCE
SCE c.c3
0.60
BOS Has cuter gneey (LataliL
RDD aeweerrtt or Suboto. D1
MCE Hoo upsemorts color Rubbet 200 1.00 (461
sct
300
StE 2.05 0.30 0.65 aesin: 1.0)
BCE c.s
ChO Itas pattrr e mudticolorss Label:1.0 D0
Frustaht
ChC bacc sottem in soldd Raabel-3.0 cuse OLSET
G.L 0.000
BLE S.43
2P Fis throat color valion Juace 5.350
BCS L.0
PIOCCMM the bally color and 1L
Papcaatt has parciing daabe 0.421
BILE
ProCCEM Nos unserpart white lasta10 3" B.40 MRSCON 5.130
Figure 13. Visualization of different images and concept uncertainty score. We display for each model the concept with the highest uncertainty alongside label for concept. For each model we display the uncertainty score alongside probability the concept being active given by each model. We find that Lero generally has higher uncertainty for more ambiguous concepts and is more certain than other concepts for concepts that are clearly visible.
23

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
2014
MEAY GiS Bac The MAR
de
TL Mio
soreprr calor
Ro0k deppat 00 ad
00 ynd
52
$-2 023 131
And COOOA E.d
CP of 1pen JAR) LEB
And
UP 042 Jas Creetouee
natteed
LM
Mana PaCC
Dedi: Pr-det 40M4
wide
Medi: ParChd
Nban No
rele
Saadta 220
and
(0Te 044 yor Puadai 2.00
DR000 Jase $774
peat 5.00 care
220 119 5001
Nope EX
Meca daygee
Ride CPC
Node CFS
NE:2: EX
pitat Teati-c0
Pode CF
Jileers COON
CAFIENO
PokOle LAPIONS
C2RME: Dayy 030 60.3
DIM pectip Ba SAL
1M 11p parc 1g Waet B.A.
Ton bok Priee partiig
020
Figure 14, Grid visualization of different images before and after applying cropping and blocking augmentations. For each model we display the concept with the highest certainty score alongside the scores of other models. Overall we find that Lero increases the confidence of the score the most when cropping causes zooming effect and decreases the most when blocking the target object.
G.3. Quantitative Uncertainty Analysis
While we previously presented examples where LCPO offers improved uncertainty quantification, we now aim to provide a more systematic evaluation. To do this, we apply random blocking augmentations across the entire training set CUB images. These augmentations are designed obseure parts of the image and should, in general, increase concept uncertainty
24

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
Block Siow: (0.3, 9.11
block Sioee (0.A, 9-4)
Block Siow: I0.6, 9.6)
BCE (Arca 3%. -
17.85
BCE (Area D-t 3%, -
3.-
BCE (Arb3 19.1%.Ars - 4.8%) Dea 6.0%)
De: a0:
CBM
COM 54.3%, 9,3%)
25-
20
20
10-
03
-122 Amn 04
3.2 2.6
6.0
6/2
26
Age
Figure 15. Distribution of changes in certainty (6mg) for correct concepts under increasing block sizes. Higher area under the curve ahove zero indicates a more appropriate increase in uncertainty. Lero and LBCE respond more consistently to occlusion than ProbCBM, whose uncertainty remains relatively unchanged.
1.0
BCE
CPO
ProBCBM
Mass Above
0.4
0.4
Figure 16. Per-concept AUC values under 4 x blocking. LECE concentrates uncertainty increases on a few concepts, while Lero distributes uncertainty more evenly across concepts, suggesting better alignment with the effects of occlusion.
by reducing visual access to relevant features. We vary the blocking size, with larger blocks expected to result in greater uncertainty due to higher likelihood of obstructing the object.
Figure 15 shows the distribution of dwng. defined as the change in certainty for correct concepts before and after blocking, across three different blocking sizes. We report the area under the curve above zero- zero-higher values indicate that the model tends to become more uncertain after blocking, as desired. Overall, we find that both Lcpo and LACF increase uncertainty more consistently following augmentation. In contrast, ProbCBM's uncertainty appears largely unaffected by the occlusion, with its scores fluctuating in seemingly random manner regardless of whether the object is visible.
While Lepo tends to yield better uncertainty setimates-particullle at the 0.4 x 0.4 blocking size-the differences between LCPO and LBCE remain modest in aggregate. To better understand the nature of these uncertainty shifts, we examine how they are distributed across individual concepts. Figure 16 presents the per-concept AUC values under 4 x blocking. We find that LACE concentrates its uncertainty increases on a small subset of concepts, leaving many concepts largely unaffected by the augmentation. In contrast, Lcpo exhibits more balanced distribution, suggesting it more evenly attributes uncertainty
25

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
increases across concepts, thus better capturing the effect of occlusion.
H. Concept Group Noise
A straightforward way to introduce structured noise across datasets is to flip labels between related concepts. This approach mirrors realistic labeling errors, such as a human annotator labeling a bird as having brown wings instead of red due to factors like lighting, zoom, or viewing angle. We explore this form of structured noise in both the CUB and Aw datasets. Following our previous setup, we examine four noise levels: p E {0.1, 0.2, 0.3, 0.4) where p here refers the probability of having concept group's label be altered.
The results found in Table 3 are consistent with our earlier findings, where we observe that LCPO is significantly less robust to noise compared to LACE. Specifically, on the CUB dataset, models trained with Lcpo achieve substantially higher task and concept accuracies than their LBce-based counterparts. In the AwA2 dataset, Lcpo-based models better preserve concept AUC; for example, CBM models trained with Lcro consistently maintain concept AUC above 0.84, while all other models drop below 0.8. These results suggest that Lcro performs better not only under uniform noise but also in the more challenging setting of structured noise.
CUB
AwA2
Noise Level
Task Accuracy
Concept AUC
Task Accuracy
Concept AUC
CBM BCE
0.741 + 0.047
0.914 + 0.020
0.879 + 0.003
0.922 + 0.002
CBM CPO
0.796 0.005
0.941 + 0.005
0.894 = 0.003
0.943 + 0.003
0.1
CEM BCE
0.728 + 0.039
0.888 + 0.016
0.886
0.004
0.937 + 0.006
CEM CPO
0.766 + 0.003
0.875 + 0.012
0.895 = 0.001
0.931 + 0.003
CBM BCE
0.719 + 0.039
0.851 + 0.018
0.880 = 0.001
0.877 + 0.001
CBM CPO
0.793 0.005
0.893 + 0.006
0.889 + 0.006
0.901 + 0.003
p= 0.2
CEM BCE
0.674 = 0.027
0.796 = 0.011
0.884
0.013
0.896 0.002
CEM CPO
0.757 + 0.002
0.844 + 0.002
0.887 + 0.004
0.873 + 0.009
CBM BCE
0.681 = 0.027
0.757 + 0.010
0.880
0.002
0.815 0.002
CBM CPO
0.784 + 0.012
0.804 + 0.007
0.883 + 0.005
0.845 + 0.018
0.3
CEM BCE
0.671 + 0.002
0.704 + 0.005
0.864
0.005
0.735 0.006
CEM CPO
0.744 + 0.002
0.809 + 0.003
0.882 = 0.002
0.772 + 0.010
CBM BCE
0.682 + 0.029
0.652 + 0.006
0.878 0.003
0.735 0.004
CBM CPO
0.780 + 0.013
0.695 + 0.006
0.877 + 0.003
0.889 + 0.003
p 0.4
CEM BCE
0.667 + 0.021
0.610 + 0.009
0.870 + 0.012
0.652 + 0.007
CEM CPO
0.736 0.005
0.715 + 0.003
0.883 + 0.005
0.685 + 0.008
Table Task Accuracy and Concept AUC for noising group groups at p E (0.11, We find that like our prior results, Lero significantly aids with concept noise across both CUB and AwA2 datasets.
I. Computational Analysis
Previously, we noted that Lcro adds minimal computational overhead to CBM-based models. Here, we provide quantitative analysis of the additional compute it requires. Table 4 shows the average training time per epoch using single RTX-4800 GPU (the same setup used for all reported experiments). We find that LCPO adds negligible amount of additional time per bonch-approximated 0.05 minutes. This additional clock time is significantly lower than that of other models such as CEM or ProbCBM, which roughly double the training time.
Furthermore, we use Py Torch's 's built-in implementation of LECE, which optimized for performance, whereas our LcpO and unoptimized. This suggests that further optimization of Lcro could reduce the already
implementation
26

---

Addressing Concept Mislabeling in Concept Bottleneck Models Through Preference Optimization
small performance gap even more.
Model
Avg. Minutes per Epoch
CBM BCE
1.132 0.012
CBM CPO
1.185 = 0.014
CEM BCE
1,403 + 0.055
CEM CPO
1.448 0.080
ProbCBM
2.310 + 0.035
Table 4. Average Time per Epoch
Table reports the total number of parameters for each model, showing that LcPo does not introduce any additional parameters. In contrast, models like CEM and ProbCBM increase the parameter count significantly, with ProbCBM requiring roughly four times more parameters.
Model
Trainable Parameters
CBM BCE
21.364,728
CBM CPO
21,364,728
CEM BCE
25,743,889
CEM CPO
25,743,855
ProbCBM
89.764,996
Table 5. Trainable Parameters
J. Sequential CBMs
While in 5.1 we explored using Lcro with joint CBMs, this is not requirement. Here, we investigate the use of LcPo with sequential CBMs. The results in Table demonstrate that LcPo outperforms LECr. for training sequential CBMs. This trend holds across all datasets, with particularly notable improvement in task accuracy on CelebA. We hypothesize this is due to the uncertainty estimate coming from LePo, which enables more expressive modeling by the decoder.
Dataset
Category
Task Accuracy
Concept AUC
SEQCBM BCE
0.730 + 0.007
0.928 + 0.003
CUB
SEQCBM CPO
0.741 + 0.003
0.931 + 0.002
SEQCBM BCE
0.898 + 0.005
0.964 + 0.000
AWA2
SEQCBM CPO
0.906 + 0.001
0.964 + 0.000
SEQCBM BCE
0.297 + 0.019
0.879 + 0.001
CELEB
SEQCBM CPO
0.325 + 0.007
0.880 0.000
Table Task Accuracy and concept AUC Sequential CBMs using Levo Lace across datasets. Consistent to joint CBMs, we find Levo can increase the unnoised performance. We hypothesize this is due to improved uncertainty estimates of the concepts.
27


Reviews:
[{"review_id": "nm6YYDZwUD", "reviewer": "Reviewer_2knp", "summary": "The paper proposes a Preference Optimization (PO) based training Paradigm - CPO for training CBMs. The paper gives a detailed analysis of the proposed method and has experiments on intervention and random label flips. The preference set is taken as observed empirical evidence while the negative sampling of concepts is taken as unpreffered.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their time and effort in helping us improve our work.\n## Weaknesses\n\n### 1 Diverse Experiments to Strengthen Conclusions\n\n\n1.1 Thank you for this valuable feedback. We agree that incorporating experiments with more structured noise would strengthen our conclusions. In our response to reviewer **sJF1**, we discuss two additional experiments:\n\nOne where noise is introduced at the concept group level (e.g., flipping only wing-related attributes in birds or altering beak colors).\n\n\nAnother where noise is introduced based on the confidence level provided by the labeler (as available in the CUB dataset).\n\n1.2\n`\nIf the model still performs almost as well when the label flips with 0.4 probability, it is deviating from its empirical evidence and relying more on the posterior. For (2) - should we actually ignore empirical evidence?\n`\n\n\n While it is possible that the model’s strong performance is partially due to reliance on the posterior rather than empirical evidence, we do not believe this fully explains the results. For example, in some settings, particularly for CEMs trained with BCE, models can still perform well even under high levels of noise.\nWe attribute this to the fact that, despite 40% of the labels being noisy, the remaining 60% provide sufficient empirical evidence for the model to fit the data—albeit imperfectly.\n\n\n\n### 2. Improving the Introduction for Clarity\nWe appreciate the reviewer’s suggestion to include a dedicated paragraph explaining Figure 1 in the introduction. While we agree that this addition would improve clarity, space constraints may limit our ability to introduce an entirely new paragraph. Instead, we propose enhancing the introduction by adding context around Figure 1, making it more digestible for the reader.\nHere, we provide the modified text we would add to the introduction to address this issue\n```\nWe propose Concept Preference Optimization (CPO), a policy optimization-inspired objective loss for CBMs. Figure 1 illustrates how CPO leverages pairwise comparisons of concept preferences to guide updates toward preferred concepts while mitigating the impact of incorrect gradients. Unlike traditional likelihood-based learning, which updates on all samples regardless of correctness, CPO selectively adjusts based on sampled preferences. This reduces sensitivity to noise by being able to mitigate incorrect gradient updates when incorrect concepts are sampled. Our analysis shows that CPO is equivalent to learning the posterior distribution over concepts, leading to more robust training. Empirically, we demonstrate that CPO not only improves CBM performance in noise-free settings but also significantly alleviates the impact of concept mislabeling.\n```\n\nQuestion\n1.  How are the hyperparameters tuned for the model?\n\nWe discuss these details in Appendix A.1, which we provide here for reference:\n\n```\nFrom App A. We use a batch size of 512 for the Celeb dataset and 256 for CUB and AwA2. We train all\nmodels using RTX8000 Nvidia-GPU. In all datasets, we train for up to 200 epochs and early stop if the validation loss has not improved in 15 epochs. For fair evaluation across methods, we tune the learning rate for CEMs, CBMs, and ProbCBM. Specifically, for CUB and AwA2 datasets, we explore learning rates ∈ {0.1, 0.01}, while for CelebA, we expand the search to ∈ {0.1, 0.01, 0.05, 0.005} due to the observed instability of CEMs at higher learning rates. Additionally, we set the hyper-parameter λ ∈ {1, 5, 10} for all methods. For CEMs and models trained using LDPO, we found RandInt beneficial, which randomly intervenes on 25% of the concepts during training. ProbCBM introduced a few extra hyperparameters, which we did not tune in this work, and we directly used the hyperparameters provided by the original authors. Similar to other models, ProbCBM employs RandInt at 50%, making it particularly sensitive to interventions, especially in concept-complete tasks such as AwA2 and CUB. The only model for which we tune additional hyper-parameters is Coop-CBM, where we adjust the weight parameter for the auxiliary loss.\n```\n \n\n\nWe hope our additional experiments and clarifications can help alleviate your concerns.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes, extensively.", "experimental_designs_or_analyses": "Yes, the experiments are appropriate.", "relation_to_broader_scientific_literature": "The new CBM training mechanism fixes known problems.", "essential_references_not_discussed": "All references are discussed well.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper addresses a well-known but rarely explored aspect of CBMs - concept mislabelling. Usually, concept labels are treated as almost always accurate but can be susceptible to mislabelling.\n2. The paper is well-written and easy to follow. The theoretical analysis and equations are clear to understand.\n\n\nWeakness:\n1. Diverse Experiments will strengthen the conclusion: For the \"Noised Evaluation\" experiment, the experiment setting tests the performance of such models on random concept flips. This evaluation leads to 2 problems - 1) The ideal real-world setting does not directly entail random label flip, but a label-confound, where a user can probably mislabel based on their perception of the image. 2) If the model still performs almost as well when the label flips with 0.4 probability, it is deviating from its empirical evidence and relying more on the posterior. For (2) - should we actually ignore empirical evidence?\n2. Confusing Writing: The paper can benefit much more from improving the flow and writing in the initial sections. A dedicated paragraph discussing the example figure in simple English can make the persuasiveness and overall appearance much more appealing. One has to shift to Section-3 for a thorough understanding directly.", "other_comments_or_suggestions": "Refer weakness", "questions_for_authors": "How are the hyperparameters tuned for the model?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jypn9BdPZQ", "reviewer": "Reviewer_sJF1", "summary": "The paper proposes training CBMs (and its variants) using Concept Preference Optimization (CPO) -  a method directly borrowing from the Preference Optimization (PO) literature", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their time and effort in helping us improve our work.\n### W 1  \nWe agree. We will address these concerns in the camera-ready version. We discuss the details in comment 2 of our response to reviewer **2knp**.  \n\n### W 2  \n  We also agree (thank you). As you and other reviewers have pointed out, experiments with more structured noise would strengthen our results. To address this, we have designed two experiments with more structured noise to the concepts than our original setting.  \n\n### Noising by Group Level\nAs you suggested, we now study mislabeling similar concepts based on concept groups in the CUB and AwA2 datasets. This means we introduce noise to semantically similar concepts, such as switching *red wings* to *brown wings* (as in your example). In this experiment, we apply noise at the group level with different noise levels,  $p\\in$ {$\\{0.1,0.2,0.3,0.4\\}$}.\n\nFull results are below. We observe that models trained using BCE experience a substantial drop in performance, whereas CPO models remain much more robust. For instance, in CUB task accuracy in CBM CPO remains relatively stable even at high noise levels.  Likewise, in AwA2 we observe CPO models are able to preserve Concept AUC performance much better compared to their BCE counterparts.\n\nUnfortunately, due to character limits in our response, we are restricted to providing links to these results. If any reviewer is uncomfortable clicking the link, we will try to provide a version of the full table in our next (final) response. \n\nCUB and AwA2 Noising by Group:\n\nhttps://ibb.co/NgDZ9sBC\n\n### Uncertainty Experiment\nAdditionally, we leverage a useful property of the CUB dataset: labelers provide confidence scores for their annotations. These scores range from $\\{1,2,3,4\\}$, where higher values indicate greater confidence. We introduce noise proportionally to these scores—labels with lower confidence are more likely to be flipped. Specifically, we apply noise at the following rates:  \n- Confidence 1 → 40% noise  \n- Confidence 2 → 30% noise  \n- Confidence 3 → 20% noise  \n- Confidence 4 → 10% noise  \n\nCUB confidence-based results:\n\nhttps://ibb.co/KcfB6qHp\n\nAs in other experiments, BCE-based models are heavily impacted by noise, leading to significant drops in task accuracy and concept AUC. In contrast, CPO-based models consistently exhibit robustness to noise, regardless of their structure.  \n\nWe thank all reviewers for suggesting experiments with more structured noise, as we believe these results significantly improve the quality of our work.  \n\n### W 3  \nThis is a good point; improving visual clarity is important. We are experimenting with converting Figure 4 into a table. However, we initially chose a plot for its space efficiency. At a minimum, we will include a table with all metrics in the appendix for the camera-ready version.  \n\nAdditionally, if we retain the figure, we may remove Prob-CBMs from the plot to enhance clarity, restricting their results to the appendix given their underperformance in this setting.  \n\n# Questions  \n\n` What exactly is the ground truth concept distribution that the preference is calculated against?...`\n\nThe preferred concepts are indeed taken from the empirical dataset. We  discuss it in lines 165-170:  \n\n*\"To circumvent this issue, we can leverage the empirical dataset and state its preference over a concept set sampled from $\\pi_\\theta$. The preference over a pair of concepts should hold specifically early on in training where the policy is suboptimal compared to the empirical data.\"*  \n\n` What are the computational overheads, if any...`\n\nWe thank the reviewer for suggesting improvements to our computational analysis. Below, we provide a detailed breakdown of the computational overhead associated with each model.  The following table reports the average time per epoch (in minutes) for all models, measured over a full run on the CUB dataset. Additionally, we include the number of trainable parameters for each model.  \n\nhttps://ibb.co/8gL3ZNLq\n\nWhile CPO introduces a small computational overhead, the increase is minimal—approximately 0.05 minutes per epoch compared to BCE. This is in contrast to CEM and ProbCBM, which significantly increase runtime. Moreover, we compare an unoptimized implementation of CPO against an optimized BCE implementation built specifically for efficiency in PyTorch. This suggests that further optimizations could reduce CPO’s overhead even more.  \n\nOn the other hand, ProbCBM has 4× the number of parameters compared to CBM and nearly doubles the training time, despite underperforming CPO in most settings.  \n\nA potential limitation of CPO is that its gradients tend to be smaller—particularly in the early stages of training—compared to BCE. This could lead to slower convergence in some cases. However, as we also mention in our response to reviewer dJqj, we do not observe this empirically.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes, all experiments are standard.", "relation_to_broader_scientific_literature": "The paper studies an important problem of CBM training with a trending topic of PO.", "essential_references_not_discussed": "All references present.", "other_strengths_and_weaknesses": "Strengths:\n1. Strong work combining interesting PO approaches with CBMs.\n2. Empirical evaluation and theoretical justification are sound.\n3. The work is a step towards increasing the robustness of CBMs.\n\nWeakness:\n1. Presentation Issues: The paper can significantly benefit from presentation issues. For example, 3 pages into the reading, there is only one motivation example which is not discussed well. I would implore authors to add some concrete examples and motivation in clear sentences to make this a more appealing paper.\n2. The experiments seem too synthetic: Even though I understand the motivation of the approach, and the experiment design - the setting is a bit too synthetic for a good analysis. In particular, label flipping with a probability can be one of the experiments, but how about mislabelling *similar* concepts, which is a much more likely scenario? For example - red wings flip to 0, brown wings flip to 1. You can utilize CLIP/Bertscore to do something like that. This will make the experiments much more \"life-like\".\n3. Better presentation of Figure 4: Fig-4, the most important figure in the paper suffers from a lack of visual intuitiveness in my opinion. The bars are just too close together to convey a strong improvement message. Why not just make a Table, like Table-1?", "other_comments_or_suggestions": "Refer Weakness.", "questions_for_authors": "1. What exactly is the ground truth concept distribution the preference is calculated against? Is it taken directly from the dataset annotation or is it done manually?\n2. What are the computational overheads if any? Are there ANY limitations at all?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UQuLfIPKAq", "reviewer": "Reviewer_3SWh", "summary": "One limitation of concept bottleneck models (CBMs) is that their training requires the set of correct concept annotations for all samples. However, concept mislabelling is inevitable due to labeling noise or subjective annotations. To this end, this paper proposes a CBM that is robust to concept-label noise. Specifically, inspired by the recent progress in Preference Optimization, Direct Preference Optimization (DPO) in particular, they introduce a novel objective function for training CBMs, named Concept Preference Optimization (CPO). Theoretically, they show the similarities and differences between the traditional BCE loss and the CPO loss, explaining why the latter is more robust to concept-label noise. In addition, they demonstrate that CPO is equivalent to learning the concept’s posterior distribution. Empirically, they show that CBMs trained with the CPO loss perform better in both un-noised and noisy environments.\n\n## Update after rebuttal\nI keep my score since the author's responses have addressed my questions.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and encouragements. Here we provide some detailed responses to some of the questions you have raised. \n\n`In Eq (6),  should $D$ be $\\mu$ ?`\n\nYes, that is a notational mistake on our end. We will fix it.\n\n`1. In Eq. (5), are c and c′ scalars or k-dimensional vectors?`\n\nIn our work, Eq. (5) uses scalars. However, this is not a strict requirement—one could use vectors to structure concepts in this way. That said, scalars are significantly less computationally demanding, as modeling structured objects would lead to $2^k$ possible concept representations, which could become impractical.  \n\n`2. In the later training stages, could c′ ≻ c occur? If so, would continued training hurt the model's ability to distinguish preference pairs?`  \n\nYes, this situation can arise later in training. To mitigate this, we apply early stopping across all models. This ensures that the model does not overfit to later-stage preferences in a way that degrades its ability to distinguish concept quality.  \n\n`3. Is the reason CPO is more robust against concept-level noise that DPO behaves like BCE when c ∼ μ+, but for c ∼ μ−, DPO yields smaller gradients?`\n\nThis intuition is correct. In **Appendix D**, we show that when noise is low (e.g., **0.1**), CPO’s gradients are indeed smaller than BCE’s. This is consistent with our experiments, where BCE performs reasonably well under **p_noise = 0.1**. However, at **p_noise = 0.3**, CPO’s gradients are **orders of magnitude smaller** than BCE’s. This aligns with our empirical findings—CPO remains relatively unaffected by high noise levels, whereas BCE struggles significantly.  \n\n`4. Could CPO be used to identify mislabeled concepts or even recover correct labels?`\n\nThis is an interesting question. While we have not explored this explicitly, we have qualitatively found that CPO’s uncertainty estimations seem more grounded to the target object in the image. To show this, we provide examples of images before and after various augmentations, along with the concept being analyzed. Our uncertainty score is based on the variance of the concept, normalized to a [0,1] range, where 0 indicates complete certainty and 1 represents maximum uncertainty. Results show that when the target object becomes obscured, CPO more effectively increases its concept uncertainty, while both BCE and ProbCBM perform significantly worse at this task. Additionally, we observe that in certain scenarios, cropping the image creates a zooming effect that makes models more confident in their assessments.\n\nhttps://ibb.co/JR9DZvfd\n\nhttps://ibb.co/219TTtLN\n\nhttps://ibb.co/23Rkwqqr", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing theoretical and empirical evidence.", "methods_and_evaluation_criteria": "The datasets used in this paper are the standard benchmarks used to evaluate the performance of CBMs, including CUB, AwA2, and CelebA.\n\nThe studied problem (i.e., training a model robust to concept-level noise) is well motivated because there indeed exists much noise in these datasets.", "theoretical_claims": "The theoretical claims and proofs seem correct to me.", "experimental_designs_or_analyses": "The experimental designs seem valid to me, because the datasets used in this paper are the standard ones and the experiments have studied two different environments—with and without concept-level noise (even with different levels of noise), which well demonstrate the effectiveness of the proposed method across various scenarios.", "relation_to_broader_scientific_literature": "This paper focuses on the robustness of CBMs against concept-level noise, which is a well-motivated problem and may inspire further research on removing concept-level noise in datasets or building noise-robust CBMs.\n\nThis paper designs a loss function based on Direct Preference Optimization (DPO) and may inspire further research on combining the field of preference optimization and the area of concept bottleneck models.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "This is a solid piece of work, clearly defining the problem, formulating the method, theoretically analyzing the properties of the proposed loss in comparison to the traditional loss, and evaluating its effectiveness across various environments.", "other_comments_or_suggestions": "In Eq (6), $D$ should be $\\mu$?", "questions_for_authors": "1. In Eq (5), are $c$ and $c^\\prime$ a scalar or a k-dimensional vector?\n\n2. Different from DPO which adopts a static preference dataset, the proposed method CPO adopts a dynamic dataset because $c^\\prime$ is sampled from the current model $\\pi_\\theta$. My question is: I understand that in the early training stage, $c \\succ c^\\prime$, because $\\pi_\\theta$ is not well-trained at that time. But, if it is possible that in the late training stage, $c^\\prime \\succ c$ could happen? If this is the case, will continuing training the model with the proposed loss hurt the model's ability to distinguish preference pairs, i.e., give a higher score to the better concept?\n\n3. Can I conclude the reason why CPO loss is more robust against the concept-level noise as follows: DPO is the same as BCE when $c\\sim\\mu^{+}$; however, when $c\\sim\\mu^-$, DPO yields smaller gradients?\n\n4. I am just curious: in the future, if it's possible to utilize CPO to identify mislabeled concepts or even find the correct labels for these concepts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ONcEItDeWl", "reviewer": "Reviewer_dJqj", "summary": "CBMs aim to improve explainability of models by making decisions based on human-interpretable concepts but often suffer from mislabeled concept data, leading to significant performance drops. To address this, the Concept Preference Optimization (CPO) objective is introduced, leveraging Direct Preference Optimization to reduce the impact of concept mislabeling. The authors compared CPO  to conventional BCE across multiple datasets to evaluate robustness against concept noise.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and helping us improve our work. \n\n`Consider visualizing gradient behavior over time...`\n\nWe agree with the reviewer that this figure can aid in illustrating our claim. A version of it is already in App D, where we empirically confirm our theoretical results. \n\n`1 Sequential CBMS`\n\nWe chose to focus on joint CBMs as they are shown to generally be the best-performing models for a variety of tasks. Below, we provide some results for the unnoised setting using sequential CBMs. We generally observe that CPO provides better results for both the sequential and joint setting, specifically in task accuracy, where we find that the added uncertainty estimates encoded into the concept representation allows for more expressive modeling to be picked up by the decoder. \n\nSequential CBM results: https://ibb.co/bMJKXcLP\n\n` 2. What happens when empirical data is systematically biased?`\n\n1.1  We agree that incorporating experiments with more structured noise would strengthen our conclusions. Specifically, in our response to reviewer **sJF1** we describe the following experiments:\n\nOne where noise is introduced at the concept group level (e.g., flipping only wing-related attributes in birds or altering beak colors).\n\nAnother where noise is introduced based on the confidence level provided by the labeler (as available in the CUB dataset).\n\n` 3.Strong assumptions`\n\nWhile our results do rely on the assumption of a uniform prior, this is the weakest prior one can have. We show in Sec 5.3 how using a prior improves the performance of the model through the added flexibility. This ability to leverage a prior (which BCE cannot) but still work using an uninformative one is beneficial. Thus, we believe that while it is an assumption that our results rely on, it is not impairing. \n\nWhile we acknowledge that concepts are often correlated, assuming conditional independence is a common approach in CBM literature, with dedicated works exploring dependency modeling (e.g., stochastic CBMs [1]). Given this, we opted for a model that assumes conditional independence.\n\nThis feedback has prompted us to reflect on whether conditional independence is necessary for the correctness of our analysis. Upon review, we found that it is not required, meaning our results can generalize to more expressive models.\n\nTo illustrate this, consider modeling the joint distribution of a concept with all other concepts, $\\pi(c_i, c_{0:i-1}|x)$ . Using the chain rule, this can be factorized into an autoregressive formulation:\n\n$$\n\\pi(c_i | c_{i-1}, x)\\pi(c_{i-1} | c_{i-2}, x) \\dots\n$$\n\nWith this factorization, we can always compute the joint distributions $\\pi(c | x)$ and $\\pi(c' | x)$, ensuring that our results in Appendix C.1 still hold even without only assuming conditional independence on x.\n\nThe main reason for the independence assumption was to simplify the derivation of the CPO loss in Eq. (6). Since we find this assumption unnecessary, we will drop this assumption from our work. \n\nHere, we present preliminary results using this autoregressive formulation. Specifically, we implement this approach with two networks: $ \\pi_{\\phi}(c_{i} \\mid c_{0:i-1}, x) $ and $ h_{\\omega}(\\tau \\mid c, x) $, where $ h_{\\omega} $ determines the order of the autoregressive decomposition.  In this process, $h_{\\omega}(\\tau \\mid c, x) $ takes as input the concept probabilities predicted by $\\pi $ and produces a softmax distribution over concepts. The concept with the highest probability is then selected and predicted by $\\pi $, and this procedure repeats in an autoregressive manner until all concept probabilities have been obtained. \n \nAR CBM Table: https://ibb.co/BH9s5jcm\n\nHere, we find that our autoregressive model beats conditionally independent models for BCE, slightly underperforming for CPO. Regardless, we find that CPO also performs better for this type of model. \n\n` 4. How does LCPO perform in early training phases? Proposition 4.2 suggests LCPO’s gradient updates are more conservative than BCE’s, potentially leading to slower convergence...`\n\nIn practice, CPO can be slower in fitting training data, but we observe that this does not negatively impact generalization. Instead, it suggests that CPO may be less prone to overfitting, which is a known issue in likelihood-based training. To illustrate this, we provide anonymized links for sample runs on the CUB dataset across two seeds.\n\nhttps://ibb.co/Z67qYySF (Validation Task Accuracy)\n\nhttps://ibb.co/4wkNn1WW (Validation C AUC)\n\nhttps://ibb.co/Xx8WNLnZ (Training task accuracy)\n\nhttps://ibb.co/chxX81Jf (Training C AUC) \n\nRegarding adaptive learning rates or entropy-based updates, any improvements seen from these strategies should also apply to CPO, as its updates are fundamentally based on data likelihood.\n\n`Computational Analysis`\n\nPlease see our response to reviewer **sJF1**\n\n\n[1] Stochastic Concept Bottleneck Models  https://arxiv.org/html/2406.19272v1", "claims_and_evidence": "The paper claims that CPO improves the robustness of CBMs by replacing correctness assumptions with a preference-based optimization approach. Theoretical results and empirical findings suggest that CPO leads to better performance under noisy concept labels than traditional BCE. The authors provide mathematical derivations to show that CPO's gradient updates remain closer to the optimal noise-free gradient than BCE's, which serves as the main theoretical justification for CPO's robustness. However, while the claims are well-motivated and backed by theoretical derivations, the scope of noise modeling is limited to empirical concept label noise and does not consider more complex, structured noise models (e.g., systematic bias or adversarial noise). Additional empirical results exploring these settings would strengthen the evidence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The primary theoretical claim is that CPO is more resilient to noise than BCE due to its gradient properties, specifically, that LCPO’s gradient updates are closer to the optimal noise-free gradient than BCE’s under constant noise conditions. The authors prove this with Proposition 4.1 and Theorem 4.3, which mathematically formalize this gradient similarity. The main limitation here is that the assumptions behind these proofs are strong and may not hold in all real-world cases. First, this assumption of independent and uniform priors for concepts is unrealistic. Many concepts are correlated (e.g., \"feathers\" and \"wings\" in birds). Second the noise model assumes random label corruption but does not consider structured noise, such as systematic bias in datasets or adversarial noise attacks. A stronger theoretical contribution could involve relaxing these assumptions and extending the analysis to more realistic noise distributions.", "experimental_designs_or_analyses": "The experimental setup appears reasonable. See below for more.", "relation_to_broader_scientific_literature": "Important topic.", "essential_references_not_discussed": "Thorough analysis.", "other_strengths_and_weaknesses": "Strengths:\n- The paper presents a novel preference-based optimization approach that relaxes strict correctness assumptions, making it highly relevant for noisy real-world settings.\n- Theoretical results are clearly derived, and mathematical proofs support key claims.\n- The empirical study demonstrates improvements on multiple real-world datasets, suggesting that CPO has practical benefits.\n\nWeaknesses:\n- Strong assumptions (e.g., uniform priors, concept independence) may limit generalizability.\n- No discussion of computational efficiency—CPO introduces an online learning approach, but the paper does not analyze how training time or model complexity compares to BCE.\n- Limited evaluation on different noise types—structured or adversarial noise is not explored.", "other_comments_or_suggestions": "- Table 1 does not follow the order of baseline introduction. Please align the presentation for clarity.\n- Consider visualizing gradient behavior over time. A plot comparing LCPO and BCE gradients over training iterations could illustrate the robustness claim more effectively.", "questions_for_authors": "- Why did the authors only evaluate the jointly trained CBM and not the sequential one?\n- What happens when empirical data is systematically biased? The paper assumes empirical preferences are more reliable than random sampling, but what if systematic annotation errors exist (e.g., domain shifts, adversarial perturbations)?\n- How would you model concept dependencies (e.g., \"wings\" and \"feathers\" in birds) ? The assumption of conditional independence between concepts is unrealistic. How could correlated concept structures be incorporated into CPO? \n- How does LCPO perform in early training phases? Proposition 4.2 suggests LCPO’s gradient updates are more conservative than BCE’s, potentially leading to slower convergence. Could adaptive learning rates (scaling updates based on entropy) address this?\n- How does CPO handle dataset biases? Since LCPO only modifies the policy when incorrect concepts are sampled, it assumes mislabeled concepts are uniformly distributed. However, in practice, certain incorrect concepts may dominate due to dataset biases. Would reweighting techniques help mitigate this issue?\n- Provide a computational efficiency analysis. Given that CPO involves online preference updates, how does its training time and memory usage compare to BCE?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this is solid work that combines interesting PO approaches with CBMs.
The work is a step towards increasing the robustness of CBMs, and it would be of interest to the ICML community
Decision: Accept (poster)