On The Quality Assurance Of Concept-Based Representations
Authors: Mateo Espinosa Zarlenga
Pietro Barbiero
Zohreh Shams
Dmitry Kazhdan
Umang Bhatt
Mateja Jamnik
Venue: ICLR 2022 Submitted (2022)
Keywords: Concept learning
Disentanglement learning
Explainability
Interpretability

Abstract: Recent work on Explainable AI has focused on concept-based explanations, where deep learning models are explained in terms of high-level units of information, referred to as concepts. In parallel, the field of disentanglement learning has explored the related notion of finding underlying factors of variation in the data that have interpretability properties. Despite their overlapping purpose, the metrics to evaluate the quality of concepts and factors of variation in the two fields are not aligned, hindering a systematic comparison. In this paper we consider factors of variation as concepts and thus unify the notations in concept and disentanglement learning. Next, we propose metrics for evaluating the quality of concept representations in both approaches, in the presence and in the absence of ground truth concept labels. Via our proposed metrics, we benchmark state-of-the-art methods from both families, and propose a set of guidelines to determine the impact that supervision may have on the quality of learnt concept representations.

Under review as a conference paper at ICLR 2022
ON THE QUALITY ASSURANCE OF CONCEPT-BASED REPRESENTATIONS
Anonymous authors
Paper under double-blind review
ABSTRACT
Recent work has focused on concept-based explanations, where deep learning models are explained in terms of high-level units of information, referred to as concepts. In parallel, the field of disentanglement learning has explored the re- lated notion of finding underlying factors of variation in the data that have inter- pretability properties. Despite their overlapping purposc, the metrics to evaluate the quality of concepts and factors variation in the two fields are not aligned, hindering systematic comparison. In this paper, we consider factors of variation as concepts and thus unify the notations in concept and disentanglement learning. Next, we propose metrics for evaluating the quality of concept representations in both approaches, in the presence and in the absence of ground truth concept la- bels. Via our proposed metrics, we benchmark state-of-the-art methods from both families, and find that, contrary to common assumption, supervision alone may not be sufficient for quality assurance of concept representations. In light of this. we propose a set of guidelines to determine the impact that different degrees of supervision may have on the quality of learnt concept representations.
INTRODUCTION
Addressing the lack of interpretability of deep neural networks (DNNs) has given rise explain- ability methods. most common of which are feature importance methods (Ribeiro et al. [016 Lund- berg & Lee (2077) that quantify the contribution of input features to certain predictions (Bhatt al.]
2020). However, input features are not necessarily the most intuitive explanations, in particular when using low-level features such as pixels. Concept-based explainability (Ghorrbani al., 2019 Koh et al. 2020 Yeh et al.) 2020 Ciravegna et al.) (2021) remedies this issue by constructing an explana- tion at a concept level, where concepts are considered high-level and semantically meaningful units of information commonly used by humans to explain their decisions. Furthermore, concepts allow users to improve a model's performance via concept interventions, in which mispredicted concepts corrected using expert knowledge (Koh et al. 2020)..
In practice, what constitutes a concept is data-dependens ranging from a group of pixels for image data (Ghorbani al., 2019 Koh et al. 2020 Yeh et al. 2020), to sequence of words and sub-graphs for text and graph- data, respectively (Yeh et al. 2020) Magister et al.) (022). While all of these definitions are specific the data modality, they commonly refer to an intermediate representation of the input data that has certain properties. Summarising concepts as intermediate representations of the data makes them analogous to factors of variation in disentanglement learning, where the assumption is that there exists generative process capable of producing a nigh-dimensional dataset using a finite number factors (Bengio et al.) 2013). Such factors constitute disentangled inter- mediate representation of the data with interpretability (Bengio et al.] 2013 Higgins et al.) 2017), fairness (Creager et al. 2019), and predictive performance (Locatello al. 201 2020b) properties. The difference between Concept Learning (CL) and Disentanglement Learning (DGL) remains in that concepts in CL are often formed based on the supervision directly from concept labels or from downstream task, whereas generative models (e.g., Variational Autoencoders (VAEs) (Kingma & Welling. 2014 Higgins et al., 2017)) that serve as the basis of DGL are un-/semi- -n-pssemi-upprrvieed and factors of variation are directly informed by the distribution of the input data.
Since CL and DGL were developed independently, much of their connection remains unexplored. In particular, the metrics used to evaluate the quality of intermediate representations in each sub-

---

Under review as a conference paper at ICLR 2022
field are not aligned, despite their overlapping goals. Metries in the concept literature (Koh al. 2020 Kazhdan et al.] 2020 Yeh et al. 2020) are mainly concerned with the properties of learnt concepts w.r.t. the downstream task. On the other hand, given the lack of downstream task, met- rics in disentanglement literature (Higgins et al.) 2017 Ridgeway & Mozer 2018 Locatello et al.) 2019) are mainly concerned with the properties of the learnt representations, referred to as latent codes, w.r.t. the ground truth factors of variation. We argue that concepts/latent codes, as surrogate to inputs, need have the following key properties: (i) They should correspond to semantically meaningful and coherent input sub-spaces; (ii) They should preserve the amount of mutual informa- tion observed in ground truth concepts or factors of variation (when available); and (iii) They should capture sufficient statistics to predict the downstream task (when available) as well as raw inputs do.
In this paper, we consider the properties (ii) and (iii) and make the following key contributions:
We unify the language and notation across CL and DGL by framing factors of variation and latent codes in DGL as ground truth concepts and concept representations in CL, respectively.
We introduce metrics for evaluating the quality of learnt concepts/codes in presence and absence access to ground truth concepts/factors of variation and when concepts/codes are correlated.
We conduct a systematic empirical comparison of state-of-the-art methods from four families of methods: supervised CL. unsupervised CL, semi-supervised DGL, and unsupervised DGL. We make the code used for our metrics, methods, and datasets available in an open- library
2 BACKGROUND AND RELATED WORK
Notation In both CL and DGL the aim is to find a low-dimensional intermediate representation that explains the downstream task(s) in CL. or the data's factors of variation in DGL. In CL., this low-dimensional representation corresponds to a matrix è E Ć C Raxk in which the i-th column constitutes a d-dimensional representation of the i-th concept. As zero-padding can be used to ensure equal length across different concept representations, for notational simplicity we assume that all concepts use a d-dimensional vector as their representation. Under this view, elements in ((:.)) E Ed are expected to have high values (under some reasonable aggregation function) if the i-th concept is considered to be activated for the input that generated this representation. For example, in the case where d T and each concept can take up to T discrete values, ((..) E [0, 1]" can represent probability distribution over all values that the i-th concept can take. As most CL methods assume = 1, for succinctness we use é in place of C(.,) when d = 1.
We adopt the same representation for latent codes in DGL and let 2 E 2 C dg be latent code matrix such that each dimension ((.,)) (or a non-overlapping subset of dimensions) encodes one, and only one, independent factor of variation zj. Nevertheless, note that in practice, d tends to be I for most DGL methods. Finally, for simplicity. we use è to refer to both learnt concept representations and latent codes. Ground truth concepts and factors of variations
referred to as E C C R
In line with (Koh et al. 2020 Kazhdan et al. 2020 Ych et al., 2020) we make use of: (i) a concept encoder function g X" + C' that maps transformation of the inputs x E X C R"", as performed by function Φ X + X", to a concept intermediate representation; and (ii) label predictor function f C + Y that maps the concept representations to downstream task's set of labels E Y C RL. These two functions can be combined to give a set of predictions for sample x E X by computing f((((x))). In DGL autoencoders, one can think of (g o (.) as the autoencoder's encoder model and of the autoencoder's decoder model as a function that approximates (go (.).
Supervised concept learning In supervised CL, access to concept labels c( E N² in addition to target labels y(1) E RR, is assumed for inputs x+) E R In other words, we have training data ((x(() c( ( where N is the number of training samples. In its most common form, supervised CL divides the prediction into two distinct steps of: (i) mapping input sample to its concept representation via a concept encoder g: and (ii) mapping a sample's concept representation to its task labels via label predictor f(.). Together, these two functions constitute Concept Bottleneck Model (CBM) (Koh et al. 2020), because their final prediction relies on the input going through the bottleneck g(o(x)) which is trained to be component-win wise aligned with c.
Code will he released after review.
2

---

Under review as a conference paper at ICLR 2022
constructs CBM from pre- trained model by building a non-trivial o(-) mapping function using the model's ' latent space. Using such latent representation instead of raw inputs typically makes CME more data efficient than CBM (Kazhdan et al., 2021). Similarly, Concept Whitening (CW) (Chen et al. 2020) constructs a CBM by introducing pluggable batch normalization module whose activations (goo)() are trained to be aligned with representative sets of binary concepts. It achieves this by forcing different feature maps of the normalization module be decorrelated and orthogonal while incentivizing activations in given axis to be high when its corresponding pre- rre-defined concept is activated.
Concept-based Model Extraction (CME) (Kazhdan et al., 2020)
Unsupervised Concept Learning Unlike supervised CL, in unsupervised CL concept annotations are not available and concepts are discovered in an unsupervised manner. Ghorbani et al.] (2019)
extract concepts from trained classifier for image data. Images belonging to each class are first segmented with multiple resolutions. The segments are then clustered as examples of class concepts and their importance scores are measured using TCAV (Kim et al., 2018).
Unlike Ghorbani et al. (2019), Completeness- Concept Discovery (CCD) (Yeh et al. 2020) is data modality agnostic and extracts class-independent concepts. CCD builds on TCAV to first extract a set of concept vectors (h }- each of which a unit vector in X'. These vectors are then used to construct a concept representation g(o(x)) = E Ć C R by setting & to TH(((xx) h()),, (())),)) B), the 3-thresholded inner product of (xx) and concept vector h(") When complete, a concept rep- resentation should contain sufficient statistics obtain high performance in the original model's
prediction task (((x)). If this is the case, then there must exist a mapping 1/ C ト X' that re- covers o(x) from (())))) Thus, CCD constructs a set of concept vectors such that f(((((((x)))) is able to achieve similar task performance that of [(((x)). Notice that although f(() is still label predictor function here, it is not applied to the raw inputs or to their concept representation.
Similarly, Self- Eellaaxxlain Neural Networks (SENNs) (Alvarez-Melis Jaakkola| 2018) learn to produce concept-based explanations without explicit concept supervision through robustness reg- ularization term that encourages differentiable model to locally act linearly. It proceeds by first learning concept representation g(x) è E RR from the encoder of an autoencoding model and, second, generating prediction f(ê) ((
(xx)x(+) using an aggregation function g to weight the importance of each concept with a linear coefficient learnt through differentiable
model (()). Concept weights 0(x) can then serve as an explanation for SENN's predicted label.
Disentanglement Learning Generative models (e.g.. VAFS (Kingma & Welling 2014)) used in DGL assume that data is generated from a set of independent factors of variation c E C. sampled from factorable distribution p(c) II, p(c;), such that a sample is generated according to the conditional distribution p p(x(c). Thus, the goal of DGL is to find a function g-)) that maps inputs to disentangled latent representation, such that a subset of non-overlapping dimensions of the latent representation corresponds to a unique factor of variation C,.
In light of recent work showing the theoretical impossibility of learning disentangled representations in an unsupervised manner (Locatello et al., 2019). a promising line of work suggests providing the inductive bias required to learn disentangled representations through weak supervision. In this work, we focus on DGL methods where weak supervision comes via pairs of observations whose corresponding ground truth factors of variation share at least one common element (Locatello al.) 2020a) and contrast them against vanilla unsupervised DGL methods such as VAES (Kingma & Welling 2014) and B- VAEs (Higgins al. 2017).
3 QUALITY ASSURANCE OF CONCEPTS
Given the varying degree of supervision in the approaches above, the quality of their concept repre- sentations is expected to vary. The quality of learnt concepts w.r.t downstream tasks has commonly been studied using predictive performance metrics such as accuracy and AUC. Similarly, there exist several metries for measuring the properties of learnt concepts w.r.t. the ground truth ones, either focusing on the overall set of learnt concepts or focusing on individual concepts (see Appendix 6. for summary). Nevertheless, these metrics have two main shortcomings.
The first shortcoming is assuming that concepts are independent, implying that a concept repre- sentation encodes information beyond that of the ground truth concept it is aligned with, it counts as
3

---

Under review as a conference paper at ICLR 2022
leakage (Mahinpei et al. 2021). Impurities caused by leakage can have far-reaching consequences in terms of human interventions, e.g., perturbations of an impure concept may impact other con- cepts unintentionally. While we agree with the danger of impurities, real-world often have dependencies, and full disentanglement is not realistic. As such, leakage as described above is not necessarily undesirable when it reflects the correlations found in ground-truth concepts, To support this, we demonstrate the lack of inter-concept independence in a real-world dataset, the Caltech-UCSD Birds-200-2011 (CUB) (Wah al. 2011a), in Appendix 6.2
argue that concepts in the
The second shortcoming is the assumption that one has access to ground truth concepts, which are used verify the properties of the extracted ones. Previous work has shown that access to ground truth concepts is limited given the difficulty of annotating samples with concept labels (Raghavan al. 2006). The rarity of real-world datasets with such annotations is evidence for this claim.
To address these shortcomings, we first propose the oracle impurity score, metric that measures the quality of concept representations when one has access to discrete ground truth concepts labels which may be correlated. We then propose the niching purity score and niching impurity score, two efficient metrics which, in conjunction, measure the quality of concept representations when one has only access to a classification task's labels.
3.1 ORACLE IMPURITY
To circumvent the need for inter-concepi independence, we take inspiration from (Mahnnpei et al., 202T]. where they informally measure concept impurity as how predictive a CBM-generatee concept probability is for the ground truth value of other independent concepts. If the pre-defined concepts are independent, then the inter-concept predictive performance should be no better than random. To generalise this assumption beyond independent concepts, we first measure the predictability of ground truth concepts w.r.t. one another. Then we measure the predictability of learnt concepts w.r.. the ground truth ones. The divergence between the former and the latter acts an impurity metric, measuring the amount of undesired information that is encoded by the learnt concepts. In order to formally introduce metric, we begin by formalising a purity matrix.
li=1 1T= and corre- sponding discrete ground truth concept annotations I' = {c(i) E N assume that I' and I are aligned clement-wise for all I E {1,.... ,k), ,, the 1-th concept representation of 2(0) encodes for the same concept as the 1-th concept label in clt), The Purity Matrix of given ground truth labels T is defined as a matrix π (f,I) E 1j. whose entries are given by:
Definition 1 (Purity Matrix). Given concept representations f = (E()) E Rdxkin
T (f.I) (1.5) AUC({ (15, (2)) (.) ((1))
where (-)-)) is classifier (e.g.. an MLP) trained to map the i-th concept's d-dimensional vector representation ((,i) to a probability distribution over all the values that concept j may take.
The (i.j)-th entry of ক (f, contains the AUC when predicting the ground truth value of concept j given the i-th concept representation. The diagonal entries of this matrix show how good a concept representation is at predicting its corresponding ground truth value, while the off-diagonal entries show how good such representation is at predicting the ground truth labels of other concepts. Intuitively, one can think of the (i,j)-th entry of this matrix as a proxy of the mutual information between the i-th concept representation and the j-th ground truth concept. While in principle several other methods could have been used to estimate this mutual information, we choose AUC primarily for two reasons: (1) it is computationally tractable and (2) it allows the construction of a metric that can be easily bounded (as discussed below). Furthermore, while in this work, as in the majority of the existing CL literature (Koh al. 2020 Chen et al. 2020 Yeh al. 2020), we focus on concepts that are binary in nature, notice that one can trivially I extend our definition to be applicable multivariate concepts by using the mean one-vs-all AUC across all possible labels a ground truth concept may take. Details on how this matrix is computed in practice can be found in Appendix 66.3 This definition of a purity matrix allows us to construct a metric for quantifying the amount of unnecessary information, or impurity, that a concept encoder encodes.
Definition 2 (Oracle Impurity Score (OIS)), Let g :::: X + I C Rd be concept encoder and let Tx (x(() E X) and T {c(O Nkin Je=o be ordered sets of testing samples and
4

---

Under review as a conference paper at ICLR 2022
their corresponding concept annotations, respectively. If. for any ordered set A we define g(A) as g(A) = {g(a) a E A}. then the OIS of g()) given data (I',1) I) is defined as:
giS(g,Tx,1)) (
x) x). x (T.
k
where we use ||A|| F to represent the Frobenius norm of matrix A.
Intuitively, the OIS measures the total deviation of an encoder's purity matrix with the purity matrix obtained from using the ground truth concept labels only (i.e., the "oracle matrix"). We opt to measure this divergence using the Frobenius norm of their difference, instead of other similarity metrics, in order to obtain a bounded output which can be easily interpreted. More specifically,
since each entry in the difference (T(((Tx).)) - (I.I)) can be at most 1/2, the upper bound of the Frobenius norm of this difference is k/2. Therefore, our metric includes a normalisation factor 2/k which guarantees that it is in [0, 1]. This allows interpreting an OIS of 1 as a complete misalignment between π (1",1) 1)) and π (I( (i.e., the i-th concept representation can predict all other concept labels except its corresponding one even when concepts are independent). An impurity score of O, on the other hand, represents perfect alignment between the two purity matrices (i.e., the i-th concept representation does not encode any unnecessary information for predicting concept i).
3.2 NICHING-BASED METRICS
In nature, niche is defined as a resource-constrann subspace of the environment that can support different types of life (Darwin 1859.. By analogy, in the context of neural networks each layer can be seen as a computational "environment" with finite amount of computational resources (i.e., the layer parameters). A "niche of neurons" can be defined as a subspace of the layer (i.e., the "environ- ment") that is critical for the accurate prediction (i.e., "the survival") of specific downstream task (i.e., the "species living in the niche").
Inspired by the theory of niching, concept bottleneck sensitivity, and sparsity (Dimanov & Jamnik 2018 Barbiero et al. 2021) measure the quality of concept representations in the absence of access to ground truth concept labels and in the presence of task-specific labels. Our metrics quantify how task- label-separable a concept repre- sentation is. For this, they first identify set of con- cepts that are important for task label, and then see if such set, referred to as the label's concept niche, is complete enough to predict the label. The pre- dictive performance of set of concepts outside the concept niche of a label identifies impurity and un- necessary leakage in the concept representation. The existence of niches is not an assumption for the met- rics to be viable in practise, but rather observation in many real-world data (see Appendix (6.6ffor an ex- ample). We start by describing concept nicher, a function that defines the dependency between a sin- gle concept and a single task label.
propose two metrics that
ON:
Figure 1: (a) given of concepts and set of classifi- cation labels, (b) a concept nicher identifies the concepts on which label depends strongly (red blocks); thus (c) a concept niche is the set of these concepts that label depends on strongly; and (d) complement concept niche the set of concepts that label does not depend on strongly.
Concept niches and their complement:
Definition (Concept nicher). Given a label predic-
tor f + Y, where Ć C jkdkk and Y C R2L, we define a concept nicher as a Boolean-valued function V {1,... ,k} (1..... ... L} + {0, 1} that returns vy(i,j) mension of f.) has a "strong" dependency, as determined by a reasonable procedure, on the i-th concept representation C(..)) and 0 otherwise.
the j-th output di-
The above definition is affected by how a dependency is defined. One efficient way of measuring the dependency, and the way that we proceed, use the absolute Pearson correlation, denoted as p. We call such an instantiation a concept-correlation nicher (CCorrN) and define it as:
(N) CComN (i. = (( ... (. (f(E())))) .f(e(N) >
5

---

Under review as a conference paper at ICLR 2022
where 3 E [0, 1] is user-defined threshold. ((.)) is not a scalar representation (i.e., d > 1), use the maximum absolute correlation coefficient between all entries in ((,4) and the target label [(c) as a representative correlation coefficient for the entire representation ((.))- For all experiments we use 3 = 0.2. A study of the impact of 3 on CCnrn is in Appendix 6.5
For each task label, the concept niche defines the set of concepts on which the label depends (i.e.. where the concept nicher outputs 1 for the label). One can intuitively see this in Figure bb), where the red blocks show the concepts that the concept nicher has identified for y1 and y/2 and Figure Tc)
shows the niches. Formally define concept niche as:
Definition 4 (Concept niche). The concept niche N5(1z) for label j. determined by label predictor f()) and concept nicher v.(.), is defined as Nz(Uz)= = {i 1 (1,... .k) and Vy (i. = =1}.
Abusing notation, we let (ug) == (I,... .A \ (yz) be the complement of the set Nj (v)) (see Figure Id)). Although the label predictor f can be any classifier, in our experiments we use a ReLU MLP with hidden layer sizes {20, 20}. In this setup, we let fly,(e,)) be the MLP resulting from pruning all connections in except those between concept representations in the niche (vr) and the j-th output of f.
We define the Niche Purity Score (NPS) and Niche Impurity Score (NIS) as measures of the pre- dictive capacity of a niche and its complement w.r.t. a specific classification label for given label predictor function, respectively. A good concept representation has a simultaneous high NPS and low NIS for each label, as high values in the latter score symptom of unnecessary leakage in concept representation.
Definition 5 (Niche Purity Score). Given label predictor 1 C + Y, concept nicher Vs. concept representations I = {cli) and corresponding ground truth clas- sification labels IY
{yl") the NPS of the i-th output of f()) is defined as The NPS of f, defined as NPS(J,u))
NPS. vz) == (UC(L((IN(vr)) (c()) (o ΣL (PS.(J, Uy)/ L, is the mean NPS across all labels.
Definition 6 (Niche Impurity Score). The NIS of concept niche Nz(1y) is the NPS of (N,(r)).
Essentially, the NPS metric measures the quality of niche (,(r)) via the AUC of predicting the ground truth label of the i-th label using only the concept representations corresponding to niche Ne(vy). A NPS score of 1 (its maximum value), thus indicates a perfect niche purity for the i-th output of function ʃ(-) while worst-case-scenario NPS of 1/2 indicates that niche is unable to do better than random when predicting its corresponding label. Similarly, the NIS metric measures the impurity of niche Ny(ry) by measuring the AUC of predicting the ground truth label of the i-th label using concept representations outside the niche corresponding to the i-th label. Therefore, NIS score of 1/2 indicates the best-case scenario impurity as concepts outside of niche (vyy) hold no useful information to predict the i-th label. In contrast, a worst-case NIS score 1 corresponds to complete impurity as the i-th label can be perfectly predicted from concepts that are irrelevant to it.
EXPERIMENTS
Methods, Model Selection, and Training We compare the quality of concept representations in various methods using our metrics. CBM (Koh et al.) 2020) is selected from the supervised CL family due to its fundamental role in CL. We focus on jointly trained CBMs, where the task-specific loss and the concept prediction loss are minimised jointly. We also add CW (Chen et al. 2020) from this family as directly treats decorrelated concept representations. From the unsupervised CL family, CCD YYhh et al., 2020) is selected due to its data agnostic nature and SENN (Alvarez- Melis & Jaakkola 2018) due to its particular mix of ideas from both DGL and CL literature. From the DGL family, we consider two weakly supervised methods, Adaptive Group Variational Au- toencoder (Ada-GVAE) and Adaptive Multilevel Variational Autoencoder (Ada- MLVAA) (Locatello) ct al. 20200a, as well as two unsupervised methods, namely vanilla Variational AutoEncoders (VAD) (Kingma & Welling 2014) and 3 VAE (with 3 = 10) (Higgins et al., 2017). For each method and metric, we report the average metric values and 95% confidence intervals obtained from 5 different random seeds. Given that the number of concepts in unsupervised CL and DGL approaches is not known upfront, we allow extra variation, where the number of learnt concepts is twice the number

---

Under review as a conference paper at ICLR 2022
of ground truth ones. We observe that model capacity, in particular the encoder's capacity, impacts the quality of learnt concepts (see Appendix 6.7). Therefore, to minimise the effects of inductive bi- ases arising from the model architecture and training hyperparameters, and training setups for all methods within the same dataset whenever possible. We include details on training and architecture hyperparameters in Appendix (68.2
use the same architecture
Datasets In order to have datasets that are compatible with both CL and DGL., we construct datasets whose samples are fully described by vector of ground truth generative factors. Moreover, to simu- late real-world scenarios, we work with tasks with a varying degree of dependencies in their concept annotations. To achieve this, we first design a parametric binary-class dataset TabularToy(6) (6)( a vari- ation of the tabular dataset proposed by (Mahinpei et al. (2021). We also construct two multiclass image-hased parametric datasets: dSprites(A) and 3dshapes(A)) A), based on dSprites (Matthey et al., 2017) and 3dshapes (Burgess & Kim 2018) datasets, respectively. They consist of 3D samples gen- erated from a vector consisting of i and k = 6 ground truth factors of variation, respectively. Both datasets have one binary concept annotation per factor of variation. Parameters å E [0. 1] and À E {0..... k- 1} control the degree of concept inter-dependencies during generation (a value of 0 represents inter-concept independence while higher values represent stronger inter-concept de- pendencies). Details on how dependencies are introduced, as well as the nature of the individual concepts and task labels for all our datasets, can be found in the Appendix 6
4.1 RESULTS
Overall set of learnt concepts is equally predictive of the downstream task across all surveyed methods. The quality of the overall set of leant concepts w.r.t. downstream tasks, measured by its mean one-vs-all task AUC, does not vary considerably across methods and stays close to 100%. This result replicates the predictive power of the raw inputs and thus shows that the grouped concept representations faithfully capture task-related information in the input space. Plots for predictive AUC of raw inputs and the overall set of learnt concepts can be found in Appendix (6.9
Supervision may not lead to overall set of learnt concepts being more predictive of individual ground truth concepts compared with no supervision (Figure Since CBM and CW benefit from explicit concept supervision, we expect their learnt representations to predict the ground truth concepts well. While this observation holds in CBM and CW when one uses CW's entire feature maps as concept representations (shown as "CW Feature Map" in plots), it does not hold for "CW MaxPool-Mean," where one reduces the CW module's ' feature maps to scalar concept scores using a max pool followed by a mean reduction (as seen in Yeh et al. (2020). ). This in alignment with results reported in Mahinpei et al. (202T). and suggests that there may be substantial information loss when reducing CW's feature maps into scalar representations. Note that this is not observed in Toy ( as its concept representations are already scalars. Interestingly, the implicit su- pervision (i.e., supervision from downstream task as opposed to explicit concept supervision) in unsupervised CL can be just effective explicit concept supervision in other methods in terms of the predictive power of the overall set of learnt concepts for individual ground truth ones. This is observed in CCD and to lesser degree in SENN. We attribute the gap between the two CCD being designed to maximize the completeness scores of its overall set of Iearnt concepts w.r.t. the task. Surprisingly, complete lack of supervision in DGL's VAE and 3-VAE appears to be more effec- tive than the weak supervision in Ada-MLVAE and Ada-GVAE. This is unexpected given that such weak supervision was introduced to address the difficulty of learning representations that capture the data's 's underlying factors of variations in an unsupervised manner (Locatello et al.) 2019).
Metries that assume concept independence may be misleading (Figure 3).
The comparisons
above focused on the quality of overall set of learnt concepts, without an indication of the quality of individual learnt concepts w.r.t. the ground truth ones. The alignment between the learnt and ground truth concepts is clear in CBM and CW. In CCD and DGL, where there is no alignment, we compute a greedy alignment between the learnt and ground truth concepts based on the predictive AUC of using a leart concept to predict each ground truth concept (i.e., a ground truth concept is assumed to be represented by the learnt one that can predict it best). Details on this are in Appendix 6.4
After the alignment, we investigate the quality of individual learnt concepts w.r.. the ground truth ones. First, we show the inadequacy of assuming concept independence compared to our oracle impurity metric proposed in Definition 2 We achieve this by comparing our metric against a variant
7

---

Under review as a conference paper at ICLR 2022
Mean Carcept AUC (8)
Tohudar Toy(S)
eSprite
3cchapen (1
VAE Ia
CCE Enusscage 1 24]
MasPadl
Figure 2: Mean concept AUC calculated by averaging over all the AUCs of predicting individual ground-truth concepts from the entire set of learnt concepts. Each AUC is calculated using ReLU MLP with hidden layers [64, 64) that is trained to predict the target concept.
of itself, labeled as "non-oracle impurity", that assumes the oracle matrix of any dataset is always a matrix with 1 in its diagonals and 1/2 in its off-diagonals. We observe that the non-oracle impurity misleadingly shows more impurity as the degree of inter-concept dependencies increases. This is due to the fact that our metric, in contrast to the non-oracle impurity, takes into account the dataset's inherent ground truth inter-concept dependencies as a baseline for what a method's purity matrix should look like. We observe the same results across other methods (see Appendix 6.10
CBM Impurity (3dshapes E3)
Ado MIVAE Impurity (3ddhapes +1)
Impority (3ddhapes--)
Assitoiig Ada srolld jusert
Jan ClB4 prou mny
CCD boack inpulee
CCD janceps preeods hepurg)
Figure 3: Impurity scores (both oracle and non-oracle) in 3dshapes( (A)-
Oracle impurity demonstrates that explicit or implicit supervision may not result in purer individual concepts than no supervision (Figure 4.. The oracle impurity metric results demon- strate that CBM's individual concepts consistently experience the least amount of impurity due to receiving explicit supervision, which is be expected. Unexpectedly, though, we observe that the same explicit concept supervision can lead to highly impure representations: CW's full feature maps concepts encode the highest amount of impurity on average, a phenomenon observed by Mahinpei et al. (222)) too. This is due to the fact that the learnt concepts are highly predictive of every ground truth one irrespective of their alignment. Thus, while explicit supervision translates better concept predictive performance, it does not necessarily translate to purer concepts. Looking into implicit su- pervision, while the overall set of learnt concepts CCD performs as well as those learnt via explicit concept supervision in CBM and CW, individual concepts do not correspond well to the ground truth ones. This indicates that the information about each ground truth concept is distributed across the overall representation rather than localized to individual concepts. As result, surprisingly, in ab- sence of extra capacity, CCD and SENN exhibit an OIS in a similar or better range than methods benefiting from explicit supervision such as CW. We attribute CCD's lower impurity, compared with SENN, to the use of a regularization term that encourages coherence between concept representa- tions in similar samples and misalignment between concept representations in dissimilar samples. More interestingly, however, SENN encodes higher impurities in its concept representations than all DGL approaches despite benefiting from explicit supervision. Within DGL approaches, not only does the lack of supervision result in better predictive performance for the overall set of learnt con- cepts as discussed before, but it astonishingly also results in lower impurity representations than those of wuakly-superrvsed DGL methods.
Niching-based metrics demonstrate that supervision may not result in task-label-eeparable concept representations compared to no supervision (Figure 4. Niching-based metrics do not differentiate between explicit and implicit supervision as much as oracle impurity, as they merely focus on task labels that are available in both approaches. Thus, simultaneous low niche impurity and high niche purity is observed in CBM and CCD, despite their differing degree of supervision. There are, however, surprising exceptions this: in contrast to CBM and CCD, high impurity and
8

---

Under review as a conference paper at ICLR 2022
Tabula Toy(i)
(5peltest
36shapes
Gold Lime
WV CCD (oumedd
Figure 4: Evaluation of different models, deployed across various tasks, using our proposed metrics.
high purity are observed in CW's full feature maps (i.e., when no scalar reduction is applied), as well as in SENN's representations. This is particularly true when concept dependencies increase, which shows the vulnerability of this approach when concepts have inter-dependencies. Performing MaxPool-Mean reduction CW's maps reduces the amount of impurities, however, it also reduces the amount of purity (i.e., the learnt concepts unable to predict their aligned ones with high confidence). Similar to oracle impurity, this further evidence that some crucial information may be lost in CW's scalar reduction, confirming the lack of supervision results in better predictive performance for the overall set of learnt concepts, but it surprisingly also results in lower impurity and higher purity that those of weakly-supervised DGL, phenomenon worth investigating in the future.
OIS result. Within DGL approaches, not only does
Oracle impurity and niching- -asee metries are robust concept inter-dependencies. The highly preserved ranking of the methods using our metrics in both settings with and without depen- dencies, indicates the robustness these metrics concept inter-dependencies as intended.
DISCUSSION AND CONCLUSION
The lack of metrics evaluate and contrast the quality of concept representations within and be- tween CL and DGL methods hinders their systematic comparison. In this paper, we address this limitation by introducing set of metries that facilitate comparison from concept representation quality perspective. Our systematic evaluation of several methods using our metrics suggests that: (i) Model design is just as important as degree of supervision: models with the same degree of su- pervision (e.g.. CBM vs. CW and CCD vs. SENN) can encode varying amounts of impurity in their concept representations, in particular when there are strong cross-concepp dependencies; (ii) set- tings where ensuring the quality of the overall set of learnt concepts is sufficient, implicit supervision from task labels can be just as effective as explicit supervision from concept labels. Thus, given cost, providing explicit supervision is not recommended; (iii) In settings where ensuring the quality of individual concept representation as well as their overall set is required, explicit supervision as used in CBM is the most recommended option, followed by CCD if the degree of cross-concept dependencies in the data is unknown; and (iv) In either of the above settings the weak supervision provided by DGL methods In fact, overall DGL methods without any supervisions tend to outperform the weakly supervised ones. Related to metrics, in setting (ii), regardless of access concept labels, niching-based metrics should be favoured to oracle impurity. Both metrics are able to capture that the overall set of learnt concept benefits from supervision, with the former metric being very efficient and tractable when the number of concepts is high (as it does not require classifier to be trained for every pair of concepts). On the contrary, in setting (iii), the high computational cost of oracle impurity justified if concept labels available. This is because, unlike niching-baa based metrics, oracle impurity distinguishes the impact of explicit vs. implicit supervision on the quality of individual learnt concepts.
recommended as a substitute for implicit or explicit supervision.
One direction that was not explored in this paper is the evaluation of the quality of concept represen- tations w.r.t. the input space. In fact, Margeloiu et al. (202T) showed that concepts from CBM often do not correspond to semantically meaningful input space. We therefore believe that future work on defining metrics that capture concept-input quality along with the metrics we proposed here can provide comprehensive assurance about the quality of learnt concept representations.

---

Under review as a conference paper at ICLR 2022
REFERENCES
David Alvarez-Melis and Tommi S Jaakkola. Towards robust interpretability with self-explaining neural networks. Neural Information Processing Systems (NeurIPS), 2018.
Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Pietro Lió, Marco Gori, and Stefano Melacci. Entropy-based logic explanations of neural networks. CoRR, abs/2106.06804, 2021.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. JEEE transactions on pattern analysis and machine intelligence, 35(8):1798-1828, 2013.
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yunhan Jia, Joydeep Ghosh, Ruchir Puri, José MF Moura, and Peter Bekersley. Explainable machine learning in de- ployment. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 648-657,20 2020.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http:///shhhhccommmeeemmm/////is 2018.
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentangle- ment in variational autoencoders. arXiv preprint arXiv: /002.v/80.04445 04942, 2018.
Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Intelligence, 2(12):772-782, 2020.
Gabriele Ciravegna, Pietro Barbiero, Francesco Giannini, Marco Gori, Pietro Lió, Marco Maggini, and Stefano Melacci. Logic explained networks. CoRR, abs92000.44,,,44 2021.
Elliot Creager, David Madras, Jörn-Henrik Jacobsen, Marissa A. Weis, Kevin Swersky, Toniann Pitassi, and Richard S. Zemel. Flexibly fair representation learning by disentanglement. In Kama- lika Chaudhuri and Ruslan Salakhutdinov (eds.), International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp. 1436-1445. PMLR, 2019.
Charles Darwin. On the Origin of Species by Means of Natural Selection, or the Preservation of Favoured Races in the Struggle for Life. 1859.
Botty Dimanov and Mateja Jamnik. Step-wise sensitivity analysis: Identifying partially distributed representations for interpretable deep learning. 2018.
Cian Eastwood and Christopher K. Williams. barba framework for the quantitative evaluation of disentangled representations. In International Conference Learning Representations (ICLR). OpenReview.net, 2018.
Amirata Ghorbani. James Wexler, James Y. Zou, and Been Kim. Towards automatic concept-based explanations. In Neural Information Processing Systems (NeurIPS), pp. 9273-9282, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. International Conference on Learning Representations (ICLR). OpenReview.net, 2017.
Dmitry Kazhdan, Botty Dimanov, Mateja Jamnik, Pietro Liò, and Adrian Weller. Now you (CME); concept-based model extraction. In Conference on Information and Knowledge Manage- ment (CIKM) Workshops, volume 2699 of CEUR Workshop Proceedings. CEUR-WS.org 2020.
Dmitry Kazhdan, Botty Dimanov, Helena Andrés-Terré, Mateja Jamnik, Pietro Liò, and Adrian Weller. Is disentanglement all you need? comparing concept-based & disentanglement ap- proaches. CORR, abs/2104.0691 2021.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pp. 2673-2683 PMLR, 2018
10

---

Under review as a conference paper at ICLR 2022
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In Jennifer G. Dy and Andreas Krause (eds.), International Conference on Machine Learning. (ICML), volume 80 of Proceedings of Machine Learning Research, pp. 0654-2663.. PMLR, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 0rXiv:1412.6980. 2014.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), International Conference on Learning Representations (ICLR), 2014.
Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pp. 5338-53388 5348. PMLR, 2020.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Rätsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised leamn ing of disentangled representations. In International Conference on Machine Learning (ICML), volume 97 of Proceedings of Machine Learning Research, pp. 41 4114-4124 PMLR, 2019.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Confer- ence on Machine Learning (ICML), volume 119 Proceedings of Machine Learning Research, pp. 6348-6359, PMLR, 2020a.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf, and Olivier Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representations (ICLR). OpenReview.net, 2020b.
Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Annual Conference Neural Information Processing Systems (NeurlPS), pp. 4764 4765-4774, 2017.
Lucie Charlotte Magister, Dmitry Kazhdan, Vikash Singh, and Pictro Liò. Gcexplainer: Human-in- the-loop concept-based explanations for graph neural networks. CoRR, abs/2107.1188 2021.
Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez. and Weiwei Pan. Promises and pitfalls of black-box concept learning models. CoRR, abs/2106.13314, 2021.
Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian
Weller. Do concept bottleneck models learn as intended? CoRR, fbb//210500428) 2021.
Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentanglement testing sprites dataset, 2017.
Hema Raghavan, Omid Madani, and Rosie Jones. Active learning with feedback on features and instances. Journal of Machine Learning Research, 7:1655-1686, 2006.
Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. "why should trust you?": Explaining the predictions of any classifier. In International Conference on data science and advanced analytics (DSAA), pp. 1 1135-1144. ACM, 2016.
Karl Ridgeway and Michael C. Mozer. Learning deep disentangled embeddings with the f-statistic loss. In Advances in Neural Information Processing Systems (NeurIPS), pp. 185-194, 2018.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011a.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011b.
Chih-Kuan Yeh, Been Kim, Sercan Ömer Arik, Chun-Liang Li, Tomas Pfister, and Pradeep Raviku- mar. On completeness-aware concept-based explanations in deep neural networks. In Neural Information Processing Systems (NeurIPS), 2020.

---

Under review as a conference paper at ICLR 2022
6 ADDITIONAL RESOURCES
6.1 METRICS RELATED TO PROPERTIES OF CONCEPT REPRESENTATIONS W.R.T. GROUND TRUTH CONCEPTS
Refer to Table for a summary of some metrics used for measuring the quality of disentangled representations which are applicable disentangled concept representations.
Table I: Metrics related to properties of concepts w.r.t ground truth concepts divided to two categories: those that capture properties of individual concepts and those that capture properties of the overall of learnt concepts w.r.t ground truth concepts. Although metries each column serve the same purpose, they are mathematically distinct.
Individual learnt concepts w.r.f. ground truth concepts
Overall set of learnt concepts w.r.. ground truth concepts
Modularity (Ridgeway & Mozer 2018): Whether each learnt concept corresponds to at most one ground truth one. Measured by deviation from ideal case where each learnt concept has high tual information with only one ground truth one and zero with others.
Explicitness (Ridgeway Whether the overall set of learnt concepts can predict each individual ground truth ones using simple (e.g.. linear) classifier. Measured by the average predictive performance concept vector.
Mozer 2018):
Mutual information gap (Chen al. 2018):
Informativeness (Eastwood & Williams 2018
Whether learnt concepts are disentangled. Mea- sured by averaging the gap in mutual informa- tion between the two learnt concepts that have the highest mutual information with ground truth concept. This metric generalises the disentangle- ment scores in Higgins et al. (2017) and Kim Mnih 20088
Whether overall concept vector can predict each ground truth concept with low prediction er- ror. Measured by average prediction error of con- cept vector.
Disentanglement (Eastwood Williams 2018): Whether each learnt concept captures at most one ground truth one. Measured by the weighted aver- age of disentanglement degree of each learnt con- cept. Such degree is calculated based entropy. where high entropy for learnt concept shows its equal importance for all ground truth ones and therefore its low disentanglement degree. The weight is calculated based the aggregation of relative importance of learnt concept predict- ing each ground truth one.
Alignment (Yeh al. [0020:: Whether the learnt concepts match the ground truth ones. Mcasured by average accuracy of predicting each ground truth concept using the learnt one that predicts it best.
6.2 MUTUAL INFORMATION IN CUB DATASET CONCEPTS
We take the Caltech-UCSS Birds-200-2011 (CUB) dataset (Wah et al. 201 formed by images of 200 types of birds, and use each sample's 's 112 binary attributes as concept annotations (Koh et al.) 2020). We find the top-10 concepts with highest mutual information with the label and compute their Pearson correlation coefficients. Our results, shown in Figure 5 highlight that many concepts have non-zero correlations. This shows the lack of inter-concept independence in real-world datasets.
6.3 PURITY MATRIX IMPLEMENTATION DETAILS
We compute the (i. j)-th entry of the purity matrix follows: split the original testing data (Xiest- Yeary Crea) into two disjoint sets, a new training set (X Y Crain) and a new testing set tosts )s using traditional 80%-20% split. We then the concept representations learnt
(X) test: Y
12

---

Under review as a conference paper at ICLR 2022
10
Figure 5: Absolute Pearson correlation coefficients of the top- concepts with highest label mutual information in CUB dataset. Notice that there exist strong correlations between some of the concepts.
for the i-th concept for samples in Xlest train a ReLU MLP (+) with single hidden layer with 32 activations to predict to truth value of the j-th ground-truth concept. In other words, we train ()) using labelled samples (9(6(x())) (sa) c/ x(1) E X A c()) Cimin-- Finally, we set (i.j)) f)th entry the purity matrix as the AUC achieved when evaluating v/y(.) on the new testing set ((( Clea)
6.4 CONCEPT ALIGNMENT IN UNSUPERVISED CONCEPT REPRESENTATIONS
In the presence of ground truth concept annotations, one can still compute a purity matrix, and therefore the OIS, even when the concept representations being evaluated were leart without di- rect concept supervision. We achieve this by finding an injective alignment A {1,2,...,,A 21,2.... ... k} between ground truth concepts E IR* and learnt concept representations è E BRdk In this setting, we A(i) = j represent the fact that the i-th ground truth concept e, is best rep- resented by the j-th learnt concept representation C(:j)- In our work, we greedily compute this alignment starting from a set of unmatched ground truth concepts I = {1, k} and a set
of unmatched learnt concept representations Z f1, k'} and iteratively constructing A by adding one match (i,j) at time. Specifically, at sime-step ++1 we match ground truth concept with learnt concept if one can predict concept from c(.j) better than every other concept representation Cggg can predict every other ground truth concept cy.. We evaluate predictability of ground truth concept from learnt concept representation c(:5) by training ReLU MLP with single hidden layer with 32 activations and evaluating its AUC on a test set. Once a match between ground truth concept i and learnt concept representation, j has been established, we set T(ttt) := I Vii and Ieamt I Mí). We repeat this process until have found match for every ground truth concept (i.e., until (t) becomes the empty set). Notice that in practice, one needs to compute the predictability of ground truth concept from concept representation, only once when building alignment A.
6.5
IMPACT OF CCORRN'S S ON NICHE PURITY, IMPURITY, AND SIZE
not a parameter to tune, it is rather knob to be used to assess the stability and robustness of the concept representations. To this aim, we compute niching scores for different values of / and generate a plot as reported in Figure ( The figure shows the impact of threshold 3 used in concept-correlation nicher (CCorıN) niche purity, impurity, and size in dSprites(A = O). Across all approaches, increasing 3 decreases niche purity and niche size, while increasing niche impurity. When 3 is high, only very few concepts can pass the threshold required to be identified as concept nicher for a label. As result, niche sizes get smaller, which translates to bigger niche complement and, subsequently, more impurity.
13

---

Under review as a conference paper at ICLR 2022
Figure 6 allows qualitative comparison of concept representations. CBM provides the most robust representation (niching scores are less sensitive to 9), while VAEs provide the least stable set of concepts. CCD representations are almost stable as CBM ones, while CW is bit closer to VAEs.
1.0
10
purty
putty Nedw
0.2
by
80
B0
02
0.5
0.0
0.1
02
0.4
(a) Joint-CBM
(b) CW MaxPool-Mcan
1.0
1.0
09
0.5
0.8
0.8
0.7
07
0.
8 0.
83
05
Nedw party
0.
Net party
Aug. nithe insurty
Niche inpurty
Avg. ridhe size
Mich size
0.6
0.5
0.1
(c) Ada-MLVAE (n latent k)
(d) Ada-GVAE latent = k)
3
09
Note purty
02
01
(e) CCD (n_concepts k)
Figure 6: The impact of CCornN' 3 on niche purity, impurity, and size in dSprites(A = 0).
To compare different models across different data sets, we select a fixed value for 3 for all models (3 = 0.2) in a range where the dependency between |3 and concept scores was approximately linear for most models.
6.6 CONCEPT NICHES IN REAL- WORLD DATA
The niching-based metrics reveal information about the task separability and minimality of concepts. To motivate why task separability and minimality are important for evaluating concept quality, show that these properties are often empirically observed in real world data.
The intuition behind concept niching is relatively simple: for the sake of example, say we have multilabel task with labels "bird" and "car". Using niching, we can isolate neurons encoding only the information about car (e.g., wheel), only the information about a bird (e.g., a feather), and only the information about both of them (e.g., the colour blue). The ideal concept representation has small niche for each task (one for "car"="wheel"" "blue" and one for "bird"=" "blue"), simplifying concept-based explanations and downstream procedures.
Figure shows the absolute values of concepts-to- tasks linear correlation coefficients in the Caltech- UCSD Birds-200-2011 dataset (CUB, (Wah et al. (011b)), The coefficients are computed on ground-truth concept and task labels. The sparsity of the matrix empirically proves that tasks indeed rely on an often small and non-overlapping set of concepts. Thus, the concept niches for each task do not tend to intersect. If this is preserved by the learnt concept representations, then the Niche Impurity Score (NIS), capturing the amount of undesirable mutual information amongst concepts, should be low, while Niche Purity Score (NPS), capturing the amount of inevitable mutual information amongst concepts, should be high. On the other hand,
representative of real-world datasets.
14

---

Under review as a conference paper at ICLR 2022
simultaneously high NIS and NPS suggests that there may be unnecessary mutual information be- tween concepts that counts as leakage.
C-to-Y correlation (CUB)
0.30
102969084 78 088660644-423602222
0.25
ground-rruth concepis
0.20
0.15
0.10
0.05
10
30
80
100
110
120
120
140
150
160
170
180
190
ground-bruth tasks
Figure Absolute values of coneepts-to-tasks lincar correlation coefficients in CUB.
6.7 EFFECT OF NETWORK CAPACITY ON ORACLE IMPURITY
We motivate the need of fixing the neural network architecture when contrasting different methods by looking at how the capacity of the selected architecture can affect a method's learnt concept representations. For this, we train a Joint-CBM in the TabularToy(E = 0) dataset (with a = 0.1) whose encoder and decoder models are simple ReLU MLPs with two hidden layers. this exper- iment, we vary the number of activations in either the encoder and decoder models by setting the number of units in their hidden layers to {capacity, capacity / units, while keeping the number of hidden units in their corresponding counterpart fixed to {128, 64). In this setting, we monitor the predictive accuracy of concept representations w.r.t. their aligned ground truth concepts as well the representations' oracle impurities. As shown in Figure 8 we observe that the encoder and decoder capacities decrease, the CBM begins to exhibit significantly higher impurity and lower concept accuracy. Notice that results also show that the encoder's 's capacity has a significantly greater effect on the quality of the learnt representations over the decoder's capacity. This motivates the need for a constant architecture with enough capacity to achieve high concept predictability and serves as a basis for our decision to fix the encoder and decoder architectures across all methods in our evaluation.
CBM Concept Accuracy & Impurity (Tabular (oy 0))
0.8
Concept Accuracy (Decoder Capacity)
Score
0.5
Concept Ace mayy (Eaceder Capacity)
Oradl Impurity (Encoder Capacity)
0.4
Orade Impurity (Dezoder Capacity)
0.2
0.0
96
123
100
132
224
256
Capacity (Activations)
Figure 8: Effect of network capacity (i.e., number of hidden activations used in the encoder and decoder) in CBM's concept accuracy and oracle impurity.
15

---

Under review as a conference paper at ICLR 2022
6.8 EXPERIMENT DETAILS
In this section we provide further details on the datasets used for our experiments as well as the architectures and training regimes used to obtain our presented results.
6.8.1 DATASETS
For benchmarking, we use three datasets: simple synthetic toy tabular dataset extending that de- fined in (Mahinpei et al. 2021). dSprites (Matthey et al., 2017), and 3dshapes (Burgess Kim 2018). In order to investigate the impact of dependency between concepts on the quality of concept representation in each method, we allow a varying degree of dependencies in concept annotations and propose the following parameterized tasks:
TabularToy(O); this dataset consists of inputs {x(1) RRT such that the j-th coordinate
of x6)) is generated by applying a non-invertible nonlinear function (=1 22 23 to 3
latent factors {z(") 22 23 ). These latent factors are sampled from multivariate normal
(6)
distribution with zero mean and a covariance matrix with d in its off-diagonal elements.
The concept annotations for each sample correspond to the binary vector c := [1
0). 1(2 ( > 0), 1(2))) > 0)] and the task we use is to determine whether at least two of the
latent variables are positive. In other words, we set (+) to 11 ((c))) + c22 + C3 (") 2 2). The individual functions used to generate each coordinate of x()) are the same as those defined
in (Mahinpei et al. (02TT. As in (Mahinpei al. (02T), ) we use total of 2, 000 generated samples during training and total of 1, 000 generated samples during testing.
dSprites(A): (A): we define task based on the dSprites dataset where each sample x(() E {0, 1],466xxx a grayscale image containing square, an ellipse, or heart with varying positions, scales, and rotations. Each sample x(9) is procedurally generated from a vector of ground truth factors of variation z [shape E {0, 2) scale E {o.... .5,0 0 e {0...- ,0...,,99,,2 x E {0, ,31,, y E {0, .-. 31H] (0 indicating an angle of rotation) and is assigned binary concept annotation vector (+)) {0,, 13. with elements c(0) := [1
2), (2 < 3).
20). (2 1 < 16)] For this task, construct
set of 8 labels from the concept annotations by setting (()) [c2 04 10 if if = 1 (where
(i) (6)
we use [b1b2]10 to indicate the base-10 representation of binary number with digits b
and b2) and ( = 4+ [cg Co 10 otherwise. Finally, we parameterize this dataset on the dependency number A E (0, 1,... 4) that indicates the number of random dependencies factors of variation (with À = 0 implying all
we introduce across the factors of variation are independent). For example, if X = 1, we introduce a conditional dependency between factor of variations 21 ("shape") and 22 ("scale") by assigning each value of &1 a random subset of values that 22 may take given 21. This subset is sampled by selecting, random for each possible value of 21, half of all the values that 22 can take. More specifically, if 21 and z2 can take a total of Ti and T2 different values, respectively. then for each {0, 1,... , Ti} constraint 22 be able take only [22/2] values from the set 22 E (a) defined as:
(a [SWR( (0. 1, [WWR [1 ..... .To 13,
37 J}, (2)
CI =1 otherwise
where SWR(A, n) stands for Sample Without Replacement and is a function that takes in a set A and number and returns a set of n elements sampled without replacement from A. This process is recursively extended for higher values of by letting the dataset generated with X = i be the same as the dataset generated by X = 1 with the addition of new conditional dependency between factor of variations z; and 2(1+1)- Finally, in order to maintain constant dataset cardinality as X varies, we subsample all allowed factor of variations in {*x+2, 22-3. ... .33] } by selecting only every other allowed value for each of them. This guarantees that once conditional dependency is added, the cardinality of the resulting dataset is the same as the previous one. Because of this, all parametric variants of this dataset consist of around 32, 000 samples.
16

---

Under review as a conference paper at ICLR 2022
3dshapes(A): we define task based the 3dshapes dataset where each sample x(i) E {0,1, 255]64x64x3 is color image containing sphere, cube, cap- sule, or a cylinder with varying component hues, orientation, and scale. Each sample x(1) is procedurally generated from a vector of ground truth factors of variation z(() = [floor_uue E {0, 1.... .9), wall_hue E {0, 1, ... ,9), object_hue E 10, 9), scale e {0, I,..., -7,, shape E {0, 1, 2, 3), orientation E {0, 1, 14]] and is assigned binary concept annotation vector c( E {0, 1) with elements c(() := [L(z)" < 1), < 5), (2)") < 5).
(2g) < (26) < 7)]. For this task, we construct a set of 12 labels from the concept annotations by setting y = [4 C2 03 10 if og =1
and y = 8 + [<] C to otherwise. As in the dSprites task defined above, we further parameterise this dataset with parameter À E {0. 5] to control the number of ran- dom conditional dependencies we introduce at construction time. The procedure used to introduce such dependencies is the same as in dSpriles()) but we use cs rather than ej for determining the set of values that we sample from. Similarly, we use the same subsampling as in d.Sprite( (A) to maintain a constant-sized dataset, resulting in all parametric variants of this dataset having around 16, 000 samples.
6.8.2 MODEL. ARCHITECTURES AND TRAINING
In all of the results we report on the TabularToy(5) dataset, for CBM and CCD we use a 4-layer ReLU MLP with activations [7, 128, 64, 3] as the concept encoder g(x) and a 4-layer ReLU MLP with activations [64, 128, 64, 1] as label predictor f()). For CW, we use the same architecture with the exception that CW module is applied to the output of the concept encoder model. For both CBM and CCD, we train each model for 300 epochs with a batch size of 32. For CW, we train each model for 300 epochs, with CW updates occurring every 20 batches, using a batch size of 128. Finally, for the mixture hyperparameter of the joint loss in CBMs, unless specified otherwise we use a value of a = 0.1.
For our CBM and CCD experiments in (Sprites(A) and 3dshapes(A), we use Convolutional Neu- ral Network (CNN) with four (Conv + BatchNorm + ReLU + MaxPool) blocks with feature maps {8, 16, 32, 64) followed by three fully-connected layers with activations {64, 64, k) for the con- cept encoder model g(x) (with & being the number of ground truth concepts in the dataset). Fur- thermore, for the label predictor model f(e) use simple 4-layer ReLU MLP with activations {k, 64, 64, L} (with L being the number of output labels in the task). For CW's concept encoder g(x) we use CNN with three (Conv BatchNorm + ReLU MaxPool) blocks with feature maps (8, 16, 32) followed by a (Conv + CW) block with 64 feature maps. For the label predictor, we use a model composed of MaxPool layer followed by five ReLU fully connected layers with activations (64, 64, 64, 64, L}. All models evaluated for CBM and CCD are trained for 100 epochs using a batch size of 32. In contrast, CW models are trained for 100 epochs using a batch size of 256 and an CW module update step every 20 batches. Finally a value of = 10 is used during joint CBM training.
For evaluating VAE, 3-VAE (6- = 10), Ada-GVAE and ALaAE, we use the same architec- ture as in CBM's and CCD's concept predictor for the encoder and the same architecture as in (Locatello et al. 2020b) for the decoder. The decoder consists of two ReL.U fully connected lay- ers with activations (256, 512) followed by four ReLU deconvolutional layers with feature maps (64,32, 32, mppu_feature (ecature_map)).. All DGL models are trained for 100 epochs using batch size of 32. Weakly supervised models are trained with a dataset consisting of 2N at least one factor of variation (with N being the number of samples in the original dataset) while unsupervised models trained with the same dataset used for CL methods. As in other methods, we train all DGL models using a default Adam optimizer (Kingma & Ba 2014), with learning rate 10-
33 pairs of images that share
When evaluating CCD, we use a threshold of 3 = 0.0 for computing concepts scores and the same regulariser parameters X 0.1, X 0.1, e = 10-5 as in Ych et al. (2020) released code for their work in Yeh et al. 2020). Finally, all CCD models, across all tasks, are trained for 100 epochs and batch size of 32 using default Adam optimizer, with learning rate 10-3
17

---

Under review as a conference paper at ICLR 2022
(oy(
dSprites (A)
3dshapes
303
:30
200
00
ALC
60
0.0
6
Jiet-CBM
(n.concepta
SEWN (n.comsspts
CW MasPook-Mian
CCD (t.concegts
2k)
SENN (n.concepps 24)
Festure
Figure 9: Downstream task predictive AUC for all datasets using original pre-trained models prior to bottleneck construction. Note that because DGL methods have downstream task supervision in their training pipelines, we do not include those methods.
Toy(1)
epprtee
3dshapes A)
13
AUI (N2
(n.atert
Fastre
litee k)
Bet Liteer 34)
EC0 eoopppe k)
(a reweptt
Figure 10: Downstream task predictive AUC for all datasets using the overall set of learnt concepts predict the task labels. Note that this plot faithfully replicates the downstream task predictive AUC of methods that received direct task supervision in their training (shown Figure
When benchmarking SENN, coder g(x) and for its corresponding decoder. Note that the decoder in this case is only used as part of the regularization term during training. For the weight model ((ē) (i.e., the "parameterizer"), use simple ReLU MLP with unit sizes (input shape, 64, 64, k}} (k being the number of concepts SENN will learn). Finally, we use additive aggregation function and use À 0.1 robustness regularization strength and ( = 2 x 10-5 as the sparsity regularization strength, as done in Alvarez- Melis & Jaakkola (2018). We train our SENN models for 100 epochs using a batch size of 32 and default Adam optimizer with learning rate 10-3
the same architecture in DGL methods for the concept
6.9 DOWNSTREAM TASK PREDICTIVE AUC
Figure shoows downstream task predictive AUC for all datasets using raw inputs, in absence of any dependencies as well as the maximum dependencies between concepts. If concept representations from various methods are good surrogates to the inputs, they need to recover the same predictive performance. Figure 10 confirms that this holds a simple ReLU MLP with hidden layers (64, 64) trained to predict the corresponding task labels using the concept representations generated by each method.
good degree by looking the task AUC of
6.10 IMPURITY SCORES (ORACLE AND NON-ORACLE)
Figure 11 confirms that, for 3dShapes(A), across all models the non-oracle impurity misleadingly shows more impurity that often increases with the dependence of concepts.
18

---

Under review as a conference paper at ICLR 2022
CBM Impurity (3dshapes(
Impurity (3dshapas(-))
CW Feature Map Impurity (3dshapes
CBM CBM (ande insuro!
Inpulit Scote
0.20
Importty Soore
Paras the iepais)
CW Fastue Mep moak opaiol
CM Mad (oadl impalin)
CW MadPositean (seaadd impasty)
64
(a) CBM
(b) CW MaxPool-Mcan
(c) CW Feature Map
Ada-MLLVEE Impurity (3dshapex(3) ())
Ads GVAE Impurity (3dshapes(A) (
CCD Impurity (3dshopes(3))
Acx-EEWK Biter
Sucedry)
Au (iter k) (oadl belandd)
0.39
My- (taveorta
1131
Soor
642
038
0.30
0.30
(cwagra- fendl impurts)
090
UCB (ceeggn free-ande mquum
(d) Ada-MLVAE
(e)
(1) CCD
Figure 11: Impurity scores (both oracle and non-oracle) in 3dshapes( (X)-
19


Reviews:
[{"review_id": "qWBUkTMlW4m", "reviewer": "Reviewer_7LHy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper introduced metrics to evaluate the concept quality generated by both methods i.e. concept-based representation learning and disentanglement learning in the scinerio of concept supervision and correlation of concepts. For this purpose, mainly three family of methods were considered i.e. concept-based representation learning both with and without concept supervision and semi-supervised disentanglement learning method. Based on this metrics, some recommendations were made regarding requirements of explicit concept supervision for concept-based representation learning methods and weak supervision for disentanglement learning methods.\n\nThis paper has below weaknesses/clarifications/suggestions:\n1)It's not very clear how the concept property (ii) is related to the contributions made based on concept correlations and availability of ground truth cocnepts. The property is definitely a desirable property for the concepts, but how the proposed metrics capture whether a set of generated concepts follow this property, specially when the ground truth concepts are not available.\n\n2)How each entry of the purity matrix is calculated? Do all the data points are required to calculate each matrix entry? Please elaborate the line \"In practice, we parameterise the family of functions ψ j by training, via gradient descent, a ReLU MLP with 32 hidden activations\" in this context.\n\n3)A very important work (https://arxiv.org/abs/1806.07538) is missing from the paper, which can be a very good representative of unsupervided concept learning methods. This work has used both label supervision and decoder based network for better concept learning. Extending in the same line, What is your opinion on (possibility of) merging these two types of methods (CL and DGL) for better concept learning? If you think it's possible, then how would the proposed metrics would be helpful here?\n\n4)Figure 8 shows the effect of network capacity. On top of that, it would be interesting to see the individual effects of capacities of encoder and the decoder. To be precise, improving the capacities of both the encoder and the decoder would result low impurity and higher cocnept accuracy, but it's important to check if capacity of the encoder has more effect on concept quality than capacity of the decoder.", "summary_of_the_paper": "The authors have put decent effort to bring concept-based representation learning and disentanglement learning together under one umbrella in terms of the quality of generated concepts in presence as well as absence of ground truth concept labels. Some related metrics were proposed for evaluation of the quality of concepts for both these methods. Based on these studies, presented in the paper, some important conclusions were made based on requirements of concept supervision and their effects on final concept quality as well as the predictive performance of the model.", "summary_of_the_review": "This is an interesting work that tried to unify concept-based representation learning and disentanglement learning and proposed corresponding metrics that helped to find some important conclusions. Answering/commenting on the points, that I posted under the main review, can significantly improve the quality of this work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fPJ2mn9yJh", "reviewer": "Reviewer_iQK1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrengths\n\n- metrics which work for correlated underlying factors are really valuable and I think a big blind spot in this space, glad to see this is a focus of this paper\n- proposed metrics have some nice underlying intuitions and seem sensible to me\n- experiments seem quite extensive, comparing many prior methods from the literature in both CL and DGL\n\nWeaknesses\n\n- paper could greatly benefit from improved clarity throughout: there are a number of points where I struggle to understand exactly what is being communicated\n- In particular, clearer descriptions of methods and how those differences manifest in the results section would be helpful - I don't have a deep understanding of all the related work discussed and I have a lot of trouble knowing what the takeaways from the results section should be. Some figures which accent the particular points in question in each section could help, or more analysis of the overall concepts in question for each experiment\n- it's not entirely clear conceptually how we're supposed to think about CL and DGL in this paper. Is the paper intended to unify the methods? Contrast them? I don't fully understand what the authors see as the main differences or similarities, or what they want to communicate to the reader on this front. It would be good to spend more time developing a framework and some common language to describe the two approaches\n- I give many more specific notes below\n\nOther\n\nSec 1:\n- the acronym DGL for disentanglement learning is strange - I'd personally prefer something like DEL but up to you\nSec 2:\n- a working example at the top of Section 2 explaining what each concept might represent would be very helpful, especially to someone like me who is familiar with the overall literature but not on the bleeding-edge methods\n-  not sure why we need both g and \\phi in this framework?\n- can you clarify the role of the downstream task here? it's never really made clear - are there many tasks? just one? do we expect to know it at training time?\n- would be useful to define TCAV\n- you state that if a function g contains sufficient statistics to perform well on a downstream task f, then g must be invertible; I don't see why this is true, since g may reduce the input \\phi to just the sufficient statistics, and much of \\phi may not be recoverable\n- you list the operating assumption of DGL, what is the analogous assumption for CL?\n- you focus on weakly supervised disentanglement, why not weakly supervised CL as well?\nSec 3:\n- some sloppiness in language at the top - I'm not sure what the \"quality of their learnt concepts\" is meant to refer to: only CL (since you used the word \"concept\")? or both, since you refer to all the \"approaches above\". This is where I think it would be useful to clarify a particular framework for how you relate these two literatures\n- may be beyond the scope of the work, but I agree that the no-leakage assumption is unrealistic - however there are some cases where we want it (the two concepts are truly related) and some cases where we don't (if it's a relic of sampling bias). Just an interesting thought, not sure if there are ramifications in your paper\n- it would be good to discuss the CUB example in the main body since it is an important point for the rest of the paper\nSec 3.1\n- Def. 1: what does it mean for two representations to be \"aligned element-wise\"?\n- Def. 1: the argmax notation loses me - is the idea that the concept should be categorical? I don't think this was specified earlier\n- I don't understand the usage of AUC here - are you assuming binary concepts?\n- Implementation concern which should probably be discussed: your estimate of \\pi depends on calculating a \\sup over functions \\psi - this will probably be only approximate, and if this is a related function class (or less powerful) than the encoder in question g, then you might systematically bias your estimate of \\pi and OIS\nSec 3.2\n- I'm not totally sure the analogy with evolutionary biology is necessary here\n- there's an assumption implicit in niching which is that all tasks you care about will necessarily rely on a specific subset of individually predictive dimensions - worth motivating this assumption, it's a little bit different from the standard DGL assumptions\n- Def. 6: technically N_j is not an input to NPS, just clean up formalization here to clarify what you mean by \"the NPS of \\not N_j (v_f)\"\nSec 4.2\n- \"in order to have datasets compatible with both CL + DGL ... construct datasets full described by ground truth gen. factors\" - I'm not sure what this means, can't you use any dataset with either of these methods?\nSec 4.3\n- clarify exactly what the metric \"Mean Concept AUC\" refers to - I'm not quite sure how it's calculated\n- why is the non-oracle impurity baseline matrix using 1/2 on its off-diagonals - wouldn't the baseline be 0 on the off-diagonals to mimic the situation where we assume no leakage?\n- you say that the non-oracle impurity \"misleadingly\" shows more impurity as dependency increases. However, it's not clear this is misleading, it's possible that the methods are truly doing worse on that data (even relative to oracle). It would be nice to have something external to tell us which is the case\n- suggestion: indicate in figures which methods are supervised, this will help interpreting them a lot\n- the last sentence of the \"oracle impurity demonstrates\" paragraph seems to contradict the first: the first says that explicit supervision \"encodes purer individual concepts\", the last says explicit supervision \"does not translate to purer concepts\"  - I'm confused which should be true\n- the reference to \"concept loudness\" should be explained to those not familiar with the literature\n- the point at the bottom about robustness to concept inter-dependencies seems really important - I would love to see this fleshed out more\nSec 5.\n- (iii) you discuss the \"weak supervision provided by DGL methods\" but aren't there supervised, weakly, and un-supervised DGL methods?\n- why should we prefer niching-based metrics?\n- this point about efficiency goes over my head, would be good to have a quick discussion in the paper\n\n", "summary_of_the_paper": "The authors propose new metrics for methods in disentanglement learning and concept learning,  which have some nice properties, including robustness to correlations in the underlying factors, and experimentally probe the properties of these metrics on a number of different proposed methods in the disentanglemnt/concept learning literatures.", "summary_of_the_review": "The authors propose metrics which address a very important problem in the relevant literatures (in particular, underlying factors which may be correlated), but the communication of the ideas and experimental results are not currently clear enough for me to accept the paper as is.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8_NK8OUhp21", "reviewer": "Reviewer_yfkD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Concept-based explanations are a relatively new idea; the first papers on this topic, including TCAV and its extensions, were post-hoc methods that did not require specialized model architectures. More recently, the concept bottleneck idea suggested training a model that explicitly encodes concepts at an intermediate layer (enabling interventions, etc). In parallel, as the authors say, there has been a rich literature on unsupervised/semi-supervised disentangled representation learning, where models are trained that may or may not capture semantically meaningful, decorrelated concepts in their latent spaces. It is natural to wonder to what extent these various methods can result in representations that are useful from a concept-based model explanation perspective. \n\nBefore getting into the authors' contributions, it's probably worth saying that the intuitive answer here (for me, at least) is that concept supervision is key, and without it we can't expect a model to learn a latent space that is well-aligned with concepts that we don't tell it about. Ultimately, this appears to be the takeaway from the experiments (see figures 2, 4, 5), but it's not a particularly exciting or surprising result; it's showing what people would expect to begin with, but with a little bit of rigor.\n\nBelow, I'll summarize and comment on the authors' two proposed metrics.\n\n### Oracle impurity score\n\nThe proposed OIS score basically assesses to what extent the learned concepts predict the ground truth concepts equivalently to how the ground truth concepts would. Some previous work suggests that concepts should not be predictive of one another because this would constitute \"leakage,\" which is problematic for experiments with concept interventions. To remedy that, because concepts in fact are related (this can likely be seen in the ground truth annotations for most datasets), the authors consider how well each ground truth concept can predict another (using AUC, for example) and then ask whether the learned concept achieves the same predictive performance.\n\nMore specifically, the authors build a matrix of predictive accuracies for all pairs of concepts, using both the ground truth and learned concepts as predictors. They then take the Frobenius norm of the difference between these two matrices and apply a couple constant factors.\n\nThis is fine, it seems like an improvement on previous metrics for measuring leakage. It also seems related to the idea of measuring how well the learned concepts can predict the ground truth concepts in a 1-1 manner, which is one of the main metrics in the CBM paper. While that metric effectively assesses the mutual information between matched pairs of learned and ground truth concepts while ignoring inter-concept relationships, this one measures (roughly) the delta in mutual information across all pairs.\n\nA couple questions and concerns about this:\n- The bounds used to guarantee that the score is in [0, 1] seem to require the use of AUC as a performance metric. What if we want to use something else, e.g. because the concept is multivariate or real-valued?\n- If you want to quantify the mutual information for a discrete concept, why not use cross entropy loss? This would approximate $H(c_i | c_j)$ and $H(c_i | \\hat c_j)$, whose difference is equal to $I(c_i | c_j) - I(c_i | \\hat c_j)$. This seems more aligned with what you're talking about than AUC.\n- The use of a Frobenius norm on the difference in matrices of prediction accuracies seems a bit heuristic. It's fine, it just seems a bit arbitrary and I wonder if there's a better way to do this depending on what the particular accuracy metric is (e.g., if we had exact mutual information values).\n- The notation in definition 1 is a pretty odd way of saying you train a model $\\psi_j$ to predict one concept from another. I would not have been able to understand this without the explanatory text below. Please consider rewriting this a different way.\n\n### Niching scores\n\nThe next metric tries to provide some measure of concept-based representation quality without requiring access to ground truth concepts. This is quite a leap, because it requires resorting to some notion of quality that departs from how previous work has assessed concept-based representations. \n\nThe authors suggest that for a good concept-based representation, we should be able to identify a small number of concepts that are predictive of each output dimension, and that the remaining concepts should not be predictive (because this would constitute leakage). I read this part of the paper a couple times and I just don't buy this argument, I don't see how the ability to \"niche\" concepts into groups that are useful/useless for each output dimension is a measure of the representation's quality. This is not what I would think of as a measure of whether my learned concepts capture what they're supposed to capture. And the ability to do well on these scores seems dataset-dependent: what if there are output dimensions where every concept is useful, or where no concept is useful? It's a nice idea to try to make a metric that doesn't require access to ground truth concepts, but I don't quite agree with the premise of this particular approach. (Though I'm curious to hear if other reviewers disagree.)\n\nSome other questions and concerns:\n- I'm not so sure about framing the fact that previous metrics require access to ground truth concepts as a weakness. First off, if you're using some of these methods (e.g., CBM) we know you have access to ground truth concepts, so it's not an unrealistic assumption. Second, existing notions of high-quality concept-based representations are defined specifically based on learned representations capturing known concepts, so access to those concepts is clearly necessary; you haven't gotten around that need, you've just opted for an orthogonal notion of representation quality.\n- Isn't a high NPS and low NIS virtually guaranteed by the construction of your concept niche? You're effectively doing feature selection to ensure that the predictive concepts are in one group and the useless ones are in another (for a given output dimension). For any representation at all, it should not be surprising if NPS is high and NIS is low. Or am I missing something here?\n\nAs explained above, I have some qualms about this metric, but given its current state here are a couple other comments:\n- When defining your \"concept nicher\" (i.e., a way to decide which concepts are relevant to a given output dimension), there are a couple problems with your use of correlation. First off, it's worth noticing that correlation is essentially a measure of the MSE from a fitted linear regression model (check the math on this for the specific relationship). With that in mind, there's a better option than taking the max score when you have multivariate learned concepts: derive the MSE from a fitted multivariate linear model that takes in all the concept dimensions. But why not allow a nonlinear model? Also, what if the label is not real-valued but a classification label? Rather than using correlation, wouldn't it be better to fit a classifier and check the cross entropy loss? Yes, this requires fitting a model, but that step is only required once per method evaluation.\n- How are we supposed to choose beta? Different beta values will result in very different outcomes. For example, if I set beta close to 0, I expect we'll see great NPS/NIS scores (because everything will be deemed relevant to every output dim). \n- Overall, there's a lot of wiggle room in your definition of \"concept niching\" and it would be nice if there were a better justified default option. On the other hand, this aspect of your paper is essentially feature selection, and that's a hard problem with approximations of varying quality and computational cost.\n\n### Experiments\n\nThe experiments section uses the proposed metrics to compare some SOTA concept-based and DGL algorithms. As mentioned at the beginning of my review, these experiments overall felt a bit too much like sanity checks (particularly figure 3), because the outcome was mostly what I would have expected to begin with. Also, it could have been nice to use some non-toy datasets.\n\n### Other comments\n\n- While you're at it evaluating various methods, why not include any unsupervised (rather than semi-supervised) DGL algorithms? It would be interesting to see how much worse these do than the other approaches (which is perhaps to be expected because of the works cited).\n- The OIS score appears to require a known alignment between learned and ground truth concepts, is that right? Am I correct in understanding that this is somehow known for semi-supervised DGL approaches but not unsupervised ones? \n- The definitions in this paper are not easy to read. They're too verbose and too long, please try to make them more concise.\n", "summary_of_the_paper": "The authors consider the question of whether recent concept-based learning algorithms, as well disentangled representation learning algorithms, result in high-quality representations. In particular, they consider what high-quality should mean in terms of the relationship with ground truth concepts and the ability to make accurate predictions for a downstream task. To this end, they propose two main metrics for representations that are explicitly or implicitly encouraged to encode concepts: 1) a score that captures how well the learned representation preserves the relationships between concepts (which may be correlated), and 2) a score that captures how well concepts can be split into groups that are useful/useless for predicting particular label dimensions.", "summary_of_the_review": "It's a good idea to assess whether unsupervised and semi-supervised DGL algorithms can learn high-quality concept-based representations, and in doing so the authors developed two metrics that they suggest can be used to compare concept-based representations both with and without access to ground truth concepts. However, the results from their study do not seem very surprising or impactful, and I was not convinced of the rationale behind these metrics (for the reasons described above). For that reason, my rating is that this work is marginally below the acceptance threshold.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper considers the question of whether recent concept-based learning algorithms, as well disentangled representation learning algorithms, result in high-quality representations. In particular, the authors consider what high-quality should mean in terms of the relationship with ground truth concepts and the ability to make accurate predictions for a downstream task. To this end, they propose two main metrics for representations that are explicitly or implicitly encouraged to encode concepts. While the premise of this paper has been appreciated by the reviewers, some concerns about the details of the metrics proposed and experimental results which have been raised by the reviewers remain post rebuttal. Given this, we are unable to recommend the acceptance of the paper at this time. We hope the authors find the reviewer feedback useful.
Decision: Reject