GlanceNets: Interpretable, Leak-proof Concept-based Models
Authors: Emanuele Marconato
Andrea Passerini
Stefano Teso
Venue: NeurIPS 2022 Accept (2022)
Keywords: explainability
concept-based models
interpretability
disentanglement
concept leakage

Abstract: There is growing interest in concept-based models (CBMs) that combine high-performance and interpretability by acquiring and reasoning with a vocabulary of high-level concepts. A key requirement is that the concepts be interpretable. Existing CBMs tackle this desideratum using a variety of heuristics based on unclear notions of interpretability, and fail to acquire concepts with the intended semantics. We address this by providing a clear definition of interpretability in terms of alignment between the model’s representation and an underlying data generation process, and introduce GlanceNets, a new CBM that exploits techniques from disentangled representation learning and open-set recognition to achieve alignment, thus improving the interpretability of the learned concepts. We show that GlanceNets, paired with concept-level supervision, achieve better alignment than state-of-the-art approaches while preventing spurious information from unintendedly leaking into the learned concepts.

GlanceNets: Interpretable, Leak-proof
Concept-based Models
Emanuele Marconato
Andrea Passerini
Department of Computer Science
Department of Computer Science
University of Pisa & University of Trento
University of Trento
Pisa, Italy
Trento, Italy
emanuele. waw.ooattouuminnnn... in
andrea andrea.passer wwwserriiiueniitnnnnn
Stefano Teso
Department of Computer Science
University of Trento Trento, Italy stefano. teso@unitn.com
Abstract
There is growing interest in concept-based models (CBMs) that combine high- performance and interpretability by acquiring and reasoning with vocabulary of high-level concepts. key requirement is that the concepts be interpretable. Exist- ing CBMs tackle this desideratum using variety of heuristics based on unclear notions of interpretability, and fail to acquire concepts with the intended seman- ties. We address this by providing clear definition of interpretability in terms of alignment between the model's representation and an underlying data generation process, and introduce GlanceNets, new CBM that exploits techniques from disen- tangled representation learning and open-set recognition to achieve alignment, thus improving the interpretability of the learned concepts. We show that GlanceNets, paired with concept-level supervision, achieve better alignment than state-of- thethheart approaches while preventing spurious concepts from unintentionally affecting its predictions. The code is available at https://gitub..co/eee aaaaaaaaaaacncalllaaaaaaa lancenet.
Introduction
Concept-based models (CBMs) are an increasingly popular family of classifiers that combine the transparency of white-box models with the flexibility and accuracy of regular neural nets [1-5]. At their core, all CBMs acquire vocabulary of concepts capturing high-level, task-relevant properties of the data, and use it compute predictions and produce faithful explanations of their decisions [6].
The central issue in CBMs is how to ensure that the concepts are semantically meaningful and interpretable for (sufficiently expert and motivated) human stakeholders. Current approaches struggle with this. One reason is that the notion of interpretability is notoriously challenging to pin down, and therefore existing CBMs rely on different heuristics-suce sparse [1], orthonormal to each other [5], or match the contents of concrete examples [3]-with unclear properties and incompatible goals. A second, equally important issue concept leakage, whereby the learned concepts end up encoding spurious information about unrelated aspects of the data, making it hard assign them clear semantics [7]. Notably, even concept- level supervision insufficient to prevent leakage [8].
encouraging the concepts to be
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

imput space
latent space
Pw
x
X
a
2
X
Pn
X
Figure 1: Left: Architecture of GlanceNets showing the encoder decoder pa, classifier pw, and open-set recogmition step. Right: At test time, GlanceNets prevent leakage by identifying and rejecting out-of-distribution inputs using combined strategy, shown here for model trained on digits "4" and "5" only: the "3" is rejected as its embedding falls far away from prototypes of the two training classes (colored blobs), while the "8" is rejected as its reconstruction loss is too large.
Prompted by these observations, we define interpretability in terms of alignment: learned concepts are interpretable if they can be mapped to (partially) interpretable data generation process using transformation that preserves semantics. This is sufficient unveil limitations in existing strategies, build an explicit link between interpretability and disentangled representations, and provide clear and actionable perspective on concept leakage. Building on our analysis, we also introduce GlanceNets (aliGned LeAk-proof coNCEptual Networks), a novel class of CBMs that combine techniques from disentangled representation learning [9] and open-set recognition (OSR) [10] to actively pursue alignment- and guarantee it under suitable assumptions and avoid concept leakage.
Summarizing, we: Provide definition of interpretability as alignment that facilitates tapping into ideas from disentangled representation learning; (ii) Show that concept leakage can be viewed from the perspective of out-of-distribution generalization; (iii) Introduce GlanceNets, novel class of CBMs that acquire interpretable representations and are robust to concept leakage; (iv) Present an extensive empirical evaluation showing that GlanceNets are accurate as state-of-the-a CBMs while attaining better interpretability and avoiding leakage.
Contributions:
2 Concept-based Models: Interpretability and Concept Leakage
Concept-based models (CBMs) comprise two key elements: (i) A learned vocabulary of & high-level concepts meant to enable communication with human stakeholders [11], and (ii) a simulatable [12] classifier whose predictions depend solely those concepts. Formally, CBM f R [c]. with [c] == (l,...,ch c). maps instances x to labels by measuring how much each concept activates on the input, obtaining an activation vector z(x) = (2) (x).... {x))) E R& aggregating the activations into per-class scores (x) using linear map [1, 5], and then passing these through softmax, i.e.,
(x) Ej Weg (x)x,
p(u x) == softmax(s(x)).
(1)
Each weight Wyj E R encodes the relevance of concept z, for class The activations themselves computed in a black-box manner, often leveraging pre-trained embedding layers, but learned so as to capture interpretable aspects of the data using variety of heuristics, discussed below.
Now, as long as the concepts are interpretable, it is straightforward extract human understandable local explanations disclosing how different concepts contributed to any given decision (x, y) by looking at the concept activations and their associated weights, thus abstracting away the underlying computations. This yields explanations of the form {Wyy, 2j (x)) j [k]} that can be readily summarized! and visualized [13, 14]. Importantly, the score of class y is conditionally independent from the input x given the corresponding explanation, i.e., 8y (x) 11. E(x, y), ensuring that the latter is faithful to the model scores. GlanceNets inherit all of these features.
Heuristics for interpretability. Crucially, CBMs are only interpretable insofar as their concepts are. Existing approaches implement special mechanisms to this effect, often pairing a traditional classification loss (such as the cross-entropy loss) with an auxiliary regularization term.
For instance, by pruning those concepts that have little effect the outcome simplify the presentation.

---

Alvarez-Melis and Jaakkola [1] acquire concepts using an autoencoder augmented with sparsification penalty encouraging distinct concepts to activate on different instances. Chen al. [5] apply geometric transforms to learn mutually orthonormal concepts that thus encode complementary information and attain comparable activation ranges. These mechanisms - sparsity and orthogonality, respectively alone cannot prevent capturing features that are not semantic in nature.
second group of CBMs tackle this issue by constraining the concepts to match concrete cases, in the hope that these are better aligned with human intuition [15]. For instance, prototype classification
networks [2], pert--rototype networks [3]. and related approaches [16-18] model concepts using prototypes in embedding space that perfectly match training examples or parts thereof. Depending on the embedding space, which ultimately determines the distance to the prototypes, concepts learned this way may activate on elements unrelated to the example they match, leading to unclear semantics [19].
Closest to our work, concept bottleneck models (CBNMs) [20, 4] align the concepts using concept- level supervision possibly obtained from a separate source, like ImageNet [21] - either sequentially or in tandem with the top-level dense layer. From a statistical perspective, this seems perfectly sensible: if the supervision is unbiased and comes in sufficient quantity, and the model has enough capacity, this strategy appears guarantee the learned and ground-truth concepts match.
Concept leakage in concept-bottleneck models. Unfortunately, concept-level supervision is nor sufficient to guarantee interpretability. Mahinpei et al. [7] have demonstrated that concepts acquired by CBNMs pick up spurious properties of the data. In their experiment, they learn two concepts 2 and 25, meant represent the 4 and 5 MNIST digits, using concept-level supervision, and then show that - surprisingly these concepts can be used to classify all other digits (i.e., MNIST images that neither 4's nor 5's) whereby learned concepts unintentionally capture information about unobserved concepts, is known as concept leakage.
or odd significantly better than random guessing. This phenomenon,
Intuitively, leakage occurs because in CBNMs the concepts end up unintentionally capturing distribu- tional information about unobserved aspects of the input, failing to provide well-defined semantics. However, a clear definition of leakage is missing, and so are strategies to prevent it. In fact, separating concept learning from classification and increasing the amount of supervision for the observed con- cepts (here, 4 and 5) is not enough [8]. A key contribution of our paper is showing that leakage can be understood from the perspective of domain shift and dealt with using open-set recognition [10].
3 Disentangling Interpretability and Concept Leakage
The main issue with heuristics used by CBMs is that they are based on unclear notions of inter- pretability. In order to develop effective algorithms, we propose to view interpretability of alignment between the machine's 's representation and that of its user. This enables us to identify conditions under which interpretability can be achieved, build links well-understood properties of representations, and leverage of learning strategies.
form
Interpretability. We henceforth focus on the (rather general) genera- tive process shown in Fig. 2: the observations X RRd are caused by n generative factors G E RR,, themselves caused by set of confounds C (including the label [22]). Notice that the generative factors can be statistically dependent due to the confounds C, but as noted by Suter et al. [23], the total causal effect [24, Def. 12] between G, and G, is zero for all i * j. The generative factors capture all information necessary to determine the observation [23, 25], so the goal is to learn concepts Z 1 that recover them. The variable T is also a confound- ing factor, but is kept separate from as it relates to concept leakage, and will be formally introduced later
C
X
Figure 2: The data genera- tion process.
We posit that a (learned) representation is only interpretable if supports symbolic communication between the model and the user, in the sense that it shares the same (or similar enough) semantics to the user's representation. The latter is however generally unobserved. We therefore make a second, critical assumption that some of the generative factors G, C G are interpretable to the user, meaning that they can be used as a proxy for the user's internal representation. Naturally, not all generative factors are interpretable [26], but in many applications some of them are. For instance,
3

---

in dSprites [27] the generative factors encode the position, shape and color of a 2D object, and in CelebA [28] the hair color and nose size of celebrity. Human observers have good grasp of such concepts.
Interpretability as alignment. Under this assumption, if the variables Z, C Z are aligned to the generative factors G₁ by a map g + z, that preserves semanties, they are themselves interpretable. Now, defining what a semantics-preserving map should look like challenging, but constructing one is not: the identity is clearly one such map, and so are maps that permute the indices and independently rescale the individual variables. One desirable property that a does not "mix" multiple G's into a single Z. E.g. if Z blends together head tilt, hair color, and nose size, users will have trouble pinning down what it means." ? This property can be formalized in terms of disenttangemmett [29, 23, 9]. This is however insufficient: we wish the map between G; and its associated factor Z, to be "simple", so as to conservatively guarantee that it preserves semantics. This makes alignment strictly stronger than disentanglement.
Motivated by these desiderata, we say that Z₁ aligned to G, if it satisfies:
(i) Disentanglement. There exists an injective map between indices [ns] + [k]] where [12] identifies the subset of generative factors indexes in G such that, for all i, i' E [nr]. + i', and j (()) it holds that fixing G; is enough to fix Z₃ regardless of the value taken by the other generative factors Gy, and
(ii) Monotonicity. The map a can be written as or(g) (mz (9x(()))) ..... "n ((x(()))) )))" where the N4 s are monotonic transformations. This holds, for instance, for linear transformations of the form (I+(1)..... (.())))))))) )T, where A E R x a matrix with no non-ze zero off off-diagonal entries. This second requirement can be relaxed depending on the application.
Notice that we do not require each G; to map to single Zj (a property known as completeness [29]: Zu is interpretable even if it contains multiple perhaps slightly different, but aligned transforma- tions of the same GІ.
Measuring alignment with DCI. Disentanglement can be measured in a number of ways [30], but most of them provide little information about how simple the map a is. order to estimate alignment, we repurpose DCI, a measure of disentanglement introduced by Eastwood and Williams [29], see also ??. According to this metric, a representation Z, disentangled if there exists a regressor that, given z,, can predict g, with high accuracy using few 2;'s 's to predict each 9i. Following [29], we use a linear regressor with parameters E Rex- on the test set assuming that it is annotated with the interpretable generative factors and corresponding learned representations and then measure how diffuse the weights associated to each latent factor are. We do this by normalizing them and computing their average Shannon entropy over all G, s, i.c.,
- -ESEIH Pj (Eretar) bij log b5) where big bjs/ EpenN bys and Pj E; bys/ Ey, by
(2)
Hence, DCI gauges the degree of mixing that linear map can attain using the learned representation Z, and as such it indirectly measures alignment, with B approximating the inverse of A.
Achieving alignment with concept-level supervision. It has been shown that disentanglemen cannot be achieved in the purely unsupervised setting [31]. This immediately entails that alignment is also impossible in that setting, highlighting core limitation of approaches like self-explainable neural networks [1]. However, disentanglement can be attained supervision about the generative factors is available, even only for a small percentage of the examples [32]. As matter of fact, supervision is used in representation learning to achieve ideniifiabiliy, a stronger condition than and that entails both of disentanglement and alignment [33]. Thus, following CBNMs, we seek alignment by leveraging concept-level supervision.
Interpretability and concept leakage. Intuitively, concept leakage occurs when model is trained on a data set on which:
(i) Some generative factors Gy G vary, while the others G G \ Gy are fixed, and
"The converse is not true: interpretable concepts with compatible semantics can be mixed without compro- mising interpretability. E.g., rotating coordinate system gives another intuitive coordinate system. Our point is that conservatively avoiding mixing helps preserve semantics.

---

(ii) The two groups of factors are statistically dependent.
For instance, in the even vs. odd experiment and play the role of Gy and the other digits of Ge- CBNMs with access to supervision on Gv tend to acquire a latent representation that approximates these factors. But, because of (ii), this representation correlates with the fixed factors Gp. This immediately explains why additional supervision Gy cannot prevent leakage, but rather has the opposite effect: the better a latent representation matches Gy, the more information it conveys about Gr.
In contrast with previous assessments [7, 8], we observe that this phenomenon can be viewed as a special form of domain shift: the training examples are sampled from a ground-truth distribution p(X, G T = 1) in which GF is approximately fixed, e.g., p(GF T = 1) = ó(g')) for some vector gy., while in the test set, the data is sampled from a different distribution pl X, G T 0) in which GF is no longer fixed. In the MNIST task, for instance, when T no concept besides 4 and 5 can occur, while all concepts except 4 and 5 can occur when T O. Here, T E {0. 1] seleets between training and test distribution, see Fig. 2. Now, CBMs have no strategy to cope with domain shift and thus cannot disambiguate between known training and unknown test concepts.
Motivated by this, we propose then to tackle concept leakage by designing a CBM specifically equipped with strategies for detecting at inference time instances that do not belong to the training distribution using OSR [10]. The idea is to estimate the value of the variable at inference time, essentially predicting whether input was sampled from distribution similar enough to the training distribution, and therefore can be handled by a model learned on this distribution, or not. This strategy proves very effective in practice, shown by our empirical evaluation (Section 5.2).
4 Addressing Alignment and Leakage with GlanceNets
GlanceNets combine VAF-like architecture [34, 35] for learning disentangled concepts with a prior and classifier designed for open-set prediction [36]. order to accommodate for non-interpretable factors, the latent representation of GlanceNets Z is split into two: (i) k concepts Z,, aligned to the interpretable generative factors G,, that are used for prediction, and (ii) & opaque factors Z, j that only used for reconstruction. Specifically, GlanceNet comprises
encoder qa (Z X) and decoder pa [X] Z), both parameterized by deep neural networks, as well as a classifier pw (Y Zz) feeding off the interpretable concepts only. The overall architecture is shown in Fig. 1.
Following other CBMs, the classifier is implemented using a dense layer with parameters W E Rexk followed by softmax activation, i.e. Pw (Y z := softmax (Wz,) Wz)), ), and the most likely label is used for prediction. The class distribution obtained by marginalizing over the encoder's 's distribution:
p( x) := (x1))
z, x) (x)() [pw (Y) zz)]]
Equality holds because Y 1L X Zy. In order to expedite the computation, we follow the general practice of approximating the integral as softmax(W (e,)(())((2]]] = softmax ( (M (x)],).
In contrast to regular VAEs, GlanceNets associate each class to a prototype in latent space through the prior p(Z Y), which is conditioned on the class and modelled as a mixture of gaussians with one component per class. The encoder, decoder, and prior are fit on data so as to maximize the evidence lower bound (ELBO) [37], defined as (xp(x.)) [C(0, x. y; B]] with:
(4) Here, PD (x, u) is the empirical distribution of the training set D = {xx; yi) 1,...,m, m). The first term of Eq. (4) is the likelihood of an example after passing it through the encoder distribution.
C(8, x. y: (3) : Ey [log (x z) log Pw (y 2.)] - , KL(gs (z x) || p(z y))
The second term penalizes the latent vectors based on how much their distribution differs from the prior and encourages disentanglement. As mentioned in Section 3, learning disentangled represen- tations is impossible in the unsupervised i.id. setting [31]. Following Locatello et al. [32], and similarly to CBNMs, we assume access to a (possibly separate) data set D = {(xe, e)) containing supervision about the interpretable generative factors G, and integrate it into the ELBO by replacing the per-example loss L in Eq. (4) with:
E Bpp(x.) [[00, x, y: {)] + 7 Ep (x.)) (xx [((z. g)]
(5)
5

---

where y > 0 controls the strength of the concept-level supervision. Following [32], the term (Z g) penalizes encodings sampled from go(z x) for differing from the annotation g. Specifically, we implement this term using the average cross-entropy loss (2 g) == Σx 9h log a + (1 gs) - (=g(1-o(=2)) where the annotations gle are rescaled to lie in [0. 1] and a is the sigmoid function.
Dealing with concept leakage. In order to tackle concept leakage, GlanceNets integrate the OSR strategy of Sun et al. [36], indicated in Fig. 1 by the "osr" block. This strategy identifies out-of-class inputs by considering the class prototype My := Ep(z v) [z] in R* defined by the prior distribution and the decoder pa x l2.. Recall that the prior is fit jointly with the encoder, decoder, and classifier by optimizing the ELBO. Once learned, GlanceNet uses the training data estimate: (7) a distance
threshold Fy. which defines a spherical subset in the latent space B, (z /| z < fl ] centered around the prototype of class y (i.e., the mean of the corresponding Gaussian mixture component), and (ii) a maximum threshold on the reconstruction eгror Thhr If new data points have reconstruction error above Thhr or they do not belong to any subset By, they are inferred as open-set instances, i.e.,
T = O. In practice, we found that choosing the thresholds as to include 95% of training examples to work well in experiments. Further details are available in Appendices ?? and ??.
4.1 Benefits and Limitations
GlanceNets can naturally be combined with different VAE- architectures for learning dis- entangled representations [38], including 8-TCVAEs [39], Info [40]. DIP-VAEs [41], and JLI-VAEs [42]. Since our experiments already show substantial benefits for GlanceNets building on 3-VAEs [43], we leave detailed study of these extensions to future work.
Like CBNMs, GlanceNets foster aligoment by leveraging supervision on the interpretable generative factors [32], possibly derived from external data set [20]. However, GlanceNets can be readily adapted to a variety of different kinds of supervision used for VAE-based models, including par- tially annotated examples [26], group information [44], pairings [45, 46] and other kinds of weak supervision [47, 48). as well as feedback from a domain expert [49]. On the other hand, CBNMs are incompatible with these approaches.
One limitation inherited from VAEs by GlanceNets is the assumption that the interpretable generative factors are disentangled from each other [23]. In practice, GlanceNets work even when this does not hold (as in our even vs. odd experiment, see Section 5.2). However, one direction of future work is to integrate ideas from hierarchical disentanglement [50].
Empirical Evaluation
In this section, we present results several tasks showing that GlanceNets outperform CBNMs [20] in terms of alignment and robustness to leakage, while achieving comparable prediction accu- racy. All experiments were implemented using Python 3 and Pytorch [51] and run on server with 128 CPUs, ITiB RAM, and 8 A 100 GPUs GlanceNets were implemented on top of the disentanglement- pytorhh [52] library. All alignment and disentanglement metries were com- puted with disentangl ement __ie [31]. Code for the complete experimental setup is available on Additional details on architectures
GitHub at the link: https://github..om www.b/.coo/maaaaae... marconato/ and hyperparameters can also be found in the Supplementary Material.
5.1 GlanceNets achieve better alignment than CBNMs
In a first experiment, we compared GlanceNets with CBNMs on three classification tasks for which supervision on the generative factors is available. In order to evaluate the impact of this supervision on the different competitors, we varied the amount of training examples annotated with it from 1% to 100%. For each increment, we measured prediction performance using accuracy, and alignment using the linear variant of DCI [29] discussed in Section 3.
Data sets. We carried out our evaluation on two data sets taken from the disentanglement literature and a very challenging real-world data set. dSprites [27] consists of x 64 black-and- white images of sprites on a flat background, where each sprite is determined by one categorical and four generative factors, namely shape, size, rotation, position_s x, and position sii__n__..___ The images were obtained by discretizing and enumerating the generative factors, for total of 3 x 6 x 40 x 32 32

---

ACCURACY
ALIGNMENT
EXPLICITNESS
DSPRITES
13
0.50
MP13D
CELEBA
0.54
63
Figure 3: GlanceNets are better aligned than CBNMs. Each row is a data set and each column re- ports a different metric. The horizontal axes indicate the % of training examples for which supervision on the generative factors is provided. Remarkably, in all data sets GlanceNets achieve substantially better alignment than CBNMs for the same amount of supervision, and achieve comparable accuracy in 14 cases out of 15.
images. MP13D [53] consists of 64 x 64 RGB rendered images of 3D shapes held by robotic arm. The generative factors are cojject_color object_shappe object _size, camera_he background_ color, and the horizontal and vertical position of the arm. The data contains 6 x 6 x 2 x 3 x 4 x 40 x 40 examples. CelebA [28] is collection of 178 x 218 RGB images of over 10k celebrities, converted to 64 x 64 by first cropping them to 178 x 178 and then rescaling. Images annotated with 40 binary generative factors including hair color, presence of sunglasses, etc. Since we are interested in measuring alignment, we considered only those 10 factors that CBNMs can fit well (in the Appendix). We also dropped all those examples for which hair color is not unique (e.g., annotated as both blonde and black), obtaining approx. 127k examples. CelebA is more challenging than dSprites and MPI3D, as it does not include all possible factor variations and the generative factors- although disentangled - are insufficient to completely determine the contents of the images. For dSprites and MPI3D, we used random 80/10/10 train/validation/eest split, while for CelebA we kept the original split [28].
We generated the ground truth labels y follows. For dSprites, labeled images according to a random but fixed linear separator defined over the continuous generative factors, chosen so as to ensure that the classes are balanced. For MPI3D and CelebA, we focused on the categorical factors instead. Specifically, we clustered all images using the algorithm of [54], for a total of 10 and 4 clusters for MPI3D and CelebA respectively, and then labeled all examples based their reference cluster. This led to slightly unbalanced classes containing different percentages of examples, ranging from 5% to 16% in MPI3D and from 21% to 29% in CelebA
Architectures. For dSprites and MPI3D, we implemented the encoder as a six layer convolutional neural net, while for CelebA we adapted the convolutional architecture of Ghosh et al. [55]. We employed a six layer convolutional architecture for the decoder in all cases, for simplicity, as changing it did not lead to substantial differences in performance. In all cases, for all CBMs (see Section 2), the classifier was implemented as a dense layer followed by a softmax activation. The very same architectures were used for both GlanceNets and CBNMs, for fairness. For each data set, we chose the latent space dimension as the total number of generative factors, where categorical ones are one hot encoded. In particular, we used 7 latent factors for dSprites, 21 for MPI3D and 10 for CelebA. Further details are included in the Supplementary Material.
Results and discussion. The results of this first experiment are reported in Fig. 3. The behavior of both competitors on dSprites and MP13D was extremely stable, owing to the fact that these data sets cover essentially exhaustive set of variations for all generative factors, so we report their hold-out
7

---

performance on the test set. Since for CelebA variance was non-negligible, we ran both methods 7 times varying the random seed used to initialize the network and report the average performance across runs and its standard deviation.
In addition alignment, we also report explicitness [29], which measures how well the linear regressor employed by DCI fits the generative factors. The higher, the better. Details on its evaluation are included in Suppl. Material.
The plots clearly show that, although the two methods achieve high and comparable accuracy in all settings, GlanceNets attain better alignment in all data sets and for all supervision regimes than CBNMs, with a single exception in CelebA using low values of supervision, for a total of 13 wins out of 15 cases. In all disentanglement data sets, there is clear margin between the alignment achieved by NetsceNets and that of CBNMs: performances vary up to maximum of 15% in dSprites, and minumum of 8% in MPI3D. In CelebA, the gap is evident with full supervision (almost 8% of difference in alignment), and GlanceNets still attain overall better scores in the 25% and 50% regime. On the other hand, performance are lower, but comparable, with 10% supervision. The case I 1% refers to an extreme situation where both CBNMs and GlanceNets struggle to align with generative factors, as is clear also from the very low explicitness.
In dSprites and MPI3D, both GlanceNets and CBNMs quickly achieve very high alignment at 1% supervision, expected [32], whereas better results in CelebA are obtained with growing supervision. Also, both models display similar stability on this data set, as shown by the error bars in the plot.
5.2 GlanceNets are leak-proof
Next, we evaluated robustness to concept leakage in two scenarios that differ in whether the un- observed generative factors are disentangled with the observed ones or not, see Section 3. In both experiments, we compare GlanceNets with CBNM and modified GlanceNet where the OSR component has been removed (denoted CG-VAB).
Leakage due to unobserved entangled factors. We start by replicating the experiment of Mahinpei et al. [7]: the goal is to discriminate between even and odd MÑIST images using a latent representation (Z4, Zs) obtained by training (with complete supervision on the generative factors) only on examples of 4's and 5's. Leakage occurs if the learned representation can be used to solve the prediction task better than random on a test set where all digits except and 5 occur." During training. we use the digit label for conditioning the prior p( Y) of the GlanceNet. More qualitative results are collected in ??.
even
odd
21.5%
25
-5
10
-10
-10
(a)
(b)
(c)
Figure 4: GlanceNets are leak-proof on MNIST. (a) Training set embedded by GlanceNet with 100; axes indicate 24 and 25 and color the concept label, i.e., 4 5. (b) Latent representations of the test images, divided in even vs. odd. Every ball in light gray denotes the region Hy z| < fy for each class prototype y. For more details, refer Section 3. (c) Information Leakage performances of the considered models: CBNM, CG- CGEVAE and GlanceNet.
Fig. 4 (a, b) illustrates the latent representations of the training and test set output by GlanceNet: since the two digits are mutually exclusive, the model has learned to map all instances along the (z) diagonal. This is where OSR kicks in: if an input is identified as open-set, T is predicted as 0 by the OSR component and the input is rejected. In all Icakage experiments, we implement rejection by predicting random label. Since MNIST is balanced, measure leakage by computing
'arreeloiu et al. [8] perform classification using multi-laye perceptron on top of z. Following the CBM literature, we linear classifier instead. Leakage occurs regardless.
8

---

the difference in accuracy between the classifier and an ideal random predictor, i..., 2 -|ace - L1: the smaller, the better. The results, shown Fig. 4 (c), show a substantial difference between GlanceNet and the other approaches. Consistently with the values reported in [7]. CBNMs are affected by considerable amount of leakage, around 28%. This is not the case for our GlanceNet: most (approx. 85%) test images are correctly identified as open-set and rejected, leading to very low (about 2%) leakage, 26% less than CBNMs. The results for CG-VAE also indicate that removing the open-set component from GlanceNets dramatically increases leakage back to around 30%. This shows that alignment and disentanglement alone are not sufficient, and that the open-set component plays a critical role for preventing leakage.
Leakage due unobserved disentangled factors. Next, we analyze concept leakage between disen- tangled generative factors using the dSprites data set. To this end, we defined a binary classification task in which the ground-truth label depends on position, x and position.
only. In particular, instances within fixed distance from (0, 0) are annotated as positive and the rest as negative, as shown in Fig. (a). In order to trigger leakage, all competitors are trained (using full concept-level supervision, as before) and position on in__on_ are almost constant (they range in small interval around (0.5, 0.5), ef. Fig. 5). leakage occurs if the learned model can successfully classify test instances where position_s and position_y are no longer fixed. More qualitative results can be found in ??.
training images where shape, size and rotation vary, but posi position...
positives negutives
102
08.2
regatives
train ro
PCA aoce
0C
poskinss negatives separation
20
2.18)
PCA anis
PCA
(a)
(b)
(c)
(d)
Figure 5: GlanceNets are leak-proof on dSprites. (a) The variations over pos_x _s__xd and pos_y y for the
training set, and for the test set, divided in positives vs. negatives. (b) PCA reduction for GlanceNet over the its five latent factors. (c) PCA reduction for CBNM; the dotted line indicates the separating hyperplane predicted in the second phase. (d) Leakage % for CBNM, CG-VAE and GlanceNet.
For both competitors, we encode shape using 3D one-hot encoding and size and rotation as continuous variables. During training, we use the shape annotation for conditioning the prior p(Z Y) of the GlanceNet. The first two PCA components of the latent representations acquired by our GlanceNet and by a CBNM are shown, rotated so as to be separable on the first axis, in Fig. 5 (b, c): in both cases, it is possible to separate positives from negatives based the obtained representations in the five latent dimensions. As shown in Fig. (d), this means that both CBNM and CG- (G-VAE suffer from very large leakage, 80% and 98%, respectively. In contrast, OSR allows us correctly identify and reject almost all test instances, leading to negligible leakage even in this disentangled setting.
6 Related Work
Concept-based explainability. Concepts lie at the heart of AI [56] and have recently resurfaced as a natural medium for communicating with human stakeholders [11]. J. In explainable AI, this was first exploited by approaches like TCAV [57], which extract local concept-based explanations from black-box models using concept-level supervision to define the target concepts. Post-hoc explanations, however, notoriously unfaithful to the model's reasoning [58-60]. CBMs, including GlanceNets, avoid this issue by leveraging concept-like representations directly for computing their predictions. Existing CBMs model concepts using prototypes [2, 3, 16, 17] or other representations [1, 20, 4, 5], but they seek interpretability using heuristics, and the quality of concepts they acquire has been called into question [61, 19, 7, 8). We show that disentangled representation learning helps in this regard.
Disentanglement and interpretability. Interpretability is one of the main driving factors behind the development of disentangled representation learning [62-64]. These approaches however make no distinction between interpretable and non-interpretable generative factors and generally focus on properties of the world, like independence between causal mechanisms [9] or invariances [43].

---

Interpretability, however, depends on human factors that are not well understood and therefore usually ignored [12, 65]. The link between disentanglement and interpretability has never been made explicit. Importantly, in contrast to alignment, disentanglement does not require that the map between matching generative and learned factors preserves semantics. We remark that other VAE--based classifiers either do not tackle disentanglement or are unconcerned with concept leakage [66, 67, 36).
Disentanglement and CBMs. Neither the literature on disentanglement nor the one on CBMs have attempted to formalize the notion of interpretability or to establish proper link between the latter and disentanglement. The work of Kazhdan et al. [68] the only one to compare techniques for disentangled representation learning and concept acquisition, however it makes no attempt at linking the two notions. Our work fills this gap.
Acknowledgments and Disclosure of Funding
The research of ST and AP was partially supported by TAILOR, a project funded by EU Horizon 2020 research and innovation programme under GA No 952215.
References
[1] David Alvarez-Melis and Tommi S Jaakkola. Towards robust interpretability with self-explaining neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 7796-7795, 2018.
[2] Oscar Li, Hao Liu, Chaofan Chen, and Cynthia Rudin. Deep learning for case-based reasoning through prototypes: A neural network that explains its predictions. In Proceedings of the AAAI Conference Artificial Intelligence, 2018
[3] Chaofan Chen, Oscar Li, Daniel Tao, Alina Barnett, Cynthia Rudin, and Jonathan K. Su. This looks like that: Deep learning for interpretable image recognition. Advances in Neural Information Processing Systems, 32:8930-8941, 2019.
[4] Max Losch, Mario Fritz, and Bernt Schiele. Interpretability beyond classification output: Semantic Bottleneck Networks. arXiv preprint 1rXiv:1907.10856 2019.
[5] Zhi Chen, Yijie Bei, and Cynthia Rudin. Concept whitening for interpretable image recognition. Nature Machine Inielligence, 2(12):772-782, 2020.
[6] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-25 215, 2019.
[7] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi- Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. In International Conference on Machine Learning: Workshop on Theoretic Foundation, Criticism, and Application Trend of Explainable AI, volume pages 1-13, 2021.
[8] Andrei Margeloiu, Matthew Ashman, Umang Bhatt, Yanzhi Chen, Mateja Jamnik, and Adrian Weller. Do concept bottleneck models learn as intended? arXiv preprint arXiv:2105.04289 2021.
[9] Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5):612666) 634, 2021.
[10] Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE transactions on pattern analysis and machine intelligence, 35(7):
1757-1772, 2012.
[11] Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and Lin Guan. Symbols as Lingua Franca for Bridging Human-Al Chasm for Explainable and Advisable Al Systems. In Proceedings of Thirty-Sixti Sixth AAAI Conference on Artificial Intelligence (AAAI), 2022.
10

---

[12] Zachary C Lipton. The mythos of model interpretability: In machine learning, the concept of interpretability both important and slippery. Queue, 16(3):31-57, 2018.
[13] Peter Hase and Mohit Bansal. Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pages 5540-5555 2020.
[14] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM computing surveys (CSUR), 51(5): 1-42, 2018.
[15] Been Kim, Rajiv Khanna, and Oluwasanmi O Koyejo. Examples criticize! criticism for interpretabilityy. Advances in neural information processing systems, 29, 2016.
enough, learn to
[16] Dawid Rymarczyk, Lukasz Struski, Jacek Tabor, and Bartosz Zieliński. ProtoPShare: Pro- totypical Parts Sharing for Similarity Discovery in Interpretable Image Classification. In Proceedings the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, page 1420-1430, 2021.
[17] Meike Nauta, Ron van Bree, and Christin Seifert. Neural prototype trees for interpretable fine- fineggrained image recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Putlern Recognition, pages 14933-14943, 2021.
[18] Gurmail Singh and Kin-Choong Yow. These do not look like those: An interpretable deep learning model for image recognition. IEEE Access, 9:41482 414493934449,, 2021.
[19] Adrian Hoffmann, Claudio Fanconi, Rahul Rade, and Jonas Kohler. This looks like that... does it? shortcomings of latent space prototype interpretability in deep networks. arXiv preprint arXiv:2105.02668 2021.
[20] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International Conference on Machine Learning, pages
PMLR, 2020.
[21] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. leee, 2009.
[22] Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv: /tt::/206.666 6471, 2012.
[23] Raphael Suter, Djordje Miladinovie, Bernhard Schölkopf, and Stefan Bauer. Robustly disen- tangled causal mechanisms: Validating deep representations for interventional robustness. In International Conference on Machine Learning. pages 6056-6065. PMLR, 2019.
[24] Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: founda- tions and learning algorithms. 2017.
[25] Abbavaram Gowtham Reddy, I. Benin Godfrey, and Vineeth N Balasubramanian. On causally disentangled representations. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[26] Aviv Gabbay, Niv Cohen, and Yedid Hoshen. An image is worth more than a thousand words: Towards disentanglement in the wild. Advances in Neural Information Processing Systems, 34, 2021.
[27] Loic Matthey, Irina Higgins, Demis Hassabis, and Alexander Lerchner. dsprites: Disentangle- ment testing sprites dataset.
2017.
[28] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang, Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) December 2015.
[29] Cian Fastwood and Christopher KI Williams. A framework for the quantitative evaluation of disentangled representations. In International Conference Learning Representations, 2018.

---

[30] Julian Zaidi, Jonathan Boilard, Ghyslain Gagnon, and Marc-André Carbonneau. Measuring disentanglement: review of metries, arXiv preprint arXiv:2012.09276, 2020.
[31] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In International Conference on Machine Learning, pages 4114-4124, 2019.
[32] Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Rätsch, Bernhard Schölkopf. and Olivier Bachem. Disentangling factors of variations using few labels. In International Conference on Learning Representations, 2020.
[33] llyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207-2217. PMLR, 2020.
[34] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International conference on machine learning. PMLR, 2014.
[35] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference deep generative models. In International conference on machine learning. PMLR, 2014.
[36] Xin Sun, Zhenning Yang, Chi Zhang, Keck Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13480-13489, 2020.
[37] Diederik P Kingma and Max Welling. An introduction to variational autoencoders. Foundations and Trends@ in Machine Learning, 12(4):307-392, 2019.
[38] Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, Narayanaswamy Siddharth, Brooks Paige, Dana H Brooks, Jennifer Dy, and tations. In The 22nd International Conference Artificial Intelligence and Statistics, pages 2525-2534. PMLR, 2019.
Meent, Structured disentangled represen-
[39] Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaud. Isolating sources of disentanglement in vaes. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 2615-2625, 2018.
[40] Shengjia Zhao, Jiaming Song. and Stefano Ermon. Infovae: Balancing learning and inference in variational autoencoders. In Proceedings of the aaai conference on artificial intelligence. volume 33, pages 5895, 2019.
[41] Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference of disentangled latent concepts from unlabeled observations. In International Conference on Learning Representations, 2018
[42] Travers Rhodes and Daniel Lec. Local disentanglement variational auto-encoders using jacobian 1_1 regularization. Advances in Neural Information Processing Systems, 34:22708- 22719, 2021.
[43] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. f-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representations, 2016.
[44] Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. Multi-level variational autoen- coder:
Learning disentangled representations from grouped observations. In -herond-Second AAAI Conference on Artificial Intelligence, 2018.
[45] Rui Shu, Yining Chen, Abhishek Kumar, Stefano Ermon, and Ben Poole. Weakly supervised disentanglement with guarantees. In International Conference on Learning Representations, 2020.
12

---

[46] Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International Conference Machine Learning, pages 6348 6359. PMLR, 2020.
[47] Aviv Gabbay and Yedid Hoshen. Latent optimization for non-adversarial representation disen- tanglement. arXiv preprint arXiv: /906. 1796, 2019.
[48] Junxiang Chen and Kayhan Batmanghelich. Weakly supervised disentanglement by pairwise similarities. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3495-3502 2020.
[49] Wolfgang Stammer, Marius Memmel, Patrick Schramowski, and Kristian Kersting, Interactive disentanglement: Learning concepts by interacting with their prototype representations. arXiv preprint (t.Xv::2//2.022200 2021.
[50] Andrew Ross and Finale Doshi-Veles disentanglement. In International Conference on Machine Learning, pages 9084-9094. PMLR, 2021.
Benchmarks, algorithms, and metries for hierarchical
[51] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, tigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[52] Amir H. Abdi, Purang Abolmaesumi, and Sidney Fels. disentanglementtpyiien pyttorh.. arXiv preprint arXiv: 1912.0544 2019.
Variational learning with
[53] Muhammad Waleed Gondal, Manuel Wuthrich, Djordje Miladinovic, Francesco Locatello, Martin Breidt, Valentin Volchkov, Joel Akpo, Olivier Bachem, Bernhard Schölkopf, and Stefan Bauer. On the transfer of inductive bias from simulation to the real world: new disentanglement dataset. H. Wallach, H. Larochelle, A. Beygelzimer, d' d'Alché-Buc, Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[54] Zhexue Huang. Clustering large data sets with mixed numeric and categorical values. In In The First Pucific-Asia Conference on Knowledge Discovery and Data Mining, pages 21-34, 1997.
[55] Partha Ghosh, Mehdi SM Sajjadi, Antonio Vergari, Michael Black, and Bernhard Schölkopf. From variational to deterministic autoencoders. arXiv preprint arXiv: 190.::993./222 /2436, 2019.
[56] Stephen Muggleton and Luc De Raedt. Inductive logic programming: Theory and methods. The Journal of Logic Programming, 19:629-679, 1994.
[57] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav). In International conference machine learning, pages 2668-2677. PMLR, 2018.
[58] Ann-Katie Kun-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus- Robert Müller, and Pan Kessel. Explanations can be manipulated and geometry is to blame. Advances in Neural Information Processing Systems, 32: 12589-13660 -3600, 2019.
[59] Stefano Teso Toward faithful explanatory active learning with self-explainable neural nets. In Proceedings of the Workshop on Interactive Adaptive Learning (IAL 2019), pages 4-16, 2019.
[60] Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified bp attributions fail. International Conference on Machine Learning, pages 9046-9057. PMLR, 2020.
[61] Meike Nauta, Annemarie Jutte, Jesper Provoost, and Christin Seifert. This looks like that, because... explaining prototypes for interpretable image recognition. In Joint European Confer- ence on Machine Learning and Knowledge Discovery in Databases, pages 441-456. Springer, 2021.
13

---

[62] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 1798-1828, 2013.
[63] Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Josh Tenenbaum. Deep convo- lutional inverse graphics network. Advances in neural information processing systems, 28, 2015.
[64] Xi Chen, Yan Duan, Rein Houthooft, John Schulman, llya Sutskever, and Pieter Abbeel. Info- GAN: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016.
[65] Tim Miller. Explanation in artificial intelligence: Insights from the social sciences, Artificial intelligence, 267:1-38, 2019.
[66] Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem. Are disentangled representations helpful for abstract visual reasoning? In NeurlPS, 2019.
[67] Weidi Xu and Haoze Sun. superviee variational autoencoders for sequence classification. ArXiv, abs/1603.0255 100504044, 14, 2016.
[68] Dmitry Kazhdan, Botty Dimanov, Helena Andres Terre, Mateja Jamnik, Pietro Liò, and Adrian Weller. Is disentanglement all you need? comparing concept-based & disentanglement ap- proaches. arXiv preprint arXiv: 104.0270 2021.
14

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] You are strongly encouraged to include a justification your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Refer sec.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes]] We described the main procedure to obtain our same results. More details will be available on the Suppl. Material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] The most important details are addresses in the main text 3. Those not relevant to discussion, like values of the hyperparameters, number of epochs and training heuristics, are included the Suppl. Material.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] We reported the errorbars for CelebA experiment, see Fig. 3
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We use existing libraries and open data sets. Sources are cited in the experiments section
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're nou're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
15

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
16


Reviews:
[{"review_id": "xmGMZKCavQ", "reviewer": "Reviewer_jG58", "summary": "\nThis work introduces a new autoencoder architecture GlanceNet and proposes to advance the interpretability of concept-based models (CBMs) by introducing a definition of interpretability that is based on a supervised aligning of learned latent concepts with the factors of the underlying data generation, some of which maybe semantically interpretable to a user.\n\nThe latent space Z induced be the encoder q_theta(Z | X) is divided into two parts, Zj and Zbar_j.  Zj is for learning of k concepts which are used for task level classification p(Y | Zj) and for aligning with supervised generative factors G, which is measured via a DCI metric from the disentanglement literature.  Zbar_j is used by the decoder to reconstruct x_hat which is in turn novelly used by an open set recognition module to learn class prototypes that determine if a test set example follows the training distribution ( T=1 ) or not ( T=0 ) and can be used for rejecting test cases.  Additionally, a prior p(z | y) of concept prototypes is learned as a mixture of gaussians (one component per class) and used to encourage disentangled representation learning via penalizing the KL divergence between it and the latent space, KL( p(z|y) || q( z | x) ).    \n\nOver three image tasks with varying amounts of supervision for generative factors (1 to 100% of examples), and under the assumption that interpretable generative factors are disentangled from each other, the authors show the architecture performs comparably or better in terms of accuracy, and better in terms of the two interpretable metics, alignment and ``explictness” which measures how well the linear regressor employed by DCI fits the generative factors using Zj.  ", "questions": "1) Its unclear what the method does when T predicts a test example is out of domain.  Is it just not used in the results in section 5.1?  \nIn 5.2 you show OSR helps with concept leakage, by rejecting 85% or almost all cases and then guessing random, which by definition will do get you better results for a binary task, but its not clear how OSR is used in 5.1 ( are samples rejected there too?).  \n\n2) What is the size of Zbar_j in these experiments?  I might be missing something but I didn’t see it in the appendix ?\n\n3) How does the accuracy of your experiments compare with other non-concept based models in general?    ", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths:\nThe paper is novel for its proposed architecture, tying the definition of interpretability via the alignment of learned concepts to generative factors, and its utilitization of OSR for use in preventing concept leakage.  It additionally is shown to be quite performative against the concept whitening paper for synthetic cases ( first two experiments dSprites and MPI3D) & the third case CelebA which is semi-synthetic because the supervised factors themselves are used to cluster and create classes for the data (so y is conditioned on z which is supervised by G ).  The writing overall is fairly clear, though the use of T in the experiments for 5.1 was unclear to me and section 5.2 as a whole could be improved in terms of clarity.  The exact setup for learning the prototypes in the OSR module could have been made a little clearer.\n\n## Weaknesses:\nThe authors need to make a more compelling case of if alignment to factors in this setting really leads to better interpretability because they make lots of assumptions to simplify the problem including \n1) having generative factor supervision in the first place which seems unrealistic and limits its applicability, and\n2) that there is no mixing of factors to create concepts since empirically they only show one to one factor -> concept mappings.  The alignment metric DCI gives higher value for fewer concepts features being used as predictors for generative factors, but even without allowing mixing, its quite possible the same factor could be transformed in many monotonic ways to create many concepts ( which is valid according to your setting ) and the metric in that case would be shown to be artificially low.\nThis latter assumption affects both the alignment and explictness metrics, and the latter is seems largely just a transform of the former after seeing its definition in the appendix and given the simplified setting.\n\nWithout human studies as to the utility of these explanations, its unclear how useful they are in this restricted setting, though I imagine they should be given the data is setup to be explained by the generative factors used. \nMore importantly for human studies of interpretability is how this work applies in general settings ( where generative factors aren’t given ) though this could be outside of the scope of this paper.\n\nOverall its an interesting idea, but the paper could be made stronger by human evaluation of whether alignment in a simplified conservative setting is more “interpretable” compared with other CBMS.  Having an experiment which is less synthetic meaning where the classes aren’t explicitly based on the supervised generative factors given could help show this work would generalize to more real world settings and increase the significance of the findings.  If not, motivation and examples for how and where this setting, where generative factors with one to one mappings to concepts are given, are impactful is very much needed to highlight the significance and applicability of this work.   \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "or_Fk_M7Zs", "reviewer": "Reviewer_mGx1", "summary": "For improving the interpretability of concepts in concept-based models (CBMs), in this paper, the author(s) proposed a model, GlanceNet, to utilize the techniques from disentangled representation learning and open-set recognition for achieving alignment and avoiding concept leakage. As the author(s) claimed that experiments on several datasets showed the advantages of the proposed method.", "questions": "As for the Weaknesses mentioned above, to be specific, I have the following questions/comments which need further explanation from the author(s). Please correct me if I am wrong. Thanks.\n\n1. Could you please provide more details about threshold $\\eta_y$? In Line 214, the author(s) mentioned that \"...choosing the thresholds as to include the 95% of training examples to work well in our experiments.\", what is the effect of different percentage samples? To include the 95% of training examples, for different $y$, did you use different values? Or the same value? Did you choose $\\eta_y$ based on the results on the test set?\n\n2. Figure 3, did different data partitions (using different random seeds) have different effects on the results? In Checklist, for Question 3 (c), the author(s) answered \"the error bars for CelebA was reported...\".  For CelebA, taking the 100% sample as an example, the maximum fluctuation reached about 0.1. For MPI3D, the difference between curves is obvious in visual effect, but the actual difference was also about 0.1. For the calculation on MPI3D, is the average based on multiple data partitions? Or is it just one partition? If the latter, is it possible that the method in this paper is not so superior?\n\n---\n---\n\nIn addition, some other tiny issues/typos:\n\n(1) Please give the full name of DCI when it first appears;\n\n(2) When using mathematical equations, sometimes there are punctuation marks and sometimes there are not. Please use a consistent format;\n\n(3) Please check and use \"em dash\" carefully in the paper;\n\n(4) The numbers on the vertical axis in Figure 4 (c) are too small;\n\n(5) What does $\\mathbf{s(x)}$ mean in Eq.(1)? What is difference of  $\\mathbf{Z}_J$ and $\\mathbf{z}_J$ in lines 141-142;\n\n(6) There are many formatting inconsistencies in the cited references. Please check and correct them carefully.\n\n\n\n\n\n\n", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is generally well written, I have the following comments:\n\n### Strengths\n---\n\n1. Defining interpretability in terms of alignment between the model’s representation and an underlying data generation process;\n\n2. Avoiding concept leakage by utilizing the viewpoint from OOD;\n\n3. Empirical experiments were performed to show the superiority of the proposed method.\n\n---\n---\n\nAlthough the discussion and analysis in this paper are interesting, I think this paper has the following deficiencies or unclear points (Please see the \"Questions\" section below for details).\n\n### Weaknesses\n---\n\n1. Some experimental designs are not very clear;\n\n2. Some experimental results need further clarification.\n\n---\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "n9qkAvuQYvM", "reviewer": "Reviewer_Xfpf", "summary": "This paper introduces GlanceNets, a novel family of concept bottleneck models that leverages disentanglement learning and open-set recognition to avoid information leakage and generate meaningful concept-based explanations for predictions. The authors motivate GlanceNet by first providing an explicit definition of interpretability in terms of concept alignment to interpretable data generative factors, establishing a connection between disengagement learning and concept learning. Then, they use this definition to define an autoencoder-based architecture that, through full or weak concept supervision, learns to extract a latent code that aligns with interpretable factors of variation (i.e., concepts) and uses this code for predicting a downstream label. With three sets of experiments, this paper shows that GlanceNets are able to maintain high task accuracy while avoiding unnecessary leakage in their learned concept encodings.", "questions": "After reading the paper a few times, I am still unclear about the following items: \n\n1. One thing that is particularly missing in the experiments is an evaluation of how well GlanceNets can predict concepts (i.e., the accuracy of using $p(g_i | x) = \\sigma(z_i)$ to predict the $i$-th interpretable generative factor). How does this accuracy compare to that of a vanilla concept bottleneck model? The alignment may implicitly capture this in some way, however, in practice, this is an important metric as it determines how accurate its explanations are.\n2. In the context of experiment 5.1, it is unclear what entails to train a vanilla Concept Bottleneck Model with only a percentage of its samples with concept annotations. Given that these models require full-concept supervision, can you please clarify what was the procedure used to train them if only a portion of its training data has concept annotations? Did you ignore the concept-prediction loss term for samples that do not have annotations in them or did you simply reduce the training set size to be that of samples that have annotations?\n3. How are the concept bottleneck models used in the paper's evaluation trained? Are they trained jointly, sequentially, or independently? I might’ve missed something but I was unable to find an answer to this in the paper and its supplementary material.\n4. My biggest concern with the evaluation of this method is that some important details of the setup for the experiments in section 5.2 are a bit unclear. For example, when training the different baselines in the MNIST task, it is unclear what the downstream task (i.e., $y$) is. From the supplementary material, it is my understanding that parity prediction is not the downstream task used to learn the concept latent codes for both CBNM and GlanceNet. Is the downstream task therefore just predicting the image’s number? This should be clarified as it is a key component in understanding what sort of supervision these encodings get when being learned. Similarly, for the dSprites experiment, what is the downstream task used to train the different baselines? From Figure 5. (a), it is unclear how samples in the square corresponding to the training set are labeled, and thus it is unclear what sort of training supervision the concept encodings get during training. Clarifying these things would certainly improve this evaluation substantially.\n\nIn terms of suggestions for the presentation, the paper is extremely well-written and clear (thanks for this!). Nevertheless, the following typos may be worth addressing for a final version:\n\n1. There is a typo in the title: “Interpretabile” should be “Interpretable”\n2. Traditionally, the acronym “CBM” is used for “Concept Bottleneck Model” rather than “Concept-based Model”. To avoid confusion, I would suggest following this convention and using CBM for “Concept Bottleneck Model” (rather then “CBNM”) and something else for “Concept-based Model” (e.g., “CM” or “CLM” for “Concept learning model” as in Mahinpei et al.).\n3. Line 193: the citation for [37] is a bit awkwardly placed.\n4. Figure 3: it would be helpful if the axes are labeled in these plots.\n5. Line 285: “Eastwood and Williams [29]” seems to be misplaced. Maybe it is intended to be just “[29]”?\n6. Line 309: “trained” should be “training”.\n7. Figure 4:  it would be helpful if the axes are labeled in these plots.\n8. Line 330: “Fig. 4 (a)” should probably be “Fig. 5 (a)”.", "limitations": "The authors do a good just at discussing some limitations of their work (e.g., how their work assumes that all interpretable generative factors are independent of each other). As for potential negative societal impacts, I was unable to find a discussion on this topic in the paper or in the checklist. Nevertheless, I do not see any immediate potential negative societal impacts from this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Thank you for the very insightful and interesting work presented in this paper. After carefully reading your paper and its supplementary material I believe that its main strengths are:\n\n1. The method is novel and it addresses an important gap in the literature by being, to the best of my knowledge, the first work to design an architecture that explicitly addresses leakage in CBMs. \n2. This work provides a clear and insightful link between disentanglement learning and concept learning.\n3. The evaluation of the proposed methods is extensive and well-motivated.\n4. The work is clearly placed within its literature and the paper offers an extensive amount of references and resources for the interested reader.\n5. The paper is very well-written and its presentation is very clear.\n\nI found the following things to be potential weaknesses of this paper:\n\n1. As opposed to conventional practice, error bars are not included in all results (e.g., dSprites and MPI3D plots in Figure 3).\n2. A few crucial design elements of the experiments are unclear (see below).\n3. The code provided as part of the Supplementary material is lacking clear instructions and/or annotations for one to know how to follow it.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bnIV17ae5vh", "reviewer": "Reviewer_SNnk", "summary": "This paper present GlanceNets as a new approach for learning interpretable DNNs whose presentations maps to a set of human aligned concepts. The primary improvement that comes with using GlanceNets is that they do not suffer from concept leakage, which is a challenge for predecessors that tried to address this problem. The GlanceNet is a VAE trained to represent 1) interpretable concepts, 2) Non-interpretable concepts, and 3) an out-of-distribution head to reject samples at test-time. Empirical assessment of the GlanceNets shows that they are not susceptible to leakage and competitive on accuracy.", "questions": "The discussion around the questions I have is provided in the weakness bullet list in the previous section. Here I give a shortened version of that discussion:\n\n- Can you give an example, in terms of the 4/5 classification problem, how the concept leakage problem is a domain shift one? Specifically, what is $\\mathbf{G_V}$ in this problem? Is it the 4 and 5 concept? If yes, what is $\\mathbf{G_F}$? Is it any other concept that can be used to classifier examples that are not 4/5? ", "limitations": "No explicit discussion of the limitations of this work in the draft. The authors can add this towards the end of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- **Tackles an important problem**: This paper addresses the important concept leakage problem, and demonstrates convincingly that the the proposed architecture addresses the problem. In addition that paper shows that the model is competitive on accuracy (figure 3), while achieving better alignment. Section 5.2 critically demonstrates that Glancenets do not suffer from concept leakage.\n\n- **Intuitive Rethinking of Interpretability as alignment**: One of the key contributions of this work is its focus on the alignment problem. Specifically, the paper seeks to make sure that the concepts relied on by a classifier are 'aligned' with the concept representations provided by a domain expert. Section 3 provides an interesting discussion on this front, which could likely be adapted by future work in this area.\n\n## Weaknesses\n- **The Interpretability as Concept leakage paragraph**: This is the key paragraph that justifies the proposed architecture in my opinion, but I don't follow the logic here. Specifically, why is the issue a domain shift problem? Couldn't it still be possible to train a CBNM where there is not domain shift but there is still concept leakage? Perhaps the authors can use the 4/5 classification example as an example here. \n\n- **It seems like the solution here punts the problem**: As far as I am aware, following the 4/5 example, representations learned from the 4/5 classification are effective for classification other instances. However, to tackle this issue this paper suggests adding a component that rejects any input that is not a 4/5, but it is unclear as whether there is still concept leakage in the representations. Specifically, my question is whether there is still leakage if you don't reject out of domain examples? I say this because it seems to me that ood identification and the leakage/alignment problem here are somewhat orthogonal. Given the definition of alignment that this paper provides, it would've been great if the paper showed that optimizing for that objective leads to a more aligned representation for glancenets.\n\n- **Notation**: This paper has a ton of notation that makes it somewhat difficult to digest. Specifically, $\\mathbf{G_I}$, $\\mathbf{g_I}$, and $g_I$ somewhat interchangeably. It'll help to unify this notation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2PsCxUkxeOs", "reviewer": "Reviewer_31JG", "summary": "The paper defines a model that the authors claim to be a concept based model that solves two key issues in the domain of concept based models and interpretable models more widely. The first is that the concepts are demonstrably interpretable. Here the authors present a (rather neat) definition of interpretability in terms of alignment, where:\n> learned concepts are interpretable if they can be mapped to a (partially) interpretable data generation process using a transformation that preserves semantics.\n\nThis is a powerful notion and the mathematical arguments around sufficient conditions for alignment. Their model,training process and evaluation metrics are then informed by these. There are some additional arguments relating to disentanglement, but these are less clear to me.\n\nThe paper presents a model that combined a disentangled (beta-)VAE model with an additional concept decoder for a subset of supervised examples on the (otherwise) hidden factors. The auxiliary loss is described as an average cross entropy loss where (for reasons that aren't entirely clear to me):\n> the annotations g k are rescaled to lie in [0, 1]\n\nThe paper also presents an argument about concept leakage. The definition of this is a little hazy but at the  first presentation it is described as:\n> whereby the learned concepts end up encoding spurious information about unrelated aspects of the\ndata, making it hard to assign them clear semantics.\n\nTheir model addresses concept leakage by modelling the distribution of learned interpretable factors and rejecting predictions at test time that are outliers for this distribution. In the evaluations later in the paper, these rejections are implemented by making random predictions. However this is more to meet their evaluation requirements than a suggestion of what to do with outlier predictions.\n\nThe paper then presents a series of evaluations comparing the proposed model (GlanceNets) and a pre-existing concept model (CBNM). These evaluations involve 3 datasets, two pre-existing synthetic benchmarks for disentangled and related models: dSprits and MPI3D, and an image dataset of celebrities (CelebA), labelled with \"interpretable features\" by crowdsourced workers. The two datasets MPI3D and Celeb were then given labels based on clustering the \"interpretable factors\".\n\nPerformance is measured by three measures, accuracy, alignment (as defined by the authors) and explictness (predictive performance on hidden/interpretable features). The proposed model outperforms (or is competitive with) CBMN on all three metrics on all three datasets. In particular, GlanceNets outperforms significantly on the alignment metric for the synthetic datasets.\n", "questions": "The leak proof experiments are a little unclear. As I read it, the representations are trained on 4s and 5s from the MNIST dataset. Then, I am guessing, a linear classifier is trained on fixed embeddings for all digits 0-9, where the target label is binary (odds/evens). If there is consistent information in the representation to allow the prediction to be made then it will be above 0.5. Is that right?\n\nI have also tried to outline my understanding in the strengths and weaknesses section. In terms of these interpretations have I misunderstood anything?", "limitations": "There could be a little more on the limitations of the approach and the limitations of the evaluations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a novel and, in my opinion, convincing argument that interpretability can be defined in terms of *alignment* with interpretable features. The subsequent mathematical arguments relating to alignment are, mostly, very clear and well argued. Equally the argument that the existing DCI metric (indirectly) measures alignment.\n\nThe arguments about concept leakage are less clear, and would really benefit from a much clearer, upfront, definition of concept leakage alongside an example. There are also other issues of clarity, including on disentanglement and some of the descriptions of their model/evaluations (see below).\n\nAlong with the issues with clarity I have some concerns about the strength of the evaluations. The results do show that the GlanceNet model improves alignment with the features/factors that are said to be interpretable, but the datasets chosen are either entirely synthetic (dSprites and MPI3D), and/or have labels that are constructed from some synthetic construction (MPI3D and CelebA). As such there is a danger that these results are confirming the authors preconceptions to some degree. At this level, I would expect to see at least one dataset where there hasn't been intervention in the data generation process. I realise the challenge that this presents researchers in this (and related fields) but it is a principle I feel is important for the advancement of the field. For instance, some kind of human study that evaluates the interpretability of learn features would be the gold standard in my opinion. For an excellent example see (Chang et al., 2009) - details below.\n\nAnother minor issue I have is with the fairness of the experiment showing concept leakage. Really, the authors are making a claim that out of sample test predictions should be rejected if the interpretable/latent factors fall outside of the centre of mass of points seen at training time. This is an excellent observation and not one that I intend to dispute. Nontheless, the evaluation then sets up their method with a (quasi-)rejection option based on this principle and compare it to methods that do not have such a reject mechanism. I don't know what a fair evaluation would look like here, but this doesn't meet those requirements in my opinion. It also feels a little contrived that the reject option is to make a random prediction. Were these random predictions included in the earlier experiments too?\n\nSo, in summary, I like this paper, and I am convinced by the main causal argument, but I think that there are some outstanding issues that mean I cannot recommend it for acceptance as it stands.\n\n\nI have a few additional comments about details in the paper that came up as I was reading. These are as follows:\n\n\nThe authors claim about concept leakage that:\n> This phenomenon, illustrated in Fig. 1 is known as concept leakage.\n\nHowever, the image seems more to demonstrate how their model avoids concept leakage rather than the phenomenon itself. The authors then say (about the literature) that:\n> a clear definition of leakage is missing\n\nThen about their paper that:\n> A key contribution of our paper is showing that leakage can\nbe understood from the perspective of domain shift and dealt with using open-set recognition\n\nBut this feels like a bit of a non-sequitur. A definition should be present and unambiguously described before it can be understood from different perspectives. In the absence of a definition then a better intuitive description would help, with some clear examples.\n\nThere is a similar issue with the definition of disentanglement. I realise that this is one of the key issues in the disentanglement literature, that there are disputed terms, but, perhaps because of this, it is important that a working definition is given at the very least. The paper later associates semantically preserving maps with something *stronger than disentanglement* but do not give a clear definition of tihs either.\n\nOn line 133 you describe what appears to be a row vector left multiplied by a matrix. Elsewhere it could be clearer where vectors are defined that they are treated as column vectors.\n\nThis sentence (line 308-310) doesn't seem to parse very clearly:\n> the goal is to discriminate between even and odd MNIST images using a latent representation\nZ = (Z 4 , Z 5 ) obtained by trained (with complete supervision on the generative factors) only on\nexamples of 4’s and 5’s.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept