Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper present GlanceNets as a new approach for learning interpretable DNNs whose presentations maps to a set of human aligned concepts. The primary improvement that comes with using GlanceNets is that they do not suffer from concept leakage, which is a challenge for predecessors that tried to address this problem. The GlanceNet is a VAE trained to represent 1) interpretable concepts, 2) Non-interpretable concepts, and 3) an out-of-distribution head to reject samples at test-time. Empirical assessment of the GlanceNets shows that they are not susceptible to leakage and competitive on accuracy.
strengths_and_weaknesses: ## Strengths

- **Tackles an important problem**: This paper addresses the important concept leakage problem, and demonstrates convincingly that the the proposed architecture addresses the problem. In addition that paper shows that the model is competitive on accuracy (figure 3), while achieving better alignment. Section 5.2 critically demonstrates that Glancenets do not suffer from concept leakage.

- **Intuitive Rethinking of Interpretability as alignment**: One of the key contributions of this work is its focus on the alignment problem. Specifically, the paper seeks to make sure that the concepts relied on by a classifier are 'aligned' with the concept representations provided by a domain expert. Section 3 provides an interesting discussion on this front, which could likely be adapted by future work in this area.

## Weaknesses
- **The Interpretability as Concept leakage paragraph**: This is the key paragraph that justifies the proposed architecture in my opinion, but I don't follow the logic here. Specifically, why is the issue a domain shift problem? Couldn't it still be possible to train a CBNM where there is not domain shift but there is still concept leakage? Perhaps the authors can use the 4/5 classification example as an example here. 

- **It seems like the solution here punts the problem**: As far as I am aware, following the 4/5 example, representations learned from the 4/5 classification are effective for classification other instances. However, to tackle this issue this paper suggests adding a component that rejects any input that is not a 4/5, but it is unclear as whether there is still concept leakage in the representations. Specifically, my question is whether there is still leakage if you don't reject out of domain examples? I say this because it seems to me that ood identification and the leakage/alignment problem here are somewhat orthogonal. Given the definition of alignment that this paper provides, it would've been great if the paper showed that optimizing for that objective leads to a more aligned representation for glancenets.

- **Notation**: This paper has a ton of notation that makes it somewhat difficult to digest. Specifically, $\mathbf{G_I}$, $\mathbf{g_I}$, and $g_I$ somewhat interchangeably. It'll help to unify this notation.
questions: The discussion around the questions I have is provided in the weakness bullet list in the previous section. Here I give a shortened version of that discussion:

- Can you give an example, in terms of the 4/5 classification problem, how the concept leakage problem is a domain shift one? Specifically, what is $\mathbf{G_V}$ in this problem? Is it the 4 and 5 concept? If yes, what is $\mathbf{G_F}$? Is it any other concept that can be used to classifier examples that are not 4/5? 
limitations: No explicit discussion of the limitations of this work in the draft. The authors can add this towards the end of the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper defines a model that the authors claim to be a concept based model that solves two key issues in the domain of concept based models and interpretable models more widely. The first is that the concepts are demonstrably interpretable. Here the authors present a (rather neat) definition of interpretability in terms of alignment, where:
> learned concepts are interpretable if they can be mapped to a (partially) interpretable data generation process using a transformation that preserves semantics.

This is a powerful notion and the mathematical arguments around sufficient conditions for alignment. Their model,training process and evaluation metrics are then informed by these. There are some additional arguments relating to disentanglement, but these are less clear to me.

The paper presents a model that combined a disentangled (beta-)VAE model with an additional concept decoder for a subset of supervised examples on the (otherwise) hidden factors. The auxiliary loss is described as an average cross entropy loss where (for reasons that aren't entirely clear to me):
> the annotations g k are rescaled to lie in [0, 1]

The paper also presents an argument about concept leakage. The definition of this is a little hazy but at the  first presentation it is described as:
> whereby the learned concepts end up encoding spurious information about unrelated aspects of the
data, making it hard to assign them clear semantics.

Their model addresses concept leakage by modelling the distribution of learned interpretable factors and rejecting predictions at test time that are outliers for this distribution. In the evaluations later in the paper, these rejections are implemented by making random predictions. However this is more to meet their evaluation requirements than a suggestion of what to do with outlier predictions.

The paper then presents a series of evaluations comparing the proposed model (GlanceNets) and a pre-existing concept model (CBNM). These evaluations involve 3 datasets, two pre-existing synthetic benchmarks for disentangled and related models: dSprits and MPI3D, and an image dataset of celebrities (CelebA), labelled with "interpretable features" by crowdsourced workers. The two datasets MPI3D and Celeb were then given labels based on clustering the "interpretable factors".

Performance is measured by three measures, accuracy, alignment (as defined by the authors) and explictness (predictive performance on hidden/interpretable features). The proposed model outperforms (or is competitive with) CBMN on all three metrics on all three datasets. In particular, GlanceNets outperforms significantly on the alignment metric for the synthetic datasets.

strengths_and_weaknesses: The paper presents a novel and, in my opinion, convincing argument that interpretability can be defined in terms of *alignment* with interpretable features. The subsequent mathematical arguments relating to alignment are, mostly, very clear and well argued. Equally the argument that the existing DCI metric (indirectly) measures alignment.

The arguments about concept leakage are less clear, and would really benefit from a much clearer, upfront, definition of concept leakage alongside an example. There are also other issues of clarity, including on disentanglement and some of the descriptions of their model/evaluations (see below).

Along with the issues with clarity I have some concerns about the strength of the evaluations. The results do show that the GlanceNet model improves alignment with the features/factors that are said to be interpretable, but the datasets chosen are either entirely synthetic (dSprites and MPI3D), and/or have labels that are constructed from some synthetic construction (MPI3D and CelebA). As such there is a danger that these results are confirming the authors preconceptions to some degree. At this level, I would expect to see at least one dataset where there hasn't been intervention in the data generation process. I realise the challenge that this presents researchers in this (and related fields) but it is a principle I feel is important for the advancement of the field. For instance, some kind of human study that evaluates the interpretability of learn features would be the gold standard in my opinion. For an excellent example see (Chang et al., 2009) - details below.

Another minor issue I have is with the fairness of the experiment showing concept leakage. Really, the authors are making a claim that out of sample test predictions should be rejected if the interpretable/latent factors fall outside of the centre of mass of points seen at training time. This is an excellent observation and not one that I intend to dispute. Nontheless, the evaluation then sets up their method with a (quasi-)rejection option based on this principle and compare it to methods that do not have such a reject mechanism. I don't know what a fair evaluation would look like here, but this doesn't meet those requirements in my opinion. It also feels a little contrived that the reject option is to make a random prediction. Were these random predictions included in the earlier experiments too?

So, in summary, I like this paper, and I am convinced by the main causal argument, but I think that there are some outstanding issues that mean I cannot recommend it for acceptance as it stands.


I have a few additional comments about details in the paper that came up as I was reading. These are as follows:


The authors claim about concept leakage that:
> This phenomenon, illustrated in Fig. 1 is known as concept leakage.

However, the image seems more to demonstrate how their model avoids concept leakage rather than the phenomenon itself. The authors then say (about the literature) that:
> a clear definition of leakage is missing

Then about their paper that:
> A key contribution of our paper is showing that leakage can
be understood from the perspective of domain shift and dealt with using open-set recognition

But this feels like a bit of a non-sequitur. A definition should be present and unambiguously described before it can be understood from different perspectives. In the absence of a definition then a better intuitive description would help, with some clear examples.

There is a similar issue with the definition of disentanglement. I realise that this is one of the key issues in the disentanglement literature, that there are disputed terms, but, perhaps because of this, it is important that a working definition is given at the very least. The paper later associates semantically preserving maps with something *stronger than disentanglement* but do not give a clear definition of tihs either.

On line 133 you describe what appears to be a row vector left multiplied by a matrix. Elsewhere it could be clearer where vectors are defined that they are treated as column vectors.

This sentence (line 308-310) doesn't seem to parse very clearly:
> the goal is to discriminate between even and odd MNIST images using a latent representation
Z = (Z 4 , Z 5 ) obtained by trained (with complete supervision on the generative factors) only on
examples of 4’s and 5’s.

questions: The leak proof experiments are a little unclear. As I read it, the representations are trained on 4s and 5s from the MNIST dataset. Then, I am guessing, a linear classifier is trained on fixed embeddings for all digits 0-9, where the target label is binary (odds/evens). If there is consistent information in the representation to allow the prediction to be made then it will be above 0.5. Is that right?

I have also tried to outline my understanding in the strengths and weaknesses section. In terms of these interpretations have I misunderstood anything?
limitations: There could be a little more on the limitations of the approach and the limitations of the evaluations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: 
This work introduces a new autoencoder architecture GlanceNet and proposes to advance the interpretability of concept-based models (CBMs) by introducing a definition of interpretability that is based on a supervised aligning of learned latent concepts with the factors of the underlying data generation, some of which maybe semantically interpretable to a user.

The latent space Z induced be the encoder q_theta(Z | X) is divided into two parts, Zj and Zbar_j.  Zj is for learning of k concepts which are used for task level classification p(Y | Zj) and for aligning with supervised generative factors G, which is measured via a DCI metric from the disentanglement literature.  Zbar_j is used by the decoder to reconstruct x_hat which is in turn novelly used by an open set recognition module to learn class prototypes that determine if a test set example follows the training distribution ( T=1 ) or not ( T=0 ) and can be used for rejecting test cases.  Additionally, a prior p(z | y) of concept prototypes is learned as a mixture of gaussians (one component per class) and used to encourage disentangled representation learning via penalizing the KL divergence between it and the latent space, KL( p(z|y) || q( z | x) ).    

Over three image tasks with varying amounts of supervision for generative factors (1 to 100% of examples), and under the assumption that interpretable generative factors are disentangled from each other, the authors show the architecture performs comparably or better in terms of accuracy, and better in terms of the two interpretable metics, alignment and ``explictness” which measures how well the linear regressor employed by DCI fits the generative factors using Zj.  
strengths_and_weaknesses: ## Strengths:
The paper is novel for its proposed architecture, tying the definition of interpretability via the alignment of learned concepts to generative factors, and its utilitization of OSR for use in preventing concept leakage.  It additionally is shown to be quite performative against the concept whitening paper for synthetic cases ( first two experiments dSprites and MPI3D) & the third case CelebA which is semi-synthetic because the supervised factors themselves are used to cluster and create classes for the data (so y is conditioned on z which is supervised by G ).  The writing overall is fairly clear, though the use of T in the experiments for 5.1 was unclear to me and section 5.2 as a whole could be improved in terms of clarity.  The exact setup for learning the prototypes in the OSR module could have been made a little clearer.

## Weaknesses:
The authors need to make a more compelling case of if alignment to factors in this setting really leads to better interpretability because they make lots of assumptions to simplify the problem including 
1) having generative factor supervision in the first place which seems unrealistic and limits its applicability, and
2) that there is no mixing of factors to create concepts since empirically they only show one to one factor -> concept mappings.  The alignment metric DCI gives higher value for fewer concepts features being used as predictors for generative factors, but even without allowing mixing, its quite possible the same factor could be transformed in many monotonic ways to create many concepts ( which is valid according to your setting ) and the metric in that case would be shown to be artificially low.
This latter assumption affects both the alignment and explictness metrics, and the latter is seems largely just a transform of the former after seeing its definition in the appendix and given the simplified setting.

Without human studies as to the utility of these explanations, its unclear how useful they are in this restricted setting, though I imagine they should be given the data is setup to be explained by the generative factors used. 
More importantly for human studies of interpretability is how this work applies in general settings ( where generative factors aren’t given ) though this could be outside of the scope of this paper.

Overall its an interesting idea, but the paper could be made stronger by human evaluation of whether alignment in a simplified conservative setting is more “interpretable” compared with other CBMS.  Having an experiment which is less synthetic meaning where the classes aren’t explicitly based on the supervised generative factors given could help show this work would generalize to more real world settings and increase the significance of the findings.  If not, motivation and examples for how and where this setting, where generative factors with one to one mappings to concepts are given, are impactful is very much needed to highlight the significance and applicability of this work.   

questions: 1) Its unclear what the method does when T predicts a test example is out of domain.  Is it just not used in the results in section 5.1?  
In 5.2 you show OSR helps with concept leakage, by rejecting 85% or almost all cases and then guessing random, which by definition will do get you better results for a binary task, but its not clear how OSR is used in 5.1 ( are samples rejected there too?).  

2) What is the size of Zbar_j in these experiments?  I might be missing something but I didn’t see it in the appendix ?

3) How does the accuracy of your experiments compare with other non-concept based models in general?    
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces GlanceNets, a novel family of concept bottleneck models that leverages disentanglement learning and open-set recognition to avoid information leakage and generate meaningful concept-based explanations for predictions. The authors motivate GlanceNet by first providing an explicit definition of interpretability in terms of concept alignment to interpretable data generative factors, establishing a connection between disengagement learning and concept learning. Then, they use this definition to define an autoencoder-based architecture that, through full or weak concept supervision, learns to extract a latent code that aligns with interpretable factors of variation (i.e., concepts) and uses this code for predicting a downstream label. With three sets of experiments, this paper shows that GlanceNets are able to maintain high task accuracy while avoiding unnecessary leakage in their learned concept encodings.
strengths_and_weaknesses: Thank you for the very insightful and interesting work presented in this paper. After carefully reading your paper and its supplementary material I believe that its main strengths are:

1. The method is novel and it addresses an important gap in the literature by being, to the best of my knowledge, the first work to design an architecture that explicitly addresses leakage in CBMs. 
2. This work provides a clear and insightful link between disentanglement learning and concept learning.
3. The evaluation of the proposed methods is extensive and well-motivated.
4. The work is clearly placed within its literature and the paper offers an extensive amount of references and resources for the interested reader.
5. The paper is very well-written and its presentation is very clear.

I found the following things to be potential weaknesses of this paper:

1. As opposed to conventional practice, error bars are not included in all results (e.g., dSprites and MPI3D plots in Figure 3).
2. A few crucial design elements of the experiments are unclear (see below).
3. The code provided as part of the Supplementary material is lacking clear instructions and/or annotations for one to know how to follow it.
questions: After reading the paper a few times, I am still unclear about the following items: 

1. One thing that is particularly missing in the experiments is an evaluation of how well GlanceNets can predict concepts (i.e., the accuracy of using $p(g_i | x) = \sigma(z_i)$ to predict the $i$-th interpretable generative factor). How does this accuracy compare to that of a vanilla concept bottleneck model? The alignment may implicitly capture this in some way, however, in practice, this is an important metric as it determines how accurate its explanations are.
2. In the context of experiment 5.1, it is unclear what entails to train a vanilla Concept Bottleneck Model with only a percentage of its samples with concept annotations. Given that these models require full-concept supervision, can you please clarify what was the procedure used to train them if only a portion of its training data has concept annotations? Did you ignore the concept-prediction loss term for samples that do not have annotations in them or did you simply reduce the training set size to be that of samples that have annotations?
3. How are the concept bottleneck models used in the paper's evaluation trained? Are they trained jointly, sequentially, or independently? I might’ve missed something but I was unable to find an answer to this in the paper and its supplementary material.
4. My biggest concern with the evaluation of this method is that some important details of the setup for the experiments in section 5.2 are a bit unclear. For example, when training the different baselines in the MNIST task, it is unclear what the downstream task (i.e., $y$) is. From the supplementary material, it is my understanding that parity prediction is not the downstream task used to learn the concept latent codes for both CBNM and GlanceNet. Is the downstream task therefore just predicting the image’s number? This should be clarified as it is a key component in understanding what sort of supervision these encodings get when being learned. Similarly, for the dSprites experiment, what is the downstream task used to train the different baselines? From Figure 5. (a), it is unclear how samples in the square corresponding to the training set are labeled, and thus it is unclear what sort of training supervision the concept encodings get during training. Clarifying these things would certainly improve this evaluation substantially.

In terms of suggestions for the presentation, the paper is extremely well-written and clear (thanks for this!). Nevertheless, the following typos may be worth addressing for a final version:

1. There is a typo in the title: “Interpretabile” should be “Interpretable”
2. Traditionally, the acronym “CBM” is used for “Concept Bottleneck Model” rather than “Concept-based Model”. To avoid confusion, I would suggest following this convention and using CBM for “Concept Bottleneck Model” (rather then “CBNM”) and something else for “Concept-based Model” (e.g., “CM” or “CLM” for “Concept learning model” as in Mahinpei et al.).
3. Line 193: the citation for [37] is a bit awkwardly placed.
4. Figure 3: it would be helpful if the axes are labeled in these plots.
5. Line 285: “Eastwood and Williams [29]” seems to be misplaced. Maybe it is intended to be just “[29]”?
6. Line 309: “trained” should be “training”.
7. Figure 4:  it would be helpful if the axes are labeled in these plots.
8. Line 330: “Fig. 4 (a)” should probably be “Fig. 5 (a)”.
limitations: The authors do a good just at discussing some limitations of their work (e.g., how their work assumes that all interpretable generative factors are independent of each other). As for potential negative societal impacts, I was unable to find a discussion on this topic in the paper or in the checklist. Nevertheless, I do not see any immediate potential negative societal impacts from this work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: For improving the interpretability of concepts in concept-based models (CBMs), in this paper, the author(s) proposed a model, GlanceNet, to utilize the techniques from disentangled representation learning and open-set recognition for achieving alignment and avoiding concept leakage. As the author(s) claimed that experiments on several datasets showed the advantages of the proposed method.
strengths_and_weaknesses: This paper is generally well written, I have the following comments:

### Strengths
---

1. Defining interpretability in terms of alignment between the model’s representation and an underlying data generation process;

2. Avoiding concept leakage by utilizing the viewpoint from OOD;

3. Empirical experiments were performed to show the superiority of the proposed method.

---
---

Although the discussion and analysis in this paper are interesting, I think this paper has the following deficiencies or unclear points (Please see the "Questions" section below for details).

### Weaknesses
---

1. Some experimental designs are not very clear;

2. Some experimental results need further clarification.

---


questions: As for the Weaknesses mentioned above, to be specific, I have the following questions/comments which need further explanation from the author(s). Please correct me if I am wrong. Thanks.

1. Could you please provide more details about threshold $\eta_y$? In Line 214, the author(s) mentioned that "...choosing the thresholds as to include the 95% of training examples to work well in our experiments.", what is the effect of different percentage samples? To include the 95% of training examples, for different $y$, did you use different values? Or the same value? Did you choose $\eta_y$ based on the results on the test set?

2. Figure 3, did different data partitions (using different random seeds) have different effects on the results? In Checklist, for Question 3 (c), the author(s) answered "the error bars for CelebA was reported...".  For CelebA, taking the 100% sample as an example, the maximum fluctuation reached about 0.1. For MPI3D, the difference between curves is obvious in visual effect, but the actual difference was also about 0.1. For the calculation on MPI3D, is the average based on multiple data partitions? Or is it just one partition? If the latter, is it possible that the method in this paper is not so superior?

---
---

In addition, some other tiny issues/typos:

(1) Please give the full name of DCI when it first appears;

(2) When using mathematical equations, sometimes there are punctuation marks and sometimes there are not. Please use a consistent format;

(3) Please check and use "em dash" carefully in the paper;

(4) The numbers on the vertical axis in Figure 4 (c) are too small;

(5) What does $\mathbf{s(x)}$ mean in Eq.(1)? What is difference of  $\mathbf{Z}_J$ and $\mathbf{z}_J$ in lines 141-142;

(6) There are many formatting inconsistencies in the cited references. Please check and correct them carefully.







limitations: Yes.
review_title: null

Meta-review: metareview: I thank the authors for their submission and active participation in the discussions. The paper presents a method for interpretable deep learning. All reviewers unanimously agree that this is a solid paper worthy of acceptance. In particular, reviewers noted that the paper tackles an important problem [SNnk], and presents a novel [jG58,Xfpf], interesting [SNnk,mGx1] and intuitive [31JG] framing of interpretability as alignment problem. Furthermore, the writing is clear [jG58,Xfpf] and the empirical validation extensive [Xfpf]. Thus, I am recommending acceptance of the paper and encourage the authors to further improve their paper based on the reviewer feedback.
confidence: Certain
Decision: Accept