Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a new version of transformer called concept transformer with the aim of improving the interpretability of its attention by computing cross attention between the inputs features and a set of concepts. This should make the model both more explainable, in the sense that it is easier for the human to interpret the attention weights of meaningful concepts, and also more faithful, in the sense that the attention scores given to particular concepts directly impact the final prediction of the model.

Update: Following the author response I'd like to keep my positive score reflecting my view of the paper.
main_review: I think the paper explores an important problem and propose an interesting model that goes in a good direction in terms of encouraging models to not just achieve stronger numerical performance but also perform explainable reasoning along the way to justify their predictions. 
* **Writing quality**: The writing clarity of the paper is good, the idea and content are easy to follow, the motivation is clearly explained, and there are good visualizations that help understanding. 
* **Related work**: the paper misses some closely related prior works that explored the idea of reasoning over concepts for reasoning over images: one paper is “The consciousness prior” by Bengio and the other is “Learning by Abstraction: the Neural State Machine” by Hudson and Manning. The former discussed the idea mainly in the conceptual level and the latter studied it also empirically. At the same time, overall the related work section gives a good coverage of prior works about interpretability and attention.
* **Model structure**: the model proposed is simple yet interesting and general and so can be easily incorporated into transformer-based models for variety tasks and in different domains.
* **Attention supervision**: I really like the idea of supervising the concept attention directly when such information exists, as in the case of CUB. I think that’s a nice advantage of the proposed approach.
* **Experiments**: The experimental section is overall extensive enough and provides good balance both over multiple datasets, some with extra supervision info (like CUBs) and some more diagnostic (like MNIST), experiments include both quantitative and qualitative results. To demonstrate the potential generality of the approach, it could be nice to explore over different domains (e.g. textual only) or tasks (e.g. VQA, or other multimodal tasks) but I feel in the context of the proposed model exploring just image classification is nevertheless alright and not too limited.
* **Concepts identification**: one potential limitation of the approach proposed is the fact that the concepts needed to be pre-defined rather than emergent through the model, which may conversely reduce the applicability or potential benefit of the approach in all cases where such info doesn’t exist. 

summary_of_the_review: Overall, I think the paper proposed a nice interesting model on an important problem, motivates the goal of the model in a compelling manner, presents the model with good detail and clarity, provides a nice mix of experiments of different types and on different datasets, and can be of benefit to the research community, and therefore I recommend its acceptance and wish best of luck to the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a transformer-based model which can enhance explainability of a deep learning model by inducing domain knowledge into the model in a form of cross-attention mechanism. This paper interestingly focuses on addressing the relationship between post-hoc  explainability and inherent  interpretable model.  The authors address the limitation of interpretable model, which focuses on the controversy over how much a human should trust model’s explanations for its decisions.

main_review: Quality and clarity: 
The quality of the paper is good and it has a good motivation. The targeted goal is interesting and meaningful and the authors achieved this by linearly enforcing domain knowledge to weight allocations of attention mechanism. The proposed methodology is well-described over the paper and introduced experimental settings are properly designed with three public datasets. The written
results seem to be reasonable.

Originality:
Although interesting, It’s a bit hard to say the introduced technique is quite new, since a simple linear combination of attention weights and concepts patches (domain knowledge) in Transformer is the main contribution. Also, the experimental results are relatively weak when comparing the main model with its baselines in accuracy performance.


Qualitative and quantitative analysis of model explainability between the proposed model and its baselines can be a good validation to show the benefit of their explanation technique. The authors also need to describe how concept patches are generated and the error bound generated from the concept patches should be considered for clarity.
summary_of_the_review: Although the paper is interesting, it's considered that the proposed technique is somewhat incremental.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work presents a new, Transformer-like architecture that has explainability built into it in the form of concepts which are then cross-attended against the input. The authors present their work in a variety of domains and show how it aids model decision interpretability. 
main_review: This paper presents a fairly straightforward yet novel modification to a vanilla Transformer that introduces concepts, which the user has to provide during training time. These concepts are then attended to via cross-attention against queries, which as usual come from the input. This allows an auxiliary, explanation loss to be used, which the authors show to be useful to increase classification accuracy. For most datasets presented, the authors' architecture is relatively competitive but does not achieve state of the art results, as I would expect from a model that baked in explainability. One cool bit is that classification accuracy does not necessarily fall with the use of the auxiliary loss (table 2), as one would expect it to.

A weakness of the work is that the concepts provide an explanation of perhaps why the model is wrong but what I expected was the ability to do some sort of fix of the model's attention so that it can be forced to output the right answer. As it is, I am not sure I am convinced that the concepts are causally linked to classification / misclassification performance. The authors provide a few examples where a misclassification happens and they show how the model attended to the wrong concepts, but is it possible this is a coincidence? The authors do not investigate in detail. It would be nice to see if the authors can change the attention weights in a natural way to explore this, such as for example altering the input image (ie, changing the color of certain pixels) so that certain concepts are / are no longer attended to and see how this affects classification performance.
summary_of_the_review: A reasonable first draft of a promising line of work but more analysis is needed to make it interesting.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a transformer-based architecture for (deep) concept models. Similar to previous approaches in this area, the authors propose a replacement of the model’s deep classifier head. Here, the authors suggest using the attention mechanism. With the introduced ConceptTransformer module they are able to utilise global as well as local (spatial) concepts.
After introducing the architecture and supervised concept learning of the model, the model’s performance is demonstrated on three benchmark datasets. Where only one makes use of local and global concepts.
main_review: This work investigates the important field of interpretable deep models. With their technical contribution, the authors present an interesting approach to extend previous approaches, such as Bottleneck Concept Models (Koh et al.), by explicit local and global concepts. 
The proposed approach seems to be very clearly described. Figure 1 and the illustrative example on MNIST describe the approach really well.

Furthermore, distinguishing between local and global concepts by the design of the architectures is a very interesting approach. Unfortunately, the paper only rudimentary focuses on this contribution. In this regard, the findings of the present work don’t show novel insights e.g. compared to Concept Bottleneck Models by Koh et al (ICML 2020) or the neuro-symbolic, object-centric approach by Stammer et al (CVPR 2021). 

In particular, the takeaway message of Table 1 is not clear to me. In the main text, the authors mention that resulting explanations from ProtoPNet "cannot be always guaranteed to be faithful to the classifier’s decision", whereas the authors claim that the ConceptTransformer guarantees this, which is not clear to me. Are you only referring to ProtoPNet or Concept Bottleneck models in general? Can you elaborate on this further? Applying the evaluation of Margeloiu et al. (ICLR Workshop 2021) would further strengthen the authors’ claims.

The intention of the final evaluation on aPY is also not clear to me. Especially since here, only global concepts are used. What is the benefit compared to a more „simple“ classifier (c.f Concept Bottleneck Models by Koh et al.)?



References: 

Koh et al. Concept Bottleneck Models, ICML 2020 (https://arxiv.org/abs/2007.04612)

Stammer et al. Right for the Right Concept: Revising Neuro-Symbolic Concepts by Interacting with their Explanations, CVPR 2021 (https://arxiv.org/abs/2011.12854)

Margeloiu et al. Do Concept Bottleneck Models Learn as Intended? ICLR Workshop 2021 (https://arxiv.org/abs/2105.04289)
summary_of_the_review: As it is now presented the paper seems only to be a minor technical extension of Concept Bottleneck Models and I would recommend that the authors focus more on presenting as well as evaluating the benefit of their ConceptTransformer. 
The authors state that the explanations by their introduced approach are plausible and faithful. However, no quantitive evaluation in this regard is performed (cf. Margeloiu et al. Do Concept Bottleneck Models Learn as Intended?). The introduced ConceptTransformers seems to be a technical extension to „simple“ linear classification heads of Concept Bottleneck Models. The advantage is not clearly described and evaluated. 
Therefore, the paper seems to me to be only a minor technical contribution.
review_title: null

Meta-review: This paper proposes a generalization of the standard Transformer attention mechanism in which keys and queries represent abstract concepts (which must be specified a priori). This in turn yields "concept embeddings" (and logits) as intermediate network outputs, providing a sort of interpretability. Reviewers agreed that this is a simple (in a good way) and interesting approach, and may lead to follow-up work that builds on this architecture. 

Some concerns regarding the relation of this method to prior work—in particular the "Concept Bottleneck" model—were raised and addressed in discussion; the authors might incorporate additional discussion in future drafts of the work.
Decision: Accept (Poster)