Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackles the trustworthiness of concept bottleneck models (CBM) by improving the accuracy-interpretability tradeoff using a concept-based architecture (Concept Embedding Model or CEM) which represents each concept as a supervised vector. Furthermore, the authors propose two metrics for evaluating concept representations.
strengths_and_weaknesses: Strengths:
1) The paper tackles an important problem of trustworthiness and accuracy-interpretability tradeoff.
2) The paper is well-organized and easy to follow.
3) The authors performed multiple experiments demonstrating that the proposed approach would work reasonably well in different scenarios and improves the accuracy-interpretability.
4) Scalable to real-world cases with lacking complete concept supervision.

Weaknesses:
1)  Lack of necessary statistical analysis. Some improvements do not seem to be significant. It would be better to provide task and concept accuracy as well as statistical significance, e.g., error bars, in a Table. 
2) Does RandInt regularizer increase the training time and cost? I think providing and comparison of the training cost (or model sizes) can also be helpful.
3) Have you tried applying RandInt regularizer to Bool-CBM, Fuzzy-CBM, or other baselines? Based on the provided results in Figure 6, it seems that a lot of improvements are due to adding the RandInt regularizer. For a fair comparison, I would like to see if adding that to the existing methods can improve their performance as well. 

questions: See weaknesses.
limitations: The authors discussed the limitations of the proposed method in Section 6.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper extends the concept bottleneck method by generating two embedding vectors for each concept; one representing the embedding when the concept is active while the other representing when the concept is inactive. These two embedding are linearly combined through a scoring function (similar to the gating mechanism) the produces a probability score of which embedding is to be used. The concept embedding vectors are then used in the downstream task for prediction. This extension increase the model capacity to encode more information about concepts. 

strengths_and_weaknesses: Strength
- easy to read 
- simple extension 

Weakness
- constraining the architecture of all other baselines as similar to the architecture of the proposed method seems to be unfair. 
- I still can not see how this model can be used in scenarios where incomplete concepts exist. In particular, in eq 1, how can the model be trained without a full supervision of concepts?
- the information bottleneck metric in section 4 is a bit unclear. more detailed explanation would be preferred. 
questions: Read my listed points in the weakness section. 
limitations: No.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Concept bottleneck models implicitly learn to explain the downstream tasks in addition to learning how to perform them. However, these models forgo predictive performance on the downstream tasks. The authors propose Concept Embedding Models (CEMs), a novel family of concept bottleneck models that address this issue.
strengths_and_weaknesses: Strengths:
- The proposed techniques perform better than CBMs and their variants on downstream tasks.
- Concept representation learned by CEMs is more aligned to the ground truth concepts and successfully captures the semantics of the images.
Weaknesses:
- CEMs assume that the datasets contain annotations of concepts which is not valid in practice and are often quite expensive to obtain.
questions: - How robust are the concepts learned? Are they robust to changes in backgrounds?
- Is it possible to relax assumptions that datasets should contain concept annotations? Maybe train the model with a contrastive loss for concepts instead of a supervised loss.
- How does the underlying architecture of the model change the concepts generated?
- How much is the additional computational overhead incurred while training CEMs as opposed to CBMs?

I would encourage the authors to include results on OAI & AwA2 too.
limitations: I would encourage the authors to list the limitations of the proposed approach.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper introduces a model named concept embedding model (CEM) based on concept bottleneck model (CBM) architecture. Compared to CBM, CEM contains an embedding generator layer that considers two embedding representations (one for activate and one for inactivate) and then produces an embedding representation for one concept. Results show that the model produces high task accuracy and interpretability at the same time compared to CBM-family models. 
strengths_and_weaknesses: Strength: This paper tries to solve a challenging research question: to design XAI models which are good at task performance and interpretability. The authors conduct experiments on multiple datasets and evaluate different models using different metrics. 

Weakness: 
(1) The proposed model is not thoroughly studied. For instance, the relationship between c_hat+ and c_hat−. This is the novelty of the proposed model compared to CBM. For instance, for one concept, do these two embeddings represent opposite concepts? 

(2) There is some unclearness about the baseline models in the paper. For example, the CEM uses m=16 to represent one concept in c_hat (bottleneck). What is the dimension of c_hat for CBMs? In Appendix A.5, it says “γ = k · (m − 1)” for Hybrid-CBM. Does it mean that the dimension of c_hat is k · (m − 1) dimension? Why not k*m as in CEM?

(3) The lack of justification for the proposed CAS. In Fig.3, the baseline model “no concepts” has an even better score than Boolean-CBM and Fuzzy-CBM. 

(4) The qualitative results are not very convincing. Fig.5 c and Appendix Fig. 5 show the samples and their nearest neighbors for one concept. However, it does not reflex information about the concept and for a well-trained classifier, it should find out the visually similar samples based on Euclidean distance in the embedding space.

questions: (1)	Line 163-164, p should be rho? 

(2)	Sec 5.4, line 312-314 suggests that CEM is robust to incorrect concepts in intervention. But is it desirable in the practice? Do authors consider the scenario where changing the important concept that can change the final decision?  If so, how it can truly reflex the user trust?

limitations: This paper does not emphasize the advantage of using two separate concept representations (c_hat+ and c_hat-), which is the novelty of the CEM, and does not evaluate the interpretability and user trust thoroughly.
review_title: null

Meta-review: metareview: This paper proposes Concept Embedding Models, which learn interpretable high-dimensional concept representations to exploit the tradeoff between accuracy, interpretability, and interventions on concepts. Reviewers vote for accepting this paper. The authors are encouraged to further improve this work based on reviewers’ comments in the camera ready and put the new experiments and discussions during the author-reviewer discussion phrase into the final revision, in particular the following:

- Add statistical significance test of experimental results
- Compare training costs and model sizes
- Better justify the proposed CAS mechanism
- Investigate the robustness of learned concepts
- Address the fairness concerns raised by reviewers in comparison with baselines

confidence: Certain
Decision: Accept