Towards a Deeper Understanding of Concept
Bottleneck Models Through End-to-EndExplanation
Jack Furby1

Daniel Cunnington2
Cardiff University, UK1

Dave Braines2

Alun Preece1

IBM Research Europe2

Introduction

Results

In our paper, we evaluate Concept Bottleneck Models (CBMs) [1], a type of Deep Neural Network that first maps
raw input(s) to a vector of human-defined concepts, before using this vector to predict a final classification. We
analyse what input features the model uses for concept predictions and which predicted concepts contribute tothe final classification.

Relevancy is generally distributed
over the entire bird and the same
input features can predict different
concepts, as shown in figure 1.

Contributions
1.Provide an understanding of the input features the models use from both the input image and predicted concepts.
2.Introduce a quantitative evaluation to measure the distance between the maximum input feature relevance and theground truth location and perform this with multiple relevancy techniques.
3.We propose using the proportion of relevance as a measurement for explaining concept importance.

We believe the dataset is notconfining the model to learn concepts
as intended.

MethodIndependent andSequential

We use Layer-wise Relevance
Propagation (LRP) [2], an
explanation technique whichhighlights features of an input whichwere relevant to a prediction We
generate saliency maps of both therelevance on the input image andconcept prediction vector.
We compare concept saliency maps
using by averaging the distance
between the most salient points andthe ground truth locations.

has_wing_shape::pointed-wings

has_back_color::brown

has_bill_shape::all-purpose

has_breast_pattern::solid0.9973

0.99800.9928

0.99780.99320.99960.8271

0.99180.9691

0.9975

Joint with sigmoid

Our models were trained on amodified version of the CUB-200 2011
(CUB) dataset [1] which used classlevel concepts; concepts are applied
to classes.

Joint without
sigmoid

Input

has_crown_color::brown

0.82850.5296

0.98900.94950.6278Figure 1: In general, relevance does not map to input features that a human would
associate each concept with.

Figure 2 shows the average distanceof saliency to the ground truth
location. The distance is around 100pixels which reinforces the model is
not focusing on concepts as 100
pixels could easily be outside of aconcept. The input images are
299x299 pixels.
In figure 3 we produce saliency maps
to show which concepts the model
uses for the final classification. OurCBMs mostly predict finalclassifications using concepts
predicted as present.
By calculating relevancy distribution
we reveal the concepts used for finalclassification predictions.

Conclusion and Future
WorkOur paper evaluates CBMs using theLRP explanation technique. Thesereveal that concepts do not map todistinct regions in the input space.
However, relevance from the finalclassification back to the concept
vector shows the model has mapped
these as expected for some CBM
training methods.

Concept contribution proportion
calculations are made possible by
LRP's requirement to conserve
relevancy as it’s propagatedbackwards through a model.
Code and additional examples
available at:github.com/JackFurby/explainableconcept-bottleneck-models

We demonstrate the ability to
calculate proportional concept
contribution to final classifications.

Figure 2: The average distance of all explanation techniques are too far from the groundtruth locations for the model to be focusing on the correct input features.

InputFuture work
●CBMs trained on instance-level
concepts and non 1 to 1 mapping ofconcepts to final classifications.
●Evaluating CBMs with a nonrelevance-based method.
●
Studying the effectiveness of CBMsand explanations in a human study.

(a) Independent(b) Sequential(c) Joint without sigmoid(d) Joint with sigmoid
Figure 3: The concept to final classification model part primarily uses concept predicted as present although, for the sequential model and joint without sigmoid model,concepts predicted as present have negative relevance.

References

Acknowledgement[1] Koh, P. W.; Nguyen, T.; Tang, Y. S.; Mussmann, S.; Pierson, E.; Kim, B.; and Liang, P. 2020. Concept Bottleneck Models. In III, H. D.; and Singh, A., eds.,
Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, 5338–5348. PMLR.
[2] Bach, S.; Binder, A.; Montavon, G.; Klauschen, F.; Müller, K.-R.; and Samek, W. 2015. On Pixel-Wise Explanations for Non-Linear Classifier Decisions byLayer-Wise Relevance Propagation. PLOS ONE, 10(7): 1–46.

This research is funded by the UK Engineeringand Physical Sciences Research Council(EPSRC) and IBM UK via an Industrial CASE
(ICASE) award.