Locality-aware Concept Bottleneck Model

Authors: sujin  jeon
inwoo  hwang
sanghack  lee
byoung-tak  zhang
Venue: NeurIPS_cc_2024_Workshop_UniReps
Published: 2024-09-22 15:44:02

Abstract: Concept bottleneck models (CBMs) are inherently interpretable models that make predictions based on human-understandable visual cues, referred to as concepts. As obtaining dense concept annotations with human labeling is demanding and costly, recent approaches utilize foundation models to determine the concepts existing in the image. However, such \textit{label-free} CBMs often fail to attend to 
concepts that are important predictive but only exist in a small region of the image  (e.g., a beak of a bird), making their decision-making less aligned with human reasoning. In this paper, we propose a novel framework, coined Locality-aware CBM (LCBM), which divides an image into smaller patches. Specifically, we use their similarity with concepts to ensure that the concept prediction of the model adheres to the relevant region and effectively captures important local concepts existing in the small region of the image. Experimental results demonstrate that LCBM accurately identifies important concepts from images and exhibits improved localization capability while maintaining high classification performance.

Main paper: 
 introduction: 
 The interpretability of deep neural networks has become an increasingly important issue as the rapid advancements of AI make them closely integrated into our daily lives. This is especially critical in fields where the reliability of models is paramount, e.g., healthcare. Since explaining the decisionmaking process of a black-box model is often challenging, recent works focus on building inherently interpretable models, i.e., whose decision-making is naturally easily understandable to humans.Concept-bottleneck model (CBM) [4, 6, 7, 17, 20] is a representative inherently interpretable architecture where a concept refers to human-recognizable visual cues. CBMs first predict the concepts existing in the image (e.g., "red color" and "round shape" in the image of an apple) and make a final prediction on the class label through the linear combination of these concepts. However, their reliance on dense annotations of which concepts are present in the image limits their scalability and practicality. This has led to the emergence of label-free CBMs [8, 12, 16, 21, 22] , which utilize foundational models such as large language models (LLMs) and vision-language models (VLMs) to determine and estimate the presence of concepts.However, the interpretability of label-free CBMs is often compromised due to their neglect of locality in two key ways. First, they often fail to focus on important local features since they analyze the entire image with VLMs to infer concept presences. This prevents the model from explaining decisions in terms of fine-grained, localized concepts, thereby limiting their interpretability. Second, it is suggested that label-free CBMs fail to properly attend to the corresponding regions in the image when predicting concepts [10, 11, 14] . This discrepancy raises concerns on their interpretability as their concept prediction does not align with the spatial locality of each concepts.To this end, we propose a novel framework, coined Locality-aware CBM (LCBM). To ensure locality, we begin by dividing the image into smaller patches. To make the prediction on each concept properly attends to the corresponding region, we use the CLIP similarity between concepts and image patches to guide the training of our model, as a higher CLIP score for a concept and a patch implies a higher probability that the concept exists in the corresponding local region in the image. We experimentally demonstrated that LCBM can effectively predict concepts from small regions while also focusing on the corresponding region during the prediction process. 
related works: 
 Concept Bottleneck Models (CBMs) The concept bottleneck model (CBM) [7] is an inherently interpretable framework that explains its predictions through the concepts, which represent canonical visual cues composing the objects and scene. It first predicts which concepts exist in the image and then makes the final prediction based solely on these predicted concepts, enabling model decisions to be explainable in terms of concept presence. Building on this foundational idea, several works have aimed to improve its reliability [6] or accuracy-interpretability trade-off [4, 17, 20] .Label-free CBMs A significant drawback of conventional CBMs is that they rely on dense annotations indicating which concepts are present in the image. This demands extensive human labor, which makes them impractical to be applied to large-scale datasets like ImageNet [3] . To address this, recent label-free CBMs [8, 12, 16, [21] [22] [23] utilize foundation models to determine the concept presence without human annotation. For example, PCBM [23] leverages CLIP [13] to estimate the presence of concepts. Label-free CBM [12] , LaBo [22] , and LM4CV [21] additionally utilize LLMs to automate the curation of a concept set. Several studies have tackled issues related to concept faithfulness [8] and completeness [16] in such models. Yet, the problem of locality still remains unresolved. 
method: 
 In this section, we present our novel framework whose prediction is inherently interpretable through a composition of distinct local parts and their attributes, collectively referred to as concepts. We first describe the curation of a concept set using LLM (Sec. 3.1). We then describe each component of our method, including two novel losses alongside the classification loss. These losses encourage that the appropriate region to be attended to when predicting a concept, while also promoting that the concept prediction is based on its presence in a specific local region. (Sec. 3.2). The overall architecture of our method is illustrated in Fig. 1 .Here, we describe the automated process of concept generation. We employ the GPT-4-omni [2] to curate a set of concepts similar to previous label-free CBMs [12, 22] . To provide a compositional explanation, we prompt GPT to generate potential concepts for the object category (e.g., birds in the CUB dataset [18] ) rather than class-specific concepts as in prior approaches. Next, we prompt GPT again to align these concepts with each class, yielding a set C = {c 1 , • • • , c K } of up to 20 relevant concepts per class.We now describe the overall architecture of our model. For each sample (x, y) where x is the input image and y is the corresponding label, we first extract features with an encoder f . This yields a feature map F ∈ R H×W ×D , where H and W are the spatial dimensions, and D is the feature dimension. We apply average pooling over the spatial dimensions to obtain F p , which is then mapped to concept logits l c ∈ R K through the linear layer, representing the predicted presence of each concept in the image. Through the final classification layer, the model outputs the class-label logits l y = l c φ(W y ) T where W y is the weights and φ is the activation function which ensures that the weights remain positive, as negative weights are difficult to interpret. We use the classification loss: where L CE is the standard cross-entropy loss and ŷ is the prediction of the model, i.e., softmax over the class-label logits l y .EQUATIONBuilding on top of this concept bottleneck architecture, we aim to improve its localization capabilities, i.e., to better capture the important local concepts and better align its concept prediction to the corresponding local region of the image. First, we crop the image into H × W patches and extract the CLIP features for each patch, resulting in a feature map F c ∈ R H×W ×Dc , where D c is the feature dimension. Similarly, we compute CLIP features for the concept set C, yielding T ∈ R K×Dc . We then obtain the similarity score matrix S = Fc T T ∈ R H×W ×K , where • indicates normalized tensors. To ensure the proper localization of concepts within the image, we compute the influence value for each concept c k as:EQUATIONWe aim to align V k , i.e., the distribution of the influence value for the concept c k over the spatial dimensions, and S k , i.e., the CLIP scores between c k and all image patches, as follows:EQUATIONwhere D KL denotes the KL-divergence, and σ and τ l represent the softmax function and its temperature, respectively.Finally, we apply max-pool over the spatial dimensions of the similarity score matrix S to produce S p ∈ R K , representing how strongly each concept c k is activated in the image. Intuitively, this allows us to effectively capture the concepts existing in the small local region of the image. We align the calculated concept logits with the actual concepts present in the image by minimizing the KL divergence between the distribution of l c and S p /τ c , as follows:EQUATION) The final loss is L total = αL class + βL local + γL concept , where α, β, γ are balancing coefficients. LfCBM [12] 0.727±0.001 0.146±0.001 0.251±0.002 0.173±0.002 0.194±0.006 LaBo [22] 0.734±0.000 0.580±0.001 0.730±0.000 0.626±0.001 -LCBM (Ours) 0.741±0.004 0.702±0.003 0.678±0.013 0.672±0.006 0.515 ±0.007 4 Experiments 
setup: 
 We used the CUB dataset [18] which contains 11,788 images across 200 bird species, with 5,994 images for training. Given many species pairs with small inter-class differences, accurately identifying fine-grained concepts in the images is crucial for correct classification. For the baselines, we compare our method with two representative label-free approaches, LfCBM [12] and LaBo [22] . For the evaluation, we measure the classification accuracy of the models on the validation set. We selected the concepts whose scores (i.e., l c × φ(W y )) are higher than the defined threshold, and considered them as the concepts predicted to be present in the image by the models.Concept evaluation To evaluate whether the predicted concepts truly exist in the image, we calculated the precision, recall, and F1-score of the predicted concepts. Ground-truth concepts for these metrics were selected from the concepts associated with the label, defined by Sec. 3.1. Whether a concept is considered ground-truth was determined by Qwen2-VL [19] , a large VLM capable of visual question answering. Specifically, we queried the VLM "Does {concept} appear in this bird image?" and regarded the concept as ground-truth when the VLM output positive answer. We then measured the text similarity between the predicted concepts and ground-truth concepts using OpenAI's text-embedding-3-large model [1] . Predicted concepts with a text similarity score higher than 0.8 were considered to be present in the image.Localization To evaluate the model's localization ability, we generated a GradCAM [15] based score map for each predicted concept. By applying a threshold to the score map, we extracted the most highly activated area corresponding to each concept. We then used the CUB dataset's ground truth annotations which specify the location point for each part (e.g., head). For each annotated part, if there exists corresponding predicted concept (e.g., 'striped black head' for head part), we calculated whether the activated area of matched concept contains the ground truth location point.The first column in Table 1 shows the classification accuracy. We used the same concept set generated by the LLM at 3.1 for all baselines. LCBM achieved the highest accuracy among the baselines, demonstrating that it maintains generalization ability while enhancing both concept prediction capability and localization.Concept evaluation As shown in Table 1 (precision, recall, F1), LfCBM performs poorly in these metrics as it considers only the entire image, making it difficult to discover fine-grained concepts. LaBo, which also uses only the entire image for reasoning, shows higher concept evaluation scores due to its weight prior. The weights between a class and its corresponding concepts (class-concepts) are larger than other weights, so the predicted concepts tend to heavily reflect the class-concepts, even if some are not present in the image. This is evident from the gap between LaBo's precision and recall: while recall is high because a large portion of class-concepts are included in the predictions, precision is much lower since many of these predicted concepts are not actually present in the image. In contrast, LCBM achieves high precision and comparable recall, indicating that it more accurately identifies the concepts that are genuinely present in the image.The fifth column of Table 1 presents the average ratio of part whose activated area of corresponding concept contains its ground truth location point. This shows that LCBM has a significantly improved ability to correctly localize the corresponding area compared to LfCBM. LaBo is excluded from this evaluation since it does not train a concept prediction layer.Fig. 2 presents the qualitative localization results, comparing our model (LCBM) and LfCBM. The results demonstrate that our model accurately predicts concepts that are aligned with the image's class and the image content itself. More importantly, our model effectively localizes four out of the five predicted concepts, with high activation in the relevant parts of the image. For instance, when the concept contains 'belly', our model strongly activates the region where the belly is located. In contrast, LfCBM struggles with localization, showing more random behavior and failing to focus on the correct areas. 
conclusion: 
 In this paper, we tackled the issue of neglecting locality in existing label-free CBMs by introducing a novel framework, LCBM. Our approach utilizes cropped image patches and incorporates two additional losses to ensure reliable prediction and localization of concepts. Experimental results demonstrate that LCBM significantly improves concept prediction accuracy and localization within the image. 
 
 Appendix: https://github.com/Trustworthy-ML-Lab/Label-free-CBMhttps://github.com/YueYANG1996/LaBo 


Decision: In light of the positive reviewers' feedback and relevancy of the submission, we are pleased to accept this paper for presentation at UniReps 2024. We kindly ask the authors to incorporate the reviewers' suggestions and feedback in the final camera-ready version of the manuscript.

Reviews: review: title: title: The work presents a new and improved procedure to build Concept Bottleneck Models (CBMs) that are also sensitive to the actual location of the concepts in the original image. The work achieves improved performance in-terms of accuracy and localisation scores.
main_review: main_review: Summary - 
The work presents a new and improved procedure to build Concept Bottleneck Models (CBMs) that are also sensitive to the actual location of the concepts in the original image. The work achieves improved performance in-terms of accuracy and localisation scores. Thus, improving the utility of CBMs by making them more location aware.

Strengths - 
1. The paper tackles an important problem, of localising concepts to specific image regions for Concept Bottleneck Models.
2. The work proposes an supposedly easy formulation to incorporate alignment and make the concepts location-aware.
3. The work achieves improved performance in terms of accuracy, precision / recall / f1 scores, and localisation scores of the concepts.
4. The single qualitative example shown also demonstrates some promise of the proposed method.

Weaknesses - 
1. The Formulation of the loss L_local and influence value V_k,h,w is not very clear, and the motivation for using this is not properly explained. A better explanation and motivation of this would be really helpful.
2. The final loss is a weighted sum of L_class , L_local, and L_concept; with weights alpha, beta and gamma. A proper ablation (backed by empirical results on a held-out dataset) of how these are chosen should be demonstrated.
3. Although the method demonstrates improved localisation scores as compared to prior works, however the mIOU scores of 0.202 are still very low, and it is unclear if these are even meaningful (which is also somewhat indicated by the qualitative results).
4. Finally the work only demonstrates the quantitative evaluations on as single (fine-grained birds dataset - CUB), and further evaluation across more datasets would strengthen the results.

Minor - 
In the abstract, the following line is not very clear or grammatically correct and is recommended to be re-written.- "However, such label-free CBMs often fail to attend to concepts that are important predictive but only exist in a small region of the image (e.g., a beak of a bird), making their decision-making less aligned with human reasoning."
title: title: Technically solid paper with clever tweak for label-free CBM
main_review: main_review: ### Summary

This paper introduces locality-aware concept bottleneck models to improve label-free CBM by dividing the image into smaller patches and using concept similarity. 

### Strengths

- The main idea of patching for locality is simple, clever, and novel.
- The results are strong, conclusive, and underline the effectiveness of the approach

### Areas for development

- This work seems closely related to concept transformers [1], which also split the images into patches. Although they require concept annotations to work but in practice you could train them to work with label-free approaches. I think it would be helpful to outline how this approach varies from this.
- It would be interesting to see an ablation on how overlapping vs. non-overlapping patches and things like patch size change the effectiveness of this method.

[1] Rigotti, Mattia, et al. “Attention-based interpretability with concept transformers.” *International conference on learning representations*. 2021.
title: title: The proposed method is straightforward, but there are some questionable parts.
main_review: main_review: ## Summary:
To tackle the issue of label-free Concept Bottleneck Models (CBMs) neglecting essential concepts that appear in only small regions of an image, the author introduced a novel framework called Locality-Aware CBM (LCBM). This approach divides the image into smaller patches and uses CLIP to extract concept labels, measuring the similarity between these labels and image patches to guide the model's training. Experiments on the CUB dataset demonstrated reasonable classification and concept localization performance.

## Strength:
- S1: The proposed method is straightforward to understand. 

## Weakness:
- W1: While the author's use of the CLIP score for training guidance appears reasonable, whether the proposed method offers an advantage over directly utilizing the CLIP or foundation model to represent the concept logit \(l_c\) is questionable. Since the proposed approach introduces an additional backbone to be trained with CLIP, it would be valuable to include additional experimental results comparing the proposed method with a pure CLIP-based configuration.

- W2: In explaining the equation for computing the class-label logits in lines 84 and 85, the author justifies using a specific activation function to ensure that the weights remain positive. However, this rationale appears inadequate, as a negative relationship between the concept and the logit can also be valid and meaningful.

- W3: The authors presented experimental results on concept evaluation and localization; however, more comprehensive experimentation is required. The CUB dataset contains approximately 300 binary concepts, which could be utilized more extensively in the concept evaluation experiments. Furthermore, the authors focused solely on the head region in the concept localization experiments, which needs to be revised and warrants broader evaluation.
title: title: Review on LCBM
main_review: main_review: This paper introduces the Locality-aware Concept Bottleneck Model (LCBM), aiming to address the limitations of label-free Concept Bottleneck Models (CBMs) in capturing local concepts that exist in small regions of images. 

Strengths :
1. Addressing a Relevant Problem: 

The paper effectively highlights the issue of locality in CBMs, where existing models often fail to capture important local details within images. By adopting a patch-based method and leveraging CLIP for concept localization, it presents a novel approach to improving the alignment between concepts and specific image regions.

2. Enhanced Localization and Interpretability: 

The experimental results demonstrate improved localization, as indicated by the Intersection over Union (IoU) metric. LCBM outperforms baseline methods like LfCBM in identifying localized features such as bird beaks and heads, with a notable increase in IoU scores. By focusing on more localized concepts, the model is able to enhance transparency in decision-making.


Weakness :

1. No Justification for Patch Size and Partitioning :

The paper does not provide a clear explanation or reasoning for the chosen patch size or the method used to partition the image. Dividing images into patches and treating each one independently assumes that each concept can be isolated in a small region, which may not always hold true. There is no discussion of why the selected patch size is optimal or how sensitive the model is to this parameter. This could result in arbitrary or suboptimal choices in practice, where an incorrect patch size might misrepresent local features or overlook important visual information spread across patches.


2. Overlooking the Effect of Concept Overlap: 

The method assumes that each patch contains distinct and isolated concepts, but in many real-world scenarios, visual concepts often overlap or are interdependent. By rigidly dividing images into patches, the model may fail to capture these interdependencies, leading to flawed predictions. This limitation overlooks the complexity of visual data, where multiple concepts may need to be considered together to achieve accurate predictions.

Conclusion:

Overall, this paper tackles an important problem in CBMs by focusing on locality, offering a solution through LCBM. While there are some limitations, particularly around the choice of patch size and handling concept overlap, the overall approach is promising and addresses key issues in the field. I believe the strengths outweigh the weaknesses, and this paper should be accepted.
title: title: The paper proposes a locality preserving CBM model which can extract granular information from the image and making a more aligned human interpretable concept reasoner.
main_review: main_review: The paper has interesting ideas but it talks about the locality of the images (i.e say beak of the bird) but doesn't provide experiments to validate this. A good example would be that if we can align what the CUB attribute list is and do a ROC classification with the extracted attributes. Though the authors did a localisation task was it only covered for 'head' or other attributes too.