Title: Towards Multi-dimensional Explanation Alignment for Medical Classification
Venue: NeurIPS 2024
Accepted: yes
Keywords: Explainable Medical Image

Interpretable ML

Explainable AI
Abstract: The lack of interpretability in the field of medical image analysis has significant ethical and legal implications. Existing interpretable methods in this domain encounter several challenges, including dependency on specific models, difficulties in understanding and visualization, and issues related to efficiency. To address these limitations, we propose a novel framework called Med-MICN (Medical Multi-dimensional Interpretable Concept Network). Med-MICN provides interpretability alignment for various angles, including neural symbolic reasoning, concept semantics, and saliency maps, which are superior to current interpretable methods. Its advantages include high prediction accuracy, interpretability across multiple dimensions, and automation through an end-to-end concept labeling process that reduces the need for extensive human training effort when working with new datasets. To demonstrate the effectiveness and interpretability of Med-MICN, we apply it to four benchmark datasets and compare it with baselines. The results clearly demonstrate the superior performance and interpretability of our Med-MICN.

Towards Multi-dimensional Explanation Alignment for Medical Classification

Abstract

The lack of interpretability in the field of medical image analysis has significant ethical and legal implications. Existing interpretable methods in this domain encounter several challenges, including dependency on specific models, difficulties in understanding and visualization, as well as issues related to efficiency. To address these limitations, we propose a novel framework called Med-MICN ( Med ical M ultidimensional I nterpretable C oncept N etwork). Med-MICN provides interpretability alignment for various angles, including neural symbolic reasoning, concept semantics, and saliency maps, which are superior to current interpretable methods. Its advantages include high prediction accuracy, interpretability across multiple dimensions, and automation through an end-to-end concept labeling process that reduces the need for extensive human training effort when working with new datasets. To demonstrate the effectiveness and interpretability of Med-MICN, we apply it to four benchmark datasets and compare it with baselines. The results clearly demonstrate the superior performance and interpretability of our Med-MICN.

1 Introduction

The field of medical image analysis has witnessed remarkable advancements, especially for the deep learning models. Deep learning models have exhibited exceptional performance in various tasks, such as image recognition and disease diagnosis [31, 46, 1], with an opaque decision process and intricate network. However, this lack of transparency is particularly problematic in the medical domain, making it challenging for physicians and clinical professionals to trust the predictions made by these deep models. Thus, there is an urgent need for the interpretability of model decisions in the medical domain [43, 13, 57].

The medical field has strict trust requirements. It not only demands high-performing models but also emphasizes comprehensibility and earning the trust of practitioners [20]. Thus, Explainable Artificial Intelligence (XAI) has emerged as a prominent research area in this field. It aims to enhance the transparency and comprehensibility of decision-making processes in deep learning models and large language models by incorporating interpretability [65, 21, 22, 64, 63, 25, 8]. Various methods have been proposed to achieve interpretability, including attention mechanisms [56, 61, 27, 26, 24], saliency maps [70, 16], DeepLIFT and Shapley values [38, 4], influence functions [34, 55]. These methods strive to provide users with visual explanations that shed light on the decision-making process of the model. However, while these post-hoc explanatory methods offer valuable information, there is still a gap between their explanations and the model decisions [42]. Moreover, these post-hoc

Ground-glass appearance

0.01

Ground-glass appearance

0.31

mlncreased density in the lung

0.04

mlncreased density in the lung0.04

Localized or diffuse persentation

10.09

Localized or diffuse persentation

0.23

Absence of lobar consolidation

0.06

Absence of lobar consolidation

10.76

Crazy-paving pattern

0.07

Crazy-paving pattern |0.01

Image class NonCOVID

Multilobar distribution

Bilateral involvement

0.01

0.04

Image class COVID

Multilobar distribution0.06

Bilateral involvement0.05

Peripheral ground-glass opacities

10.03

Peripheral ground-glass opacities

0.62

0.00

0.020.040.060.08

0.0

0.2

0.4

0.6

Concept preditions

Concept preditions

CCs

Saliency map

Concept reasoning rules

Concept reasoning rules

Ground-glass appearance

0.06

Ground-glass appearance

0.07

mlncreased density in the lung

[0.05

mlncreased density in the lung

10.25

Localized or diffuse persentation

0.07

Localized or diffuse persentation

0.33

Absence of lobar consolidation

0.09

Absenceof lobarconsolidation

0.75

Crazy-paving pattern0.03

Crazy-paving pattern

[0.14

Multilobar distribution

0.11

Multilobar distribution

0.83

Image class NonCOVID

Bilateral involvement0.05

Image class COVID

Bilateral involvement

10.67

Peripheral ground-glass opacities

10.66

Peripheral ground-glass opacities

0.81

0.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

0.8

Concept preditions

Concept preditions

CC4Cs C6C

²

Concept reasoningrules

Saliency map

Concept reasoningrules

explanations are generated after the model training and cannot actively contribute to the model fine-tuning process, hindering them from being a faithful explanation tool.

Thus, there is increasing interest among researchers in developing self-explanatory methods. Among these, concept-based methods have garnered significant attention [50, 2, 35]. Concept Bottleneck Model (CBM) [35] initially predicts a set of pre-determined intermediate concepts and subsequently utilizes these concepts to make predictions for the final output, which are easily understandable to humans. Concept-based explanations provided by inherently interpretable methods are generally more comprehensible than post-hoc approaches. However, most existing methods treat concept features alone as the determination of the predictions. This approach overlooks the intrinsic feature embeddings present within medical images, thus degrading accuracy [44]. Moreover, while these concepts are human-understandable, they lack semantic meanings, thus questioning the faithfulness of their interpretability [39]. To improve further human trust, several recent works [3] aim to leverage syntactic rule structures to concept embeddings. However, there are still several potential issues. First, unlike CBMs, current concept models with logical rules mainly focus on the supervised concept case, which is quite strict for biomedical images as concept annotation is expensive. Second, while current concept models (with logical rules) provide interpretations via concepts, we found that the importance of these concepts is misaligned with other explanations, especially the explanation given by saliency maps [70, 16]. This will lead to a possible reduction in human trust when using these models.

To address these challenges, we introduce a new and innovative end-to-end concept-based framework called the Med-MICN ( Med ical M ulti-dimensional I nterpretable C oncept N etwork), as illustrated in Figure 3. As shown in Figure 2, Med-MICN is an end-to-end framework that leverages Large Multimodals (LMMs) to generate concept sets and perform auto-annotation for medical images, thereby aligning concept labels with images to overcome the high cost associated with medical concepts annotation. In contrast to typical concept-based models, our interpretation is notably more diverse and precise (shown in Figure 1). Specifically, we map the image features extracted by the backbone through a concept encoder to obtain concept prediction and concept embeddings, which are then input into the neural symbolic layers for interpretation. This also establishes alignment between image information and concept embeddings by utilizing a concept encoder, leading to the derivation of predictive concept scores. Furthermore, we align concept semantics with concept embeddings by incorporating neural symbolic layers. Thus, we effectively align image information with concept semantics and concept saliency maps, achieving comprehensive multidimensional alignment. Additionally, unlike most concept-based methods, we use concept embeddings to complement the original image features, which enhances classification accuracy without any post-process. Our main contributions can be summarized as follows:

We proposed an end-to-end framework called Med-MICN, which leverages the strength of different XAI methods such as concept-based models, neural symbolic methods, saliency maps, and concept semantics. Moreover, Med-MICN generates rules and utilizes concept embeddings to complement the intrinsic medical image features, which improves accuracy.

Med-MICN offers an alignment strategy that includes text and image information, saliency maps, and concept semantics. It is model-agnostic and can easily transfer to other models. Our outputs are interpreted in multiple dimensions, including concept prediction, saliency maps, and concept reasoning rules, making it easier for experts to identify and correct errors.

Through extensive experiments on four benchmark datasets, Med-MICN demonstrates superior performance and interpretability compared with other concept-based models and the black-box model baselines.

2 Related Work

Concept Bottleneck Model. The Concept Bottleneck Model (CBM) [35] has emerged as an innovative deep-learning approach for image classification and visual reasoning by incorporating a concept bottleneck layer into deep neural networks. However, CBM faces two significant challenges. Firstly, its performance often falls short of the original models without the concept bottleneck layer, attributed to incomplete information extraction from the original data to bottleneck features. Secondly, CBM extensively depends on meticulous dataset annotation. To solve these problems, researchers have delved into potential solutions. For example, [7] have extended CBM into interactive prediction settings by introducing an interaction policy to determine which concepts to label, ultimately improving the final predictions. Additionally, [41] has addressed the limitations of CBM by proposing a novel framework called Label-free CBM, which offers promising alternatives. Post-hoc Concept Bottleneck models [66] can be applied to various neural networks without compromising model performance, preserving interpretability advantages. Despite much research in the image field [18, 33, 32, 45, 47, 36, 28, 23, 37], concept-based method for the medical field remains less explored, which requires more precise results and faithful interpretation. [9] used a conceptual alignment deep autoencoder to analyze tongue images representing different body constituent types based on traditional Chinese medicine principles. [35] introduced CBM for osteoarthritis grading and used ten clinical concepts such as joint space narrowing, bone spurs, calcification, etc.

However, previous research heavily relies on expert annotation datasets or often focuses solely on concept features to make predictions while overlooking the intrinsic feature embeddings within images. Furthermore, while the concept neural-symbolic model has been explored in the graph domain [3], its application to images, particularly in the medical domain, has been largely absent. Additionally, our work addresses these gaps by proposing an end-to-end framework with an alignment strategy that leverages various explainable methods, including concept-based models, neural-symbolic methods, saliency maps, and concept semantics, to provide comprehensive solutions to these challenges.

Explanation in Medical Image. Research on the interpretability of deep learning in medical image processing provides an effective and interactive approach to enhancing medical knowledge and assisting in disease diagnosis. User studies involving physicians have revealed that doctors often seek explanations to understand AI results, especially when the outcomes are related to their own hypotheses or differential diagnoses [60]. They also turn to explanations to resolve conflicts when their judgments differ from those of AI [6], thereby enhancing the intelligence of medical models. Previous studies have visualized lesion areas through methods such as heatmaps [59] and attention visualization [12], aiding in the identification of lesion regions and providing visual evidence. Additionally, utilizing language model-based methods like LLM or LMM to generate medical reports complements the interpretation of model results (ChatCAD [58], XrayGPT [52], Med-PaLM [49]). Saliency maps have emerged as the most common and clinically user-friendly explanation for medical imaging tasks [53, 68, 69]. Recent research underscores the importance of understanding the pivotal features influencing AI predictions, particularly when clinicians must compare AI decisions with their own clinical assessments in cases of decision incongruity [54]. In addition to the image's intrinsic feature recognition, assisted discrimination methods based on concept injection are widely employed in assisted medical diagnosis [35, 7]. Compared to relying solely on self-supervised training, conceptual feature-based supplementation integrates expert knowledge, offering more accurate assistance for interpreting detection results.

However, previous research on medical images relies on single-dimensional explanations, potentially lacking sufficient decision information for physicians. Furthermore, erroneous single-dimensional explanations could significantly impact physicians' judgments. Thus, there is a pressing need for a multi-dimensional explanatory framework where explanations across various dimensions complement each other. In instances of incorrect explanations, physicians can turn to explanations in alternative dimensions to aid their judgment.

(a)

GPT-4V

Concept Set

TextEmbeddings

Bilateral involvement

Ｆ2

Bilateralinvolvement

Class Names&Imgs

Multilobardistribution

Encoder

Text

Multilobardistribution

切3

AlveolarOpacities

AlveolarOpacities

NonCOVID

Crazy-paving pattern

Crazy-pavingpattern

tR

(q)

Concept Heatmap

ConceptScore

Concept

Vector

Label

Bilateralinvolvement

ImageFeature

ooo

0.821

Threshold

tP

0.13!

Multilobardistribution

Pooling

Image

ooO

1

Encoder

t

**.

yHxWxD

Crazy-pavingpattern

0.611

t

3 Preliminaries

Concept Bottleneck Models. To introduce the original Concept Bottleneck Models, we adopt the notations used by [35]. We consider a classification task with a concept set denoted as C = C 1 , C 2 , . . . , C N and a training dataset represented as { ( x i , y i , c i ) } i M =1 . Here, for i ∈ [ M ] , x i ∈ R d represents the feature vector, y i ∈ R d z denotes the label (with d z corresponding to the number of classes), and c i ∈ R d c represents the concept vector. In this context, the j -th entry of c i represents the weight of the concept p j . In CBMs, our goal is to learn two representations: one that transforms the input space to the concept space, denoted as g : R d → R d c , and another that maps the concept space to the prediction space, denoted as f : R d c → R d z . For any input x , we aim to ensure that its predicted concept vector ˆ c = g ( x ) and prediction ˆ y = f ( g ( x )) are close to their underlying counterparts, thus capturing the essence of the original CBMs.

Fuzzy Logic Rules. As described by [17, 3], continuous fuzzy logic extends upon traditional Boolean logic by introducing a more nuanced approach to truth values. Rather than being confined to the discrete values of either 0 or 1, truth values are represented as degrees within the continuous range of {0, 1}. Conventional Boolean connectives including t-norm ∧ : [0 , 1] × [0 , 1] ↦→ [0 , 1] , t-conorm ∨ : [0 , 1] × [0 , 1] ↦→ [0 , 1] , negation ¬ x = 1 -x . The logical connectives, including ¬ , ∨ , ∧ , ⇒ , ⇐ , ⇔ , are utilized to convey the logical relationships between concepts and their representations. For example, consider the problem of deciding whether an X-ray lung image has COVID, given the vocabulary of concepts "ground-glass opacities (GO)," "Localized or diffuse presentation (LDP)," and "lobar consolidation (LC)." A simple decision rule can be y ⇔ c GO ∧ ¬ c LC . From this rule, we can deduce that (1) Having both "no LC" and "GO" is relevant to having COVID. (2) Having LDP is irrelevant to deciding whether COVID exists.

Featureextract

Embeddings

ConceptEmb

Classification

Concept

Encoder

fc(x)

Prediction

Backbone

classy

FC

：

y1,y2..,yM

Ob

Prediction

conceptC

(Notations)

f(x1)

f(xM)

C1,C2 .., CM

fc (x1)

fc(xM)

ConceptSemantic

Concept Labels c: C1,..., Cm

Alignment

Neural symbolic layer

Neural

E {0,1}N

VLMs

Auto

Anno

C E {0,1}N

m =0,1,2 ...M

Op : backbone

①c : concept encoder

yneural

Med-MICN

Rules yneural, ,..,yneural,M

4 Medical Multi-dimensional Interpretable Concept Network

Here, we present Med-MICN (Figure 3), a novel framework that constructs a model in an automated, interpretable, and efficient manner. (i) In traditional CBMs, the concept set is typically generated through annotations by human experts. When there is no concept set and concept labels, we first introduce the automated concept labeling alignment process (Figure 2). (ii) Then, the concept set (output by LLMs such as GPT4-V) is fed into the text encoder to obtain word embedding vectors. Our method utilizes Vision-Language Models (VLMs) to encode the images and calculate cosine distances to generate heatmaps. We apply average pooling to these heatmaps to obtain a similarity score aligned with the concept set through a threshold to obtain concept labels. (iii) Next, we extract image features using a feature extraction network and then map them through a concept encoder to obtain concept embeddings. (iv) We finally use these concept embeddings as input into the neural symbolic layers to generate concept reasoning rules and incorporate them as complementary features to the intrinsic spatial features for predictions, proving multi-dimensional interpretation alignment. We provide details for each component of Med-MICN as follows.

4.1 Concept Set Generation and Filtering

Given M c classes of target diseases or pathology, the first step of our paradigm is to acquire a set of useful concepts related to the classes. A typical workflow in the medical domain is to seek help from experts. Inspired by the recent work, which suggests that instruction-following large language models present a new alternative to automatically generate concepts throughout the entire process [41, 40]. We propose to generate a concept set using LMMs, such as GPT-4V, which has extensive domain knowledge in both visual and language, to identify the crucial concepts for medical classification. Figure 2 (a) illustrates the framework for concept set generation. Details are in Appendix A.

4.2 VLMs-Med-based Concept Alignment

Generating Concept Heatmaps. Suppose we have a set of N useful concepts C = { C 1 , C 2 , . . . , C N } obtained from GPT-4V. Then, the next step is to generate pseudo-(concept)

labels for each image in the dataset corresponding to its concept set. Inspired by the satisfactory performance in concept recognition within the medical field demonstrated by VLMs [62], we use BioViL [5] to generate the pseudo-labels for the concepts of each image. Figure 2 (b) illustrates the detailed automatic annotation process of concepts.

Given an image x and a concept set, its feature map V ∈ R H × W × D and the text embedding t i ∈ R D for each concept are extracted as follows:

where Θ V and Θ T are the visual and text encoders, t i is the embedding of the i -th concept in the concept pool, H and W are the height and width of the feature map.

Given V and t i , we can obtain a heatmap P i , i.e., a similarity matrix that measures the similarity between the concept and the image can be obtained by computing their cosine distance:

where h, w are the h -th and w -th positions in the heatmaps, and P h,w,i represents a local similarity score between V and t i . Then, we derived heatmaps for all concepts, denoted as { P 1 , P 2 , . . . , P N } .

Calculating Concept Scores. As average pooling performs better in downstream medical classification tasks [62], we apply average pooling to the heatmaps to deduce the connection between the image and concepts: s i = 1 H · W ∑ H h =1 ∑ W w =1 H h,w,i . Intuitively, s i is the refined similarity score between the image and concept c i . Thus, a concept vector e can be obtained, representing the similarity between an image input x and a set of concepts: e = ( s 1 , . . . , s N ) T .

Alignment of Image and Concept Labels. To align images with concept labels, we determine the presence of a concept attribute in an image based on a threshold value derived from an experiment. If the value s i exceeds this threshold, we consider the image to possess that specific concept attribute and set the concept label to be True . We can obtain concept labels for all images c = { c 1 , . . . , c M } , where C i ∈ { 0 , 1 } N is the concept label for the i -th sample. Finally, to ensure the truthfulness of concepts, we discard all concepts for which the similarity across all images is below 0.45. To achieve higher annotation accuracy, we only annotated 20% of the data for fine-tuning the model to adapt to different datasets. We sampled 10% of the pseudo-labels generated and compared them with expert annotations [11]. It is notable that in the case where the concept set and concept labels are given, we can directly skip Section 4.1 and 4.2.

4.3 Multi-dimensional Alignment

In Section 4.2, we get the concept labels for all input images. However, as we mentioned in the introduction, such representation might significantly degrade task accuracy [39, 67]. To overcome this issue, recently [67] propose using concept embeddings, which increase the task accuracy of concept-based models while weakening their interpretability. Motivated by this, we use these concept embeddings to increase the accuracy and leverage our concept label to enhance interpretability. In the following, we provide details.

Concept Embeddings. For the training data X = { ( x m , y m ) } M m =1 , we use a backbone network (e.g., ResNet50) to extract features F = { f ( x m ) } M m =1 . Then, for each feature, it passes through a concept encoder [67] to obtain feature embeddings f c ( x m ) and concept embeddings ˆ c m . The specific process can be represented by the following expression:

where Θ b and Θ c represent the backbone and concept encoder, respectively.

To enhance the interpretability of concept embeddings, we utilize binary cross-entropy to optimize the accuracy of concept extraction by computing L c based on ˆ c = { ˆ c m } M m =1 and concept labels c in Section 4.2:

Neural-Symbolic Layer . Our next goal is to use concept embedding to learn concept rules for prediction, which is motivated by [3]. We aim to generate rules involving the utilization of two sets of feed-forward neural networks: Φ( · ) and Ψ( · ) . The output of Φ( · ) signifies the assigned role of each concept, determining whether it is positive or negative (such as 'no LC' and 'GO'). On the

other hand, the output of Ψ( · ) represents the relevance of each concept, indicating whether it is useful within the context of the sample feature (such as 'LDP'). The overall process can be divided into three distinct parts: (i) Learning the role (positive or negative) of each concept called Concept Polarity . For each prediction class j , there exists a neural network Φ j ( · ) . This network takes each concept embedding as input and produces a soft indicator, a scalar value in the [0 , 1] range. This soft indicator represents the role of the concept within the formula; (ii) Learning the relevance of each concept called Concept Relevance . Similar to concept polarity, for each prediction class j , a neural network Ψ j ( · ) is utilized; (iii) Output the logical reasoning rules of the concept and the contribution scores of concepts. For each class j , we combine the previous concept polarity vector I o,j and the concept relevance vector I r,j to obtain the logical inference output. This is achieved by the following expression (Details are in Appendix B):

4.4 Final Objective

In this section, we will discuss how we derive the class of medical images and the process of network optimization. First, we have the loss L c in (1) for enhancing the interpretability of concept embeddings. Also, as the concept embeddings are input into the neural-symbolic layer to output logical reasoning rules of the concept and prediction ˆ y neural,m in (2) for x m , we also have a loss between the predictions given by concept rule and ground truth, which corresponds to the interpretability of our neural-symbolic layers. In the context of binary classification tasks, we employ binary cross-entropy (BCE) as our loss function. For multi-class classification tasks, we use crossentropy as the measure. Using binary classification as an example, we calculate the loss L neural by comparing the output ˆ y from the neural-symbolic layer to the label y as follows:

Classification Loss. Note that as L neural in (3) is purely dependent on the concept rules rather than feature embeddings, we still need a loss for final prediction performance. In a typical classification network, the process involves obtaining the feature f ( x m ) and passing it through a classification head to generate the classification results. What sets our approach apart is that we fuse the previously extracted f c ( x m ) with the f ( x m ) using a fusion module as input to the classification ahead. This can be expressed using the following formula:

Note that W F represents a fully connected neural network. For training our classification model, we use categorical cross-entropy loss, which is defined as follows:

Formally, the overall loss function of our approach can be formulated as:

where λ 1 , λ 2 are hyperparameters for the trade-off between interpretability and accuracy.

5 Experiment

In this section, we introduce the experimental settings, present our superior performance, and showcase the interpretability of our network. Due to the space limit, additional experimental details and results are in the appendix C.

5.1 Experimental Setting

Datasets. We consider four benchmark medical datasets: COVID-CT [29] for CT images, DDI [10] for dermatology images, Chest X-Ray [14], and Fitzpatrick17k [15] for a dermatological dataset with skin colors.

Baselines. Wecompared our model with other state-of-the-art interpretable models, such as Label-free CBM[41] and DCR[3], to highlight the robustness of our interpretability capabilities. Furthermore, we conducted comparisons with black-box models, such as SSSD-COVID [51].

Evaluation Metrics. In this study, we concentrate on medical image classification. To comprehensively evaluate our classification model, we employ a range of key metrics, including binary accuracy, precision, recall, F1-Score, and AUC value. Specifically, we calculate each metric for positive and negative samples in each dataset, alongside utilizing AUC for further insight into classification performance.

Experimental Setup. We employed three different models as our backbones: ResNet-50 [19], VGG19 [48], and DenseNet169 [30]. To demonstrate the superiority of our method, we use the same backbones without any additional processing. For the concept encoder, we utilized the same structure as described in [35]. Before training, we obtained pseudo-labels for the concepts in each image using the automatic labeling process. Although previous work has shown that operations like super-resolution reconstruction can improve model accuracy on our dataset, to demonstrate the superiority of our method, we used the original images as our input. During training, we adopted the Adam optimizer with a learning rate of 5e-5 throughout the training stages. For hyperparameter selection, we set λ 1 = λ 2 = 0 . 1 .

5.2 Model Utility Analysis

Med-MICN delivers superior performance. In Table 1, our method achieved different improvements on various backbones for the COVID-CT dataset. Taking Acc as an example, it increased by 3.39% with ResNet50 and by 3.45% with VGG compared to backbones. Compared to SSSD-COVID, our method outperforms in terms of Acc and F1, with improvements of 4.68% and 7.15%, demonstrating the superiority of our method in enhancing model accuracy. Additionally, our method possesses interpretability, which is not achievable by SSSD-COVID. On the other hand, our method achieved significant improvements on different backbones for the DDI dataset. For instance, Acc increased by 6.01% with VGG. Despite the significant differences between the two datasets in terms of modality, our method demonstrated significant effects on both datasets, indicating its good generalizability. Details are shown in Table 4, 5, 6, and 7 in Appendix.

Meanwhile, when compared to existing well-performing interpretable models, our approach demonstrates significant advantages in terms of accuracy and other metrics across different datasets. This indicates that our joint prediction of image categories using both concept and image feature spaces outperforms predictions based solely on a limited concept space.

5.3 Model Interpretability Analysis

Explanation across multiple dimensions. In our approach, we discover and generate concept reasoning rules based on the neural-symbolic layer. Analyzing these rules enhances the interpretability of our network. In addition, our approach derives concept prediction scores through the concept encoder, and during evaluation, it also produces saliency maps. Through multi-dimensional explanations, we can observe the basis for the model decision from different perspectives. In concept score prediction, we can observe how the model maps data to concept dimensions, allowing doctors to observe and correct concept results, thereby rectifying prediction errors caused by incorrect concept predictions. In concept reasoning rules, we can deduce the model classification criteria, further explaining the model decisions. Additionally, during the evaluation process, we can generate saliency maps for the

data, providing an intuitive understanding of how pixels in the image influence the image classification result aligned with semantic concepts. Compared to traditional concept-based models, our model interpretation offers advantages in richness and accuracy. Additional visualization examples can be found in Appendix D.2.

Cyst

10.16

Cyst

0.16

Pigmented

0.05

Pigmented

0.05

Wheal

10.05

Wheal

10.05

Erythema

10.63

Erythema

10.73

White

0.03

White

0.03

Translucent

0.02

Translucent-0.02

Warty

0.17

10.71

Warty

0.11

0.15

Xerosis

10.48

Xerosis

0.05

Fissure

0.14

Fissure

0.04

Exudate

Vesicle

Papule

0.03

10.13

10.54

y1←Cg

Exudate-0.03

Papule-0.01

Vesicle-0.01

0.0

0.2

0.4

0.6

0.0

0.2

0.4

0.6

Concept Preditions

ConceptPreditions

Saliency map(ResNet50)

CBM

Med-MICN

Cyst

0.06

Cyst

10.09

Pigmented

0.08

Pigmented

10.18

Wheal

0.11

Erythema

Wheal

White

0.05

0.59

10.71

Translucent

Erythema

White

0.03

0.03

10.72

Translucent

i0.13

0.81

0.62

Warty

0.13

Warty

10.53

Xerosis

10.05

Xerosis

0.02

Fissure

0.09

Exudate

Fissure

Papule

Vesicle

0.04

10.17

10.23

0.31

Exudate

Vesicle

Papule

0.04

0.03

0.07

0.0

0.2

Concept Preditions

0.4

0.6

y1=C6Cg

0.0

0.2

Concept Preditions

0.4

0.6

0.8

Saliency map(ResNet50)

CBM

Med-MICN

As shown in Figure 4, it is evident that relying solely on single-dimensional interpretable strategies, such as saliency maps or concept embedding enhancements, does not furnish adequate interpretability to effectively address the problem. However, by integrating multi-dimensional strategies, the model can align the information of each dimension, thus obtaining more comprehensive interpretable information and ultimately yielding more correct prediction results. Specifically, when feature extraction is solely reliant on saliency maps, obtaining accurate attention in the feature region often proves challenging, and conceptual information tends to be unstable when supplemented solely by concepts. In contrast, our proposed multi-dimensional interpretable strategy transcends dependence on a single interpretable strategy, opting instead for a more generalized multi-dimensional augmentation approach. This approach enables the model to complement the single-dimensional methods and achieve heightened accuracy.

5.4 Ablation Study

The ablation experiments presented in Table 2, conducted with Resnet50 as the backbone, reveal significant contributions from both L c and L neural to the classification result. To illustrate, considering the comprehensive index AUC in the DDI dataset, utilizing only L c yeilds a 3.79% improvement, while relying solely on L neural does not notably enhance performance. However, employing both

simultaneously leads to an 8.71% improvement. This observation underscores the complementary nature of concept and neural logic rules in enhancing model performance. Besides, additional ablation studies investigating the effect of concept filters and comparing VLMs labeling methods with other Med-CLIP approaches are provided in Appendix D.3. Additionally, we conducted a sensitivity analysis for both baselines and Med-MICN on the DDI dataset, as depicted in Figure 14 in the appendix, showcasing the robustness of our model against perturbations. Furthermore, computational cost analysis (presented in Table 9 in the appendix) was conducted. Experimental findings indicate that Med-MICN incurs minimal computational cost compared to the baseline model while achieving improvements in both accuracy and interpretability.

6 Conclusion

This paper proposes a novel end-to-end interpretable concept-based model called Med-MICN. Combining medical image classification, neural symbolic solving, and concept semantics, Med-MICN achieves superior accuracy and multi-dimensional interpretability. Our comprehensive experiments demonstrate consistent enhancements over other baselines, highlighting its potential as a generalized and faithful interpretation model for medical images.

Acknowledgments

Di Wang and Lijie Hu are supported in part by the funding BAS/1/1689-01-01, URF/1/4663-01-01, REI/1/5232-01-01, REI/1/5332-01-01, and URF/1/5508-01-01 from KAUST, and funding from KAUST - Center of Excellence for Generative AI, under award number 5940. Di Wang and Lijie Hu are also supported by the funding of the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI).

References

Ravi Aggarwal, Viknesh Sounderajah, Guy Martin, Daniel SW Ting, Alan Karthikesalingam, Dominic King, Hutan Ashrafian, and Ara Darzi. Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis. NPJ digital medicine , 4(1):65, 2021.

Anonymous. Faithful vision-language interpretation via concept bottleneck models. In The Twelfth International Conference on Learning Representations , 2024.

Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Mateo Espinosa Zarlenga, Lucie Charlotte Magister, Alberto Tonda, Pietro Lio, Frederic Precioso, Mateja Jamnik, and Giuseppe Marra. Interpretable neural-symbolic concept reasoning. In Proceedings of the 40th International Conference on Machine Learning , volume 202, pages 1801-1825. PMLR, 2023.

Daniel Beechey, Thomas MS Smith, and Özgür ¸ Sim¸ sek. Explaining reinforcement learning with shapley values. In International Conference on Machine Learning , pages 2003-2014. PMLR, 2023.

Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez-Valle, Hoifung Poon, and Ozan Oktay. Making the Most of Text Semantics to Improve Biomedical Vision-Language Processing , page 1-21. Springer Nature Switzerland, 2022.

Carrie J Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg S Corrado, Martin C Stumpe, et al. Human-centered tools for coping with imperfect algorithms during medical decision-making. In Proceedings of the 2019 chi conference on human factors in computing systems , pages 1-14, 2019.

Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37(5), pages 5948-5955, 2023.

Keyuan Cheng, Gang Lin, Haoyang Fei, Lu Yu, Muhammad Asif Ali, Lijie Hu, Di Wang, et al. Multi-hop question answering under temporal knowledge editing. arXiv preprint arXiv:2404.00492 , 2024.

Yinglong Dai and Guojun Wang. Analyzing tongue images using a conceptual alignment deep autoencoder. IEEE Access , 6:5962-5972, 2018.

Roxana Daneshjou, Kailas Vodrahalli, Roberto A Novoa, Melissa Jenkins, Weixin Liang, Veronica Rotemberg, Justin Ko, Susan M Swetter, Elizabeth E Bailey, Olivier Gevaert, et al. Disparities in dermatology ai performance on a diverse, curated clinical image set. Science advances , 8(31):eabq6147, 2022.

Roxana Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto A. Novoa, and James Zou. Skincon: A skin disease dataset densely annotated by domain experts for fine-grained debugging and analysis. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Jie Du, Kai Guan, Yanhong Zhou, Yuanman Li, and Tianfu Wang. Parameter-free similarityaware attention module for medical image classification and segmentation. IEEE Transactions on Emerging Topics in Computational Intelligence , 2022.

Marzyeh Ghassemi, Luke Oakden-Rayner, and Andrew L Beam. The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health , 3(11):e745-e750, 2021.

Praveen Govi. CoronaHack-Chest X-Ray-Dataset, 2020.

Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1820-1828, 2021.

Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in Neural Information Processing Systems , 35:1140-1156, 2022.

Petr Hájek. Metamathematics of fuzzy logic , volume 4. Springer Science & Business Media, 2013.

Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems , 35:23386-23397, 2022.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.

Andreas Holzinger, Georg Langs, Helmut Denk, Kurt Zatloukal, and Heimo Müller. Causability and explainability of artificial intelligence in medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 9(4):e1312, 2019.

Yihuai Hong, Yuelin Zou, Lijie Hu, Ziqian Zeng, Di Wang, and Haiqin Yang. Dissecting fine-tuning unlearning in large language models. arXiv preprint arXiv:2410.06606 , 2024.

Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Hasti Seifi, and Jiayuan Xie. Unimeec: Towards unified multimodal emotion recognition and emotion cause. arXiv preprint arXiv:2404.00403 , 2024.

Lijie Hu, Tianhao Huang, Huanyi Xie, Chenyang Ren, Zhengyu Hu, Lu Yu, and Di Wang. Semi-supervised concept bottleneck models. arXiv preprint arXiv:2406.18992 , 2024.

Lijie Hu, Tianhao Huang, Lu Yu, Wanyu Lin, Tianhang Zheng, and Di Wang. Faithful interpretation for graph neural networks. arXiv preprint arXiv:2410.06950 , 2024.

Lijie Hu, Liang Liu, Shu Yang, Xin Chen, Hongru Xiao, Mengdi Li, Pan Zhou, Muhammad Asif Ali, and Di Wang. A hopfieldian view-based interpretation for chain-of-thought reasoning. arXiv preprint arXiv:2406.12255 , 2024.

Lijie Hu, Yixin Liu, Ninghao Liu, Mengdi Huai, Lichao Sun, and Di Wang. Improving faithfulness for vision transformers. arXiv preprint arXiv:2311.17983 , 2023.

Lijie Hu, Yixin Liu, Ninghao Liu, Mengdi Huai, Lichao Sun, and Di Wang. Seat: stable and explainable attention. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37(11), pages 12907-12915, 2023.

Lijie Hu, Chenyang Ren, Zhengyu Hu, Cheng-Long Wang, and Di Wang. Editable concept bottleneck models. arXiv preprint arXiv:2405.15476 , 2024.

Qiongjie Hu, Hanxiong Guan, Ziyan Sun, Lu Huang, Chong Chen, Tao Ai, Yueying Pan, and Liming Xia. Early ct features and temporal lung changes in covid-19 pneumonia in wuhan, china. European journal of radiology , 128:109017, 2020.

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4700-4708, 2017.

Daniel S. Kermany, Daniel S. Kermany, Michael H. Goldbaum, Wenjia Cai, Carolina C. S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei, Jacqueline Pei, Magdalene Yin Lin Ting, Jie Zhu, Christina M. Li, Sierra Hewett, Sierra Hewett, Jason Dong, Ian Ziyar, Alexander Shi, Runze Zhang, Lianghong Zheng, Rui Hou, William Shi, Xin Fu, Xin Fu, Yaou Duan, Viet Anh Nguyen Huu, Viet Anh Nguyen Huu, Cindy Wen, Edward Zhang, Edward Zhang, Charlotte L. Zhang, Charlotte L. Zhang, Oulan Li, Oulan Li, Xiaobo Wang, Michael A. Singer, Xiaodong Sun, Jie Xu, Ali R. Tafreshi, M. Anthony Lewis, Huimin Xia, and Kang Zhang. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell , 172:1122-1131.e9, 2018.

Mert Keser, Gesina Schwalbe, Azarm Nowzad, and Alois Knoll. Interpretable model-agnostic plausibility verification for 2d object detectors using domain-invariant concept bottleneck models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3890-3899, 2023.

Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. arXiv preprint arXiv:2306.01574 , 2023.

Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions, 2020.

Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning , pages 5338-5348. PMLR, 2020.

Songning Lai, Lijie Hu, Junxiao Wang, Laure Berti-Equille, and Di Wang. Faithful visionlanguage interpretation via concept bottleneck models. In The Twelfth International Conference on Learning Representations , 2023.

Jia Li, Lijie Hu, Zhixian He, Jingfeng Zhang, Tianhang Zheng, and Di Wang. Text guided image editing with automatic concept locating and forgetting. arXiv preprint arXiv:2405.19708 , 2024.

Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems , 30, 2017.

Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi-Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314 , 2021.

Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capabilities of gpt-4 on medical challenge problems, 2023.

Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations , 2023.

Georgina Peake and Jun Wang. Explanation mining: Post hoc interpretability of latent factor models for recommendation systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2060-2069, 2018.

Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence , 1(5):206-215, 2019.

Anirban Sarkar, Deepak Vijaykeerthy, Anindya Sarkar, and Vineeth N Balasubramanian. A framework for learning ante-hoc explainable models via concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10286-10295, 2022.

Yoshihide Sawada and Keigo Nakamura. Concept bottleneck model with additional unsupervised concepts. IEEE Access , 10:41758-41765, 2022.

Fahad Shamshad, Salman Khan, Syed Waqas Zamir, Muhammad Haris Khan, Munawar Hayat, Fahad Shahbaz Khan, and Huazhu Fu. Transformers in medical imaging: A survey. Medical Image Analysis , page 102802, 2023.

Ivaxi Sheth and Samira Ebrahimi Kahou. Auxiliary losses for learning generalizable conceptbased models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

K Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations (ICLR 2015) . Computational and Biological Learning Society, 2015.

Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Publisher correction: Large language models encode clinical knowledge. Nature , 620(7973):19-19, 2023.

Sanchit Sinha, Mengdi Huai, Jianhui Sun, and Aidong Zhang. Understanding and enhancing robustness of concept-based models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 15127-15135, 2023.

Zhiyong Tan, Yuhai Yu, Jiana Meng, Shuang Liu, and Wei Li. Self-supervised learning with self-distillation on covid-19 medical image classification. Computer Methods and Programs in Biomedicine , 243:107876, 2024.

Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radiographs summarization using medical vision-language models. arXiv preprint arXiv:2306.07971 , 2023.

Vo Hoang Thien, Vo Thi Hong Tuyet, and Nguyen Thanh Binh. Object contour in medical images based on saliency map combined with active contour model. In 8th International Conference on the Development of Biomedical Engineering in Vietnam: Proceedings of BME 8, 2020, Vietnam: Healthcare Technology for Smart City in Low-and Middle-Income Countries , pages 717-732. Springer, 2022.

Sana Tonekaboni, Shalmali Joshi, Melissa D McCradden, and Anna Goldenberg. What clinicians want: contextualizing explainable machine learning for clinical end use. In Machine learning for healthcare conference , pages 359-380. PMLR, 2019.

Bas HM Van der Velden, Hugo J Kuijf, Kenneth GA Gilhuijs, and Max A Viergever. Explainable artificial intelligence (xai) in deep learning-based medical image analysis. Medical Image Analysis , 79:102470, 2022.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature , 620(7972):47-60, 2023.

Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. Chatcad: Interactive computer-aided diagnosis on medical image using large language models. arXiv preprint arXiv:2302.07257 , 2023.

Mengying Xiao, Liyuan Zhang, Weili Shi, Jianhua Liu, Wei He, and Zhengang Jiang. A visualization method based on the grad-cam for medical image segmentation model. In 2021 International Conference on Electronic Information Engineering and Computer Science (EIECS) , pages 242-247. IEEE, 2021.

Yao Xie, Melody Chen, David Kao, Ge Gao, and Xiang'Anthony' Chen. Chexplain: enabling physicians to explore and understand data-driven, ai-enabled medical imaging analysis. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems , pages 1-13, 2020.

Haoxuan Xu, Songning Lai, Xianyang Li, and Yang Yang. Cross-domain car detection model with integrated convolutional block attention mechanism. Image and Vision Computing , 140:104834, 2023.

An Yan, Yu Wang, Yiwu Zhong, Zexue He, Petros Karypis, Zihan Wang, Chengyu Dong, Amilcare Gentili, Chun-Nan Hsu, Jingbo Shang, et al. Robust and interpretable medical image classifiers via concept bottleneck models. arXiv preprint arXiv:2310.03182 , 2023.

Shu Yang, Muhammad Asif Ali, Cheng-Long Wang, Lijie Hu, and Di Wang. Moral: Moe augmented lora for llms' lifelong learning. arXiv preprint arXiv:2402.11260 , 2024.

Shu Yang, Lijie Hu, Lu Yu, Muhammad Asif Ali, and Di Wang. Human-ai interactions in the communication era: Autophagy makes large models achieving local optima. arXiv preprint arXiv:2402.11271 , 2024.

Shu Yang, Shenzhe Zhu, Ruoxuan Bao, Liang Liu, Yu Cheng, Lijie Hu, Mengdi Li, and Di Wang. What makes your model a low-empathy or warmth person: Exploring the origins of personality in llms. arXiv preprint arXiv:2410.10863 , 2024.

Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In The Eleventh International Conference on Learning Representations , 2022.

Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, et al. Concept embedding models. arXiv preprint arXiv:2209.09056 , 2022.

Jiajin Zhang, Hanqing Chao, Giridhar Dasegowda, Ge Wang, Mannudeep K Kalra, and Pingkun Yan. Overlooked trustworthiness of saliency maps. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 451-461. Springer, 2022.

Yi Zhang, Mingming Jin, and Gang Huang. Medical image fusion based on improved multiscale morphology gradient-weighted local energy and visual saliency map. Biomedical Signal Processing and Control , 74:103535, 2022.

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2921-2929, 2016.

A Concept Set Generation and Filtering

A.1 Concept Generation with GPT-4V

In this section, we will explain our process for generating concept sets using GPT-4V. Specifically, we ask GPT-4V the following:

[User]:

List the useful visual attributes to recognize {class} in a {type} in this {image}:

List the things most commonly seen around a {class} in this {image}:

List in bullet points with no extra context.

Note that the image here refers to the images we random sample from the dataset, and type represents the modality of medical image (e.g. X-rays, CT). Full prompts and example outputs are illustrated in the appendix D.2. Combining the concepts received from different classes and images gives us a large, somewhat noisy set of initial concepts, which we further improve by filtering. We found using GPT-4V to generate initial concepts to perform better than using GPT-3, which was used in Label-free CBM [41]. We initiate the procedure with the following conversation:

[User]: List the useful visual attributes to recognize COVID in a CT in this image. [GPT-4V]: When assessing CT images for signs of COVID-19, radiologists often look for certain visual attributes that can suggest the presence of the disease. Here are some of the key visual attributes they may look for:

Ground-glass opacities (GGOs) : These are hazy areas that do not obscure the underlying lung structures. They are called "ground-glass" because they resemble the appearance of frosted glass.

Consolidation : ...

In addition, we can proceed with the following question:

[User]: List the things most commonly seen around COVID in this image. [GPT-4V]: Based on the image you have provided and considering the common radiological findings associated with COVID-19, here are the things that might be commonly seen:

Possible consolidation areas.

Absence of significant pleural effusions.

......

Then, we can use the following inquiry to structure the output from GPT-4V:

[User]: List in bullet points with no extra context. [GPT-4V]:

Ground-Glass Opacities (GGOs)

Bilateral and Multifocal Lesions

Crazy-Paving Pattern

Reverse Halo or Atoll Sign

Absence of Pleural Effusion

Septal Thickening

Rare Lymphadenopathy

Vascular Enlargement

Traction Bronchiectasis

After obtaining the initial concept set following the filtering process, our rules and concepts on COVID-CT and DDI datasets are as follows (see Table 3).

A.2 Concept Set Filtering

After concept set generation, a concept set with some noise can be obtained. The following filters are set to enhance the quality of the concept :

The length of concept : To keep concepts simple and avoid unnecessary complication, We remove concepts longer than 30 characters in length.

Similarity : We measure this with cosine similarity in a text embedding space. We removed concepts too similar to classes or each other. The former conflicts with our interpretability goals, while the latter leads to redundancy in concepts. We set the thresholds for these two filters at 0.85 and 0.9, respectively, ensuring that their similarities are below our threshold.

Remove concepts we cannot project accurately : Remove neurons that are not interpretable from the BioViL [5]. This step is actually described in section 4.2.

B Neural-symbolic Layer

We give the details with examples for neural-symbolic layer [3].

Concept Polarity. For each prediction class j , there exists a neural network Φ j ( · ) . This network takes each concept embedding as input and produces a soft indicator, a scalar value in the [0 , 1] range. This soft indicator represents the role of the concept within the formula. As an illustration, consider a specific concept like 'Crazy-paving pattern'. If its value after passing through Φ j ( · ) is 0.8, it indicates that the 'Crazy-paving pattern' has a positive role with a score of 0.8 for class j . We use the notation I o,i,j to represent the soft indicator for concept c i .

Concept Relevance. For each prediction class j , a neural network Ψ j ( · ) is utilized. This network takes each concept embedding as input and produces a soft indicator, a scalar value within the range of [0 , 1] , representing the relevance score within the formula. To illustrate, let us consider a specific concept such as 'Multilobar distribution'. If its value after passing through Ψ j ( · ) is 0 . 2 , it implies that the relevance score of 'Multilobar distribution' in the inference of class j is 0 . 2 . We denote the soft indicator for concept c i as I r,i,j .

Prediction via Concept Rules. Finally, for each class j , we combine the previous concept polarity vector I o,j and the concept relevance vector I r,j to obtain the logical inference output. This is

achieved by the following expression:

Intuitively, the aforementioned concept reasoning rules allow us to determine the relevance of each concept and whether a concept has a positive or negative role in predicting the label as class j . Intuitively ¬ I o,i,j ∨ I r,i,j means if it is irrelevant, then we set it to 1, and if it is relevant, then we set it to be the relevance score. It is strange at first glance. However, since finally we will take the conjunction (or minimum) for all concepts to get ˆ y j , so we will filter the i -th concept if ¬ I o,i,j ∨ I r,i,j = 1 , i.e., if it is irrelevant. Thus, we take the neg here. Since we need to consider each concept for the final prediction, we finally use ∧ for all concepts.

C More Experimental Setup

C.1 Training Setting

Our model exhibits remarkable efficiency in its training process. We utilized only a single GeForce RTX 4090 GPU, and the training duration did not exceed half an hour. We configured the model to run for 100 epochs with a learning rate set at 5e-5. Additionally, all images were resized to a uniform dimension of (256, 256).

C.2 Datasets

COVID-CT. The COVID-CT dataset was obtained from [29] and comprises 746 CT images, consisting of two classes (349 COVID and 397 NonCOVID). We divided this dataset into a training set and a test set with an 8:2 ratio, and the data were split accordingly.

DDI. DDI [10] is a dataset comprising diverse dermatology images designed to assess the model's ability to correctly identify skin diseases. It consists of a total of 656 images, including 485 benign lesion images and 171 malignant lesion images. These images are divided into training and test sets, with an 80% and 20% split, respectively.

Chest X-Ray. The Chest X-Ray [14] dataset comprises 2D chest X-ray images of both healthy and infected populations. It aims to support researchers in developing artificial intelligence models capable of distinguishing between chest X-ray images of healthy individuals and those with infections. The dataset includes 5933 images, divided into 5309 training images and 624 testing images.

Fitzpatrick17k. Fitzpatrick17k [15] dataset is a dermatological dataset that includes a wide range of skin colors. In order to better compare the model performance, we filtered the malignant and nonmalignant classes in 3230 images that have been relabeled by SkinCon [11]. And we divided the training and test set according to an 8:2 ratio.

C.3 Baseline Models

SSSD-COVID. SSSD-COVID incorporates a Masked Autoencoder (MAE) for direct pre-training and fine-tuning on a small-scale target dataset. It leverages self-supervised learning and selfdistillation techniques for COVID-19 medical image classification, achieving performance levels surpassing many baseline models. Our method is trained exclusively on the COVID-CT dataset without considering the effects of introducing knowledge from other datasets. Notably, our approach outperforms SSSD-COVID in terms of performance. Furthermore, SSSD-COVID falls under the category of black-box models, indicating that our method, to some extent, overcomes the accuracy degradation issue introduced by the incorporation of concepts.

Label-free CBM. Label-free CBM is a fully automated and scalable method for generating concept bottleneck models. It has demonstrated outstanding performance on datasets like ImageNet. In our comparisons, our model outperforms this model significantly in terms of accuracy. Regarding interpretability, our model not only possesses the same level of interpretability as this model but also

provides explanations in multiple dimensions, such as concept reasoning and saliency maps. This helps doctors gain a more diverse and precise understanding of the model's decision-making process when using our model.

DCR. Deep Concept Reasoner (DCR), a concept-based model combining neural-symbolic methods, has demonstrated promising interpretability performance on simple datasets such as XOR, Trigonometry, and MNIST-Addition. However, its performance tends to degrade on more complex or challenging datasets.

C.4 Definition of Metrics

Accuracy is the ratio of the number of correctly categorized samples to the total number of samples:

Where TP denotes true positive, TN denotes true negative, FP represents false negative, and FN represents false negative.

The precision is the proportion of all samples classified as positive categories that are actually positive categories:

Recall is the proportion of samples that are correctly categorized as positive out of all samples that are actually positive categories:

The F1 score is the reconciled mean of precision and recall:

AUC denotes the area under the ROC curve, which is a curve with True Positive Rate (Recall Rate) as the vertical axis and False Positive Rate (False Positive Rate) as the horizontal axis.

D More Experimental Results

D.1 Utility Evaluation

We provide our detailed utility evaluation for four datasets in Table 4, 5, 6, and 7.

Table 4: Results for COVID-CT. We conducted ten repeated experiments and calculated the average results for each metric. Red and blue indicate the best and the second-best result. Our approach outperforms the baseline backbone models while also providing interpretability. When compared to other state-of-the-art interpretable models, our model exhibits significantly higher accuracy.

D.2 Visualization of Concept Predictions

More samples of instance-level predictions for COVID-CT and DDI datasets are visualized in Figure 5, 6, 7, 8, 9, 10, 11, and 12. We also append the corresponding rules to it, which can reflect the multi-dimensional interpretation better. Taking the COVID-CT sample as an example, it can be found that the model predicts the concept score prediction accurately, and for the COVID samples, the correlation of the first three concepts is greater, which leads to a higher prediction score of their concepts, thus generating real concept rules, and then assists the model judgment. This can also be reflected in the saliency map of the model in the inference stage, where the phenomena described by the relevant concepts can get higher attention in the saliency map, thus further reflecting the multidimensional interpretation of Med-MICN.

D.3 Ablation Study: Effect of Concept Filters

In this section, we will discuss how each step in our proposed concept filtering affects the results of our method. In general, our utilization of filters has two main goals: First, improving the interpretability of our models. Second, improving computational efficiency and complexity by reducing the number of concepts.

Although our initial goal was not to improve model accuracy (a model with more concepts is generally larger and more powerful [41]), the more precise concepts after filtering make the concatenated features more effective for classification and slightly improve the accuracy. To evaluate the impact of each filter, we trained our models separately on COVID-CT and DDI while removing one filter at a time and one without using any filter at all. The results are shown in the Table 8. From Table 8, it is noticeable that our model's accuracy does not exhibit high sensitivity to the choice of filters. On the COVID-CT dataset, the accuracy of our model remains largely unaffected by the choice of filters. Furthermore, employing filters on the DDI dataset results in improved model accuracy. This phenomenon arises due to the relatively sparse nature of concepts within the DDI dataset images, where increasing the number of concepts did not yield superior solutions.

D.4 Sensitivity Analysis

We also performed a sensitivity analysis for baselines and Med-MICN in the DDI dataset. we set various attacks under δ ∈ 4 / 255 , 6 / 255 , 8 / 255 , 10 / 255 , 12 / 255 and attack radius ρ a ∈ { 0 , 2 / 255 , 4 / 255 , 6 / 255 , 8 / 255 , 10 / 255 } . in Figure 14, the results show that our model consists of stronger robustness against perturbation. This is evidenced by the marginal decrease in test accuracy as the attack radius increases. The model's detection performance fluctuates slightly at ρ a of 4/255, with this variability diminishing as the perturbation level rises, underscoring the robustness of our model against significant disturbances.

D.5 Computational Cost

We conducted a computational cost analysis for Med-MICN and the baseline models. By inputting a random tensor of size (1, 3, 244, 224) into the model and computing the FLOPs and parameters, the results are presented in Table 9. Experimental evidence demonstrates that Med-MICN incurs only negligible computational cost compared to the baseline models while it achieves improvements in accuracy and interpretability.

Ground-glass appearance

10.04

mlncreased densityinthelung

0.12

Localized or diffuse persentation

0.15

Absenceoflobarconsolidation

10.08

Crazy-paving pattern

10.09

Image class NonCOVID

Multilobardistribution

0.12

Bilateralinvolvement

0.07

Peripheral ground-glass opacities

10.33

0.0

0.1

0.2

0.3

Concept preditions

C C2456

Conceptreasoningrules

Ground-glass appearance

10.06

mlncreased densityinthelung

0.07

Localized ordiffusepersentation

0.04

Absenceoflobarconsolidation

10.08

Crazy-paving pattern

10.69

Image classNonCOVID

Multilobardistribution

0.04

Bilateral involvement-

0.11

Peripheralground-glassopacities

0.07

0.0

0.2

0.4

0.6

Concept preditions

CC 25 6C

Conceptreasoningrules

Ground-glass appearance

0.07

mlncreaseddensityinthelung

0.09

Localizedordiffusepersentation

0.13

Absenceoflobarconsolidation

0.71

Crazy-pavingpattern

0.05

Image class COVID

Multilobardistribution

0.73

Bilateralinvolvement

10.69

Peripheral ground-glass opacities

0.82

0.0

0.2

0.4

0.6

0.8

Concept preditions

y1 = Co Ac1^c2 ^C4

Saliency map

Concept reasoning rules

Ground-glass appearance

0.06

mlncreased densityin thelung

0.11

Localized ordiffusepersentation

10.05

Absenceof lobarconsolidation

10.18

Crazy-paving pattern

0.29

Image class COVID

Multilobardistribution

0.68

Bilateralinvolvement

0.81

Peripheral ground-glass opacities

0.72

0.0

0.2

0.4

0.6

0.8

Concept preditions

y1 ≈ Co ^c1^C2

Conceptreasoningrules

Ground-glass appearance

10.09

mlncreased densityin the lung

10.35

Localizedordiffusepersentation

10.05

Absence of lobar consolidation

10.18

Crazy-paving pattern

10.33

Image class NoncOvVll

Multilobar distribution0.01

Bilateralinvolvement

[0.15

Peripheralground-glassopacities

0.35

0.0

0.1

0.2

0.3

Concept preditions

C  C2 C4 Cs C

Concept reasoningrules

Ground-glassappearance

10.07

mlncreased density in thelung

0.15

Localizedordiffusepersentation

0.13

Absenceoflobarconsolidation

10.68

Crazy-paving pattern

10.43

Image class NoncOVII

Multilobardistribution

0.51

Bilateralinvolvement

10.05

Peripheral ground-glass opacities

0.05

0.0

0.2

0.4

0.6

Concept preditions

Saliency map

Concept reasoning rules

Ground-glass appearance

10.06

mlncreaseddensityinthelung

0.05

Localized or diffusepersentation

0.07

Absenceoflobarconsolidation

10.69

Crazy-paving pattern0.03

Image class COVID

Multilobardistribution

0.71

Bilateral involvement0.05

Peripheralground-glassopacities

10.86

0.0

0.2

0.4

0.6

0.8

Concept preditions

y1 = Co Ac2 ^C4

Saliency map

Concept reasoning rules

Ground-glassappearance

0.03

mlncreased density in the lung

0.15

Localized or diffusepersentation

0.25

Absenceoflobarconsolidation

10.68

Crazy-paving pattern

0.13

Image class COVID

Multilobar distribution

0.81

Bilateralinvolvement

0.75

Peripheral ground-glass opacities

0.76

0.0

0.2

0.4

0.6

0.8

Concept preditions

y1  Co cc2c4

Saliency map

Concept reasoningrules

Cyst

0.09

Pigmented

10.05

Wheal

10.16

Erythema

0.04

White

0.03

Translucent

0.06

Warty

0.05

10.77

ImageclassNonMalignant

Xerosis

0.01

Fissure

0.12

Exudate

0.03

Vesicle

Papule

0.04

0.11

0.0

0.2

0.4

0.6

0.8

ConceptPreditions

y C  2  C4 Cs c> s1o1112

Saliency map

Conceptreasoningrules

Cyst

10.07

Pigmented

0.11

Wheal

0.06

Erythema

10.74

White

0.03

Translucent

0.06

0.12

Warty

0.05

Xerosis

0.07

Image class NonMalignant

Fissure

0.05

Exudate

0.04

Papule

10.61

Vesicle-0.01

0.0

0.2

0.4

0.6

ConceptPreditions

yo Co C2 C C4 s C6 C> CsC1oC11 12

Saliency map

Concept reasoning rules

Cyst

0.06

Pigmented-10.01

Wheal-0.01

Erythema

10.73

White

0.01

Translucent-0.01

0.6

Warty

0.06

Image class Malignant

Xerosis

Fissure

0.03

0.13

Exudate

0.03

Papule

0.11

Vesicle

0.01

0.0

0.2

0.4

0.6

ConceptPreditions

y1 ∈C6Cg

Saliency map

Concept reasoningrules

Cyst

0.07

Pigmented

0.11

Wheal

0.06

Erythema

0.04

White

0.03

Translucent-0.01

10.81

Warty

0.13

Xerosis

0.04

Fissure

0.09

Image class Malignant

Exudate

10.75

Papule

0.11

Vesicle

0.01

0.0

0.2

0.4

0.6

0.8

Concept Preditions

y1C2C6

Saliency map

Concept reasoningrules

Cyst

10.06

Pigmented

10.02

Wheal

10.07

Erythema

10.03

White

0.01

Translucent

10.01

10.11

Warty

0.04

Image class NonMalignant

Xerosis

10.04

Fissure

10.12

Exudate

10.05

Papule

10.12

Vesicle

10.03

0.00

0.05

0.10

ConceptPreditions

y Co 1 2  C4 C5 c> sCg1o1112

Saliency map

Conceptreasoningrules

Cyst

10.29

Pigmented

10.05

Wheal

10.13

Erythema

10.03

White

10.03

Translucent

0.06

10.05

Warty

0.03

Image class NonMalignant

Xerosis

10.02

10.12

Exudate

10.07

Papule

10.21

Vesicle

10.24

0.0

0.1

0.2

0.3

ConceptPreditions

Co C C2 3 C4 Cs C CCg 1011 C12

Concept reasoning rules

Saliency map

Cyst

10.76

Pigmented

0.05

Wheal

0.05

Erythema

0.03

White-0.03

Translucent-0.02

10.41

Warty

10.05

Xerosis

0.04

Image class Malignant

Fissure-10.02

Exudate

0.03

Papule

10.22

Vesicle

10.71

0.0

0.2

0.4

0.6

ConceptPreditions

y1 CoC12

Saliency map

Concept reasoningrules

Cyst

0.07

Pigmented

0.11

Wheal

0.06

Erythema

0.04

White

0.03

Translucent-0.01

10.81

Warty

0.13

Xerosis

0.04

Image class Malignant

Fissure

10.09

Exudate

10.75

Papule

0.11

Vesicle

0.01

0.0

0.2

0.4

0.6

0.8

ConceptPreditions

y1C2C6

Saliency map

Concept reasoningrules

vesicle:0.001

Peripheralground-glassopacities:0.735

papule:0.183

exudate:0.692

Absenceoflobarconsolidation:0.453

fissure:0.061

Localizedor diffusepresentation:0.340

xerosis:0.225

COVID:3.521

warty:0.111

brown:0.209

Crazy-paving pattern:0.895

White:0.227

Malignant:3.702

pigmented:0.934

Bilateralinvolvement:0.826

erythema:0.372

Multilobardistribution:0.109

wheal:0.116

mlncreased densityin thelung:0.158

translucent:0.571

Ground-glass appearance:0.005

(a) COVID

(b) SKIN

0.82

0.80

0.78

0.76

司

0.74

Clean

Baseline

Ours-4/255

0.72

■-

Ours-6/255

Ours-8/255

0.70

Ours-10/255

Ours-12/255

0

2

4

6

8

10

AtackRadius(/255)

E Limitation

In the case of the "neural symbolic" method, additional acceleration techniques may be required when dealing with large sample sizes. However, it is worth noting that medical datasets tend to be relatively small.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims presented in the abstract and introduction accurately reflect the paper's contributions and scope. We proposed an end-to-end framework called MedMICN, which leverages the strengths of various XAI methods such as concept-based models, neural symbolic methods, saliency maps, and concept semantics. In Section 1 and Abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: in Section E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have provided a comprehensive exposition of our network architecture, encompassing experimental parameter details. Additionally, our dataset is publicly available. Thus, our methodology exhibits reproducibility. In Section C and Section 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: Code [Yes] DATA [No]

Justification: We will soon open-source the code, and the datasets used are publicly available for download by anyone. In Section C and Section 5.1.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: All detailed specifications are presented in the experimental section. In Section C and Section 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We conducted ten repeated experiments and calculated the average results for each metric. In Section C and Section 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We detailed this in Section C.1 and Section D.5.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research fully complies with the NeurIPS Code of Ethics. We have carefully reviewed and ensured adherence to all relevant standards.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We pointed out the positive role our model will play in assisting doctors in making decisions regarding diseases in the future. In Section 1 and Section E.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Yes, we have cited the original papers that produced the code packages or datasets, provided the version information and URLs when possible, and included the name of the license for each asset. In Section C and Section 5.1.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This work introduces a novel end-to-end concept-based framework called Med-MICN, which is quite inspiring and important for the next XAI era due to its multi-dimensional, powerful interpretability, and efficient performance. Furthermore, they propose an automated process for efficiently and accurately obtaining concept labels, which are costly to acquire in reality.

STRENGTHS:
1. This paper introduces a novel attempt that creatively integrates medical image classification, neural symbolic solving, and concept semantics. This integration enables multidimensional interpretability, making it more comprehensive and powerful compared to previous interpretable models.

2. Med-MICN demonstrates superior accuracy and interpretability across datasets from different modalities.

3. The ample experiments are convincing, as the authors conducted experiments on multiple datasets and baselines to validate the model's interpretability and accuracy, while also providing rich and impressive visualization results.

WEAKNESSES:
1. The caption of the picture is not detailed enough, as shown in figure 3.

2. The author considers interpretability on the basis of concept embedding, neural-sybolic reasoning and saliency map. The author lacks the reason why it is necessary to analyze the interpretation from these aspects and whether there are other more dimensions.

QUESTIONS:
See weeknesses.

LIMITATIONS:
NA
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the explainable classification of medical images with multi-dimensional explanation. The proposed Med-MICN framework contains four modules including feature extraction, auto-annotation, concept embedding, and neural symbolic layer. The incorporation of fuzzy logic rules is novel in the interpretation study of deep neural networks. The experimental results show that Med-MICN surpasses the baseline networks such as vgg and resent, and also outperforms concept bottleneck models (CBM). The generated attention maps and predicted concepts match better than traditional CBM.\n\nSTRENGTHS:\nSee summary\n\nWEAKNESSES:\n1. The whole presentation of the paper should be further improved including mathematical symbols, figures 2 and 3, and equations. The symbols should be better matched with Figures 2 and 3 to improve readability. It is difficult to understand these figures without reading the contents. Some symbols like Cosine should be given in the figures. Detailed notations are needed in the figures or in the captions.\n2. It is better to give pseudo codes of training and inferences to help understand the whole workflow.\n3. At the bottom of Page 3, \"In instances ... judgment\", why do the authors claim alternative explanations can aid judgment? Any references? More explanations may confuse the physicians.\n4. In line 131, does $y_{i}$ the one-hot labels?\n5. In the Concept labeling alignment module, it is not clear why the authors use \"average pooling to heatmaps\". I think this will lose some spatial localizations of lesions, for example, one heatmap with multiple lesions and another with one lesion, both of them may have similar average pooling results.\n6. Under the line 183, the $V$ is three-dimensional, and why here it has two subscripts.\n7. In line 194, does the representation be $c \\in \\{C_{1}, ..., C_{M}\\}$ ?\n8. In line 142, the logic symbols should be defined may be in supplementary.\n9. Lack of training details. In line 277, how the learning rate decay? Any data augmentations were used?\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nThe whole framework of Med-MICN is complex, and the so-called multi-dimensional explanations are attention maps and predicted concepts which have been proposed in traditional CBM models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces Med-MICN, an explainable framework for medical image classification. This framework leverages a concept bottleneck framework combined with a neural symbolic reasoning framework to generate simple explanations for its predictions. In general, strong performance is demonstrated.\n\nSTRENGTHS:\nThe overall framework is interesting, and the idea of “multi-dimensional explainability” seems widely applicable. The ability to see some explanation for each concept’s logic seems useful, and the decision rules generated are quite simple. The performance of Med-MICN is quite compelling, with superior accuracy across multiple medical datasets.\n\nWEAKNESSES:\nIn general, the notation used in the paper is inconsistent and somewhat ill defined. For example, in section 3, N is used to denote the number of sample in a dataset, and k the number of concepts. In section 4, N becomes the number of concepts, and M the number of samples. Lower case k is then used to index along the height dimension of a feature map. Several objects ($I_{o, j}$ and $I_{r,j}$, for example) are not clearly described upon their introduction.\n\nThere are a few seemingly key hyper parameters ($\\lambda_1$, $\\lambda_2$, the threshold used for binarizing concept vectors), but the experimental details do not indicate that there was a validation/development set used to select these values.\n\nIf these two points can be addressed, I believe this paper will warrant acceptance.\n\nQUESTIONS:\nThere appear to be some notational issues in section 4.2 — initially H and W are used as the height and width of the feature map, but in the definition of $H_{p,k,i}$, it seems that P and K are used to fill the same roll.\n\n“to ensure the truthfulness of 196 concepts, we discard all concepts for which the similarity across all images is below 0.45. “ — does this refer to all training images?\n\nThe equation following line 214 is quite confusing — what is the dimensionality of each of the three values returned by $\\theta_c$? How does it return $f(x_m)$?\n\nDuring training, which components are trained jointly? It seems like $L_{neural}$ only applies to the neural symbolic component, while $L_c$ and $L_{task}$ apply only to the concept extraction component and some additional classification fully connected layer.\n\nFigure 4 is somewhat unclear — which cells in the figure should be compared to which?\n\nLIMITATIONS:\nDiscussion of limitations is appropriate.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed a novel interpretable model called Med-MICN in this work. This method manages medical image classification tasks with multi-dimensional aspects with neural symbolic and concept semantics. With the help of LLMs, the work performed superior on four medical benchmark datasets. In the ablation study, the authors presented that the performance of the proposed method performed better with the complementary multi-dimensional loss functions in evaluating the classification result.\n\nSTRENGTHS:\n- The paper proposed a novel method that combines extra information (text and logic rules) to enhance the classification performance and the interpretability of the model outputs. This enables the interpretability of the deep learning classification model with multi-dimensional information.\n- Instead of post-hoc interpretable methods, the authors integrate fuzzy logic rules into the proposed method. This gives a clear decision rule for image classification and interpretability.\n- The method description is relatively clear and easy to follow. The ablation study gave a good overview of the usage of multi-dimensional information.\n\nWEAKNESSES:\n- The authors show that the proposed method performed superior to the baseline methods in image classification (Tab. 1). However, there is a missing part on the evaluation of the interpretability. How correct are those heatmaps generated from the proposed model, do they align with the labels or the clinicians?\n\n- Also the comparison of the interpretability between the proposed method and the baseline methods is missing. In Tab. 1, the authors mentioned those methods are interpretable, but they did not evaluate the interpretability. This could be crucial for medical applications.\n\n- For section 4.3, the description of Neural-Symbolic Layer is not easy to follow. What is the difference between the functions of \"Concept Polarity\" and \"Concept Relevance\"? It seems like their outputs are the same.\n\nQUESTIONS:\n- The authors used LLM (GPT-4V) in their proposed method. Would it be better or make a difference, if the LLM is a more medical-specific model?\n\n- How do authors deal with different numbers of concept sets from the LLM? If I understood correctly, the LLM could output several different concepts and does not necessarily fit into the input size of the text encoder. (Figure 2.a)\n\n- Did authors try with different $\\lambda_1$ and $\\lambda_2$?\n\nLIMITATIONS:\n- The proposed method provides an opportunity to interpret the medical image classification results and in the meanwhile also increase the classification performance. However, heatmaps look somehow blurry and may not be trustable (Fig 1).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}