Towards Multi-dimensional Explanation Alignment for Medical Classification
Authors: Lijie Hu
Songning Lai
Wenshuo Chen
Hongru Xiao
Hongbin Lin
Lu Yu
Jingfeng Zhang
Di Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Explainable Medical Image
Interpretable ML
Explainable AI
Primary area: machine_learning_for_healthcare

Abstract: The lack of interpretability in the field of medical image analysis has significant ethical and legal implications. Existing interpretable methods in this domain encounter several challenges, including dependency on specific models, difficulties in understanding and visualization, and issues related to efficiency. To address these limitations, we propose a novel framework called Med-MICN (Medical Multi-dimensional Interpretable Concept Network). Med-MICN provides interpretability alignment for various angles, including neural symbolic reasoning, concept semantics, and saliency maps, which are superior to current interpretable methods. Its advantages include high prediction accuracy, interpretability across multiple dimensions, and automation through an end-to-end concept labeling process that reduces the need for extensive human training effort when working with new datasets. To demonstrate the effectiveness and interpretability of Med-MICN, we apply it to four benchmark datasets and compare it with baselines. The results clearly demonstrate the superior performance and interpretability of our Med-MICN.

Towards Multi-dimensional Explanation Alignment for Medical Classification
Lijie Hu* Songning L,,3, Wenshuo Chen":l.2, Hongru Xiao* Hongbin Lin", Lu Yu Jingfeng Zhang and Di Wang Provable Responsible Al and Data Analytics (PRADA) Lab "King Abdullah University of Science and Technology HHKUST(GZ) "Tongji University 'The University of Auckland 6 AAnt Group, 'RIKEN Center for Advanced Intelligence Project (AIP)
Abstract
The lack of interpretability in the field of medical image analysis has significant eth- ical and legal implications. Existing interpretable methods in this domain encounter several challenges, including dependency standing and visualization, as well as issues related to efficiency. To address these limitations, we propose novel framework called Med-MICN (Medical Multi- dimensional Interpretable Concept Network). Med-MICN provides interpretability alignment for various angles, including neural symbolic reasoning, concept mantics, and saliency maps, which are superior to current interpretable methods. Its advantages include high prediction accuracy, interpretability across multiple dimensions, and automation through an end-to-end concept labeling process that re- duces the need for extensive human training effort when working with new datasets. To demonstrate the effectiveness and interpretability of Med-MICN, we apply it four benchmark datasets and compare with baselines. The results clearly demonstrate the superior performance and interpretability of our Med-MICN.
specific models, difficulties in under-
1
Introduction
The field of medical image analysis has witnessed remarkable advancements, especially for the deep learning models. Deep learning models have exhibited exceptional performance various tasks, such as image recognition and disease diagnosis [31 46, []. with opaque decision process and intricate network. However, this lack of transparency is particularly problematic in the medical domain, making it challenging for physicians and clinical professionals trust the predictions made by these deep models. Thus, there is an urgent need for the interpretability model decisions in the medical domain 43 13 177.
The medical field has strict trust requirements. not only demands high-performing models but also emphasizes comprehensibility and earning the trust of practitioners [20]. Thus, Explainable Artificial Intelligence (XAI) has emerged as a prominent research area in this field. aims to enhance the transparency and comprehensibility of decision-making processes in deep learning models and large language models by incorporating interpretability [65 21 22
8]. Various methods
have been proposed to achieve interpretability. including attention mechanisms [56] 61 27 26 24).
saliency maps [70] [6]. DeepLIFT and Shapley values [38]
4] influence functions 34 55 These
methods strive to provide users with visual explanations that shed light the decision- making process of the model. However, while these post-hoc explanatory methods offer valuable information, there is still gap between their explanations and the model decisions 42 Moreover, these post-hoc
"The first three authors contributed equally to this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Ground ploss-ples oposereticn
Ground glass appearanss
bed orfifiio pesseration
leoabred difine prrsentation
Atsence of leter censet dution
D6
Aasence librr concolsation
0.78
Chacyparing pettern
Cuorppoing putsen
Image dass NonCOVID
Jus loter bution
image
Hustibeer sktrisution
Invoivement
plaau graandgghas
Porpharll ground
Concept preditone
Concept preditons
A niyy
miy may A
A
Sallency map
Concept ressening rules
Salenny map
Concept rules
Greandifae appeamance
Ground-g appecrence
strernased denatyy lung bab.s
minerweeed deriity lung
lober consoidation
locar consol
0.75
Crotepouing patter no 03
Crazy pooiig pattem
00.34
Multiober estribution
MLEikear dist bus
Image dass NonCOMID
insalvemenn
Image class COVID
Blattral invevvrment
ground
Concept preditions
Coscest
Ys -6 Amrry A moy mi, mōs mi,
Yt S G AGAG
Safeecy map
Concept reasoning rules
Sallency map
Concept reasoning
Figure 1: Med-MICN demonstrates multidimensional interpretability. encompassing concept score prediction, concept reasoning rules, and saliency maps, achieving alignment within the interpretative framework. The 'Peripheral ground-glass opacities' is co. and along the y-axis, it sequentially becomes C7...,,77
explanations are generated after the model training and cannot actively contribute to the model fine-tuning process, hindering them from being faithful explanation tool.
Thus, there increasing interest among researchers in developing self-explanatory methods. Among these, concept-based methods have garnered significant attention (50) 2 355. Concept Bottleneck
Model (CBM) 35 initially predicts a set of pre-determined intermediate concepts and subsequently utilizes these concepts to make predictions for the final output, which are easily understandable to humans. Concept-based explanations provided by inherently interpretable methods are generally more comprchensible than post-hoc approaches. However, most existing methods treat concept features alone as the determination of the predictions. This approach overlooks the intrinsic feature embeddings present within medical images, thus degrading accuracy [4). Moreover, while these concepts are human-unders aeiraaannnndrstaaaall.... they lack semantic meanings, thus questioning the faithfulness of their interpretability 39] To improve further human trust, several recent works [] aim to leverage syntactic rule structures to concept embeddings. However, there are still several potential issues. First, unlike CBMss, current concept models with logical rules mainly focus the supervised concept case, which is quite strict for biomedical images as concept annotation is expensive. Second, while current concept models (with logical rules) provide interpretations via concepts, we found that the importance of these concepts is misaligned with other explanations, especially the explanation given by saliency maps 770 16] This will lead possible reduction in human trust when using these models.
To address these challenges, we introduce new and innovative end-to-end concept-based framework called the Med-MICN (Medical Multi dineti-ddmensional Interpretable Concept Network), as illustrated in Figure 3 As shown in Figure 2 Med-MICN is an end-to-end framework that leverages Large Multimodals (LMMs) to generate concept sets and perform auto-annotation for medical images, thereby aligning concept labels with images overcome the high cost associated with medical concepts annotation. In contrast to rypical concept-based models, our interpretation notably more diverse and precise (shown in Figure []. Specifically, we map the image features extracted by the backbone through concept encoder to obtain concept prediction and concept embeddings, which are then input into the neural symbolic layers for interpretation. This also establishes alignment between image information and concept embeddings by utilizing concept encoder, leading to the derivation of predictive concept scores. Furthermore, we align concept semanties with concept embeddings by incorporating neural symbolic layers. Thus, we effectively align image information with con- cept semantics and concept saliency maps, achieving comprehensive multidimensional alignment. Additionally, unlike most concept-based methods, we use concept embeddings to complement the original image features, which enhances classification accuracy without any post-process. Our main contributions can be summarized as follows:

---

We proposed an end-to-end framework called Med-MICN, which leverages the strength of different XAI methods such as concept-based models, neural symbolic methods, saliency maps, and concept semantics. Moreover, Med-MICN generates rules and utilizes concept embeddings to complement the intrinsic medical image features, which improves accuracy.
Med-MICN offers an alignment strategy that includes text and image information, saliency maps, and concept semantics. It is model-agnostic and can easily transfer to other models. Our outputs are interpreted in multiple dimensions, including concept prediction, saliency maps, and concept reasoning rules, making it casier for experts to identify and correct errors.
Through extensive experiments on four benchmark datasets, Med-MICN demonstrates superior performance and interpretability compared with other concept-based models and the black-box model baselines.
2 Related Work
Concept Bottleneck Model. The Concept Bottleneck Model (CBM) 35 has emerged as an in- novative deep-learning approach for image classification and visual reasoning by incorporating a concept bottleneck layer into deep neural networks. However, CBM faces two significant chal- lenges. Firstly, its performance often falls short of the original models without the concept bottleneck layer, attributed to incomplete information extraction from the original data to bottleneck features. Secondly, CBM extensively depends on meticulous dataset annotation. To solve these problems, researchers have delved into potential solutions. For example, 7 have extended CBM into interactive prediction settings by introducing an interaction policy to determine which concepts to label, ulti- mately improving the final predictions. Additionally, 44 has addressed the limitations of CBM by proposing a novel framework called Label-free CBM, which offers promising alternatives. Post-hoc Concept Bottleneck models [66] can be applied various neural networks without compromising model performance, preserving interpretability advantages. Despite much research in the image field $7), concept-based method for the medical field remains less explored, which requires more precise results and faithful interpretation. [9] used a conceptual alignment deep autoencoder analyze tongue images representing different body constituent types based on traditional Chinese medicine principles. (35] introduced CBM for osteoarthritis grading and used ten clinical concepts such as joint space narrowing, bone spurs, calcification, etc.
18
33
32
45
36
28
23
However, previous research heavily relies on expert annotation datasets or often focuses solely on concept features to make predictions while overlooking the intrinsic feature embeddings within images. Furthermore, while the concept neural-symbolic model has been explored in the graph domain 3. its application to images, particularly in the medical domain, has been largely absent. Additionally, our work addresses these gaps by proposing an end-to-end framework with an alignment strategy that leverages various explainable methods, including concept-based models, neural-symbolic methods, saliency maps, and concept semantics, to provide comprehensive solutions to these challenges.
Explanation in Medical Image. Research on the interpretability of deep learning in medical image processing provides an effective and interactive approach to enhancing medical knowledge and assisting in disease diagnosis. User studies involving physicians have revealed that doctors often seek explanations to understand AI results, especially when the outcomes are related to their own hypotheses or differential diagnoses [60]. They also turn to explanations to resolve conflicts when their judgments differ from those of Al (6, thereby enhancing the intelligence of medical models. Previous studies have visualized lesion areas through methods such as heatmaps (59] and attention visualization [2), aiding in the identification of lesion regions and providing visual evidence. Additionally, utilizing language model-based methods like LLM or LMM to generate medical reports complements the interpretation of model results (ChatCAD (58). XrayGPT [52), Med-PaLM 44)). Saliency maps have emerged as the most common and clinically user-friendly explanation for medical imaging tasks (53
68 69 Recent research underscores the importance of understanding the pivotal features influencing AI predictions, particularly when clinicians must compare AI decisions with their own clinical assessments in cases of decision incongruity [54]. In addition to the image's intrinsic feature recognition, assisted discrimination methods based on concept injection are widely employed in assisted medical diagnosis 35 7 Compared to relying solely on self-supervised training, conceptual feature-based supplementation integrates expert knowledge, offering more accurate assistance for interpreting detection results.
3

---

However, previous research on medical images relies on single-dimensions explanations, potentially lacking sufficient decision information for physicians. Furthermore, erroneous single-dimensional explanations could significantly impact physicians' judgments. Thus, there is pressing need for multi-dimensional explanatory framework where explanations across various dimensions complement each other. In instances of incorrect explanations, physicians can turn to explanations in alternative dimensions to aid their judgment.
(a)
Concapt Set
Text Embeddings
involvement
Class Names Imgs
Multlobar distribution
Text Encoder
62
Alveolar Opacities
Aleelor Opsiite
1
NonCCiviD
COVID
paving pattern
Cranyposing patters
(b)
Score
Concept
Label
Bltirrl incleamen
Image Feature
O
D
0.82
0.33
Pooling
Image
Encoder
Vil-WoD
0.61
(a) module, output rich dimensional interpretable conceptual information for the specified disease through the multimodal model and convert the conceptual information into text vectors through the text embedding module; (b) module, access the image to the image embedder to get the image features, and then match with the conceptual textual information get the relevant attention region. Then, we get the influence score of the relevant region information through pooling, and finally send it the filter sieve out the concept information with weak relevance get the disease concept of image information.
Figure 2:
Preliminaries
Concept Bottleneck Models. To introduce the original Concept Bottleneck Models, we adopt the notations used by [35). We consider a classification task with a concept set denoted as Crand training dataset represented { Wr, a)) Here, for i [M]. x; Rd represents the feature vector, E Rdd denotes the label (with de corresponding to the number of classes), and & E Rdd represents the concept vector. In this context, the j-th entry of e represents the weight of the concept Pý- In CBMs, our goal is to learn two representations: one that transforms the input space to the concept space, denoted as Rd - Rd,,, and another that maps the concept space to the prediction space, denoted as f Rd + Rdd, For any input æ, we aim to ensure that its predicted concept vector è g(x)) and prediction ý (((x)) are close to their underlying counterparts, thus capturing the essence of the original CBMs.
Fuzzy I Logic Rules. As described by 17 continuous fuzzy logic extends upon traditional Boolean logic by introducing a more nuanced approach truth values. Rather than being confined to the discrete values of either 0 or 1, truth values are represented as degrees within the continuous range of [0, 1). Conventional Boolean connectives including t-norm A [o, 1] x (0, 1] + [0, 1]. t-conorm V: [0. 1]x x J0. 1] - [0, 1]. negation - = 1-x. The logical connectives, including -, V, +, +>, are utilized to convey the logical relationships between concepts and their representations. For example, consider the problem of deciding whether an X-ray lung image has COVID, given the vocabulary of concepts "ground-glass opacities (GO)," "Localized or diffuse presentation (LDP)." and "lobar consolidation (L.C)." " A simple decision rule can be < cGo A -CLC- From this rule, we can deduce that (1) Having both "no L.C" and "GO" is relevant to having COVID, (2) Having LDP irrelevant to deciding whether COVID exists.

---

Feature extract
Embeddings
Concept Emb
Classification
Concept Encoder
(x)
Prediction class y 9,
FC layer
Backbone
FC
Prediction
concept
(Notations)
((x)
((rw)
4 Eu
f (xx)
fc (Kx)
Concept Semantic Alignment
Concept Labels Cp m. CM
Je Ctr) R
0
Neural Cimbolic layer
f(mm) ga
Neural
e (0.1p*
VLMs
Auto Anno
(0,
0.I.2...M
Symbolic
9, backbone
0e concept encoder
Med-MICN
necural
Rules 9nsurat1 Smeanas
Figure 3: Overview of the Med-MICN Framework. The Med-MICN framework consists of four pri- mary modules: (1) Feature Extraction Module: In the initial step, image features are extracted using a backbone network to obtain pixel-level features. (2) Concept Embedding Module: The extracted features are fed into the concept embedding module. This module outputs concept embeddings while passing through category classification linkage layer to obtaín predicted category information. (3) Concept Semantic Alignment: Concurrently, a Vision-Language Model (VLM) is used to annotate image features, generating concept category annotations aligned with the predicted categories. (4) Neural Symbolic Layer: After obtaining the concept embeddings, they are input into the Neural Symbolic layer to derive conceptual rules. Finally, the concept embeddings obtained from module (2) concatenated with the original image embeddings and fed into the final category prediction layer to produce the ultimate prediction results.
4 Medical Multi-dimennional Interpretable Concept Network
Here, we present Med-MICN (Figure 3p. novel framework that constructs model in an automated, interpretable, and efficient manner.
(i) In traditional CBMs, the concept is typically generated through annotations by human experts. When there is no concept set and concept labels, we first introduce the automated concept labeling alignment process (Figure 2}. (ii) Then, the concept set (output by LLMs such as GPT4-V) is fed into the text encoder to obtain word embedding vectors. Our method utilizes Vision Language Models (VLMs) encode the images and calculate cosine distances to generate heatmaps. We apply average pooling to these heatmaps to obtain similarity score aligned with the concept set through a threshold to obtain concept labels. (iii) Next, we extract image features using feature extraction network and then map them through concept encoder to obtain concept embeddings. (iv) We finally use these concept embeddings as input into the neural symbolic layers to generate concept reasoning rules and incorporate them complementary features to the intrinsic spatial features for predictions, proving multi-dimensional interpretation alignment. We provide details for each component of Med-MICN follows.
4.1 Concept Set Generation and Filtering
Given M,, classes of target diseases or pathology, the first step of our paradigm is to acquire a set of useful concepts related the classes. A typical workflow in the medical domain is seek help from experts. Inspired by the recent work, which suggests that instruction- following large language models present new alternative automatically generate concepts throughout the entire process [41] 40). We propose to generate a concept set using LMMs, such as GPT-4V, which has extensive domain knowledge in both visual and language, identify the crucial concepts for medical classification. Figure 2 (a) illustrates the framework for concept set generation. Details are in Appendix
4.2
VLMs--Med-based Concept Alignment
Generating Concept Heatmaps. Suppose we have set of N useful concepts C {C1, C2...., CN. obtained from GPT-4V. Then, the next step is to generate pseudo-(concept)
5

---

labels for each image in the dataset corresponding to its concept set. Inspired by the satisfactory performance in concept recognition within the medical field demonstrated by VLMs [62]. we use
BioViL. S to generate the pseudo-labels for the concepts of each image. Figure (b) illustrates the detailed automatic annotation process of concepts.
Given an image z and concept set, its feature map V E ER #XxxDD and the text embedding & E R for each concept are extracted as follows:
V = Or(),,
4 (r(),,,, i= N
where Oy and r are the visual and encoders. t is the embedding of the i-th concept in the concept pool, H and W are the height and width of the feature map.
Given V and ti, between the concept and the image can be obtained by computing their cosine distance:
obtain heatmap P., i.e., similarity matrix that measures the similarity
VA.
h H, w = L...., W
||e/|| 1|V
where h, the h-th and w-th positions in the heatmaps, and Pr represents local similarity score between V and 44. Then, we derived heatmaps for all concepts, denoted as {P1, P PN):
Calculating Concept Scores. As average pooling performs better in downstream medical classi- fication tasks 662 we apply average pooling to the hcatmaps to deduce the connection between the image and concepts:
I Σ 1 Ew Hh,w,i. Intuitively, 8; is the refined similarity H-W
score between the image and concept qę. Thus, concept vector can be obtained, representing the similarity between an image input x and a set of concepts: e = ($1, (NN"....N)).
Alignment of Image and Concept Labels. To align images with concept labels, we determine the presence of concept attribute in an image based on a threshold value derived from an experiment. the value 8; exceeds this threshold, and set the concept label to be True. We can obtain concept labels for all images c = {ei,... Cad }. where C, {0. | is the concept label for the i-th sample. Finally, to ensure the truthfulness of concepts, we discard all concepts for which the similarity across all images is below 0.45. To achieve higher annotation accuracy,
consider the image possess that specific concept attribute
only annotated 20% of the data for fine-tuning the model to adapt to different datasets. We sampled 10% of the pseudo-labels generated and compared them with expert annotations [T]. It is notable that in the case where the concept set and concept labels are given, we can directly skip Section 4.1 and 4.2
4.3 Multi-dimennionall Alignment
In Section 4.2, we get the concept labels for all input images. However, as we mentioned in the introduction, such representation might significantly degrade task accuracy 39 67]. To overcome this issue, recently [67] propose using concept embeddings, which increase the task accuracy of concept-based models while weakening their interpretabiliity Motivated by this, embeddings to increase the accuracy and leverage our concept label to enhance interpretability. the following, provide details.
these concept
LM Concept Embeddings. For the training data {Tim ym m= we use a backbone network (e.g., ResNet50) to extract features F (f JM Jm=1- Then, for each feature, it passes through concept encoder [67] to obtain feature embeddings (2 and concept embeddings čn The specific process can be represented by the following expression:
f(xm) (x(mm), fe(#m), C... (c(f(mm) for [M],
where O and θ, represent the backbone and concept encoder, respectively.
To enhance the interpretability of concept embeddings, utilize binary cross-entropy to optimize the accuracy of concept extraction by computing Le based on č = { and concept labels in Section 4.2
L. BC)((,)).
(I)
Neural-Symbolic Layer. Our next goal is to use concept embedding to learn concept rules for prediction, which is motivated by (3] We aim to generate rules involving the utilization of two sets of feed-forward neural networks: (+.) and (.).. The output of +(.) signifies the assigned role of each concept, determining whether it is positive negative (such as "no LC" and "GO"). On the

---

other hand, the output of (-) represents the relevance of each concept, indicating whether it is useful within the context the sample feature (such as "LDP"). The overall process can divided into three distinct parts: (i) Learning the role (positive or negative) of each concept called Concept Polarity. For each prediction class j, there exists a neural network Φ This network takes each concept embedding as input and produces soft indicator, a scalar value in the [0, 1] range. This soft indicator represents the role of the concept within the formula; (ii) Learning the relevance of each concept called Concept Relevance. Similar to concept polarity, for each prediction class j. neural network V (-) is utilized; (iii) Output the logical reasoning rules of the concept and the contribution scores of concepts. For each class J, combine the previous concept polarity vector Io.j and the concept relevance vector Ir to obtain the logical inference output. This is achieved by the following expression (Details are in Appendix B):
Új - (- V VTaa)) min {max(1 Tonyy Tagh).
iE[N]
4.4 Final Objective
In this section, we will discuss how we derive the class of medical images and the process of network optimization. First, we have the loss Le in ( for enhancing the interpretability of concept embeddings. Also, as the concept embeddings are input into the neural-symbolic layer to output logical reasoning rules of the concept and prediction Inearal,m. in (2) for Im, we also have loss between the predictions given by concept rule and ground truth, which corresponds to the interpretability of neural-symbolic layers. In the context of binary classification tasks, employ binary cross-entropy (BCE) as our loss function. For multi-class classification tasks, we use cross- entropy as the measure. Using binary classification as an example, we calculate the loss Lnearol by comparing the output ý from the neural- neural-symbolic layer to the label as follows:
Lncural BCC (neuraii y)
Classification Loss. Note that as Lneural in 3 purely dependent on the concept rules rather than feature embeddings, we still need loss for final prediction performance. In typical classification network, the process involves obtaining the feature fx²m) and passing it through classification head to generate the classification results. What sets our approach apart is that we fuse the previously extracted f=(mm) with the f (Im) using a fusion module as input the classification ahead. This can be expressed using the following formula:
Jm WP- Concat( (f fe
Note that Wr represents fully connected neural network. For training our classification model, we categorical cross-entropy loss, which is defined follows:
Ltask = CF y)
Formally, the overall loss function of our approach can be formulated as:
[ Lrask \ L. + A2 Lneural:
where W X2 are hyperparameters for the trade-off between interpretability and accuracy.
5 Experiment
In this section, we introduce the experimental settings, present our superior performance, and showcase the interpretability of our network. Due to the space limit, additional experimental details and results are in the appendix C
5.1 Experimental Setting
Datasets. We consider four benchmark medical datasets: COVID-CT 29 for CT images, DDI 10 for dermatology images, Chest X-Ray [14) and Fitzpatrick17k 15 for a dermatological dataset with skin colors.
Baselines. We compared our model with other state-of-the-art interpretable models, such as Label-free 4BM and DCR to highlight the robustness of our interpretability capabilities. Furthermore, conducted comparisons with black-box models, such as SSSD-COVID (51).
7

---

Evaluation Metries. In this study, we concentrate on medical image classification. To comprehen- sively evaluate our classification model, we employ a range of key metrics, including binary accuracy. precision, recall, F1 F1-Score, and AUC value. Specifically, we calculate each metric for positive and negative samples in each dataset, alongside utilizing AUC for further insight into classification performance.
Experimental Setup. We employed three different models as our backbones: ResNet-50 [1]] VGG19 48) and DenseNet169 BQ) To demonstrate the superiority of our method, we use the same backbones without any additional processing. For the concept encoder, we utilized the same structure as described in [35]. Before training, we obtained bheudo-labelss for the concepts in each image using the automatic labeling process. Although previous work has shown that operations like super-resolution reconstruction can improve model accuracy on our dataset, to demonstrate the superiority of our method, we used the original images as our input. During training, we adopted the Adam optimizer with learning rate of Se-5 throughout the training stages. For hyperparameter selection, we set A1 X2 0.1.
COVID-CT
DDI
Chest X-Ray
Fitzpatrick17k
Method Backbone
Interpretability
Acc.(%)
F1(S)
(cc.
(1(9)
Acc.(S))
F1(%)
Acc.(S)
F1(S)
ResNet50
81.36
81.67
77.27
72.77
75.64
71.72
80.79
80.79
x
VGG19
79.60
79.88
76.52
70.12
81.41
77.56
75.37
75.37
Baseline
16nseNet169
85.59
85,59
78.03
69.51
69.55
61.66
76.85
76,83
SSSD-COVID
81.76
80.00
Label Free CBM
69.49
69.21
70.34
69.21
71.21
70.84
75.24
75.41
DCR
55.93
51.41
76.52
65.32
62.02
41.33
68.05
66.12
ResNe150
84.75
84.75
81.82
76.33
78.37
74.42
82.76
83.03
Ours
VGG19
83.05
84.37
82.58
78.07
88,30
88.16
77.34
77.53
16nseNet169
86.44
87.15
79.55
69.79
73.88
65.70
80.79
81.11
Table 1:
Utility results. We conducted ten repeated experiments and calculated the average results for each metric. Red and blue indicate the best and the second-best result. Our approach outperforms the baseline backbone models while also providing interpretability. Our model exhibits significantly higher accuracy compared to other state-of-the-art interpretable models.
5.2 Model Utility Analysis
Med-MICN delivers superior performance. In Table our method achieved different improve- ments on various backbones for the COVID-CT dataset. Taking Acc as an example, it increased by 3.39% with ResNet50 and by 3.45% with VGG compared backbones. Compared to SSSD-COVID, our method outperforms in terms of Acc and FI, with improvements of 4.68% and 7. 7.15%, demonstrat- ing the superiority of our method in enhancing model accuracy. Additionally, our method possesses interpretability, which is not achievable by significant improvements on different backbones for the DDI dataset. For instance, Ace increased by 6.01% with VGG. Despite the significant differences between the two datasets in terms of modality. our method demonstrated significant effects on both datasets, indicating its good generalizability. Details are shown in Table
On the other hand, our method achieved
6 and 7 in Appendix.
Mcanwhile, when compared to existing well- peelllerfonmnn interpretable models, our approach demon- strates significant ad advantages in terms of accuracy and other metrics across different datasets. This indicates that our joint prediction of image categories using both concept and image feature spaces outperforms predictions based solely on limited concept space.
5.3 Model Interpretabiliiyy Analysis
Explanation across multiple dimensions. In our approach, we discover and generate concept rea- soning rules based on the neural-symbolic layer. Analyzing these rules enhances the interpretability of our network. In addition, our approach derives concept prediction scores through the concept encoder, and during evaluation, it also produces saliency maps. Through multi-dimensione explanations, we can observe the basis for the model decision from different perspectives. In concept score prediction, we can observe how the model maps data to concept dimensions, allowing doctors to observe and correct concept results, thereby rectifying prediction errors caused by incorrect concept predictions. In concept reasoning rules, we can deduce the model classification criteria, further explaining the model decisions. Additionally, during the evaluation process, can generate saliency maps for the

---

Setting
Datuset
Px Lneural
ACC(%)
Precision! %)
Recall( %))
F1(%)
AUC(%)
Interpretability
82.20
82.92
82.21
82.55
82.64
COVID-CT
83.05
83.62
83.16
83.01
83,16
81.36
82.11
81.38
81,20
81.81
84.75
84.77
84.88
84.75
84.77
78.03
74.97
66.88
69.24
67.41
DDI
79.55
75.36
71.47
72.73
71.20 11.20
81.82
76.56
76.17
76.33
76.12
72.28
77.63
64.15
63.72
64.15
Chest X-Ray
70.03
73.83
61.84
61.25
62.39
78.37
80.38
73.12
74.42
73.12
78.32
79.06
Fitzpatrick17k
79. 0.880
80.60
79.81
80.20
80.31
80.79
81.28
80.82
81.28
81.07
82.76
82.84
83.23
83.03
82.99
Table Experimental results from ablation studies on each loss function demonstrate that each loss function is indispensable for both accuracy and interpretability.
data, providing an intuitive understanding of how pixels in the image influence the image classification result aligned with semantic concepts. Compared traditional concept- based models, model interpretation offers advantages in richness and accuracy. Additional visualization examples can be found in Appendix D.2
Cyst
Cyst
Plamented 0:05
0.05
Wheal
0.00
Wheal =0.05
Erythema
White
50.03
White 80.03
Translucent
Translucent 10.02
Brown
0,71
Warty 0.05
0.15
Xeros'
0.40
Fssure
0.13
Nassre 0 0.
Papule
Papuie 10
00.03
Vesicle
0.0
0.2
0.4
0.6
0.4 Concapt Preditions
Concept Preditions
Saliency map(ResNet5O))
CBM
Med-MICN
Cyst Wheal 0.11
0.00
Plamented
0.18
Erytheme =0.03
Erytheme
0.59
(Nite
Translucent
m0.03
0.13
Xeros
Brown
10,02
0.53
Xerosis
Warty
0.09
0.11 0.13
Exucate
0.17
Exudate 0.07
Weicle
0.0
0.2
Concept Preditions
0.23
0.4
0.6
V1 Ac
Vesicle
0.0
#0.03
0.2
Concept Preditions
0.4
0.6
0.8
Saliency map(ResNet50)
CBM
Med-MICN
Figure 4: Comparison of single-dimensional and multi-dimensional interpretability methods.
As shown in Figure is evident that relying solely on single-dimensional interpretable strategies, such as saliency maps or concept embedding enhancements, does not furnish adequate interpretability effectively address the problem. However, by integrating multi-dimensional strategies, the model can align the information of each dimension, thus obtaining more comprehensive interpretable information and ultimately yielding more correct prediction results. Specifically, when feature extraction is solely reliant saliency maps, obtaining accurate attention in the feature region often proves challenging. and conceptual information tends to be unstable when supplemented solely by concepts. In contrast, our proposed multi-dimensional interpretable strategy transcends dependence on single interpretable strategy, opting instead for more generalized multi multi-dimmensonall augmentation approach. This approach enables the model to complement the single-dimensions methods and achieve heightened accuracy.
5.4 Ablation Study
The ablation experiments presented Table conducted with Resnet50 as the backbone, reveal sig- nificant contributions from both L. and Lncara! to the classification result. To illustrate, considering the comprehensive index AUC in the DDI dataset, utilizing only L. yeilds a 3.79% improvement, while relying solely Lneural does not notably enhance performance. However, employing both

---

simultaneously leads to an 8.71% improvement. This observation underscores the complementary nature of concept and neural logic rules in enhancing model performance. Besides, additional ablation studies investigating the effect of concept filters and comparing VL.Ms labeling methods with other Med-CI.. approaches are provided in Appendix [D.3] Additionally, we conducted sensitivity analysis for both baselines and Med-MICN on the DDI dataset, depicted in Figure [4] in the appendix, showcasing the robustness of our model against perturbations. Furthermore, computational cost analysis (presented in Table 9 the appendix) was conducted. Experimental findings indicate that Med-MICN incurs minimal computational cost compared to the baseline model while achieving improvements in both accuracy and interpretability.
6
Conclusion
This paper proposes novel end-to-end interpretable concept-based model called Med-MICN. Com- bining medical image classification, neural symbolic solving, and concept semanties, Med-MICN achieves superior accuracy and dinlii-dimesionnll interpretability. Our comprehensive experiments demonstrate consistent enhancements over other baselines, highlighting its potential as generalized and faithful interpretation model for medical images.
Acknowledgments
Di Wang and Lijie Hu are supported in part by the funding BAS/1/1689-01-01, URF/1/4663-01-01 REI/I/5232-01- RE1/1/5332-01-0,, and URF/1/5508-01 from KAUST, and funding from KAUST Center of Excellence for Generative AI, under award number 5940. Di Wang and Lijie Hu also supported by the funding of the SDAIA-KAUST Center of Excellence in Data Science and Artificial Intelligence (SDAIA-KAUST AI).
References
[1] Ravi Aggarwal, Viknesh Sounderajah, Guy Martin, Daniel SW Ting, Alan Karthikesalingam, Dominic King, Hutan Ashrafian, and Ara Darzi. Diagnostic accuracy of deep learning in medical imaging: systematic review and meta-analysis. NPJ digital medicine, 4(1):65, 2021.
[2] Anonymous. Faithful vision-language interpretation via concept bottleneck models. In The Twelfth International Conference on Learning Representations, 2024.
[3] Pietro Barbiero, Gabriele Ciravegna, Francesco Giannini, Mateo Espinosa Zarlenga, Lucie Char- lotte Magister, Alberto Tonda, Pietro Lio, Frederic Precioso, Mateja Jamnik, and Giuseppe Marra. Interpretable neural-symbolic concept reasoning. In Proceedings of the 40th Interna- tional Conference on Machine Learning, volume 202, pages 1801-1825. PMLR, 2023.
[4] Daniel Beechey, Thomas MS Smith, and Özgür Şimşek. Explaining reinforcement learning with shapley values. In International Conference on Machine Learning, pages 2003-2014. PMLR, 2023.
[5] Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C. Castro, Anton Schwaighofer, Stephanie Hyland, Maria Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez- Valle, Hoifung Poon, and Ozan Oktay. Making the Most of Text Semantics to Improve Biomedical VasionnLanguage Processing, page 1-21. Springer Nature Switzerland, 2022.
[6] Carrie J Cai, Emily Reif, Narayan Hegde, Jason Hipp, Been Kim, Daniel Smilkov, Martin Wattenberg, Fernanda Viegas, Greg Corrado, Martin C Stumpe, et al. Human-center tools for coping with imperfect algorithmss during medical decision-man making. In Proceedings of the 2019 chi conference on human factors in computing systems, pages 1-14, 2019.
[7] Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37(5), pages 5948-5955, 2023.
[8] Keyuan Cheng, Gang Lin, Haoyang Fei, Lu Yu, Muhammad Asif Ali, Lijie Hu, Di Wang, al. Multi-hop question answering under temporal knowledge editing. arXiv preprint arXiv:2404.00492. 2024.
10

---

[9] Yinglong Dai and Guojun Wang, Analyzing tongue images using conceptual alignment deep autoencoder. IEEE Access, 6:5962-5972, 2018.
[10] Roxana Daneshjou, Kailas Vodrahalli, Roberto A Novoa, Melissa Jenkins, Weixin Liang, Veronica Rotemberg, Justin Ko, Susan M Swetter, Elizabeth E Bailey, Olivier Gevaert, et al. Disparities in dermatology ai performance on diverse, curated clinical image set. Science advances, 8(3 :ebb))ebb,,147, 2022.
[11] Roxana Daneshjou, Mert Yuksekgonul, Zhuo Ran Cai, Roberto A. Novoa, and James Zou. Skincon: A skin disease dataset densely annotated by domain experts for fine- tie-grained debugging and analysis. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[12] Jie Du, Kai Guan, Yanhong Zhou, Yuanman Li, and Tianfu Wang. Parameter-free similarity- aware attention module for medical image classification and segmentation. IEEE Transactions on Emerging Topics in Computational Intelligence, 2022.
[13] Marzyeh Ghassemi, Luke Oakden-Raynet and Andrew L Beam. The false hope of current approaches to explainable artificial intelligence in health care. The Lancel Digital Health, ((1)):775--770, 2021.
[14] Praveen Govi. CoronaHack- Chehhh X- E-Raa-DDttsee 2020.
[15] Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri. Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1820-1828, 2021.
[16] Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmentation. Advances in Neural Information Processing Systems, 35:1140-1156 2022.
[17] Petr Hájek. Metamathematics of fuzzy logic, volume 4. Springer Science & Business Media, 2013.
[18] Marton Havasi, Sonali Parbhoo, and Finale Doshi-Veler. Addressing leakage in concept bottleneck models. Advances in Neural Information Processing Systems, 35:23386-23397, 2022.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, 2015.
[20] Andreas Holzinger, Georg Langs. Helmut Denk, Kurt Zatloukal, and Heimo Müller. Causability and explainability of artificial intelligence in medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 94): 1312. 2019.
[21] Yihuai Hong, Yuelin Zou, Lijie Hu, Ziqian Zeng, Di Wang, and Haiqin Yang. Dissecting fine-tuning unlearning in large language models. arXiv preprint arXiv:2410.06600 2024.
[22] Guimin Hu, Zhihong Zhu, Daniel Hershcovich, Hasti Seifi, and Jiayuan Xie. Unimeec: Towards unified multimodal emotion recognition and emotion cause. arXiv preprint arXiv:2404.0044 00403, 2024.
[23] Lijie Hu, Tianhao Huang, Huanyi Xie, Chenyang Ren, Zhengyu Hu, Lu Yu, and Di Wang. Semi-supervised concept bottleneck models. arXiv preprint arXiv:2406. 18992, 2024.
[24] Lijie Hu, Tianhao Huang, Lu Yu, Wanyu Lin, Tianhang Zheng, and Di Wang. Faithful interpre- tation for graph neural networks. arXiv preprint arXiv:2410.06950. 2024.
[25] Lijie Hu, Liang Liu, Shu Yang, Xin Chen, Hongru Xiao, Mengdi Li, Pan Zhou, Muhammad Asif Ali, and Di Wang. A hopfieldian view-based interpretation for chain-of-thought reasoning. arXiv preprint arXiv:2406.1 12255, 2024.
(26) Lijie Hu, Yixin Liu, Ninghao Liu, Mengdi Huai, Lichao Sun, and Di Wang. Improving faithfulness for vision transformers. arXiv preprint arXiv:2311.1798, 2023.
11

---

[27] Lijie Hu, Yixin Liu, Ninghao Liu, Mengdi Huai, Lichao Sun, and Di Wang. Seat: stable and explainable attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37(11), pages 1 12907-12915, 2023.
[28] Lijie Hu, Chenyang Ren, Zhengyu Hu, Cheng-Long Wang, and Di Wang. Editable concept bottleneck models. arXiv preprint arXiv:2405. 15476, 2024.
[29] Qiongjie Hu, Hanxiong Guan, Ziyan Sun, Lu Huang, Chong Chen, Tao Ai, Yueying Pan, and Liming Xia. Early features and temporal lung changes covid-19 pneumonia in wuhan, china. European journal of radiology, 128:109017, 2020.
[30] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[31] Daniel S. Kermany, Daniel S. Kermany, Michael H. Goldbaum, Wenjia Cai, Carolina C. S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang. Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei, Jacqueline Pei, Magdalene Yin Lin Ting, Jie Zhu, Christina M. Li, Sierra Hewett, Sierra Hewett, Jason Dong, Ian Ziyar, Alexander Shi, Runze Zhang, Lianghong Zheng, Rui Hou. William Shi, Xin Fu, Xin Fu, Yaou Duan, Viet Anh Nguyen Huu, Viet Anh Nguyen Huu, Cindy Wen. Edward Zhang, Edward Zhang, Charlotte L. Zhang, Charlotte L. Zhang, Oulan Li, Oulan Li, Xiaobo Wang. Michael A. Singer, Xiaodong Sun, Jie Xu, Ali R. Tafreshi, M. Anthony Lewis, Huimin Xia, and Kang Zhang, Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell, 172:1122-1131..d 2018.
[32] Mert Keser, Gesina Schwalbe, Azarm Nowzad, and Alois Knoll. Interpretable model-agnostic plausibility verification for 2d object detectors using els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3890-3899, 2023.
concept bottleneck mod-
[33] Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. arXiv preprint arXiv:2306.01574 2023.
[34] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions, 2020.
[35] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning, pages 5338-5348 PMLR, 2020.
[36] Songning Lai, Lijie Hu, Junxiao Wang, Laure Berti-Equille, and Di Wang. Faithful vision- language interpretation via concept bottleneck models. In The Twelfth International Conference on Learning Representations, 2023.
[37] Jia Li, Lijie Hu, Zhixian He, Jingfeng Zhang, Tianhang Zheng, and Di Wang. Text guided image editing with automatic concept locating and forgetting. arXiv preprint ar arXiv:2405. 19708, 2024.
[38] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[39] Anita Mahinpei, Justin Clark, Isaac Lage, Finale Doshi Velez, and Weiwei Pan. Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106.13314, 2021.
[40] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capa- bilities of gpt-4 on medical challenge problems, 2023.
[41] Tuomas Oikarinen, Subhro Das, Lam M Nguyen, and Tsui-Wei Weng. Label-free concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2023.
12

---

[42] Georgina Peake and Jun Wang. Explanation mining: Post hoc interpretability of latent factor models for recommendation systems. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 2060-2069, 2018.
[43] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature machine intelligence, 1(5):206-215, 2019.
[44] Anirban Sarkar, Deepak Vijaykeerthy, Anindya Sarkar, and Vineeth N Balasubramanian. A framework for learning ante-hoc explainable models via concepts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10286-10295. 2022.
[45] Yoshihide Sawada and Keigo Nakamura. Concept bottleneck model with additional unsuper- vised concepts. IEEE Access, 10:41758-41765, 2022.
[46] Fahad Shamshad, Salman Khan, Syed Waqas Zamir, Muhammad Haris Khan, Munawar Hayat, Fahad Shahbaz Khan, and Huazhu Fu. Transformers in medical imaging: A survey, Medical Image Analysis, page 102802, 2023.
[47] Ivaxi Sheth and Samira Ebrahimi Kahou, Auxiliary losses for learning generalizable concept- based models. In Thirty- Conference Neural Information Processing Systems, 2023.
[48] K Simonyan and A Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In 3rd International Conference on Learning Representations (ICLR 2015). Computational and Biological Learning Society, 2015.
[49] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Publisher correction: Large language models encode clinical knowledge. Nature, 620(79733:19-19, 2023.
[50] Sanchit Sinha, Mengdi Huai, Jianhui Sun, and Aidong Zhang. Understanding and enhancing robustness of concept-hasee based models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15127-15135, 2023.
[51] Zhiyong Tan, Yuhai Yu, Jiana Meng, Shuang Liu, and Wei Li. Self-supervised learning with self-distillation on covid-19 medical image classification. Computer Methods and Programs in Biomedicine, 243: 107876, 2024.
[52] Omkar Thawkar, Abdelrahman Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Shahbaz Khan. Xraygpt: Chest radiographs summarization using medical vision-language models. arXiv preprini arXiv:2306.07977 2023.
[53] Vo Hoang Thien, Vo Thi Hong Tuyet, and Nguyen Thanh Binh. Object contour in medical images based on saliency map combined with active contour model. In 8th International Conference on the Development of Biomedical Engineering in Vietna: Proceedings of BME 8, 2020, Vietnam: Healthcare Technology for Smart City in Low-and Middle-Income Countries, pages 717-732. Springer, 2022.
[54] Sana Tonekaboni, Shalmali Joshi, Melissa D McCradden, and Anna Goldenberg. What clinicians want: contextualizing explainable machine learning for clinical end use. In Machine learning for healthcare conference, pages 359-380. PMLR, 2019.
[55] Bas HM Van der Velden, Hugo J Kuijf, Kenneth GA Gilhuijs. and Max A Viergever. Explainable artificial intelligence (xai) in deep learning- based medical image analysis. Medical Image Analysis, 79: 79:107,70, 2022.
[56] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[57] Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. Scientific discovery in the age of artificial intelligence. Nature, 620(7972):47-60, 2023.
13

---

[58] Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. Chatcad: Interactive computer-aided diagnosis on medical image using large language models. arXiv preprint arXiv:2302.07257, 2023.
[59] Mengying Xiao, Liyuan Zhang, Weili Shi, Jianhua Liu, Wei He, and Zhengang Jiang. A visualization method based on the grad-cam for medical image segmentation model. In 2021 In- ternttional Conference Electronic Information Engineering and Computer Science (EIECS), pages 242-247. IEEE, 2021.
[60] Yao Xie, Melody Chen, David Kao, Ge Gao, and Xiang' Anthony' Chen. Chexplain: enabling physicians explore and understand data-driven, ai-enabled medical imaging analysis. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1-13, 2020.
[61] Haoxuan Xu, Songning Lai, Xianyang Li, and Yang Yang. Cross-domain car detection model with integrated convolutional block attention mechanism. Image and Vision Computing. 140: 104834, 2023.
[62] An Yan, Yu Wang. Yiwu Zhong, Zexue He, Petros Karypis, Zihan Wang, Chengyu Dong, Amilcare Gentili, Chun-Nan Hsu, Jingbo Shang, et al. Robust and interpretable medical image classifiers via concept bottleneck models. arXiv preprint arXiv:2310.03182 2023.
[63] Shu Yang, Muhammad Asif Ali, Cheng-Long Wang. Lijie Hu,, and Di Wang. Moral: Moe augmented lora for Ilms' lifelong learning. arXiv preprint 210::2242..12600 2024.
[64] Shu Yang, Lijie Hu, Lu Yu, Muhammad Asif Ali, and Di Wang. Human-ai interactions in the communication era: Autophagy makes large models achieving local optima. arXiv preprint arXiv:2402.11277. 2024.
[65] Shu Yang. Shenzhe Zhu, Ruoxuan Bao, Liang Liu, Yu Cheng, Lijie Hu, Mengdi Li, and Di Wang. What makes your model a low-empathy or warmth person: Exploring the origins of personality in llms. arXiv preprint arXiv:2410. 10863, 2024.
[66] Mert Yuksekgonul, Maggie Wang, and James Zou, Post-hoc concept bottleneck models. The Eleventh International Conference Learning Representations, 2022.
[67] Mateo Espinosa Zarlenga, Pietro Barbiero. Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Zohreh Shams, Frederic Precioso, Stefano Melacci, Adrian Weller, al. Concept embedding models. arXiv preprint arXiv:2209.09005 2022.
[68] Jiajin Zhang, Hanqing Chao, Giridhar Dasegowda, Ge Wang, Mannudeep K Kalra, and Pingkun Yan. Overlooked trustworthiness of saliency maps. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 451 351-461. Springer, 2022.
[69] Yi Zhang, Mingming Jin, and Gang Huang. Medical image fusion based on improved multi- scale morphology madient-weigghed local energy and visual saliency map. Biomedical Signal Processing and Control, 74:103535, 2022.
[70] Bolei Zhou, Aditya Khosla, Agata I Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2921-2929, 2016.
14

---

A Concept Set Generation and Filtering
A.1 Concept Generation with GPT-4V
In this section, we will explain process for generating concept sets using GPT-4V. Specifically, we ask GPT-4V the following:
[User]:
List the useful visual attributes to recognize [class) in a (type) in this (image]:
List the things most commonly seen around a (class) in this (image]:
List in bullet points with no extra context.
Note that the image here refers to the images we random sample from the dataset, and type represents the modality of medical image (e.g. X-rays, CT). Full prompts and example outputs illustrated in the appendix D.2 Combining the concepts received from different classes and images gives us large, somewhat noisy set of initial concepts, which we further improve by filtering. We found using GPT-4V to generate initial concepts to perform better than using GPT-3, which was used in Label-free CBM 41. We initiate the procedure with the following conversation:
[User]: List the useful visual attributes to recognize COVID in a CT in this image. [GPT-4V]: When assessing CT images for signs of COVID-19, radiologists often look for certain visual attributes that can suggest the presence of the disease. Here are some of the key visual attributes they may look for:
Ground-glass opacities (GGOs): These are hazy areas that do not obscure the underlying lung structures. They are called grround-glass because they resemble the appearance frosted glass.
2. Consolidation:
In addition, we can proceed with the following question:
List the things commonly seen around COVID in this image. [GPT-4V]: Based on the image you have provided and considering the common radiological findings associated with COVID-19, here the things that might be commonly seen:
[User]:
1. Possible consolidation areas.
2. Absence of significant pleural effusions.
3.
Then, we can use the following inquiry to structure the output from GPT-4V:
[User]: List in bullet points with no extra context. [GPT-4V]:
Ground-Glass Opacities (GGOs)
Bilateral and Multifocal Lesions
Crazy-Pavine Pattern
Reverse Halo or Atoll Sign
Abxence of Pleural Effusion
Septal Thickening
Rare Lymphadenopathy
Vascular Enlargement
Traction Bronchiectasis
15

---

After obtaining the initial concept set following the filtering process, our rules and concepts on COVID-CT and DDI datasets follows (see T)
Rules COVID-CT
DDI
Peripheral ground-glase -lass opacities
Vesicle
C1
Bilateral involvement
Papule
C2
Multilobur distribution
Exudate
Frazy-paving pattern
Fissure
Absenee of lobar consolidation
Xerosis
65
Localized or diffuse persentation
Warty
46
Increased density in the lung
Brown
Ground-glan glass appearance
Translucent
08
White
0g
Erythema
C10
Wheal
611
Pigmented
612
Cyst
Table 3: We have recorded the concept set generated through GPT-4V and subsequently cleaned through filtering. This concept will be used in our training process.
A.2 Concept Set Filtering
After concept generation, concept with noise be obtained. The following filters set to enhance the quality of the concept:
1. The length of concept: To keep concepts simple and avoid unnecessary complication, We remove concepts longer than 30 characters in length.
2. Similarity: We measure this with cosine similarity in text embedding space. We removed concepts too similar to classes each other. The former conflicts with our interpretability goals, while the latter leads to redundancy in concepts. We set the thresholds for these two filters 0.85 and 0.9, respectively, ensuring that their similarities are below our threshold.
3. Remove concepts we cannot project accurately: Remove neurons that are not interpretable from the Bio (5). This step actually described in section 4.2
B Neural-symbolic Layer
We give the details with examples for neural-symbolic layer
Concept Polarity. For each prediction class j. there exists a neural network (,))). This network takes each concept embedding input and produces soft indicator, scalar value in the [0, 1] range. This soft indicator represents the role of the concept within the formula. As an illustration, consider specific concept like "Crazy-paving pattern". If its value after passing through (.)) is 0.8, it indicates that the "Crazy-paving pattern" has a positive role with a score of 0.8 for class j. We use the notation lori to represent the soft indicator for concept C..
Concept Relevance. For each prediction class j. a neural network V (-(-) is utilized. This network takes each concept embedding input and produces soft indicator, scalar value within the range of [0, 1), representing the relevance score within the formula. To illustrate, let us consider a specific concept such "Multilobar distribution". If its value after passing through , (-) is 0.2, it implies that the relevance score of "Multilobar distribution" in the inference of class j is 0.2. We denote the soft indicator for concept 6; Tr,ij-
Prediction via Concept Rules. Finally, for each class j. we combine the previous concept polarity vector Lo and the concept relevance vector Lng to obtain the logical inference output. This is
16

---

achieved by the following expression:
f - (- V Injis) min {max Loci.d 4r503 [EIN]
Intuitively, the aforementioned concept reasoning rules allow us to determine the relevance of each concept and whether a concept has positive or negative role in predicting the label as class Intuitively -Ta,.j Tr.i.j means if it is irrelevant, then we set it to 1, and if it is relevant, then we set it to be the relevance score. It is strange at first glance. However, since finally we will take the conjunction (or minimum) for all concepts to get Új, so we will filter the i-th concept -Loos V Ir, = 1, i... if it is irrelevant. Thus, we take the neg here. Since need to consider each concept for the final prediction, we finally use for concepts.
C More Experimental Setup
C.1 Training Setting
Our model exhibits remarkable efficiency in its training process. We utilized only a single GeForce RTX 4090 GPU, and the training duration did not exceed half hour. We configured the model to run for 100 epochs with a learning rate set at 5e-5. Additionally, all images were resized to a uniform dimension of (256, 256).
C.2 Datasets
COVID-CT. The COVID-CT dataset was obtained from 22 and comprises 746 CT images, consisting of two classes (349 COVID and 397 NonCOVID). We divided this dataset into training set and a test set with an 8:2 ratio, and the data were split accordingly.
DDL. DDI 10 is dataset comprising diverse dermatology images designed to assess the model's ability to correctly identify skin diseases. It consists of a total of 656 images, including 485 benign lesion images and 171 malignant lesion images. These images are divided into training and test sets, with an 80% and 20% split, respectively.
Chest X-Ray. The Chest X-Ray 14 dataset comprises 2D chest X-ray images of both healthy and infected populations. It aims to support researchers in developing artificial intelligence models capable of distinguishing between chest X-ray images of healthy individuals and those with infections. The dataset includes 5933 images, divided into 5309 training images and 624 testing images.
Fitzpatrick17k. Fitzpatrick 17k (15] dataset is dermatological dataset that includes wide range of skin colors. In order to better compare the model performance, we filtered the malignant and nonmalignant classes 3230 images that have been relabeled by SkinCon [I]]. And we divided the training and test set according to an 8:2 ratio.
C.3 Baseline Models
SSSD-COVID. SSSD-COVID incorporates Masked Autoencoder (MAE) for direct pre-training and fine-tuning on small-scale target dataset. It leverages self-supervised learning and self- distillation techniques for COVID-19 medical image classification, achieving performance levels surpassing many baseline models. Our method is trained exclusively on the COVID-CT dataset without considering the effects of introducing knowledge from other datasets. Notably, our approach outperforms SSSD-COVID in terms of performance. Furthermore, SSSD-COVID falls under the category of black-box models, indicating that our method, to some extent, overcomes the accuracy degradation issue introduced by the incorporation of concepts.
Label-free CBM. Label-free CBM is fully automated and scalable method for generating concept bottleneck models. It has demonstrated outstanding performance on datasets like ImageNet. In our comparisons, our model outperforms this model significantly in terms accuracy. Regarding interpretability, our model not only possesses the same level of interpretability as this model but also
17

---

provides explanations in multiple dimensions, such as concept reasoning and saliency maps. This helps doctors gain a more diverse and precise understanding of the model's decision decision-making process when using our model.
DCR. Deep Concept Reasoner (DCR), a concept-based model combining neural-symbolic meth- ods. has demonstrated promising interpretability performance on simple datasets such as XOR, Trigonometry. and MNIST-Addition, However, its performance tends to degrade on more complex or challenging datasets.
C.4 Definition of Metries
Accuracy is the ratio of the number of correctly categorized samples to the total number samples:
TP TN
Acc
TP+ TN + FN
Where TP denotes true positive, TN denotes true negative, FP represents false negative, and FN represents false negative.
The precision is the proportion of all samples classified as positive categories that are actually positive categories:
TP
Precision
TP + FP
Recall the proportion of samples that are correctly categorized positive out of all samples that are actually positive categories:
TP TP + FN
Recall
The Fl score is the reconciled mean of precision and recall:
2 Precision Recall
Precision + Recall
AUC denotes the area under the ROC curve, which is a curve with True Positive Rate (Recall Rate) the vertical axis and False Positive Rate (False Positive Rate) the horizontal axis.
D
More Experimental Results
D.1
Utility Evaluation
We provide our detailed utility evaluation for four datasets in Table 5 and
Method
Backbone
Acc.(%)
Precision( %)
Recall %
F1(%)
AUC.(%)
Interpretabiliit
ResNet50
81.36
82.28
81.44
81.67
81.85
VGG19
79.60
81.82
78.93
79.88
80.26
Baseline
66nseNet169
85,59
85.60
85.60
85,59
85.60
SSSD-COVID
81.76
81.82
78.26
80.00
88.21
Label Free CBM
69.49
68 18.62
69.82
69.21
64,84
DCR
55.93
58.38
55,43
51.41
55.43
ResNer50
84.75
84.77
84.88
84.75
84.77
Ours
VGG19
83.05
86.74
82.93
84.37
84.26
DenseNet169
86.44
87.27
86.41
87.15
87.92
Table 4: Results for COVID-CT. We conducted ten repeated experiments and calculated the average results for each metric. Red and blue indicate the best and the second-best result. Our approach outperforms the baseline backbone models while also providing interpretability. When compared to other state-of-the-art interpretable models, our model exhibits significantly higher accuracy.
18

---

Method
Backbone
Acc(%%
Precision( %)
Recall( %)
F1(%)
AUC-(%)
Interpretability
ResNe:50
77.27
72.37
73.19
72.77
72.51
VGG19
76.52
72.92
68.54
70.12
68.80
Baseline
DenseNet169
78.03
74.37
67.41
69.51
68.76
Label Free CBM
70.34
68.62
69.82
69.21
69.49
DCR
76.52
71.79
63.88
65.32
63.88
ResNet50
81.82
76.56
76.17
76.33
76.12
Ours
VGGI9
82.58
81.59
76.05
78.07
75.63
DenseNer169
79.55
77.68
67.64
69.79
67.64
Table 5: Results for DDI. We conducted ten repeated experiments and calculated the average results for each metric. Red and blue indicate the best and the second-best result. Our approach outperforms the baseline backbone models while also providing interpretability. When compared to other state-of- the-art interpretable models, our model exhibits significantly higher accuracy.
Method
Backbone
Acc. %)
Precision( %)
Recall
F1(%)
AUC.(%)
Interpretability
ResNet50
75.64
75.01
70.77
71.72
70.88
x
VGG19
81.41
88.56
75.51
77.56
75.94
x
Baseline
NenseNer169
69.55
70.37
62.05
61.66
62.12
Label Free CBM
71.21
71.89
71.45
70.84
74.12
62.02
51.50
41.33
50.56
ResNet50
78.37
80.38
73.12
74.42
73.12
Ours
VGGI9
88.30
92.59
85.43
88.16
87.09
DenseNer169
73.88
81.24
65.85
65.70
66.28
Table 6: Results for Chest X-Ray. We conducted ten repeated experiments and calculated the average results for each metric. Red and blue indicate the best and the second becond-best result. Our approach outperforms the baseline backbone models while also providing interpretability. When compared to other state-of-the-att interpretable models, our model exhibits significantly higher accuracy.
Method
Backbone
Acc.(%
Precision(
Recall(
FI(%)
AUC(%)
Interpretability
ResNet50
80.79
80.81
80.81
80.79
80.81
x
VGG19
75.37
75
75.34
75.37
75.39
Baseline
DenseNet169
76.85
77.05
76.91
76.83
76.91
Label Free CBM
75.24
75.15
74.92
75.41
75.02
DCR
68.05
67.55
65.33
66.12
67.01
ResNet50
82.76
82.84
83.23
83.03
82.99
Ours
VGG19
77.34
77.72
77.33
77.53
77.58
DenseNet169
80.79
82.12
80.89
81.11
81.38
Table 7: Results for Fitzpatrick17k, We conducted ten repeated experiments and calculated the average results for each metric. Red and blue indicate the best and the second-best result. Our approach outperforms the baseline backbone models while also providing interpretability. When compared to other sfate-of-the- the-aar interpretable models, our model exhibits significantly higher accuracy.
19

---

D.2 Visualization of Concept Predictions
More samples of instance-level predictions for COVID-CT and DDI datasets are visualized in Figure 10 and [2] We also append the corresponding rules to it, which can reflect the multidimennionall interpretation better. Taking the COVID-CT sample as an example, can be found that the model predicts the concept score prediction accurately, and for the COVID samples, the correlation of the first three concepts is greater, which leads to a higher prediction score of their concepts, thus generating real concept rules, and then assists the model judgment. This can also be reflected in the saliency map of the model in the inference stage, where the phenomena described by the relevant concepts can get higher attention in the saliency map, thus further reflecting the multidimensional interpretation of Med-MICN.
5
6
D.3 Ablation Study: Effect of Concept Filters
In this section, we will discuss how each step in our proposed concept filtering affects the results of our method. In general, our utilization filters has two main goals: First, improving the interpretability of our models. Second, improving computational efficiency and complexity by reducing the number of concepts.
Although our initial goal was not to improve model accuracy (a model with more concepts is generally larger and more powerful 4T). the more precise concepts after filtering make the concatenated features more effective for classification and slightly improve the accuracy. To evaluate the impact of each filter, we trained our models separately on COVID-CT and DDI while removing one filter at time and one without using any filter at all. The results are shown in the Table 8 From Table 88 it noticeable that our model's accuracy does not exhibit high sensitivity to the choice of filters. On the COVID-CT dataset, the accuracy of our model remains largely unaffected by the choice of filters. Furthermore, employing filters on the DDI dataset results in improved model accuracy. This phenomenon arises due the relatively sparse nature of concepts within the DDI dataset images, where increasing the number of concepts did not yield superior solutions.
COVID-CT
DDI
Filters
Accuracy(%)
#concept
Accuracy(%5
#concept
All filters
86.44
82.58
13
No length filter
86.32
82.53
16
No similarity filter
86.43
82.46
26
projection filter
86.21
81.68
No filters at
86.19
81.60
49
Table Effect of our individual concept filters on the final accuracy and number of concepts utilized by our models.
D.4 Sensitivity Analysis
We also performed a sensitivity analysis for baselines and Med-MICN in the DDI dataset. we set various attacks under 5 4/255, 6/ 6/255, 8/255, 10/255, 12 22/255 and attack radius pa {0, 2/255, 4/255, 6/255, 8/255, (./255). in Figure 14 the results show that our model consists of stronger robustness against perturbation. This is evidenced by the marginal decrease in test accuracy as the attack radius increases. The model's 's detection performance fluctuates slightly at Pa of 4/255, with this variability diminishing the perturbation level rises, underscoring the robustness of our model against significant disturbances.
D.5
Computational Cost
We conducted a computational cost analysis for Med-MICN and the baseline models. By inputting random tensor of size (1, 3, 244, 224) into the model and computing the FLOPs and parameters, the results are presented in Table Experimental evidence demonstrates that Med-MICN incurs only negligible computational cost compared to the baseline models while it achieves improvements in accuracy and interpretability.
20

---

Ground-glass appearance
mincreased density the Localized or diftuse persentation
0.12
10.15
Absence lobar consolidation
Crazy-paving pettern
0000
Image NonCOVID
Multilobar distribution
0.12
Peripheral ground-giass opacities
0.33
0.0
0.1
0.2
0.3
Concept preditions
Yo = 76y A m6, A -6 A - A 7c A -c A -c A -67
Sallency map
Concept reasoning
Ground glassd-glass appearance 0006
mincreased density in the lung
Localized or diffuse persentation 0.04
Absence of lobar consolidation
00008
Crazy-paving pattern
class NonCOVID
Multilobar distribution
0.04
Bilateral involvement
0.11
Peripheral ground-glass
0.07
Concept preditions
Yn + Ccn A mc; A mcy A mc A mcs A mcg A mCy
Saliency map
Concept reasoning rules
Figure 5: Samples classified as NonCOVID.
Ground-g mincreased density in the lung 0.09
0.07
persentation
Absence of lobar consolidation
0.71
Crazy-paving pattem
0..55
daass COVID
Multilcbar distribution
Bilateral invalvement Peripheral ground-gless opacities
0.69
0.82
0.0
0.2
0.4
0.6
0.8
Concept preditions
cp Ac,
Saliency map
Concept reasoning
Ground-g jass appearance 0.06
mincreased densiity in the lung
Localized persentation
0.05
Absence of lobar consolidation
0.18
Crazy paving pattem
0.29
Image class COVID
0.68
allaterai Involvement
0.81
Peripheral ground glass opacities
0.0
0.2
0.4
0.6
0.8
Concept preditions
Y12 Co Ac,Acz
Saliency map
Concept reasaning rules
Figure 6: Samples classified COVID.
21

---

Ground-gless appeerance mincreased density the Localized or dilluse persentation
0.33
Absence of lobar consolidation Crazy-paving pattern
0.10
Image class NonCOVII
Multilobar distribucion 80.01
Bllateral involvement
Paripheral ground-glass opacities
0.33
0.0
0.1
0.2
0.3
Concept preditions
Yo = -6p A mc még A ma A mC4 / mCs A mCg m6p
map
Concept ressaning rules
Ground-glass appearance 0.07
mincreased density lung
Localized difluse persentation
0.13
Absence lobar consolidation
Crazy-paving pattern Multilobar distribution
0.43
Image class NonCOVII
Bllateral involvement
0.05
Peripheral ground-glass opacities
Concept praditions
Yo A mG A ma mEy A
Saliency map
Concept reasoning rules
Figure 7: Samples classified as NonCOVID.
Ground glass appearance 00.06
mincreased density lung
Localized diffuse persentation
00.07
Absence lobar consolidation
paving pattern 10
00.03
class COVID
Multilobar distribution
0.71
involvement
0.05
Peripheral ground-gless opacities
0.86
0.0
0.2
0.4
0.6
0.8
Cancept precitions
Y1 = do Neg Acg
Saliency map
Concept reasoning rules
Ground-glase glass appearance |00.03
mincreased density lung
Locelized diffuse persentation Absance lobar consolidation
0.25
0.68
Crazy-paving pattern
0..13
Image class COVID
Multilebar distribution
0.81
invalvement Peripheral ground-glan glass opacities
0.76
0.0
0.2
0.4
0.6
0.8
Concest preditions
Y1 AcgAAc
Saliency map
Concapt reasoning rules
Figure 8: Samples classified as COVID.
Method
Flops(M)
Params(M)
ResNet50
4133.74
25,56
VGG16
15470.31
138.36
DenseNet169
3436.12
14.15
Med-ICNSS(ReNNe500)
4134.79
26.60
Med-ICNS(VVGi6)
15471.36
139.40
Mcd-ICNSSDDemsNee110) 3436.32
14.35
Table 9: Computational costs. Results indicate that Med-MICN incurs minimal computational cost
compared to the baseline model while achieving improvements in both accuracy and interpretability.
22

---

Cyst .0.09
Pigmented 0016
...16
Erythema 0.00
Translucent 00.66
Brown
00.77
Image class NonMalignant
Xarasis 00.01
Rasure m0.03
Papule
11
Vesicle
Concept
Yo
mby
moy
Saliency map
Concept nules
Cyst 0.07
Pigmented
Wheal 0.06
Erythama
0.74
m0.03
Transucent 0.06
Brown 0.05
Xarasis 0.07
Image class NonMallgnant
Fissure 0.04
Papule
0..1
Concept Preditions
Yo mlg A mtg A may me, A mi; A mey A may mitgh mesh miya m6iz
Saliency map
Concept reasoning rules
Figure 9: Samples classified as NonMalignant.
Pigmented 0.06
Erythema
00.73
White Translucent 10.01 Brown Warty 0003
0.6
Malignant
Rosure 0.13
Exudate 0.11
Vesicle 0..01
0.2
Concept Preditions
6 Ac,
Saliency map
Concept reasoning
00000 0.07
Pigmented
01
Wheal
Erythern =0 0.04
White m0.03
Brown 0.11
0.01
Warty 00.04
Rispure 0.09
Image class Malignant
Exudate
0.11
Vesicle
00.01
0.0
0.2
0.6
0.8
Concept Preditions
Y1 A.
Saliency map
Concept reasoning rules
Figure 10: Samples classified as Malignant.
23

---

Cyst
00.06
Pigmented
Wheal
0.07
Erythema
0.01
Transiucent
0.01
Brown
0.04
Image class NonMalignant
Xerasis
0.04
Exudate
0.05
Papule
0.12
0.10
Preditions
may
A-c;
mo,h - -8 mG
Saliency map
Concept reasaning
Cyst
0.29
Rigmented
0.13
Erythama 0.03
White 0.03
Brown
0.05 05
Image class NonMalignant
Warty
0.03
fissure
0.12
Exudate
0.07
Papule
0.24
0.0
0.1
0.2
0.3
Concept Preditions
-0y Amti -sy A mt, A mt, -c; may A mish nowh Ai
Saliency map
Concept reasoning rules
Figure 11: Samples classified as NonMalignant.
Cyst
0.76
Pigmeniled 0.05
Erythema 10.03
White
Translucent 10.02
Brown
0.41
Xerosis 0 0.04
Image class Malignant
Alssure 0.002
Papule
0,22
Vesicle
Concept
Y1 44
Concept reasoning
Sallency map
00007 0.07
Pigmented
01
Wheal
Erythern =0
0.04
White m0.0
Brown
0.01
Warty 00.04
0.11
Image Malignant
Rispure 0.09
Exudate
0.11
Vesicle
00.01
0.0
0.2
0.6
0.8
Concept Preditions
Y1 A.
Sallency map
Concept reasoning rules
Figure 12: Samples classified as Malignant.
24

---

vecite 6.001
Peripheral ground-ght gounddglass opacities:
papule: 0.183
exudate: 0.892
consolidation.
fleure: 0.061
Locelized
0.225
COVID: 3.521
warty
brown: 0.209
cazy-gpvvig pattern: 0.895
White: 0.227
Malignant: 3.702
pigmented: 0.934
arythema:
Multloburr distrbuttorr 8.109
0.116
minoreased density in the lungi 0.158
translecnn:: 0.571
Ground glass appearance: 0.005
COVID
SKIN
Figure 13: Interpretation of learned linear weights for COVID-CT (left) and DDI (right) dataset. The model can discover concepts that are crucial for classification.
0.82
0.80
Accuracy
0.78
0.75
0.74
Baseline
Ours-4/255
0.72
Ours-8/255
Our5-10/255
Ours-12/255
10
Atack Radius(/255)
Figure 14: Sensitivity analysis.
E
Limitation
In the case of the "neural symbolic" method, additional acceleration techniques may be required when dealing with large sample sizes. However, it is worth noting that medical datasets tend to be relatively small.
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims presented in the abstract and introduction accurately reflect the paper's contributions and scope. We proposed end-to-end framework called Med- MICN, which leverages the strengths of various XAI methods such as concept-base based models, neural symbolic methods, saliency maps, and concept semanties. In Section 1 and Abstract. Guidelines:
Justification:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: in Section B
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
26

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have provided comprehensive exposition of our network architecture, encompassing experimental parameter details. Additionally, our dataset is publicly available. Thus, our methodology exhibits reproducibility. In Section and Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
27

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: Code [Yes] DATA [No]
Justification: We will open-source the code, and the datasets used publicly available for download by anyone. Section C and Section 5.1
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nnis.cc)
aablic/guudees/Codeebbbmsss caspssionoolyyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppplclguuiieeeeaec
Stteee//CooSSSemsssS.Si.a www.ssssiicggyoo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All detailed specifications are presented in the experimental section, In Section C and Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conducted ten repeated experiments and calculated the average results for each metric. In Section Clddd Section 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We detailed this in Section C.I and Section D.5
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https::/neurr/ httppppp...ppp..../iccceeeeeeeaa
Answer: [Yes]
Justification: Our research fully complies with the NeurIPS Code of Ethics. We have carefully reviewed and ensured adherence to all relevant standards.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We pointed out the positive role our model will play in assisting doctors in making decisions regarding diseases in the future. Section T and Section E
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Yes, we have cited the original papers that produced the code packages or datasets, provided the version information and URLs when possible, and included the name of the license for each asset. Section C and Section 5.I
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
30

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "76Wno82Gsj", "reviewer": "Reviewer_Lw8m", "summary": "The authors proposed a novel interpretable model called Med-MICN in this work. This method manages medical image classification tasks with multi-dimensional aspects with neural symbolic and concept semantics. With the help of LLMs, the work performed superior on four medical benchmark datasets. In the ablation study, the authors presented that the performance of the proposed method performed better with the complementary multi-dimensional loss functions in evaluating the classification result.", "questions": "- The authors used LLM (GPT-4V) in their proposed method. Would it be better or make a difference, if the LLM is a more medical-specific model?\n\n- How do authors deal with different numbers of concept sets from the LLM? If I understood correctly, the LLM could output several different concepts and does not necessarily fit into the input size of the text encoder. (Figure 2.a)\n\n- Did authors try with different $\\lambda_1$ and $\\lambda_2$?", "limitations": "- The proposed method provides an opportunity to interpret the medical image classification results and in the meanwhile also increase the classification performance. However, heatmaps look somehow blurry and may not be trustable (Fig 1).", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper proposed a novel method that combines extra information (text and logic rules) to enhance the classification performance and the interpretability of the model outputs. This enables the interpretability of the deep learning classification model with multi-dimensional information.\n- Instead of post-hoc interpretable methods, the authors integrate fuzzy logic rules into the proposed method. This gives a clear decision rule for image classification and interpretability.\n- The method description is relatively clear and easy to follow. The ablation study gave a good overview of the usage of multi-dimensional information.", "weaknesses": "- The authors show that the proposed method performed superior to the baseline methods in image classification (Tab. 1). However, there is a missing part on the evaluation of the interpretability. How correct are those heatmaps generated from the proposed model, do they align with the labels or the clinicians?\n\n- Also the comparison of the interpretability between the proposed method and the baseline methods is missing. In Tab. 1, the authors mentioned those methods are interpretable, but they did not evaluate the interpretability. This could be crucial for medical applications.\n\n- For section 4.3, the description of Neural-Symbolic Layer is not easy to follow. What is the difference between the functions of \"Concept Polarity\" and \"Concept Relevance\"? It seems like their outputs are the same.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' recognition of our work and your constructive comments. Please find below our detailed responses to the queries you have raised. We hope you could consider increasing our overall score if our response addresses your concerns :)\n\n> W1, W2: \"evaluation of the interpretability\"\n\n**Response**: The questions you raised are very insightful. Although quantifying interpretability remains a challenging task with no well-established metrics, our extensive examples demonstrate that the interpretability information provided by our model is meaningful. This is supported by numerous examples presented in our paper, particularly in Figures 1 and 5-12. While aligning saliency maps with features has been a longstanding issue in interpretability, we have compared Med-MICN with baseline saliency maps. As illustrated in Figure 1 of the attached PDF, we used the most distinctive skin modality for clarity. The results show that our method better highlights pathological regions and achieves a significant improvement in alignment compared to the baseline.\n\n> W3: \"For section 4.3, the description of Neural-Symbolic Layer is not easy to follow....\".\n\n**Response**: *Concept Polarity* focuses on determining the role of a concept as either positive or negative for a given prediction class. The output of the neural network $\\Phi_j(\\cdot)$ is a soft indicator value between 0 and 1, which signifies the degree to which the concept contributes positively (closer to 1) or negatively (closer to 0) to the prediction. For example, if the output for the concept is 0.8 for class $j$, it suggests that this concept has a strong positive influence on predicting class j.\n\n*Concept Relevance* assesses how useful or significant a concept is within the context of the sample feature for a prediction class. The neural network $\\Psi_j(\\cdot) $also outputs a soft indicator value between 0 and 1, but this value represents the concept's relevance score, indicating how much it impacts the prediction. If the output for the concept is 0.2 for class $j$, it means that this concept has a relatively low relevance to predicting class j.\n\nWhile both functions output scalar values between 0 and 1, Concept Polarity indicates the concept's positive or negative influence, and Concept Relevance measures the concept's overall importance or significance in the prediction. These two outputs are combined in Equation (2) to generate the logical reasoning rules for the concept.\n\n> Q1: \"The authors used LLM (GPT-4V) in their proposed method. Would it be better...\"\n\n**Response**: Thanks to your suggestion, we have fully considered the generalization and accuracy of different diseases when using multimodal models for questioning. Here, we have added two medical multimodal models for comparison experiments (Due to word count limitations, the results from each model will be uniformly presented in the appendix of the paper). The first model is XrayGPT, The XrayGPT model is limited by the form of training data for different kinds of disease images, such as plain view lung CT map (dataset COVID) can not output accurate medical conceptual information. Moreover, for chest CT images, its prediction about different classes of viruses such as COVID output information is also affected and lacks generalization to different disease types.\n\nBesides, we also use the better generalized Llava Med model for comparison. For the COVID dataset, it is capable of obtaining certain medical category information, however, the output disease load categories are not comprehensive compared to the methods used in this framework, and the comprehensiveness of the information.\n\n> Q2: \"How do authors deal with different numbers of concept sets from the LLM? ....\"\n\n**Response**: It is important to note that each concept within the concept set is processed individually by the text encoder to obtain a corresponding text embedding, rather than inputting the entire concept set simultaneously. The LLM handles varying input sizes through different lengths of concepts, such as \"Peripheral ground-glass opacities\" and \"Bilateral involvement.\" By obtaining the text embedding for each individual concept, we can derive a concept heatmap for each image and subsequently assign the appropriate concept labels.\n\n> Q3: \"Did authors try with different λ1 and λ2?\"\n\n**Response**: Thank you for your valuable feedback. We conducted experiments within reasonable parameter ranges for λ1 and λ2, as detailed in the table below. The table shows the accuracy of various backbones with different combinations of λ1 and λ2. When λ1 is fixed, a larger λ2 is preferred, and similarly, when λ2 is fixed, a larger λ1 is preferred. This indicates that each loss function contributes to the model’s predictive accuracy. Additionally, we need to adjust these coefficients for different backbones to achieve optimal performance. For example, with VGG, setting λ1 = 1 and λ2 = 0.1 resulted in a 7.41% increase in accuracy.\n\n\n| Dataset  | Model    | Baseline |  λ1=0.1, λ2=0.1 |  λ1=0.1, λ2=0.01 |  λ1=0.1, λ2=0.05 |  λ1=0.1, λ2=0.5 |  λ1=0.1, λ2=1.0 |  λ1=0.01, λ2=0.1 |  λ1=0.05, λ2=0.1 |  λ1=0.5, λ2=0.1 |  λ1=1.0, λ2=0.1 |\n|----------|----------|----------|:---------------:|:----------------:|:----------------:|:---------------:|:---------------:|:----------------:|:----------------:|:---------------:|:---------------:|\n|          | RN50     | 81.36    |    **84.75**    |       83.90      |       82.20      |      82.25      |      83.05      |       81.33      |       83.90      |      83.92      |      82.53      |\n| COVID-CT | DenseNet | 85.59    |      86.44      |       84.75      |       84.74      |    **87.29**    |      85.59      |       84.09      |       84.75      |      84.12      |      84.28      |\n|          | VGG      | 79.60    |      83.05      |       79.66      |       80.52      |      80.51      |      85.60      |       80.85      |       79.66      |      86.44      |    **87.01**    |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2SnRP3JqcL", "reviewer": "Reviewer_7tbr", "summary": "This work introduces Med-MICN, an explainable framework for medical image classification. This framework leverages a concept bottleneck framework combined with a neural symbolic reasoning framework to generate simple explanations for its predictions. In general, strong performance is demonstrated.", "questions": "There appear to be some notational issues in section 4.2 — initially H and W are used as the height and width of the feature map, but in the definition of $H_{p,k,i}$, it seems that P and K are used to fill the same roll.\n\n“to ensure the truthfulness of 196 concepts, we discard all concepts for which the similarity across all images is below 0.45. “ — does this refer to all training images?\n\nThe equation following line 214 is quite confusing — what is the dimensionality of each of the three values returned by $\\theta_c$? How does it return $f(x_m)$?\n\nDuring training, which components are trained jointly? It seems like $L_{neural}$ only applies to the neural symbolic component, while $L_c$ and $L_{task}$ apply only to the concept extraction component and some additional classification fully connected layer.\n\nFigure 4 is somewhat unclear — which cells in the figure should be compared to which?", "limitations": "Discussion of limitations is appropriate.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "The overall framework is interesting, and the idea of “multi-dimensional explainability” seems widely applicable. The ability to see some explanation for each concept’s logic seems useful, and the decision rules generated are quite simple. The performance of Med-MICN is quite compelling, with superior accuracy across multiple medical datasets.", "weaknesses": "In general, the notation used in the paper is inconsistent and somewhat ill defined. For example, in section 3, N is used to denote the number of sample in a dataset, and k the number of concepts. In section 4, N becomes the number of concepts, and M the number of samples. Lower case k is then used to index along the height dimension of a feature map. Several objects ($I_{o, j}$ and $I_{r,j}$, for example) are not clearly described upon their introduction.\n\nThere are a few seemingly key hyper parameters ($\\lambda_1$, $\\lambda_2$, the threshold used for binarizing concept vectors), but the experimental details do not indicate that there was a validation/development set used to select these values.\n\nIf these two points can be addressed, I believe this paper will warrant acceptance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' recognition of our work and your constructive comments. Please find below our detailed responses to the queries you have raised. We hope you could consider increasing our overall score if our response addresses your concerns :)\n\n> W1,Q1: \"Notations..\"\n\n**Response**: We have placed the modified table with the notation changes in the PDF. The corresponding original equations have the following modifications:\n- On line 131: N -> M\n- On line 133,135,136: k -> $d_c$\n- On line 130: c = {$p_1, \\cdots, p_k$\\} -> $\\mathcal{C} = ${$C_1, C_2, \\dots, C_N $} \n- On line 173: $\\mathcal{C} = ${$c_1, c_2, \\dots, c_N $} -> $\\mathcal{C} = ${$C_1, C_2, \\dots, C_N $}\n- Throughout the text, replace all heatmap H with P.\n- On line 184: $H_{p,k,i} = \\frac{t_i^T V_{p,k}}{||t_i||\\cdot ||V_{p,k}||},\\quad p = 1, \\ldots , P, \\quad k = 1, \\ldots , K $ -> $P_{h,w,i} = \\frac{t_i^T V_{h,w}}{||t_i||\\cdot ||V_{h,w}||},\\quad h = 1, \\ldots , H, \\quad w = 1, \\ldots , W $\n- On line 189: $s_i = \\frac{1}{H \\cdot W} \\sum_{h=1}^{H} \\sum_{w=1}^{W} P_{h,w,i}$\n- On line 195: $c =$ {$C_1, \\ldots, C_M$} -> $c = ${$c_1, \\ldots, c_M$}\n- On line 212: $D$ -> $X$ (Here, the dataset and the dimensional representation in the previous text conflict)\n- On line 215: $f_c(x_{m}), \\hat{c_m}, f(x_m) = \\Theta_c(\\Theta_b(x_m))$ for $i \\in [M]$ -> $f(x_m) = \\Theta_b(x_m), f_c(x_{m}),\\hat{c_m} = \\Theta_c(f(x_m))$ for $i \\in [M]$\n- More detail of $I_{o,j}$ and $I_{r,j}$\n\n> W2: \"hyperparameters..\"\n\n**Response**: Thank you for your valuable feedback. We conducted experiments on reasonable parameter ranges for λ1 and λ2, as detailed in the table below. The table shows the accuracy of various backbones with different combinations of λ1 and λ2. Notably, the combination of λ1=0.1 and λ2=0.1 (used in the paper) performs well overall. When λ1 and λ2 are too small, accuracy generally decreases, indicating that λ1 and λ2 play a positive role in model optimization. Conversely, as λ1 and λ2 increase, different backbones exhibit varying performance. Overall, λ1 and λ2 values between 0.1 and 1 yield good model performance and show significant improvement compared to the baseline.\n| Dataset  | Model | Baseline |  λ1=0.1, λ2=0.1 |  λ1=0.1, λ2=0.01 |  λ1=0.1, λ2=0.05 |  λ1=0.1, λ2=0.5 |  λ1=0.1, λ2=1.0 |  λ1=0.01, λ2=0.1 |  λ1=0.05, λ2=0.1 |  λ1=0.5, λ2=0.1 |  λ1=1.0, λ2=0.1 |\n|-----|---|------|:-------:|:-----:|:-----:|:-----:|:----:|:----:|:---:|:----:|:----:|\n|       | RN50  | 81.36    |    **84.75**    | 83.90 | 82.20 | 82.25 | 83.05 | 81.33 | 83.90 | 83.92 | 82.53 |\n| COVID-CT | DenseNet | 85.59 | 86.44 | 84.75 | 84.74 | **87.29** |      85.59      |       84.09      |       84.75      | 84.12 | 84.28 |\n|       | VGG | 79.60 | 83.05 | 79.66 |       80.52      |      80.51      |      85.60      |       80.85      |       79.66      |      86.44      |    **87.01**    |\n\nWe conducted experiments to determine the appropriate threshold values. We sampled 100 samples and the results are as follows:\n\n| Dataset  | 0.1   | 0.2   | 0.3   | 0.4   | 0.45      | 0.5   | 0.6   |\n|----------|-------|-------|-------|-------|-----------|-------|-------|\n| COVID-CT | 59.13 | 63.25 | 76.25 | 84.00 | **90.13** | 88.50 | 86.50 |\n\n>  Q2: \"all training images..\"\n\n**Response**: We annotated 20% of the training data for fine-tuning the model and labeled all images, including both training and testing images. This is because we need to use the automatically generated concept labels during inference.\n\n> Q3: \"equation..\"\n\n**Response**: Thank you for pointing this out. The expression in our paper was somewhat unclear. We have revised it as follows:\nFor each image $x_m$, after inputting it through the backbone, we obtain $f(x_m)$, represented as $f(x_m) = \\theta_b(x_m)$, where $f(x_m) \\in \\mathbb{R}^{d}$. Next, we input the image feature $f(x_m)$ into the concept encoder $\\theta_c$. It is important to note that $\\theta_c$ returns two values: one is the predicted concept $\\hat{c}_m$ and the other is the concept feature $f_c(x_m)$. The dimensions of these are $\\hat{c}_m \\in ${$0, 1$}$^N$ $\\in \\mathbb{R}^{d_c}$ and $f_c(x_m) \\in \\mathbb{R}^{d_c}$, respectively. In the paper, $d$ and $d_c$ is set to 1000, and $ N $ depends on the number of concepts.\n\n> Q4: \"during training..\"\n\n**Response**: During training, all modules are trained together. As shown in Table 1, the result of using only the neural-symbolic layer (DCR) is very poor, indicating that joint training is necessary. It is also worth noting that while $\\mathcal{L_{neural}}$ primarily affects the output of the neural-symbolic layer, $ \\mathcal{L_c} $ impacts the concept prediction results, and $ \\mathcal{L}_{task} $ affects the classification results. Due to the nature of backpropagation, these losses also optimize the concept encoder and backbone. As evident from the ablation studies in Table 2, removing any of these losses leads to a decline in overall model performance.\n\n> Q5: \"Figure 4..\"\n\n**Response**: A full comparison of our method with current saliency and CBM methods in terms of interpretable dimensions and effectiveness is shown in Fig. 4. CBM class methods lack interpretation of saliency map and rules dimensions. saliency map class methods lack interpretation of concept and rules dimensions. For this medical classification task, single dimensions such as saliency map methods do not accurately capture the features of disease images, and CBM-type methods for concept extraction of image information carry false predictions. And both of them lack explanatory information about the conceptual rules. Our method is able to predict interpretable information in three dimensions, saliency map with concept and rules, which provides more accurate multi-dimensional information, as reflected in the more accurate capture of disease features in image information, clearer concept output, and logical rule information generated based on the predicted concepts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2wnxyQ6ZSM", "reviewer": "Reviewer_WJ93", "summary": "This paper focuses on the explainable classification of medical images with multi-dimensional explanation. The proposed Med-MICN framework contains four modules including feature extraction, auto-annotation, concept embedding, and neural symbolic layer. The incorporation of fuzzy logic rules is novel in the interpretation study of deep neural networks. The experimental results show that Med-MICN surpasses the baseline networks such as vgg and resent, and also outperforms concept bottleneck models (CBM). The generated attention maps and predicted concepts match better than traditional CBM.", "questions": "See weakness.", "limitations": "The whole framework of Med-MICN is complex, and the so-called multi-dimensional explanations are attention maps and predicted concepts which have been proposed in traditional CBM models.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "See summary", "weaknesses": "1. The whole presentation of the paper should be further improved including mathematical symbols, figures 2 and 3, and equations. The symbols should be better matched with Figures 2 and 3 to improve readability. It is difficult to understand these figures without reading the contents. Some symbols like Cosine should be given in the figures. Detailed notations are needed in the figures or in the captions.\n2. It is better to give pseudo codes of training and inferences to help understand the whole workflow.\n3. At the bottom of Page 3, \"In instances ... judgment\", why do the authors claim alternative explanations can aid judgment? Any references? More explanations may confuse the physicians.\n4. In line 131, does $y_{i}$ the one-hot labels?\n5. In the Concept labeling alignment module, it is not clear why the authors use \"average pooling to heatmaps\". I think this will lose some spatial localizations of lesions, for example, one heatmap with multiple lesions and another with one lesion, both of them may have similar average pooling results.\n6. Under the line 183, the $V$ is three-dimensional, and why here it has two subscripts.\n7. In line 194, does the representation be $c \\in \\{C_{1}, ..., C_{M}\\}$ ?\n8. In line 142, the logic symbols should be defined may be in supplementary.\n9. Lack of training details. In line 277, how the learning rate decay? Any data augmentations were used?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' recognition of our work and your constructive comments. Please find below our detailed responses to the queries you have raised. We hope you could consider increasing our score if our response addresses your concerns :)\n\n> W1: \"captions and notations...\"\n\n**Response**: We have added notations to Figure 2, 3 and also enhanced the captions for Figures 2, 3, and 4 to improve readability. We have placed the modified table with the notation changes in the PDF. The corresponding original equations have the following modifications:\n- On line 131: N -> M\n- On line 133,135,136: k -> $d_c$\n- On line 130: c = {$p_1, \\cdots, p_k$\\} -> $\\mathcal{C} = ${$C_1, C_2, \\dots, C_N $} \n- On line 173: $\\mathcal{C} = ${$c_1, c_2, \\dots, c_N $} -> $\\mathcal{C} = ${$C_1, C_2, \\dots, C_N $}\n- Throughout the text, replace all heatmap H with P.\n- On line 184: $H_{p,k,i} = \\frac{t_i^T V_{p,k}}{||t_i||\\cdot ||V_{p,k}||},\\quad p = 1, \\ldots , P, \\quad k = 1, \\ldots , K $ -> $P_{h,w,i} = \\frac{t_i^T V_{h,w}}{||t_i||\\cdot ||V_{h,w}||},\\quad h = 1, \\ldots , H, \\quad w = 1, \\ldots , W $\n- On line 189: $s_i = \\frac{1}{H \\cdot W} \\sum_{h=1}^{H} \\sum_{w=1}^{W} P_{h,w,i}$\n- On line 195: $c =$ {$C_1, \\ldots, C_M$} -> $c = ${$c_1, \\ldots, c_M$}\n- On line 212: $D$ -> $X$ (Here, the dataset and the dimensional representation in the previous text conflict)\n- On line 215: $f_c(x_{m}), \\hat{c_m}, f(x_m) = \\Theta_c(\\Theta_b(x_m))$ for $i \\in [M]$ -> $f(x_m) = \\Theta_b(x_m), f_c(x_{m}),\\hat{c_m} = \\Theta_c(f(x_m))$ for $i \\in [M]$\n- More detail of $I_{o,j}$ and $I_{r,j}$\n\n> W2: \"It is better to give pseudo codes of training and inferences...\"\n\n**Response**: The pseudocode has been supplemented, as seen in Figure 2 of the PDF.\n\n> W3: \"At the bottom of Page 3...\"\n\n**Response**: It has been demonstrated that unidimensional interpretive aids such as saliency map [1,2], concept [3,4], and other information will contribute to medical decision making. To the best of our knowledge, our work is the first to propose a deeper aiding strategy for medical aids using multidimensional interpretable means. The advantage of this multidimensional aid is not only in the cumulative nature of the multiple facilitation strategies but also in the fact that when there is an error in one dimension of interpretability, the information from the other dimensions is able to correct that erroneous information, thus providing more accurately interpreted information.\n\nIn addition, we also combine neural symbolic and fuzzy logic to further the connection between the concepts, which makes the interpretation of concepts more auxiliary meaning.\n\n> W4: \"In line 131, does $y_i$ the one-hot labels?\"\n\n**Response**: Yes, $y_i$ is the one-hot labes.\n\n> W5: \"It is not clear why the authors use \"average pooling to heatmaps\"...\"\n\n**Response**:\n|Method|COVID-CT|DDI|\n|--|:---:|:--:|\n|Convolution|**89.81**|80.95|\n|Linear|69.84|73.92|\n|Average pooling|89.79|**91.89**|\n\nThank you for the valuable suggestions. Our approach is based on experimental results, which show that average pooling is comparable to or even better than other methods.\n\n> W6: \"Under the line 183, the $V$ is three-dimensional, and why here it has two subscripts.\" \n\n**Response**: Yes, it is a 3-dimensional tensor. Here, we use $V_{h,w}$ to represent $V_{h,w,\\cdot}$, which is a vector whose length is $D$. The term $ V_{h,w} $ is dot-multiplied with $ t_i^T $ to obtain an intermediate result of size $ \\mathbb{R}^{D \\times D} $. We then apply pooling to obtain the result for the heatmap at the position $(h, w)$.\n\n> W7: \"In line 194, does the representation be ....\"\n\n**Response**: The meaning of $c$ is the set of all image concept labels, where $c=${$c_1, . . . , c_M$}. This notation is correct, with each $c_i \\in ${$0, 1$} $^N$.\n\n> W8: \"In line 142, the logic symbols should be defined may be in supplementary.\"\n\n**Response**: Thank you. We will define them in supplementary. For example:\n- Negation ($\\neg $): This symbol represents the inverse of a truth value. In fuzzy logic, the negation of a degree of truth $x$ is computed as $\\neg x = 1 - x$. For example, if $x$ represents a 0.7 probability of a statement being true, then $\\neg x$ would be 0.3, indicating a 0.3 probability of the statement being false.\n- T-norm (Conjunction, $\\land$): The t-norm, or triangular norm, is a continuous version of the AND operation. It takes two truth values $x$ and $y$ and combines them to produce a new truth value in the range $[0, 1]$. Common t-norms include the minimum function ($\\min(x, y)$) and the product function ($x \\cdot y$). In the example given, $c_{GO} \\land \\neg c_{LC}$ would represent the conjunction of \"GO\" being present and \"LC\" not being present.\n- T-conorm (Disjunction, $\\lor$): The t-conorm, or triangular conorm, is a continuous version of the OR operation. It also takes two truth values $x$ and $y$ and combines them, typically using functions like the maximum function ($\\max(x, y)$) or the sum function ($x + y - xy$), ensuring the result is within $[0, 1]$. \n\n> W9: \"Lack of training details...\"\n\n**Response**: To highlight the significant performance of our method, we did not use learning rate decay; the learning rate was kept constant at 5e-5 throughout the entire training process. For data augmentation, we only used the most basic operations: resize (256), center crop (224), and normalization.\n\n*Reference*\n\n[1] Van der Velden et al. \"Explainable artificial intelligence (XAI) in deep learning-based medical image analysis.\" Medical Image Analysis 79 (2022): 102470.\n\n[2] Patrício et al. \"Explainable deep learning methods in medical image classification: A survey.\" ACM Computing Surveys 56.4 (2023): 1-41.\n\n[3] Patrício et al. \"Coherent concept-based explanations in medical image and its application to skin lesion diagnosis.\" CVPR 2023.\n\n[4] Eminaga et al. \"PlexusNet: A neural network architectural concept for medical image classification.\" Computers in biology and medicine 154 (2023): 106594.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bKoKXlFsrI", "reviewer": "Reviewer_gQUT", "summary": "This work introduces a novel end-to-end concept-based framework called Med-MICN, which is quite inspiring and important for the next XAI era due to its multi-dimensional, powerful interpretability, and efficient performance. Furthermore, they propose an automated process for efficiently and accurately obtaining concept labels, which are costly to acquire in reality.", "questions": "See weeknesses.", "limitations": "NA", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This paper introduces a novel attempt that creatively integrates medical image classification, neural symbolic solving, and concept semantics. This integration enables multidimensional interpretability, making it more comprehensive and powerful compared to previous interpretable models.\n\n2. Med-MICN demonstrates superior accuracy and interpretability across datasets from different modalities.\n\n3. The ample experiments are convincing, as the authors conducted experiments on multiple datasets and baselines to validate the model's interpretability and accuracy, while also providing rich and impressive visualization results.", "weaknesses": "1. The caption of the picture is not detailed enough, as shown in figure 3.\n\n2. The author considers interpretability on the basis of concept embedding, neural-sybolic reasoning and saliency map. The author lacks the reason why it is necessary to analyze the interpretation from these aspects and whether there are other more dimensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewers' recognition of our work and your constructive comments. Please find below our detailed responses to the queries you have raised. We hope you could consider increasing our overall score if our response addresses your concerns :)\n\n> W1: \"The caption of the picture is not detailed enough, as shown in figure 3.\"\n\n**Response**: Thank you for your suggestions. We have provided additional details for each figure caption as follows:\n\n- Figure 2: (a) module, output rich dimensional interpretable conceptual information for the specified disease through the multimodal model and convert the conceptual information into text vectors through the text embedding module; (b) module, access the image to the image embedder to get the image features, and then match with the conceptual textual information to get the relevant attention region. Then, we get the influence score of the relevant region information through pooling, and finally send it to the filter to sieve out the concept information with weak relevance to get the disease concept of image information.\n- Figure 3: Overview of the Med-MICN Framework. The Med-MICN framework consists of four primary modules: (1) **Feature Extraction Module**: In the initial step, image features are extracted using a backbone network to obtain pixel-level features. (2) **Concept Embedding Module**: The extracted features are fed into the concept embedding module. This module outputs concept embeddings while passing through a category classification linkage layer to obtain predicted category information. (3) **Concept Semantic Alignment**: Concurrently, a Vision-Language Model (VLM) is used to annotate the image features, generating concept category annotations aligned with the predicted categories. (4) **Neural Symbolic Layer**: After obtaining the concept embeddings, they are input into the Neural Symbolic layer to derive conceptual rules.  Finally, the concept embeddings obtained from module (2) are concatenated with the original image embeddings and fed into the final category prediction layer to produce the ultimate prediction results.\n- Figure 4: A full comparison of our method with current saliency and CBM methods in terms of interpretable dimensions and effectiveness is shown in Fig. 4. CBM class methods lack interpretation of saliency map and rules dimensions. saliency map class methods lack interpretation of concept and rules dimensions. For this medical classification task, single dimensions such as saliency map methods do not accurately capture the features of disease images, and CBM-type methods for concept extraction of image information carry false predictions. And both of them lack explanatory information about the conceptual rules. Our method is able to predict interpretable information in three dimensions: saliency map with concept and rules, which provides more accurate multi-dimensional information, as reflected in the more accurate capture of disease features in image information, clearer concept output, and logical rule information generated based on the predicted concepts.\n\n> W2: \"The author considers interpretability on the basis of concept embedding, neural-sybolic reasoning and saliency map. The author lacks the reason why it is necessary to analyze the interpretation from these aspects and whether there are other more dimensions.\"\n\n**Response**:\n\n**concept**: This approach focuses on understanding the internal representation of high-level concepts in a model.  By analyzing the learned embeddings, researchers can gain insights into how the model associates different input features and concepts.  It helps in understanding the semantic meaning of the model's hidden layers and can reveal if the model has learned meaningful abstractions.\n\n**neural-symbolic**: This combines the power of neural networks with neural symbolic logic, aiming to make models more transparent by grounding their decisions in symbolic rules via concept.  It allows for more explicit concept reasoning processes, which can be more interpretable than purely connectionist models.\n\n**saliency map**: These highlight the parts of the input that are most influential in the model's decision.  By visualizing these maps, one can see which features or pixels are driving the prediction, providing a more intuitive understanding of the model's behavior.\n\nThere are also indeed other dimensions:\n\n(1) LIME (Local Interpretable Model-agnostic Explanations) \n\n(2) SHAP (SHapley Additive exPlanations)\n\nThese are post-hoc explainability methods, and these techniques could be less scalable, less suitable for large models, and may struggle to align with specific tasks or samples.\n\nOur framework, however, goes beyond these limitations.  By integrating concept-based and attention map explanations, we offer a more intrinsic understanding of the model's decision-making process.  Furthermore, the introduction of neural symbolic methods allows us to delve into clearer concept reasoning and decision-making, thus providing a third dimension of interpretability.\n\nSpecifically, we have successfully aligned the attention-based image features with concept-based semantics, ensuring a direct correspondence between the model's perception and the underlying meaning.  Moreover, the neural-symbolic approach enables us to elucidate the interactions among concepts, which is crucial in understanding complex medical scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their insightful feedback. We have revised and supplemented our work in the following two aspects.

1. **Notations and Captions**: We have added notations to Figure 2, 3 and also enhanced the captions for Figures 2, 3, and 4 to improve readability. We have placed the modified table with the notation changes in the PDF. The corresponding original equations have the following modifications:

- On line 131: N -> M
- On line 133,135,136: k -> $d_c$
- On line 130: c = {$p_1, \cdots, p_k$\} -> $\mathcal{C} = ${$C_1, C_2, \dots, C_N $} 
- On line 173: $\mathcal{C} = ${$c_1, c_2, \dots, c_N $} -> $\mathcal{C} = ${$C_1, C_2, \dots, C_N $}
- Throughout the text, replace all heatmap H with P.
- On line 184: $H_{p,k,i} = \frac{t_i^T V_{p,k}}{||t_i||\cdot ||V_{p,k}||},\quad p = 1, \ldots , P, \quad k = 1, \ldots , K $ -> $P_{h,w,i} = \frac{t_i^T V_{h,w}}{||t_i||\cdot ||V_{h,w}||},\quad h = 1, \ldots , H, \quad w = 1, \ldots , W $
- On line 189: $s_i = \frac{1}{H \cdot W} \sum_{h=1}^{H} \sum_{w=1}^{W} P_{h,w,i}$
- On line 195: $c =$ {$C_1, \ldots, C_M$} -> $c = ${$c_1, \ldots, c_M$}
- On line 212: $D$ -> $X$ (Here, the dataset and the dimensional representation in the previous text conflict)
- On line 215: $f_c(x_{m}), \hat{c_m}, f(x_m) = \Theta_c(\Theta_b(x_m))$ for $i \in [M]$ -> $f(x_m) = \Theta_b(x_m), f_c(x_{m}),\hat{c_m} = \Theta_c(f(x_m))$ for $i \in [M]$
- More detail of $I_{o,j}$ and $I_{r,j}$

2. **Hyperparameters and Experimental Details**: We conducted experiments on reasonable parameter ranges for λ1 and λ2, as detailed in the table below. The table shows the accuracy of various backbones with different combinations of λ1 and λ2. Notably, the combination of λ1=0.1 and λ2=0.1 (used in the paper) performs well overall. When λ1 and λ2 are too small, accuracy generally decreases, indicating that λ1 and λ2 play a positive role in model optimization. Conversely, as λ1 and λ2 increase, different backbones exhibit varying performance. Overall, λ1 and λ2 values between 0.1 and 1 yield good model performance and show significant improvement compared to the baseline.

| Dataset  | Model    | Baseline |  λ1=0.1, λ2=0.1 |  λ1=0.1, λ2=0.01 |  λ1=0.1, λ2=0.05 |  λ1=0.1, λ2=0.5 |  λ1=0.1, λ2=1.0 |  λ1=0.01, λ2=0.1 |  λ1=0.05, λ2=0.1 |  λ1=0.5, λ2=0.1 |  λ1=1.0, λ2=0.1 |
|----------|----------|----------|:---------------:|:----------------:|:----------------:|:---------------:|:---------------:|:----------------:|:----------------:|:---------------:|:---------------:|
|          | RN50     | 81.36    |    **84.75**    |       83.90      |       82.20      |      82.25      |      83.05      |       81.33      |       83.90      |      83.92      |      82.53      |
| COVID-CT | DenseNet | 85.59    |      86.44      |       84.75      |       84.74      |    **87.29**    |      85.59      |       84.09      |       84.75      |      84.12      |      84.28      |
|          | VGG      | 79.60    |      83.05      |       79.66      |       80.52      |      80.51      |      85.60      |       80.85      |       79.66      |      86.44      |    **87.01**    |

We conducted experiments to determine the appropriate threshold values. We sampled 100 samples and the results are as follows:

| Dataset  | 0.1   | 0.2   | 0.3   | 0.4   | 0.45      | 0.5   | 0.6   |
|----------|-------|-------|-------|-------|-----------|-------|-------|
| COVID-CT | 59.13 | 63.25 | 76.25 | 84.00 | **90.13** | 88.50 | 86.50 |

Meta-review: After discussion, all reviewers agree to accept this paper. 
The authors are encouraged to further improve this work by addressing feedback from the reviewers, in particular, the following:

1. Add evaluation of the generated heatmaps
2. Compare the interpretability between the proposed method and the baseline methods
3. Improve the notations
Decision: Accept (poster)