Mixed Prototype Correction for Causal Inference in Medical Image Classification

Authors: yajie  zhang
zhi-an  huang
zhiliang  hong
songsong  wu
jibin  wu
kay chen tan
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 02:18:13

Abstract: The heterogeneity of medical images poses significant challenges to accurate disease diagnosis. To tackle this issue, the impact of such heterogeneity on the causal relationship between image features and diagnostic labels should be incorporated into model design, which however remains underexplored. In this paper, we propose a mixed prototype correction for causal inference (MPCCI) method, aimed at mitigating the impact of unseen confounding factors on the causal relationships between medical images and disease labels, so as to enhance the diagnostic accuracy of deep learning models. The MPCCI comprises a causal inference component based on front-door adjustment and an adaptive training strategy. The causal inference component employs a multi-view feature extraction (MVFE) module to establish mediators, and a mixed prototype correction (MPC) module to execute causal interventions. Moreover, the adaptive training strategy incorporates both information purity and maturity metrics to maintain stable model training. Experimental evaluations on four medical image datasets, encompassing CT and ultrasound modalities, demonstrate the superior diagnostic accuracy and reliability of the proposed MPCCI. The code will be available at https://github.com/Yajie-Zhang/MPCCI.

Main paper: 
 introduction: 
 Medical image classification provides essential support to the clinicians and other medical professionals in diagnosing and treating patients by analyzing lesion features of the human body within medical images [11] . With wide applications in the real world, this problem has been extensively researched. In the past few decades, especially driven by the application of deep learning technologies, a significant body of methods have been developed, which can generally be categorized into detection-based approaches [10, 32] , segmentation-based approaches [28, 44, 45] and feature extraction approaches [22, 23, 29, 30] . Despite the remarkable advancements in previous studies, classifying images in the medical domain remains much more challenging than it is for natural images. This is primarily attributed to the inherently complex nature of the medical images. Compared with the natural images, the medical images often contain more noise and artifacts, due to the limitations of current imaging technologies, such as weak X-ray penetration of the equipment, presence of gas in the body, and motion artifacts [12] .In addition to these two factors that have received more attention in previous research, lesion heterogeneity is also a crucial challenge to the classifiers, which however is much less exploited. Lesion heterogeneity in the medical imaging refers to the variability in features and appearances of the same disease, encompassing variations in terms of shape, size, density, intensity, texture, and other lesion characteristics. An exemplar illustration is given in Fig. 1 . As shown in the figure, breast cancer may exhibit pronounced heterogeneity in ultrasound images, with lesions varying significantly in appearance. For example, the lesion shapes may be oval, round, or irregular, and the complexity is further exacerbated when combined with other varied attributes as shown in the table below the images. The impact of heterogeneity on model performance has been acknowledged in previous studies [27, 49] , and there have been some attempts to alleviate its negative effects through variance pooling structures [5] and data augmentation [52] . However, these methods often yield suboptimal outcomes due to the lack of a thorough examination on the root causes of heterogeneity.Lesion heterogeneity is caused by various factors, such as the diverse origins of cancer cells, variable gene expressions, and patientspecific susceptibilities [3] . These factors have significant influences upon the prediction of a diagnostic label from the medical image. For example, the patients with genetic predispositions are more susceptible to the illness [3] . Hence in this work, we propose to model these factors by leveraging causal inference [37] for enhanced medical image classification. To enunciate our idea, we establish a structural causal model for the medical image classification task, as shown in Fig. 2 . In this figure, the underlying causes of heterogeneity, denoted as C, involve various factors as above mentioned; the medical images X and the diagnostic outcomes Y are both impacted by the heterogeneity cause factors C. In formulation, there exist X → Y, denoting the causal path that image X contains the lesion representations related to the given label Y, and X ← C → Y representing the backdoor path that X and Y exhibit spurious correlation through C. That is, the factors in C affect the characterization of medical imaging (X ← C); in addition, these factors influence the probability of a patient contracting a particular disease (C → Y). Following the Pearl's causal inference theory [37] , when we try to find the causal effect of X on Y, we want the nodes we condition on to block any "backdoor" path in which one end has an arrow to X, because such paths may make X and Y dependent, but are obviously not transmitting causal influences from X; and if we do not block them, they will confound the effect that X has on Y. Therefore, we should adjust the confounders "C" to block the backdoor path for better inference from X to Y. However, given the inherent difficulty or even impossibility of quantifying these confounders, their adverse impact upon the image-based diagnostic procedures remains elusive and unaddressed.In this work, we propose a novel approach for enhanced medical image classification, named mixed prototype correction for causal inference (MPCCI), which mitigates the influences of the confounding factors on the medical diagnosis by exploiting front-door adjustment (FDA) [37] . The FDA introduces a mediator variable, denoted as A in Fig. 3 , between the causal path of variables X and Y, to adjust the causal pathway between them, which well addresses the immeasurability of the elusive confounding factors. To implement MPCCI, we first design a multi-view feature extraction (MVFE) module with spatial-channel attention that allows these multi-view features to serve as mediators in FDA to link the causal effect from images to labels. We also develop a mixed prototype correction (MPC) module that exchanges some features of the multi-view features and the multi-view prototypes to effectively apply causal intervention on the mediators. The multi-view prototypes contain meta-knowledge of various disease categories, and the causal intervention mechanism that exchanges features with them can correct the spurious association between X and Y formed by the confounders. To improve the smoothing of the feature exchange process, an adaptive training strategy is presented, comprising two key components: information purity (IP) and maturity (MT). The IP module is used to measure the proportion of noise in the feature exchange process, and MT is used to measure the stability of the model to noise in different training stages.Experiments on four medical datasets verify the effectiveness of the proposed MPCCI on diagnosing covid, breast cancer, lymph node metastasis, and thyroid. In summary, the contributions of this work are as followings:• This work is the first attempt to conduct cause-effect analysis to alleviate the immeasurable condounders for enhancing medical image classification. The proposed method solves the problem by applying an FDA strategy, treating the multiview features as mediators to infer the causalities from images to labels. The model performance is mainly dependent on its architecture design as well as the scale of the training data. Some methods adopt advanced architectures, such as AlexNet [26] , ResNet [20] , VGG [47] , and ViT [32] for good classification performance. In recent year, some works propose to extract and integrate multi-scale information to improve classification accuracy, utilizing feature pyramid networks [45] , dilated convolutions [14] , and attention mechanisms [19, 30] , etc. There are also models fusing local and global features [8, 31] to achieve lifted efficacy in medical image classification. These methods are all based on the assumptions of sufficient high-quality training data, which are not always true.In addition to these model-centric approaches, some other methods broaden the diversity and volume of training data to enhance model generalization by utilizing GANs [4] , variational autoencoders [6] , MixUp [13] , and diffusion models [25] . These datacentric approaches also demonstrate strong effectiveness on medical image classification tasks. However, either the model-centric models, or the data-centric ones, fall short on addressing disease heterogeneity and its related confounding factors, which hampers their performance. 
causal inference: 
 The goal of causal inference is to unravel the complex causal relationships between variables, far beyond the mere correlations [50] . It serves as a powerful tool for understanding the roots and implications of phenomena and thereby supports informed decisionmaking and interventions. For its potent analytical power, causal inference has been applied in various domains, such as medical image classification [38] , domain generalization [36] , and medical image segmentation [7, 34] . In medical image analysis, some methods [7, 34, 36] treat complex organ co-occurrences and background phenomena such as pseudo artifacts as observable confounding factors, and leverage the backdoor adjustment strategy [37] for causal intervention. Some works [38] harness counterfactual reasoning for medical image analysis by crafting counterfactual samples to neutralize the effects of observable confounding factors. These causal inference based methods achieve promising results. Yet, they tend to focus on observable confounding factors, which constrains their effectiveness in handling cases with unobservable confounders.In this work, we propose to utilize the FDA strategy to mitigate the impact of unmeasured confounders for better medical image classification performance. 3 Cause-Effect AnalysisX A Y C X A Y C (b) (c) X A Y C (a)In this section, we provide a brief analysis of the causal relationships among the elements in our tasks, namely the input image 𝑋 , multiview features 𝐴, image label 𝑌 , and confounders 𝐶, using a structural causal model (SCM) illustrated in Fig. 3 (a). We also describe how FDA is used in this context. The main causal relationships in Fig. 3 (a) include 𝑋 → 𝐴 → 𝑌 , 𝐶 → 𝑋 , and 𝐶 → 𝑌 . 1) For 𝑋 → 𝐴 → 𝑌 , the input image 𝑋 is fed into a deep neural network to extract multi-view features 𝐴, which are then used to predict the label 𝑌 . 2) For 𝐶 → 𝑋 , the confounders 𝐶 like genetics, origins of cancer cells, patient habits, etc. influence the lesion manifestation 𝑋 . 3) For 𝐶 → 𝑌 , the confounders 𝐶 can also affect the disease category 𝑌 of a patient. For example, the patients with genetic predisposition for breast cancer have a higher risk of malignancy. Note that there are two paths connecting 𝑋 and 𝑌 : the frontdoor path 𝑋 → 𝐴 → 𝑌 and the backdoor path 𝑋 ← 𝐶 → 𝑌 . The existence of the backdoor path makes it difficult to evaluate the true causality from 𝑋 to 𝑌 through deep networks. If 𝐶 is measurable, the backdoor adjustment can be used to eliminate the link of 𝐶 ← 𝑋 . However, since most of 𝐶 in this work are not measurable, we turn to use front-door adjustment [37] to estimate the causality from 𝑋 to 𝑌 . To achieve this, FDA employs a mediator 𝐴 to transmit knowledge of 𝑋 to 𝑌 through the front-door path, and then evaluates the causalities from 𝑋 to 𝑌 by combining the causal effects of 𝑋 to 𝐴 and 𝐴 to 𝑌 , i.e., to estimate the probabilities 𝑃 (𝐴|𝑑𝑜 (𝑋 )) and 𝑃 (𝑌 |𝑑𝑜 (𝐴)), respectively. The 𝑑𝑜-operation represents an active intervention to a cause rather than a passive observation.The P(A|do(X)) represents the causal relationship between 𝑋 and 𝐴, as illustrated in Fig. 3 (b) . Since the path of 𝑋 ← 𝐶 → 𝑌 ← 𝐴 is blocked by the collider [37] , we can writeEQUATIONThe P(Y|do(A)) (Fig. 3 (c )) pursues the true causality between 𝐴 and 𝑌 without confounders 𝐶. There are two paths from 𝐴 to 𝑌 : 𝐴 → 𝑌 and the backdoor path 𝐴 ← 𝑋 ← 𝐶 → 𝑌 . Due to the existence of the backdoor path, we need to cut off the link between 𝐴 and 𝑋 by controlling 𝑋 , and we can write 𝑃 (𝑌 |𝑑𝑜 (𝐴)) asEQUATIONThrough layer-by-layer causal effect calculation, the causality from 𝑋 to 𝑌 can be represented as where 𝑥 ′ is an index of summation in 𝑃 (𝑌 |𝑑𝑜 (𝐴)).EQUATION 
methodology: 
 In this section, we introduce the proposed Mixed Prototype Correction for Causal Inference (MPCCI) approach in medical image classification. As shown in Fig. 4 , it involves multi-view feature extraction (MVFE) and mixed prototype correction (MPC) modules to implement the FDA strategy. Additionally, we present an adaptive training approach, incorporating the information purity (IP) and the maturity (MT), to alleviate the noise at MPC. The IP module quantifies the noise proportion during the feature exchange process, while MT assesses the model's robustness to noise across various training phases.The MVFE module is responsible for generating multi-view features 𝐴, which serve as the mediator in Fig. 3 (a) and are used to implement 𝑃 (𝐴|𝑑𝑜 (𝑋 )) in Eq. ( 1). First, we input the image into a convolutional neural network (CNN) such as ResNet18 [18] to obtain feature maps E = 𝑓 𝑏 (𝑥) ∈ R 𝐷 ×𝐻 ×𝑊 , where 𝑓 𝑏 is the function of the CNN, and 𝐷, 𝐻 , 𝑊 represent the number of channels, height, and width of E, respectively. To extract multi-view features from the E, we employ two parallel paths. We apply global average pooling to E to obtain a global feature vector g ∈ R 𝐷 and construct expert networks [21] with spatial-channel attention following CBAM [43] . Spatial-channel attention adopted in CNN allows for the adaptive weighting of feature maps across both spatial and channel dimensions. This enables the network to selectively focus on informative features, enhancing its ability to learn and identify complex visual patterns. To ensure that each expert network learns different features of an image, they are initialized with different parameters. The formulation of the expert networks can be expressed asEQUATIONwhere 𝑓 𝑘 𝑎 represents the k-view feature vector a 𝑘 generated by the k-th function of spatial-channel attention.After extracting the mediator A, it is crucial to ensure that the learned multi-view features are distinct across classes. To achieve this goal, the multi-view features and global feature are concatenated and then fed into the classifier (a fully-connected layer is used in this work) denoted as 𝑓 𝑐 , to produce the predicted label y of the image 𝑥:EQUATIONwhere || represents the concatenation operation, 𝐶 is the number of categories, and 𝐾 is the number of expert networks. The crossentropy loss is used to optimize 𝑓 𝑐 :EQUATIONwhere l ∈ R 𝐶 denotes the ground-truth label of 𝑥. If 𝑥 belongs to the c-th category, 𝑙 𝑐 = 1; otherwise 𝑙 𝑐 = 0.The MPC module aims to correct the side-effects of confounders and further explore the causality of 𝐴 on 𝑌 by estimatingEQUATIONHowever, it is infeasible to collect all possible 𝑥 ′ (lesions that might appear in reality) with 𝐴 for predicting 𝑌 . Thus we use mixed multi-view prototypes to approximate 𝑥 ′ . Multi-view prototype learning [48] is an emerging machine learning technique that aims to learn a set of prototypes across different views to capture the underlying structure of representative examples for each category. Specifically, we use the 𝑐-th class-specific average multi-view features to approximate the 𝑐-th multi-view prototypes, denoted asS 𝑐 = {s 𝑐 1 , • • • , s 𝑐 𝐾 } ∈ R 𝐾 ×𝐷 .We then generate the mixed multi-view prototypes Ŝ, which partially come from the source multi-view prototypes (𝑐-th) and another random counterparts (𝑐 ′ -th), to expressEQUATIONwhere v ∈ {0, 1} 𝐾 represents the random exchanging index vector of S 𝑐 . Namely, each 𝑣 𝑘 represents whether the 𝑘-th prototype in S 𝑐 should be exchanged by the 𝑘-th prototype in S 𝑐 ′ . Since Ŝ captures lesions that have distinct characteristics in specific view features, it can serve as a substitute for 𝑥 ′ . To predict label 𝑌 , we fuse Ŝ with A via a fusion module. Cross-attention [30] is a mechanism that enables neural networks to capture the interdependent relationship between two heterogeneous features using a learnable similarity matrix. In this work, we incorporate cross-attention with the feature mapping function ℎ(•) into the fusion module to explore this independence. The output is fused multi-view features denoted as Â:EQUATIONWe can concatenate Â with global feature vector g to predict label as ŷ = 𝑓 𝑐 (g, Â) using Eq. ( 5).We utilize an adaptive training strategy to maintain stable model training. Since Ŝ randomly mixes the 𝑐-th and 𝑐 ′ -th multi-view prototypes, there is a possibility that Â is predicted as the 𝑐 ′ -th category. In this study, we hypothesize that two factors are related to this situation: 1) the information purity (IP) in Ŝ, and 2) the maturity (MT) of the fusion module.The IP refers to the amount of prototype information from the source category contained in Ŝ. If Ŝ contains a large amount of prototype information from other categories, the probability of Â being predicted as other categories increases. Therefore, we use v 𝐾 and(1-v) 𝐾to represent the possibility of Â being predicted as the 𝑐-th class and 𝑐 ′ -th class, respectively.The MT represents the ability of the fusion module to accurately fuse label-related prototype information to Â. When the fusion module cannot fuse multi-view prototypes well, the probability of accurately predicting ŷ𝑐 decreases. We assume that MT increases with the process of network iterative optimization, and we denote MT as 𝛼 = 𝛼 0 + 𝑐𝑢𝑟 _𝑒𝑝𝑜𝑐ℎ 𝑡𝑜𝑡𝑎𝑙_𝑒𝑝𝑜𝑐ℎ , where 𝛼 0 represents the initialized maturity. Based on these two factors, we can write the probabilities of ŷ to be ŷ𝑐 and ŷ𝑐 ′ :EQUATION)Based on this adaptive training strategy, the optimization goal of MPC can be formulated as:EQUATIONwhere 𝑃 (𝑥 ′ ) is set to a uniform distribution 1 𝑁 because 𝑥 ′ is generated from a random mixture with equal probability.By combining the MVFE, MPC, and the adaptive training strategy, the overall loss function L, which is the optimization objective to be minimized during training iterations, is a combination of the original loss L 𝑜 and the fusion loss L 𝑓 :EQUATION)where 𝜆 is a hyperparameter that controls the relative weight of the fusion loss. To ease the understanding of MPCCI, the pseudo-code is presented as Algorithm 1.Algorithm Compute L 𝑜 via Eq. (/* MPC */ 10:S 𝑐 𝑘 ← 1 𝑛 𝑐 (a 𝑐 𝑘 1 + • • • + a 𝑐 𝑘 𝑛 𝑐) // The multi-view prototypes 11: Compute L 𝑓 via Eq. (11) 18:Ŝ ← vS 𝑐 + (1 -v)S 𝑐 ′ //Update 𝑓 by minimizing L = L 𝑜 + 𝜆L 𝑓 19: end for 
experiment: 
 We conduct comprehensive experiments to evaluate the performance of the porposed MPCCI approach. At below, we first introduce the datasets used for experiments, evaluation protocols, compared methods, and implementation details. Then, we report and analyze the quantitative results obtained across four medical datasets. Moreover, the validation of heterogeneity cause C and data set analysis are conducted to systematically evaluate MPCCI. We also take further experimental analysis to assess the capabilities of MPCCI. This analysis encompasses an examination of the number of features, the function of the mixing mechanism, and the visualization results.Four medical image datasets are utilized for the evaluation of MPCCI. The details of the datasets are provided as follows:The CT COVID-19 [33] dataset comprises 7,593 COVID-19 CT images sourced from 466 patients, 6,893 normal CT images from 604 patients, and 2,618 CAP CT images from 60 patients. For experimentation purposes, a total of 14,486 images from the normal and COVID-19 categories are selected. These images are randomly partitioned into training, validation, and test sets at a ratio of 7:1:2.The BUSI [2] is a publicly available dataset consisting of 780 ultrasound images of three classes, i.e., normal, benign, and malignant. In this work, we follow the setting of MIB Net [42] which achieves the best result reported to date and use only the benign images (437) and malignant images (210). Conforming to the protocol of MIB Net, the dataset is partitioned into training, validation, and test sets at a ratio of 8:1:1.The FJPH is a dataset established by ourselves for predicting the likelihood of lymph node metastasis. The data inside are obtained anonymously from a local hospital (Fujian Provincial Hospital) to ensure the privacy of all involved patients. It consists of 889 ultrasound images categorized into two classes: metastasis (500 ultrasound images) and non-metastasis (389 ultrasound images). We employ this dataset to demonstrate the versatility of the proposed method in different ultrasound imaging scenarios. Adhering to the settings of BUSI, we partition this dataset into training, validation, and test sets at a ratio of 8:1:1.The FJTU is a thyroid ultrasound dataset established from the Fujian Provincial Hospital for four sub-types of thyroid, e.g., thyroid adenoma (TA), follicular carcinoma (FC), follicular variant of PTC (FV-PTC), and medullary carcinoma (MC). It consists of 1,969 ultrasound images from 290 patients. Five-fold cross-validation is utilized for this dataset. In addition, a subset of the data, FJTU-H (349 images from FC and 174 images from FV-PTC), includes gender information and 9 heterogeneous attributes annotated by professional doctors. The attribute distribution exhibits severe heterogeneity within the same category as in Fig. 5 . The FJTU-H is utilized for dataset analysis of heterogeneity and validation of heterogeneity cause C. Compared methods. In order to comprehensively validate the effectiveness of MPCCI, we make comparisons with various methods. Initially, we select four representative deep learning models with backbone architectures of ResNet18 [18] , VGG16 [40] , ViT [9] , and Mamba [15] for image processing. These four methods utilize distinct feature extraction mechanisms to identify image features, facilitating the assessment of MPCCI performance across different architectures. Additionally, we employ several representative supervised learning methods for image classification, such as CAB-Net [17] and CAD_PE [24] , along with data augmentation techniques like MixupNet [51] and MixStyleNet [54] , as well as invariant feature learning method Fishr [39] , to further evaluate the performance of MPCCI. It is noteworthy that for the BUSI dataset, we also incorporate state-of-the-art modality-specific methods for breast cancer disease, including TNTs [16] , BVA Net [46] , HoVer-Trans [35] , and MIB Net [42] , for comparative analysis with MPCCI.Evaluation metrics. We evaluate MPCCI using four commonly used metrics in classification tasks: accuracy (Acc), precision (P), recall (R), and F1-score (F1).In our experiments, we utilize ResNet18 as the backbone of MPCCI. All medical images are resized to 128 × 128 pixels. The model is trained using the SGD optimizer, with learning rates set to 0.0001/0.0001/0.001/0.0001 for the CT COVID-19, BUSI, FJPH, and FJTU datasets respectively. All experiments are conducted on a single Nvidia RTX3090 GPU, with batch sizes set to 10/10/128/10. The hyperparameter 𝛼 is set to 0.5, while 𝜆 is set to 0.1/0.1/1/0.1 for the four datasets respectively.For the baseline methods, we reproduce the source codes of ResNet18 [18] , VGG16 [40] , ViT [9] , Mamba [15] , CABNet [17] , MixupNet [51] , MixStyleNet [54] , Fishr [39] , and CAD_PE [24] . All experimental results are based on the average of five experiments conducted with different random seeds. The results of TNTs [16] , BVA Net [46] , HoVer-Trans [35] , and MIB Net [42] are directly cited from the original papers, as their source codes are not publicly accessible.Table 1 , Table 2 , and Table 3 present the overall performance of MPCCI and the compared methods. On CT COVID-19 dataset, our MPCCI method exhibits strong performance, underscoring its efficacy and robustness in COVID-19 image classification tasks. Specifically, it achieves the highest performances of 96.10% and 96.28% on precision and F1-score respectively. On the BUSI dataset, MPCCI achieves the best average results of four evaluation metrics, demonstrating its superiority. Compared to MIB Net, which takes advantages of multi-task learning method in both classification and segmentation, our MPCCI outperforms it with improvements of 0.26%, 0.99%, 2.94%, and 2.19% in accuracy, precision, recall, and F1-score, respectively. This demonstrates that our approach can perform well in diagnosing breast cancer in ultrasound images with only instance-level labels. On the FJPH dataset, the performance of MPCCI consistently surpasses the runner-up by 1.36%, 1.20%, and 3.83% in accuracy, recall, and F1-score, respectively. The results highlight its consistent superiority across various datasets and underscores its potential for practical applications in medical image analysis. On the FJTU dataset, we test the accuracy for the four sub-types. Compared to SOTA and baseline methods, the proposed MPCCI achieves the best performance across all categories. While our method generally outperforms the compared methods, it occasionally lags behind by 1% to 2% in Precision or Recall. This discrepancy may stem from our method's suboptimal performance in handling specific categories. A potential improvement direction is to leverage prototype learning for enhancing the classification boundaries [41] .Ablation Study. We conduct ablation studies on CT COVID-19 and FJPH datasets to explore the individual contributions of each component of the proposed MPCCI. The four components evaluated are MVFE, MPC, and its two contained criteria (i.e., IP & MT). The results of ablation studies are presented in Table 4 with AB1: ResNet18 (baseline), AB2: AB1+MVFE, AB3: AB2+MPC, AB4: AB3+IP, and AB5: MPCCI (AB4+MT). The results show that both MVFE and MPC contribute to the improvements in performance compared to the baseline, demonstrating that MPCCI based on FDA can effectively evaluate the causal effect from image to label. Moreover, the results also reveal that it is crucial to consider both IP and MT factors simultaneously, as the performance drops significantly when only IP is considered.Validation of Heterogeneity Cause C. The gender/age is the unobserved confounding factor C [1] . Inspired by this work [1] , the FJTU-H dataset is divided into male and female groups. Subsequently, we conduct generalization tests on these two groups separately. The higher generalizability indicates that the algorithm is less influenced by the confounding factor of gender. The results are shown in Table 5 . It can be seen that MPCCI is less affected by the confounding factor C (gender).Data Set Analysis. We utilize the FJTU-H dataset to validate the effectiveness of our method in addressing heterogeneity. Two sets of control experiments are conducted: random splitting and splitting by low/high heterogeneity. The Pearson correlation in low and high heterogeneity groups are 0.8 and 0.5, respectively. The experimental results are shown in Table 6 . It can be found that in the random splitting group, the baseline method (ResNet18) and our MPCCI method achieve similar results. However, in the splitting by heterogeneity group, our method significantly outperforms the baseline, demonstrating its effectiveness in handling the heterogeneity issue.Experimental Extensions. In this section, we aim to address three questions to provide more detailed analysis of the proposed method: 1) What is the optimal number of expert networks required to optimize feature views? 2) How do the mixing mechanism and fusion module contribute to the performance of MPC? 3) Can the interpretability of MPCCI be quantitatively assessed? To answer the first question, we conduct experiments by changing the number of expert networks from one to nine on the FJPH dataset and observe the performance. The results in Fig. 6 (a) show that the optimal performance is achieved with five expert networks, indicating that an increased number of views does not correlate directly with enhanced performance. To address the second question, we assess the performance of "MPC1" mode (where the mixing mechanism is removed, and A is fused with only the source multi-view prototypes S 𝑐 ) and "MPC2" mode (where the fusion module is removed, and Â is calculated by randomly mixing A and S 𝑐 ). From Fig. 6 (b), we observe that both the mixing mechanism and the fusion module are necessary for MPC, demonstrating its ability to simulate the feature representation of lesions in various states. To answer the third question, we visualize the class activation maps (CAM) [53] for four samples in the FJPH and CT COVID-19 datasets using baseline (ResNet18), Fishr, and MPCCI in Fig. 7 . The lesions in the original images have been marked by professional doctors and shown with red circles. By comparing the locations of the lesions and visualization results, we can see that MPCCI can attend to the entire lesion, while ResNet18 and Fishr can only focus on a small part of the lesion or miss it entirely. Therefore, the CAM results of MPCCI can provide interpretable results to doctors, thereby facilitating medical diagnosis. 
conclusion: 
 In this paper, we propose a novel approach MPCCI for enhanced medical image classification by addressing the unmeasurable confounding factors present in medical imaging analysis. Leveraging FDA, MPCCI estimates the total causal effect of an image on its corresponding label, thus mitigating the negative effects of the confounders. The proposed approach comprises an MVFE module with spatial-channel attention, allowing multi-view features to serve as mediators in FDA, and an MPC module to effectively apply causal intervention on the mediators. An adaptive training strategy, including IP and MT, is introduced to maintain the stable training during the feature exchange process. Experimental results on four medical datasets demonstrate the effectiveness of MPCCI, achieving high accuracy, precision, recall, and F1-score in diagnosing COVID-19, breast cancer, lymph node metastasis, and thyroid.In the future, we plan to conduct extensive validation studies across a wider spectrum of medical conditions and imaging modalities. By rigorously evaluating the performance of MPCCI on diverse datasets encompassing a myriad of medical scenarios, we aim to demonstrate its efficacy and versatility in facilitating accurate and reliable diagnostic decision-making. 
 
 Appendix:  


Decision: The paper introduces the Mixed Prototype Correction for Causal Inference (MPCCI), a novel method addressing challenges related to image heterogeneity in medical image classification. By integrating causal inference with adaptive training strategies, the method aims to mitigate the impact of unseen confounding factors and enhance diagnostic accuracy. The experimental evaluation on three medical image datasets demonstrates the effectiveness of MPCCI in diagnosing diseases such as COVID-19, breast cancer, and lymph node metastasis.
Despite some limitations, such as the need for more comprehensive comparative experiments and detailed explanations of hyperparameter selection, the method demonstrates significant potential for clinical application. The strengths of the paper, including its methodological innovation, rigorous experimental design, and superior performance, outweigh the limitations. The authors have effectively addressed most of the concerns raised during the review process, making this submission a valuable contribution to the field and suitable for presentation as a poster at ACM Multimedia 2024.

Reviews: review: paper_summary: paper_summary: The present paper introduces a novel method named Mixed Prototype Correction for Causal Inference (MPCCI) to address challenges related to image heterogeneity in medical image classification. By integrating causal inference with adaptive training strategies, this method aims to mitigate the impact of unseen confounding factors and enhance the accuracy of deep learning models in disease diagnosis.
In the experimental section, the authors evaluated the performance of MPCCI on three medical image datasets, including CT and ultrasound image modalities. The experimental results demonstrate that MPCCI exhibits superior diagnostic accuracy and reliability in diagnosing diseases such as COVID-19, breast cancer, and lymph node metastasis. The method outperformed comparative approaches in terms of accuracy, precision, recall, and F1 score.
strength_weakness: strength_weakness: #### Accurate Problem Identification
The paper effectively addresses the issues of heterogeneity and unseen confounding factors in medical image classification. By presenting a targeted solution, it holds significant potential for improving the accuracy of disease diagnosis.

#### Methodological Innovation
By incorporating causal inference theory, specifically the front-door adjustment strategy, to address confounding factors in medical image analysis, the study introduces a novel approach that offers a fresh perspective on the field.

#### Rigorous Experimental Design
The authors conducted extensive experimental validation across multiple datasets, including different modalities of medical images. This rigorous approach enhances the credibility of the research findings.

#### Superior Performance
Compared to existing techniques, MPCCI demonstrates superior performance across multiple evaluation metrics, including accuracy, precision, recall, and F1 score.

#### Clinical Application Potential
The proposed method shows promising performance in diagnosing various diseases, indicating its potential application in clinical practice.

#### Excellent Results Presentation
In terms of results visualization, the paper provides clear and informative images that vividly illustrate the advantages of the proposed method and its focus on lesion areas. This makes the data presentation exceptionally transparent and easy to understand.
limitations: limitations: #### Incomplete Comparative Experiments
The study lacks comprehensive comparative experiments regarding the selection of CNN structures in the MCFE module. The paper primarily presents results based on ResNet18 without comparing them to other CNN architectures. It is crucial to explain whether the choice of feature maps impacts subsequent results and to justify why ResNet18 was ultimately selected as the base network for feature extraction.

#### Insufficient Data Explanation
Given that the paper's primary aim and innovation lie in integrating causal inference and adaptive training strategies to mitigate the impact of unseen confounding factors and enhance the accuracy of deep learning models in disease diagnosis, it fails to demonstrate whether the three datasets used actually encompass these issues. For instance, in the CT COVID-19 dataset, are there significant variations in CT images among infected individuals? The paper does not provide detailed information on this aspect, making it difficult for readers to assess whether the proposed method effectively addresses these problems.

#### Performance Limitations
While the method proposed in the paper performs well across various metrics, it is not always the best compared to other models. There are instances where other models outperform the proposed method in specific metrics.

#### Hyperparameter Selection
The paper does not offer detailed information on hyperparameter selection, such as learning rate, batch size, and choice of optimizer. It is essential to explain how these hyperparameters were tuned and determined.

#### Explanation of Heterogeneity Cause C
The paper mentions the heterogeneity cause C, but it remains unclear whether this C is automatically learned by the model or inherent in the data. Moreover, there is no professional validation of this cause C. It is necessary to determine whether C is accurately identified, its accuracy, and its effectiveness. Incorporating visual outputs would enhance the interpretability of the model's structure.

### Specific Questions and Recommendations

1. **Comparative Analysis of CNN Structures**: Have comparative experiments been conducted to evaluate different CNN structures within the MCFE module? The paper only shows results for ResNet18 but lacks comparisons with other CNN architectures. Please provide an explanation of whether the feature maps influence the subsequent results and justify why ResNet18 was chosen as the base network for feature extraction.

2. **Data Set Analysis**: Does the paper sufficiently address whether the three datasets used exhibit issues related to heterogeneity and unseen confounding factors? For example, in the CT COVID-19 dataset, are there significant differences in CT images among infected individuals? The paper should detail this aspect to help readers evaluate the method's effectiveness in addressing these problems.

3. **Performance Metrics**: While the proposed method performs well across various metrics, it is not consistently the best. Can you provide a more detailed comparison and analysis of where other models outperform your method and discuss potential improvements?

4. **Hyperparameter Details**: The paper should include detailed information on the selection and tuning of hyperparameters, such as learning rate, batch size, and optimizer choice. How were these hyperparameters adjusted to optimize performance?

5. **Validation of Heterogeneity Cause C**: The paper mentions a heterogeneity cause C but lacks clarity on whether it is model-generated or data-inherent. There is no professional validation of this cause. Could you elaborate on the accuracy and effectiveness of identifying C, and possibly include visual outputs to enhance the model's interpretability?
paper_summary: paper_summary: This work introduces a novel medical image classification model, the Mixed Prototype Correction for Causal Inference (MPCCI). To enhance interpretability, the model employs the front-door adjustment scheme to clarify the causal relationship between medical images and disease labels. An adaptive training strategy is implemented using the Information Purity (IP) and Maturity metrics to stabilize the training process. Numerical experiments on three medical image datasets demonstrate that MPCCI achieves high diagnostic accuracy.
strength_weakness: strength_weakness: - MPCCI offers a novel approach to interpreting medical image classification models.
- The adaptive training strategy ensures stable model training.
- Ablation studies demonstrate performance improvements with the front-door adjustment (FDA) framework.
limitations: limitations: - Some of the methods compared in the experiments are outdated.
- While the MPCCI model shows performance improvement on the FJPH dataset classification, it does not consistently outperform all compared methods across other datasets.
paper_summary: paper_summary: This paper proposes a causal inference-based mixed prototype correction model to improve classification performance by reducing the influence of confounding factors on the causal relationship between image features and labels. A multi-view feature extraction module is used to establish mediators, and a mixed prototype correction model is employed for causal intervention. To ensure the stability of the model, an adaptive training startegy is proposed. It is the first attempt to address the impact of medical image heterogeneity on disease diagnosis. The experimental results demonstrate the contributions of each module to the model's performance, and compared to other methods, the proposed approach achieves significant improvements.
strength_weakness: strength_weakness: The method takes a causal analysis perspective, which is innovative in the field of medical image analysis.
The method achieves good performance on multiple datasets with different types of diseases, indicating a certain level of generalization ability.
The MVFE module with spatial-channel attention mechanism enhances the model's representational power. And the MPC module  improves the classification accuracy.
The Class Activation Mapping (CAM) visualization provides the interpretability of the model.
limitations: limitations: The limitations of the method should be further discussed.
The performance improvement on the CT COVID-19 and BUSI datasets is small.
It may need to further validate on a broader range of disease types and imaging modalities. 
The paper  dose not provide the source code for  other researchers to replicate and verify the study's results.