Building Trust in Decision with Conformalized Multi-view Deep Classification

Authors: xiaodong  yue
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 13:09:08

Abstract: Uncertainty-aware multi-view deep classification methods have markedly improved the reliability of results amidst the challenges posed by noisy multi-view data, primarily by quantifying the uncertainty of predictions. Despite their efficacy, these methods encounter limitations in real-world applications: 1) They are limited to providing a single class prediction per instance, which can lead to inaccuracies when dealing with samples that are difficult to classify due to inconsistencies across multiple views. 2) While these methods offer a quantification of prediction uncertainty, the magnitude of such uncertainty often varies with different datasets, leading to confusion among decision-makers due to the lack of a standardized measure for uncertainty intensity. To address these issues, we introduce Conformalized Multi-view Deep Classification (CMDC), a novel method that generates set-valued rather than single-valued predictions and integrates uncertain predictions as an explicit class category. Through end-to-end training, CMDC minimizes the size of prediction sets while guaranteeing that the set-valued predictions contain the true label with a user-defined probability, building trust in decision-making. The superiority of CMDC is validated through comprehensive theoretical analysis and empirical experiments on various multi-view datasets.

Main paper: 
 introduction: 
 In practical environments encompassing image analysis, computer vision, data mining and multimedia applications, objects are frequently characterized by numerous modalities or feature varieties, jointly categorized as multi-view data [45] . This phenomenon has engendered the evolution of multi-view learning, a field that strives to construct superior methods for integrating multiple views to augment performance. Leveraging the achievements of deep learning, Multi-view Deep Learning (MDL) has emerged as a significant research trajectory aimed at acquiring shared multi-view representations through Deep Neural Networks (DNNs) [2, 20, 38, 42] .While multi-view deep learning demonstrates notable effectiveness in real-world scenarios, such developments have largely thrived within the closed-world assumption, which assumes an ideal scenario where data views are error-free and training aligns perfectly with testing distributions. However, the complex realities of the real world often challenge this idealized assumption, especially with the presence of noise-corrupted multi-view data, commonly referred to as outliers, which introduce risks during the testing phase. In response to this challenge, recent research has introduced uncertainty-aware multi-view deep classification methods [15, 16, 28, 29, 50, 51] that aim to assess uncertainty levels in predictions to enhance the reliability of multi-view results. Despite their evident value, these approaches encounter limitations that may elevate risks in real-world applications, particularly those of high stakes (e.g., medical diagnosis) that require critical decision-making.Specifically, imagine you're a doctor facing a critical medical decision, such as diagnosing pancreatic cancer, based on diagnostic information from a multi-view deep classifier. In scenarios where medical imaging is challenging to classify due to inconsistencies among different views or when encountering images of rare diseases not present in the training data, what output would you expect from the classifier to make the best decision? Existing uncertaintyaware multi-view methods typically produce a single-valued prediction with a degree of uncertainty. However, in such scenarios: 1) Information gathered from inconsistent views often leads to conflicting evidence among different classes (i.e., the Normal class and Serous Cystic Neoplasm (SCN) class as illustrated in Figure 1(a) ), making it challenging to reach a conclusive decision. 2) Although the uncertainty density distribution of rare diseases may differ from that of the original imaging, determining whether an image belongs to a rare disease, referred to as outliers in machine learning, can be difficult due to the small difference between outlier and normal data with a small magnitude (valued range from [0 -0.3]), as illustrated in Figure 1 (b). In contrast, the desired output should: 1) Provide possible prediction sets instead of deterministic single-valued predictions, allowing for the ruling in or ruling out of harmful decisions (such as diagnosing Pancreatic Ductal Adenocarcinoma (PDAC), a malignant tumor category). 2) Have the ability to make a clear judgment as to whether the samples belong to outliers in real-world environments across different datasets. 3) Provide a theoretical guarantee of coverage, ensuring that the true label is included within the prediction sets with a high probability.To this end, we introduce a novel Conformalized Multi-view Deep Classification (CMDC) method. It is designed to offer prediction sets, which are subsets of candidate labels explicitly including outlier classes for classification tasks, thereby enhancing decisionmaking reliability. Drawing inspiration from the Evidential Neural Network (ENN) [34] and the theory of Conformal Prediction (CP) [3] , our model initially captures the second-order probability of the class distribution to explicitly estimate prediction uncertainty caused by outliers, thus expanding the output space by incorporating outlier classes. Leveraging an effective multi-view fusion strategy and the conformal score function, we measure the similarity between calibration samples and testing inputs, enabling the construction of suitable prediction sets. Through end-to-end training, our method minimizes the size of prediction sets while providing a coverage guarantee that the set-valued predictions contain the true label with a user-defined probability, thereby building trust in decision-making. In summary, the specific contributions of this paper are as follows:(1) We introduce a novel multi-view model aimed at providing reliable prediction sets, which are subsets of candidate labels that explicitly include outlier class, thereby enhancing the reliability of decisions. This approach marks a new paradigm in multi-view deep classification. (2) The proposed model presents a unified framework designed to ensure that the true label is contained within the prediction sets. This is achieved through the incorporation of a suitable multi-view fusion strategy and conformal score function, both of which are implemented in an optimizable (learnable) manner. (3) Theoretical analysis demonstrates that our model effectively decreases the empirical risk of learning outcomes as the number of views increases, while also controlling the conformal risk within a small margin. 
related works 2.1 multi-view deep learning: 
 In recent years, numerous multi-view deep learning methods have emerged, integrating multi-view learning with DNNs to achieve high performance [26, 27, 46, 48, 49] . Traditional methods include Canonical Correlation Analysis (CCA) combined with DNNs [2, 42, 43] . CCA aims to maximize the correlation between different views to find a common representation. Deep CCA (DCCA) [2] focuses on capturing nonlinear relationships, while Deep Canonically Correlated AutoEncoder (DCCAE) [42] trains autoencoders to obtain common representations. Deep Tensor Canonically Correlation Analysis (DTCCA) [43] learns complex nonlinear transformations of multiple views (more than two) of data to ensure that the resulting representations are linearly correlated in high order.Recently, several uncertainty-based multi-view classification methods [15, 16, 28, 29, 44, 50, 51] have been proposed. Han et al. [15, 16] focus on uncertainty estimation using Dempster's rule of combination to produce reliable results, while Liu et al. [29] treat the fusion of multiple views as an opinion aggregation process. Despite their success in multi-view deep classification, these methods fall short in addressing the reliability of decisions in real-world environments. One limitation is their restriction to providing a single class prediction per instance, which may lead to inaccuracies when dealing with samples that are difficult to classify due to inconsistencies across multiple views. Additionally, while these methods quantify prediction uncertainty, its magnitude often varies with different datasets, causing confusion among decision-makers due to the lack of a standardized measure for uncertainty intensity. In contrast, our method generates set-valued predictions instead of single-valued ones and integrates uncertain predictions as an explicit class category, thereby enabling more reliable decisions.Uncertainty estimation is pivotal for ensuring trustworthy decisionmaking and has been extensively explored in deep learning [1, 11] . Two primary types of uncertainties, namely aleatoric uncertainty and epistemic uncertainty, serve as indicators of the likelihood that predictions given by models are incorrect. Uncertainty-based deep learning models can generally be classified into Bayesian and non-Bayesian models. Bayesian deep learning approaches [18] facilitate the interpretation of model parameters by replacing deterministic weight parameters with distributions. These methods employ various approximation techniques, such as Monte Carlo (MC) dropout [12] , Markov Chain Monte Carlo (MCMC) [33] , Variational Inference (VI) [39] , and Laplacian approximations [31] , to estimate uncertainty. However, Bayesian methods often entail high computational costs. To address this, several non-Bayesian methods have been proposed. For instance, Deep Ensemble [19, 24] combines multiple deep neural networks to estimate predictive uncertainty. More recently, another approach termed as ENN [34] , directly models uncertainty using subjective logic [21] and evidence theory [7, 35] instead of indirectly through weights. Simultaneously, several uncertainty estimation algorithms based on the Dirichlet distribution have been introduced [30, 34] , aiming to estimate uncertainty in distribution shifts. Given the robustness of subjective logic and evidence theory in estimating uncertainty caused by outliers without additional computational burden, we employ the Evidential Neural Network (ENN) as the backbone of our uncertainty-aware module.Conformal Prediction (CP) serves as an uncertainty quantification tool in machine learning, aimed at enhancing the reliability of models [3, 8, 36, 37] . At its core, CP involves a statistical calibration technique for predictive models, offering finite-sample guarantees on the predicted uncertainty. By specifying a permissible level of risk, conformal calibration adjusts the model's predictions into predictive sets or intervals. These sets are designed to contain the true test labels or targets at least as frequently as dictated by the specified risk level. There are two primary steps in CP. Firstly, in the prediction step, a trained model computes conformity scores, which gauge the similarity between calibration examples and a testing input. Secondly, in the calibration step, these conformity scores from a set of calibration examples are utilized to determine a threshold for constructing prediction sets that adhere to the coverage constraint (e.g., 1 -𝜏 = 90%). Recent studies have proposed conformity scores based on ordered probabilities in CP for single-view classification [4, 32] . However, there has been limited focus on multi-view classification within an end-to-end training setup, especially regarding the integration of the outlier class into the prediction set through the development of an effective conformal score function. This gap in the literature motivates the development of the work presented in this manuscript. 
method: 
 In this section, we start with notations and problem statement, followed by detailing the proposed CMDC and its key components.Considering a multiclass classification task on a multi-view datasetX = 𝑥 𝑣 𝑖 𝑉 𝑣=1 , 𝑦 𝑖 𝑁 𝑖=1with 𝑉 views, where 𝑥 𝑣 𝑖 ∈ X denotes the 𝑣 𝑡ℎ view of the 𝑖 𝑡ℎ sample, and 𝑦 𝑖 ∈ Y = {1, . . . , 𝐾 } represents the associated class label. Our goal is to learn a conformalized multiview model ℎ 𝑥 𝑣 𝑖 𝑉 𝑣=1 ; 𝜃 that maps 𝑥 𝑣 𝑖 𝑉 𝑣=1 to a prediction set 𝐶 𝜃 𝑥 𝑣 𝑖 𝑉 𝑣=1 ⊆ 2 𝐾+1 containing the true label 𝑦 𝑖 with a coverage probability of 1 -𝜏, where the (𝐾 + 1) 𝑡ℎ class represents the outlier category. It's evident that achieving a large coverage with a small set size will result in better prediction performance.To realize this goal, we need to address three key challenges: 1) Explicitly estimating prediction uncertainty caused by outliers in multi-view setting; 2) Developing an effective fusion strategy to enhance performance, including classification accuracy and prediction set size, through the fusion of multiple views; 3) Guaranteeing that prediction sets contain the true label and correctly identify outlier classes with a user-defined probability (𝑖.𝑒., 1 -𝜏 = 90%).To this end, we draw inspiration from both the ENN and CP and propose a novel conformalized multi-view deep classification model. This model not only ensures the effective fusion of multiple views but also provides prediction sets with guaranteed coverage, thereby enhancing the precision and reliability of classification results in the face of challenges posed by inconsistent views or outliers.To incorporate the outlier category into the possible label sets, providing intuitive assessment to decision-makers, we expand the multi-view output space 𝑦 𝑖 ⊆ 2 𝐾 to 𝑦 𝑖 ⊆ 2 𝐾+1 . To achieve this, two aspects need assurance: 1) A reliable tool to identify outliers in the input space; 2) Ensuring that the probabilities of the outlier class and the other normal classes sum to one. To this end, we first assess the quality of data views through uncertainty estimation in ENN, a common practice in existing state-of-the-art works [15, 16, 28, 29] .Formally, let ℎ 𝑣 𝑥 𝑣 𝑖 ; 𝜃 be the output of the penultimate layer (i.e., logits layer) of a neural network for the sample 𝑥 𝑣 𝑖 in single view. Instead of using softmax to predict a single categorical estimation, we use another activation function 𝑎 (•) (i.e., ReLU) to capture a non-negative evidence 𝒆 𝑣 𝑖 = 𝑎 ℎ 𝑣 𝑥 𝑣 𝑖 ; 𝜃 , 𝒆 𝑣 𝑖 = 𝑒 𝑣 𝑖1 , . . . , 𝑒 𝑣 𝑖𝐾 , over 𝐾 categories. Then we have:EQUATIONwhere 𝑏 𝑣 𝑖𝑘 is the belief mass of the 𝑘 𝑡ℎ class based on the collected evidence, 𝑢 𝑣 𝑖 is the estimated prediction uncertainty that indicates the lack of total evidence, and𝑆 𝑣 𝑖 = 𝐾 𝑘=1𝑒 𝑣 𝑖𝑘 + 𝐾. When the evidence is low, leading to a low belief mass 𝑏 𝑣 𝑖𝑘 and high uncertainty 𝑢 𝑣 𝑖 in prediction. Conversely, when the evidence is high and concentrated in one category, the uncertainty will be low.According to the Eq.( 1 O 𝑖 = O 1 𝑖 ⊕ O 2𝑖 can be achieved by the following rule:𝑏 𝑖𝑘,𝑘≠𝐾+1 = 𝑏 1 𝑖𝑘,𝑘≠𝐾+1 𝑏 2 𝑖 (𝐾+1) + 𝑏 2 𝑖𝑘,𝑘≠𝐾+1 𝑏 1 𝑖 (𝐾+1) 𝑏 1 𝑖 (𝐾+1) + 𝑏 2 𝑖 (𝐾+1) -𝑏 1 𝑖 (𝐾+1) 𝑏 2 𝑖 (𝐾+1) , 𝑏 𝑖 (𝐾+1) = 𝑏 1 𝑖 (𝐾+1) 𝑏 2 𝑖 (𝐾+1) 𝑏 1 𝑖 (𝐾+1) + 𝑏 2 𝑖 (𝐾+1) -𝑏 1 𝑖 (𝐾+1) 𝑏 2 𝑖 (𝐾+1).(EQUATIONWithout loss of generality, let O 𝑖 = 𝒃 𝑖 = {𝑏 𝑖𝑘 } 𝐾+1 𝑘=1 be multi-view output space fused from 𝑉 views, we can easily expand the above rule to multiple views fusion:EQUATIONFollowing the above definition, we also can derive the multi-view evidence 𝒆 𝑖 = [𝑒 𝑖1 , . . . , 𝑒 𝑖𝐾 ], over 𝐾 categories using the Eq.(1).According to the above augmented multi-view output space, we will introduce how to formalize the conformalized multi-view predictor to provide reliable prediction sets. EQUATIONHere, 𝜋 𝑥 EQUATIONAnd the calculation of q should ensure the following property for the predicted label ŷ𝑡𝑒𝑠𝑡 :EQUATIONRemark 1. Notably, in contrast to the inclusion of the predicted label sets ŷ𝑖 ⊆ 2 𝑘 , 𝑘 ∈ [1, . . . , 𝐾] entailing true labels for supervising calibration on held-out dataset, the inclusion of ŷ𝑖 = {𝐾 + 1} introduces an unsupervised dimension due to the lack of true outlier labels. Thus we should guarantee it does not return too many false positives. To this end, we consider it as a hypothesis testing problem. Points that are rejected as outliers have a p-value less than 𝜏 for the null hypothesis of exchangeability with the calibration data. Then we have the Eq.( 5) and Eq.( 6) to guarantee the method have a suitable error control as shown in Proposition 2.In this section, we outline our approach to learning a model that produces reliable prediction sets in a multi-view setting. To enhance clarity, we structure the learning process as a multi-task learning problem, comprising three distinct objectives:(1) Maximizing the integration of joint information from multiple views to support correct class identification.(2) Minimizing information while increasing uncertainty when predictions are incorrect.(3) Minimizing the size of the prediction sets.At a high level, the first part (1) involves fitting our multi-view data to the proposed model, thereby enhancing its ability to capture the information for accurate predictions. The second part (2) enforces a prior to eliminate incorrect information, thereby calibrating the estimated confidence to obtain a well-calibrated model. The third part (3) aims to optimize the efficiency of prediction sets. Notably, we do not incorporate a coverage loss as the proposed model theoretically ensures that the true label is contained in prediction sets with a probability of 1 -𝜏.In summary, for the model ℎ 𝑥 𝑣 𝑖 𝑉 𝑣=1 ; 𝜃 , the overall objective function is formulated as follows:EQUATIONwhere ℓ 𝑎𝑐𝑐 (𝜃 ) 𝑖 represents the classification loss term, ℓ 𝑐𝑎𝑙 (𝜃 ) 𝑖 is the confidence calibration loss term and ℓ 𝑠𝑖𝑧𝑒 (𝜃 ) 𝑖 is the set size regularization term. Specifically, for a training sample 𝑥 𝑣 𝑖 𝑉 𝑣=1 with one-hot label 𝒚 𝑖 , we treat Dirichlet distribution 𝐷𝑖𝑟 (𝒑 𝑖 |𝒆 𝑖 + 1 ) as a prior on the likelihood, obtain the negative log likelihood loss:EQUATIONwhere 𝑆 𝑖 = 𝐾 𝑘=1 𝑒 𝑖𝑘 + 𝐾. Notably, due to the absence of a true outlier label, our focus in this part is solely on the classification of the remaining classes.To enhance the ability of uncertainty estimation, we propose a confidence calibration loss that regularizes training by applying a confidence prior 𝑐 𝑖 = 1 -𝑢 𝑖 as a penalty, aiming to minimize information for inaccurate predictions:EQUATIONThis loss intuitively imposes a penalty whenever there is an error in the prediction and scales with the confidence of our inferred posterior, which shrinks the evidence for incorrect classes to 0. Larger confidence values are not penalized as long as the prediction is close to the target. Then, for minimizing the size of prediction sets, we introduce a set size regularization term:EQUATION)To avoid the indicator functions, we use approximation [6] by the sigmoid function 𝜎 (𝑥) = 1/1 + exp (-𝑥) to make the set size regularization term differentiable:EQUATION)Here, we set the 𝛾 = 1 in order to not penalize singletons. Furthermore, we use 𝜆 𝑡 1,2 = 𝜆 0 exp {-(ln 𝜆 0 /𝑇 ) 𝑡 } as an annealing factor, where 𝜆 0 is a small positive constant, 𝑡 is the current number of training epoch, and 𝑇 is the total number of training epochs. As the 𝑡 increasing to 𝑇 , 𝜆 𝑡 will be exponentially from 𝜆 0 to 1. 
theoretical analysis: 
 In this part, we theoretically prove the effectiveness of our model. We begin by stating a proposition indicating that integrating more views leads to lower empirical risk compared to using fewer views.Proposition 1 (Classification Performance Improvement).Let 𝑅 ℎ 𝑥 𝑣 𝑖 • 𝑣=1 ; 𝜃 = 1 𝑁 𝑁 𝑖=1ℓ 𝑎𝑐𝑐 (𝜃 ) 𝑖 be the empirical risk for the data with any subset of views. Under the regularization of confidence calibration, the inequalityEQUATIONis satisfied with 𝑉 ′ ∈ N * ≤ 𝑉 .Next, we provide a theoretical guarantee that the conformal risk from the CMDC method is bounded by a user-defined value. ; 𝜃 is defined as 𝐶𝑅 (𝜃 ), we have:EQUATION)where 𝐶𝑅 𝑚𝑖𝑠 (𝜃 ) indicates the risk of miscoverage that true label is not contained in the prediction set, and 𝐶𝑅 𝑜𝑢𝑡𝑙𝑖𝑒𝑟 (𝜃 ) is the risk of points are wrongly identified as outliers, defined as:𝐶𝑅 𝑚𝑖𝑠 (𝜃 ) = E P 𝑦 𝑡𝑟𝑢𝑒 𝑡𝑒𝑠𝑡 ∈ {1, . . . , 𝐾 } ∉𝐶 𝜃 𝑥 𝑣 𝑡𝑒𝑠𝑡 𝑉 𝑣=1. ( 14)𝐶𝑅 𝑜𝑢𝑡𝑙𝑖𝑒𝑟 (𝜃 ) = E P { ŷ𝑡𝑒𝑠𝑡 = {𝐾 + 1}} ∈𝐶 𝜃 𝑥 𝑣 𝑡𝑒𝑠𝑡 𝑉 𝑣=1.Notably, the above guarantee also ensures that:EQUATION)Finally, we give a generalization analysis to demonstrate the effectiveness of our model.Let 𝑋 be a set of 𝑁 samples with label 𝑌 , 𝜃 ∈ B 𝑑 be the parameter of loss function in a finite 𝑑-dimensional unit ball. Define generalization risk as:EQUATION)Let 𝜃 * = argmax 𝜃 ∈B 𝑑 𝑅 (𝜃 ) be the optimal parameter in the unit ball, 𝜃 = argmax 𝜃 ∈𝐴 𝑅 (𝜃 ) be the optimal parameter of empirical risk among a candidate set 𝐴. With probability at least 1 -𝛿 we have,EQUATION)Proposition 3 shows the generalization of CMDC can approach the optimal result in the order 𝑂 √︁ 𝑑 ln (𝑁 )/𝑁 , where 𝑑 indicates the number of parameters in our model and 𝑁 denotes the number of samples. In summary, this section demonstrates the effectiveness of our method in terms of the classification performance, conformal risk control and the generalization. 
experiments: 
 We extensively evaluate the proposed method on real-world multiview datasets, considering metrics such as Accuracy (ACC), Expected Calibration Error (ECE) [14] , Inefficiency (Ineff), and Coverage (Cov), where Ineff measures the size of prediction sets and Cov measures the coverage of true label contained in the prediction sets. Implementations. For all datasets except MRNet, we utilize fully connected networks, while MRNet employs ResNet-18 as its backbone. We set 𝜏 = 0.1, which means guaranteeing the coverage with a probability of 90%. The Adam optimizer [22] is used to train the network, where 𝑙 2 -norm regularization is set to 1𝑒 -5 . We use 5-fold cross-validation to select the learning rate from 1𝑒 -5 , 3𝑒 -4 , 1𝑒 -3 , 3𝑒 -3 . For all multi-view datasets, the data is partitioned into training (70%), testing (20%), and calibration (10%) sets. Furthermore, we run 5 times for each method to report the average values in Figures or the mean values and standard deviations in Tables. The model is implemented by PyTorch on one GeForce RTX 4090 GPU with 24GB memory.To investigate the effectiveness of our model, we conducted comparisons with several popular models for multi-view deep classification. Given the widespread use of CCA-based methods in multi-view learning, we initially compared our approach with three representative CCA-based methods: DCCA [2] , DCCAE [42] , DTCCA [43] . These methods employ CCA to derive latent representations, followed by classification using a support vector classifier (SVC). Next, we compared our model with three advanced multi-view representation learning approaches: CPM-Nets [47] , DUA-Nets [13] , and MVTCAE [17] . As in [17] , logistic regression served as the basic classifier in these methods, utilizing the learned representations as input. Finally, we contrasted our model with four state-of-the-art uncertainty-based methods: TMC [15] , TMDOA [29] , ETMC [16] , and SMDC [28] . Notably, these four methods share the same neural network architecture as ours.The comprehensive experimental results are presented in Table 1 . It is evident that our approach showcases state-of-the-art performance across all multi-view datasets except for the CUB dataset, as indicated by the ACC metric. The reason CMDC doesn't achieve the best performance on the CUB dataset is that our method utilizes a part of the training data as the calibration dataset during training. Specifically, we train our method on 70% of the training It is evident that although the estimated uncertainty of these two cases is higher compared to other test points, making a clear judgment regarding their outlier status is challenging due to the relatively small value of prediction uncertainty. In contrast, our method provides a threshold q, facilitating straightforward judgments. 2 , which denote the highest accuracy among individual single-views, we conduct evaluations across all views for our primary components. As illustrated in Table 2 , our method consistently surpasses all other combinations, thus confirming the efficacy of our major technical components. across diverse multi-view datasets, signifying enhanced alignment between model confidence and actual outcomes. Furthermore, ℓ 𝑠𝑖𝑧𝑒 reduces the size of prediction sets, enhancing the reliability of results. Additionally, Cov demonstrates that our method achieves coverage with 1 -𝜏 = 90%.In this section, we delve deeper into examining the practical significance of our method in real-world settings.Uncertainty Estimation with Simulated Noises. Initially, we evaluate the capability of our method in estimating uncertainty. We introduce Gaussian noise 𝝐 to our samples, effectively rendering them as 'polluted' samples, i.e., 𝒙 𝒗 𝒊 = 𝒙 𝑣 𝑖 + 𝜂𝝐, where 𝜂 represents the noise intensity, and 𝝐 is sampled from a Gaussian distribution N (0, I). Figure 3 visually represents samples affected by Gaussian noise (𝜂 = 100) across the three planes in the MRNet dataset. The estimated density of uncertainty in the MRNet dataset for the ACL task is depicted in Figure 5 . It can be observed that when all views are affected by high-level noise, the polluted multi-view samples can be regarded as outliers. Consequently, the overall uncertainty increases significantly compared to the clean multi-view data, validating the good performance of our method in estimating prediction uncertainty caused by outliers.Real-world Case Studies. Next, we apply our method to the real-world medical dataset MRNet to detect whether knee MRI scans of patients belong to the ACL injury category. Figure 2 showcases the prediction uncertainty density distribution estimated by five state-of-the-art uncertainty-aware multi-view deep classification methods for the imperfect Case 1194 and Case 1212 in the MRNet dataset. The experimental results from Figure 2 indicate that the uncertainty values from all five methods cannot provide clear evidence to support the doctor in making accurate diagnoses, posing a high risk due to the small difference between the estimated uncertainty values of Case 1194 and Case 1212 and those of the other samples. Our model addresses this issue by providing clear outlier class judgments using the threshold q with statistical guarantees.Furthermore, we also present the probability maps of Case 1212 in the MRNet dataset as illustrated in Figure 4 , showing the class probability and uncertainty degree across five uncertainty-based methods. It is evident that the difference between the probability of the ACL class and the Normal class is small due to the conflicts among the noisy views in Case 1212, resulting in random predictions. Our model addresses this issue by providing prediction sets using the thresholds p and q to mitigate the risks posed by inconsistencies or uncertainties from the samples, thereby enhancing the reliability of multi-view results. 
conclusion: 
 In this study, we introduce a novel Conformalized Multi-view Deep Classification (CMDC) model aimed at generating reliable prediction sets that encompass the outlier class. This facilitates clear assessments of uncertain predictions attributed to outliers, thereby enhancing trust in decision-making. Leveraging an effective multiview fusion strategy and conformal score function, the CMDC model provides theoretical assurances regarding the inclusion of the true label in prediction sets with a user-defined probability, while also controlling risk within a narrow margin. Empirical investigations highlight the superior performance of our approach across various multi-view datasets. 
 
 Appendix:  


Decision: As all the reviewers have given positive scores, I recommend accepting this paper.

Reviews: review: paper_summary: paper_summary: This paper combines CP and EDL into the multiview classification tasks.
strength_weakness: strength_weakness: 1. The theories are solid.
2. The experiments are extensive.
limitations: limitations: 1. Why we need to introduce Conformalized Predictions into  Multiview Classification? Its not clear.
2. Comparison with ECML?
 Xu C, Si J, Guan Z, et al. Reliable Conflictive Multi-View Learning[J]. arXiv preprint arXiv:2402.16897, 2024.
3. Reliability experiments, i.e. density figures are limited to specific datasets. Can you provide more results?
paper_summary: paper_summary: This paper introduces a novel method for multi-view deep learning, named Conformalized Multi-view Deep Classification (CMDC), which aims to enhance the reliability of decisions made from multiple views. Unlike existing uncertainty-aware multi-view deep learning methods that generate single-valued predictions with vague confidence levels, CMDC produces set-valued predictions that explicitly include an outlier category. This approach ensures that the true label is contained within the prediction set with a probability of at least 1-τ. Both empirical and theoretical analyses validate the superiority of the proposed method.
strength_weakness: strength_weakness: (1) The innovation presented in this paper is significant. Through an end-to-end training, CMDC effectively combines the best features of both ENN and CP approaches, offering a compelling new perspective for advancing the field of multi-view deep learning.

(2) The proposed method achieves superior performance compared to previous methods.

(3) The authors provide a comprehensive theoretical analysis to support the effectiveness of the proposed method.
limitations: limitations: (1) Figure 1 doesn't illustrate the idea of this paper but highlights an existing problem. Adding an overall framework diagram of the proposed method would aid in providing a clearer understanding of the paper.

(2) There are typographical errors present, for example, in Equation (15), $\notin$ should be replaced with $\in$.

(3) Does the size of the validation sets affect the coverage of the proposed method? A discussion regarding the impact on coverage with small-scale datasets would be interesting.
paper_summary: paper_summary: This paper introduces conformalized multi-view deep classification and aims to address the limitations of current uncertainty-aware multi-view deep classification methods. Compared with the former research, the authors focus on set-level predictions rather than single-level ones. The proposed method minimizes the size of prediction sets while guaranteeing true labels with probability, and thus more reliable decision-making. The authors also conduct a series of experiments to validate the superiority of the proposed method.
strength_weakness: strength_weakness: 1. The paper is easy to follow.
2. Introducing conformal multi-view deep classification into this task is indeed interesting.
limitations: limitations: 1. I am curious why to fuse as eq.2. Were there any other forms to fuse the outputs?
2. In my view, multi-view classification aims to utilize multiple views with uncertainty to make the final classification more reliable and precious. According to L423-426, it seems that the correct classification or incorrect classification is quite crucial. However, how does it figure out whether the classification is correct or incorrect without labels in test scenarios?
3. In Table 1, the proposed method fails in CUB. 
4. After reading the whole paper and reviewing related work, it seems that the novelty of this paper is limited. The authors introduce the conformal prediction into the multi-view deep classification task while more from intuition and lack of theoretical justification. It should be
clarified.

I am willing to increase my score if the concerns are adequately answered.
paper_summary: paper_summary: ## **1. Summary.**
This work addresses the two challenges of inaccurate single-class prediction and non-standard uncertainty measurement in multi-view classification, and combines uncertainty and anomaly categories to perform multi-view deep classification.
strength_weakness: strength_weakness: ## **2. Strengths.**
i) This work has a reasonable motivation, a clear starting point, and comprehensive related work.

ii) Compared to previous works such as TMC or ETMC, the introduction of anomaly classes expands the boundaries of trustworthy multi-view classification.

iii) Extensive theoretical analysis and experimental verification have demonstrated the effectiveness of the proposed method.
limitations: limitations: ## **3. Some minor confusion.**
After carefully reading the paper and appendix, I believe it is a carefully prepared work. However, I still have some minor confusion and suggestions regarding the theoretical analysis and other details, and I hope to discuss and reach a consensus with the authors.

**Theoretical analysis of this work:** It is worth acknowledging that the authors provided some analysis to ensure the correctness and generalization of the proposed method at the theoretical level, which is commendable. However, **1)** Regarding **Propositions 1-3**, after the authors show them in the main text, further explanations should be provided to help readers understand why the proposed proposition can be achieved, such as with **Proposition 1**, how to achieve the "**Classification Performance Improvement**"? After **Proposition 2**, how to keep the "**Conformal Risk Control Guarantee**"? Based on the **Proposition 3**, how to obtain the "**Generalization**"? More explanations can allow readers to clearly and intuitively feel the theoretical efforts of the proposed method. 
**2)** A small question is, why is **Proposition 1** proven starting from $\ell_{acc}(\cdot)$? Because according to the reader's intuitive perception, the improvement of classification performance should start from $\ell_{overall}(\cdot)$ to prove the proposed statement. Similarly, why prove generalization **Proposition 3** from $\ell_{acc}(\cdot)$? Shouldn't it be the cumulative risk of global losses $\ell_{overall}(\cdot)$? 
**3)**  What is the $\tau$ in **Proposition 2**? I hope the authors can further clarify to solve my confusion.

**Other small details:** **1)** Algorithms 1-3 provide a good summary of the proposed method, but it is presented from an algorithmic perspective. It is best to add a framework diagram (similar to Figure 1) to visually present the proposed method.
**2)** The theoretical analysis lacks an important part: Computational complexity, which should also be considered for inclusion in the main text to enhance the rigor of the proposed method.

**Overall, this is a good work, motivation, innovation, method presentation, and experiments are all carefully presented. And if the authors can provide effective reply on my concerns, I am willing to increase my rating.**