Dynamic Evidence Decoupling for Trusted Multi-view Learning

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 04:04:38

Abstract: Multi-view learning methods often focus on improving decision accuracy, while neglecting the decision uncertainty, limiting their suitability for safety-critical applications. To mitigate this, researchers propose trusted multi-view learning methods that estimate classification probabilities and uncertainty by learning the class distributions for each instance. However, these methods assume that the data from each view can effectively differentiate all categories, ignoring the semantic vagueness phenomenon in real-world multi-view data. Our findings demonstrate that this phenomenon significantly suppresses the learning of view-specific evidence in existing methods. We propose a Consistent and Complementary-aware trusted Multi-view Learning (CCML) method to solve this problem. We first construct view-opinions using evidential deep neural networks, which consist of belief mass vectors and uncertainty estimates. Next, we dynamically decouple the consistent and complementary evidence. The consistent evidence is derived from the shared portions across all views, while the complementary evidence is obtained by averaging the differing portions across all views. We ensure that the opinion constructed from the consistent evidence strictly aligns with the ground-truth category. For the opinion constructed from the complementary evidence, we only require it to reflect the probability of the true category, allowing for potential vagueness in the evidence. We compare CCML with state-of-the-art baselines on one synthetic and six real-world datasets. The results validate the effectiveness of the dynamic evidence decoupling strategy and show that CCML significant outperforms baselines on accuracy and reliability. We promise to release the code and all datasets on GitHub and show the link here.

Main paper: 
 introduction: 
 In real-world scenarios, different data modalities or features could be treated as multiple views. For example, in autonomous vehicle systems, cameras and lidars collect images and points; in the field of healthcare, a patient's comprehensive condition can be assessed through multiple types of examinations. Multi-view learning Figure 1 : Visualization of the dynamic semantic vagueness phenomenon. The ground-truth category of the first instance is "apple pie". When considering the image view alone, it becomes challenging to differentiate between the categories "waffles" and "apple pie". For the second instance, both views provide explicit differentiation between the categories. aims to synthesize both consistent and complementary information from these multiple views, leading to a more comprehensive understanding of the data [4, 20, 32] . It has generated significant and wide-ranging influence across multiple research areas, including classification [17] , clustering [13, 29] , retrieval [41] and large language models [22] .Most existing multi-view learning methods primarily emphasize enhancing decision accuracy, often overlooking the crucial aspect of decision uncertainty. This limitation significantly restricts the applicability of multi-view learning in safety-critical scenarios, such as autonomous vehicle systems. Therefore, to predict the reliability of the results, researchers have proposed many multi-view uncertainty quantification methods in recent years. The pioneering work [9] , Trusted Multi-view Classification (TMC), calculates and aggregates the evidence [27] of all views. TMC utilizes this aggregated evidence to parameterize the class distributions, enabling the estimation of class probabilities and uncertainties. In order to train the model effectively, TMC requires the estimated class probabilities to align with the ground-truth labels. Building upon this research, researchers have proposed novel evidence aggregation paradigms such as sum [21] , (weighted) average [35] , element-wise dot product [10] , etc. These methods enhance the reliability and robustness in the presence of various challenges, such as feature noise [8, 25, 42] , incomplete views [33] , etc.Regrettably, the evidence aggregation paradigms in these methods rely on an assumption: the data of each view can distinguish all categories. However, real-world multi-view data exhibit the semantic vagueness phenomenon, which means that one view may exhibit ambiguity or uncertainty in its categorization. For example, as shown in Figure 1 , the ground-truth category of the first instance is "apple pie", while the image view is difficult to differentiate between the categories "waffles" and "apple pie". For this view, existing evidence aggregation paradigms encourage only the evidence of the category "apple pie" is large due to the common evidence complying with this. This overlooks the fact that the evidence for the category "waffles" is also substantial in the data, which would significantly impact the overall learning process and compromise its effectiveness. This motivates us to delve into the semantic vagueness problem in trusted multi-view learning.In this paper, we propose a Consistent and Complementaryaware trusted Multi-view Learning (CCML) for this problem. First, we construct view-specific evidential Deep Neural Networks (DNNs) to learn the view-specific evidence, which represents the level of support for each category obtained from the data. In the multiview fusion stage, we dynamically decouple the consistent and complementary evidence. The consistent evidence is derived from the consistent portions across all views, while the complementary evidence is obtained by averaging the differing portions from all views. This separation allows us to capture both the shared information and the unique aspects of each view. In the training stage, we enforce strict alignment between the opinion constructed from the consistent evidence and the ground-truth category. This is achieved by adjusting the probabilities of the true and false categories, as well as enhancing the separation between them. As for the opinion constructed from the complementary evidence, we only require it to reflect the probability of the true category, allowing for potential vagueness in the evidence. In the test stage, we aggregate the consistent and complementary evidence to make a decision.The main contributions of this work are summarized as follows: 1) we identify the semantic vagueness phenomenon in multi-view data, which can significantly suppress the learning of view-specific evidence in existing trusted multi-view learning methods; 2) we propose the CCML method to tackle this problem. CCML effectively addresses the negative impact of semantic vagueness through two key strategies: dynamically decoupling the consistent and complementary evidence, and allowing potential vagueness in the complementary evidence; 3) we conduct empirical comparisons between CCML and state-of-the-art trusted multi-view learning baselines on a synthetic toy dataset and six real-world datasets. The experimental results not only validate the effectiveness of the proposed dynamic decoupling strategy but also demonstrate that CCML surpasses the baseline methods in terms of accuracy and reliability. 
related work: 
 The proposed CCML is a new uncertainty-aware multi-view fusion methods. Therefore, in this section, we briefly review two lines of related work, multi-view fusion and uncertainty-aware deep learning, to better motivate this work.Multi-view fusion is highly effective in a wide range of tasks, as it combines information from multiple sources or modalities [6, 12, 39] . Based on the fusion strategy employed, existing deep multi-view fusion methods can be broadly categorized into two main pipelines: feature fusion [1, 19] and decision fusion [14] . Feature fusion methods aim to capture the interactions between different views at the feature level. For example, canonical correlation analysis and its variants [1, 40] maximize the correlation of the multi-view latent representations. Matrix factorization methods [34] decode the multi-view common representation to view-specific data via basis matrices. Following this line, Xu et al. explicitly model consistent and complementary information [2] at the highest abstraction level by the group sparseness constraint [34] . More recently, researchers have used deep neural networks to decouple the complex consistent and complementary information at representation level [4, 37] . A major challenge of feature fusion methods is that low-quality views may adversely affect the representation of other views. Decision fusion methods [5, 23, 26, 28] mitigate this problem by integrating the decision results from different views. We follow this line and propose a new method for decoupling consistent and complementary information at the decision level.Traditional deep learning methods have made remarkable progress in various domains. However, they are unable to provide uncertainty estimates in predictions, which is increasingly important in real-world scenarios. To tackle this challenge, researchers have proposed uncertainty-aware deep learning methods. One approach is Bayesian Neural Networks [24] , which considers the variation in results caused by variations in the data distribution as uncertainty. However, the high computational cost associated with Bayesian neural networks limits their practical applications. As a result, researchers begin to explore more efficient methods for estimating uncertainty. The Monte Carlo dropout method [7] is one such approach. It involves using multiple instances of input data to obtain multiple prediction results, from which uncertainty measures can be calculated. Another method, EDL [27] , calculates categoryspecific evidence and considers the lack of evidence as a source of uncertainty using a single deep neural network. Recently, TMC [9] extends EDL to the multi-view learning area. Following this line, researchers have proposed various evidence aggregation paradigms, including sum [21] , (weighted) average [35] , element-wise dot product [10] , etc. However, these paradigms often overlook the phenomenon of semantic vagueness present in real-world multi-view data, which can significantly hinder the learning of view-specific evidence. The proposed CCML effectively addresses the issue by allowing for potential vagueness in the complementary evidence. 
the method: 
 In this section, we first introduce the trusted multi-view classification problem and semantic vagueness phenomenon, then present the pipeline and loss function of CCML in detail.In this section, we introduce the trusted multi-view classification problem and semantic vagueness phenomenon in detail. For the 𝐶 classification problem, considering a dataset 𝐷 = {{𝒙 𝑣 𝑛 } 𝑉 𝑣=1 , 𝒚 𝑛 } 𝑁 𝑛=1 has 𝑁 instances with 𝑉 views, where 𝒙 𝑣 𝑛 denotes the feature vector for the 𝑣-th view of the 𝑛-th instance, and the one-hot vector 𝒚 𝑛 ∈ {0, 1} 𝐶 denotes the ground-truth label of the 𝑛-th instance. The semantic vagueness phenomenon indicates that some elements of {𝒙 𝑣 𝑛 } may not well distinguish certain categories. For example, as shown in Figure 1 , the image view of the first instance is hard to differentiate between the categories "waffles" and "apple pie". The goal is to accurately predict 𝒚 𝑛 and provide the associated prediction uncertainties 𝑢 𝑣 𝑛 ∈ [0, 1] to measure the decision reliability.As shown in Figure 2 , the overall architecture is a decision-level fusion pipeline, which consists of the view-specific evidence learning stage and the evidential multi-view fusion stage. The view-specific DNNs {𝑓 𝑣 (•)} 𝑉 𝑣=1 learn the view-specific evidence, which indicates the level of support for each category based on the data. In the fusion stage, we dynamically decouple the consistent evidence (𝒆 𝑐𝑜𝑛 ) and complementary evidence (𝒆 𝑐𝑚𝑝 ). 𝒆 𝑐𝑜𝑛 is extracted from the consistent portions across all views, while 𝒆 𝑐𝑚𝑝 is obtained by averaging the differing portions across all views. In the training stage, we establish different principles for 𝒆 𝑐𝑜𝑛 and 𝒆 𝑐𝑚𝑝 , respectively. For the testing stage, we aggregate 𝒆 𝑐𝑜𝑛 and 𝒆 𝑐𝑚𝑝 to make a decision.Learning. Many conventional multiview learning methods utilize softmax layers to produce standard outputs to address multi-classification problems in neural classification networks. However, these methods only provide class probabilities without considering the reliability of the results [11] . Due to the single-point estimation paradigm of softmax scores, they tend to produce overconfident outputs, even when the predictions are incorrect. To address these issues and achieve accurate uncertainty prediction, we employ EDL [27] by replacing the softmax layer with a ReLU activation to obtain non-negative evidence.We also introduce the subjective logic [15] framework to form opinions. In this framework, the parameter 𝜶 of the Dirichlet distribution 𝐷𝑖𝑟 (𝒑|𝜶 ) is associated with the belief distribution in the framework of evidence theory, where 𝒑 is a simplex representing the probability of class assignment. We collect evidence {e 𝑣 𝑛 }, using view-specific evidential DNNs {𝑓 𝑣 (•)} 𝑉 𝑣=1 . The corresponding Dirichlet distribution parameters are𝜶 𝑣 = 𝒆 𝑣 + 1 = [𝛼 𝑣 1 , • • • , 𝛼 𝑣 𝐶 ]𝑇 . After obtaining the distribution parameters, we can calculate the subjective opinion, 𝝎 𝑣 = (𝒃 𝑣 , 𝑢 𝑣 ) of the view including the quality of beliefs 𝒃 𝑣 and the quality measure of uncertainty 𝑢, where 𝒃 𝑣 = (𝜶 𝑣 -1)/𝑆 𝑣 = 𝒆 𝑣 /𝑆 𝑣 , 𝑢 𝑣 = 𝐾/𝑆 𝑣 , and 𝑆 𝑣 = 𝐾 𝑘=1 𝛼 𝑣 𝑘 is the Dirichlet intensity.After training evidential DNNs to learn view-specific evidence, we observe that the evidence produced by vagueness views contains support for multiple categories. Among these categories, the true categories would generate relatively higher amounts of evidence than other categories. Therefore, we cannot require only the groundtruth category has large evidence. We utilize this property in the subsequent evidence fusion stage.Existing trusted multi-view learning methods face limitations when addressing the phenomenon of semantic vagueness. This is primarily because, during the generation of view-specific evidence, the indistinguishable error categories in vague views produce a similar amount of evidence as the indistinguishable categories. Many methods struggle to distinguish between equal amounts of evidence for correct and incorrect categories. As a result, we are motivated to tackle this problem by decoupling the consistent and complementary evidence and allowing for potential vagueness in the complementary evidence. For views 𝑣 𝑖 and 𝑣 𝑗 , the view-specific deep learning network of view 𝑣 𝑖 will produce similar amounts of evidence for classes 𝑐 1 and 𝑐 2 , while the network of view 𝑣 𝑗 will produce similar amounts of evidence for classes 𝑐 2 and 𝑐 3 when classifying instances of class 𝑐 2 , Effective classification should be achieved through complementary information from view 𝑣 𝑖 and 𝑣 𝑗 respectively (𝑣 𝑖 determines that the instance is one of 𝑐 1 or 𝑐 2 , and 𝑣 𝑗 determines that the instance is one of 𝑐 2 or 𝑐 3 , the final classification result should be 𝑐 2 utilizing the complementary information of 𝑣 𝑖 and 𝑣 𝑗 ).Therefore, we need to decouple view-specific evidence of different views into consistent and complementary evidence. One previous work [36] addresses the semantic vagueness phenomenon by setting fixed relation degradation layers for semantic vagueness categories. However, the extent of semantic vagueness varies among different instances, as reflected by the difference in consistency and complementarity. It is necessary to use dynamic decoupling strategies according to the differences of consistency and complementarity in the classification tasks, instead of using fixed relations. For this purpose, we propose a dynamic consistent and complementary evidence decoupling strategy. Definition 3.1. (Consistent and Complementary Evidences). For the view-specific evidences {𝒆 𝑣 = (𝑒 𝑣 1 , . . . , 𝑒 𝑣 𝐶 )} 𝑉 𝑣=1 . The consistent evidence 𝒆 𝑐𝑜𝑛 and complementary evidence 𝒆 𝑐𝑚𝑝 are defined as:EQUATIONEQUATIONEQUATIONwhere 𝒆 𝑐𝑜𝑛 denotes the minimum evidence of each class supported by each view. 𝒆 𝑐𝑚𝑝 denotes the complementary evidence.The consistent evidence 𝒆 𝑐𝑜𝑛 is obtained by aggregating the consistent portions from all views, while the complementary evidence 𝒆 𝑐𝑚𝑝 is calculated as the average of the differing portions across all views. This distinction is made because the complementary evidence of all views {𝒆 𝑣 -𝒆 𝑐𝑜𝑛 } 𝑉 𝑣=1 capture the varying information between different views, encompassing both complementarity and conflict. It is important to note that this information does not always enhance the accuracy of results; instead, it can introduce additional uncertainty. Consequently, retaining the entire set of complementary evidence is not advisable. Instead, a more suitable approach is to preserve it by taking the average [35] . By employing this strategy, we dynamically separate the consistent and complementary evidence from the view-specific evidence. Finally, we aggregate the two pieces of evidence on average to form the final opinion, which is used in the test stage. In the following section, we will outline how our deep learning networks are trained using distinct principles for handling consistent and complementary evidence.In this section, we will present the training process of CCML. The objective is to deal with view-specific, consistent, and complementary evidence separately. Specifically, the consistent evidence should closely approximate the true label, while the complementary evidence serves as a supplementary component, allowing for more relaxed constraints. We will elaborate on these components below.View-specific Loss Function. The evidential DNNs are obtained by converting the softmax layer of traditional DNNs into ReLU. Therefore, we obtain the non-negative outputs as evidence. We introduce an adjusted cross-entropy loss function to ensure that all views can generate appropriate non-negative view-specific evidence for classification:EQUATIONwhere 𝜓 (•) is the digamma function. The view-specific loss function of 𝒙 𝑣 𝑛 is defined as:EQUATIONwhere 𝜶 𝑣 𝑛 = 𝒆 𝑣 𝑛 + 1 is the parameters of the corresponding Dirichlet distribution, 𝒆 𝑣 𝑛 = 𝑓 𝑣 (𝒙 𝑣 𝑛 ) represent the evidence vector predicted by the network.The consistent evidence is obtained by aggregating the consistent portions of all views. Consequently, we impose a stringent alignment between the opinion constructed from the consistent evidence and the ground-truth category. To accomplish this, we adjust the probabilities assigned to the true and false classes while simultaneously enhancing the distinction between them. To achieve this objective, we introduce two principles: the Error Reduction Principle and the Separability Principle.Error Reduction Principle. The error reduction principle highlights that during the training process, the evidence generated for incorrect categories may inadvertently increase due to the limited availability of counterexamples. This misleading evidence has the potential to introduce challenges to the classification process. Therefore, to reduce evidence for incorrect labels, we introduce the Kullback-Leibler (KL) divergence into the loss function:EQUATIONwhere 𝐷 (𝒑 𝑛 |1) is the uniform Dirichlet distribution, 𝜶 𝑛 = 𝒚 𝑛 + (1 -𝒚 𝑛 ) ⊙ 𝜶 𝑛 is the Dirichlet distribution parameter after removing the evidence of the ground-truth category from the original parameter 𝜶 𝑛 . 𝜆 𝑡 = 𝑚𝑖𝑛(1, 𝑡/𝑇 ) ∈ [0, 1] is annealing coefficient, acting as the balance factor. As the training process progresses, 𝜆 𝑡 continues to increase, enhancing the influence of KL divergence, to prevent premature convergence of misclassified instances to the uniform distribution.Separability Principle. The Separability Principle emphasizes the importance of creating a significant distinction between the evidence supporting different categories during the classification process. This principle allows the classifier to more accurately distinguish between different categories. For instance, consider the belief masses of opinions 𝒃 1 = (0.4, 0.4) and 𝒃 2 = (0.3, 0). The total amount of belief mass in 𝒃 2 is more than 𝒃 1 . However, the greater degree of separation of 𝒃 2 makes it more contribution to the classification result than 𝒃 1 . To enhance classification performance, we need to increase the degree of separation during the training process. Therefore, we quantify the degree of separation of the belief masses supporting different classes in opinion using the separation degree.Definition 3.2. (Separation Degree). For the subjective opinion 𝝎 = (𝒃, 𝑢, 𝒂), where 𝒃 = (𝑏 1 , . . . , 𝑏 𝐶 ), the separation degree can be defined as:EQUATION)Our goal is to increase the separation degree of consistent opinions. There are two approaches to increase it: the first approach involves adding a constraint to maximize the degree of separation. However, this approach may lead to an increase in the total amount of evidence across all categories. This is contrary to our original intention of controlling uncertainty based on the total amount of evidence. Therefore, we design the following method.We first convert consistent evidence into consistent opinions 𝝎 𝑐𝑜𝑛 = (𝒃 𝑐𝑜𝑛 , 𝑢 𝑐𝑜𝑛 , 𝒂 𝑐𝑜𝑛 ) and adjust it to obtain the final consistent opinion 𝝎 𝑐𝑜𝑛 = ( 𝒃 𝑐𝑜𝑛 , 𝑢 𝑐𝑜𝑛 , 𝒂 𝑐𝑜𝑛 ):EQUATIONwhere 𝑏 𝑐𝑜𝑛 𝑐 = 𝑝𝑜𝑤 (𝑏 𝑐𝑜𝑛 𝑐 , 𝛽) ∈ [0, 1] is the belief mass supporting classes 𝑐, 𝛽 is a hyper-parameter which is bigger than 1. By the power operation 𝑝𝑜𝑤 (𝑏 𝑐𝑜𝑛 𝑐 , 𝛽), the separation degree would increase. The reason behind this is the increasing disparity in the confidence mass that supports each category. We also demonstrate theoretically that this approach can increase the separation degree, which is elaborated in the appendix. Therefore, we only need this simple operation to achieve the increase of separation degree without changing the total amount of evidence.Under the guidance of the above two principles, we can derive the consistent loss function:EQUATIONwhere 𝜶 𝑐𝑜𝑛 represents the corresponding parameters of the final Dirichlet distribution of the consistent opinion. The consistent loss function can refine the consistent evidence. After increasing the separation degree, the cross-entropy loss function minimizes the model's deviation from the true label, and the KL loss function reduces the evidence for incorrect categories. The combination of these two loss functions can maximize the impact of consistent evidence on the model training process.Vagueness Principle. Complementary evidence usually represents complementary or even conflicting information between different views. Its reliability is generally lower than that of the consistent evidence. As a supplement to the consistent evidence, the complementary evidence need not necessarily enhance the degree of separation or reduce false labels. It accounts for potential vagueness in multi-view data. Therefore, it is only required to reflect the probability of the true category.According to this principle, we define the complementary loss function as follows:EQUATIONwhere𝜶 𝑐𝑚𝑝 𝑛 = 𝒆 𝑐𝑚𝑝 𝑛+ 1 is the corresponding Dirichlet parameter of the complementary evidence.. By synthesizing the above objectives, the overall loss function for a specific instance {𝒙 𝑣 𝑛 } 𝑉 𝑣=1 is formulated as:EQUATIONwhere 𝛿, 𝛾 > 0 are hyper-parameters. 
experiment: 
 In this section, we show the empirical results of CCML in making trusted decisions for multi-view inputs. We first apply CCML to a synthetic toy example to investigate its performance in solving the semantic vagueness problem, then we evaluate CCML on six realworld multi-view datasets and compare it with existing multi-view learning methods.The major advantage of CCML compared with pioneer uncertaintyaware methods is the ability to perceive consistent and complementary information between different views. Therefore, we conducted a set of comparative experiments with TMC in the Toy Dataset to investigate the effectiveness of CCML in solving semantic vagueness questions and to explicitly achieve higher accuracy results. We set view 1 cannot distinguish categories 𝑐 2 and 𝑐 3 and view 2 cannot distinguish categories 𝑐 1 and 𝑐 2 , respectively. Specifically, the toy dataset consists of 2 views, each with 1200 data instances {𝒙 𝑣 𝑛 } 1200 𝑛=1 belonging to 3 categories, 𝑐 1 , 𝑐 2 , and 𝑐 3 , with 400 data instances in each category. The underlying latent space has 9 dimensions, with three for each category. The first 3 dimensions and the last 3 dimensions are private to categories 𝑐 1 and 𝑐 3 respectively, and the middle dimensions are a shared dimension for 𝑐 2 and 𝑐 3 . Each element of {𝒗 𝑣 𝑛 } 1200 𝑛=1 is the sum of a number sampled from a gamma-distributed Γ(1, 0.9), the noise sampled from Gaussian distribution 𝑁 (0, 0.1), and a consistent term of 0.5. We randomly generated 12 × 9 basis matrices 𝑾 𝑣 for the two views, with elements drawn from a uniform distribution 𝑈 (0.4, 1), We randomly set 30 percent of the elements to be zero to simulate the real-world multi-view mapping pattern. Then we generated a noise matrix 𝒁 𝑣 , and the elements of 𝒁 𝑣 drawn from by the Gaussian distributions 𝑁 (0, 0.5) and 𝑁 (0, 1), respectively. We use the equation 𝒙 𝑣 𝑛 = 𝑾 𝑣 𝒗 𝑣 𝑛 + 𝒛 𝑣 to generate data instances. In the Toy Dataset, we set the last 3 columns of 𝑾 1 to be 0 and the first 6 columns of 𝑾 2 to be 0, respectively. Therefore, the data instances in view 1 cannot distinguish between categories 𝑐 2 and 𝑐 3 , and the data instances in view 2 cannot distinguish between categories 𝑐 1 and 𝑐 2 . The Toy Dataset represents a strong 3 .In addition, to obtain a better understanding of the advantages of CCML over TMC in addressing semantic vagueness problems, we have visualized the evidence produced by CCML and TMC for instances in category 𝑐 2 , as illustrated in Figure 4 .Based on the experimental results, we can obtain the following conclusions: (1) The accuracy rate of CCML on the Toy Dataset is 98.07%, which is significantly higher than the accuracy rate of TMC on the same dataset, which is 94.73%. This indicates that CCML demonstrates superior performance in addressing semantic vagueness problems. (2) Upon observation in Figure 4 , it is evident that when facing semantic vagueness problems, the evidence generated by a single view in the TMC method is subdued due to the inherent ambiguity of the view. Consequently, this results in insufficient evidence. The insufficiency arises because vague views fail to distinguish between certain categories, resulting in decision conflicts within the TMC framework, and ultimately, an inadequate evidence supply for both categories. In contrast, CCML can effectively generate sufficient evidence from each view and effectively leverage the information from vague views through the consistent evidence component. As a result, CCML constructs accurate and effective evidence supporting the 𝑐 2 category. Datasets. HandWritten 1 comprises 2000 instances of handwritten numerals ranging from '0' to '9', with 200 patterns per class. It is represented using six feature sets. Scene15 2 includes 4485 images from 15 indoor and outdoor scene categories. We extract three types of features HOG, LBP, and GIST. CUB [30] consists of 11788 instances associated with text descriptions of 200 different categories of birds, we focus on the first 10 categories and extract image features using GoogleNet and corresponding text features using doc2vec. LandUse [38] comprises 2100 images from 21 classes. We extract HOG and SIFT features as two views. PIE 3 contains 680 facial instances belonging to 68 classes. We extract intensity, LBP, and Gabor as 3 views. Colored-MNIST 4 includes 1200 instances of numerals with RGB coloured backgrounds which consist of three colours (red, green, blue) for each number. We extract RGB and HOG features as two views.1 https://archive.ics.uci.edu/dataset/72/multiple+features 2 https://figshare.com/articles/dataset/15-Scene_Image_Dataset/7007177/1 3 http://www.cs.cmu.edu/afs/cs/project/PIE/MultiPie/Multi-Pie/Home.html 4 https://www.kaggle.com/datasets/youssifhisham/colored-mnist-dataset/ Compared Methods. (1) Single-view uncertainty aware methods contain: EDL (Evidential Deep Learning) [27] and IEDL [3] which is the SOTA method that combines Fisher's information matrix. (2) Multi-view feature fusion methods contain: DCCAE (Deep Canonically Correlated AutoEncoders) [31] is the classical method, which employs autoencoders to seek a common representation, DCP (Dual Contrastive Prediction) [18] is the SOTA method that obtains a consistent representation. (3) Multi-view decision fusion methods contain: CALM (Enhanced Encoding and Confidence evaluating framework) [42] takes advantage of cross-view consistency and diversity to improve the efficacy of the learned latent representation, ETMC (Enhanced Trusted Multi-view Classification) [10] , addresses the uncertainty estimation problem and produces reliable classification results. RCML (Reliable Conflictive Multiview Learning) [35] is the SOTA method that proposed a fusion strategy for solving conflictive problems.Implementation Details. We briefly introduce the details of the experiment. We utilize fully connected networks with a ReLU layer to extract view-specific evidence. The Adam optimizer [16] is used to train the network, where L2-norm regularization is set to 1𝑒 -5 . We employ 5-fold cross-validation to select the learning rate from the options of 3𝑒 -3 , 1𝑒 -3 , 3𝑒 -4 , 1𝑒 -4 . In all datasets, 20% of the instances are allocated as the test set. The average performance is reported by running each test case five times. obtain that: (1) multi-view methods generally outperform singleview methods, which illustrates the necessity of using multiple views in classification tasks. (2) There is a large difference in accuracy among different methods on the Colored-MNIST dataset, demonstrating significant variation in the ability of different methods to solve the semantic vagueness problem. (3) For the majority of real-world datasets, CCML shows performance comparable to state-of-the-art methods and has outstanding performance on Colored-MNIST datasets. (4) This result indicates that CCML significantly improves the ability to handle the semantic vagueness phenomenon while ensuring good performance on general classification tasks. The reason would be attributed to the consistent and complementary dynamic decoupling method, we would further verify this in the next analysis and other experiments.Ablation Study. To validate the effectiveness of the evidence decoupling strategy, separability principle, and error evidence reduction principle, we construct a detailed ablation study that performs different combinations of these modules to achieve degradation methods. Specifically, to verify the effectiveness of the evidence decoupling module, we compared the approach with the three modules removed to using only that module. The effectiveness of the other two modules was validated based on the decoupling module respectively. We conduct the degradation methods above on the dataset Colored-MNIST. The results are shown in Table 3 . From the result, we can observe the outstanding effectiveness of our dynamic decoupling strategy. Therefore, to further validate the effectiveness 4 . Compared to other methods, CCML achieves higher accuracy because it decouples and processes consistent and complementary evidence separately, giving higher confidence and increasing the separation degree for consistent evidence while averaging complementary evidence. This allows CCML to adjust the corresponding uncertainties based on the consistency between different views, instead of considering only one type of evidence or applying the same separation processing strategy to both consistent and complementary evidence. From the ablation study, we verified the effectiveness of each module of CCML.Parameter Analysis. The separation increase module can improve the model's performance in solving classification tasks. We verify the influence of the separation increase module on the model by changing the value of the hyperparameter 𝛽. Specifically, we gradually increase the hyperparameter 𝛽 from 1 to 5 and observe CCML's performance on all datasets as shown in Figure 5 . The results show that the accuracy of the model increases first and then decreases with the change of 𝛽. In particular, the effectiveness of 𝛽 in the semantic vagueness phenomenon is demonstrated on the dataset Colored-MNIST by the significantly increased accuracy. We can obtain the point that when the 𝛽 is too large, it will have a negative impact on the model, and the appropriate 𝛽 value can improve the performance of the model to a certain extent.Uncertainty Estimation. To further evaluate the estimated uncertainty, we used the original dataset CUB and constructed out-ofdistribution instances. We consider the original test instances as in-distribution data and add Gaussian noise 𝑁 (0, 𝑰 ) with intensity 𝜂 to the test instances in one view, constructing out-of-distribution test instances. Specifically, given the noise vector 𝝐 sampled from the Gaussian distribution, the out-of-distribution test instances 𝒙 𝑖 = 𝒙 𝑖 + 𝜂𝝐 𝑖 . The uncertainty associated with out-of-distribution data is expected to be higher compared to that of in-distribution data. The noise intensity increases in the sequence (𝜂 = 0.1, 1, 2, 5). We perform CCML on the data with added Gaussian noise and visualize the uncertainty, as shown in Figure 7 , where the blue curves represent in-distribution instances and the red curves represent outof-distribution instances with noise intensity 𝜂. The results show that as the intensity of noise increases, the overall distribution of the uncertainty also increases, demonstrating that the data have higher uncertainty with greater noise. This also demonstrates the ability of our method to estimate uncertainty.Noise Views Impact Analysis. To verify the robust classification ability of CCML for noise views, we construct noise instances and conduct CCML and other baseline methods. Specifically, we select about half of the views and add Gaussian noise with distribution 𝑁 (0, 𝜎 2 ) to the instances of these noise views, where 𝜎 is the standard deviation. The result is shown in Figure 6 . From the experimental results, we can observe that as the standard deviation of Gaussian noise increases, the classification performance of all methods deteriorates. However, the deterioration in CCML's accuracy is noticeably smaller than that for other baseline methods in most cases. The possible reason for this is that CCML can dynamically adjust its evidence obtained between different views. In the case of more noise within views, the consistency of evidence between viewpoints decreases, which prevents the training process from overly trusting the noise and thereby generating incorrect classification results. 
conclusion: 
 In this paper, we propose a CCML method to solve the semantic vagueness problem in trusted multi-view learning. CCML tries to dynamically decouple the consistent and complementary evidence from the view-specific evidence. It further processes consistent and complementary evidence according to different principles to achieve classification results and reliability. The experimental results on the synthetic toy dataset and six real-world datasets verified the effectiveness of the proposed decouple strategy and the performance superiority of CCML. 
 
 Appendix:  


Decision: This paper originally received mixed ratings, especially many comments on confusing formulations. However, after rebuttal, these questions have been resolved and all reviewers give a positive review. Thus AC recommends acceptance. Upon acceptance, it is recommended to include a discussion on tasks beyond classification.

Reviews: review: paper_summary: paper_summary: This paper proposes a trusted multi-view deep learning method through Dynamic Evidence Decoupling, which aims to mitigate the impact of the semantic vagueness phenomenon. Specifically, the method devises a consistent and complementary-aware multi-view fusion strategy to enhance the multi-view classification performance.
strength_weakness: strength_weakness: 1. Trusted multi-view deep learning is an interesting topic.

2. The experimental evaluation has demonstrated good performance in terms of accuracy.
limitations: limitations: 1. If I understand correctly, there appears to be a significant technical error: $\widetilde {{w^{con}}}$ cannot ensure that the sum of the $\widetilde {{b^{con}}}$ and the uncertainty $u^{con}$ equals 1 under a uniform prior $a^{con}=1$, which contradicts the principles of subjective logic.

2. The innovation of this paper is limited; several works [1-4] have been proposed to mitigate the impact of conflicting views through consistent constraint [1], dissonance uncertainty [3] and conflict fusion [2, 4]. However, there is no convincing evidence to demonstrate CCML's strengths compared to existing papers, especially the works of [2, 4].

3. Under the constraint of KL-divergence, which shrinks the incorrect evidence to zero and enhances the correct evidence, is it necessary to introduce a consistent loss function, or a complementary loss function? It would be better to provide a deeper empirical and theoretical analysis to validate the effectiveness of this method.

4. What role does uncertainty play in multi-view fusion? It seems that there is no improvement to enhance the reliability of the results, only to enhance the accuracy.

5. Why should the formulation of $e^{con}$ in Eq.(1a) be multiplied by $V$? How can consistent evidence be converted into consistent opinions? What is the meaning of $u^{con}$ compared to the overall $u$? How is $a^{con}$ defined? All of these questions should be clarified in the manuscript.

6. The experiment on toy samples should be compared with existing works [2,4] that address the same problem, in order to enhance the strength of this method, rather than focusing on the TMC.

[1] Trusted multiview deep learning with opinion aggregation. AAAI 2022.

[2] Uncertainty-Aware Multiview Deep Learning for Internet of Things Applications. TII 2023.

[3] Safe multi-view deep classification. AAAI 2023.

[4] Reliable Conflictive Multi-View Learning. AAAI 2024.
paper_summary: paper_summary: This paper demonstrates that the semantic vagueness phenomenon significantly suppresses the learning of view-specific evidence in existing methods. Then, they propose a CCML method to solve this problem by dynamically decoupling the consistent and complementary evidence. Experiment results validate the effectiveness of the dynamic evidence decoupling strategy and show that CCML significantly outperforms baselines on accuracy and reliability.
strength_weakness: strength_weakness: -This paper is well motivated. It points out a intersting problem, semantic vagueness phenomenon, in existing trusted multi-view learning methods. The authors also show 
intuitive examples in figure 1 to explain this.

- The proposed method is convenient and effective. The proposed CCML dynamically decouples the consistent and complementary evidence, and allowing potential vagueness in the complementary evidence.

- This paper is well written and the experiment shows excellent performance.
limitations: limitations: - The authors claim that "the evidence aggregation paradigms in these methods
rely on an assumption: the data of each view can distinguish all categories." It is better to detailedly claim the reason behind this conclusion.

- This is only an advise. The proposed method is only suitable to classification  task?
paper_summary: paper_summary: ## **1. Summary.**
This paper propose a Consistent and Complementary-aware trusted Multi-view Learning (CCML) method to solve the problem of semantic vagueness phenomenon in real-world multi-view data.
strength_weakness: strength_weakness: ## **2. Strengths.**
i) This work is different from traditional uncertainty estimation fusion methods. It distinguishes evidence through the principles of consistency and complementarity, and designs corresponding losses. Thus, this approach has a certain degree of innovation. 

ii) A certain degree of theoretical analysis provides theoretical support for the proposed framework. 

iii) The experimental design is reasonable and rich.
limitations: limitations: ## **3. Some details worth discussing.**
After carefully reading this paper, I have identified some limitations that can improve the quality of the paper and hope to discuss them with the authors.

**Contribution of this work:** The author mentioned "Decision fusion methods [5, 23, 26, 28] mitigate this problem by integrating the decision results from different views. We follow this line and propose a new method for decoupling consistent and complementary information at the decision level." in related work, the fact is indeed so. As for this paper, when we move our perspective to the loss section, we will find that it is very similar to the loss in work [1-2], but plays a different role. In other words, the contribution of the paper lies only in proposing an improved evidence theory fusion method based on the principles of consistency and complementarity. From this perspective, the contribution of this paper is incremental. But after reading, I have a clear understanding of the design concept of this paper, and it seems that the work is also easy for other readers to understand. Therefore, it is necessary to clarify the similarities and differences between this paper and other similar works, such as [1-2]. Therefore, I suggest the authors add a separate section to focus on discussing the similarities and differences between this work and other similar methods in terms of fusion models and losses.

[1] Trusted multi-view classification. ICLR 2021.

[2] Trusted multi-view classification with dynamic evidential fusion. TPAMI 2022.

**About theory:** I am pleased to see the author's efforts in theoretical support, including the proof in the Appendix. However, I also hope to see a theoretical analysis of the computational complexity of the proposed method. Furthermore, it would be even better if it could include an overall algorithmic process, which would be user-friendly for readers.

**About experiments:** **i)** Can the authors add some larger multi-view datasets for testing to show the performance under large-scale case? For example, NoisyMNIST (with 30,000 samples).
**ii)** Is it fair to only use accuracy (ACC) to reflect "trustworthiness" in trusted multi-view learning? The use of other indicators may bring progress to further trusted explanations, such as ECE, MCE [3].

[3] On calibration of modern neural networks. ICML 2017.

**Other details:** Some styles can be more unified to make reading more enjoyable for readers, such as subtitle "Experimental Setup" and "Implementation Details", they all can use command "\subsubsection" to form a unified style.
Some discussions on future work also need to be added. This can enable readers to clearly grasp the research trends of follow-up work.

**Overall, the ideas presented in this paper are somewhat novel, but there is still room for improvement. If authors can adopt some of the suggestions, I am willing to keep my rating.**
paper_summary: paper_summary: This work proposes a trusted multi-view classification framework that separates consistent and complementary evidences based on the existence of semantic vagueness phenomenon in multi-view learning.
strength_weakness: strength_weakness: ##  **Advantages.**
1. The motivation is clear and reasonable.
2. The method is easy to understand.
3. The experiments on simulated and real datasets show that the proposed method has good multi-view fusion effects.
limitations: limitations: ## **Some obstacles encountered.**
However, after reading the entire paper and the code in the supplementary materials, I still encountered obstacles in understanding this paper. 

### **1. The correctness of the code implementation.**

**In the code of model.py:** Let's put aside the confusing naming of variables, **(1)** In the function of **"def Evidence_DC"**, why is the **"alpha_con = E_con + 1"** added twice (after codes "E[v] = torch.sub(E[v], E_con)" and "alpha_a = E_a + 1")? Because according to the definition of ecidence theory [1-2], an evidence and its parameter can be presented as $\boldsymbol{\alpha}$=$\boldsymbol{e}$+1, as defined in this paper, too. It can be inferred that **$\boldsymbol{\alpha}^{con}$**=**$\boldsymbol{e}^{con}$**+1. But obviously, if the code is added with a **1** twice here, it clearly does not comply with the rules of evidence theory [1-2]. Therefore, I reasonably speculate that there is a certain technical error here.

**(2)** In addition, after the code "E_div = torch.div(E_div, len(alpha))", there are also some confusing codes appearing for readers. From the line "S_con = torch.sum(alpha_con, dim=1, keepdim=True)" to "# print(E_con[0])". Due to its chaotic naming, I reluctantly understood after carefully reading the code that this was to update Eq.(6) and obtain new evidence $\boldsymbol{e}^{con}$ and its parameters $\boldsymbol{\alpha}^{con}$. However, what confuses me the most are "Sum0_con = torch.sum(E_con, dim=1, keepdim=True)" and "Sum1_con = torch.sum(E_con, dim=1, keepdim=True)", and I understand that they are updating according to Eq.(6). Why do they count as the sum of categories of evidence? Shouldn't it be the sum of parameter categories? Because by definition in ecidence theory [1-2], $\boldsymbol{b}^{con}$ ($\tilde{\boldsymbol{b}^{con}}$ is the same idea) is related to Dirichlet intensity $S^{con}$ and $S^{con}$ is related to the sum of $\boldsymbol{\alpha}^{con}$ as $\sum_{k=1}^{K}\alpha_{k}^{con}$, rather than $\sum_{k=1}^{K} e^{con}_{k}$. 

**These two points seriously affect the correctness and reproducibility of the proposed method implementation, and I hope the author can clarify my these two concerns.**

[1] Upper and lower probabilities induced by a multivalued mapping. The Annals of Mathematical Statistics 1967.

[2] Evidential deep learning to quantify classification uncertainty. NIPS 2018.

### **2. Some concerns about the motivations.**
**About the motivations:**  The authors used a bimodal application example in the abstract and introduction to introduce the challenge of semantic vagueness, which is intuitively reflected in Fig. 1. However, when I moved on to the experimental setup section, I found that the datasets used by the authors should be divided into two different types of multi-views: Modal-mixing (CUB) and feature-descriptor-mixing (HandWritten, Scene15, LandUse, PIE, Colored-MNIST). And the previous motivation sections only shows the modal-mixing case.
The phenomenon of semantic vagueness in the feature-descriptor-mixing case should also be explained with a simple example and added into Fig. 1. 

**About the toy example:** In addition, the only visualization example in the experiment, Fig. 3, only shows the semantic vagueness challenge they try to simulate. But there is no display of the t-SNE effect after using the proposed method. The ideal effect after classification should be more distinguishable. This can reverse verify that the model indeed solves this challenge during multi-view fusion learning as they claim, but the authors did not further show it.

**In summary, there are still some concerns in this paper for me. If the authors can perfectly solve my confusion in code and other aspects of this paper, I am willing to change my score.**