Trusted Multi-View Classification

1 INTRODUCTION . Multi-view data , typically associated with multiple modalities or multiple types of features , often exists in real-world scenarios . State-of-the-art multi-view learning methods achieve tremendous success across a wide range of real-world applications . However , this success typically relies on complex models ( Wang et al. , 2015a ; Tian et al. , 2019 ; Bachman et al. , 2019 ; Zhang et al. , 2019 ; Hassani & Khasahmadi , 2020 ) , which tend to integrate multi-view information with deep neural networks . Although these models can provide accurate classification results , they are usually vulnerable to yield unreliable predictions , particularly when presented with views that are not well-represented ( e.g. , information from abnormal sensors ) . Consequently , their deployment in safety-critical applications ( e.g. , computer-aided diagnosis or autonomous driving ) is limited . This has inspired us to introduce a new paradigm for multi-view classification to produce trusted decisions . For multi-view learning , traditional algorithms generally assume an equal value for different views or assign/learn a fixed weight for each view . The underlying assumption is that the qualities or importance of these views are basically stable for all samples . In practice , the quality of a view often varies for different samples which the designed models should be aware of for adaption . For example , in multi-modal medical diagnosis ( Perrin et al. , 2009 ; Sui et al. , 2018 ) , a magnetic resonance ( MR ) image may be sufficient for one subject , while a positron emission tomography ( PET ) image may be required for another . Therefore , the decision should be well explained according to multi-view inputs . Typically , we not only need to know the classification result , but should also be able to answer ∗Corresponding author : Changqing Zhang “ How confident is the decision ? ” and “ Why is the confidence so high/low for the decision ? ” . To this end , the model should provide in accurate uncertainty for the prediction of each sample , and even individual view of each sample . Uncertainty-based algorithms can be roughly divided into two main categories , i.e. , Bayesian and non-Bayesian approaches . Traditional Bayesian approaches estimate uncertainty by inferring a posterior distribution over the parameters ( MacKay , 1992a ; Bernardo & Smith , 2009 ; Neal , 2012 ) . A variety of Bayesian methods have been developed , including Laplace approximation ( MacKay , 1992b ) , Markov Chain Monte Carlo ( MCMC ) ( Neal , 2012 ) and variational techniques ( Graves , 2011 ; Ranganath et al. , 2014 ; Blundell et al. , 2015 ) . However , compared with ordinary neural networks , due to the doubling of model parameters and difficulty in convergence , these methods are computationally expensive . Recent algorithm ( Gal & Ghahramani , 2016 ) estimates the uncertainty by introducing dropout ( Srivastava et al. , 2014 ) in the testing phase , thereby reducing the computational cost . Several non-Bayesian algorithms have been proposed , including deep ensemble ( Lakshminarayanan et al. , 2017 ) , evidential deep learning ( Sensoy et al. , 2018 ) and deterministic uncertainty estimate ( van Amersfoort et al. , 2020 ) . Unfortunately , all of these methods focus on estimating the uncertainty on single-view data , despite the fact that fusing multiple views through uncertainty can improve performance and reliability . In this paper , we propose a new multi-view classification algorithm aiming to elegantly integrate multiview information for trusted decision making ( shown in Fig . 1 ( a ) ) . Our model combines different views at an evidence level instead of feature or output level as done previously , which produces a stable and reasonable uncertainty estimation and thus promotes both classification reliability and robustness . The Dirichlet distribution is used to model the distribution of the class probabilities , parameterized with evidence from different views and integrated with the Dempster-Shafer theory . In summary , the specific contributions of this paper are : ( 1 ) We propose a novel multi-view classification model aiming to provide trusted and interpretable ( according to the uncertainty of each view ) decisions in an effective and efficient way ( without any additional computations and neural network changes ) , which introduces a new paradigm in multi-view classification . ( 2 ) The proposed model is a unified framework for promising sample-adaptive multi-view integration , which integrates multi-view information at an evidence level with the DempsterShafer theory in an optimizable ( learnable ) way . ( 3 ) The uncertainty of each view is accurately estimated , enabling our model to improve classification reliability and robustness . ( 4 ) We conduct extensive experiments which validate the superior accuracy , robustness , and reliability of our model thanks to the promising uncertainty estimation and multi-view integration strategy . 2 RELATED WORK . Uncertainty-based Learning . Deep neural networks have achieved great success in various tasks . However since most deep models are essentially deterministic functions , the uncertainty of the model can not be obtained . Bayesian neural networks ( BNNs ) ( Denker & LeCun , 1991 ; MacKay , 1992b ; Neal , 2012 ) endow deep models with uncertainty by replacing the deterministic weight parameters with distributions . Since BNNs struggle in performing inference and usually come with prohibitive computational costs , a more scalable and practical approach , MC-dropout ( Gal & Ghahramani , 2016 ) , was proposed . In this model , the inference is completed by performing dropout sampling from the weight during training and testing . Ensemble based methods ( Lakshminarayanan et al. , 2017 ) train and integrate multiple deep networks and also achieve promising performance . Instead of indirectly modeling uncertainty through network weights , the algorithm ( Sensoy et al. , 2018 ) introduces the subjective logic theory to directly model uncertainty without ensemble or Monte Carlo sampling . Building upon RBF networks , the distance between test samples and prototypes can be used as the agency for deterministic uncertainty ( van Amersfoort et al. , 2020 ) . Benefiting from the learned weights of different tasks with homoscedastic uncertainty learning , ( Kendall et al. , 2018 ) achieves impressive performance in multi-task learning . Multi-View Learning . Learning on data with multiple views has proven effective in a variety of tasks . CCA-based multi-view models ( Hotelling , 1992 ; Akaho , 2006 ; Wang , 2007 ; Andrew et al. , networks ( 1 ) . The obtained evidence parameterizes the Dirichlet distribution ( 2 ) to induce the classification probability and uncertainty ( 3 ) . The overall uncertainty and classification probability are inferred by combining the beliefs of multiple views based on the DST ( 4 ) . The combination rule and an example are shown in Definition 4 and ( b ) , respectively . Given two sets of beliefs ( blue and green blocks ) , we recombine the compatible parts of the two sets ( brown blocks ) and ignore the mutually exclusive parts ( white blocks ) of the two sets to obtain the combined beliefs . 2013 ; Wang et al. , 2015a ; 2016 ) are representative ones that have been widely used in multi-view representation learning . These models essentially seek a common representation by maximizing the correlation between different views . Considering common and exclusive information , hierarchical multi-modal metric learning ( HM3L ) ( Zhang et al. , 2017 ) explicitly learns shared multi-view and view-specific metrics , while AE2-Nets ( Zhang et al. , 2019 ) implicitly learn a complete ( view-specific and shared multi-view ) representation for classification . Recently , the methods ( Tian et al. , 2019 ; Bachman et al. , 2019 ; Chen et al. , 2020 ; Hassani & Khasahmadi , 2020 ) based on contrastive learning have also achieved good performance . Due to its effectiveness , multi-view learning has been widely used in various applications ( Kiela et al. , 2018 ; Bian et al. , 2017 ; Kiela et al. , 2019 ; Wang et al. , 2020 ) . Dempster-Shafer Evidence Theory ( DST ) . DST , which is a theory on belief functions , was first proposed by Dempster ( Dempster , 1967 ) and is a generalization of the Bayesian theory to subjective probabilities ( Dempster , 1968 ) . Later , it was developed into a general framework to model epistemic uncertainty ( Shafer , 1976 ) . In contrast to Bayesian neural networks , which indirectly obtain uncertainty through multiple stochastic samplings from weight parameters , DST directly models uncertainty . DST allows beliefs from different sources to be combined with various fusion operators to obtain a new belief that considers all available evidence ( Sentz et al. , 2002 ; Jøsang & Hankin , 2012 ) . When faced with beliefs from different sources , Dempster ’ s rule of combination tries to fuse their shared parts , and ignores conflicting beliefs through normalization factors . A more specific implementation will be discussed later . 3 TRUSTED MULTI-VIEW CLASSIFICATION . It has been shown that using a softmax output as confidence for predictions often leads to high confidence values , even for erroneous predictions since the largest softmax output is used for the final prediction ( Moon et al. , 2020 ; van Amersfoort et al. , 2020 ) . Therefore , we introduce an evidencebased uncertainty estimation technique which can provide more accurate uncertainty and allow us to flexibly integrate multiple views for trusted decision making . 3.1 UNCERTAINTY AND THE THEORY OF EVIDENCE . In this subsection , we elaborate on evidential deep learning to quantify the classification uncertainty for each of multiple views , which simultaneously models the probability of each class and overall uncertainty of the current prediction . In the context of multi-class classification , Subjective logic ( SL ) ( Jøsang , 2018 ) associates the parameters of the Dirichlet distribution ( Definition A.1 in the Appendix ) with the belief distribution , where the Dirichlet distribution can be considered as the conjugate prior of the categorical distribution ( Bishop , 2006 ) . Accordingly , we need to determine the concentration parameters , which are closely related to the uncertainty . We elaborate on the Subjective logic ( Jøsang , 2018 ) , which defines a theoretical framework for obtaining the probabilities ( belief masses ) of different classes and overall uncertainty ( uncertainty mass ) of the multi-classification problem based on the evidence collected from data . Note that evidence refers to the metrics collected from the input to support the classification ( step 1 in Fig . 1 ( a ) ) and is closely related to the concentration parameters of Dirichlet distribution . Specifically , for the K classification problems , subjective logic tries to assign a belief mass to each class label and an overall uncertainty mass to the whole frame based on the evidence . Accordingly , for the vth view , the K + 1 mass values are all non-negative and their sum is one : uv + K∑ k=1 bvk = 1 , ( 1 ) where uv ≥ 0 and bvk ≥ 0 indicate the overall uncertainty and the probability for the kth class , respectively . For the vth view , subjective logic connects the evidence ev = [ ev1 , · · · , evK ] to the parameters of the Dirichlet distribution αv = [ αv1 , · · · , αvK ] ( step 2 in Fig . 1 ( a ) ) . Specifically , the parameter αvk of the Dirichlet distribution is induced from evk , i.e. , α v k = e v k + 1 . Then , the belief mass b v k and the uncertainty uv ( step 3 in Fig . 1 ( a ) ) are computed as bvk = evk Sv = αvk − 1 Sv and uv = K Sv , ( 2 ) where Sv = ∑K i=1 ( e v i + 1 ) = ∑K i=1 α v i is the Dirichlet strength . Eq . 2 actually describes the phenomenon where the more evidence observed for the kth category , the greater the probability assigned to the kth class . Correspondingly , the less total evidence observed , the greater the total uncertainty . The belief assignment can be considered as a subjective opinion . Given an opinion , the mean of the corresponding Dirichlet distribution p̂v for the class probability p̂vk is computed as p̂vk = αvk Sv ( Frigyik et al. , 2010 ) . Differences from traditional deep-neural-network classifiers . Firstly , the output of traditional neural network classifiers can be considered as a point on a simplex , while Dirichlet distribution parametrizes the density of each such probability assignment on a simplex . Therefore , with the Dirichlet distribution , SL models the second-order probability and uncertainty of the output . Secondly , the softmax function is widely used in the last layer of traditional neural network classifiers . However , using the softmax output as the confidence often leads to over-confidence . In our model , the introduced SL can avoid this problem by adding overall uncertainty mass . Existing methods ( Gal & Ghahramani , 2016 ; Lakshminarayanan et al. , 2017 ) usually require additional computations during inference to output uncertainty . Since the uncertainty is obtained during the inference stage , it is difficult to seamlessly train a model with high accuracy , robustness and reasonable uncertainty in a unified framework . Accordingly , the limitations underlying existing algorithms ( e.g. , inability to directly obtain uncertainty ) also limits their extension to trusted multi-view classification . For clarity , we provide typical examples under a triple classification task to illustrate the above formulation . Let us assume that e = 〈40 , 1 , 1〉 and accordingly we have α = 〈41 , 2 , 2〉 . The corresponding Dirichlet distribution , shown in Fig . 2 ( a ) , yields a sharp distribution centered on the top of the standard 2-simplex . This indicates that sufficient evidence has been observed to ensure accurate classification . In contrast , let us assume that we have the evidence e = 〈0.0001 , 0.0001 , 0.0001〉 , which is little evidence for classification . Accordingly , we obtain the Dirichlet distribution parameter α = 〈1.0001 , 1.0001 , 1.0001〉 and the uncertainty mass u ≈ 1 . As shown in Fig . 2 ( b ) , in this case , the evidence induces quite a flat distribution over the simplex . Finally , when e = 〈5 , 5 , 5〉 , there is also a high uncertainty , as shown in Fig . 2 ( c ) , even though the overall uncertainty is reduced compared to the second case . As shown in Fig . 2 ( d ) , we can convert a Dirichlet distribution into a standard 3-simplex ( a regular tetrahedron with vertices ( 1,0,0,0 ) , ( 0,1,0,0 ) , ( 0,0,1,0 ) and ( 0,0,0,1 ) in R4 ) based on the subjective logic theory ( Eq . 1 and Eq . 2 ) , where the point ( M ) in the simplex corresponding to { { bk } 3k=1 , u } indicates an opinion . Accordingly , the expectation value p̂ of the Dirichlet distribution is the projection ofM on the bottom .

Summary: This paper proposes a reliable multi-view classification mechanism equipped with uncertainty, called Trusted Multi-View Classification. The goal is to dynamically assess the quality of different views for different samples to provide reliable uncertainty estimation. The idea is clear and well-motivated. The authors perform empirical studies on diverse datasets to conclude that the proposed algorithm is effective, robust and reliable.