Trusted Multi-View Classification
Authors: Zongbo Han
Changqing Zhang
Huazhu Fu
Joey Tianyi Zhou
Venue: ICLR 2021 Poster (2021)
Keywords: Multi-Modal Learning
Multi-View Learning
Uncertainty Machine Learning

Abstract: Multi-view classification (MVC) generally focuses on improving classification accuracy by using information from different views, typically integrating them into a unified comprehensive representation for downstream tasks. However, it is also crucial to dynamically assess the quality of a view for different samples in order to provide reliable uncertainty estimations, which indicate whether predictions can be trusted. To this end, we propose a novel multi-view classification method, termed trusted multi-view classification, which provides a new paradigm for multi-view learning by dynamically integrating different views at an evidence level. The algorithm jointly utilizes multiple views to promote both classification reliability (uncertainty estimation during testing) and robustness (out-of-distribution-awareness during training) by integrating evidence from each view. To achieve this, the Dirichlet distribution is used to model the distribution of the class probabilities, parameterized with evidence from different views and integrated with the Dempster-Shafer theory. The unified learning framework induces accurate uncertainty and accordingly endows the model with both reliability and robustness for out-of-distribution samples. Extensive experimental results validate the effectiveness of the proposed model in accuracy, reliability and robustness.

Published as a conference paper at ICLR 2021
TRUSTED MULTI-VIEW CLASSIFICATION
Zongbo Han, Changqing Zhang,
Huazhu Fu
College of Intelligence and Computing
Inception Institute of Artificial Intelligence Abu Dhabi, UAE hzfu@ieee.org
Tianjin University
Tianjin, China
{zongbo, baangbo,zhangginn ju en
Joey Tianyi Zhou
Institute of High Performance Computing A Singapore
joey. tianyi. zhou@ zhou@gmail com
ABSTRACT
Multi-view classification (MVC) generally focuses on improving classification accuracy by using information from different views. typically integrating them into unified comprehensive representation for downstream tasks. However, it is also crucial to dynamically assess the quality of a view for different samples in order provide reliable uncertainty estimations, which indicate whether predictions can be trusted. To this end, we propose a novel multi-view classification method, termed trusted multi-view classification, which provides new paradigm for multi-view learning by dynamically integrating different views at an evidence level. The algorithm jointly utilizes multiple views to promote both classification reliability and robustness by integrating evidence from each view. To achieve this, the Dirichlet distribution is used to model the distribution of the class probabilities, parameterized with evidence from different views and integrated with the Dempster- Shafer theory. The unified learning framework induces accurate uncertainty and accordingly endows the model with both reliability and robustness for out-of- distribution samples. Extensive experimental results validate the effectiveness of the proposed model in accuracy, reliability and robustness.
INTRODUCTION
Multi-view data, typically associated with multiple modalities or multiple types of features, often exists in real-world scenarios. State-of-the-art multi-view learning methods achieve tremendous success across wide range of real-world applications. However, this success typically relies on complex models (Wang et al. 2015a; Tian et al.) 2019; Bachman et al. 2019 Zhang et al., 2019 Hassani & Khasahmadi] 2020), which tend to integrate multi-view information with deep neural networks. Although these models can provide accurate classification results, they are usually vulnerable to yield unreliable predictions, particularly when presented with views that are not well-represented (e.g., information from abnormal sensors). Consequently, their deployment in safety-critical applications (e.g., computer-aidee diagnosis autonomous driving) is limited. This has inspired us to introduce new paradigm for multi-view classification to produce trusted decisions.
For multi-view learning, traditional algorithms generally assume an equal value for different views or assign/learn fixed weight for each view. The underlying assumption is that the qualities or importance of these views are basically stable for all samples. In practice, the quality of view often varies for different samples which the designed models should be aware of for adaption. For example, multi-modal medical diagnosis (Perrin et al. 2009; Sui et al.] (018). magnetic resonance (MR) image may be sufficient for one subject, while positron emission tomography (PET) image may be required for another. Therefore, the decision should be well explained according to multi-view inputs. Typically, we not only need to know the classification result, but should also be able to answer
"Coreepooning author: Changqing Zhang

---

Published as a conference paper at ICLR 2021
"How confident is the decision?" and "Why is the confidence so high/low for the decision?". To this end. the model should provide in accurate uncertainty for the prediction of each sample, and even individual view of each sample.
Uncertainty-basee -base algorithms can be roughly divided into two main categories, i.e., Bayesian and non-Bayesian approaches. Traditional Bayesian approaches estimate uncertainty by inferring
posterior distribution over the parameters (MacKay 1992a; Bernardo & Smith 2009] Neal 2012))
A variety of Bayesian methods have been developed, including Laplace approximation (MacKay 1992b), Markoy Chain Monte Carlo (MCMC) (Neal 2012) and variational techniques (Graves) 1
Ranganath et al.) 2014; Blundell et al. 2015). However, compared with ordinary neural networks, due to the doubling of model parameters and difficulty in convergence, these methods are computationally expensive. Recent algorithm (Gal & Ghahramani] 2016) estimates the uncertainty by introducing dropout (Srivastava et al. 2014) in the testing phase, thereby reducing the computational cost. Several non-Bayesian algorithms have been proposed, including deep ensemble (Laakshiinarayanan et al., 2017), evidential deep learning (Sensoy et al. 2018) and deterministic uncertainty estimate (van Amerstoort et al., 2020). Unfortunately, all of these methods focus on estimating the uncertainty on single-veew data, despite the fact that fusing multiple views through uncertainty can improve performance and reliability.
In this paper, we propose new multi view classification algorithm aiming elegantly integrate multi- view information for trusted decision making (shown in Fig. [[a)). Our model combines different views at evidence level instead of feature output level done previously, which produces stable and reasonable uncertainty estimation and thus promotes both classification reliability and robustness. The Dirichlet distribution used to model the distribution of the class probabilities, parameterized with evidence from different views and integrated with the Dempster-Shafee theory. In summary, the specific contributions of this paper are:
(1) We propose novel multi-view classification model aiming to provide trusted and inter- pretable (according to the uncertainty of each view) decisions in an effective and efficient way (without any additional computations and neural network changes), which introduces a new paradigm in multi-view classification.
(2) The proposed model is unified framework for promising sample-adaptive multi-view integration, which integrates multi-view information at an evidence level with the Dempster- Shafer theory in an optimizable (learnable) way.
(3) The uncertainty of each view is accurately estimated, enabling our model to improve classification reliability and robustness.
(4) We conduct extensive experiments which validate the superior accuracy, robustness, and reliability of our model thanks the promising uncertainty estimation and multi-view integration strategy.
RELATED WORK
Lincertainty-basee Learning. Deep neural networks have achieved great success in various tasks.
However since most deep models are essentially deterministic functions, the uncertainty of the model cannot be obtained. Bayesian neural networks (BNNs) (Denker & LeCun 1991] MacKay] 19926]
Neal 2012) endow deep models with uncertainty by replacing the deterministic weight parameters with distributions. Since BNNs struggle in performing inference and usually come with prohibitive computational costs, a more scalable and practical approach, MC-dropout (Gal & Ghahramani 2016))
was proposed. In this model, the inference is completed by performing dropout sampling from the weight during training and testing. Ensemble based methods (Lakshminaray and integrate multiple deep networks and also achieve promising performance. Instead of indirectly modeling uncertainty through network weights, the algorithm (Sensoy et al., 2018)) introduces the subjective logic theory to directly model uncertainty without ensemble or Monte Carlo sampling. Building upon RBF networks, the distance between test samples and prototypes can be used as the
et al.) (017)) train
agency for deterministic uncertainty (van Amersfoort et al. 2020). Benefiting from the learned
weights of different tasks with homoscedastic uncertainty learning, (Kendall et al. 2018) achieves impressive performance in multi-task learning.
Multi- View Learning. Learning on data with multiple views has proven effective in a variety of tasks. CCA-based multi-view models (Hotelling. 1992 Akaho 2006 Wang 2007 Andrew et al.
2

---

Published as a conference paper at ICLR 2021
Dirichlat Distribution aj)
b
b2
D4
b
628
View-
= [e), cl."",, o8]
Evidence
Uncertainty
(bi, ub)")
sps Bunequoo
55
65
Uncertainty = (b, by. (p.st
Evidence [c].e)..,.
Uncertainty
View
M (b.
Dirichlet Distribution
u²
a" [ot..r.....n
Overview of the trusted multi-view classification
(b) Combining belief's
Figure 1: Illustration of our algorithm. (a) The evidence of each view is obtained using neural networks ()). The obtained evidence parameterizes the Dirichlet distribution (2) induce the classification probability and uncertainty (Q). The overall uncertainty and classification probability are inferred by combining the beliefs of multiple views based on the DST (4). The combination rule and an example are shown in Definition 4 and (b), respectively. Given two sets of beliefs (blue
and green blocks), we recombine the compatible parts of the two sets (brown blocks) and ignore the mutually exclusive parts (white blocks) of the two sets obtain the combined beliefs.
2013; Wang al., 2013a 2016) are representative ones that have been widely used in multi-view representation learning. These models essentially seek a common representation by maximizing the correlation between different views. Considering common and exclusive information, hierarchical multi-modal metric learning (HM3L) (Zhang et al. (017)) explicitly learns shared multi-vien and view-specific metrics, while AE Zhang et al. 2019) implicitly learn a complete (view-specific and shared multi-view) representation for classification. Recently, the methods (Tian et al. 2019 Bachman et al., 2019 Chen et al.) 2020: Hassani & Khasahmadi] (0200) based on contrastive learning have also achieved good performance. Due to its effectiveness, multi-view learning has been widely
used in various applications (Kiela et al. 2018 Bian et al. 2017 Kiela et al. 2019 2020).
Wang et al.
Dempster-Shafer Evidence Theory (DST). DST, which is a theory on belief functions, was first proposed by Dempster (Dempster 1967) and is generalization of the Bayesian theory subjective probabilities (Dempster 1968). Later, it was developed into a general framework to model epis- temic uncertainty (Shafer 1976). In contrast Bayesian neural networks, which indirectly obtain uncertainty through multiple stochastic samplings from weight parameters, DST directly models uncertainty. DST allows beliefs from different sources to be combined with various fusion operators to obtain a new belief that considers all available evidence (Sentz et al. 2002) Jøsang & Hankin 2012). When faced with beliefs from different sources, Dempster's rule combination tries to fuse their shared parts, and ignores conflicting beliefs through normalization factors. A more specific implementation will be discussed later.
TRUSTED MULTI-VIEN CLASSIFICATION
has been shown that using softmax output as confidence for predictions often leads high confidence values, even for erroneous predictions since the largest softmax output is used for the final prediction (Moon al. 2020 van Amersfoort al., 2020). Therefore, we introduce an evidence- based uncertainty estimation technique which can provide more accurate uncertainty and allow us to flexibly integrate multiple views for trusted decision making.
3.1 UNCERTAINTY AND THE THEORY OF EVIDENCE
In this subsection, we elaborate on evidential deep learning to quantify the classification uncertainty for each of multiple views, which simultaneously models the probability of each class and overall uncertainty of the current prediction. In the context of multi-class classification, Subjective logic (SL) (Jøsang 2018) associates the parameters of the Dirichlet distribution (Definition A. in the Appendix)
3

---

Published as a conference paper at ICLR 2021
M
b3
b
b2
p
(a) Confident prediction
(b) Out of distribution
(c) High uncertainty
(d) Opinion (M)
Figure 2: Typical examples of Dirichlet distribution. Refer to the text for details.
with the belief distribution, where the Dirichlet distribution can be considered as the conjugate prior of the categorical distribution (Bishop) 2006).
Accordingly, we need to determine the concentration parameters, which are closely related to the uncertainty. We elaborate on the Subjective logic (Jøsang) 2018), which defines a theoretical framework for obtaining the probabilities (belief masses) different classes and overall uncertainty (uncertainty mass) of the multi-classification problem based on the evidence collected from data. Note that evidence refers to the metrics collected from the input to support the classification (step I in Fig. [(a)) and is closely related to the concentration parameters of Dirichlet distribution. Specifically, for the K classification problems, subjective logic tries to assign belief mass to each class label and overall uncertainty mass the whole frame based the evidence. Accordingly, for the wh view, the K + mass values are all non-negative and their sum is one:
K
1,
where u" 2 0 and og 2 indicate the overall uncertainty and the probability for the hth class, respectively.
For the wh view, subjective logic connects the evidence e" Dirichlet distribution ax [a", a* (step ② in Fig. the Dirichlet distribution is induced from ex. i.e. ag 6 Then, the belief mass by and the
[ey.
e to the parameters of the Specifically, the parameter oj of
((a)
uncertainty u" (step 3 in Fig. [(a)) are computed as
K S
and
"
5t
S
where S" = EK 11 a. is the Dirichlet strength. Eq. actually describes the phenomenon where the more evidence observed for the btth category, the greater the probability assigned to the 1tth class. Correspondingly, the less total evidence observed, the greater the total uncertainty. The belief assignment can be considered as a subjective opinion. Given an opinion, the mean of the corresponding Dirichlet distribution p" for the class probability P is computed as
P
(Frigyik et al. 2010).
5
Differences from traditional deep-neurrll-newwork classifiers. Firstly, the output of traditional neural network classifiers can be considered as point on simplex, while Dirichlet distribution a simplex. Therefore, with the Dirichlet distribution, SL models the second- -oderrr-rerrrreoo probability and uncertainty of the output. Secondly, the softmax function is widely used in the last layer of traditional neural network classifiers. However, using the softmax output as the confidence often leads to over-confidence. In our model, the introduced SI. can avoid this problem by adding overall uncertainty mass. Existing methods (Gal Ghahramani 2016 Lakshminarayanan et al. 2017) usually require additional computations during inference to output uncertainty. Since the uncertainty is obtained during the inference stage, it is difficult to seamlessly train model with high accuracy, robustness and reasonable uncertainty in a unified framework. Accordingly, the limitations underlying existing algorithms (e.g., inability to directly obtain uncertainty) also limits their extension trusted multi-view classification.
parametrizes the density of each such probability assignment
For clarity, we provide typical examples under a triple classification task to illustrate the above formulation. Let us assume that (40, and accordingly we have (41,2, 2). The
4

---

Published as a conference paper at ICLR 2021
corresponding Dirichlet distribution, shown in Fig. a). yields sharp distribution centered the top of the standard 2-simplex. This indicates that sufficient evidence has been observed to ensure accurate classification. In contrast, let assume that we have the evidence which is little evidence for classification. Accordingly, we obtain the Dirichlet distribution parameter (1.0001, 1.0001, 1.0001) and the uncertainty mass 1. As shown in Fig. ( in this case, the evidence induces quite a flat distribution over the simplex. Finally, when e (5, 5, 5). there also a high uncertainty, as shown in Fig. _(c), even though the overall uncertainty is reduced compared to the second case. As shown in Fig. d)), we can convert a Dirichlet distribution into standard 3-simplex (a regular tetrahedron with vertices (1.0,0,0), (0,1,0,0), (0,0.1.0) and (0,0,0.1) in Rt) based on the subjective logic theory (Eq. and 2), where the point (M) in the simplex corresponding to (be ) ] indicates an opinion. Accordingly, the expectation value p of the Dirichlet distribution is the projection of M on the bottom.
(0.0001, 0.0001, 0.0001),
3.2 DEMPSTER'S S RULE OF COMBINATION FOR MULTI- VIEW CLASSIFICATION
Having introduced evidence and uncertainty for the single-view case, we now focus on their adaptation to data with multiple views. The Dempster- Shafer theory of evidence allows evidence from different sources to be combined arriving at degree of belief (represented by mathematical object called the
belief function) that takes into account all the available evidence (see Definition 3.1 Specifically,
we need to combine V independent sets of probability mass assignments {M" where M"
[[6]] u") J. to obtain a joint mass M = HE u (step 4 in Fig. I(a).
Definition 3.1 (Dempster's combination rule for two independent sets of masses) The combination
(called the joint mass) M = {(by
[bb])=1> 1k-1: u} is calculated from the two sets of masses M
( u²) and M = {(0) 1= (2) in the following manner:
M M² @ M².
(3)
The more specific calculation rule can be formulated as follows:
1 - + bb.u² + bgu²), I-C u'u,
bk
(4)
6/63 is measure of the amount of conffict between the two mass sets (the white where C C= -iigi
blocks in Fig. 1(b)), and the scale factor T-c is used for normalization.
The joint opinion M is formed based on the fusion of opinions M and M The joint belief mass of class l (bg) and overall uncertainty (4) correspond to the brown blocks in Fig. [(b] Intuitively, the combination rule ensures: (1) when both views are of high uncertainty (large u and w²), the final prediction must be of low confidence (small by): ): (2) when both views are of low uncertainty (small ul and u²), ), the final prediction may be of high confidence (large by);; (3) when only view is of low uncertainty (only u' or u2 is large), the final prediction only depends on the confident view.
Then, given data with V different views, we can obtain the Afterwards, we can combine the beliefs from different views with Dempster's rule of combination. Specifically, fuse the belief mass and uncertainty mass between different views with the following rule:
mass for each view.
M = M + M @ ..
(5)
After obtaining the joint mass M {(bx) u). according to Eq. 2 the corresponding joint evidence from multiple views and the parameters of the Dirichlet distribution are induced as
K S Cb by and og = e 1.
u
Based on the above combination rule, we can obtain the estimated multi -iew joint evidence e and the corresponding parameters of joint Dirichlet distribution a produce the final probability of each class and the overall uncertainty.
Advantages of using subjective logic compared with softmax. Compared with softmax output, using subjective uncertainty is more suitable for the fusion of multiple decisions. Subjective logic provides additional mass function (u) that allows the model distinguish between lack of evidence. In our model, subjective logic provides the degree of overall uncertainty of each view, which is important for trusted classification and interepretability some extent.
5

---

Published as a conference paper at ICLR 2021
3.3 LEARNING TO FORM OPINIONS
In this section, we will discuss how to train neural networks to obtain evidence for each view, which can then be used obtain the corresponding masses [M*)y= and MM. The neural networks can capture the evidence from input to induce a classification opinion (Kiela et al. 2018). and the conventional neural-network-bbie
classifier can be naturally transformed into the evidence-basee classifier with minor changes. Specifically, the softmax layer of conventional classifier replaced with an activation function layer (i.e., RELU) ensure that the network outputs non-negative values, which are considered as the evidence vector e. Accordingly, the parameters of the Dirichlet distribution can be obtained.
setrralnetworr-baee
For conventional deural-network-basse classifiers, the cross-entropy loss is usually employed:
K
Loe = Σ Mij log(pis),
j=1
where Pij is the predicted probability of the ith sample for class j. For our model, given the evidence of the ith sample obtained through the evidence network, we can get the parameter a (i.e., af c + 1) of the Dirichlet distribution and form the multinomial opinions D(p:)a;) (;p:)a;). where p; is
class assignment probabilities on a simplex. After a simple modification on Eq. [ we have the
adjusted cross-entropy loss:
K
K II j=1
K
(ac) =
Σ j=1
Biy log (pt))
c Pi
dp:
Σ j=1
- yo ( (ar)))
(8)
B (a.))
where v²()) is the digamma function. Eq. 8 is the integral of the cross-entropy loss function the simplex determined by ax The above loss function ensures that the correct label of each sample generates more evidence than other classes, however, it cannot guarantee that less evidence will be generated for incorrect labels. That is to say, in our model, we expect the evidence for incorrect labels shrink to O. To this end, the following KL. divergence term is introduced:
KL [D (p:)a,)
|| (P:
I( a.s.).
EK= (ā - 1) [et (āxk) - vb (E ass)) )s
= log
T(K) II r(āa))
where a; = y, + (1 yz) a is the adjusted parameter of the Dirichlet distribution which can avoid penalizing the evidence of the groundtruth class to 0, and I(-) is the gamma function.
Therefore, given parameter a; of the Dirichlet distribution for each sample the sample-specific loss is
((a;) = Cace + X [D (ā;) ||D (p:[1))
(10)
where A > is the balance factor. practice, we can gradually increase the value of X so as to prevent the network from paying too much attention to the KL divergence in the initial stage of training, which may result a lack of good exploration of the parameter space and cause the network to output flat uniform distribution.
To ensure that all views can simultaneously form reasonable opinions and thus improve the overall opinion, we use multi-task strategy with following overall loss function:
N
Loveralt
Σ
C(as) + c(a))
(11)
i=1
B=1
The optimization process for the proposed model summarized in Algorithm [ the Appendix).
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
In this section, we conduct experiments on six real-world datasets: Handwritten
CUB Wah et al.
2011), Caltech101 (Fei-Fei et al. 2004) PIEN 1cene (Fei-Fei & Perona 2005 and HMDB

---

Published as a conference paper at ICLR 2021
(Kuehne et al. 2011). We first compare algorithm with single-view classifiers to validate the effectiveness of our algorithm utilizing multiple views. Then, we apply existing classifiers to multi-view features and conduct experiments under different levels of noise to investigate their ability in identifying multi-view OOD samples. Details of these datasets and experimental setting can be found in the appendix.
Compared methods. We compare the proposed method with following models: (a) MCDO (monte carlo dropout) (Gal & Ghahramani 2013) casts dropout network training as approx- imate inference in Bayesian neural network: (b) DE (deep
1.00
ensemble) (Lakshminaray ynnan et al. (017)) is simple, non- Bayesian method which involves training multiple deep models; (c) UA (uncertainty-awaal attention) (Heo et al., 2018) gener- ates attention weights following a Gaussian distribution with learned mean and variance, which allows heteroscedastic uncertainty to be captured and yields a more accurate cali- bration of prediction uncertainty; (d) EDL (evidential deep Learning) (Sensoy et al. 2018)) designs a predictive distribution for classification by placing Dirichlet distribution the class probabilities.
Accuracy
Handwritten
0.85
PIE
0.80
Caltech101
Scene15
HMDB
0.0
0.2
0.4
as
0.8
1.0
Uncertainty threshold
Figure 3: Accuracy with uncer- tainty thresholding.
4.2 EXPERIMENTAL RESULTS
Comparison with imcertainty-basee algorithms using the best view. We first compare algo- rithm with current uncertainty- based classification methods. The detailed experimental results are shown in Table 1 Since most existing uncertainty-baset classification methods use single-view data, we report the results of each method with the best-performing view in terms of both accuracy and AUROC (Hand & Till (001I) to comprehensively compare our method with others. As shown in Table ! our model outperforms other methods on all datasets. Taking the results on PIE and 5cenel5 examples, our method improves the accuracy by about 7.6% and 14.8% compared the second-best models (EDL/MCDO) in terms of accuracy respectively. Although our model is clearly more effective than single-view uncertainty-brat based models, it natural further ask what happens if all algorithms utilize multiple views?
Data
Metric
MCDO
DE
UA
EDL
Ours
ACC
97.3711.880 98.3010.31
97.4511.84
97.67
98.51:10.15
Handwritten
AUROC
99.70++0.07
99.79+00.05
99.67++0.10
99.83++0.02
99.97--0.00
ACC
89.7810.52
90.1910.51
89.7511.43
89.5011.17
91.0011.422
CUB
AUROC
99.29-10.03
98.7710..3
98.6910..99
98.7110003
99.06:10.03
ACC
84.09+1..45
70.29+3.17
83.70+2..70
80.86
91.99-1.01
PIE
AUROC
98.90.10.31
95.71110.88
98.06:10.56
98.7410..17
99.6910.05
ACC
91.73+-0.58
91.60+F0.82
92.37+0072
90.84:0.56
92.93--0.20
Caltech101
AUROC
99.91+0.001
0.01
99.85+00.55
99.74+0.03
99.90++0.01
ACC
12.96.11.17
39.12-10.80
41.20.11.34
46.41
67.74.00.6
Scene1.5
AUROC
92.90+00.31
74.64+00477
85.26+0.32
91.41
95.94:10.02
ACC
$2.92.11.28
57.93
53.32.11.39
59.88.1.119
65.26
HMDB
AUROC
93.57110.28
94.0110.21
91.68.10.69
94.00.1..25
96.18
Table 1: Evaluation of the classification performance.
Comparison with uncertainty-baset algorithms using multiple views. To further validate the effectiveness of our model in integrating different various views, we concatenate the original features of multiple views for all comparison methods. We add Gaussian noise with different levels of standard deviations (o) to half of the views. The comparison results are shown in Fig. 4 As can be observed that when the data free of noise, our method can achieve competitive results. After introducing noise the data, the accuracy of all the comparison methods significantly decreases. Fortunately, benefiting from the uncertainty- based fusion, the proposed method is aware the view-specific noise and thus achieves impressive results all datasets. Therefore, the effectiveness for both clean and
7

---

Published as a conference paper at ICLR 2021
noisy multi-vies view data is well validated. However, it will be more convincing to explicitly investigate the performance uncertainty estimation.
0.9
0.8
Accuracy
0.7
0.0
Accuracy
Accuracy
0.6
0.6
EDL
EDL
EDL
0.5
0.4
0.4
DE
10t tor
10 1ot 10" 10t
161 1ot 107 10r 10 10r 15 10t 10
101 10l iot 10s 30
Standard deviation (a)
Standa rd deviation (a)
Standard deviation (e)
(a) Handwritten
(b) CUB
(c) PIE
Accuracy
0.53
Accuracy
0.6
Acouracy
0.00
0.5
EDL
EDL
EDL
LAA.
MEDO
0.4
MCDO
NEDO
0.70
Dr
DE
DE
Ours
Ours
Ours
19 lor 2ot jof 10 jor 10" Standard deviation
10t 10r 1or 10f 10 10t 19"
10t 10t 20 20 to 1or jot
Standard deviation (a)
Standard
(d) Caltech101
(c) Secne15
(f) HMDB
Figure 4: Performance comparison on multi-view data with different levels of noise.
Uncertainty estimation. To evaluate the uncertainty estimation, we visualize the distribution of fooout-of-distribttioo samples in terms of uncertainty. We consider the original samples in- distribution data, while the samples with Gaussian noise are viewed as out-of-distriibtion data. Specifically, add Gaussian noise with the fixed level of standard deviations (a 10) 50% of the test samples. The experimental results are shown in Fig. 5 According to the results. the following observations are drawn: (1) Datasets with higher classification accuracy (e.g., Handwritten) are usually associated with lower uncertainty for the in-distribution samples. (2) In contrast, datasets with lower accuracy are usually associated with higher uncertainty for the in-distribution samples. (3) Much higher uncertainties are usually estimated for out-of-distribute observations imply the reasonability our model in estimating uncertainty, since can facilitate discrimination between these classes. Fig. 3 shows that our algorithm provides much more accurate predictions as the prediction uncertainty decreases. This implies that trusted decisions are supported based on the output (classification and its corresponding uncertainty) of our model.
samples on all datasets. These
CONCLUSION
In this work, we propose novel trusted multi-view classification (TMC) algorithm which, based on the Dempster-Shafee evidence theory, can produce trusted classification decisions on multi-view data. Our algorithm focuses on decision-making by fusing the uncertainty of multiple views, which is essential for making trusted decisions. The TMCC model can accurately identify the views which are risky for decision making, and exploits informative views in the final decision. Furthermore, our model can produce the uncertainty of current decision while making the final classification, providing intepretability. The empirical results validate the effectiveness of the proposed algorithm in classification accuracy and citto--distribution identification.
ACKNOWLEDGMENTS
This work was supported in part by National Natural Science Foundation China (No. 61976151, No. 61732011) 1), and the Natural Science Foundation of Tianjin of China (No. 19ICYBJC15000.
8

---

Published as a conference paper at ICLR 2021
indiennbsuion out-of-dstbbution
Invditir bucson
out-sf-ditrrbuuoo
out-of-d stribuutttibutioo
Density
Densiry
Denslyy
Uncortainty
Uncortaintyy
Uncortainty
(a) Handwritten
(b) CUB
(c) PIE
in distrfoution
1n-dicorooua ousion
In 1n-i6trioctonn
out-oll-disttn solldditttbutiin
nut-al-diinbbttee
oull-at-d sinbuution
Deriiy
Demiiy
Demiiy
0.0
1.0
1.0
0.0
0.4
Uncertainty
Uncertainty
Uncertainty
(d) Caltech101
(c) Scene
(f) HMDB
Figure 5: Density of uncertainty.
REFERENCES
Shotaro Akaho. kernel method for canonical correlation analysis. arXiv preprint cx/0609071, 2006.
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In International Conference on Machine Learning, pp. 1247-1255, 2013.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing
mutual information across views. In Advances in Neural Information Processing Systems, pp. 15535-15544 2019.
José M Bernardo and Adrian FM Smith. Bayesian theory, volume 405. John Wiley & Sons, 2009.
Yunlong Bian, Chuang Gan, Xiao Liu, Fu Li, Xiang Long, Yandong Li, Heng Qi, Jie Zhou, Shilei Wen, and Yuanqing Lin. Revisiting the effectiveness of off-the-shed temporal modeling approaches for large-scale video classification. arXiv preprint arXiv: 1xxv:1708 1708. 038000 2017.
Christopher M Bishop. Pattern recognition and machine learning. springer, 2006.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference Machine Learning, pp. 1613-1622, 2015.
Ting Chen, Simon Kombllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
AP Dempster. Upper and lower probabilities induced by multivalued mapping. The Annals of Mathematical Statistics, pp. 325-339, 1967.
Arthur P Dempster. A generalization of bayesian inference. Journal of the Royal Statistical Society:
Series (Methodological), 30(2):205 1968.
John S Denker and Yann LeCun. Transforming neural-net output levels to probability distributions.
In Advances in Neural Information Processing Systems, pp. 853-859, 1991.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0 04805, 2018.
Li Fei-Fei and Pietro Perona. A bayesian hierarchical model for learning natural scene categories. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR' 005, volume 2, pp. 524-531. IEFE, 2005.

---

Published as a conference paper at ICLR 2021
Li Fei-Fei, Rob Fergus, and Pietro Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference Computer Viston and Pattern Recognition workshop, pp. 178-178. IEEE, 2004.
Bela A Frigyik, Amol Kapila, and Maya R Gupta. Introduction the dirichlet distribution and related processes, Department of Electrical Engineering. University of Washignton. UWEE7R-2010-000 (0006):1-27, 2010.
Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with bernoulli approxi- mate variational inference. arXiv preprint /506.02/500/23 2015.
Yarin Gal and Zoubin Ghahramani. Dropout bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, pp. 1050-1059, 2016.
Alex Graves. Practical variational inference for neural networks. In Advances Neural Information Processing Systems, pp. 2348-24-80 2356, 2011.
David Hand and Robert J Till. A simple generalisation of the area under the curve for multiple class classification problems. Machine learning, 45(2):171-186, 2001.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. arXiv preprint wwXiv:2006.05582, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 770-778, 2016.
Jay Heo, Hae Beom Lee, Saehoon Kim, Juho Lee, Kwang Joon Kim, Eunho Yang, and Sung Ju Hwang. Uncertainty-aware attention for reliable interpretation and prediction. In Advances in Neural Information Processing Systems, pp. 909-918, 2018.
Harold Hotelling. Relations between two sets of variates. In Breakthroughs in statistics, pp. 162-190. Springer, 1992.
Audun Jøsang. Subjective Logic: A formalism.for reasoning under uncertainty. Springer Publishing Company, Incorporated, 2018.
Audun Jøsang and Robin Hankin. Interpretation and fusion of hyper opinions in subjective logic. In 2012 15th International Conference on Information Fusion, pp. 1225-1232. IEEE, 2012.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7482-7491, 2018.
Douwe Kiela, Edouard Grave, Armand Joulin, and Tomas Mikolov. Efficient large-scale multi-modal 2018.
classification. arXiv preprint
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. arXiv preprint arXiv: 02950, 2019.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In 2011 International Conference on Computer Vision, pp. 2556-2563. IEEE, 2011.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
David JC MacKay. Bayesian methods for adaptive models. PhD thesis, California Institute of Technology, 1992a.
10

---

Published as a conference paper at ICLR 2021
David JC MacKay. A practical bayesian framework for backpropagation networks. Neural computa- tion, 4(3):448 472, 1992b.
Jooyoung Moon, Jihyo Kim, Younghak Shin, and Sangheum Hwang. Confidence-aware learning for deep neural networks. International Conference on Machine Learning, 2020.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Richard J Perrin, Anne M Fagan, and David M Holtzman. Multimodal techniques for diagnosis and prognosis of alzheimer's disease. Nature, 461(7266):11-922,, 2009.
Rajesh Ranganath, Sean Gerrish, and David Blei. Black box variational inference. In Artificial Intelligence and Statistics, pp. 814 822. PMLR, 2014.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In Advances in Neural Information Processing Systems, pp. 3179-3189, 2018.
Kari Sentz, Scott Ferson, et Combination of evidence in Dempster-Shafer theory, volume 4015. Citeseer, 2002.
Glenn Shafer. A mathematical theory of evidence, volume 42, Princeton university press, 1976.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In European Conference on Computer vision, pp. 746-760. Springer, 2012.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-199) 2014.
Jing Sui, Shile Qi, Theo GM van Erp, Juan Bustillo, Rongtao Jiang, Dongdong Lin, Jessica A Turner, Eswar Damaraju, Andrew R Mayer, Yue Cui, et al. Multimodal neuromarkers in schizophrenia via cognition-guidee guided mri fusion. Nature communications, 9(1):1-14, 2018.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv: 1904.00899 2019.
Joost van Amersfoort, Lewis Smith, Yee Whye Teh, and Yarin Gal. Uncertainty estimation using single deep deterministic neural network. In International Conference on Machine Learning. 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
Chong Wang. Variational bayesian approach to canonical correlation analysis. IEEE Transactions Neural Networks, 18(3):905-91) 9100,,9999,,00000 2007.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In International Conference on Machine Learning, pp. 1083-1092, 2015a.
Weiran Wang, Xinchen Yan, Honglak Lee, and Karen Livescu. Deep variational canonical correlation analysis. arXiv preprint arXiv:/610.0000 1610.03454 2016.
Weiyao Wang. Du Tran, and Matt Feiszli. What makes training multi-modal classification networks hard? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12695-12705, 2020.
Xin Wang, Devinder Kumar, Nicolas Thome, Matthieu Cord, and Frederic Precioso. Recipe recogni- tion with large multimodal food dataset. In 2015 IEEE International Conference on Multimedia Expo Workshops (ICMEW), pp. 1-6. IEEE, 201
Changqing Zhang, Zongbo Han, Huazhu Fu, Jocy Tianyi Zhou, Qinghua Hu, et al. Cpm-nets: Cross partial multi-view networks. In Advances in Neural Information Processing Systems, pp. 559-569, 2019.
Heng Zhang, Vishal M Patel, and Rama Chellappa. Hierarchical multimodal metric learning for multimodal classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3057-3065, 2017.
11


Reviews:
[{"review_id": "pSfB3nhjBpj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose using Dempster-Shafer evidence theory to build a more trusted multi-view classification method.\n\nThe article is fairly well structured, apart from the literature review, which somewhat does not discuss some of the related works properly, for example, the probabilistic CCA based methods and the optimal transport based papers [Dixin Luo, 2020].  The article has the unfortunate tendency to contains a few unproven (or wrong) claims. For instance, \"[...] accurate classification results, they are usually vulnerable to yielding incorrect predictions, particularly when presented with views that are not well-represented [...]\", which would deserve a citation or some empirical evidence. Specifically, I think some of the Bayesian versions of CCA methods are able to handle such a situation by providing uncertainty, and the authors did not include them as the baselines in the experimental results as well. It is also claimed that the proposed method is \"optimal\" sample-adaptive multi-view integration, which I could not see any proof in a theoretical manner. \n\nMy main concern with this paper is the experimental results. First of all, the paper is a kind of Bayesian multi-view learning method and should compare with Bayesian and deep CCA-based method as well. Second, the paper used a Dirichlet distribution for each view. Why HDP cannot be included as another baseline. Regarding the results, the authors emphasize that the performance has been improved by around 15%. This is not surprising as those methods are just single-view based methods. \n\nConcerns/questions/comments:\n\n1- The authors added Gaussian noise to half of the views. It is not clear to me that some views are noisy and some of them are not noisy. All of the features in one view are noisy, or all of the features are noisy. \n\n2- Dirichlet distribution is somehow more suitable for the situation that one sample can belong to multiple classes. Did the authors consider that? \n\n3- The integration methods, such as those proposed based on CCA and optimal transport (OT), try to combine different views somehow to achieve the best performance. However, the proposed method seems to investigate each view separately and then combine them. My concern is that when the results are noisy in all views, how the method performs. Is it helpful to combine them in this way? Adding some empirical experimental results would be great.\n\n4- It is not clear to me that if the authors assume different views are well aligned or not. If not, can the proposed method handle such a situation?\n\n5- The experiments if Figure 4 aim to show that the proposed method is more robust to the noisy environment. However, adding $\\sigma = 10^9$ does not mean anything, and I think the method's performance is because it is not considering the view with the noise. But in real practice, when we talk about multi-view, both views are reasonably noisy. I prefer to see such a comparison. At least, a comparison in that way is also needed. \n\n6- A comparison of the methods in semi-supervised classification would be great. \n\n7- How the results change if you remove one of the views from a multiple-view (more than 2 views) dataset. \n\n8- Last but not least, Table 1 seems to be a comparison with one-view based methods. This is not enough to show the proposed methods are good enough. \n\n\n----------- UPDATE I ----------\n\nI thank the authors for their response to my concerns/comments. It seems like I have to defend my position for suggesting a rejection of the paper. While the response of the authors has clarified some aspects, some comments have not been adequately addressed.\n\nStill, the experiments and related works are my main concern with this paper. \n\nR1: Based on this response, It seems the paper is trying to tackle the negative transferability problem. While this is an important problem, the authors need to discuss that in the related works, for example [1]. However, in a multi-view learning problem, the main issue is that each view includes some features which can help the classification goal. In other words, part of the features in all views are noisy. At least, you need to add this experiment in my point of view.\n\nR10: [2,3,4] are a few examples that used DP for classification purposes. When one of the main arguments of the paper is adding uncertainty estimation using DP, discussing other related papers are important. In some cases, you can also compare with them.  Besides that, the authors can compare with HDP similar to what they did for the CCA-based methods.\n\nIt would be great if the authors could report the classification results based on only using one view for Figure 4 as a baseline. The authors might also want to check the paper [5]. \n\n[1] Bayesian multi-domain learning for cancer subtype discovery, NeurIPS 2018.\n\n[2] Multi-Task Learning for Classification with Dirichlet Process Priors, Journal of Machine Learning Research 2007.\n\n[3] Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification, NeurIPS 2018.\n\n[4] Factorial Multi-Task Learning : A Bayesian Nonparametric Approach, ICML 2013.\n\n[5] Hierarchical Optimal Transport for Robust Multi-View Learning, NeurIPS 2020.\n\n----------- UPDATE II ----------\nI have read this paper again and went through the author's responses. While I appreciate the authors for their responses and believe there is some novelty in including uncertainty quantification, I'm still not confident with their experimental designs and literature review.\n\nBased on my understanding, this method is not a multi-view learning in a classical way. It is better to say that the proposed method is somehow an ensemble approach. Going through the model, the paper deals with each view separately and then tries to combine the classification results of each view by weighting them. There is some novelty here, where the weighting is adapting based on the uncertainty. However, in a classical multi-view, the problem is that you have different views, where features of each view are randomly noisy. So, the goal is how one can combine information from different views to improve the classification results. Still, I cannot see that. \n\nIn the experiments, the authors make N views to be noisy, and the other N(+1) views are clean. Then they are trying to classify each view (somehow) separately and because they are reducing the weight of noisy views, their performance (slightly) will be improved. For example, please check table 1. For instance, the Handwritten dataset, the performance of DE, the only ensemble model as the baselines, performed almost similar 99.79 vs 99.97 ( 98.30 vs 98.51). This improvement can be due to tuning the other methods, and the authors did discuss this in neither the main text nor the supplement.\n\nIn Figure 4, when the noise is small, the different models' performances are almost the same. Also, \\sigma = 10^9 is not meaningful and with some sort of variance comparisons, one can find that that view is corrupted.\n\nThe authors did not address if the method can work in real/harder situations when all features and views are noisy. The authors responded they do not have any restriction on that; however, they did not show either empirically or theoretically that the proposed method can handle this situation more suitable. I agree that the authors have tried on the real dataset, however, to see the performance comparison, you somehow need to randomly corrupt different features on different views.\n\nSince the method does seem to be a multi-view ensemble learning, I would have rather expected to try a more common approach for achieving the same goal first, for example to use one of the available Bayesian single-view methods for each view and they combine the results and see how the results look like.\n\nThe authors claim that the negative transfer effect is not related here, which I can't entirely agree with. This term has been used in multi-task learning as well. The reason that no one uses this term in multi-view learning is that this is not a real scenario in multi-view learning. One view is degrading the classification performance because some views are noisy. If we do not use those very noisy views, the performance should be improved.\n\nI also want to point out that their authors show that removing each view can degrade their model's performance. And this is desire, especially when the number of views is small, as having three views means three models in their ensemble architecture. It is not clear that this improvement is due to better information sharing or more complex methods.\n\nRegarding the CCA-based method, one can use (Optimal) Bayesian Classification on learned space to get the uncertainty as well. Although, I agree that the proposed method is somehow an end-to-end learning method. \n\nI would suggest the authors somehow re-organize/write the paper as a multi-view ensemble learning method and compare it with those multi-view ensemble methods as the main focus of the paper. You also need to discuss [5] as it is very related to your work. Although, it is a Bayesian but not an ensemble method. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "dSAJJLl9Zaj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper has proposed a novel trust-based multi-view classifier. The idea of transferring classification output to the parameter of Dirichlet distribution to give uncertainty in output is novel and interesting. The model has interestingly used Dirichlet’s strength to define the weight of a view. \n \nPaper is well written and mathematically sound. To my knowledge using Dirichlet distribution for bringing uncertainty in output in multi-view multiclass classification is new.\n\nThe proposed method has been also compared with recent existing works in the task of a single view and multiple view classification. \n\nThe proposed model has been motivated as classification with uncertainty in multi-view case. But in the Experiment section, it shows that even for single view setup the method is outperforming the existing models. It will be good to discuss the reason behind that. Why it is better than other methods in a single view case too?\n\nDetails of the experimental setup are important for the reproducibility of experimental results. It would have been better to have in the main paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JCHdIdUuUQ9", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a reliable multi-view classification mechanism equipped with uncertainty, called Trusted Multi-View Classification. The goal is to dynamically assess the quality of different views for different samples to provide reliable uncertainty estimation. The idea is clear and well-motivated. The authors perform empirical studies on diverse datasets to conclude that the proposed algorithm is effective, robust and reliable. \n\nStrengths: \n+ It is interesting to conduct multi-view classification by dynamically integrating different views at an evidence level, which provides a novel and flexible way in multi-view classification.\n+ The way of using Dempster-Shafer theory for integrating evidences in a unified and learnable framework is quite neat. \n+ The paper is well-written and clearly presented. \n+ Strong and sufficient empirical results are provided.\n\nMinor comments: \n+ It is reasonable to use the subjective logic theory to directly model uncertainty, however, beyond the advantages mentioned, it will be better if intuitive comparison or discussion between using u (overall uncertainty) and softmax scores in multi-view learning could be provided. \n+ The authors could present failure cases which may be associated with high uncertainties (ideally).\n+ For the results (Table 2) of the end-to-end experiments, are the data used original or being corrupted manually? \n\n\nOverall, the paper is very well motivated and easy to follow. The assumptions and decisions are well supported. The stepwise experiments are helpful and provide good insights to evaluate the proposed algorithm. The method seems to be of great potential in real-world (cost-sensitive) applications. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new idea for multi-view classification: using a Dirichlet distribution over the views to model uncertainty.

The paper appears to be clear, well written and sound.
Also, the experimental comparison is thorough.

The authors have given pertinent responses to the reviewers' questions, including w.r.t comparing against Bayesian/deep CCA in terms of accuracy.

Overall, this is a good paper.

Decision: Accept (Poster)