Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper has proposed a novel trust-based multi-view classifier. The idea of transferring classification output to the parameter of Dirichlet distribution to give uncertainty in output is novel and interesting. The model has interestingly used Dirichlet’s strength to define the weight of a view. 
 
Paper is well written and mathematically sound. To my knowledge using Dirichlet distribution for bringing uncertainty in output in multi-view multiclass classification is new.

The proposed method has been also compared with recent existing works in the task of a single view and multiple view classification. 

The proposed model has been motivated as classification with uncertainty in multi-view case. But in the Experiment section, it shows that even for single view setup the method is outperforming the existing models. It will be good to discuss the reason behind that. Why it is better than other methods in a single view case too?

Details of the experimental setup are important for the reproducibility of experimental results. It would have been better to have in the main paper.

review_title: Interesting use of Dirichlet distribution for bringing uncertainty in output in multi-view multiclass classification

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose using Dempster-Shafer evidence theory to build a more trusted multi-view classification method.

The article is fairly well structured, apart from the literature review, which somewhat does not discuss some of the related works properly, for example, the probabilistic CCA based methods and the optimal transport based papers [Dixin Luo, 2020].  The article has the unfortunate tendency to contains a few unproven (or wrong) claims. For instance, "[...] accurate classification results, they are usually vulnerable to yielding incorrect predictions, particularly when presented with views that are not well-represented [...]", which would deserve a citation or some empirical evidence. Specifically, I think some of the Bayesian versions of CCA methods are able to handle such a situation by providing uncertainty, and the authors did not include them as the baselines in the experimental results as well. It is also claimed that the proposed method is "optimal" sample-adaptive multi-view integration, which I could not see any proof in a theoretical manner. 

My main concern with this paper is the experimental results. First of all, the paper is a kind of Bayesian multi-view learning method and should compare with Bayesian and deep CCA-based method as well. Second, the paper used a Dirichlet distribution for each view. Why HDP cannot be included as another baseline. Regarding the results, the authors emphasize that the performance has been improved by around 15%. This is not surprising as those methods are just single-view based methods. 

Concerns/questions/comments:

1- The authors added Gaussian noise to half of the views. It is not clear to me that some views are noisy and some of them are not noisy. All of the features in one view are noisy, or all of the features are noisy. 

2- Dirichlet distribution is somehow more suitable for the situation that one sample can belong to multiple classes. Did the authors consider that? 

3- The integration methods, such as those proposed based on CCA and optimal transport (OT), try to combine different views somehow to achieve the best performance. However, the proposed method seems to investigate each view separately and then combine them. My concern is that when the results are noisy in all views, how the method performs. Is it helpful to combine them in this way? Adding some empirical experimental results would be great.

4- It is not clear to me that if the authors assume different views are well aligned or not. If not, can the proposed method handle such a situation?

5- The experiments if Figure 4 aim to show that the proposed method is more robust to the noisy environment. However, adding $\sigma = 10^9$ does not mean anything, and I think the method's performance is because it is not considering the view with the noise. But in real practice, when we talk about multi-view, both views are reasonably noisy. I prefer to see such a comparison. At least, a comparison in that way is also needed. 

6- A comparison of the methods in semi-supervised classification would be great. 

7- How the results change if you remove one of the views from a multiple-view (more than 2 views) dataset. 

8- Last but not least, Table 1 seems to be a comparison with one-view based methods. This is not enough to show the proposed methods are good enough. 


----------- UPDATE I ----------

I thank the authors for their response to my concerns/comments. It seems like I have to defend my position for suggesting a rejection of the paper. While the response of the authors has clarified some aspects, some comments have not been adequately addressed.

Still, the experiments and related works are my main concern with this paper. 

R1: Based on this response, It seems the paper is trying to tackle the negative transferability problem. While this is an important problem, the authors need to discuss that in the related works, for example [1]. However, in a multi-view learning problem, the main issue is that each view includes some features which can help the classification goal. In other words, part of the features in all views are noisy. At least, you need to add this experiment in my point of view.

R10: [2,3,4] are a few examples that used DP for classification purposes. When one of the main arguments of the paper is adding uncertainty estimation using DP, discussing other related papers are important. In some cases, you can also compare with them.  Besides that, the authors can compare with HDP similar to what they did for the CCA-based methods.

It would be great if the authors could report the classification results based on only using one view for Figure 4 as a baseline. The authors might also want to check the paper [5]. 

[1] Bayesian multi-domain learning for cancer subtype discovery, NeurIPS 2018.

[2] Multi-Task Learning for Classification with Dirichlet Process Priors, Journal of Machine Learning Research 2007.

[3] Dirichlet-based Gaussian Processes for Large-scale Calibrated Classification, NeurIPS 2018.

[4] Factorial Multi-Task Learning : A Bayesian Nonparametric Approach, ICML 2013.

[5] Hierarchical Optimal Transport for Robust Multi-View Learning, NeurIPS 2020.

----------- UPDATE II ----------
I have read this paper again and went through the author's responses. While I appreciate the authors for their responses and believe there is some novelty in including uncertainty quantification, I'm still not confident with their experimental designs and literature review.

Based on my understanding, this method is not a multi-view learning in a classical way. It is better to say that the proposed method is somehow an ensemble approach. Going through the model, the paper deals with each view separately and then tries to combine the classification results of each view by weighting them. There is some novelty here, where the weighting is adapting based on the uncertainty. However, in a classical multi-view, the problem is that you have different views, where features of each view are randomly noisy. So, the goal is how one can combine information from different views to improve the classification results. Still, I cannot see that. 

In the experiments, the authors make N views to be noisy, and the other N(+1) views are clean. Then they are trying to classify each view (somehow) separately and because they are reducing the weight of noisy views, their performance (slightly) will be improved. For example, please check table 1. For instance, the Handwritten dataset, the performance of DE, the only ensemble model as the baselines, performed almost similar 99.79 vs 99.97 ( 98.30 vs 98.51). This improvement can be due to tuning the other methods, and the authors did discuss this in neither the main text nor the supplement.

In Figure 4, when the noise is small, the different models' performances are almost the same. Also, \sigma = 10^9 is not meaningful and with some sort of variance comparisons, one can find that that view is corrupted.

The authors did not address if the method can work in real/harder situations when all features and views are noisy. The authors responded they do not have any restriction on that; however, they did not show either empirically or theoretically that the proposed method can handle this situation more suitable. I agree that the authors have tried on the real dataset, however, to see the performance comparison, you somehow need to randomly corrupt different features on different views.

Since the method does seem to be a multi-view ensemble learning, I would have rather expected to try a more common approach for achieving the same goal first, for example to use one of the available Bayesian single-view methods for each view and they combine the results and see how the results look like.

The authors claim that the negative transfer effect is not related here, which I can't entirely agree with. This term has been used in multi-task learning as well. The reason that no one uses this term in multi-view learning is that this is not a real scenario in multi-view learning. One view is degrading the classification performance because some views are noisy. If we do not use those very noisy views, the performance should be improved.

I also want to point out that their authors show that removing each view can degrade their model's performance. And this is desire, especially when the number of views is small, as having three views means three models in their ensemble architecture. It is not clear that this improvement is due to better information sharing or more complex methods.

Regarding the CCA-based method, one can use (Optimal) Bayesian Classification on learned space to get the uncertainty as well. Although, I agree that the proposed method is somehow an end-to-end learning method. 

I would suggest the authors somehow re-organize/write the paper as a multi-view ensemble learning method and compare it with those multi-view ensemble methods as the main focus of the paper. You also need to discuss [5] as it is very related to your work. Although, it is a Bayesian but not an ensemble method. 
review_title: evidence level based multi-view classification

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes a reliable multi-view classification mechanism equipped with uncertainty, called Trusted Multi-View Classification. The goal is to dynamically assess the quality of different views for different samples to provide reliable uncertainty estimation. The idea is clear and well-motivated. The authors perform empirical studies on diverse datasets to conclude that the proposed algorithm is effective, robust and reliable. 

Strengths: 
+ It is interesting to conduct multi-view classification by dynamically integrating different views at an evidence level, which provides a novel and flexible way in multi-view classification.
+ The way of using Dempster-Shafer theory for integrating evidences in a unified and learnable framework is quite neat. 
+ The paper is well-written and clearly presented. 
+ Strong and sufficient empirical results are provided.

Minor comments: 
+ It is reasonable to use the subjective logic theory to directly model uncertainty, however, beyond the advantages mentioned, it will be better if intuitive comparison or discussion between using u (overall uncertainty) and softmax scores in multi-view learning could be provided. 
+ The authors could present failure cases which may be associated with high uncertainties (ideally).
+ For the results (Table 2) of the end-to-end experiments, are the data used original or being corrupted manually? 


Overall, the paper is very well motivated and easy to follow. The assumptions and decisions are well supported. The stepwise experiments are helpful and provide good insights to evaluate the proposed algorithm. The method seems to be of great potential in real-world (cost-sensitive) applications. 

review_title:  A reliable multi-view classification mechanism equipped with uncertainty

Meta-review: The paper introduces a new idea for multi-view classification: using a Dirichlet distribution over the views to model uncertainty.

The paper appears to be clear, well written and sound.
Also, the experimental comparison is thorough.

The authors have given pertinent responses to the reviewers' questions, including w.r.t comparing against Bayesian/deep CCA in terms of accuracy.

Overall, this is a good paper.

Decision: Accept (Poster)