Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a new uncertainty estimation framework with Gaussian Process for multi-view/modal data. The framework is basically a product-of-expert model. The model is trained via a variational inference algorithm to approximate the multi-view posterior distributions. Experiments on both synthetic and real-worl data are conducted. 
strengths_and_weaknesses: # Strengths


# Weaknesses
1. The paper is hard-to-follow, especially in Sec. 2. [L63] The latent function $f_v$ is not defined. Inferring from the context, the dimension of $f_v$ is $N$, i.e., the number of training samples. However, in many real-world dataset, $N$ can be ten thousand to one million, in which case the covariance is intractable. [L66] The kernel function indicates that the inputs $x_{v,i}$ are real numbers, while in practice the inputs are often high dimensional images or discrete tokens (like captions). [L114] What is the $c$ in $\tilde{y}_{i, c}$? After reading the whole section, the reviewer cannot find the origin of **Gaussian Process** and thinks the term is inappropriate.
2. The experiments are not convictive. There is no metrics for synthetic dataset. The rationality of synthetic dataset needs to be justified. Why can the two moon curves be considered as two views of the samples from the same class? Why are the red and blue classes not symmetric? 

Typos:
[title] Uncertainty Estimation for Multi-View Data
[L69] applyinh -> applying

questions: The whole part of methodolgy needs to be clarified and improved. Please see the problems in Weaknesses.
limitations: The limitations and potential negative societal impact are not explained in the paper.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper is about uncertainty quantification for multi-view inputs, where multiple views are input to a model and a decision (classification or regression) must be made by combining decisions made with each individual input/view.

The authors propose a new kind of Gaussian Process, the Multi-View GP, that works by combining predictions from individual views with a Product of Experts (PoE) and training/estimating the posterior predictive distribution using variationa inference. The main purpose seems to be robustness to noise in some of the views, and out of distribution detection.

The contributions of this work are:
- A concept for uncertainty quantification on multi-view data using Gaussian Processes and Variational Inference.
- The proposed framework improves on the state of the art for multi-view/multi-modality classification under noise corrupted inputs, calibration, and out of distribution detection, on several datasets.
- Multi-view posterior distributions are approximated using variational inference in a theoretically grounded way, and predictions for each view are inversely weighted to their uncertainty, meaning that only the most certain views contribute the most to the final output of the model.
strengths_and_weaknesses: Strengths
- The paper deals with the problem of multi-view/multi-modal inputs to machine learning models, in particular Gaussian Processes, combined with uncertainty quantification. As the authors say, this specific combination (multi-view + uncertainty) is often not well explored. I believe the ideas presented in this paper are novel, in particular the use of Gaussian Processes for multi-view/multi-modal data, and the results show that this combination is robust to noise.
- This work is very well written, the ideas flow very clearly and it was a pleasure to read. I only have some minor comments below.
- The formulation used in the paper makes sense. First there is one Gaussian Process for each input view/modality, and the predictions are combined using Product of Experts. The joint predictive distribution is approximated using variational inference, in a very standard framework by maximizing the ELBO. At prediction time, the different views/modalities are weighted by the negative entropy of the predictive distribution, which gives more weight to more certain views/modalities. The whole formulation is very well theoretically grounded and connected to the literature.
- Overall the baselines selected are appropriate for comparison. I have no doubts about the evaluation, but in Weaknesses I mention some additional baselines that could be a good comparison.
- One experiment uses 6 common multi-view datasets, and adds Gaussian noise to half of the views, simulating a case of corrupt data or sensor failure. In this experiment, the proposed method MGP clearly outperforms all baselines in terms of accuracy as noise is increased, but more notably MGP seems to be very robust to noise in some of the views, with minor drops in accuracy as the noise standard deviation is increased.
- Another experiment tests out of distribution detection in a synthetic multi-view dataset based on CIFAR10-C, where views are different corruptions of the dataset, and SVHN/CIFAR100 are used as out of distribution datasets. In this setup, for the in-distribution data, MGP has the best calibration error, no significant degradation in accuracy, and best accuracy when corrupting half views with Gaussian Noise. For the OOD detection setting, MGP has the best AUC.
- The paper has a very nice and detailed supplementary material with proofs and additional results and details.
- Overall I believe that this paper is well executed, the ideas are novel, and the results show that it is a very good improvement over the state of the art. I believe the paper should be accepted.

Weaknesses
- One major weakness is that the proposed method seems only to be defined for classification, but I believe it is possible to extend it to regression, but the paper does not evaluate or describe the formulation for a regression setting.

~~- I think in terms of baselines, some methods like deep ensembles could also  be trained in a similar way to weight each view prediction with the negative entropy, in order to show if this mechanism is important to improve performance, particularly in noisy views. A deep ensemble could also in theory be combined with Bayes by Backprop and be trained using the Variational approximation, not sure if this would work but the authors could mention if this is viable and why it was not tried.~~

Minor issues
- In Fig 1 and 2, it would be nice to have a proper legend for the colorbar on the right, indicating what exactly each heatmap is representing.
- I think the paper is using the ECE metric without defining it, I am aware it is the Expected Calibration Error, and is clearly defined in the supplementary, but some indication of what it is, like a reference, should be in the main paper.
questions: - How was Product of Experts chosen? [27] shows that there are other options, like Mixture of Experts, and Naive Local Experts, so was there an ablation using these other choices to combine individual views?
- Would it be possible to also use Deep Ensembles (+ some approximate Bayesian NN) instead of Gaussian Processes for each of the views, while keeping the variational posterior distribution?
limitations: The supplementary material contains a nice section on societal impacts, mostly about the approximation to uncertainty which could be misleading, and  to raise awareness of the importance of uncertainty quantification for safety and trustability of AI systems.
 
There are some obvious limitations that could be mentioned or discussed, such as that it provides an approximation (due to approximate GP, variational inference, etc), and that the formulation is made for classification. The authors mention that the weight computation can be suboptimal, which I believe is true.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed Multi-view Gaussian Process (MGP) for classification with multi-view and multi-modal data. The MGP modeled each view of data using Gaussian Process (GP) whose classification results are aggregated using product of expert (PoE). Variational inference (VI) is used for model learning (estimation of posterior distribution of latent function $f_v$) and inference (estimation of class probability given test data). Experiment with synthetic data and multiple real-world dataset shows that proposed method achieves comparable in-distribution classification accuracy and better robustness to noise than other state-of-the-art methods that also considers uncertainty in data. The proposed method also showed better performance in out-of-distribution detection (trained with CIFAR10-C and detect samples from SVHN and CIFAR100). 
strengths_and_weaknesses: ## Strengths

1. The paper extended GP to model multi-view data and utilize PoE for combining results of different views. The overall framework seems to be novel. The adoption of VI provides a principled way to solve the learning and inference problems. Though this is not groundbreaking as it has been adopted for GP in the past such as [14].
2. Empirical evaluation on multiple real-world datasets show that the proposed method is much more robust than other uncertainty-aware methods in robustness to noise and out-of-distribution data.

## Weakness

1. There is no discussion on how uncertainty is actually computed in MGP. In addition, although the authors did provide illustration in Figure 2 on uncertainty values, there is no equivalent illustration or analysis on real datasets. 
2. Although the proposed method to improve the robustness and ability to detect OOD samples of classification model. It comes with a cost of reduced in-domain classification accuracy (at least not as competitive as some other state-of-the-art e.g. DE). It is unclear whether this is a limitation of the classification model (or representation power) or the limitation of the proposed inference framework.
3. The metrics Expected Calibration Error (ECE) is introduced without definition or citation. It is preferred to at least describe the intuition and add citation to the metrics. 
questions: 1. How is the $M$ value determined in Eq. (1) and subsequent experiments?
2. In algorithm 1 step 3, how is $q(f_v)$ computed by Eq. (7)? Is there an analytical solution to the integration?
3. In OOD experiment, how is the three views used for training selected? Can we expect similar results if we attempt training and evaluation with a different choice of views such as in a manner of k-fold cross validation?
4. Can the authors provide some discussion on the computation cost vs. accuracy given Monte Carlo estimation is used for Eq. (14) during inference?
5. In Figure 3, the performance of TMC appears to be comparable to DE (EF) but worse than DE (LF), which is different from what reported in [10], which also had a much wider range of noise magnitude. Can the authors comment on what may contribute to this difference?
6. Typo: Line 69 'applyinh' should be 'applying'
limitations: The author provided a discussion on societal impacts in the supplementary materials. I have no concern on the limitations and potential negative societal impact of the work.
review_title: null

Meta-review: metareview: Two out of three reviewers learn to accept, with one of them championing the paper. The reviewer that’s most critical did not engage with either the authors or with the reviewers / AC during the discussion period. Furthermore, the review itself is pretty sparse in details and justifications on the criticism. Upon looking at the manuscript I agree with the other two positive reviewers that the work is innovative and the idea of combining multiple views under Product of Experts framework is interesting. Using Sparse GPs as an underlying mechanism does address some of the concerns about computational complexity of GPs. Additionally, the experimental evaluation is detailed enough and highlights the potential of the proposed model. Given all the above and including the observation that the manuscript is well-written with a principled inference mechanism, I am recommending the manuscript of acceptance at NeurIPS.
confidence: Certain
Decision: Accept