Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a “Complete Bayesian Model” that combines the benefits of Parametric Bayesian Models and Evidential Bayesian Models. To this end, the authors propose a technique to learn an input-dependent prior. The experiments showcase the efficacy of the method compared to the baselines.
main_review: Strengths: 

 - The idea is interesting and well motivated.
 - The experimental results highlight the improvement over the baselines in terms of a series of metrics. 
 - The paper is generally well-written.

Comments:
 
 - I believe the paper would benefit from experiments on a more diverse set of  (not necessarily very large) architectures (e.g., ResNet20).
 - It would be interesting to see results on real-world data beyond images, e.g.,  IMDB sentiment classification.
 - I appreciate the illustrative examples in 1-D and 2-D. It would be interesting to see a similar experiment on a small-scale image dataset (e.g., MNIST).
summary_of_the_review: Overall, this is an interesting work. I lean towards acceptance, especially if the comments that were raised are addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes Evidential Turing Process which is a combination of Neural Turing Machines and Neural Processes. The authors theoretically motivate and empirically demonstrate that their proposed method leads to better in-distribution uncertainty estimation as well as improved out-of-distribution detection.
main_review: Even though I am far from an expert in the relevant literature, I find this paper very well written and was able to follow most of the presentation. It clearly introduces the relevant concepts and explains the underlying motivation for the proposed methods. 

I find the experimental section in the main text a bit brief. I personally find this justifiable as allows the manuscript to explain and motivate the proposed method in sufficient detail. I do want to encourage the authors to be a bit more restraint with marking the best method by bold letters in Table 1. If two or methods lead to the same results within one (or even three) standard deviations, one should either mark all of these methods as performing best or none of them. 

A few minor suggestions:
-Equation (1) ends with both a full stop and comma.
-Equation (4) misses a "(" in the subscript of the expectation value



summary_of_the_review: I find the paper very well written. The proposed method appears to be theoretically well-motivated and, empirically, leads to improved robustness. 

I want to stress however that I have only a very limited overview of the relevant field of research and I can thus not confidently state a definite opinion, especially concerning the novelty of the manuscript.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces the category of complete Bayesian models, with the goal of providing both in-domain uncertainty quantification (that is, calibration) and and out-of-domain detection. Concretely, the authors develop the evidential Turing process as an implementation of complete Bayesian model. The approach is experimentally validated by experiments on FMNIST, CIFAR10 and SVHN, showing that the evidential Turing process is competitive on all criteria. 
main_review: This paper has several strengths to acknowledge:
- I really appreciate the top-down approach of starting with general principles, which are progressively transformed into an actual implementation which obtains competitive performance. 
- The category of complete Bayesian models deserves its name of "complete" as it covers a very large range of models (see Section 4).
- Creating a unified approach to uncertainty quantification is a valuable and ambitious goal.

But I am unfortunately not yet convinced by the validation of the main claims of the paper:

*"**Main Hypothesis.** A Bayesian model which is capable of quantifying both uncertainties in isolation, as well as their interaction explicitly is a guiding principle for predictors that can succeed in in-domain calibration and out-of-domain detection simultaneously."*

*"**Our Solution Proposal.** We conjecture that a prior can effectively depend on an input if it receives context information during training time, stores it into an external memory, and retrieves relevant parts of it using the conditioned input example as a query."*

- The experimental validation is undermined by the very small models, self-described as "LeNet5-size", with only a few layers. The reported prediction error seem unreasonably high (15+% on CIFAR10 for instance). Please clarify any misunderstanding concerning this point. 
- There is no experimental validation for the choice of a Turing process. What about the alternative of using a simple neural network predicting $\pi$ from input $x$ to amortize the prior $p(\pi|\theta, x)$?
- Despite a lot of math in the paper, the theoretical justification that is actually relevant seems rather limited. A large part of the equations appear to be somewhat decorative. For instance:
  - As far as I understand, the expressions of variational free energy (4) (5) (9) (13) (17) (19) don’t support any argument of the paper 
  - As a second example, Definition 1, (18) and (26) are not contributions of this paper, but describe already existing components (Turing processes, neural processes, the attention mechanism)


summary_of_the_review: I really appreciate the top-down approach of the paper. Unfortunately, I am neither really convinced by the experimental validation which considers too small models, nor by the theoretical presentation that sometimes feel a bit decorative. Hence my weak reject recommendation, with confidence rated at 3 because I am not familiar with evidential Bayesian models. I am looking forward to discussion with the authors to clarify any misunderstanding.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a complete Bayesian model that combines strengths of parametric and evidential Bayesian models.  An instance of this model is an Evidential Turing Process (ETP) -- a model in which the prior on the global variable that governs the likelihood function is input-dependent, and this dependence is established in a way similar to the Neural Turing Machines. Unlike Neural Processes (NPs), it doesn't need a context set during test time as the context is extracted from the training set and stored in a memory variable. Experiments show that  ETPs are somewhat better than BNNs and evidential NPs.
main_review: I don't think I understood the paper well enough to form a strong opinion about this model. 
What lacks for me is a clear motivation for the following:
1. Why the Complete Bayesian Model is constructed this way. Is this the only plausible way to combine parametric and evidential approaches? 
2. In the evidential DL papers that I've read, it seemed common to use entropies instead of variances to measure the spread of the distributions. Is there a reason for using variances here?
3. I wish there was some more intuition on what the interaction term in Eq.10 is doing. Model uncertainty and data uncertainty are quite easy to grasp: in the evidential DL literature they are usually visualized on a simplex (for Dirichlet priors), which makes them readily understandable. Is there a similar illustration that could explain the interaction term?
4. Also, how important is the interaction term?  I wish there were ablation studies that measure its effect in isolation.
5. I disagree with the approach of verifying the Main Hypothesis by showing that experimentally ETPs work better. These are non-trivial models that probably depend on a number of hyperparameters, so tuning ETP slightly better can have a greater effect on the performance compared to the effect of having a genuinely better model. A proper ablation is thus needed to account for this.
6. Definition 1, in my opinion, needs a better explanation. To me, it seems like any exchangeable process would have these properties. M', as I understand it, is a set of posterior parameters that can be computed from the prior parameters and the observations. What makes these properties so special that they define a Turing Process is unclear to me. 
7. If we take an NP and encode the whole training set as a context, which we then use during test time, would that work? I mean, is it really necessary to have a rather complex memory mechanism of the Neural Turing Process?
8. I am underwhelmed by the experimental results: Table 1 is hard to understand, so it's not obvious why ETP is better. Also, if ECE is an important metric, which I assume is, then it would be best to explain it in the text.   Robustness experiments are pretty weird: I don't know how to interpet them or what was expected. If these experiments are included, then I think more than 4 lines of text are required to explain them. 

Finally, I was surprised by the abrupt ending of the paper: I wish there was some sort of discussion. I can only assume that the paper was submitted in a rather unfinished state. 
        
summary_of_the_review: Given the points above, I think that the paper lacks clarity. As I mentioned earlier, I cannot form a strong opinion before I understand the motivation for the model and the experiments.
review_title: null

Meta-review: It seems the reviewers are in an agreement that the work seems interesting, well motivated, and results are meaningful. The main complaints or issues that remain is the amount of rewriting involved, which might be hard for the reviewers to track, and maybe question regarding the results given for e.g. the choice of architectures for CIFAR-10 making numbers harder to interpret. 

However, I do see that the quality of the manuscript is quite good (and multiple reviewers commented on this), and the idea seems natural to me. I think the results, especially with the addition of CIFAR-100 and IMDB seem sufficient, and given the overall positive feeling of the reviewers over the work with no major concerns, I am happy to recommend this paper for acceptance.
Decision: Accept (Poster)