Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper is an empirical investigation into the role of architecture and objective choices in Neural Process (NP) models when the amount of conditioning data is limited. Specifically, they investigate the question of well-calibrated uncertainty. 

Clarity: The overall quality of the writing is clear, but missing details/explanations leave some claims unjustified and therefore lines of argument difficult to follow.

Originality: Limited -- the paper is mostly an empirical investigation, with the modifications to existing NPs being (a) max-pooling and (b) SIVI on the posterior z’s.

Significance: Seems limited (see more on “Cons” section and “Questions/comments”), though I am wondering whether the authors could have done more to emphasize the main contributions of this work. The paper’s significance for me has been hampered by lack of details and clear takeaway/implications of the results.

Pros: There’s been a lot of work on NPs and their variants, but less so on empirically investigating how and when they work well. There is also a range of systematic empirical evaluations both in the main text and supplement, which I appreciated.

Cons: There are some assumptions/statements that would be beneficial to elaborate upon in the paper. First, calibrated uncertainty is defined to be “high sample diversity for small conditioning sets; and sharp-looking, realistic images for any size of conditioning set.” This statement, introduced early on in the paper without much justification, is a point that the authors repeatedly return to, and I found myself wondering why (especially since NPs are built for prediction/regression, so a lot of the prior work on calibration for classification models should hold here such as (Murphy 1973), (Gneiting et. al 2007), (Kuleshov et al., 2018)). What is the benefit of reasoning about calibration in the generative modeling case (e.g. with Inception Scores)?

Additionally, why should a more flexible approximate posterior be more beneficial for better calibrated uncertainty? (Is this a log-likelihood argument, since the log-likelihood decomposes to calibration + “sharpness”?) More broadly, my biggest problem was that the paper makes claims about improving calibration (e.g. samples are obtained from “well calibrated posteriors” in Figure 6) without formally defining how to evaluate “good calibration,” what it means to have “calibrated uncertainty,” etc. I would appreciate if the authors could clear up any misunderstandings I may have had about the work.

Questions/comments:
- (Heusel et. al 2017) show that the inception score (IS) is not a valid metric for CelebA -- would the authors report FID scores instead?
- Regarding the comment in Section 4 about posterior contraction: aren’t NPs exchangeable by design via the iid latent variable z (Korshunova et. al 2019, among many others)? I thought that invoking (conditional) de Finetti via the iid latent variable is what allows NPs to model (exchangeable) stochastic processes.
- It would be helpful to formally define “posterior contraction” for the reader -- is it referring to a reduction in posterior uncertainty?
- Intuitively, why would sample diversity decrease as the size of the conditioning set grows? For example, if I have a dataset of 10 examples of only cats and dogs and increase its size to 1000 (which say, also includes examples of sheep), shouldn’t that also increase my sample diversity as well?
- Shouldn’t the arrow going from z -> y_i in Figure 2 be reversed?

-------------
UPDATE: I have read the authors' rebuttal and revised draft and have raised my score to a 5.
review_title: contributions not clear

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper aims at increasing the sample diversity of neural processes when the condition set is small, while maintaining visual fidelity. The low-data regime is arguably where neural processes are most interesting, and in that regard the paper is right to turn to this setting. The discussion on how different aggregation functions affect the predictive uncertainty of the neural process is also appreciated, as is the experiment on regressing the size of the condition set based on the latent embedding.

Unfortunately, the experiments section does not paint a clear enough picture. While the experiments show us that the proposed modifications have some benefits, it is not so clear how much each part contributes. Especially the contribution of the SIVI bound is hard to judge. As it stands the paper feels a bit incomplete in this regard. For that reason I cannot recommend accepting the paper at this stage, though I am willing to revise my score based on the authors' response. Specifically, I'd appreciate if the paper could make a clear case for adopting the hierarchical latent variable structure and the SIVI bound, as these add complexity to the method (while the max-aggregator does not).

### Pros

* The paper deals with a relevant issue. Neural processes are most interesting when the condition set is small, and this scenario has so far been largely ignored.
* The discussion on the choice of aggregator is useful, as are the experiments on the variational posterior entropy and the prediction of the context set size.

### Cons

* It is unclear how much each modification (SIVI and max-pooling) contribute. The experimental results compare SIVI+max pooling with NP+max pooling, but SIVI+mean is omitted. It's also notable that NP+max seems to work better than SIVI+max in the CelebA dataset. Some discussion would be helpful here, as I don't see any reason why a hierarchical latent structure should hurt in any case, barring optimization difficulties.
* I am not familiar with SIVI and I don't expect the average reader to be either. I'd appreciate some discussion on the choice of using it.

### Other comments

* The inception score sounds like the mutual information between the class label and the generated image. I expect that stating this would help some readers.
* Perhaps it would help to look at each component in isolation and in different settings, e.g. outside the image domain. I can see how max-pooling might be good for images, while other aggregation methods might have an edge in, e.g. a dataset of robot joint trajectories. Other people have investigated the choice of aggregation method, and reading this work reminded me of work by [Soelch et al.](https://arxiv.org/abs/1903.07348), which might be interesting to the authors.
review_title: This paper proposes to improve neural processes by changing the aggregation operator and using a hierarchical latent structure. These decisions are aimed at improving the performance in the low-data regime.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes an improvement of the standard NP by using a mixture distribution \q_{\phi}, semi-implicit variational inference, and max pooling to capture the multimodel structure of the posterior distribution. Replacing one normal Gaussian distribution with a mixture (of Gaussians, normally) is a widely-adopted idea in latent variable models including NP; the adopted semi-implicit variational inference was originally developed in Yin and Zhou ICML 2018, and no further improvement on this inference method is proposed in this manuscript; max pooling is one of three commonly used pooling methods, i.e., max, min, and mean pooling. using one of them to replace another is simple but the explanation of the reason why max pooling is better is interesting and profound. So, the improvement is weak although it is shown to be effective by the empirical study. More importantly, the authors have investigated the posterior contraction of NP. It is interesting. The relationship between the two parts of the objective function of NP has been discussed related to the posterior contraction, both parts have contributed to the contraction apart from their classical explanation on reconstruction and regularization. To my best knowledge, it is the first work to discuss the posterior contraction of NP. It is a classical property in Bayesian and this link will enable further theoretical analysis for NP. 
review_title: This paper proposes an improvement of the standard NP, and the authors have investigated the posterior contraction of NP. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose an extension to Neural Processes (NPs), where they use max-pooling instead of mean-pooling as the aggregation function and use a mixture distribution with hierarchical VI for the decoder. They show that this slightly improves the predictive likelihoods, but crucially strongly improves the diversity of posterior samples when the conditioning set is small, which they argue is an important feature of such models.

Major comments:
- What is the actual impact of using the SIVI? In most experiments, it looks like NP+max performs just as well.
- What would happen if one would use mean-pooling AND max-pooling and just concatenate the two to yield an aggregated representation? Wouldn't that combine the best of both worlds and the downstream decoder could learn which representation to use for the mean and the variance prediction?
- Could these ideas (SIVI, max-pooling) also be combined with more modern NP architectures (like Attentive NPs or Convolutional NPs)?

Minor comments:
- It is argued that the max-pooling is naturally better at capturing useful information for estimating the posterior variances. But what about the posterior mean? Shouldn't the mean-pooling be better for that?
- In Tab. 1, "NP+max" seems to be the best-performing model. Why is it not shown in Tab. 6?

Summary:
I think the focus on the diversity of posterior samples is very interesting and highlights some important property of these kinds of models. However, given the relative simplicity of the proposed extensions, I feel like they are not studied extensively enough. For the paper to provide a clear value for the community, I think it would be good to extend the experiments to cover the whole combination space of {NP, ANP, SIVI} x {mean-pooling, max-pooling, mean+max-pooling}, so that it becomes clearer what the influence of the different design choices are in combination with each other.
review_title: Interesting idea, but the evaluation is not quite thorough enough

Meta-review: This paper analyzes some design choices for neural processes, paying particular attention to their small-data performance, uncertainty, and posterior contraction.  This is certainly a worthwhile project, and R3 found the analysis interesting, giving the paper a score of 8.  However, R1, R2, and R4 found the experimental validation to be incomplete and insufficient to support the paper's broader recommendations.  As the paper is investigating the various combinations of implementations, I tend to agree with R1, R2, and R4 that this paper---while having some interesting ideas---needs a bit more precision and breadth to its experiments.
Decision: Reject