Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: This paper applied the technique of martingale posterior distributions from Fong et al., 2021 to neural processes (Garnelo et al., 2018). The key idea is to use NP's NLL as the loss function to construct the distribution of the parameter of interest, which, provided observations, has an interpretation of posterior given some assumptions from Fong et al. 2021.
strength_and_weaknesses: This paper is conceptually very interesting but the writing is not on par with the idea. See the section below for more details.
clarity,_quality,_novelty_and_reproducibility: Clarity: the authors used a very specific type of expressions throughout the natural language part of this paper, and I found it difficult to understand without guessing on the way.

What does "each task" correspond to in "here each task in the data stream consists of a meta-training set of input-output pairs and also a meta-validation set"?

Why does "taking the meta-training set as an input" mean? How can a neural network take a dataset as input? I had to guess if it means training a neural net on a meta-training set.

And what does "map the meta-training set into a fixed dimensional global latent variable with a Gaussian posterior approximation" mean here? I guessed that the authors had a mental model of the operation that the neural process method is accomplishing, an abstraction perhaps used more often in functional analysis etc, but that's just not the kind of language that a machine learning researcher would typically use.

The introduction of "future data" was very abrupt. Its meaning was not explained as I've never seen such expression in the Bayesian machine learning literature. It's perhaps only clear to people who might have read Fong et al., 2021.

The entire introduction is not an "introduction". It is a summary of the paper and only made sense to me after reading the rest of this paper. E.g. the view of meta learning for this problem was introduced in section 2.2, and the concept of tasks in section 2.1, and  I think it could be much improved by writing a motivation and some high level overviews of NP and martingale posterior (and why they make sense to be combined for that motivation).

The rest of this work is mostly math expressions and very few explanations in words (I wished there were more explanations for the illustration which is supposed to be intuitive), making it difficult to understand for perhaps most machine learning researchers who specialize more on deep learning. Given that this is a machine learning conference on representation learning, I don't know how well this paper can convey its core ideas to the attendees without having the readers getting into the weed. But writing aside, I think if people read the math descriptions, they would get it.

Quality: I think it's good overall. I didn't spot critical mistakes or logical issues. The authors conducted extensive experiments on multiple tasks and presented the promise, as well as potential limitations.

The notations are well defined and consistent in the paper, except some parts of sec 3. The definitions of the problems on NP, martingale posterior and MPNP became clearer once I accepted the notations and expressions.

Some questions/comments I had while reading:
1. How can Y be iid given f in Eq 2? Each element of Y has a distribution parameterized by its corresponding x. For Eq 1 to hold, independence seems to be enough.
2. how is \sigma_\theta(x) represented in f_{dec}?
3. which sequence of variables in section 2.3 exactly is a martingale?
4. what are the weak conditions for the consistency with Bayesian posterior in sec 2.3? And are those supposed to be satisfied for the proposed method? How practical are these conditions?
5. what's the relation between conformal prediction and martingale posterior? They seem to both rely on how far Z' is from Z?
6. Are the samples from Z\cup Z' iid? If so, why so? If not, why not?
7. c was introduced as a set of indices together with Z_c and Z_t, but in sec 3.1, it's unclear what the relations are for Z', Z_c and the unmentioned Z_t. They seem to overlap but it wasn't explained. By the definition of Z', another guess I had was that Z' is somehow a perturbed version of Z_c or Z. This is super confusing. Please clarify.
8. Explain what exactly is \delta_z in sec 3.1.
9. \pi_N in sec 2.3 was defined to be the martingale posterior of theta. But in sec 3.1, how is this posterior computed exactly? It seems that the distribution of theta was implicitly represented as samples from CNPs. But it's unclear how these samples are generated (or how Z' samples can be generated to ensure the distribution is well defined).
10. Explain all variables in Fig 1. IMHO Fig 1 in the current form produces more confusion than illustration.

For the Bayesian optimization results, I think the authors should add a GP based baseline that uses classic acquisition functions such as expected improvements or GP-UCB or probability of improvements given their popularity. Note that the same problem setup applies to https://arxiv.org/abs/2109.08215 as it can make use of the tasks \tau for pre-training in a similar fashion as Eq 19, except the inner part is NLL for a GP which is straightforward to implement. Similarly, I think it's worth comparing to that pre-trained GP for the 1D regression task.

Novelty: as far as I can tell, this work is novel. It is non trivial to extend NP to the martingale posterior setting.

Reproducibility: code was provided though I didn't run it.
summary_of_the_review: I like this paper but I'm somewhat afraid that it is not going to be well received at ICLR due to its writing style. While I hope the authors could focus more on writing as a way to convey information easily (in addition to compactly), I believe this work should be accepted as it presents an interesting and solid extension of neural processes by adopting the martingale posterior interpretation of uncertainty. This introduces new opportunities in deep learning based uncertainty estimation.


=====After rebuttal=======

I've read the rebuttal and updated the score accordingly. I appreciate the clarifications and the (ongoing) effort to make this paper more readable.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces a novel Neural Process (NP) variant which uses martingale posterior distributions to account for epistemic uncertainty. Martingale posteriors are a recent generalization of Bayesian inference proposed by Fong et al. (2021), in which valid posteriors are specified entirely in terms of the predictive distribution function (rather than relying on the traditional prior-likelihood construction). Martingale Posterior Neural Processes (MPNPs) are compared against existing NP variants on a set of different tasks, and overall outperforming the baselines.
strength_and_weaknesses: ## Strengths

- The paper explores an interesting new direction to account for epistemic uncertainty in NPs.
- Across a range of different tasks MPNPs show competitive empirical performance over baselines.
- The paper may open a new avenue for further improvements of NP methods.
- Background and prior work are clearly laid out.

## Weaknesses

- It would be informative to see how MPNPs scale with higher dimensionality. For example, empirical comparisons on a high-D regression task complementing the 1D one.
- The results of the Lotka-Volterra task would deserve further analysis: Why is BNP/BANP seemingly more apt at dealing with misspecification than MPNPs? My understanding is that model data-mismatch is a problem general to Bayesian inference, i.e., should also affect B(A)NP.
clarity,_quality,_novelty_and_reproducibility: ## Originality

The authors explore a novel direction to increase NP's flexibility in modelling epistemic uncertainty based on the recent work of Fong et al. (2021).

## Clarity

The paper is well-written and easy to follow. As stated above, I appreciated the clear outline of background and prior work. 

## Reproducibility

It seems that sufficient details for reproducibility are provided; I have not reviewed the included code.
summary_of_the_review: The paper provides a novel extension of NPs with strong empirical performance based on martingale posteriors and seems like a valuable contribution to the conference.

Two possible points for improvement are 1) inclusion of higher-D tasks, and 2) further analysis of model-data misspecification.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The proposed method uses the martingale posterior technique in place of Bayesian inference for the first time to construct neural processes, i.e. stochastic processes generated using neural networks. Martingale posterior approach models the predictive distribution straight ahead, by passing the need for an interim approximate Bayesian inference step. The combination of two approaches, neural processes and martingale posterior inference, makes intuitive sense. Its implementation appears to be in order and some experiment results are strong.

strength_and_weaknesses: Strengths:

i) The idea is overall interesting and novel. Application of the martingale posterior idea to neural processes is a rather straightforward and almost literal but anyway useful and suitable combination of two methodologies.

ii) Bayesian optimization results are strong and image completion results are OK.

Weaknesses:

i) The consistent outperformance of BNP/BANP over MPNP/MPANP weakens the central hypothesis of the paper. Why would one expect that the martingale posterior approach to outperform Bayesian inference in that particular setup and why was it not the case?

ii) The comparisons appear to be against relatively old versions of NPs. I wonder how the proposed method compares against more recent versions of NPs than ANPs (2018) and BNPs (2020), for instance Evidential Turing Processes (2022).

https://openreview.net/forum?id=84NMXTHYe-

iii) I find that the adaptation of the MPNP idea to CANP a bit dilutes the main message of the paper. It is after all a heavy pipeline with many components. The contribution of martingale posterior inference to performance would be more directly measurable when one uses a simple pipeline. It is also a bit misleading to have a Fig 1 which describes some architecture details that do not have much relevance to martingale posteriors. It could be shifted to the appendix and replaced by another Fig 1, maybe a plate diagram that gives a gist of the presented inference technique.

iv) It is great that the paper points out the limitations of the presented method, but would be even better if it also gave an educated guess on which properties of the method cause them. For instance the model-data mismatch and the decoder input ignorance. How would the authors explain these seemingly unexpected outcomes?

clarity,_quality,_novelty_and_reproducibility: The paper is written in a precise, technically correct, and easy-to-follow scientific language. It also provides sufficient details to reproduce the experiments.

The proposed idea, using martingale posterior inference for neural processes, is novel, not only as being the first combination of the two ideas but also as being a useful and meaningful combination of them.
summary_of_the_review: This is solid work that presents a novel and interesting idea backed with a sufficient set of experimental results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper develops a novel method for producing predictive uncertainty in neural processes using a Martingale posterior formulation.  This is validated empirically, comparing to existing neural process variants, on a variety of experiments including regression tasks, Bayesian optimization and image completion.  The reviewers all found the proposed approach compelling, technically strong, interesting and novel.  In general, they found the empirical evaluation compelling as well.  As a result, the reviewers were all in agreement on acceptance (10, 8, 8).  The Martingale posterior approach seems well suited to the problem and the reviewers felt that it could provide exciting avenues for future work.  Therefore the recommendation is to accept.
justification_for_why_not_higher_score: I think this could be an oral presentation.  It seems technically strong and novel, and producing stochastic processes with good uncertainty using neural networks is a relatively new and exciting direction.  One reviewer gave a 10 while two stayed at 8.  One of the  reviewers found that the comparisons were to older variants of neural processes and could have had stronger baselines.  I'd say the paper could be a bit technically niche for the general ICLR community.
justification_for_why_not_lower_score: The paper is technically strong and novel, and producing stochastic processes with good uncertainty using neural networks is a relatively new and exciting direction.
Decision: Accept: notable-top-25%