Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper shows that a standard CNP (trained with the diagonal covariance assumption) can be used, at test time, to sample target values capturing their non-diagonal covariance structure. This is done by sampling one target point at a time from the CNP and feeding that predicted target point back into the context set of the CNP. This is termed as running ‘CNP in auto-regressive (AR) mode’.
strength_and_weaknesses: ### Pros

1. Paper adopts the simplest class, CNP, to demonstrate their key idea — thus their main result seems to be free from the nuances of a complex architecture.
2. Paper shows that CNPs in AR mode are a more flexible class than GNP.
3. The paper clearly mentions the limitations such as the lack of consistency under marginalization and permutation. Also, care may be taken when feeding more points into the context set than shown during training. It is good for practitioners to know about these limitations and exercise caution when deploying it. 
4. Paper also proposes how one can trade off speed and expressiveness in CNPs in AR mode by sampling target points block-wise.
5. Empirically, they show that CNP in AR mode outperforms the usual CNP that comes with diagonal covariance over target points. It also tends to do better/comparable to other baselines e.g. GNP or LNP. The fact it does so despite carrying the simplicity of CNPs seems like a good takeaway.

### Weaknesses/Questions

1. I do not fully understand the procedure for obtaining the smooth samples on the bottom part of page 4. 
2. Paper does not analyze how the performance is affected if we auto-regressively sample more target points than shown during training. How quickly does it get worse?
3. Another important question not analyzed is whether TNP (a recent state-of-the-art) trained in diagonal-covariance-mode (i.e. TNP-D) would benefit similarly at test-time to what is shown in this paper for CNPs.
4. When trying to sample $N$ target points in the AR model, we sample the points $1, 2, \ldots, N$ sequentially. It would be good to show whether prediction error worsens going from point $1$ to point $N$ due to the accumulation of prediction errors in the context set.
clarity,_quality,_novelty_and_reproducibility: **Novelty.** The work seems to distinguish itself well from the prior works. It is motivated by the following limitations of the prior works: i) Standard CNPs assume a diagonal covariance among the target points — a limitation of CNP resolved by this work. ii) GNPs can model non-diagonal covariance structure but are restricted to Gaussian processes — a limitation of GNP but not of the proposed approach. iii) Latent-variable NPs (LNP) can also model non-diagonal covariance structure but are trained via an approximation of the likelihood objective — a limitation of LNP but not of the proposed approach.  

**Reproducibility.** The work is reproducible since the CNP code is available and this work simply proposes sampling target points auto-regressively.

**Clarity and Quality.** The writing seems error-free. But given that the idea is simple and perhaps impactful, the writing and the presentation of math seem to make the idea a bit inaccessible to the first-time reader. After my first reading of the paper, I missed catching the key idea (CNP in AR mode) that I was only able to realize after some re-reads.
summary_of_the_review: Several results (e.g., equation 7) rely on mathematical proofs which I think I (the reviewer) do not have the right expertise to review properly. Besides that, I think the paper is simple and clever. But due to my lack of expertise in judging the math and the possibility that there is some paper that already shows some part of what the paper shows (e.g. because it is so simple), I will exercise caution and rate it as 6 with confidence 2.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper presents Autoregressive Conditional Neural Processes, a way to use conditional neural processes in an autoregressive way, by feeding newly sampled points back into the CNP and re-evaluating. The paper vouches that using CNPs in this autoregressive way gives better predictive performance, since there are no independence assumptions. Furthermore, there are no changes that need to be done to the training or architecture, since this process only affects the inference routine.

The paper notes that this process does give up the property of consistency, since the predictions can depend on the different permutations chosen during inference. Also, sampling can be done in parallel using blocks of size N, to trade off conditional dependence and speed.

strength_and_weaknesses: *strengths*
- related work section is comprehensive (maybe also worth including [this](https://arxiv.org/abs/2206.03992))
- paper is overall well written, with thoughtful discussion
- a variety of experimental results

*weaknesses*
- the idea of unrolling autoregressively seems to be already well known at this point, in many recent works, and in some of the works already cited (e.g. esp [[1]](https://arxiv.org/abs/2207.04179) cf C.1, and also [[2]](https://arxiv.org/abs/2102.04426) [[3]](https://arxiv.org/abs/2110.02037) [[4]](https://arxiv.org/abs/2205.13554)). Therefore, I'm not sure that the ideas of this paper are novel enough to be a useful contribution to the community.
  - e.g. [[3]](https://arxiv.org/abs/2110.02037) Section 3.1 has a more in-depth discussion about parallelizing blocks, and presents a dynamic programming method from [[5]](https://arxiv.org/abs/2106.03802) for choosing how to parallelize. It's not in the stochastic process framework, but it does seem very similar.
clarity,_quality,_novelty_and_reproducibility: The clarity is reasonable, and the results look very much reproducible.
The novelty is questionable, as discussed above.
summary_of_the_review: In summary, the paper proposes to augment conditional neural processes with an autoregressive inference procedure, by feeding samples back into the model. This method improves predictive performance without requiring retraining of the model.

Based on my personal opinion, I think this (or similar) techniques have already been used in many recent works, so I'm not sure that it is novel enough to be of interest. However, I'm not against acceptance if other reviewers find the idea novel, since otherwise the paper is well written with reasonable experimentation.


=============

I have read the other reviews and the author's response.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper investigates an autoregressive (AR) evaluation scheme for Conditional Neural Processes (CNP) and its variants. The work is motivated by the fact that standard CNP does not model correlations between prediction targets. Incorporating these correlations is tackled in prior work by extending the *model*, which typically leads to complex training procedures. In contrast, the proposed method changes the *evaluation procedure* by defining an autoregressive, non-factorized joint distribution over targets, while leaving the model/training procedure of CNPs untouched. In a range of experiments, the paper shows that AR-CNPs show strong performance.
strength_and_weaknesses: Strengths:

- The paper studies an interesting and relevant problem.
- The paper is very well written and easy to follow.
- The experimental section is very well fleshed out.

Weaknesses: there are only minor open points that I encourage the authors to elaborate on, cf. below.

clarity,_quality,_novelty_and_reproducibility: **Novelty:** The paper fills a blank spot in the literature: as discussed in the paper, AR schemes have been used for visualization of CNP predictions (e.g., [1]), but there exists no thourough investigation of the theoretical and empirical properties of AR-induced predictive distributions.

**Quality/Clarity:** The paper is very well structured and well written. The visualizations are of high quality and help the reader grasp the concepts. I did not check all the math in depth, but the theoretical exposition appears concise and correct. The empirical evaluation is extensive and contains comparisons on a range of synthetic and interesting real-world datasets with the relevant SOTA methods.

**Reproducibility:** The paper contains detailed descriptions of the datasets and experimental settings. Nevertheless, I have some open questions regarding the evaluation methodology, cf. below. Unfortunately, the github links are not valid, so the source code cannot be accessed.

**Open questions:** I ask the authors to elaborate on the following points:

- As discussed in the paper, the AR scheme leads to predictive distributions that are not permutation invariant w.r.t. set of input locations. I would like to see an extended discussion of this point, as I feel that the order can be crucial for the quality of the AR predictive distribution. In particular, (i) how would, e.g., Fig. 3 look like if the targets are fed into the AR-scheme from *right to left* instead of from *left to right*?, (ii) which order was chosen in the experiments?, (iii) how to choose the order in practice?/how to determine an "optimal" order?, (iv) how sensitive is the predictive performance to the order (in the author’s experience)?
- If I understand Sec. E correctly, you sampled new tasks for each training epoch. While I know that this is a standard approach, I would be interested in a comparison between CNP and AR-CNP when trained on a meta dataset with *finitely* many tasks, as is done, e.g., in [1]. Preferably, I would be interested in results on a range of different meta data set sizes in terms of the number of tasks (results on a simple 1D experiment that can be quickly trained would be enough). I think this is an interesting evaluation, because it sheds light on the question whether a lot of training data is required for the resulting model to exhibit good performance when evaluated using the AR scheme.

[1] Volpp et al., "Bayesian Context Aggregation for Neural Processes", ICLR 2021

summary_of_the_review: The paper studies an interesting way of constructing expressive predictive distributions for CNP models at test time which will be of interest for the community. Both the theoretical and empirical parts are sound and well fleshed out. I have some open questions that I ask the authors to elaborate on, so I might further increase my score after the discussion period. Nevertheless, I think the paper is ready for publication at ICLR, and, thus, I recommend acceptance. 


_________________________________________


Update after rebuttal: I increase my score, cf. my comment below.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes an algorithmic technique to convert CNPs with factorized Gaussian likelihoods with diagonal covariance into autoregressive likelihoods. The technique is conceptually simple: augment the model predictions into the context set of a CNP before making the next prediction. There are some subtle important details here, such as removing the noise in the prediction before augmentation and picking a right order of target points for prediction -- there are no "clean" answers here but the authors do a good job at discussing the potential issues that might arise. There are also theoretical limitations in this scheme, such as the lack of consistency under marginalization and permutation, which have been acknowledged by the authors. Nevertheless, I think there is empirical merit in the scheme, as is also evident by the experiments done by the authors comparing their autoregressive CNPs with the default CNPs. 

One outstanding concern with the current work is the performance of their approach with TNPs (Nguyen&Grover, ICML 2022). TNPs also study both diagonal and autoregressive factorizations of their approach and are the current SOTA in the NP space. While the current work is distinct in converting a diagonal CNP into an autoregressive one post-training, the reviewers are curious if the performance benefits surpass TNPs. This is a valid question from both a practical and conceptual perspective and one that I encourage the authors to study empirically in their final version.


justification_for_why_not_higher_score: (1) Their autoregressive approach sacrifices key theoretical guarantees of standard CNPs. (2) Lack of comparison with TNPs which is the most relevant autoregressive CNP work here.
justification_for_why_not_lower_score: Neat and clever trick that is worth future study. Good experiments even though baselines are limited.
Decision: Accept: poster