Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Neural processes are a recent proposed type of flexible stochastic processes that models distributions of functions. The distribution of functions implicitly captured by a global latent variable Z that are fed into neural networks. Unlike NPs, the function-space view of Gaussian processes model the function distribution directly as a Gaussian using mean and covariance functions. This work proposes to combine the strength of the two by using attention-based neural networks to model the conditional mean and covariance functions, getting rid of the global latent variable Z. 
main_review: ## Summary

Neural processes are a recent proposed type of flexible stochastic processes that models distributions of functions. The distribution of functions implicitly captured by a global latent variable Z that are fed into neural networks. Unlike NPs, the function-space view of Gaussian processes model the function distribution directly as a Gaussian using mean and covariance functions. This work proposes to combine the strength of the two by using attention-based neural networks to model the conditional mean and covariance functions, getting rid of the global latent variable Z. 

## Strengths

* Fair amount of work seems to have been devoted to the design of the multi-headed attention architectures and they seem to work very well in the toy examples shown (better than architectures used in previous works including NP and Attentive NP).

## Weaknesses

* The main concern that I have after reading the text is the proposed model construction may not lead to a proper stochastic process. Specifically, the Kolmogorov extension theorem requires that all marginal distributions of the stochastic process to be consistent with each other. However, in the proposed model, I can't see how this could not be violated. How do you ensure that the marginal distributions will match for (f(x2), f(x3)) in the cases where X_T = (x1, x2, x3) and X_T = (x2, x3, x4)?

* The description of Neural Processes in the paper is very confusing and gave me the feeling that the nature of the model is mis-interpreted. The whole background section on NPs are focused on the architecture, or what the authors referred as "deterministic path" and "latent path", without mentioning why these architecture emerges (they are there because we want to do amortized inference). The change of latent variable name $z_T$ to $z_C$ is also problematic as there is only one shared latent variable in the definition of NPs. If you take a look at Garnelo et al. (2018b), there is only $z$. It also seems to me that the evidence lower bound given in (2) is not well-understood that there is an approximation made to the prior so that p(z|context) can be replaced by q(z|context). I strongly encourage the authors to revisit the NP paper and revise the whole submission so that it does not contain misleading information to readers.

* Besides, there seems to be confusion about NPs and conditional NPs. The proposed model are eventually trained in a way similar to conditional NPs (using the conditional distribution), which is different from other NPs trained by the conditional evidence lower bound with approximated priors.

* The figures are generally not very descriptive. It is unclear from Figure 2 how the attention module works (which is the main contribution of the work). In Figure 3, it is confusing to concat $h_c$ and $h_t$ horizontally. Shouldn't it be concatenated in the first axis?

* The empirical results are weak and are missing important baselines. I would expect ConvCNPs and GNPs to work equally well on the 1D regression examples. The proposed method also generates quite a few artifacts in the 2d image experiments. This might be related to the fact that the proposed model may not satisfy the consistency requirements of stochastic processes.

## Overall suggestions

I vote for rejection mainly due to the questionable validity of the proposed model as a stochastic processes, as the parameterization can violate conditions of the Kolmogorov extension theorem.

## Questions and minor points

* In section 5, you mentioned that the Gaussian Neural Processes share similar motivation to your work, but termed their seek of translation equivariance as weaknesses. However, is it also fair to say their construction ensures the model is a valid stochastic process but yours doesn't?
* You may want to mention $X_C \subset X_T$ at the beginning of section 2.1.
* Using \mathrm to wrap "softmax" and "Concat" makes it look better.
* It is better to remove claims like "we can predict DKNP will better adapt to even more change points composed of multiple kernel types than NP variants. 
summary_of_the_review: I vote for rejection mainly due to the questionable validity of the proposed model as a stochastic processes, as the parameterization can violate marginal consistency required by the Kolmogorov extension theorem.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Decoupled Kernel Neural Processes (DKNPs), which is a variant of neural processes first proposed a few years ago. The main contribution is to decouple the process of learning the mean and the covariance functions separately. Empirical evaluations are provided to demonstrate its usefulness. 
main_review: Perhaps it is because of my lack of background on neural processes, but from reading this paper and from a brief scan of the literature, it seems that the major leap forward for neural processes was made by [Kim et al., 2019] with the use of attention. The authors make heavy use of attention in this paper too, and although experiments show that improvements are made upon ANP, I think only time will tell whether this decoupling of mean and covariance functions will prove to be a major advancement in the long term. As I believe is typical with deep learning papers, theoretical results are non-existent, but empirical results are sufficiently extensive for this not to be a drawback. 
However, from my very limited judgment, the proposed method is well-motivated, and has intuitively appealing properties, such as those outlined around equations (5) and (6), and I am leaning towards acceptance. 

As an aside, there are some English errors which I think should be corrected. For example, in the abstract, "we introduce a new neural stochastic processes" should be "we introduce a new class of neural stochastic processes", and just above the start of Section 2, it says "a neural stochastic processes that explicitly learn..." which should also be corrected similarly. 
summary_of_the_review: Despite my lack of knowledge of the relevant literature, I still lean towards acceptance, because I like the idea of decoupling the mean and covariance function and believe this idea may lead to more advancements in the future.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes Decoupled Kernel Neural Processes (DKNPs), a new neural stochastic process, which improves uncertainty estimation by learning a separate mean and kernel function to directly model the covariance between output variables in a data-driven manner. It is demonstrated that DKNPs can improve uncertainty estimation in terms of conditional likelihood and diversity in generated samples in 1-D and 2-D regression tasks, compared to other concurrent NP variants.
main_review: Strengths:

(1) The motivation of this paper is clear and interesting, improving uncertainty estimation for the highly complex data distribution.

(2) The model architecture looks straightforward. 

Weaknesses:

(1) The novelty is limited. In the architecture, the path for generating the mean vector is the deterministic path of ANPs. Learning kernel function by the attention mechanism is a common strategy in many NPs-based papers.  

(2) What does the “decoupled” mean in the paper? The separate mean and variance? Actually, they shouldn’t be decoupled since they share kernel matrices, e.g., K(Xc, Xc) and K(Xc, Xt).

(3) The comparison between the proposed method and baselines is not fair. In the test stage, the proposed model takes all test samples as inputs simultaneously to predict the full covariance matrix. However, other models based on NPs task one batch of test samples as input. Could the author provide the results of the proposed method under the same test setting as other NPs-based methods? 

(4) Mean squared error (MSE) is also an important evaluation of regression tasks. Providing MSE in the experiments will make the results more convincing. 

(5) In Figure 1 (left), 4 and 6, the ground truth (the blue line for mean and the shade for variance) should be added. It will provide a straightforward comparison between each prediction and the ground truth.

(6) I am confused with this statement: “Note that without any context, …, allowing us to view the prior learned by DKNPs, unlike conventional NPs”. What’s the prior of your model? More details are needed to make the paper self-contained.


summary_of_the_review: The motivation is interesting, but the model somewhat lacks novelty.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors proposed a novel extension of the neural process by explicitly and separately modeling the mean and the covariance of output variables via multi-head attention. The general NP has the problem that it doesn't well calibrate the output uncertainty when the data sample size is small. However, DKNP seems to mimic the uncertainty calibration behavior similar to GP which is encouraging. The experimental results look convincing with an extensive comparison over state-of-the-art NP variants. 

main_review: 
I think the paper is well-written with sufficient background introduction for people to follow and understand the difference between DKNP and ANP. The comparison between GP and DKNP is helpful for understanding. I think it is an acceptable paper to the community with a significant contribution.

Some detailed comments:

1. It would be helpful to add some computational analysis. One drawback of GP is its scalability to large-scale datasets. But the GP's benefit is its good uncertainly calibration with small data. Where does DKNP locate in this spectrum, also compared with NPs?

2. One confusing point is in figure 1, the mean is achieved only given the three black context points, right? If not seeing the target points, how does DKNP manage to achieve a periodic mean estimate? There is no corresponding explanation of figure 1 in the main text. 

3. Is permutation invariance also guaranteed?

4. In the original ANP paper, they showed examples of the image completion task with a reasonable diversity. How comes the diversity in figure 9 here is so bad for ANP?




summary_of_the_review: I think the model is novel and has a significant contribution to the community. The connection between DKNP and GP is interesting.
review_title: null

Meta-review: This paper proposes Decoupled Kernel Neural Processes (DKNPs), a new neural stochastic process, which learns a separate mean and kernel function to directly model the covariance between output variables. Numerical experiments on 1-D regression and 2-D image completion are provided.

There  was a concern that the original version of the proposed model was not a valid stochastic process, since the consistency condition of Kolmogorov Extension Theorem might not be satisfied. The authors fixed this by replacing multihead mixed attention (MMA) in the covariance path with multihead cross-attention.
Overall, reviewers find the work interesting, but there remain concerns that the novelty is limited and that  the current work lacks sufficient experimental evaluation.
Decision: Reject