Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present a class of neural process models that are able to produce correlated predictions while amenable to exact, simple and scalable maximum likelihood optimization supporting multiple outputs. By using invertible transformations (gaussian copula), the model is able to capture non-Gaussian output distributions. Experiments with artificial and real data (EEG and climate), highlight the predictive ability of the proposed model.
main_review: The authors introduce a relatively straightforward extension for Gaussian neural processes in which both mean and (half of the) covariance functions are specified as neural networks, and the covariance function is either explicitly calculated as an inner product in (7) or as a squared exponential covariance function modulated in magnitude by an auxiliary neural network and calculated using the outputs of a neural network rather than the input data itself (x_t, x_c, y_c). Their multi-output and non-Gaussian strategy follows the modulated kernel and Gaussian copula formulation, respectively. Comprehensive experiments on both artificial and real data demonstrate the advantages of the proposed model over the related, but more computationally expensive, fullConvGNP model. Moreover, on real data, the proposed model outperforms both mean field, convNP and MOGP approximations.

Something that is not discussed in the paper is the setting of the length scale of the squared exponential kernel.

Considering that one of the motivations of the proposed approach is how prohibitive existing approaches are, having estimates of computational cost and/or runtime experiments could be a welcome addition to the paper.
summary_of_the_review: The proposed approach though technically simple relative to existing literature in neural and Gaussian process literature it is well motivated, technically sound and with comprehensive experimental results that support the improved predictive performance claimed by the authors.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: There is a long line of recent interesting work on neural processes, a scalable and more flexible alternative to GPs for performing prediction at a set of test points (x1, ..., xm) given a conditioning set ((x, y)_1, ..., (x, y)_n). This mapping is learned via meta-learning. 

This paper addresses a core issue of the popular conditional neural process: the predictions at each test point are conditionally independent given the conditioning set. This is an inappropriate modeling assumption for many real-world datasets. In response, the authors propose to go beyond a non-diagonal Gaussian to describe the joint distribution. For example, they use some structured Gaussian covariances (linear, kvv) and also a Gaussian copula model. 

They demonstrate both qualitatively and quantitatively that modeling these dependencies improves the performance of the model on a variety of datasets spanning application domains.



main_review: 
When I saw 'Tractable Dependent Predictions' in the title, I assumed that a normalizing flow was being used to capture the joint distribution. This is a modern, flexible family of density estimators for which computing the likelihood is tractable. I was disappointed when I found that these weren't used in the paper. The copula model is very close to a full normalizing flow model. Can you explain what would be necessary to extend your method to a full normalizing flow?

I found the discussion of fullconvgp inadequate, as it seems like the most relevant baseline for capturing these dependencies. It would be very helpful if you included an equation or two. I shouldn't need to dig into the literature to understand the difference between your method and prior work. What exactly is the difference between fullconvgp and convgnp?

I was disappointed that there were no error bars. What sources of variance are you averaging over?

Can you make the analysis of fig 3 more quantitative? Right now, the assertion that the samples are better is too qualitative. Can you check, for example, a q-q plot at a few x axis locations?

I'm curious how your model behaves when the data truly has diagonal covariance. I expect that there are off-diagonal artifacts due to overfitting. Can you run a quick experiment to check this?




summary_of_the_review: I feel that the contribution is fairly incremental and I'm disappointed that they did not consider full normalizing flow models. However, I found the exposition engaging and the experiments thorough.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The manuscript proposes variants of Neural process (NP), which can model correlation in the input (and in the output for multiouput regression). The main idea is to directly parameterize the mean and the covariance functions of a Gaussian predictive via neural networks. The authors also propose to use Copulae to handle non-Gaussian marginal distributions.  
main_review: Strengths 
1. The proposed methods are easy to understand and implement. 
2.  The way of modeling dependencies in the input (or the output) through a covariance function looks simple but indeed improves the performance.
3. Using Copulae, the proposed methods can handle non-Gaussian distributions

Weaknesses 
1. I think the novelty of the proposed methods is not striking. The proposed method looks like a direct implementation of a Gaussian process with deep kernel learning in the meta learning setting (through the form of NP): just including the global representation vector \mathbb{r} into the mean and the covariance functions for a GP.    

2. The descriptions (in Introduction) related the reference [Bruinsma et al., 2021] looks a little bit confusing. First, the name “Gaussian Neural Process” (and the concept of modeling a Gaussian predictive using neural networks) is already introduced in the reference paper. Second, it sounds that the proposed methods are belonging to the same class as the FullConvGNP [Bruinsma et al., 2021] is (due to the same class name “Gaussian Neural Process”). Are the proposed methods also translation equivariant as the FullConvGNP is? If this is true, it is required to include proofs. If not, I think the introduction section should be carefully revised to clearly differentiate the proposed methods from the FullConvGNP [Bruinsma et al., 2021].      

summary_of_the_review: I like that fact that the authors include multiple data sets that can show the effectiveness of modeling dependencies in the input (or the output). However, I think the novelty of the proposed methods is not that impressive. 

Minor comments

In the end of page 1: “CNPs model their respective outputs y_m and y_m’ independently, that is y_m \prep y_m’”. It would be better if the mathematical definition of the symbol “\prep” was included in the manuscript. 

The current version of the manuscript is not printable (on Windows 10). I tried to print the manuscript out from Adobe reader and Chrome several times but failed to do it.
review_title: null

Meta-review: This paper proposes a class of neural processes that lifts the limitations of conditional neural processes (CNPs) and produces dependent/correlated outputs but that, as CNPs, is inherently scalable and it is easy to train via maximum likelihood. The proposed model is extended to multi-output regression and to capture non-Gaussian output distributions. Results are presented on synthetic data, an electroencephalogram dataset and on a climate modeling problem. The paper parameterizes the prediction map as a Gaussian, where the mean and covariance are determined using neural networks. Non-Gaussian prediction maps are obtained using copulas. 

Technically speaking, the reviewers found the approach to be incremental and only marginally significant and I agree with them. Issues such as estimates of computational cost, using fixed lengthscales for the covariances and relationships/using normalizing flows have been addressed by the authors satisfactorily. Empirically, the contribution of the paper is somewhat significant, as it provides similar flexibility to other more computationally expensive processes and more general assumptions than conditional neural processes.
Decision: Accept (Poster)