Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this work generative models using a GP as prior and a deep network as likelihood (GP-DGMs) are considered. In the VAE formalism for inference, the novelty of this paper is located in the encoder: It is sparse and the posterior can be computed even when part of the observations are missing. Sparsity is obtained using inducing inputs and the missing observations are handled through the use of deep sets, i.e. the observations aren't given as a vector, but as a permutation-invariant set of (index, value) pairs.

The idea is  interesting and well executed, and the experimental results are certainly good when compared with the provided baselines. However, there are many unclear details in the experiments. I was left with quite a few unanswered questions:
- In the cases in which no imputation is needed, how does the proposed SGP-VAE compare with a standard encoder (instead of one based on deep sets). I.e., are we paying a price for the ability to handle missing data, or are we getting even better results than with a traditional encoder?
- How does the model compare with prior similar work, such as the GPPVAE? In principle, it should be easy to improve on this baseline.
- How good is the model itself? Here we see results based on amortized inference, but what if we weren't using it? Even though this case would be slow and maybe not able to handle the entire dataset, this can be done in a subset. In general, it is unclear how much is being lost by a) amortized inference; b) sparsity; c) use of deep sets (in this latter case, it could that something is won instead).
- In the Japanese Weather experiment, when the standard VAE is used (with no mean imputation), how are the unobserved values being dealt with?
- In the Japanese Weather experiment, the baselines don't seem very strong. A standard VAE and independent GPs don't seem to be particularly suited to this spatiotemporal weather prediction.
- In general, I'm not sure where the advantage seen in these experiments is coming from. For instance, when compared with published results on the same data (e.g., GPAR on EEG), the SGP-VAE doesn't seem to offer a big advantage. In the other experiments, no strong baselines for that specific  data seem to be provided.
- I would be curious to know in which of these datasets the model itself is superior (if we were to skip the encoder and use, say, MCMC for inference) and in which ones it isn't.

Thanks for your response, clarifying.

review_title: Interesting encoder, missing some  empirical results.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper attempts to enhance the inference in multi-output GPs on previously unobserved data using amortised variational inference. This model is proposed to overcome the shortcomings in current GP-DGMs. The major concerns regarding this paper are summarized as below.

One of the two main contributions in this paper is introducing sparse GP via amortized variational inference in order to replace the time-consuming GP, the novelty of which however seems to be incremental.

The modeling of task dependency should be highlighted. In my understanding, this paper employs a MLP to mix K latent functions in a nonlinear manner through equation 1 for each output. The NN parameters are shared across tasks, which however might be a too strong assumption and thus could deteriorate the performance of multi-task learning.

The comparative results in Table 1 cannot showcase the superiority of GP-VAE in comparison to the state-of-the-art multi-output GP. It performs worse than GPAR in terms of NLL on the EEG dataset, and is comparable to GPAR on the two datasets in terms of SMSE.

review_title: Novelty and comparative study should be highlighted

final_score: rating: 6: Marginally above acceptance threshold
review_content: The model proposed by the paper is a multi-task, multi-output GP, with likelihoods driven by a decoder network. While this model is not new --- same as that of Pearce 2020 as acknowledged by the authors the related work section --- this paper includes the sparse approximation and also handles missing data.

The paper can be made clearer:
1. The paper introduces "partial inference network" in section 2.1. Since this plays a central part in the contribution of the paper, I hope the authors can expand on this. For example, in what way is it "partial" and how is it different from normal inference networks, and what are the challenges. From a GP and Bayesian inference perspective, missing data is no problem, as acknowledge by the authors in section 2.3. Hence, why is it that the normal mechanism fails here? Although section 2.3 is section dedicated to this, it is insufficient. 

2. N and P in the last para of section 2.1 are not defined.

3. The first and third shortcomings of sparse GP frameworks claimed by the authors in the last three sentences of section 2.1 do not really exist in existing sparse approximations to GP. Some clarification is needed here.

4. Some figures accompanying the different approaches in section 2.3 will be very helpful.

5. The section title for section 2 is "spatial-temporal". It is good to lay out what is the spatial and what is the temporal aspect for each of the data sets in section 4. For example, I do not really find the "temporal" dimension in Jura. One way around this is to have a more general section title.

The paper is contributes to the community and the work is correct, so *accept*. It is *not a stronger accept* because I find the "sparse" contribution minimal given the vast literature of sparse approximations to GP; and I find the "missing-data" contribution confusing.







review_title: The paper proposes a multi-task multi-output GP, with sparse approximation and imputation of data, making use of inference networks.

Meta-review: The paper proposes a method for inference in models with GP priors and neural network likelihoods for multi-output modelling, dealing with the problem of scalability and missing data. The paper builds upon previous work on inducing variables for scalability on GP models and inference networks for amortization (reducing the number of parameters to estimate) and dealing with missing data.

There are several concerns about the paper in terms of generality/flexibility of the approach, as the proposed model shares the NN parameters across tasks and the results on the small datasets do not show improvements wrt baseline such as GPAR. The authors’ comments provide somewhat satisfactory replies to these issues. Nonetheless, the major drawback of this paper is its novelty as the ideas on the paper have been explored extensively in the GP literature. Although the authors do make a case for scalability when using inference networks, there are other previous works that perhaps the authors are unaware of, for example, https://arxiv.org/abs/1905.10969   and even more sophisticated inference algorithms than can serve as truly state-of-the-art competing approaches (for example based on stochastic gradient Hamiltonian Monte Carlo, https://arxiv.org/abs/1806.05490). 
Decision: Reject