Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper describes a method for model neural for neural processes considering  translation-equivariant embeddings.

The paper seems to be quite specific topic. Maybe, the author could add more empirical results to it to show the impact on translation-equivariant examples. The theoretical claims seem to be valid. So the question is a bit open what are the applications. The empirical results are also narrow as there is not much other competitive work. The results seem to be increment extension to previous work.  

The work looks solid to me, currently I am probably not able to appreciate and judge  relevance to its full extend. I would judge, it is more of interest to view specific people working on this - maybe, the authors could for the final version make this more clear.   

The questions that should be more addressed maybe is also the  applications - why is this relevant and how does it improve your specific cases. Why do we want to develop this. State of the art is quite relative if authors come from a quit narrow area which not much papers on the topic and data sets. 

One of the main points of the paper did not get clear how does translation-equivariant helps to solve or improve the empirical results. Could you add some examples where this improves results. 

I remain ambivalent. It seems to be solid work with not much convincing applications and somewhat incremental. Maybe the authors might address this in their introduction more. The motivation remains unclear to me and hence difficult to judge its potential and impact.   


review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: -- Summary 

This paper considers the problem of developing neural processes which
are translation-equivariant. The authors derive a necessary and sufficient
functional form that the neural process \Phi function must exhibit
in order to be permutation invariant, continuous and translation
equivariant.

Using the derived functional form, the authors construct a
translation-equivariant neural process, the convolutional conditional
neural process.

Results in several experimental settings are given: 1d synthetic
experiments, an astronomical time-series modelling experiment, a
sim2real experiment, and several image completion experiments.  All
the experiments show performance improvements over the AttnCNP, the
main baseline tested against. In the astronomy setting the authors
test against the winning kaggle entry, against which they get better
log likelihood. The authors give several qualitative experiments,
including image completion tasks from a small number of pixels.

Proofs of all the theorems and full details of all the experiments
are given in the appendix, along with ablations of the model.


-- Review

Overall I found this paper very impressive. It is clear how the theoretical
results motivate the choice of architecture. The fact that Theorem 1
completely characterises the design of all translation-equivariant
neural processes is a remarkable result which precisely specifies the
degrees of freedom available when constructing a convolutional NP.

The implementation gives state of the art results against
the AttnCNP while using fewer parameters on a variety of tasks. The image
completion tasks are impressive.

It seems that the authors close an open question posed in (Zaheer 2017)
regarding how to form embeddings of sets of varying size by embedding
the sets into an RKHS instead of a finite-dimensional space. This in itself
is an interesting idea, and I am interested to see how this embedding method
be applied outside of the CNP framework.

The experimental results are comprehensive and diverse, showing good
performance on both toy examples and more real-world problems. The ablations
and qualitative comparisons in the appendix are helpful in showing where
the ConvCNP outperforms the AttnCNP.

My main criticism of the work is that it's very dense, requiring a few
passes to really grasp the theoretical contribution and the concrete
architecture used in the ConvCNP. I would recommend enlarging figure 1
(b), which is illuminating but quite cluttered due to the small
size. Perhaps the section on multiplicity could be moved to the
appendix to make space as it seems for all real-world datasets the
multiplicity would be equal to 1. 


Misc Comments

- It would be good to have a brief discussion of why the ConvCNPPXL performs
very badly on the ZSMM task, while being the best performing method in all
of the other tasks. I couldn't find such a discussion.
- Did the authors try emitting a 36-dimensional joint covariance matrix over the
six-dimensional output in the plasticc experiment?
- In the synthetic experiments, for the EQ and weak periodic kernels it would
be nice to see the `ground truth' log-likelihood given by the actual GP,
just to have some idea of what the upper bound of LL could be.
- In appendix C.2 Figure 6, what is the difference between the `true function' and the
`Ground Truth GP'? I thought the true function was a gp...
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The paper introduces ConvCNP, a new member of the neural process(NP) family that models translational equivariance in the data, which uses convolutions and stationary kernels to aggregate the context data into a functional representation.

This problem is well-motivated as there are various domains where such an inductive bias is desirable, such as spatio-temporal data and images, and will help especially with predictions for out-of-distribution tasks. This inductive bias was never built into NPs, and it remained unanswered whether the NP can learn such a behaviour. This paper shows that the answer is negative and that one needs to make modifications to create such inductive bias.

The architecture of the ConvCNP is motivated by theory that completely characterises the set of translation equivariant functions Phi that maps sets of (x,y) pairs to bounded continuous functions that map x to y (disclaimer: I haven’t read through the proof in the appendix, so will not make any claims on its correctness). Theorem 1 defines the set of such functions using rho, phi and psi, and the choices for each on on-the-grid data and off-the-grid data are listed in Section 4. There are ablation studies in Appendix D.4 that justify the choices.

Overall the paper is very well-written and clear for the most part, with helpful pseudo-code and well-laid out quantitative + qualitative results, and a very detailed appendix that allows replicating the setup. The evaluation is extensive, and the results are significant.
    - The results on 1D synthetic data show a noticeable improvement of the ConvCNP compared to the AttnCNP, with improved interpolation as well as accurate extrapolation for the weakly periodic function. I do think however that a more competitive baseline for AttnCNP would have been to parameterise the logits of the attention weights as a periodic function with learnable length scale (e.g. stationary periodic kernel), since this is another way of building in periodicity into the model. Arguably this is more explicit and restrictive than the translational equivariance built into ConvCNP, but would have made for a more interesting comparison.
    - Having said that, I like how the evaluation was performed on a variety of stochastic processes - previous literature only used GP + EQ kernel, but here more challenging non-smooth functions such as GP + Matern kernels and sawtooth functions are explored - and it’s very convincing to see the outstanding performance of ConvCNPs here.
    - It’s also nice to see results on regression tasks on real data (sections 5.2, 5.3), which was never explored in the NP literature as far as I know. 5.2 shows that ConvCNPs can be competitive against other methods that model stochastic processes, and 5.3 shows an instance of where ConvCNPs do a reasonable job whereas (Attn)CNP fails.
    - The results on images is also extensive, covering 6 different datasets (including the 2 zero shot tasks), and show convincing qualitative and quantitative results. The zero shot tasks are nice examples that explicitly show the consequences of not being able to model translation equivariance in more realistic images composed of multiple objects/faces.

I have several comments/questions regarding the disccusion & related work section:
    - One link that might be worth pointing out regarding functional representation of context is that ANP (or AttenCNP) can also be seen as giving a functional representations of the context; the ANP computes a target-specific representation of the context, which can be seen as a function of the target inputs.
    - I think it’s incorrect to say that latent-variable extensions enforce consistency. Even with the latent variable, if the encoder is seen as part of the model, then the NP isn’t consistent (pointed out in the last paragraph of section 2.1 in the ANP paper). So there still are issues regarding AR sampling. There does however seem to exist variants of NPs that satisfy consistency e.g. https://arxiv.org/abs/1906.08324
    - What is preventing the incorporation of a latent variable in the ConvCNP? Is this just something that can be easily done but you haven’t tried, or do you see any non-trivial issues that arise when doing so e.g. maintaining translation equivariance?

Other minor comments:
    - Are there any guidelines on choice of filter size of CNN in the image case? E.g. have you chosen the filter size of ConvCNP such that the receptive field is smaller than the image, whereas it’s bigger for ConvCNPXL? It’s not clear why having a bigger receptive field allows to capture non-stationarity, and it would be helpful to expand on that, perhaps in the appendix.
    - Also it’d help for the sake of clarity to explain why AttnCNP uses significantly more memory than ConvCNP, i.e. because memory for self-attention is O(N^2) where N=HW is the number of inputs, whereas for convolutions it’s O(HW).
    - I think it’d also help to state explicitly in the body that AttnCNP is ANP without the latent path when it is introduced.
    - typos: first paragraph of Section 2: Z_M <- Z_m (twice), finitely <- infinitely, Appendix D.1: separabe <- separable

Overall, I think this is a very strong submission and I vote for its acceptance.
review_title: Official Blind Review #2

Meta-review: This paper presents Convolutional Conditional Neural Process (ConvCNP), a new member of the neural process family that models translation equivariance. Current models must learn translation equivariance from the data, and the authors show that ConvCNP can learn this as part of the model, which is much more generalisable and efficient. They evaluate the ConvCNP on several benchmarks, including an astronomical time-series modelling experiment, a sim2real experiment, and several image completion experiments and show excellent results. The authors wrote extensive responses the the reviewers, uploading a revised version of the paper, and there was some further discussion. This is a strong paper worthy of inclusion in ICLR and could have a large impact on many fields in ML/AI. 
Decision: Accept (Talk)